Recipes

Hosts named in from need an agent.

A database other services depend on

Check the database port from its host, and declare which services need it:

[[service]]
name = "postgres"
hosts = ["db-1"]

[[service]]
name = "api"
hosts = ["vps-1"]
depends_on = ["postgres"]

[[check]]
name = "Postgres port"
service = "postgres"
tcp = "127.0.0.1:5432"
from = "db-1"

[[check]]
name = "API health"
service = "api"
url = "https://api.acme.com/health"
expect = [{ json = "$.db", equals = "ok" }]

When Postgres is down, the API’s alert joins the Postgres incident, and api appears under Also affected instead of notifying separately. If the API check fails first, the incident moves to Postgres when its check fails too.

The port check passes while Postgres accepts connections. The JSON assertion also catches failed queries if the API’s health endpoint reports "db": "ok" (Assertions).

If both services run on one host and two or more signals measured on that host fail, the host becomes the incident’s root. See grouping and depends_on.

Home DNS through Pi-hole

Put the Pi-hole host in a dns service, look up a public name from it, and check its port by address:

[[service]]
name = "dns"
hosts = ["pihole"]

[[service]]
name = "jellyfin"
hosts = ["nas"]
depends_on = ["dns"]

[[check]]
name = "Pi-hole lookup"
service = "dns"
dns = "example.com"
every = "1m"
from = "pihole"

[[check]]
name = "Pi-hole port"
service = "dns"
tcp = "192.168.1.2:53"

[[check]]
name = "Jellyfin"
service = "jellyfin"
url = "http://jellyfin.lan:8096/health"
from = "nas"

• DNS checks run every 5 min by default; every = "1m" finds an outage sooner.
• The port check uses an address, so it does not need the DNS it checks. It runs from the server, which must be in the home network; otherwise set its from to an agent host that is.
• With depends_on = ["dns"], home services’ alerts join the dns incident when Pi-hole stops. Name-resolution errors show as the tls_or_dns likely cause.
• If the Pi-hole host goes offline, its agent falls silent and the host becomes the incident’s root.

Resolve through Pi-hole alone

A DNS check asks the nameserver lines of /etc/resolv.conf on the host that runs it, in order; the next one answers when one fails or does not respond.

• On the Pi-hole host, /etc/resolv.conf can list Pi-hole alone, as nameserver 127.0.0.1.
• With systemd-resolved, /etc/resolv.conf usually lists only its stub, 127.0.0.53, which forwards to resolved’s own servers (resolvectl dns); those must be Pi-hole alone. The stub caches answers, so the lookup can pass for a while after Pi-hole stops.
• In Docker, the agent reads the container’s /etc/resolv.conf, which with network_mode: host Docker copies from the host’s resolver settings; with systemd-resolved it can list resolved’s servers instead of the stub. The image has no shell, so print the file with docker cp <container>:/etc/resolv.conf - | tar -xO.

Another home host that resolves through Pi-hole alone can run the lookup. Its failing checks count as signals on that host, so with two or more, that host becomes the incident’s root instead of dns.