# Recipes

Monitor a database that other services need, and home DNS through Pi-hole.

Hosts named in `from` need an [agent](https://docs.kogut.dev/agent/installing/).

## A database other services depend on

Check the database port from its host, and declare which services need it:

```toml title="kogut.toml"
[[service]]
name = "postgres"
hosts = ["db-1"]

[[service]]
name = "api"
hosts = ["vps-1"]
depends_on = ["postgres"]

[[check]]
name = "Postgres port"
service = "postgres"
tcp = "127.0.0.1:5432"
from = "db-1"

[[check]]
name = "API health"
service = "api"
url = "https://api.acme.com/health"
expect = [{ json = "$.db", equals = "ok" }]
```

When Postgres is down, the API's alert joins the Postgres incident, and `api` appears
under **Also affected** instead of notifying separately. If the API check fails first,
the incident moves to Postgres when its check fails too.

The port check passes while Postgres accepts connections. The JSON assertion also
catches failed queries if the API's health endpoint reports `"db": "ok"`
([Assertions](https://docs.kogut.dev/monitoring/checks/#assertions)).

If both services run on one host and two or more signals measured on that host fail, the
host becomes the incident's root. See [grouping](https://docs.kogut.dev/alerts/how-alerts-work/#grouping)
and [`depends_on`](https://docs.kogut.dev/reference/configuration/#service).

## Home DNS through Pi-hole

Put the Pi-hole host in a `dns` service, look up a public name from it, and check its
port by address:

```toml title="kogut.toml"
[[service]]
name = "dns"
hosts = ["pihole"]

[[service]]
name = "jellyfin"
hosts = ["nas"]
depends_on = ["dns"]

[[check]]
name = "Pi-hole lookup"
service = "dns"
dns = "example.com"
every = "1m"
from = "pihole"

[[check]]
name = "Pi-hole port"
service = "dns"
tcp = "192.168.1.2:53"

[[check]]
name = "Jellyfin"
service = "jellyfin"
url = "http://jellyfin.lan:8096/health"
from = "nas"
```

- DNS checks run every 5 min by default; `every = "1m"` finds an outage sooner.
- The port check uses an address, so it does not need the DNS it checks. It runs from the
  server, which must be in the home network; otherwise set its `from` to an agent host
  that is.
- With `depends_on = ["dns"]`, home services' alerts join the `dns` incident when
  Pi-hole stops. Name-resolution errors show as the `tls_or_dns`
  [likely cause](https://docs.kogut.dev/incidents/correlation/).
- If the Pi-hole host goes offline, its agent falls silent and the host becomes the
  incident's root.

### Resolve through Pi-hole alone

A DNS check asks the `nameserver` lines of `/etc/resolv.conf` on the host that runs it,
in order; the next one answers when one fails or does not respond.

> [!WARNING]
> The host in `from` must resolve through Pi-hole alone. A fallback resolver listed after
> it, such as the router or a public server, answers instead and hides the outage.

- On the Pi-hole host, `/etc/resolv.conf` can list Pi-hole alone, as
  `nameserver 127.0.0.1`.
- With systemd-resolved, `/etc/resolv.conf` usually lists only its stub, `127.0.0.53`,
  which forwards to resolved's own servers (`resolvectl dns`); those must be Pi-hole
  alone. The stub caches answers, so the lookup can pass for a while after Pi-hole stops.
- In Docker, the agent reads the container's `/etc/resolv.conf`, which with
  `network_mode: host` Docker copies from the host's resolver settings; with
  systemd-resolved it can list resolved's servers instead of the stub. The image has no
  shell, so print the file with `docker cp <container>:/etc/resolv.conf - | tar -xO`.

Another home host that resolves through Pi-hole alone can run the lookup. Its failing
checks count as signals on that host, so with two or more, that host becomes the
incident's root instead of `dns`.