# Checks

Check types, assertions, results, and scheduling.

A check probes a target on a schedule and records its state, response time, and failure
reason. Each check belongs to a service. Checks run from the server unless `from` names
an [agent host](https://docs.kogut.dev/monitoring/internal-services/).

## Types

| Type | Target key | Observation |
|---|---|---|
| HTTP | `url` | Status, body, JSON values, and response time |
| TCP | `tcp` | Whether a port accepts connections |
| DNS | `dns` | Whether a name resolves through DNS |
| Ping | `ping` | ICMP packet loss and round-trip time |
| Domain | `domain` | Registration expiry from the registry |
| Command | the agent's checks file | The exit status, a number, and a message from a program on the host |

HTTPS checks also watch [certificate expiry](https://docs.kogut.dev/monitoring/certificates-and-domains/).
[Heartbeats](https://docs.kogut.dev/monitoring/heartbeats/) receive pings from jobs instead. Domain checks run only on
the server; HTTP, TCP, DNS, and ping checks also run on agents. Update agents before
assigning ping checks or HTTP request headers to them.

Keys, defaults, and limits are in the
[configuration reference](https://docs.kogut.dev/reference/configuration/#check).

## Try a check before adding it

`kogut check` runs one check now, by the same rules, without a server:

```sh
kogut check https://api.acme.com/health --expect 'json = "$.db", equals = "ok"'
```

```text
up  https://api.acme.com/health  200 in 142 ms, certificate expires in 61 days
```

`--expect` takes an assertion as written in `expect`. `--name` runs a check from the
file. For a check that runs `from` an agent, run `kogut check` on that host. See
[`kogut check`](https://docs.kogut.dev/reference/cli/#kogut-check).

## Checks in the panel

Admins add checks with **Add check** on **Overview** or a service's page. New or changed
checks run immediately, except domain checks, which wait for their daily slot. Deleting a
check deletes its history.

Checks marked `kogut.toml` change only in the file, and command checks only in their
host's checks file. File and panel checks share names; see
[Shared names](https://docs.kogut.dev/reference/configuration/#file-handling).

### A check's page

The page shows:

- The target, state, interval, and any certificate or registration expiry date.
- Uptime over 24 hours, 30 days, and 12 months.
- The latest response time and its 24-hour average, or the latest heartbeat ping or
  domain run.
- Recent runs with their state, response time, and failure reason. Hovering a
  [blip](https://docs.kogut.dev/alerts/how-alerts-work/#timing), a failed run that the next run cleared,
  shows what else happened at the time, such as "the api container on vps-1 restarted",
  and what visitors saw. Blips that recur at the same time of day show as one line; see
  [Recurring blips](https://docs.kogut.dev/incidents/correlation/#recurring-blips).
- A response time chart with p50 and p95, or for a command check a chart of its number.
- **Failures and changes**: failed runs and check or certificate state changes.

Failed runs are kept for the [retention period](https://docs.kogut.dev/reference/configuration/#retention);
state changes are kept longer.

## HTTP checks

HTTP checks send no body and use HTTP/1.1 with `User-Agent: kogut/<version>`. `GET` and
`HEAD` follow up to five redirects; other methods follow none. Redirects from HTTPS to
HTTP fail. Checks read at most 1 MiB of the body.

Certificates are validated against kogut's bundled Mozilla root store, not the system's.
Host lookup uses `/etc/hosts`, then `/etc/resolv.conf`. DNS checks query DNS directly, so
a hosts-file entry cannot hide a broken record.

### Request headers

Request headers can be set only in `kogut.toml`, and can use environment variables:

```toml
[[check]]
name = "Orders API"
service = "api"
url = "https://api.acme.com/v1/orders?limit=1"
headers = { Authorization = "Bearer ${ORDERS_API_TOKEN}" }
```

For Basic authentication, send the credentials in a header, because a user name or
password in the URL is rejected:

```toml
headers = { Authorization = "Basic ${BASIC_AUTH}" }
```

Set `BASIC_AUTH` to the Base64 encoding of `user:password`, such as the output of
`printf '%s' 'user:password' | base64`.

Custom headers replace matching built-in ones. Connection and body headers, such as
`Connection` and `Content-Length`, are rejected. Header values are not logged, shown in
the panel, or stored, and redirects to another origin omit them. See the
[header rules](https://docs.kogut.dev/reference/configuration/#headers).

## DNS checks

A DNS check passes when the name has at least one A or AAAA record. It asks the servers in
`/etc/resolv.conf` on the host that runs it, in order; there is no setting for another
resolver. For a home resolver such as Pi-hole, see
[Recipes](https://docs.kogut.dev/monitoring/recipes/#home-dns-through-pi-hole).

## Ping checks

Each run sends three ICMP echo requests, 250 ms apart. Response time is the average of
the replies.

| Replies | Result |
|---|---|
| All three | Up, unless a response time assertion fails |
| Some | Degraded, with the number of missing replies |
| None | Down, with the error class `timeout` |

Ping supports only `response_time_below` assertions. It uses unprivileged ICMP sockets,
so the process's group must be in `net.ipv4.ping_group_range`; otherwise runs fail with
"this system does not allow ping without privileges". systemd defaults and Docker 20.10
or newer allow every group. Elsewhere:

```sh
sudo sysctl -w net.ipv4.ping_group_range="0 2147483647"
```

Names resolve through `/etc/hosts`, then DNS. Ping uses the first address and tries the
next only if it cannot send requests. A firewall can block ICMP while services still
work.

## Assertions

All assertions in `expect` must pass:

```toml
[[check]]
name = "API health"
service = "api"
url = "https://api.acme.com/health"
every = "30s"
expect = [
	{ status = "2xx" },
	{ json = "$.db", equals = "ok" },
	{ body_not_contains = "maintenance" },
	{ response_time_below = "800ms", else = "degraded" },
]
```

The syntax is in the [assertion reference](https://docs.kogut.dev/reference/configuration/#expect).

Without a status assertion, HTTP checks require `2xx`. A `429` or `503` that fails the
status assertion makes the result degraded and skips body assertions. JSON equality
includes the type, so `0` does not match `"0"`. A failed response time assertion marks
the result down unless `else = "degraded"` is set.

## Command checks

A command check runs a program on an agent's host and turns its exit status into the
state. It is defined only in the host's `/etc/kogut-agent/checks.toml`; `kogut.toml`, the
panel, and the API cannot add, change, or run one. See
[Command checks](https://docs.kogut.dev/agent/command-checks/).

## Failures and scheduling

Failure messages include the observed value or the connection error. Confirmation,
recovery, and flapping are in [How alerts work](https://docs.kogut.dev/alerts/how-alerts-work/).

Checks are spread across their intervals. At most four run against one host at once, and
32 in total. A run that would overlap its predecessor is skipped. A run is stopped 2 s
after its timeout. Missed runs are not replayed after a restart. A check without a result
for three intervals becomes unknown, and kogut raises
[one alert](https://docs.kogut.dev/alerts/how-alerts-work/#when-checks-stop-running) for all such checks.