Checks

A check probes a target on a schedule and records its state, response time, and failure reason. Each check belongs to a service. Checks run from the server unless from names an agent host.

Types

HTTPS checks also watch certificate expiry. Heartbeats receive pings from jobs instead. Domain checks run only on the server; HTTP, TCP, DNS, and ping checks also run on agents. Update agents before assigning ping checks or HTTP request headers to them.

Keys, defaults, and limits are in the configuration reference.

Try a check before adding it

kogut check runs one check now, by the same rules, without a server:

kogut check https://api.acme.com/health --expect 'json = "$.db", equals = "ok"'

up  https://api.acme.com/health  200 in 142 ms, certificate expires in 61 days

--expect takes an assertion as written in expect. --name runs a check from the file. For a check that runs from an agent, run kogut check on that host. See kogut check.

Checks in the panel

Admins add checks with Add check on Overview or a service’s page. New or changed checks run immediately, except domain checks, which wait for their daily slot. Deleting a check deletes its history.

Checks marked kogut.toml change only in the file, and command checks only in their host’s checks file. File and panel checks share names; see Shared names.

A check’s page

The page shows:

• The target, state, interval, and any certificate or registration expiry date.
• Uptime over 24 hours, 30 days, and 12 months.
• The latest response time and its 24-hour average, or the latest heartbeat ping or domain run.
• Recent runs with their state, response time, and failure reason. Hovering a blip, a failed run that the next run cleared, shows what else happened at the time, such as “the api container on vps-1 restarted”, and what visitors saw. Blips that recur at the same time of day show as one line; see Recurring blips.
• A response time chart with p50 and p95, or for a command check a chart of its number.
• Failures and changes: failed runs and check or certificate state changes.

Failed runs are kept for the retention period; state changes are kept longer.

HTTP checks

HTTP checks send no body and use HTTP/1.1 with User-Agent: kogut/<version>. GET and HEAD follow up to five redirects; other methods follow none. Redirects from HTTPS to HTTP fail. Checks read at most 1 MiB of the body.

Certificates are validated against kogut’s bundled Mozilla root store, not the system’s. Host lookup uses /etc/hosts, then /etc/resolv.conf. DNS checks query DNS directly, so a hosts-file entry cannot hide a broken record.

Request headers

Request headers can be set only in kogut.toml, and can use environment variables:

[[check]]
name = "Orders API"
service = "api"
url = "https://api.acme.com/v1/orders?limit=1"
headers = { Authorization = "Bearer ${ORDERS_API_TOKEN}" }

For Basic authentication, send the credentials in a header, because a user name or password in the URL is rejected:

headers = { Authorization = "Basic ${BASIC_AUTH}" }

Set BASIC_AUTH to the Base64 encoding of user:password, such as the output of printf '%s' 'user:password' | base64.

Custom headers replace matching built-in ones. Connection and body headers, such as Connection and Content-Length, are rejected. Header values are not logged, shown in the panel, or stored, and redirects to another origin omit them. See the header rules.

DNS checks

A DNS check passes when the name has at least one A or AAAA record. It asks the servers in /etc/resolv.conf on the host that runs it, in order; there is no setting for another resolver. For a home resolver such as Pi-hole, see Recipes.

Ping checks

Each run sends three ICMP echo requests, 250 ms apart. Response time is the average of the replies.

Ping supports only response_time_below assertions. It uses unprivileged ICMP sockets, so the process’s group must be in net.ipv4.ping_group_range; otherwise runs fail with “this system does not allow ping without privileges”. systemd defaults and Docker 20.10 or newer allow every group. Elsewhere:

sudo sysctl -w net.ipv4.ping_group_range="0 2147483647"

Names resolve through /etc/hosts, then DNS. Ping uses the first address and tries the next only if it cannot send requests. A firewall can block ICMP while services still work.

Assertions

All assertions in expect must pass:

[[check]]
name = "API health"
service = "api"
url = "https://api.acme.com/health"
every = "30s"
expect = [
	{ status = "2xx" },
	{ json = "$.db", equals = "ok" },
	{ body_not_contains = "maintenance" },
	{ response_time_below = "800ms", else = "degraded" },
]

The syntax is in the assertion reference.

Without a status assertion, HTTP checks require 2xx. A 429 or 503 that fails the status assertion makes the result degraded and skips body assertions. JSON equality includes the type, so 0 does not match "0". A failed response time assertion marks the result down unless else = "degraded" is set.

Command checks

A command check runs a program on an agent’s host and turns its exit status into the state. It is defined only in the host’s /etc/kogut-agent/checks.toml; kogut.toml, the panel, and the API cannot add, change, or run one. See Command checks.

Failures and scheduling

Failure messages include the observed value or the connection error. Confirmation, recovery, and flapping are in How alerts work.

Checks are spread across their intervals. At most four run against one host at once, and 32 in total. A run that would overlap its predecessor is skipped. A run is stopped 2 s after its timeout. Missed runs are not replayed after a restart. A check without a result for three intervals becomes unknown, and kogut raises one alert for all such checks.