Command checks

Run your own programs on a host as checks, from a file only the host can change.

A command check runs a program on the agent’s host and turns its exit status into up, degraded, or down: replication lag, a queue backlog, or the age of the last backup.

Commands come only from a file on the host. The server cannot add, change, or start them, and receives each check’s name, service, interval, and results, never the command line.

The checks file

The agent reads /etc/kogut-agent/checks.toml when it starts, or the absolute path in KOGUT_CHECKS, which must exist. Without the default file, the agent runs no command checks. Restart the agent after changing the file.

[[check]]
name = "Replication lag on db-2"
service = "postgres"
command = ["/usr/local/bin/replication-lag", "--warn", "30", "--fail", "300"]
every = "30s"
timeout = "5s"

[[check]]
name = "Backups on db-2"
service = "backups"
command = ["/usr/local/bin/backup-age"]
every = "1h"

Key	Default	Meaning
`name`	required	Shown in the panel and in alerts. Unique across kogut, see Names.
`service`	required	The service the check belongs to. A service that does not exist yet is created.
`command`	required	The program and its arguments, as a list. The program is an absolute path or a name on the agent’s `PATH`.
`every`	`1m`	How often the command runs, from `10s` to `1d`.
`timeout`	`10s`	How long a run may take, up to `1m` and no longer than `every`.

A file holds at most 32 checks. Unknown keys are errors.

No shell runs the command: pipes, &&, *, and $VAR reach the program as they are. Put shell code in a script.

When the file has a mistake, the agent does not start:

kogut: /etc/kogut-agent/checks.toml: 5:9: check[0].every: every must be between 10s and 1d
  see https://docs.kogut.dev/agent/command-checks/#the-checks-file

kogut agent check-file checks a file before a restart.

What a command reports

The exit status is the state:

Exit status	State
`0`	Up
`1`	Degraded
`2`	Down
Any other, or killed by a signal	Down, with `exited with status 7` or `killed by a signal`

A run that takes longer than timeout or prints more than 4 KiB is killed with every process it started, and the check is down. A program that cannot start is down with the reason, such as program not found.

The first line of standard output can start with a number, such as 42, -0.5, or 1e3, which is sent with the run. The rest of the line is the run’s message, up to 200 characters. Other lines and standard error are ignored.

This script reports the age of the newest backup:

/usr/local/bin/backup-age

#!/bin/sh
# Degraded after 26 hours without a backup, down after 50.
newest=$(find /srv/backups -name '*.tar.zst' -printf '%T@\n' | sort -n | tail -n 1)
if [ -z "$newest" ]; then
  echo "no backups in /srv/backups"
  exit 2
fi
hours=$(( ($(date +%s) - ${newest%.*}) / 3600 ))
echo "$hours hours since the last backup"
[ "$hours" -ge 50 ] && exit 2
[ "$hours" -ge 26 ] && exit 1
exit 0

In kogut

At startup and hourly, the agent tells the server which command checks its file has. Each becomes a check of its service, run from that host, of kind command. A check removed from the file is removed with its history when the agent restarts. The panel cannot change or delete these checks.

Runs alert and join incidents like any other check. While the agent sends nothing, its command checks are unknown and do not alert; the agent’s own alert covers it. Each run’s number is rolled up per minute, hour, and day and charted on the check’s page.

Names

Check names are unique across kogut, so a file on several hosts needs a name per host, such as Backups on db-2. When another check already has the name, the command check is not added, and the host page names both. A check in kogut.toml always takes the name, even from an existing command check. Rename the command check and restart the agent; a name that becomes free is taken within an hour.

Who can change the file

The agent refuses to start when the file or its directory is writable by its group or by everyone, or belongs to the user the agent runs as. A symbolic link is judged by its target and the target’s directory. Make root the owner:

sudo mkdir -p /etc/kogut-agent
sudo install -m 644 -o root -g root checks.toml /etc/kogut-agent/checks.toml

The agent does not check the programs the file names; keep them owned by root too.

Where commands run

A command runs as the agent, with the agent’s permissions:

With the install script, that is the service’s dynamic user under ProtectSystem=strict, in the agent’s groups such as docker. It can read most of the system but write only to the agent’s state directory and a private /tmp. See Systemd protections.
In the container image, the command runs inside the container, which has no shell or tools. Mount the file at /etc/kogut-agent/checks.toml and the programs it names, or install the agent on the host for scripts.

The command gets an empty standard input and only PATH and LC_ALL=C. The host token and the rest of the agent’s environment never reach it.