# Command checks

Run your own programs on a host as checks, from a file only the host can change.

A command check runs a program on the agent's host and turns its exit status into up,
degraded, or down: replication lag, a queue backlog, or the age of the last backup.

Commands come only from a file on the host. The server cannot add, change, or start
them, and receives each check's name, service, interval, and results, never the command
line.

## The checks file

The agent reads `/etc/kogut-agent/checks.toml` when it starts, or the absolute path in
[`KOGUT_CHECKS`](https://docs.kogut.dev/reference/environment-variables/#agent), which must exist. Without
the default file, the agent runs no command checks. Restart the agent after changing the
file.

```toml
[[check]]
name = "Replication lag on db-2"
service = "postgres"
command = ["/usr/local/bin/replication-lag", "--warn", "30", "--fail", "300"]
every = "30s"
timeout = "5s"

[[check]]
name = "Backups on db-2"
service = "backups"
command = ["/usr/local/bin/backup-age"]
every = "1h"
```

| Key | Default | Meaning |
|---|---|---|
| `name` | required | Shown in the panel and in alerts. Unique across kogut, see [Names](#names). |
| `service` | required | The service the check belongs to. A service that does not exist yet is created. |
| `command` | required | The program and its arguments, as a list. The program is an absolute path or a name on the agent's `PATH`. |
| `every` | `1m` | How often the command runs, from `10s` to `1d`. |
| `timeout` | `10s` | How long a run may take, up to `1m` and no longer than `every`. |

A file holds at most 32 checks. Unknown keys are errors.

No shell runs the command: pipes, `&&`, `*`, and `$VAR` reach the program as they are.
Put shell code in a script.

When the file has a mistake, the agent does not start:

```text
kogut: /etc/kogut-agent/checks.toml: 5:9: check[0].every: every must be between 10s and 1d
  see https://docs.kogut.dev/agent/command-checks/#the-checks-file
```

[`kogut agent check-file`](https://docs.kogut.dev/reference/cli/#kogut-agent-check-file) checks a file
before a restart.

## What a command reports

The exit status is the state:

| Exit status | State |
|---|---|
| `0` | Up |
| `1` | Degraded |
| `2` | Down |
| Any other, or killed by a signal | Down, with `exited with status 7` or `killed by a signal` |

A run that takes longer than `timeout` or prints more than 4 KiB is killed with every
process it started, and the check is down. A program that cannot start is down with the
reason, such as `program not found`.

The first line of standard output can start with a number, such as `42`, `-0.5`, or
`1e3`, which is sent with the run. The rest of the line is the run's message, up to 200
characters. Other lines and standard error are ignored.

This script reports the age of the newest backup:

```sh title="/usr/local/bin/backup-age"
#!/bin/sh
# Degraded after 26 hours without a backup, down after 50.
newest=$(find /srv/backups -name '*.tar.zst' -printf '%T@\n' | sort -n | tail -n 1)
if [ -z "$newest" ]; then
  echo "no backups in /srv/backups"
  exit 2
fi
hours=$(( ($(date +%s) - ${newest%.*}) / 3600 ))
echo "$hours hours since the last backup"
[ "$hours" -ge 50 ] && exit 2
[ "$hours" -ge 26 ] && exit 1
exit 0
```

## In kogut

At startup and hourly, the agent tells the server which command checks its file has.
Each becomes a check of its service, run from that host, of kind `command`. A check
removed from the file is removed with its history when the agent restarts. The panel
cannot change or delete these checks.

Runs alert and join incidents like any other check. While the agent sends nothing, its
command checks are unknown and do not alert; the agent's own alert covers it. Each run's
number is rolled up per minute, hour, and day and charted on the check's page.

### Names

Check names are unique across kogut, so a file on several hosts needs a name per host,
such as `Backups on db-2`. When another check already has the name, the command check is
not added, and the host page names both. A check in `kogut.toml` always takes the name,
even from an existing command check. Rename the command check and restart the agent; a
name that becomes free is taken within an hour.

## Who can change the file

The agent refuses to start when the file or its directory is writable by its group or by
everyone, or belongs to the user the agent runs as. A symbolic link is judged by its
target and the target's directory. Make root the owner:

```sh
sudo mkdir -p /etc/kogut-agent
sudo install -m 644 -o root -g root checks.toml /etc/kogut-agent/checks.toml
```

The agent does not check the programs the file names; keep them owned by root too.

## Where commands run

A command runs as the agent, with the agent's permissions:

- With the [install script](https://docs.kogut.dev/agent/installing/), that is the service's dynamic user under
  `ProtectSystem=strict`, in the agent's groups such as `docker`. It can read most of the
  system but write only to the agent's state directory and a private `/tmp`. See
  [Systemd protections](https://docs.kogut.dev/agent/permissions/#systemd-protections).
- In the [container image](https://docs.kogut.dev/agent/docker/), the command runs inside the container, which has
  no shell or tools. Mount the file at `/etc/kogut-agent/checks.toml` and the programs
  it names, or install the agent on the host for scripts.

The command gets an empty standard input and only `PATH` and `LC_ALL=C`. The host token
and the rest of the agent's environment never reach it.