kogutdocs

Monitoring

Heartbeats

Detect missed or failed jobs with heartbeat pings.

A heartbeat catches a missed or failed cron job, backup, or worker run. The job calls a secret URL, and kogut marks the heartbeat down when the ping is late or reports failure.

Create the heartbeat

  1. Open Add check on Overview or a service’s page. Choose Heartbeat.
  2. Choose a period or a five-field cron schedule. Panel cron schedules use UTC.
  3. Set the grace period to cover the job’s runtime and any expected delay.

For a cron schedule in local time, set its timezone in kogut.toml:

kogut.toml
[[heartbeat]]
name = "Nightly backup"
service = "postgres"
cron = "0 3 * * *"
timezone = "Europe/Warsaw"
grace = "1h"

For a periodic worker, use every instead of cron. Syntax, defaults, and limits are in the heartbeat reference. A new heartbeat stays quiet until its first ping.

Copy the ping URL

  1. As an admin, click Ping URL in the heartbeat’s row on the service’s page.
  2. Click Create ping URL and copy the URL. kogut shows it only once.
  3. Store it as a secret in the job’s environment, for example as KOGUT_PING_URL.

Set the server’s public URL first; otherwise the URL uses the address you opened the panel with. Creating another URL invalidates the old one. Anyone with the URL can report success or failure, but cannot read data.

Through the API, with a write token and the heartbeat’s public id from GET /api/v1/services/<name>:

curl -fsS -X POST "https://kogut.acme.com/api/v1/checks/<id>/ping-url" \
	-H "Authorization: Bearer $KOGUT_TOKEN"

The response contains url and replaced, which is true when an earlier URL existed.

Keep an existing push URL

Jobs that call a push URL in the form /api/push/<token>?status=up can keep it. Set the token on the heartbeat in kogut.toml and change only the host in the job’s URL:

kogut.toml
[[heartbeat]]
name = "Nightly backup"
service = "postgres"
every = "1d"
ping_token = "${NIGHTLY_BACKUP_PUSH_TOKEN}"

Both GET and POST work:

  • No status, an empty one, or status=up reports success. Any other value, or more than one status, reports failure.
  • msg and ping are accepted and ignored.
  • An accepted ping returns 200 with {"ok":true}; an unknown token returns 404 with {"ok":false,"msg":"..."}.

The same token also works at /ping/<token>, and a panel ping URL works at /api/push/<token>. A heartbeat can have both a ping_token and a panel URL; creating a panel URL replaces only the earlier panel URL. A ping_token that another heartbeat’s panel URL uses is an error, and the file is not applied. Removing ping_token stops its token at the next reload.

Report the job’s result

Call the URL after success. Add /start to report a start, /fail for failure, or an exit code from /0 to /255, where only 0 is success. Both GET and POST work. Accepted pings return 200 with OK, unknown tokens 404. Send pings to kogut’s own address, not to the status page domain.

On a host with the kogut binary, kogut run sends /start, runs the command, sends its exit code, and exits with it:

0 3 * * * KOGUT_PING_URL=https://kogut.acme.com/ping/btp_... kogut run -- pg_dump -Fc -f /var/backups/acme.dump acme

Give the URL in KOGUT_PING_URL, not --ping: other users can see command-line arguments in ps.

Elsewhere, use curl, and ping only after the command succeeds:

0 3 * * * /usr/local/bin/backup.sh && curl -fsS -m 10 --retry 5 "$KOGUT_PING_URL"

To catch a job that starts but never finishes, send /start before the command and its exit code after:

backup.sh
#!/bin/sh
curl -fsS -m 10 --retry 5 "$KOGUT_PING_URL/start" >/dev/null

pg_dump -Fc acme > /var/backups/acme.dump
code=$?

curl -fsS -m 10 --retry 5 "$KOGUT_PING_URL/$code" >/dev/null
exit "$code"

The start ping sets a finish deadline one grace period later. A failed or late heartbeat goes down without confirmation, and the next successful ping restores it.

Use a systemd timer

Wrap the service’s command with kogut run:

/etc/systemd/system/backup.service
[Service]
Type=oneshot
EnvironmentFile=/etc/kogut/backup.env
ExecStart=/usr/local/bin/kogut run -- /usr/local/bin/backup.sh

Without the kogut binary, use curl:

/etc/systemd/system/backup.service
[Service]
Type=oneshot
EnvironmentFile=/etc/kogut/backup.env
ExecStartPre=-/bin/sh -c 'curl -fsS -m 10 --retry 5 "$KOGUT_PING_URL/start"'
ExecStart=/usr/local/bin/backup.sh
ExecStartPost=-/bin/sh -c 'curl -fsS -m 10 --retry 5 "$KOGUT_PING_URL"'
ExecStopPost=/bin/sh -c 'test "$SERVICE_RESULT" = success || curl -fsS -m 10 --retry 5 "$KOGUT_PING_URL/fail"'

Put KOGUT_PING_URL in /etc/kogut/backup.env. The - prefix keeps a failed ping from failing the job.

Use GitHub Actions

Store the URL as a repository secret and report the result even when earlier steps fail:

- name: Report to kogut
  if: always()
  run: curl -fsS -m 10 --retry 5 "${{ secrets.KOGUT_PING_URL }}/${{ job.status == 'success' && '0' || '1' }}"