# Heartbeats

Detect missed or failed jobs with heartbeat pings.

A heartbeat catches a missed or failed cron job, backup, or worker run. The job calls a
secret URL, and kogut marks the heartbeat down when the ping is late or reports failure.

## Create the heartbeat

1. Open **Add check** on **Overview** or a service's page. Choose **Heartbeat**.
2. Choose a period or a five-field cron schedule. Panel cron schedules use UTC.
3. Set the grace period to cover the job's runtime and any expected delay.

For a cron schedule in local time, set its timezone in `kogut.toml`:

```toml title="kogut.toml"
[[heartbeat]]
name = "Nightly backup"
service = "postgres"
cron = "0 3 * * *"
timezone = "Europe/Warsaw"
grace = "1h"
```

For a periodic worker, use `every` instead of `cron`. Syntax, defaults, and limits are in
the [heartbeat reference](https://docs.kogut.dev/reference/configuration/#heartbeat). A new heartbeat stays
quiet until its first ping.

## Copy the ping URL

1. As an admin, click **Ping URL** in the heartbeat's row on the service's page.
2. Click **Create ping URL** and copy the URL. kogut shows it only once.
3. Store it as a secret in the job's environment, for example as `KOGUT_PING_URL`.

Set the [server's public URL](https://docs.kogut.dev/reference/configuration/#server) first; otherwise the
URL uses the address you opened the panel with. Creating another URL invalidates the old
one. Anyone with the URL can report success or failure, but cannot read data.

Through the API, with a `write` token and the heartbeat's public id from
`GET /api/v1/services/<name>`:

```sh
curl -fsS -X POST "https://kogut.acme.com/api/v1/checks/<id>/ping-url" \
	-H "Authorization: Bearer $KOGUT_TOKEN"
```

The response contains `url` and `replaced`, which is true when an earlier URL existed.

## Keep an existing push URL

Jobs that call a push URL in the form `/api/push/<token>?status=up` can keep it. Set the
token on the heartbeat in `kogut.toml` and change only the host in the job's URL:

```toml title="kogut.toml"
[[heartbeat]]
name = "Nightly backup"
service = "postgres"
every = "1d"
ping_token = "${NIGHTLY_BACKUP_PUSH_TOKEN}"
```

Both `GET` and `POST` work:

- No `status`, an empty one, or `status=up` reports success. Any other value, or more
  than one `status`, reports failure.
- `msg` and `ping` are accepted and ignored.
- An accepted ping returns `200` with `{"ok":true}`; an unknown token returns `404` with
  `{"ok":false,"msg":"..."}`.

The same token also works at `/ping/<token>`, and a panel ping URL works at
`/api/push/<token>`. A heartbeat can have both a `ping_token` and a panel URL; creating a
panel URL replaces only the earlier panel URL. A `ping_token` that another heartbeat's
panel URL uses is an error, and the file is not applied. Removing `ping_token` stops its
token at the next reload.

## Report the job's result

Call the URL after success. Add `/start` to report a start, `/fail` for failure, or an
exit code from `/0` to `/255`, where only `0` is success. Both `GET` and `POST` work.
Accepted pings return `200` with `OK`, unknown tokens `404`. Send pings to kogut's own
address, not to the status page domain.

On a host with the `kogut` binary, [`kogut run`](https://docs.kogut.dev/reference/cli/#kogut-run) sends
`/start`, runs the command, sends its exit code, and exits with it:

```sh
0 3 * * * KOGUT_PING_URL=https://kogut.acme.com/ping/btp_... kogut run -- pg_dump -Fc -f /var/backups/acme.dump acme
```

Give the URL in `KOGUT_PING_URL`, not `--ping`: other users can see command-line
arguments in `ps`.

Elsewhere, use `curl`, and ping only after the command succeeds:

```sh
0 3 * * * /usr/local/bin/backup.sh && curl -fsS -m 10 --retry 5 "$KOGUT_PING_URL"
```

To catch a job that starts but never finishes, send `/start` before the command and its
exit code after:

```sh title="backup.sh"
#!/bin/sh
curl -fsS -m 10 --retry 5 "$KOGUT_PING_URL/start" >/dev/null

pg_dump -Fc acme > /var/backups/acme.dump
code=$?

curl -fsS -m 10 --retry 5 "$KOGUT_PING_URL/$code" >/dev/null
exit "$code"
```

The start ping sets a finish deadline one grace period later. A failed or late heartbeat
goes down without confirmation, and the next successful ping restores it.

### Use a systemd timer

Wrap the service's command with `kogut run`:

```ini title="/etc/systemd/system/backup.service"
[Service]
Type=oneshot
EnvironmentFile=/etc/kogut/backup.env
ExecStart=/usr/local/bin/kogut run -- /usr/local/bin/backup.sh
```

Without the `kogut` binary, use `curl`:

```ini title="/etc/systemd/system/backup.service"
[Service]
Type=oneshot
EnvironmentFile=/etc/kogut/backup.env
ExecStartPre=-/bin/sh -c 'curl -fsS -m 10 --retry 5 "$KOGUT_PING_URL/start"'
ExecStart=/usr/local/bin/backup.sh
ExecStartPost=-/bin/sh -c 'curl -fsS -m 10 --retry 5 "$KOGUT_PING_URL"'
ExecStopPost=/bin/sh -c 'test "$SERVICE_RESULT" = success || curl -fsS -m 10 --retry 5 "$KOGUT_PING_URL/fail"'
```

Put `KOGUT_PING_URL` in `/etc/kogut/backup.env`. The `-` prefix keeps a failed ping from
failing the job.

### Use GitHub Actions

Store the URL as a repository secret and report the result even when earlier steps
fail:

```yaml
- name: Report to kogut
  if: always()
  run: curl -fsS -m 10 --retry 5 "${{ secrets.KOGUT_PING_URL }}/${{ job.status == 'success' && '0' || '1' }}"
```