# Notification channels

Configure and test incident notifications.

Add a channel in **Settings → Notifications**, or add `[[alert]]` to `kogut.toml`
([all keys](https://docs.kogut.dev/reference/configuration/#alert)). **Overview** warns when there are fewer
than two channels. Use two providers, so one provider's outage does not stop all
notifications.

## Add a channel in the panel

1. As an admin, choose **Add channel**.
2. Enter a name, choose the type, and fill in its connection details.
3. Save, then choose **Send test**. A test sends at once, without retries, one at a time.
   A failure appears beside the button and marks the channel as failing.

File channels are read-only in the panel. They are named by their
[`name`](https://docs.kogut.dev/reference/configuration/#alert), or else by type and position, such as `ntfy`
and `ntfy-2`; do not use these names for panel channels. A file channel with a panel
channel's name and the same settings takes it over; see
[Move panel objects into the file](https://docs.kogut.dev/reference/configuration/#move-panel-objects-into-the-file).

In **Edit**, an empty secret field keeps the stored secret. **Remove the stored access
token** and **Remove the stored signing secret** clear optional secrets; clearing the
username removes an SMTP password. **Delete channel** also drops its queued messages.

Panel secrets, including Slack and Discord URLs, are readable in the database and backups,
but the panel and API never return them. Only admins and `write` tokens see ntfy topic
and webhook URLs; viewers and `read` tokens see names, types, and delivery states. To keep
secrets out of the database, use `${ENV_NAME}` in `kogut.toml`.

## ntfy

Subscribe to a hard-to-guess [ntfy](https://ntfy.sh) topic of 1 to 64 letters, digits,
`-`, or `_`, and add its URL:

```toml title="kogut.toml"
[[alert]]
type = "ntfy"
url = "https://ntfy.sh/acme-ops-7f3k2"
```

For an access-controlled server, add `token = "${NTFY_TOKEN}"`, sent as a bearer token.
Messages are plain text, high priority for problems and default priority for recovery.

## Slack

Create an incoming webhook for a Slack channel and store its URL in the environment:

```toml title="kogut.toml"
[[alert]]
type = "slack"
url = "${SLACK_WEBHOOK_URL}"
```

## Discord

Create a webhook under the channel's **Integrations → Webhooks** and store its URL:

```toml title="kogut.toml"
[[alert]]
type = "discord"
url = "${DISCORD_WEBHOOK_URL}"
```

Messages use embeds, red for problems and green for recovery, without mentions.

## Telegram

1. Send `/newbot` to [@BotFather](https://t.me/BotFather) and copy the bot token.
2. Add the bot to a group or start a chat with it, then send a message.
3. Open `https://api.telegram.org/bot<token>/getUpdates` and copy `chat.id`. Group IDs
   start with `-`.
4. Add the channel:

   ```toml title="kogut.toml"
   [[alert]]
   type = "telegram"
   bot_token = "${TELEGRAM_BOT_TOKEN}"
   chat_id = "-1001234567890"
   ```

## Email

Use an existing SMTP server:

```toml title="kogut.toml"
[[alert]]
type = "email"
smtp = "smtp.acme.com:587"
username = "kogut@acme.com"
password = "${SMTP_PASSWORD}"
from = "kogut@acme.com"
to = ["ops@acme.com", "oncall@acme.com"]
```

Use plain ASCII addresses without display names. TLS and authentication options are in
[configuration](https://docs.kogut.dev/reference/configuration/#alert). kogut checks TLS certificates against
public authorities and sends passwords only after TLS starts. For a local relay, omit
credentials and set `tls = "none"` and `allow_private = true`.

One plain-text message goes to all recipients. Recipients the SMTP server refuses, even
temporarily, are not retried and appear only in the server log, not as a failing channel.

## Webhooks

The endpoint receives JSON `POST` requests:

```toml title="kogut.toml"
[[alert]]
type = "webhook"
url = "https://hooks.acme.com/kogut"
secret = "${KOGUT_WEBHOOK_SECRET}"
```

| Field | Content |
|---|---|
| `title`, `body` | Title and plain-text message |
| `type` | `failure`, `success` for recovery, or `info` for a test |
| `kind` | `incident_opened`, `incident_updated`, `incident_reminder`, `incident_resolved`, or `incident_test` for incident messages and tests |
| `idempotency_key` | `incident:<id>:<transition>:<channel>`, or `test:<time>` for a test |
| `incident` | `id` as a string, `url`, `title`, `state` (`open` or `resolved`), `opened_at` in UTC epoch milliseconds, root `service`, and root `host`. ID is null for tests. URL is null without a public URL. Service and host may be null. |
| `failing` | Signals with `check`, `service`, `since` in server-local `HH:MM`, and `error`. Empty after resolution. |
| `summary` | Cause sentence and update or reminder lines |
| `causes` | Objects with `rule`, `score` from 0 to 1, and `text` |
| `affected` | Other affected services and signals |
| `sent_at` | When this attempt was sent, in UTC epoch milliseconds. A retry has a newer value. |

Every request carries `X-Kogut-Timestamp`, the `sent_at` value. `idempotency_key` stays
the same across retries, and a timeout can deliver twice, so ignore keys you have
already processed.

### Verify the signature

With `secret` set, each request carries `X-Kogut-Signature: sha256=<hex>`. It is the
HMAC-SHA256, keyed with the secret, of the `X-Kogut-Timestamp` value, a `.`, and the raw
request body: `{timestamp}.{body}`. Without `secret`, the signature header is left out.

Before parsing JSON:

1. Recompute the signature over the timestamp header and the raw request bytes, and compare
   it in constant time.
2. Reject the request if the timestamp is more than 5 minutes from your clock, so a
   captured request cannot be replayed later.
3. Drop `idempotency_key` values you have already processed, which stops repeats within
   the 5 minutes.

```js title="verify.js"
import { createHmac, timingSafeEqual } from "node:crypto";

const TOLERANCE_MS = 5 * 60 * 1000;

export function verify(rawBody, signatureHeader, timestampHeader, secret) {
	const timestamp = Number(timestampHeader);
	if (!Number.isInteger(timestamp)) return false;
	if (Math.abs(Date.now() - timestamp) > TOLERANCE_MS) return false;
	const expected =
		"sha256=" +
		createHmac("sha256", secret).update(`${timestampHeader}.`).update(rawBody).digest("hex");
	const a = Buffer.from(expected);
	const b = Buffer.from(signatureHeader ?? "");
	return a.length === b.length && timingSafeEqual(a, b);
}
```

### Apprise

To reach other notification services, use an
[Apprise API](https://github.com/caronc/apprise-api) server. Its notify endpoint reads
kogut's `title`, `body`, and `type` and ignores the rest.

1. Run Apprise with persistent configuration and a loopback port. It has no sign-in:

   ```sh
   docker run -d --name apprise -p 127.0.0.1:8000:8000 -v apprise-config:/config caronc/apprise
   ```

2. Save your [Apprise URLs](https://github.com/caronc/apprise/wiki), separated by commas,
   under a key:

   ```sh
   curl -d 'urls=pover://USER_KEY@APP_TOKEN' http://127.0.0.1:8000/add/kogut
   ```

3. Point a webhook at that key:

   ```toml title="kogut.toml"
   [[alert]]
   type = "webhook"
   url = "http://127.0.0.1:8000/notify/kogut"
   allow_private = true
   ```

4. Choose **Send test** to check delivery through Apprise.

If both run in Docker, put them on a shared network, omit `-p`, and use
`http://apprise:8000/notify/kogut`; loopback in kogut's container does not reach Apprise.

## Delivery and failures

Messages persist in a database queue. Delivery is ordered per channel. HTTP sends use
JSON `POST` requests with a 10-second timeout and no redirects. SMTP uses one connection
per message, with 10 s to connect and 60 s for the conversation.

Connection failures, timeouts, HTTP `408`, `429`, and `5xx`, and SMTP `4xx` replies retry
after 1, 5, 15, and 60 min, five attempts in total. Other failures are permanent. A
failing channel shows the reason in **Settings → Notifications**, with secrets hidden, and
is never disabled. An older message about an incident never arrives after a newer one.

Private, loopback, link-local, and other non-public destinations are refused. kogut
checks the address on every request and connects to that address. To reach an internal
server, set `allow_private = true` on an ntfy, webhook, or email channel; Slack, Discord,
and Telegram do not accept it.