Alerts
Notification channels
Configure and test incident notifications.
Add a channel in Settings → Notifications, or add [[alert]] to kogut.toml
(all keys). Overview warns when there are fewer
than two channels. Use two providers, so one provider’s outage does not stop all
notifications.
Add a channel in the panel
- As an admin, choose Add channel.
- Enter a name, choose the type, and fill in its connection details.
- Save, then choose Send test. A test sends at once, without retries, one at a time. A failure appears beside the button and marks the channel as failing.
File channels are read-only in the panel. They are named by their
name, or else by type and position, such as ntfy
and ntfy-2; do not use these names for panel channels. A file channel with a panel
channel’s name and the same settings takes it over; see
Move panel objects into the file.
In Edit, an empty secret field keeps the stored secret. Remove the stored access token and Remove the stored signing secret clear optional secrets; clearing the username removes an SMTP password. Delete channel also drops its queued messages.
Panel secrets, including Slack and Discord URLs, are readable in the database and backups,
but the panel and API never return them. Only admins and write tokens see ntfy topic
and webhook URLs; viewers and read tokens see names, types, and delivery states. To keep
secrets out of the database, use ${ENV_NAME} in kogut.toml.
ntfy
Subscribe to a hard-to-guess ntfy topic of 1 to 64 letters, digits,
-, or _, and add its URL:
[[alert]]
type = "ntfy"
url = "https://ntfy.sh/acme-ops-7f3k2"For an access-controlled server, add token = "${NTFY_TOKEN}", sent as a bearer token.
Messages are plain text, high priority for problems and default priority for recovery.
Slack
Create an incoming webhook for a Slack channel and store its URL in the environment:
[[alert]]
type = "slack"
url = "${SLACK_WEBHOOK_URL}"Discord
Create a webhook under the channel’s Integrations → Webhooks and store its URL:
[[alert]]
type = "discord"
url = "${DISCORD_WEBHOOK_URL}"Messages use embeds, red for problems and green for recovery, without mentions.
Telegram
-
Send
/newbotto @BotFather and copy the bot token. -
Add the bot to a group or start a chat with it, then send a message.
-
Open
https://api.telegram.org/bot<token>/getUpdatesand copychat.id. Group IDs start with-. -
Add the channel:
kogut.toml [[alert]] type = "telegram" bot_token = "${TELEGRAM_BOT_TOKEN}" chat_id = "-1001234567890"
Use an existing SMTP server:
[[alert]]
type = "email"
smtp = "smtp.acme.com:587"
username = "kogut@acme.com"
password = "${SMTP_PASSWORD}"
from = "kogut@acme.com"
to = ["ops@acme.com", "oncall@acme.com"]Use plain ASCII addresses without display names. TLS and authentication options are in
configuration. kogut checks TLS certificates against
public authorities and sends passwords only after TLS starts. For a local relay, omit
credentials and set tls = "none" and allow_private = true.
One plain-text message goes to all recipients. Recipients the SMTP server refuses, even temporarily, are not retried and appear only in the server log, not as a failing channel.
Webhooks
The endpoint receives JSON POST requests:
[[alert]]
type = "webhook"
url = "https://hooks.acme.com/kogut"
secret = "${KOGUT_WEBHOOK_SECRET}"| Field | Content |
|---|---|
title, body | Title and plain-text message |
type | failure, success for recovery, or info for a test |
kind | incident_opened, incident_updated, incident_reminder, incident_resolved, or incident_test for incident messages and tests |
idempotency_key | incident:<id>:<transition>:<channel>, or test:<time> for a test |
incident | id as a string, url, title, state (open or resolved), opened_at in UTC epoch milliseconds, root service, and root host. ID is null for tests. URL is null without a public URL. Service and host may be null. |
failing | Signals with check, service, since in server-local HH:MM, and error. Empty after resolution. |
summary | Cause sentence and update or reminder lines |
causes | Objects with rule, score from 0 to 1, and text |
affected | Other affected services and signals |
sent_at | When this attempt was sent, in UTC epoch milliseconds. A retry has a newer value. |
Every request carries X-Kogut-Timestamp, the sent_at value. idempotency_key stays
the same across retries, and a timeout can deliver twice, so ignore keys you have
already processed.
Verify the signature
With secret set, each request carries X-Kogut-Signature: sha256=<hex>. It is the
HMAC-SHA256, keyed with the secret, of the X-Kogut-Timestamp value, a ., and the raw
request body: {timestamp}.{body}. Without secret, the signature header is left out.
Before parsing JSON:
- Recompute the signature over the timestamp header and the raw request bytes, and compare it in constant time.
- Reject the request if the timestamp is more than 5 minutes from your clock, so a captured request cannot be replayed later.
- Drop
idempotency_keyvalues you have already processed, which stops repeats within the 5 minutes.
import { createHmac, timingSafeEqual } from "node:crypto";
const TOLERANCE_MS = 5 * 60 * 1000;
export function verify(rawBody, signatureHeader, timestampHeader, secret) {
const timestamp = Number(timestampHeader);
if (!Number.isInteger(timestamp)) return false;
if (Math.abs(Date.now() - timestamp) > TOLERANCE_MS) return false;
const expected =
"sha256=" +
createHmac("sha256", secret).update(`${timestampHeader}.`).update(rawBody).digest("hex");
const a = Buffer.from(expected);
const b = Buffer.from(signatureHeader ?? "");
return a.length === b.length && timingSafeEqual(a, b);
}Apprise
To reach other notification services, use an
Apprise API server. Its notify endpoint reads
kogut’s title, body, and type and ignores the rest.
-
Run Apprise with persistent configuration and a loopback port. It has no sign-in:
docker run -d --name apprise -p 127.0.0.1:8000:8000 -v apprise-config:/config caronc/apprise -
Save your Apprise URLs, separated by commas, under a key:
curl -d 'urls=pover://USER_KEY@APP_TOKEN' http://127.0.0.1:8000/add/kogut -
Point a webhook at that key:
kogut.toml [[alert]] type = "webhook" url = "http://127.0.0.1:8000/notify/kogut" allow_private = true -
Choose Send test to check delivery through Apprise.
If both run in Docker, put them on a shared network, omit -p, and use
http://apprise:8000/notify/kogut; loopback in kogut’s container does not reach Apprise.
Delivery and failures
Messages persist in a database queue. Delivery is ordered per channel. HTTP sends use
JSON POST requests with a 10-second timeout and no redirects. SMTP uses one connection
per message, with 10 s to connect and 60 s for the conversation.
Connection failures, timeouts, HTTP 408, 429, and 5xx, and SMTP 4xx replies retry
after 1, 5, 15, and 60 min, five attempts in total. Other failures are permanent. A
failing channel shows the reason in Settings → Notifications, with secrets hidden, and
is never disabled. An older message about an incident never arrives after a newer one.
Private, loopback, link-local, and other non-public destinations are refused. kogut
checks the address on every request and connects to that address. To reach an internal
server, set allow_private = true on an ntfy, webhook, or email channel; Slack, Discord,
and Telegram do not accept it.