# How alerts work How signals become incidents and when kogut sends notifications. kogut groups related failures into [incidents](https://docs.kogut.dev/incidents/correlation/) and notifies about the incident, not each check. ## States Checks, heartbeats, certificates, host thresholds, [containers](#containers), and agents produce signals: | State | Meaning | |---|---| | Up | Working as expected | | Degraded | Working, but slow, rate-limited, or over a soft limit | | Down | Failing | | Unknown | Not enough information to judge, such as a silent agent or a check that has not run | Degraded and down signals open alerts. Unknown never notifies. States and counters survive restarts, so a restart does not repeat an alert. ## Grouping A new alert joins an open incident with activity in the last 10 min on the same service, a directly related service (one dependency away), or the same host. If a host's agent is silent for 2 min or two or more signals on that host fail, the host becomes the incident's root. Otherwise, the deepest failing dependency becomes the root. Services that depend on it appear under **Also affected** instead of notifying separately. ## Timing These rules are fixed, except for the host limits, which are defaults: - Checks need two consecutive failures. The second run follows after 30 s, or the check interval if shorter. Checks, certificates, host thresholds, and containers recover after two successes and 3 min since the first success. - A single check failure that the second run clears is a blip. It sends no message and opens no alert. The timeline records it as a `blip` event with the failure and its [likely causes](https://docs.kogut.dev/incidents/correlation/), also shown on hover over the failed run on the check page. Each check records at most one blip in 10 min, and all checks together at most 30 in an hour. Blips of checks that need the internet are not recorded while kogut's [connection](#when-kogut-loses-its-internet-connection) is in doubt: while a round of probes fails, or when one failed within a minute before. - Heartbeats, certificates, host thresholds, and agents need one failure. Heartbeats and agents recover after one success. Heartbeats cannot alert before their first ping. - HTTP 429 and 503 that fail a status assertion mean degraded. Certificates become degraded within 14 days of expiry and down within 7 days. - CPU and memory above 90%, or swap above 50%, must persist for 5 min. A disk above 90% degrades immediately. Temperature has no alert by default. Set other limits, turn an alert off, or leave a mount out of the disk alert per host, in a [`[[host]]`](https://docs.kogut.dev/reference/configuration/#host) table or under **Alerts** on the host page. An agent becomes down after 2 min without data. A CPU or memory alert names the three containers or systemd units that used the most in those 5 min, such as `95%, above 90% for 5 min; top: postgres 2.1 GB (+1.5 GB), api 600 MB`, with the change from their [usual use](https://docs.kogut.dev/incidents/correlation/#largest-users-of-a-saturated-host) in brackets when known. - The first notification waits 30 s for related alerts. A failure that resolves during this wait stays on the timeline but sends no message. - Changed incidents send updates at most every 5 min. An incident resolves after all its alerts have been up for 5 min. - Unacknowledged incidents receive reminders at intervals of 1 h, 2 h, 4 h, and so on, capped at 24 h. Acknowledgement stops reminders, not updates or the resolution. - More than four state changes in 30 min marks a signal as flapping. Ten minutes without a change clears that state. If all failing signals are flapping, only the resolution can notify. There are no quiet hours. [Silences and maintenance](https://docs.kogut.dev/alerts/silences-and-maintenance/) hold messages without stopping checks or changing history. **Acknowledge** in the panel, or `POST /api/v1/incidents//ack`, stops reminders. ## Containers Each container of a service has a signal of its own, such as `Container api on vps-1`. A container belongs to a service when its name contains the service name, or when the service is the only one on its host. Other containers have no signal. The container is down when: - it restarted 3 times within 10 min, even if it reads `running` between restarts; - two samples in a row show it exited or dead with a non-zero exit code, or stopped by the OOM killer; - two samples in a row show it unhealthy, whatever its health was before. The alert names the reason, such as `restarted 4 times in 10 min`, `exited with code 137`, or `unhealthy`. A container that exited with code 0 is not failing. A restart loop stays down until its restarts are 10 min old, then recovers like a check. While the agent is silent or cannot read the container list, container signals are unknown. A removed container stops being judged and its alert closes. Update agents to report exit codes and OOM kills. With an older agent, the restart and health rules still work, but an exited container is never down. ## When kogut loses its internet connection While a server-run check targets a public address, kogut probes its own connection every 30 s. Probe targets and how to turn probing off are in [configuration](https://docs.kogut.dev/reference/configuration/#server). One failed round holds messages for incidents made only of server-run public checks. Two consecutive failed rounds mark kogut offline: - One connection alert replaces the apparent public outages. Channels that cannot be reached follow the normal [delivery retries](https://docs.kogut.dev/alerts/channels/#delivery-and-failures). - Failed public checks become unknown and do not count as downtime. kogut also changes failures that began within a minute before the first failed round to unknown and closes their alerts. - Public-check-only incidents hold their messages until connectivity returns. Checks of non-public targets, agent-run checks, heartbeats, and agents continue as usual. Non-public targets include loopback, private, link-local, and `100.64.0.0/10` addresses, names without a dot, and names under `localhost`, `local`, `internal`, `lan`, `home.arpa`, `localdomain`, or `test`. Domain expiry checks always use the internet. When a probe answers, the connection alert resolves and checks start afresh. A public check that still fails runs again after 30 s. If a public check passes while probes fail, kogut treats the probes as blocked. Failures count normally, and the connection alert stays open without notifications. kogut restores this judgement from stored results after a restart. ## When checks stop running kogut expects a result from every check within three of its intervals plus its timeout. A check without one becomes unknown. One alert, such as `kogut has not run 2 checks since 14:07`, covers all such checks and updates as more join. It resolves once each of them has a result again. A `checks_not_run` entry on the timeline lists the checks. - Counting starts at the last result, when the check was added or changed, or when kogut started, whichever is latest. A restart alone raises nothing. An alert open before a restart stays open until its checks run. - Agent-run checks count only while their agent reports. A silent agent raises its own alert instead. Checks on an agent that has never connected are not counted. - Heartbeats have their own deadlines and are not counted. Domain checks count with their daily interval. - [Silences and maintenance](https://docs.kogut.dev/alerts/silences-and-maintenance/) hold the alert when they cover every check it lists, by its service, its host, or everything. Counting continues during the window, and a held alert is sent when the window ends. ## kogut outages A stopped or stuck server cannot notify. Run kogut apart from the servers it watches, and use [kogut edge](https://docs.kogut.dev/high-availability/kogut-edge/) or a [heartbeat URL](https://docs.kogut.dev/high-availability/heartbeat-url/) to get an alert when kogut stops. ## Message contents Messages contain the state and start time, up to five failing signals with their observed errors, the likely cause, affected services, and an incident link: > website is down since 14:07 (3 signals failing) > > Storefront failing since 14:07: status 503, expected 2xx.\ > Checkout failing since 14:07: status 503, expected 2xx.\ > API health failing since 14:08: no answer within 10000 ms.\ > Likely cause: deploy of api 3f9c2e1 at 14:02.\ > Also affected: api. > > Open incident Longer lists end with the remaining count. Updates name what still fails, reminders state the elapsed time, and recoveries give the outage duration. Cause wording follows the [correlation rules](https://docs.kogut.dev/incidents/correlation/). Times use the server's time zone. Incident links need a public server URL. Both settings are in [configuration](https://docs.kogut.dev/reference/configuration/#server). ## Message destinations Every incident message goes to every [channel](https://docs.kogut.dev/alerts/channels/), from the panel or the file. There is no routing by service or severity. Daily summaries and deploy reports stay on the timeline, not in channels.