Likely causes

kogut ranks likely causes with fixed rules, not a model. Each finding has a rule, a score, and evidence. The rules use each service’s hosts and dependencies from the service configuration, so a deploy of api can explain a failing website that depends on it.

Likely cause rules

Rules examine the time around the first failure. The incident shows the top three likely causes. Scores of at least 0.7 are “likely”, scores of at least 0.85 are “very likely”, and lower scores “coincide” with the failure.

Rule	Evidence required	Score
tls_or_dns	The failing check reports a TLS or name-resolution error	1.0
recent_deploy	A deploy of an affected service or its dependency, within 30 minutes before failure, including new container images	0.9 within 5 minutes, falling to 0.4 at 30 minutes
recent_deploy	A deploy up to 5 minutes after failure starts	0.6
recent_change	A change of an affected service, its dependency, or an incident host, within 30 minutes before failure	0.9 within 5 minutes, falling to 0.4 at 30 minutes
recent_change	Such a change up to 5 minutes after failure starts	0.6
recent_change	A change with neither a service nor a host, in the same window	at most 0.5, so it only coincides
host_saturation	CPU or memory above 90%, swap above 50%, or disk above 95%, in at least 3 of the 5 minutes before a service first fails	0.8
host_restarted	A host of the incident restarts	0.8
error_spike	At least 20 JavaScript errors in a 5-minute window, at least 3 standard deviations above the baseline error rate, and at least twice that rate	0.7
oom_killed	The OOM killer stops a service container	0.9, or 0.5 on a dependency
container_restart	A service container restarts, is recreated (for example after pulling a new image), or becomes unhealthy	0.7, or 0.5 on a dependency
unit_failed	A service’s systemd unit fails	0.7, or 0.5 on a dependency
missed_heartbeat	A heartbeat of the service or on an incident host is late	0.5, or 0.4 on a dependency

A change matches when its service or host is an affected service, a dependency, or a host of the incident. A change with neither never counts as likely. It reads as its title and time, such as “Likely cause: database migration at 14:05”. At the same score, a deploy ranks before a change.

Error windows, container events, unit failures, and host restarts must fall between 10 minutes before and 5 minutes after the first failure. Containers and units belong to services by name.

A container is recreated when a new container with the same name replaces it, as after a changed Compose file, or an image update reported by an agent without image ids. The cause then reads “the api container on vps-1 was recreated from api:latest”, or “was updated to api:2” when the tag changed. Restarts of the same container lead the summary. A new image id is a deploy instead.

oom_killed reads like “OOM killer stopped api on vps-1 at 03:12”. It matches when the container runtime says the OOM killer stopped the container, or when the host’s count of OOM kills rises while a container restarts or stops. It replaces container_restart for that container and is a cause, not an alert of its own. Older agents do not report OOM kills.

The last four rules also count the same events on a dependency, direct or not, on any host, at the lower score, such as “the postgres container on db-1 restarted, on dependency postgres”. When the service’s own events match too, those set the summary and the score.

A deploy the agent noticed from a new container image adds a line with the image to roll back to, such as “Previous image: ghcr.io/acme/api@sha256:3f9c…”. See Container images.

A host restart is seen when its agent connects with a different boot ID, and counts from the moment the agent is back. Agents that do not report a boot ID never trigger host_restarted.

The same rules run for a blip, whose timeline event lists its causes.

Evidence names the observation, such as “Memory on vps-1 above 90% in 4 of the 5 minutes before the first failure, up to 98%”. Rule names also appear in webhooks, MCP, and Copy for AI.

A full disk

A disk above 90% opens its own host alert at once. host_saturation looks at the 5 minutes before the first failure of a service, not of the disk alert, and reads like “disk on vps-1 at 97%”, naming the host, not the mount.

Largest users of a saturated host

When host_saturation finds CPU or memory, its evidence also names the three containers or systemd units that used the most of it in the 5 minutes before the first failure, such as “Largest memory users on vps-1: postgres 2.1 GB, 1.5 GB more than usual; api 600 MB”. CPU is in percent of one CPU and its change in points. Swap and disks have no such line.

“Usual” is the average of the same hour of the week in the last four weeks, from the hourly rollups of container and unit CPU and memory. It needs at least two of those weeks; until then the line gives only the value. Those rollups keep the 30 largest containers and units of each host at a time, and follow the hourly_buckets retention. Units need an agent that reports their CPU and memory.

Baselines and traffic impact

A baseline stores the running mean and variance for each service, metric, and hour of the week. It is updated hourly, without incident windows.

Each baseline needs four weekly samples. Before it is ready, the error-spike rule uses absolute limits: at least 20 errors in 5 minutes and an error rate of at least 1% of page views. Traffic drops are not reported until the baseline is ready.

With a ready baseline, page views below half the usual level produce traffic_drop. It shows under the incident’s impact, never as a likely cause.

Visitor impact

For a blip, a host threshold such as high CPU, and an incident, kogut counts what visitors saw on the sites of the services involved. A host threshold covers every service on that host. The counts need no baseline:

• The span runs from the start of the minute of the first failure, or of when the value went above its limit, to the recovery. kogut compares it with a span of the same length just before.
• It counts page views and JavaScript errors, and the 75th percentile of LCP, INP, and CLS from the raw Web Vitals reports.
• The change in page views shows as a share, such as “40% fewer page views”, only when the span before had at least 20 page views. Below that, only the counts show, such as “3 page views (5 before), 12 JavaScript errors (0 before)”.
• Without sites, or without any visitor in either span, nothing is added.

The numbers are stored as visitor_impact in the evidence of the blip, threshold, incident_opened, and incident_resolved events. The incident timeline shows them under the event, and MCP and Copy for AI include them.

Recurring blips

After each blip, kogut looks at the check’s blips of the last 7 days. They form a pattern when blips fall inside the same 30-minute window of the server’s local time on at least 3 of those days. The window slides, so 02:50 and 03:10 count together, and a window across midnight counts as one night. Local time follows daylight saving changes.

The pattern names what ran with the blips on more than half of the matched days: a heartbeat run from its start ping to its next ping, or a restart of a service container or a failed unit, from 10 minutes before a blip to 5 minutes after it. Heartbeats count when they belong to the check’s service or to a service on the same host. Runs are known only for heartbeats that send a start ping.

The first time a pattern appears, kogut records a blip_pattern event with the rule recurring_blips, the window, the days matched, the blip times, and what ran with them. Later blips that fit the same pattern update its numbers, time of day, and summary instead of adding events, as long as its time of day moves by less than 30 minutes. The check page and the MCP service tool show the pattern as one line, such as “3 of the last 7 nights around 03:00, together with backup”, until fewer than 3 of the last 7 days have a blip in the window. Blip events are kept like other timeline events. Heartbeat pings follow [retention] check_results, so a value under 7 days can hide the heartbeat that ran with older blips.

Evidence on the incident page

The incident page shows duration, page views, JavaScript errors, response-time and host-resource charts, likely cause evidence, and a timeline. Memory and CPU charts of the containers and systemd units on the incident’s host follow, with up to six lines: the service’s own first, then those that restarted or became unhealthy, then those that used the most memory. The timeline starts 30 minutes before the first failure and includes deploys, state changes, container and host restarts, and new errors. A host that comes back after a restart shows as “vps-1 was unreachable 02:10–04:01 and restarted at 04:01”.

When an incident opens, kogut asks the agents on affected services’ hosts for logs: up to 200 lines and 64 KiB per container or systemd unit, from 10 minutes before the first failure. Answers more than 15 minutes late are ignored. Logs are not collected otherwise.

Log excerpts have no redaction. They also appear in MCP and copied incident text.