# Likely causes

The rules and evidence kogut uses to rank likely causes of an incident.

kogut ranks likely causes with fixed rules, not a model. Each finding has a rule, a score,
and evidence. The rules use each service's hosts and dependencies from the
[service configuration](https://docs.kogut.dev/reference/configuration/), so a deploy of `api` can explain a
failing `website` that depends on it.

## Likely cause rules

Rules examine the time around the first failure. The incident shows the top three likely
causes. Scores of at least 0.7 are "likely", scores of at least 0.85 are "very likely",
and lower scores "coincide" with the failure.

| Rule | Evidence required | Score |
|---|---|---|
| `tls_or_dns` | The failing check reports a TLS or name-resolution error | 1.0 |
| `recent_deploy` | A deploy of an affected service or its dependency, within 30 minutes before failure, including [new container images](https://docs.kogut.dev/deploys/containers/) | 0.9 within 5 minutes, falling to 0.4 at 30 minutes |
| `recent_deploy` | A deploy up to 5 minutes after failure starts | 0.6 |
| `recent_change` | A [change](https://docs.kogut.dev/deploys/changes/) of an affected service, its dependency, or an incident host, within 30 minutes before failure | 0.9 within 5 minutes, falling to 0.4 at 30 minutes |
| `recent_change` | Such a change up to 5 minutes after failure starts | 0.6 |
| `recent_change` | A change with neither a service nor a host, in the same window | at most 0.5, so it only coincides |
| `host_saturation` | CPU or memory above 90%, swap above 50%, or [disk above 95%](#a-full-disk), in at least 3 of the 5 minutes before a service first fails | 0.8 |
| `host_restarted` | A host of the incident restarts | 0.8 |
| `error_spike` | At least 20 JavaScript errors in a 5-minute window, at least 3 standard deviations above the baseline error rate, and at least twice that rate | 0.7 |
| `oom_killed` | The OOM killer stops a service container | 0.9, or 0.5 on a dependency |
| `container_restart` | A service container restarts, is recreated (for example after pulling a new image), or becomes unhealthy | 0.7, or 0.5 on a dependency |
| `unit_failed` | A service's systemd unit fails | 0.7, or 0.5 on a dependency |
| `missed_heartbeat` | A heartbeat of the service or on an incident host is late | 0.5, or 0.4 on a dependency |

A change matches when its `service` or `host` is an affected service, a dependency, or a
host of the incident. A change with neither never counts as likely. It reads as its title
and time, such as "Likely cause: database migration at 14:05". At the same score, a deploy
ranks before a change.

Error windows, container events, unit failures, and host restarts must fall between 10
minutes before and 5 minutes after the first failure. Containers and units belong to
services [by name](https://docs.kogut.dev/agent/discovery/#service-matching).

A container is recreated when a new container with the same name replaces it, as after
a changed Compose file, or an image update reported by an agent without image ids. The
cause then reads "the api container on vps-1 was recreated from api:latest", or "was
updated to api:2" when the tag changed. Restarts of the same container lead the summary.
A new image id is a [deploy](https://docs.kogut.dev/deploys/containers/) instead.

`oom_killed` reads like "OOM killer stopped api on vps-1 at 03:12". It matches when the
container runtime says the OOM killer stopped the container, or when the host's count of
OOM kills rises while a container restarts or stops. It replaces `container_restart` for
that container and is a cause, not an alert of its own. Older agents do not report OOM
kills.

The last four rules also count the same events on a dependency, direct or not, on any
host, at the lower score, such as "the postgres container on db-1 restarted, on dependency
postgres". When the service's own events match too, those set the summary and the score.

A deploy the agent noticed from a new container image adds a line with the image to roll
back to, such as "Previous image: ghcr.io/acme/api@sha256:3f9c…". See
[Container images](https://docs.kogut.dev/deploys/containers/#rollback-hint).

A host restart is seen when its agent connects with a different boot ID, and counts from
the moment the agent is back. Agents that do not report a boot ID never trigger
`host_restarted`.

The same rules run for a [blip](https://docs.kogut.dev/alerts/how-alerts-work/#timing), whose timeline event
lists its causes.

Evidence names the observation, such as "Memory on vps-1 above 90% in 4 of the 5 minutes
before the first failure, up to 98%". Rule names also appear in webhooks, MCP, and
[Copy for AI](https://docs.kogut.dev/incidents/copy-for-ai/).

### A full disk

A disk above 90% opens its own [host alert](https://docs.kogut.dev/alerts/how-alerts-work/#timing) at once.
`host_saturation` looks at the 5 minutes before the first failure of a service, not of the
disk alert, and reads like "disk on vps-1 at 97%", naming the host, not the mount.

### Largest users of a saturated host

When `host_saturation` finds CPU or memory, its evidence also names the three containers
or systemd units that used the most of it in the 5 minutes before the first failure, such
as "Largest memory users on vps-1: postgres 2.1 GB, 1.5 GB more than usual; api 600 MB".
CPU is in percent of one CPU and its change in points. Swap and disks have no such line.

"Usual" is the average of the same hour of the week in the last four weeks, from the
hourly rollups of container and unit CPU and memory. It needs at least two of those weeks;
until then the line gives only the value. Those rollups keep the 30 largest containers
and units of each host at a time, and follow the `hourly_buckets`
[retention](https://docs.kogut.dev/reference/configuration/#retention). Units need an agent that reports
their CPU and memory.

## Baselines and traffic impact

A baseline stores the running mean and variance for each service, metric, and hour of the
week. It is updated hourly, without incident windows.

Each baseline needs four weekly samples. Before it is ready, the error-spike rule uses
absolute limits: at least 20 errors in 5 minutes and an error rate of at least 1% of page
views. Traffic drops are not reported until the baseline is ready.

With a ready baseline, page views below half the usual level produce `traffic_drop`. It
shows under the incident's impact, never as a likely cause.

## Visitor impact

For a [blip](https://docs.kogut.dev/alerts/how-alerts-work/#timing), a host threshold such as high CPU, and
an incident, kogut counts what visitors saw on the sites of the services involved. A
host threshold covers every service on that host. The counts need no baseline:

- The span runs from the start of the minute of the first failure, or of when the value
  went above its limit, to the recovery. kogut compares it with a span of the same length
  just before.
- It counts page views and JavaScript errors, and the 75th percentile of LCP, INP, and CLS
  from the raw Web Vitals reports.
- The change in page views shows as a share, such as "40% fewer page views", only when
  the span before had at least 20 page views. Below that, only the counts show, such as
  "3 page views (5 before), 12 JavaScript errors (0 before)".
- Without sites, or without any visitor in either span, nothing is added.

The numbers are stored as `visitor_impact` in the evidence of the `blip`, `threshold`,
`incident_opened`, and `incident_resolved` events. The incident timeline shows them under
the event, and MCP and [Copy for AI](https://docs.kogut.dev/incidents/copy-for-ai/) include them.

## Recurring blips

After each [blip](https://docs.kogut.dev/alerts/how-alerts-work/#timing), kogut looks at the check's blips
of the last 7 days. They form a pattern when blips fall inside the same 30-minute window of
the server's local time on at least 3 of those days. The window slides, so 02:50 and 03:10
count together, and a window across midnight counts as one night. Local time follows
daylight saving changes.

The pattern names what ran with the blips on more than half of the matched days: a
[heartbeat](https://docs.kogut.dev/monitoring/heartbeats/) run from its start ping to its next ping, or a
restart of a service container or a failed unit, from 10 minutes before a blip to 5
minutes after it. Heartbeats count when they belong to the check's service or to a service
on the same host. Runs are known only for heartbeats that send a start ping.

The first time a pattern appears, kogut records a `blip_pattern` event with the rule
`recurring_blips`, the window, the days matched, the blip times, and what ran with them.
Later blips that fit the same pattern update its numbers, time of day, and summary instead
of adding events, as long as its time of day moves by less than 30 minutes. The check page
and the MCP `service` tool show the pattern as one line, such as "3 of the last 7 nights
around 03:00, together with backup", until fewer than 3 of the last 7 days have a blip in
the window. Blip events are kept like other timeline events. Heartbeat pings follow
[`[retention] check_results`](https://docs.kogut.dev/reference/configuration/#retention), so a value under 7
days can hide the heartbeat that ran with older blips.

## Evidence on the incident page

The incident page shows duration, page views, JavaScript errors, response-time and
host-resource charts, likely cause evidence, and a timeline. Memory and CPU charts of the
containers and systemd units on the incident's host follow, with up to six lines: the
service's own first, then those that restarted or became unhealthy, then those that used
the most memory. The timeline starts 30 minutes before the first failure and includes
deploys, state changes, container and host restarts, and new errors. A host that comes
back after a restart shows as "vps-1 was unreachable 02:10–04:01 and restarted at 04:01".

When an incident opens, kogut asks the agents on affected services' hosts for logs: up to
200 lines and 64 KiB per container or systemd unit, from 10 minutes before the first
failure. Answers more than 15 minutes late are ignored. Logs are not collected otherwise.

Log excerpts have no redaction. They also appear in MCP and copied incident text.