# Beszel

Install kogut agents alongside Beszel, replace its network monitors and alerts, and recreate its setup from data.db.

kogut does not import Beszel's settings or history. Install the agent alongside Beszel's
and compare the data before switching. To rebuild systems, alerts, channels, and network
monitors from Beszel's database, see
[Recreate the configuration from data.db](#recreate-the-configuration-from-datadb).

## Connect each server

1. Open **Hosts → Add a server** in kogut and use the system's name from Beszel.
2. Run the generated install command on that server, or see
   [Installing the agent](https://docs.kogut.dev/agent/installing/).
3. Check the host's metrics and containers in kogut.

The agent needs no inbound port.

## Run both side by side

Both agents can run on one host. Beszel's agent listens on port 45876 for the hub, or
connects out to `HUB_URL`. kogut's agent only connects out. Both use host networking and
read the Docker socket read-only:

```yaml title="compose.yml"
services:
  beszel-agent:
    image: henrygd/beszel-agent
    restart: unless-stopped
    network_mode: host
    volumes:
      - ./beszel_agent_data:/var/lib/beszel-agent
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      LISTEN: 45876
      KEY: "<public key>"
      HUB_URL: "<hub url>"
      TOKEN: "<token>"

  kogut-agent:
    image: ghcr.io/traceworks-ai/kogut:0.1
    command: ["agent"]
    restart: unless-stopped
    network_mode: host
    pid: host
    group_add:
      - "${DOCKER_GID:?set DOCKER_GID to the group id of /var/run/docker.sock}"
    environment:
      - KOGUT_SERVER=https://kogut.acme.com
      - KOGUT_TOKEN=${KOGUT_TOKEN}
      - KOGUT_HOST_ROOT=/host
      - KOGUT_STATE_DIR=/data
    volumes:
      - agent-state:/data
      - /:/host:ro,rslave
      - /var/run/docker.sock:/var/run/docker.sock:ro

volumes:
  agent-state:
```

Keep the `beszel-agent` service as Beszel generated it. Put the host token in `.env` as
`KOGUT_TOKEN`, then start both with
`DOCKER_GID=$(stat -c %g /var/run/docker.sock) docker compose up -d`. kogut's agent reads
every filesystem through `/host`, so it needs no `/extra-filesystems` mounts.
[The agent in Docker](https://docs.kogut.dev/agent/docker/) explains each setting of `kogut-agent`, and
how to use a socket proxy or Podman.

## Link hosts to services

Add the host to each service that runs on it:

```toml title="kogut.toml"
[[service]]
name = "api"
hosts = ["vps-1"]
```

A host named in the file is created if needed; connect its agent from the panel. See
[`[[service]]`](https://docs.kogut.dev/reference/configuration/#service).

## Compare the metrics

The agent samples every 15 s. Values can differ from Beszel's where the definitions
differ:

| Beszel | kogut |
|---|---|
| System | Host, with one agent each. |
| CPU | Percent of all cores in use, excluding idle, I/O wait, and steal time. Beszel counts steal time as busy, so a virtual machine with steal reads lower in kogut. kogut has no I/O wait or steal chart. |
| Memory | `MemTotal - MemAvailable` from `/proc/meminfo`. Kernels before 3.14 use total minus free, buffers, and cache. |
| Swap | `SwapTotal - SwapFree`. A host without swap has no swap chart. |
| Disk usage | Each disk-backed filesystem once, as `df` shows it: used space over used plus available. Network filesystems, `tmpfs`, and `overlay` are left out. The chart and the alert use the fullest one. |
| Disk I/O | Bytes read and written per second, summed over physical disks. Partitions, RAID arrays, and device-mapper volumes are not counted twice. |
| Bandwidth | Bytes received and sent per second, summed over network cards. Bridges, bonds, VLANs, tunnels, and container links are left out. |
| Load average | 1, 5, and 15 min, on a chart without an alert. |
| Temperatures | `hwmon` sensors. The chart shows the hottest one. |
| Containers | State, health, restarts, exit code, image, CPU as a percent of one CPU, and memory without inactive file cache. No network per container. |
| systemd services | State, restarts, CPU, and memory, when the agent is installed on the host. A failed unit is a [likely cause](https://docs.kogut.dev/incidents/correlation/), not an alert. |
| Alert thresholds per system | [`[[host]]`](https://docs.kogut.dev/reference/configuration/#host) or **Alerts** on the host page. CPU, memory, swap, and temperature must stay above their limits for 5 min. A full disk alerts at once. Beszel's minutes per alert have no equivalent. |
| GPU, battery, S.M.A.R.T. | No equivalent. |

## Replace network monitors

Each Beszel network monitor becomes one `[[check]]` that runs from the agent of the same
system. Set `from` to the host's name in kogut, and list that host in a service's
`hosts`:

| Beszel monitor | `[[check]]` key |
|---|---|
| ICMP | `ping = "<target>"` |
| TCP | `tcp = "<target>:<port>"` |
| HTTP | `url = "<target>"` |
| DNS | `dns = "<target>"` |

Beszel's interval becomes `every`, which is at least `"10s"`:

```toml title="kogut.toml"
[[service]]
name = "api"
hosts = ["vps-1"]

[[check]]
name = "Gateway"
service = "api"
ping = "10.0.0.1"
from = "vps-1"

[[check]]
name = "Postgres port"
service = "api"
tcp = "10.0.0.5:5432"
from = "vps-1"

[[check]]
name = "API health"
service = "api"
url = "http://10.0.0.6:8080/health"
from = "vps-1"
every = "30s"

[[check]]
name = "Internal DNS"
service = "api"
dns = "db.internal"
from = "vps-1"
```

A monitor that Beszel added to several systems is a copy on each of them. In kogut, write
one check per probing host, such as `Postgres port from vps-2`, because each check runs
from one place. See [Choose where to probe](https://docs.kogut.dev/monitoring/internal-services/#choose-where-to-probe).
For a public target, leave out `from` and the server runs the check.

DNS monitors that ask a specific server and HTTP monitors that skip certificate
verification have no equivalent. DNS checks ask the servers in `/etc/resolv.conf` of the
host that runs them, and HTTPS checks validate certificates against kogut's own root
store. See [DNS checks](https://docs.kogut.dev/monitoring/checks/#dns-checks).

### Check more than the status

Beszel's HTTP monitors send `GET` and count an error or a status of 400 or higher as
loss. kogut requires a `2xx` status by default, after up to 5 redirects for `GET` and
`HEAD`. An endpoint that answers with a redirect kogut does not follow, such as one from
a `POST` check, needs `{ status = "3xx" }`.

HTTP checks can also set the method, request headers, and assertions:

```toml title="kogut.toml"
[[check]]
name = "Orders API"
service = "api"
url = "http://10.0.0.6:8080/v1/orders?limit=1"
from = "vps-1"
method = "GET"
headers = { Authorization = "Bearer ${ORDERS_API_TOKEN}" }
expect = [
	{ status = 200 },
	{ json = "$.db", equals = "ok" },
	{ body_not_contains = "maintenance" },
	{ response_time_below = "500ms", else = "degraded" },
]
```

`${ORDERS_API_TOKEN}` comes from the server's environment. For Basic authentication, see
[Request headers](https://docs.kogut.dev/monitoring/checks/#request-headers). [Assertions](https://docs.kogut.dev/monitoring/checks/#assertions)
explains how they combine, and [`expect`](https://docs.kogut.dev/reference/configuration/#expect) lists them
all.

### Alerts from checks

Beszel alerts when a monitor's packet loss over the last hour passes a threshold. kogut
has no loss threshold. A check becomes down after two failed runs in a row, and recovers
after two successes and 3 min. A ping check with some replies missing is degraded. See
[How alerts work](https://docs.kogut.dev/alerts/how-alerts-work/#timing).

When an agent goes silent, its checks become unknown, not down, and the host opens an
incident of its own. See
[Handle an offline agent](https://docs.kogut.dev/monitoring/internal-services/#handle-an-offline-agent).

> [!NOTE]
> Update agents before giving them ping checks or request headers. Ping runs without root
> but needs the agent's group in `net.ipv4.ping_group_range`. See
> [Ping checks](https://docs.kogut.dev/monitoring/checks/#ping-checks).

## Replace alerts and notifications

Beszel's per-system CPU, memory, disk, and temperature thresholds map to a
[`[[host]]`](https://docs.kogut.dev/reference/configuration/#host) table, or to **Alerts** on the host page
for hosts added in the panel. CPU, memory, and temperature must stay above their limits
for 5 min. Bandwidth and load average appear on charts without alerts. A container of a
service that restarts in a loop, exits with an error, or stays unhealthy raises an
[alert](https://docs.kogut.dev/alerts/how-alerts-work/#containers), and its restarts and OOM kills become
[likely causes](https://docs.kogut.dev/incidents/correlation/).

Add channels under **Settings → Notifications** or in `[[alert]]` tables, with file
secrets in `${NAME}` variables. Convert Shoutrrr URLs to channel settings:

| Beszel | kogut |
|---|---|
| `discord://<token>@<id>` | `type = "discord"`, `url = "https://discord.com/api/webhooks/<id>/<token>"` |
| `telegram://<token>@telegram?chats=<chat>` | `type = "telegram"`, `bot_token` from `<token>`, `chat_id` from `chats` |

See [Notification channels](https://docs.kogut.dev/alerts/channels/) and
[How alerts work](https://docs.kogut.dev/alerts/how-alerts-work/).

## Recreate the configuration from data.db

Beszel keeps its systems, alerts, notification settings, and network monitors in
`beszel_data/data.db`, a SQLite database. Read a copy, not the file the hub is writing.

1. Stop the hub, copy its data directory, and start the hub again. The copy includes
   `data.db-wal` when it exists, which holds the latest changes.

   ```sh
   docker compose stop beszel
   cp -a beszel_data beszel-copy
   docker compose start beszel
   ```

2. Export the configuration from the copy. These queries match Beszel 0.20 and 0.21:

   ```sh
   db=beszel-copy/data.db
   sqlite3 -readonly -json "$db" \
     "SELECT name, host, port FROM systems ORDER BY name" > systems.json
   sqlite3 -readonly -json "$db" \
     "SELECT s.name AS system, a.name AS alert, a.value, a.min, u.email AS user
      FROM alerts a JOIN systems s ON s.id = a.system LEFT JOIN users u ON u.id = a.user
      ORDER BY s.name, a.name, u.email" > alerts.json
   sqlite3 -readonly -json "$db" \
     "SELECT u.email AS user, json_extract(us.settings, '$.emails') AS emails,
        json_extract(us.settings, '$.webhooks') AS webhooks
      FROM user_settings us JOIN users u ON u.id = us.user" > notifications.json
   sqlite3 -readonly -json "$db" \
     "SELECT s.name AS system, m.protocol, m.target, m.port, m.server, m.interval, m.enabled
      FROM network_monitors m JOIN systems s ON s.id = m.system
      ORDER BY s.name, m.target" > monitors.json
   ```

   `alerts.json` lists each alert's name, such as `CPU` or `Disk`, its threshold in
   `value`, in `min` the minutes it must last, and the user who set it. Each Beszel user
   sets their own alerts, so a system can have one row per user for the same alert.
   `monitors.json` has the DNS server of each monitor in `server`. If sqlite3 reports
   `no such column: m.server`, your Beszel version has no server setting: remove
   `m.server,` from the last query. `notifications.json` holds Shoutrrr URLs with their
   tokens, so keep it private and delete it afterwards.

3. Create a `write` token under **Settings → Tokens**, or with `POST /api/v1/tokens` and
   `{"scope": "write", "label": "beszel"}` from an admin's session.

To have an assistant create the hosts, checks, and channels over the
[HTTP API](https://docs.kogut.dev/reference/http-api/), give it the four files and this prompt:

```text
Recreate my Beszel setup in kogut at https://kogut.acme.com over its HTTP API,
documented at https://docs.kogut.dev/reference/http-api/. Send the header
"Authorization: Bearer $KOGUT_TOKEN" with every request. systems.json, alerts.json,
notifications.json, and monitors.json come from Beszel's data.db. Read kogut.toml and
GET /api/v1/hosts and /api/v1/settings first, and show me the plan before changing
anything.

1. Hosts: POST /api/v1/hosts with {"name": <system name>} for each system kogut does
   not have. Ask me before posting an existing host, because that replaces its token.
   Give me each install command it returns.
2. Thresholds: for each host added in the panel, PUT /api/v1/hosts/{name}/alerts with
   {cpu, memory, swap, disk, temperature, ignoreMounts}. Start from 90, 90, 50, 90,
   null, and [], and take whole-number values from the CPU, Memory, Disk, and
   Temperature alerts in alerts.json. When users set different values for the same
   system and alert, ask me which to use. List the other alerts as not carried over.
   Hosts in kogut.toml get a [[host]] table instead.
3. Checks: for each enabled monitor, POST /api/v1/checks with kind "ping" for icmp,
   "tcp" with target "<target>:<port>", "http", or "dns"; "from" set to the system's
   host; "everyMs" set to interval × 1000, at least 10000; and "expect": []. Leave out
   DNS monitors with a server set, because kogut asks the host's own resolvers. Use
   one service per system, named after it, unless I name others. Name each check after
   its target and host. Then link each host to its service with
   POST /api/v1/discovered/monitor and {"host", "service", "check": null}.
4. Channels: POST /api/v1/channels for each address in emails and each URL in
   webhooks. Convert discord:// and telegram:// URLs with the table at
   https://docs.kogut.dev/coming-from/beszel/#replace-alerts-and-notifications. Ask me
   for SMTP settings and about any other scheme. Name channels such as ops-email, never
   a bare type such as email or discord-2.
5. Keep panel and file objects apart. Never reuse a name from kogut.toml. Checks that
   need request headers or Basic authentication, and checks of services defined in
   kogut.toml, go into kogut.toml as [[check]] tables with secrets as ${NAME},
   because the API cannot set headers.

Finish with a list of what you created and what you left out.
```

## Remove Beszel

People sign in to kogut with a password, or through
[OpenID Connect](https://docs.kogut.dev/sign-in/openid-connect/) or [forward auth](https://docs.kogut.dev/sign-in/forward-auth/).
Every account sees every host; see [Roles](https://docs.kogut.dev/sign-in/roles/).

Keep Beszel if you need GPU metrics, per-user systems, alerts on bandwidth or load, or
alerts on a network monitor's hourly packet loss. Otherwise, check all hosts, the checks
that replace network monitors, and channels in kogut, stop Beszel's agents, then stop
its hub. kogut's history starts when its agents connect.