AkurAI-Build

Latest change e11509261b6a2aa4ea9ef67250b0be01bfb4f53b - Restore paths swept in from uncommitted WIP by Ólafur Búi Ólafsson

---
name: akurai-monitor-ops
description: Operates AkurAI Monitor on EC2 through akurai-monitorctl. Use for health, hosts, applications, metrics, logs, alert rules/events, recipients, webhook channels, Pi-bun agent channels, and channel tests.
compatibility: Runs on Titan and forwards to the AkurAI EC2 VM through the maintained akurai-ec2 CLI. Collection agents run only on Titan and the production EC2 VM — no other host is monitored.
license: MIT
---

# AkurAI Monitor operations

Use `akurai-monitorctl` for every monitor read and mutation. Do not edit MariaDB directly, `/etc/rust-monitor.env`, systemd units, or Nginx directly. The canonical source is `~/Projects/AkurAI-Monitor`; production is `https://monitor.olibuijr.com`, service `rust-monitor`, loopback port `8800`.

## Monitored hosts

Exactly two hosts are collected, no others:
- **`local`** (`host_id=1`) — the production EC2 VM itself (`monitor.olibuijr.com`). `rust-monitor` samples its own cpu/mem/disk/load/net/uptime in-process (`src/collectors.rs`) plus `akurai-dns` metrics tapped from journald; no separate agent is needed since the service runs on that host.
- **`titan`** (`host_id=8`, address `100.88.0.9`) — this workstation. A standalone collection agent, `scripts/titan-monitor-agent`, samples the same metric namespace from `/proc` and pushes each point to `POST /api/manage/metrics` with the management bearer token. Installed as `titan-monitor-agent.service` (oneshot) + `.timer` (60s cadence) under `/etc/systemd/system/`, config at `/etc/titan-monitor-agent.env` (`MONITOR_API_TOKEN` synced from PassVault entry 25, `MONITOR_HOST_ID=8`). Alert rules mirror the EC2 host's (`Titan High CPU/Memory`, `Titan Disk Almost Full`).

Metric namespace, identical on both hosts: `cpu.usage`, `mem.*`, `disk.root.*`, `load.*`, `net.<iface>.*`, `uptime.seconds`, plus per-entity metrics sampled once per persist interval (60s) rather than the fast live tick: `proc.<name>.{mem_kb,count}` (top `MONITOR_PROC_TOP_N`, default 15, processes by RSS, aggregated by sanitized command name), `systemd.<unit>.{active,mem_kb,restarts}` (allowlisted by `MONITOR_SYSTEMD_MATCH` prefixes, default = `MONITOR_JOURNAL_MATCH`) plus a host-wide `systemd.units.failed` count that needs no allowlist, and `docker.<container>.mem_kb` plus `docker.containers.running` (Docker CLI; no-ops cleanly when Docker is absent — the Titan agent's `.service` unit needs `SupplementaryGroups=docker` for socket access under `DynamicUser`).

Before registering a new host, confirm it should join this scope — don't add a third host or leave documentation implying one exists.

```sh
akurai-monitorctl health
akurai-monitorctl status
akurai-monitorctl events
akurai-monitorctl hosts list
akurai-monitorctl applications list
akurai-monitorctl metrics list --limit 50
akurai-monitorctl logs list --limit 50
akurai-monitorctl ingestion health
akurai-monitorctl logs recent --hours 1 --limit 100
akurai-monitorctl alerts list
akurai-monitorctl recipients list
akurai-monitorctl channels list
```

## Log ingestion

`ingestion health` reports every configured journald unit and tailed file, last activity, current error, cumulative error count, and stale state. `status=degraded` means an active ingestion error; `status=stale` means the pipeline has no current error but at least one source has been quiet longer than `MONITOR_LOG_STALE_SECS` (default 900).

`logs recent` reads the bounded log API rather than the management table:

```sh
akurai-monitorctl logs recent --source akurai-dns.service --hours 24 --limit 100
akurai-monitorctl logs recent --hours 24 --limit 100 --before-id LAST_ID
```

Limits are clamped to 1–1000 and `next_before_id` is the stable pagination cursor. Journald resumes from its last committed cursor; an unusable cursor falls back to `MONITOR_JOURNAL_REPLAY_SECS` (default 300) and cursor deduplication. File tails detect inode replacement and truncation, retain incomplete final lines until completed, and report rate-limited read errors.

### Structured log contract

Log rows carry `host_id, application_id, source, line, level, target, pid, origin (server|client), fields (JSON), ts`. All but `source`/`line` are optional; legacy rows have NULLs. The journald follower fills `level` from `PRIORITY` and `pid` from `_PID`. `logs recent` and `GET /api/logs` accept `--source`/`level=` filters and return every field. trace/debug retention is shorter (`MONITOR_LOG_DEBUG_RETENTION_DAYS`, default 2) and total rows are hard-capped (`MONITOR_LOG_MAX_ROWS`, default 2,000,000).

Apps ship rich structured logs via the shared **`akurai-log`** integration (AkurAI Build repo, `/home/olafurbui/Projects/akurai-log`): a Rust `tracing` layer (`akurai_log::init("app")`) for standalone Rust apps, `js/monitor-client.js` (browser) and `js/monitor-server.js` (Node/Bun + `/api/client-logs` proxy) for JS. Env: `MONITOR_INGEST_URL`, `MONITOR_INGEST_TOKEN`, `MONITOR_HOST_ID`, `MONITOR_APP_ID`, `MONITOR_LOG_LEVEL`. Zero-touch alternative: add a unit to `MONITOR_JOURNAL_UNITS` (server logs flow with level/pid, no code change). Framework (zero-external-crate) apps need a Framework-native shipper, not the crate.

Production management commands require `MONITOR_API_TOKEN` in `/etc/rust-monitor.env`. Store it in PassVault and synchronize it only through stdin:

```sh
printf '%s' "$MONITOR_API_TOKEN" |
  akurai-ec2 secret-sync rust-monitor.service MONITOR_API_TOKEN /etc/rust-monitor.env
```

Never print the token or put it in an argument. The post-deploy validator checks ingestion health plus the presence of a recent persisted log.

## CRUD contract

Every managed resource supports `list`, `get ID`, `create FIELD=VALUE...`, `update ID FIELD=VALUE...`, and `delete ID --yes`:

```sh
akurai-monitorctl hosts list
akurai-monitorctl metrics create host_id=8 name=custom.metric value=1
akurai-monitorctl alerts create name='High load' host_id=8 metric_name=load.5m operator=gt threshold=4 duration_secs=300
akurai-monitorctl hosts update 8 enabled=false
```

Values are JSON scalars: numbers, `true`, `false`, `null`, or strings. Alert operators are `gt`, `lt`, and `eq`.

## Recipients and channels

A channel is a transport connection. A recipient selects the destination on that connection. Enabled recipients receive triggered and resolved alerts.

Webhook:

```sh
akurai-monitorctl channels create name=ops-webhook kind=webhook endpoint=https://example.invalid/monitor token_env=MONITOR_CHANNEL_TOKEN_WEBHOOK
akurai-monitorctl recipients create channel_id=1 name=operations target=on-call
akurai-monitorctl channels test 1
```

Pi-bun error/alert reporting (display-only: lands as a message in the named pi-bun channel under an "Error"/"System" author, never triggers an agent reply — there is no OMP-prompt/command delivery today; that is planned future work):

```sh
akurai-monitorctl channels create name=pi-bun-errors kind=pi-bun endpoint=http://100.88.0.9:4173 token_env=MONITOR_CHANNEL_TOKEN_PI_BUN
akurai-monitorctl recipients create channel_id=2 name=errors target=errors
akurai-monitorctl channels test 2
```

For `pi-bun`, `target` is the destination pi-bun **channel name** (e.g. `errors`; `channel_targets`/the UI's receiver picker resolves options from pi-bun's `/api/channels`). Every notification POSTs to `{endpoint}/api/ingest-error`, display-only. Its endpoint is restricted to Titan's `100.88.0.9` AkurAI-VPN address; webhook endpoints require HTTPS and must resolve only to public addresses. Optional channel secret names must start with `MONITOR_CHANNEL_TOKEN_` and exist in the protected Monitor service environment. Store or synchronize their values through AkurAI-PassVault; never put a value in a CLI argument, the database, source, logs, chat, or this skill.

A `PI-BUN` channel is also auto-provisioned on every startup (`ensure_configured_channels`, endpoint/token/channel from `MONITOR_PIBUN_INGEST_URL`/`_TOKEN`/`_CHANNEL`, default target `errors`) **with an enabled recipient seeded automatically** — triggered/resolved alerts report into pi-bun `#errors` with zero manual setup once `MONITOR_PIBUN_INGEST_TOKEN` is set. `channels list`/`recipients list` shows it; deleting or disabling that recipient stops alert delivery.

A channel test must return `sent` greater than zero before treating the connection as ready. HTTP errors, missing environment variables, missing recipients, and Pi-bun login/command failures fail closed.