AkurAI-Monitor / skills/akurai-monitor/SKILL.md

Latest change b7ced4d1047f9fefbebc8df92ac853910850811f - Version polling from titan: VM health tunnel unit and docs by Ólafur Búi Ólafsson

---
name: akurai-monitor
description: Operates AkurAI Monitor (runs on titan as rust-monitor-titan) through akurai-monitorctl. Use for health, hosts, applications, metrics, logs, alert rules/events, recipients, webhook channels, and channel tests.
compatibility: Runs on Titan. akurai-monitorctl forwards through the akurai-ec2 CLI to the VM, whose loopback :8800 is a reverse tunnel back to the titan monitor. Collection agents run only on Titan and the production EC2 VM — no other host is monitored.
license: MIT
---

# AkurAI Monitor operations

Use `akurai-monitorctl` for every monitor read and mutation. Do not edit MariaDB directly, `/etc/rust-monitor.env`, systemd units, or Nginx directly. The canonical source is `~/Projects/AkurAI-Monitor`; production is `https://monitor.olibuijr.com`, served by `rust-monitor-titan` on titan (port `8800`; the VM nginx reaches it through the `akurai-monitor-tunnel` reverse tunnel). The EC2 `rust-monitor` service is retired and disabled: never release to it or re-enable it.

## Deploy

Run `./deploy.sh` from an operator shell on titan. It builds the musl release, installs `bin/rust-monitor` and `ui/` into `~/.local/opt/rust-monitor` (keeping `.prev` copies), restarts `rust-monitor-titan`, rolls back automatically if `/api/health` does not come back healthy, then runs `scripts/validate.sh`. The forge has no deploy job: forge jobs are sandboxed and cannot restart the service.

## Version polling

The version poller fetches each enabled application's `health_url` from titan, so VM apps must use URLs reachable from titan: their public `https://<host>/api/health`, or for apps without a public health route (akurai-drive frontend, akurai-dns) the forward tunnel `akurai-vm-health-tunnel` (`scripts/akurai-vm-health-tunnel.service`, titan 127.0.0.1:18095 and :18053). A `127.0.0.1:<port>` URL means titan, not the VM.

## Monitored hosts

Exactly two hosts are collected, no others:
- **`local`** (`host_id=1`) — the production EC2 VM itself (`monitor.olibuijr.com`). `rust-monitor` samples its own cpu/mem/disk/load/net/uptime in-process (`src/collectors.rs`) plus `akurai-dns` metrics tapped from journald; no separate agent is needed since the service runs on that host.
- **`titan`** (`host_id=8`, address `100.88.0.9`) — this workstation. A standalone collection agent, `scripts/titan-monitor-agent`, samples the same metric namespace from `/proc` and pushes each point to `POST /api/manage/metrics` with the management bearer token. Installed as `titan-monitor-agent.service` (oneshot) + `.timer` (60s cadence) under `/etc/systemd/system/`, config at `/etc/titan-monitor-agent.env` (`MONITOR_API_TOKEN` synced from PassVault entry 25, `MONITOR_HOST_ID=8`). Alert rules mirror the EC2 host's (`Titan High CPU/Memory`, `Titan Disk Almost Full`).

Metric namespace, identical on both hosts: `cpu.usage`, `mem.*`, `disk.root.*`, `load.*`, `net.<iface>.*`, `uptime.seconds`, plus per-entity metrics sampled once per persist interval (60s) rather than the fast live tick: `proc.<name>.{mem_kb,count}` (top `MONITOR_PROC_TOP_N`, default 15, processes by RSS, aggregated by sanitized command name), `systemd.<unit>.{active,mem_kb,restarts}` (allowlisted by `MONITOR_SYSTEMD_MATCH` prefixes, default = `MONITOR_JOURNAL_MATCH`) plus a host-wide `systemd.units.failed` count that needs no allowlist, and `docker.<container>.mem_kb` plus `docker.containers.running` (Docker CLI; no-ops cleanly when Docker is absent — the Titan agent's `.service` unit needs `SupplementaryGroups=docker` for socket access under `DynamicUser`).

Before registering a new host, confirm it should join this scope — don't add a third host or leave documentation implying one exists.

```sh
akurai-monitorctl health
akurai-monitorctl status
akurai-monitorctl events
akurai-monitorctl hosts list
akurai-monitorctl applications list
akurai-monitorctl metrics list --limit 50
akurai-monitorctl logs list --limit 50
akurai-monitorctl ingestion health
akurai-monitorctl logs recent --hours 1 --limit 100
akurai-monitorctl alerts list
akurai-monitorctl recipients list
akurai-monitorctl channels list
```

## Log ingestion

`ingestion health` reports every configured journald unit and tailed file, last activity, current error, cumulative error count, and stale state. `status=degraded` means an active ingestion error; `status=stale` means the pipeline has no current error but at least one source has been quiet longer than `MONITOR_LOG_STALE_SECS` (default 900).

`logs recent` reads the bounded log API rather than the management table:

```sh
akurai-monitorctl logs recent --source akurai-dns.service --hours 24 --limit 100
akurai-monitorctl logs recent --hours 24 --limit 100 --before-id LAST_ID
```

Limits are clamped to 1–1000 and `next_before_id` is the stable pagination cursor. Journald resumes from its last committed cursor; an unusable cursor falls back to `MONITOR_JOURNAL_REPLAY_SECS` (default 300) and cursor deduplication. File tails detect inode replacement and truncation, retain incomplete final lines until completed, and report rate-limited read errors.

### Structured log contract

Log rows carry `host_id, application_id, source, line, level, target, pid, origin (server|client), fields (JSON), ts`. All but `source`/`line` are optional; legacy rows have NULLs. The journald follower fills `level` from `PRIORITY` and `pid` from `_PID`. `logs recent` and `GET /api/logs` accept `--source`/`level=` filters and return every field. trace/debug retention is shorter (`MONITOR_LOG_DEBUG_RETENTION_DAYS`, default 2) and total rows are hard-capped (`MONITOR_LOG_MAX_ROWS`, default 2,000,000).

Apps ship rich structured logs via the shared **`akurai-log`** integration (AkurAI Build repo, `/home/olafurbui/Projects/akurai-log`): a Rust `tracing` layer (`akurai_log::init("app")`) for standalone Rust apps, `js/monitor-client.js` (browser) and `js/monitor-server.js` (Node/Bun + `/api/client-logs` proxy) for JS. Env: `MONITOR_INGEST_URL`, `MONITOR_INGEST_TOKEN`, `MONITOR_HOST_ID`, `MONITOR_APP_ID`, `MONITOR_LOG_LEVEL`. Zero-touch alternative: add a unit to `MONITOR_JOURNAL_UNITS` (server logs flow with level/pid, no code change). Framework (zero-external-crate) apps need a Framework-native shipper, not the crate.

Production management commands require `MONITOR_API_TOKEN` in `/etc/rust-monitor.env`. Store it in PassVault and synchronize it only through stdin:

```sh
printf '%s' "$MONITOR_API_TOKEN" |
  akurai-ec2 secret-sync rust-monitor.service MONITOR_API_TOKEN /etc/rust-monitor.env
```

Never print the token or put it in an argument. The post-deploy validator checks ingestion health plus the presence of a recent persisted log.

## CRUD contract

Every managed resource supports `list`, `get ID`, `create FIELD=VALUE...`, `update ID FIELD=VALUE...`, and `delete ID --yes`:

```sh
akurai-monitorctl hosts list
akurai-monitorctl metrics create host_id=8 name=custom.metric value=1
akurai-monitorctl alerts create name='High load' host_id=8 metric_name=load.5m operator=gt threshold=4 duration_secs=300
akurai-monitorctl hosts update 8 enabled=false
```

Values are JSON scalars: numbers, `true`, `false`, `null`, or strings. Alert operators are `gt`, `lt`, and `eq`.

## Recipients and channels

A channel is a transport connection. A recipient selects the destination on that connection. Enabled recipients receive triggered and resolved alerts.

Webhook:

```sh
akurai-monitorctl channels create name=ops-webhook kind=webhook endpoint=https://example.invalid/monitor token_env=MONITOR_CHANNEL_TOKEN_WEBHOOK
akurai-monitorctl recipients create channel_id=1 name=operations target=on-call
akurai-monitorctl channels test 1
```

Webhook is the only channel kind. `target` is passed through as `recipient.target` in the POSTed JSON (`{"recipient": {"name", "target"}, "alert": {...}}`). Webhook endpoints require HTTPS and must resolve only to public addresses. Optional channel secret names must start with `MONITOR_CHANNEL_TOKEN_` and exist in the protected Monitor service environment; the value is sent as a bearer token. Store or synchronize their values through AkurAI-PassVault; never put a value in a CLI argument, the database, source, logs, chat, or this skill.

A channel test must return `sent` greater than zero before treating the connection as ready. HTTP errors, missing environment variables, and missing recipients fail closed.

## Alert resolution

An open alert resolves when its samples stop violating the rule, or when its metric (scoped to the rule's host/application) has had no sample for `MONITOR_ALERT_STALE_SECS` (default five collection intervals, 300s at `MONITOR_INTERVAL=60`; never shorter than the rule's `duration_secs`). The alert log records the latter as `RESOLVED ... reason=stale`.

## Testing alert rules

Never push synthetic metrics or temporary rules into the live Monitor: every open alert there shows as CRIT on the desktop HUD. Cover evaluator changes with `cargo test` (DB-backed tests use the scratch MariaDB in `MONITOR_TEST_DB_URL`, the `akurai-mariadb-test` container on titan :3307). For an end-to-end check run `cargo test --test alert_e2e -- --ignored`: it starts a throwaway Monitor on a free loopback port against its own scratch database, pushes synthetic samples, checks trigger and stale resolution, and drops the database afterwards.