popagent

Latest change ac87cc5fc89df21b21c6d2b8938cb47d98ed176c - Expose Popagent process memory metrics by AkurAI Build

# Observability

Mastra observability is self-hosted through `MastraStorageExporter` and a dedicated PostgreSQL v-next storage domain. `OBSERVABILITY_DATABASE_URL` is required and must never point at the application database.

## Signals and correlation

- Complete sampling records traces, automatic token/latency metrics, structured logs at info and above, and user feedback. The overview also reports the current Popagent process RSS and heap usage from Bun's native `process.memoryUsage()`; these are point-in-time values, not historical aggregates or model/database memory. Mastra currently persists spans without `serviceName`; read APIs therefore filter by bounded time/status only rather than hiding those rows behind a service-name predicate.
- Every chat turn, task, and schedule execution receives one valid Mastra trace
  ID. Chat metadata persists it under `metadata.custom.traceId` beside the stable
  turn ID. Task execution creates the trace before lifecycle start and carries
  that same ID through execution, completion, and pre-stream failure evidence.
- Only safe scalar request-context keys are extracted. Runtime settings and objects are excluded.
- The AI SDK finish event's provider usage is persisted under `metadata.custom.usage`; the UI reads token counts from that object.

## Privacy

Mastra's sensitive-data filter remains enabled. Secret-tool values have a persisted-span redaction contract, including generic `value` fields. Trace read APIs never return span input or output bodies; operators receive span type/name/status/duration and filtered error evidence only. Do not add raw prompts, credentials, browser profiles, hook payloads, runtime settings, or tool objects to trace metadata or logs.

## Read APIs and retention

Protected APIs expose overview KPIs, bounded trace lists/details, warning/error logs, health, and trace-linked feedback. Reads force a maximum 30-day window and bounded pagination. Daily maintenance retains ordinary signals for `POPAGENT_OBSERVABILITY_RETENTION_DAYS` (default 30) and feedback for `POPAGENT_OBSERVABILITY_FEEDBACK_RETENTION_DAYS` (default 90). Estimated cost is intentionally absent until 9Router model IDs are verified against Mastra's pricing registry.

The tracked native OMP extension exposes these protected reads as `popagent_observability` with `overview`, `traces`, `trace`, `logs`, and `health` actions. It uses `POPAGENT_URL` and optional `POPAGENT_API_KEY`, bounds history to 720 hours and page size to 100, and cannot write observability data.

The UI presents last-24-hour run/error/token/latency KPIs, current process RSS and heap usage, expandable span timelines, correlated warnings/errors, explicit loading/empty/failure states, and trace-linked thumbs feedback.

## Contract ownership

- `src/observability.test.ts` — configuration, query bounds, correlation, redaction, and feedback contracts.
- `src/maintenance.test.ts` — signal and feedback retention.
- `src/server.test.ts` — protected health/read/feedback routes.
- `src/ui/ObservabilityPage.tsx` and final browser verification — routed UI behavior.

Observability storage failures must remain visible through health and structured stdout without redirecting writes to application PostgreSQL.