popagent

Latest change 2fb6f198c4c71ef37dffc8ac5dca8482068a7bfc - Add governed AkurAI Build maintenance by AkurAI Build

# Observability

Mastra observability is self-hosted through `MastraStorageExporter` and a dedicated PostgreSQL v-next storage domain. `OBSERVABILITY_DATABASE_URL` is required and must never point at the application database.

## Signals and correlation

- Complete sampling records traces, automatic token/latency metrics, structured logs at info and above, and user feedback. The overview also reports the current Popagent process RSS and heap usage from Bun's native `process.memoryUsage()`; these are point-in-time values, not historical aggregates or model/database memory. Mastra currently persists spans without `serviceName`; read APIs therefore filter by bounded time/status only rather than hiding those rows behind a service-name predicate.
- Every chat turn, task, and schedule execution receives one valid Mastra trace
  ID. Chat metadata persists it under `metadata.custom.traceId` beside the stable
  turn ID. Task execution creates the trace before lifecycle start and carries
  that same ID through execution, completion, and pre-stream failure evidence.
  Self-update workflow runs use that task trace as the workflow run and
  correlation ID, and persist bounded phase transitions and evidence without
  adding prompts, credentials, or tool bodies to observability metadata.
- Only safe scalar request-context keys are extracted. Runtime settings and objects
  are excluded.
- The AI SDK finish event's provider usage is persisted under `metadata.custom.usage`; the UI reads token counts from that object.

## Privacy

Mastra's sensitive-data filter remains enabled. Secret-tool values have a persisted-span redaction contract, including generic `value` fields. Trace read APIs never return span input or output bodies; operators receive span type/name/status/duration and filtered error evidence only. Do not add raw prompts, credentials, browser profiles, hook payloads, runtime settings, or tool objects to trace metadata or logs.

## Read APIs and retention

Protected APIs expose overview KPIs, bounded trace lists/details, warning/error logs, health, and trace-linked feedback. Reads force a maximum 30-day window and bounded pagination. Daily maintenance retains ordinary signals for `POPAGENT_OBSERVABILITY_RETENTION_DAYS` (default 30) and feedback for `POPAGENT_OBSERVABILITY_FEEDBACK_RETENTION_DAYS` (default 90). Estimated cost is intentionally absent until 9Router model IDs are verified against Mastra's pricing registry.

- Workflow phase transitions are task-owned durable evidence, not raw span
  payloads. `GET /api/tasks/:id/workflow` exposes only the bounded typed phase
  projection; trace reads continue to omit prompts, credentials, and tool
  inputs/outputs.

## Contract ownership

- `src/observability.test.ts` — configuration, query bounds, correlation,
  redaction, and feedback contracts.
- `src/maintenance.test.ts` — signal and feedback retention.
- `src/server.test.ts` — protected health/read/feedback routes.
- `src/ui/ObservabilityPage.tsx` and final browser verification — routed UI behavior.

Observability storage failures must remain visible through health and structured stdout without redirecting writes to application PostgreSQL.