popagent

public
Latest change 7cecf6a89b6f39dae8c5678f5b369014199aeb3c - Add self-hosted Mastra observability by AkurAI Build
# Mastra observability implementation

## 1. Foundation

- [x] Add Mastra observability and structured logger dependencies.
- [x] Configure a dedicated PostgreSQL observability service and durable volume.
- [x] Document `OBSERVABILITY_DATABASE_URL`, service environment, and retention settings.
- [x] Upgrade storage to `PostgresStoreVNext` with explicit application and observability connections.
- [x] Configure `PinoLogger`, `Observability`, `MastraStorageExporter`, redaction, logging, complete sampling, serialization bounds, and request-context metadata.
- [x] Flush observability and close stores during graceful shutdown.

## 2. Correlation and privacy

- [x] Generate one valid trace ID per chat turn or background task.
- [x] Carry trace ID, execution source, session, turn, workspace, model, task, and schedule correlation safely.
- [x] Add trace tags and metadata without exporting runtime objects or settings.
- [x] Persist trace ID in assistant message metadata alongside usage and turn ID.
- [x] Verify secret-tool inputs and outputs are redacted from exported spans, including the tools' generic `value` field omitted by Mastra's default field list.

## 3. Query and feedback API

- [x] Add shared DTOs for overview, trace lists/details, logs, health, and feedback.
- [x] Add a small observability service around Mastra's storage-domain methods.
- [x] Require bounded time ranges and bounded pagination for observability reads.
- [x] Add API-key-protected overview, trace, log, health, and feedback routes.
- [x] Validate feedback and attach it to a persisted trace.

## 4. Retention and operational logging

- [x] Prune trace, metric, log, score, and feedback partitions from daily maintenance.
- [x] Replace relevant unstructured server/runtime console calls with structured, trace-aware logging.
- [x] Surface observability storage failures through health and retain exporter failure/drop logging on structured stdout.
- [x] Omit estimated model cost until 9Router model-price mapping is verified.

## 5. Minimal UI

- [x] Add an Observability settings route and navigation item.
- [x] Show bounded run/error/token/latency KPIs, recent traces, and warning/error logs.
- [x] Provide loading, empty, and error states with accessible controls.
- [x] Show/copy trace IDs on assistant responses where available.
- [x] Add thumbs-up/down feedback linked to assistant trace IDs.

## 6. Contracts and verification

- [x] Add deterministic configuration, correlation, route, query-bound, feedback, retention, and redaction tests.
- [x] Run narrow tests while implementing: 20 passed.
- [x] Run `bunx tsc --noEmit`; observability changes are clean. The feature baseline retains one unrelated pre-existing `src/browser-settings.ts` type error that concurrent browser work on `main` is replacing.
- [x] Run the complete suite against two disposable PostgreSQL containers: 140 passed; only two pre-existing `workspace.test.ts` assertions failed because they hard-code `/popagent/workspace` and do not accept a Git worktree path.
- [x] Re-run `src/server.test.ts` after route additions against isolated application/observability databases: 22 passed.
- [x] Verify the Observability UI with mocked API data at 1280×720 and 390×844, including trace expansion, KPI/log states, routing, and zero horizontal overflow.
- [x] Update `AGENTS_AGENT.md`, `AGENTS_UI.md`, `AGENTS_RULES.md`, and `.env.example`.
- [ ] Commit the feature branch, merge it into `main`, preserve concurrent browser work, and remove the worktree.