popagent

Latest change b818f7a5b3d0cc81e44c1449fde912c0c2f1a916 - Move observability PostgreSQL to port 5435 by AkurAI Build

# Agent backend

Framework: **Mastra** (`@mastra/core`). Docs: https://mastra.ai/docs

## Model provider

- LLM gateway: local **9Router** (`http://127.0.0.1:20128/v1`, OpenAI-compatible),
  via `createOpenAICompatible` from `@ai-sdk/openai-compatible`.
  Env: `NINEROUTER_URL`, `NINEROUTER_API_KEY` (see `.env.example`).
- The default model is persisted in the singleton
  `popagent_agent_runtime_settings` row. `GET /api/models` merges that value with
  the gateway's current model IDs and context-window metadata; do not hard-code
  the catalog, default, or context limits in UI code.
- Per-request selection: `POST /api/chat` accepts an optional `model` body field;
  when omitted, the server reads the persisted default.

## HTTP server (`src/server.ts`)

`Bun.serve` with `routes`:

- `/` — the chat UI (`src/ui/index.html`, bundled by Bun at serve time).
- `GET /api/models` — `{ models, contextWindows, defaultModel }` from 9Router.
- `GET/POST /api/sessions` — list active/archived or create sessions.
- `GET/PUT/PATCH/DELETE /api/sessions/:id` — load/save, rename, archive/restore,
  or permanently delete a session and its matching Mastra thread data.
- `GET /api/sessions/:id/hooks` — ordered hook audit runs for the session.
- `GET/POST /api/memories` and `PATCH/DELETE /api/memories/:id` — long-term
  memory management; retention and mutability semantics live in
  [AGENTS_MEMORY.md](AGENTS_MEMORY.md).
- `POST /api/chat` — AI SDK UI-message stream via `handleChatStream` from
  `@mastra/ai-sdk` + `createUIMessageStreamResponse` from `ai`. The route accepts
  only the session id, optional selected model, workspace, and newest UI message
  from the client; callers cannot inject Mastra tools or execution options. The
  chat id maps to the `popagent-user` memory thread. A typed `RequestContext`
  carries one database-backed runtime-settings snapshot through the supervisor
  and delegated specialists. The route also dispatches managed lifecycle hooks,
  injects native cross-session recall, and adds turn IDs to assistant metadata
  so the UI can correlate hook activity.
- `GET/PATCH /api/settings/memory` — read or update persisted Observational
  Memory compaction settings; changes apply on the next agent turn.
- `GET/PATCH /api/settings/agent-runtime` — read or update the default model,
  execution/delegation bounds and feedback, and background-task controls.
- `GET /api/agents` and `PATCH /api/agents/:id` — list persisted system agents
  and update their names, delegation descriptions, instructions, workspace and
  browser access, delegation eligibility, and tool membership.
- `GET/PATCH /api/settings/browser` — read or toggle the global persisted browser
  capability; changes apply immediately without restarting.
- `GET/POST /api/agents/:id/skills` and
  `PATCH/DELETE /api/agents/:id/skills/:skillId` — manage database-backed Agent
  Skills.
- `GET/POST /api/tasks` and `POST /api/tasks/:id/cancel` — manage persistent
  background agent work.
- `GET/POST /api/schedules` and `PATCH/DELETE /api/schedules/:id` — manage
  recurring schedules. Clients submit cron expressions; timezone is
  server-controlled and is not part of the writable API.


Wire DTOs shared with the UI live in `src/api-types.ts`; do not redeclare those
response shapes in React code.

## Decisions

- One user-facing agent (`popagent`) supervises three internal Mastra subagents:
  `researcher`, `implementer`, and `reviewer`, using Mastra's documented
  `Agent.agents` supervisor pattern. The preferred delivery sequence is research,
  implementation, then independent review, but the supervisor retains ownership,
  may skip stages that do not apply, and synthesizes every result; specialists do
  not delegate directly to one another and are not exposed as chat agents.
  All four agents have persisted display names, delegation descriptions,
  complete role instructions, source URLs, workspace/browser access, delegation
  eligibility, and tool membership in `popagent_agents`; `agent.ts` contains no
  role-prompt or capability fallback. `src/agent-role-seed.sql` applies the
  versioned initial role data from agency-agents once, preserving pre-existing
  guidance, after which PostgreSQL is canonical and edits apply on the next
  run. Custom tool groups and their contract tests live under `src/tools/`;
  tightly coupled secret operations intentionally share one module. No teams or
  workflows.
- Model access uses Mastra's provider interface; no direct vendor SDK imports
  appear in app code, so the provider stays swappable. Each execution records
  the selected route in Mastra `RequestContext`; delegated specialists resolve
  that route dynamically instead of falling back to the server default.
- `popagent_agent_runtime_settings` is the canonical singleton for the default
  model, supervisor/specialist step ceilings, concurrent tool-call limit,
  processor retries, final-response feedback, delegation context/result bounds,
  delegation failure/truncation text, and background-task concurrency/timing.
  One settings snapshot is carried through each execution. The final step is
  reserved for a response, specialist calls cannot raise their configured step
  ceiling, and suspension-capable calls remain sequential under Mastra's
  `called` concurrency policy.
- Browser, memory-compaction, and agent-runtime singleton rows are seeded exactly
  once by their versioned SQL migrations recorded in
  `popagent_data_migrations`. Their configurable columns have no database
  defaults, and startup never recreates a row after its migration marker exists.
  PostgreSQL is the only runtime source; a missing row is a configuration error,
  not a signal to restore code defaults.
- Delegation returns only specialist text to the supervisor: nested tool
  payloads and run metadata stay out of model context. Context slicing, result
  truncation, and failure feedback use the execution's persisted settings
  snapshot before hook audit and supervisor synthesis.
- `AgentWorkspaceStore` persists repository definitions in
  `popagent_agent_workspaces`. Repository paths are relative to
  `POPAGENT_WORKSPACE_ROOT` (default `workspace/`) and must resolve inside that
  root. Every session, background task, and schedule carries a `workspace_id`.
  Request context resolves the supervisor and implementer to a contained writable
  Mastra Workspace for that repository; researcher and reviewer resolve a
  read-only view of the same path with no shell tools. Writable file tools require
  a current tool-level read before overwriting or editing an existing file.
  `LocalFilesystem` containment and the writable `LocalSandbox` bubblewrap
  boundary prevent access outside the selected repository.
- Web search uses the local SearXNG container at `SEARXNG_URL` (default
  `http://127.0.0.1:8889`). The `webSearch` runtime tool (`createTool` ID
  `web-search`) requests SearXNG's JSON API
  and is available to the supervisor and every subagent. Keep search
  credential-free and self-hosted; do not add a hosted search API dependency.
- Browser automation uses open-source `@mastra/agent-browser` in fixed headless
  mode. Global enablement lives in `popagent_browser_settings`; per-agent
  `interactive`, `read-only`, or `none` access lives in `popagent_agents`.
  Each execution applies both current settings before Mastra resolves tools.
  Read-only browser instances omit click, typing, keyboard, select, dialog,
  drag, and JavaScript evaluation tools; navigation, snapshots, scrolling,
  waiting, and screenshots remain available. These exclusions are a
  non-writable safety boundary, not agent configuration.
- Agent Skills are inline Mastra skills resolved dynamically from
  `popagent_agent_skills` for the supervisor and each specialist on every run.
  The role data migration seeds two source-attributed, role-appropriate skills
  per agent; all skill instructions, references, provenance, and enablement live
  in PostgreSQL. Enabled skills provide Mastra's `skill`, `skill_read`, and
  `skill_search` tools; disabled skills remain editable but are excluded from
  agent execution. Skill names follow the Agent Skills lowercase-hyphen
  specification, references are JSONB, and user edits are never reseeded.
- Background tasks and schedules persist in `popagent_agent_tasks` and
  `popagent_agent_schedules`. Rows retain their originating workspace and
  materialized schedule tasks inherit the schedule workspace, so recovery never
  changes the repository boundary. On startup, the runtime reads persisted
  task concurrency, polling, timeout, and restart-staleness settings; a runtime
  settings update reconfigures polling and admission without restarting.
  Running tasks retain the timeout captured when they began. At restart,
  staleness is measured from `started_at`: stale work fails with error
  `stale after restart`; newer interrupted work is requeued and increments its
  recovery count. Timeout aborts the execution signal before persisting a failed
  result. Completion atomically handles both `running` and `cancelling`: a task
  finishing after cancellation is persisted as cancelled, never completed. API
  cancellation moves active work through an observable `cancelling` state,
  aborts execution through `AbortSignal`, and records `cancelled` only after the
  runtime acknowledges the abort. The Mastra stream receives this signal via
  `abortSignal`, so model and tool work can stop. Recurring schedules are
  materialized transactionally and committed tasks are enqueued only after that
  transaction succeeds.
- `src/agent-execution.ts` gives tasks the same prompt, delegation, and tool
  hooks, browser activation, cross-session recall, lifecycle processors,
  configured execution bounds, final-response guard, and episode retention as
  chat turns. The task runtime adds correlated `TaskStart`/`TaskStop` events
  around that execution. Schedule calculations use the server computer's
  `Intl.DateTimeFormat().resolvedOptions().timeZone`. Keep UI history limits
  separate from the unbounded queued-work recovery query.
- Durable memory uses `@mastra/memory` with `@mastra/pg` `PostgresStoreVNext`
  (`src/storage.ts`). `DATABASE_URL` is required for application and memory data;
  `OBSERVABILITY_DATABASE_URL` is required for the isolated observability domain.
  Production application PostgreSQL is `popagent-postgres` on `127.0.0.1:5433`;
  observability PostgreSQL is `popagent-observability-postgres` on
  `127.0.0.1:5435`. Never route observability writes into the application store.
- Repository schema initialization shares concurrent work, retains successful
  completion, and resets after a failed attempt so transient database outages do
  not poison a store until process restart.
- Session display metadata and AI SDK messages live in PostgreSQL. Sessions have
  explicit archive timestamps and a title lock so a user rename survives later
  message saves. Session rows retain their repository workspace; list APIs can
  filter by it. Mastra also persists each conversation as a memory thread. Do not
  reintroduce SQLite.
- Long conversations use Mastra Observational Memory in thread scope. The
  Observer compacts old messages and tool results into observations and the
  Reflector bounds the observation log; original messages remain in PostgreSQL.
  Settings in `popagent_memory_settings` control enablement, observation and
  reflection thresholds, recent raw-history retention, and idle buffering.
  Each request sends only its newest UI message because Mastra reconstructs
  history from the thread. Retrieval remains enabled for exact source-message
  recall. Observer and Reflector calls use the selected 9Router model rather
  than Mastra's hosted default.
- Permanent chat deletion uses Mastra's `Memory.deleteThread()` before removing
  thread state and the session row, so messages and vector artifacts follow the
  framework lifecycle. Cross-session retention semantics live in
  [AGENTS_MEMORY.md](AGENTS_MEMORY.md).
- Bulk chat cleanup requires a data-only backup of `popagent_sessions`,
  `mastra_threads`, `mastra_messages`, and `mastra_thread_state`. Reconcile
  orphaned `popagent-user` Mastra threads that have no `popagent_sessions` row;
  test chat requests can create such threads independently of UI sessions.
  Verify active/archived API lists and preserve `popagent_memories`.
- Bootstrap configuration comes from the ignored `.env`: `DATABASE_URL` for the
  Bun process, `POSTGRES_PASSWORD` for Docker Compose, and a base64-encoded
  32-byte `POPAGENT_SECRET_KEY` for secret-tool operations.
- Application-managed secrets live encrypted in `popagent_secrets`
  (`src/secrets.ts`), scoped by resource and exact name. AES-256-GCM authenticates
  the resource ID and name as additional data. Runtime tools `storeSecret`,
  `recallSecret`, and `updateSecret` create, read, and replace respectively; hook
  matchers use these registry keys rather than their hyphenated `createTool` IDs.
- The chat stream merges specialist iteration telemetry as transient AI SDK
  `data-agentActivity` parts beside the supervisor response. Persist those parts
  with the assistant message so reopened sessions retain delegation evidence.
  The AI SDK finish event's provider usage is attached under
  `metadata.custom.usage`; the UI reads `inputTokens` from that usage object.
  Every chat turn, task, and schedule execution also receives one valid Mastra
  trace ID. Chat assistant metadata persists it under `metadata.custom.traceId`
  beside the stable turn ID so feedback and hook audit can be correlated.
- Mastra observability is self-hosted through `MastraStorageExporter` and the
  dedicated PostgreSQL v-next observability domain. It records traces, automatic
  token/latency metrics, structured logs at info and above, and user feedback.
  Sampling is complete for this single-user workload. Only safe scalar request
  context keys are extracted; runtime settings and runtime objects are excluded.
  Mastra's sensitive-data filter must remain enabled, and secret-tool values must
  have a persisted-span redaction contract test. Read APIs force a maximum
  30-day time window and bounded pagination. Daily maintenance retains ordinary
  signals for `POPAGENT_OBSERVABILITY_RETENTION_DAYS` (default 30) and feedback
  for `POPAGENT_OBSERVABILITY_FEEDBACK_RETENTION_DAYS` (default 90). Estimated
  cost is not an application KPI until 9Router model IDs are verified against
  Mastra's pricing registry.
- API authentication is controlled by `POPAGENT_API_KEY`. When set, every `/api/*` request must include the matching `x-popagent-key`; the root page and static assets remain public. Secret recall results are redacted in hook/audit payloads while remaining available to the model.

- Session writes use optimistic revision locking: clients send `revision`; each successful save increments it atomically, and stale writes return `409 { error: "revision conflict", session }`. Session lists are capped at 200 rows.
- Mutation routes validate bounded JSON bodies and reject unknown models against the cached 9Router catalog; catalog outages fail open with a warning.
- Native long-term memory is separate from Mastra thread history. See
  [AGENTS_MEMORY.md](AGENTS_MEMORY.md). Hook protocol and policy live in
  [AGENTS_HOOKS.md](AGENTS_HOOKS.md).

## Contract-test exceptions

- Storage contracts require disposable local PostgreSQL.
- `src/agent.test.ts` and chat-stream scenarios require live 9Router models on
  Titan.