popagent

Latest change edb622d3ee540a33d881bd65e982a13f1381e0fd - Add signal management CLI support by Ólafur Búi Ólafsson

# Agent backend

Framework: **Mastra** (`@mastra/core`). Docs: https://mastra.ai/docs

## Model provider

- LLM gateway: local **9Router** (`http://127.0.0.1:20128/v1`, OpenAI-compatible),
  via `createOpenAICompatible` from `@ai-sdk/openai-compatible`.
  Env: `NINEROUTER_URL`, `NINEROUTER_API_KEY` (see `.env.example`).
- The default model is persisted in the singleton
  `popagent_agent_runtime_settings` row. `GET /api/models` merges that value with
  the gateway's current model IDs and context-window metadata; do not hard-code
  the catalog, default, or context limits in UI code.
- Per-request selection: `POST /api/chat` accepts an optional `model` body field;
  when omitted, the server reads the persisted default.
- The local `qwen3.5-4b` llama.cpp route accepts one leading system message.
  The provider request adapter merges only that model's consecutive leading
  string system messages; other models and malformed/non-string requests pass
  through unchanged. Bun's maximum 255-second idle timeout leaves enough time
  for slow local models to emit their first stream event.

## HTTP server (`src/server.ts`)

`Bun.serve` with `routes`:

- `/` — the chat UI (`src/ui/index.html`, bundled by Bun at serve time).
- `GET /api/models` — `{ models, contextWindows, defaultModel }` from 9Router.
- `GET/POST /api/sessions` — list active/archived or create sessions.
- `DELETE /api/workspaces/:id/sessions` — permanently deletes every active and
  archived chat in one workspace through the normal per-session lifecycle.
- Workspace registration and documentation CRUD/index/search routes are owned by [AGENTS_DOCUMENTATION.md](AGENTS_DOCUMENTATION.md).
- Browser settings, health, sessions, takeover, profiles, recordings, and screencast routes are owned by [AGENTS_BROWSER.md](AGENTS_BROWSER.md).
- Native channel settings, history, event, post, and cancellation routes are owned by [AGENTS_CHANNELS.md](AGENTS_CHANNELS.md).
- Observability overview, traces, logs, health, and feedback routes are owned by [AGENTS_OBSERVABILITY.md](AGENTS_OBSERVABILITY.md).
- `GET /api/health` — protected deployment readiness for application and observability PostgreSQL. Model, search, and browser outages are intentionally not deployment blockers.
- `GET/PUT/PATCH/DELETE /api/sessions/:id` — load/save, rename, archive/restore,
  or permanently delete a session and its matching Mastra thread data.
- `GET /api/sessions/:id/hooks` — ordered hook audit runs for the session.
- `GET/POST /api/memories` and `PATCH/DELETE /api/memories/:id` — long-term
  memory management; retention and mutability semantics live in
  [AGENTS_MEMORY.md](AGENTS_MEMORY.md).
- `POST /api/chat` — AI SDK UI-message stream via `handleChatStream` from
  `@mastra/ai-sdk` + `createUIMessageStreamResponse` from `ai`. The route accepts
  only the session id, optional selected model, workspace, and newest UI message
  from the client; callers cannot inject Mastra tools or execution options. The
  chat id maps to the `popagent-user` memory thread. A typed `RequestContext`
  carries one database-backed runtime-settings snapshot through the supervisor
  and delegated specialists. The route also dispatches managed lifecycle hooks,
  injects native cross-session recall, and adds turn IDs to assistant metadata
  so the UI can correlate hook activity.
- `GET/PATCH /api/settings/memory` — read or update persisted Observational
  Memory compaction settings; changes apply on the next agent turn.
- `GET/PATCH /api/settings/agent-runtime` — read or update the default model,
  execution/delegation bounds and feedback, and background-task controls.
- `GET/PATCH /api/settings/autonomy` — read or update the persisted evolution,
  skill-creation, and contained self-update controls.
- `GET /api/autonomy/signals`, `POST /api/autonomy/signals/:id/ignore`,
  `POST /api/autonomy/signals/:id/retry`, `GET /api/autonomy/revisions`, and
  `POST /api/autonomy/revisions/:id/revert` — inspect bounded recent evidence,
  ignore pending/dead-letter evidence, retry dead-letter evidence from attempt zero,
  inspect append-only learned changes, or append a compensating revert.
- `GET /api/agents` and `PATCH /api/agents/:id` — list persisted system agents
  and update their names, delegation descriptions, instructions, workspace and
  browser access, delegation eligibility, and tool membership.
- `GET/PATCH /api/settings/browser` — read or toggle the global persisted browser
  capability; changes apply immediately without restarting.
- `GET/POST /api/agents/:id/skills` and
  `PATCH/DELETE /api/agents/:id/skills/:skillId` — manage database-backed Agent
  Skills.
- `GET/POST /api/tasks` and `POST /api/tasks/:id/cancel` — manage persistent
  background agent work.
- `GET/POST /api/schedules` and `PATCH/DELETE /api/schedules/:id` — manage
  recurring schedules. Clients submit cron expressions; timezone is
  server-controlled and is not part of the writable API.


Wire DTOs shared with the UI live in `src/api-types.ts`; do not redeclare those
response shapes in React code.

## Decisions

- One user-facing agent (`popagent`) supervises three internal Mastra subagents:
  `researcher`, `implementer`, and `reviewer`, using Mastra's documented
  `Agent.agents` supervisor pattern. The preferred delivery sequence is research,
  implementation, then independent review, but the supervisor retains ownership,
  may skip stages that do not apply, and synthesizes every result; specialists do
  not delegate directly to one another and are not exposed as chat agents.
  All four agents have persisted display names, delegation descriptions,
  complete role instructions, source URLs, workspace/browser access, delegation
  eligibility, and tool membership in `popagent_agents`; `agent.ts` contains no
  role-prompt or capability fallback. `src/agent-role-seed.sql` applies the
  versioned initial role data from agency-agents once, preserving pre-existing
  guidance, after which PostgreSQL is canonical and edits apply on the next
  run. Custom tool groups and their contract tests live under `src/tools/`;
  tightly coupled secret operations intentionally share one module. No teams or
  workflows.
- Each persisted agent profile has an optional model override. The read-only
  Researcher and Reviewer default to the proven low-latency
  `cx/gpt-5.3-codex-spark`; the Orchestrator and Implementer inherit the
  request-selected or persisted default model. Agent Settings applies model
  edits on the next run.
- Model access uses Mastra's provider interface; no direct vendor SDK imports
  appear in app code, so the provider stays swappable. Each execution records
  the selected route in Mastra `RequestContext`; delegated specialists resolve
  that route dynamically instead of falling back to the server default.
- `popagent_agent_runtime_settings` is the canonical singleton for the default
  model, supervisor/specialist step ceilings, concurrent tool-call limit,
  processor retries, final-response feedback, delegation context/result bounds,
  delegation failure/truncation text, and background-task concurrency/timing.
  One settings snapshot is carried through each execution. The final step is
  reserved for a response, specialist calls cannot raise their configured step
  ceiling, and suspension-capable calls remain sequential under Mastra's
  `called` concurrency policy.
- Browser, memory-compaction, agent-runtime, and autonomy singleton rows are
  seeded exactly once by versioned SQL migrations recorded in
  `popagent_data_migrations`. Their configurable columns have no database
  defaults, and startup never recreates a row after its migration marker exists.
  PostgreSQL is the only runtime source; a missing row is a configuration error,
  not a signal to restore code defaults.
- Delegation returns only specialist text to the supervisor: nested tool
  payloads and run metadata stay out of model context. Context slicing, result
  truncation, and failure feedback use the execution's persisted settings
  snapshot before hook audit and supervisor synthesis.
- `AgentWorkspaceStore` persists repository definitions in
  `popagent_agent_workspaces`. Repository paths are relative to
  `POPAGENT_WORKSPACE_ROOT` (default `workspace/`) and must resolve inside that
  root. The default registration points at the ignored `workspace/popagent` Git
  checkout; `deploy.sh` clones the public repository when that checkout is absent.
  Every session, background task, and schedule carries a `workspace_id`.
  Request context resolves the supervisor and implementer to a contained writable
  Mastra Workspace for that repository; researcher and reviewer resolve a
  read-only view of the same path with no shell tools. Every role receives
  Mastra's native `mastra_workspace_lsp_inspect` tool for TypeScript/JavaScript
  hover, diagnostics, definitions, and implementations. Agent tool inputs remain
  repository-relative. A contained `LocalFilesystem.exists` adapter translates
  only Mastra's absolute internal LSP marker probes back beneath the repository;
  other absolute and escaping paths remain rejected. LSP servers use a host
  process manager rooted at the contained repository because bubblewrap cannot
  expose host absolute URIs to the language server; agent shell commands remain
  in the network-disabled `LocalSandbox`. The application pins
  `typescript-language-server`.
  Writable file tools require a current tool-level read before overwriting or
  editing an existing file. `LocalFilesystem` containment and the bubblewrap
  boundary prevent access outside the selected repository.
- Failed, cancelled, and dead-lettered tasks can be operator-resolved through `POST /api/tasks/:id/resolve` with a bounded evidence summary. Resolution preserves the task record, changes it to completed, and clears stale failure/retry fields; queued, running, and already-completed tasks reject resolution.
- PostgreSQL-backed project-folder Markdown documentation and workspace-filtered RAG use Mastra `MDocument`, AI SDK embeddings, `PgVector`, and local-model reranking; source, indexing, and retrieval contracts live in [AGENTS_DOCUMENTATION.md](AGENTS_DOCUMENTATION.md).
- Web search uses the local SearXNG container at `SEARXNG_URL` (default
  `http://127.0.0.1:8889`). The `webSearch` runtime tool (`createTool` ID
  `web-search`) requests SearXNG's JSON API
  and is available to the supervisor and every subagent. Keep search
  credential-free and self-hosted; do not add a hosted search API dependency.
- Browser automation, network policy, profiles, recordings, session lifecycle, and verification contracts live in [AGENTS_BROWSER.md](AGENTS_BROWSER.md).
- Agent Skills are inline Mastra skills resolved dynamically from
  `popagent_agent_skills` for the supervisor and each specialist on every run.
  The role data migration seeds two source-attributed, role-appropriate skills
  per agent; all skill instructions, references, provenance, and enablement live
  in PostgreSQL. Enabled skills provide Mastra's `skill`, `skill_read`, and
  `skill_search` tools; disabled skills remain editable but are excluded from
  agent execution. Skill names follow the Agent Skills lowercase-hyphen
  specification, references are JSONB, and ownership is exposed as
  `evolutionManaged` in the Agent Skill DTO. Every ordinary user/API update
  clears `evolution_managed`; once edited, a learned skill is user-owned and
  evolution cannot reclaim or revert it.
- Autonomous learning persists in `popagent_evolution_signals`,
  `popagent_learned_overlays`, and append-only `popagent_evolution_revisions`.
  Each signal retains the real turn trace ID plus bounded, secret-redacted
  Goal/Outcome or Goal/Failure evidence with a failure classification.
  Signals are deduplicated before a periodic worker claims due batches with
  `FOR UPDATE SKIP LOCKED`; every run first recovers abandoned claims, and
  concurrent worker start/stop transitions are serialized.
- Reflection uses a tool-free Mastra agent and the configured 9Router model.
  It consumes Mastra's streaming structured-output contract because 9Router
  returns SSE for this route. The model can select only a closed, code-owned
  strategy enum; it cannot author
  prompt or skill text. Every selection must cite claimed evidence for the same
  agent. Application locks and rereads the autonomy singleton and
  rechecks cancellation in the same transaction as learned mutation and signal
  completion. A disabled global or target policy wins before mutation: claimed
  signals become ignored and the claim attempt is released rather than retried
  or dead-lettered. Other failures use capped exponential backoff and terminal
  dead-lettering.
- Learned overlays are stored separately from `popagent_agents.instructions`
  and composed after the immutable user-managed base on every agent resolution.
  Automatic skill changes are limited to rows marked `evolution_managed`; a
  user-managed skill with the same name is never overwritten. Automatic learning
  cannot change tool membership, workspace/browser access, secrets, hooks,
  authentication, execution bounds, or user-managed instructions. Secret-bearing
  and protected-boundary proposals fail without partial mutation. Reverts append
  a compensating revision with `revertsRevisionId`; a partial unique index permits
  at most one compensation per source revision. Revert restores the prior
  effective overlay or skill only when that source is still the latest target
  version and the current managed target still matches it.
- Background tasks and schedules persist in `popagent_agent_tasks` and
  `popagent_agent_schedules`. Both carry immutable `source: 'user' |
  'self-update'` ownership and their originating workspace; materialized tasks
  inherit both fields, so retry and recovery never change provenance or the
  repository boundary.
  Tasks persist `attempt_count`, `max_attempts` (default three),
  `next_attempt_at`, `last_error_class`, and `dead_lettered_at`. Due
  queued claims atomically increment attempts. Only contained `self-update`
  tasks may automatically retry timeout, provider/rate-limit, browser, database,
  or infrastructure failures: every attempt starts from a fresh clone and only
  a verified commit can escape it. User tasks fail after an execution error
  rather than replaying unknown side effects. Hook denial, cancellation/abort,
  validation, permission/containment, missing configuration, and unknown
  failures never retry; exhausted self-update work becomes terminal
  `dead-letter` with durable explanatory progress.
- Task polling rediscovers delayed work. Restart recovery preserves the prompt,
  model, session, schedule, and workspace; it requeues only non-stale running
  work below `maxAttempts`, dead-letters non-stale exhausted work, fails stale
  work, and finalizes cancelling work as cancelled. Runtime start/stop operations
  are serialized; a stop requested during startup lets initialization and one
  recovery pass finish but prevents polling or execution from being resurrected.
- Each claimed attempt retains its executor promise. Timeout, cancellation, and
  shutdown abort the signal, then wait a bounded acknowledgement interval while
  retaining the occupied execution slot. Cancellation becomes durable only after
  executor settlement; timeout becomes retry-eligible only after settlement. An
  executor that ignores abort remains non-runnable in `running` or `cancelling`,
  emits `interrupted`, holds its slot, and blocks same-runtime restart recovery
  rather than overlapping a retry.
- Durable completion, failure, dead-letter, and cancellation transitions precede
  terminal lifecycle delivery. Every attempt uses one correlated turn and trace
  ID; completed events carry the outcome and failures carry their classified
  error. `TaskStop`, channel, hook, browser-close, and evolution observers are
  best effort after the transition: observer failure is logged and cannot emit a
  contradictory failed terminal state. Intermediate retries still emit failed
  `TaskStop` before a later `TaskStart`; final exhaustion emits `dead-letter`.
- `DELETE /api/tasks/:id` removes only terminal `completed`, `failed`,
  `cancelled`, or `dead-letter` rows. It returns conflict for active work and
  not-found for missing rows; task removal never cancels execution or rolls
  back repository changes.
- Schedules persist `maxAttempts`; create/update preserve it and transactional
  materialization copies it into each task under row locking, with enqueue only
  after commit. The retry/schema migration holds a PostgreSQL advisory
  transaction lock, rechecks its version marker after locking, and records the
  marker only after the schema and one-time timezone/next-run backfill succeed.
  Ordinary initialization never recalculates an existing schedule after that
  marker. `src/agent-execution.ts` gives task work the same prompt,
  delegation/tool hooks, browser activation, recall, lifecycle processors,
  execution bounds, final-response guard, and episode retention as chat turns.
  Schedule calculations use the server timezone; keep UI history limits separate
  from the unbounded queued-work recovery query.
- `SelfUpdateScheduler` serially reconciles one `source='self-update'` schedule
  per registered workspace and cadence. A PostgreSQL partial unique index owns
  that `(workspace_id)` slot; ordinary schedule create is always `source='user'`
  and ordinary update/delete match only user rows, so display names cannot claim
  or mutate system schedules.
  Scheduler lifecycle operations share one serialized subscription/reconcile/timer
  state; a stop requested during initial reconciliation prevents later timer
  resurrection, and a subsequent explicit start can initialize again.
- Scheduling requires enabled global/self-update policy, read-write supervisor
  and implementer, delegation-eligible implementer and reviewer, reviewer
  repository access, and a resolved writable Git repository. Materialization
  copies source and `maxAttempts` while atomically rereading enabled autonomy
  policy; disabling self-update therefore blocks new work before reconciliation.
  Trusted self-update tasks propagate `executionSource='self-update'`.
- Execution first requires a clean registered checkout and captures its HEAD,
  branch, and remote digest, then works in a secret-free, network-disabled local
  clone on a unique `autonomous/self-update/*` branch. Fixed runtime policy,
  independent of prompts and managed hooks, removes memory mutation and
  `useBrowserSecret`; denies hidden/credential/protected paths and remote,
  publish, deploy, restart, or unrestricted shell operations; and keeps reviewer
  workspace access read-only. The registered checkout is never the agent's
  writable surface.
- Completion requires successful implementer and reviewer delegations and accepts
  only a clean no-op or a clean descendant commit on the prepared branch with
  unchanged remotes and an allowed final diff. Accepted commits are imported as
  local refs without changing the registered checkout; rejected, cancelled, or
  failed work removes the contained clone and cannot be marked complete. The
  runtime never pushes, publishes, deploys, restarts, changes remotes, or exposes
  Git/browser credentials.
- Durable memory uses `@mastra/memory` with `@mastra/pg` `PostgresStoreVNext`
  (`src/storage.ts`). `DATABASE_URL` is required for application and memory data;
  `OBSERVABILITY_DATABASE_URL` is required for the isolated observability domain.
  Production application PostgreSQL is `popagent-postgres` on `127.0.0.1:5433`;
  observability PostgreSQL is `popagent-observability-postgres` on
  `127.0.0.1:5435`. Never route observability writes into the application store.
- Repository schema initialization shares concurrent work, retains successful
  completion, and resets after a failed attempt so transient database outages do
  not poison a store until process restart.
- Session display metadata and AI SDK messages live in PostgreSQL. Sessions have
  explicit archive timestamps and a title lock so a user rename survives later
  message saves. Session rows retain their repository workspace; list APIs can
  filter by it. Mastra also persists each conversation as a memory thread. Do not
  reintroduce SQLite.
- Long conversations use Mastra Observational Memory in thread scope. The
  Observer compacts old messages and tool results into observations and the
  Reflector bounds the observation log; original messages remain in PostgreSQL.
  Settings in `popagent_memory_settings` control enablement; observation and
  reflection thresholds; raw-history retention; async observation/reflection
  buffering and safety thresholds; idle/provider-change activation; shared
  budgets; previous-observation context limits; temporal markers; attachment
  forwarding; exact-history retrieval scope; and custom Observer/Reflector
  guidance. Shared budgets disable async buffering as required by Mastra.
  Each request sends only its newest UI message because Mastra reconstructs
  history from the thread. Observations remain thread-scoped; retrieval can be
  restricted to that thread or browse all threads for `popagent-user`. Observer
  and Reflector calls use the selected 9Router model rather than Mastra's hosted
  default. Configuration changes apply on the next turn.
- Delegated specialist messages use isolated generated resource/thread IDs. The pinned `@mastra/core` patch backfills those IDs onto provider response messages before persistence; removing it reintroduces `Thread ID is required` observability errors. `src/agent.test.ts` exercises the real delegated streaming path.
- Permanent chat deletion uses Mastra's `Memory.deleteThread()` before removing
  thread state and the session row, so messages and vector artifacts follow the
  framework lifecycle. Workspace-wide deletion runs that same lifecycle for
  every active and archived session, including SessionEnd hooks, browser closure,
  and hook-audit cleanup. Cross-session retention semantics live in
  [AGENTS_MEMORY.md](AGENTS_MEMORY.md).
- Administrative direct-database cleanup requires a data-only backup of
  `popagent_sessions`, `mastra_threads`, `mastra_messages`, and
  `mastra_thread_state`. Reconcile orphaned `popagent-user` Mastra threads that
  have no `popagent_sessions` row; test chat requests can create such threads
  independently of UI sessions. Verify active/archived API lists and preserve
  `popagent_memories`.
- Bootstrap configuration comes from the ignored `.env`: `DATABASE_URL` for the
  Bun process, `POSTGRES_PASSWORD` for Docker Compose, and a base64-encoded
  32-byte `POPAGENT_SECRET_KEY` for application-managed secret encryption.
- Application-managed secrets live encrypted in `popagent_secrets`
  (`src/secrets.ts`), scoped by resource and exact name. AES-256-GCM authenticates
  the resource ID and name as additional data. `popagent_secret_bindings` stores
  metadata-only exact HTTPS origin bindings. The authenticated `/api/secrets`
  interface provisions, rotates, binds, lists, and deletes credentials without
  returning values.
- Agents have no plaintext secret create, update, or recall tools.
  `useBrowserSecret` resolves a bound value server-side and fills only a browser
  element whose page and frame origins both equal the configured origin. Its
  model-visible result contains only the secret name, destination origin, and
  success status.
- The chat stream merges specialist iteration telemetry as transient AI SDK
  `data-agentActivity` parts beside the supervisor response. Persist those parts
  with the assistant message so reopened sessions retain delegation evidence.
  Usage and trace correlation metadata are defined in [AGENTS_OBSERVABILITY.md](AGENTS_OBSERVABILITY.md).
- Observability storage isolation, correlation, redaction, APIs, and retention live in [AGENTS_OBSERVABILITY.md](AGENTS_OBSERVABILITY.md).
- API authentication is controlled by `POPAGENT_API_KEY`. When set, every `/api/*` request must include the matching `x-popagent-key`; the root page and static assets remain public. Retired plaintext secret tool arguments/results are centrally redacted at chat ingress, session and Mastra-message persistence, hook dispatch, and transcript rendering.

- Session writes use optimistic revision locking: clients send `revision`; each successful save increments it atomically, and stale writes return `409 { error: "revision conflict", session }`. Session lists are capped at 200 rows.
- Mutation routes validate bounded JSON bodies and reject unknown models against the cached 9Router catalog; catalog outages fail open with a warning.
- Native long-term memory is separate from Mastra thread history. See
  [AGENTS_MEMORY.md](AGENTS_MEMORY.md). Hook protocol and policy live in
  [AGENTS_HOOKS.md](AGENTS_HOOKS.md).

## Contract-test exceptions

- `src/models.test.ts` owns the local Qwen request-shape compatibility contract;
  `src/server.test.ts` owns the slow-stream idle-timeout configuration.
- Storage contracts require disposable local PostgreSQL.
- `src/agent.test.ts` and chat-stream scenarios require live 9Router models on
  Titan.


## Live BifrOSt tool

Agents with persisted browser access receive `bifrostNavigator`, a single typed gateway to the local visible BifrOSt Navigator MCP catalog. The tool accepts an exact `browser_*` operation plus its catalog arguments, preserves MCP text/image/resource content, enforces `interactive` versus `read-only` role access server-side, and is absent during trusted self-update. BifrOSt remains a user-owned desktop process; Popagent connects only to its protected Unix socket and does not launch or stop it.