popagent

Latest change e0970dab600e00afab3d85491d7db6318b584557 - Prevent false autonomous containment failures by AkurAI Build

# Agent backend

Framework: **Mastra** (`@mastra/core`). Docs: https://mastra.ai/docs

## Model provider

- LLM gateway: local **9Router** (`http://127.0.0.1:20128/v1`, OpenAI-compatible),
  via `createOpenAICompatible` from `@ai-sdk/openai-compatible`.
  Env: `NINEROUTER_URL`, `NINEROUTER_API_KEY` (see `.env.example`).
- The default model and model source are persisted in the singleton
  `popagent_agent_runtime_settings` row. The source can use the configured
  9Router model, force Titan's local Ornith route, or force 9Router's
  `auto/coding:free` route backed by currently available OpenRouter free coding
  models. `GET /api/models` merges the effective default with the gateway's
  current model IDs and context-window metadata; do not hard-code the catalog,
  default, or context limits in UI code.
- Per-request selection: `POST /api/chat` accepts an optional `model` body field;
  when omitted, the server reads the persisted effective default.
- Titan's effective production source is local
  `titan/ornith-1.0-9b-mtp-q4_k_m`. The `researcher`, `implementer`, and
  `reviewer` profiles also use that route for token-heavy local coding work.
  Ornith runs with two independent 65,536-token llama.cpp contexts.
- `src/models.ts` admits at most `POPAGENT_LOCAL_MODEL_CONCURRENCY` local
  generations at once (default `2`, matching llama.cpp). Additional local calls
  wait FIFO without their own timeout, aborted waiters are removed, and a slot
  remains leased until the response stream ends or is cancelled. Remote models
  bypass this queue.
- A changed llama.cpp alias becomes selectable only after OmniRoute imports the
  upstream `/models` catalog.
- Bun's maximum 255-second idle timeout leaves enough time for slow local models
  to emit their first stream event.

## HTTP server (`src/server.ts`)

`Bun.serve` with `routes`:

- `/` — the chat UI (`src/ui/index.html`, bundled by Bun at serve time).
- `GET /api/models` — `{ models, contextWindows, defaultModel }` from 9Router.
- `GET/POST /api/sessions` — list active/archived or create sessions.
- `DELETE /api/workspaces/:id/sessions` — permanently deletes every active and
  archived chat in one workspace through the normal per-session lifecycle.
- Workspace registration and documentation CRUD/index/search routes are owned by [AGENTS_DOCUMENTATION.md](AGENTS_DOCUMENTATION.md).
- Browser settings, health, sessions, takeover, profiles, recordings, and screencast routes are owned by [AGENTS_BROWSER.md](AGENTS_BROWSER.md).
- Native channel settings, history, event, post, and cancellation routes are owned by [AGENTS_CHANNELS.md](AGENTS_CHANNELS.md).
- Observability overview, traces, logs, health, and feedback routes are owned by [AGENTS_OBSERVABILITY.md](AGENTS_OBSERVABILITY.md).
- `GET /api/health` — protected deployment readiness for application and observability PostgreSQL. Model, search, and browser outages are intentionally not deployment blockers.
- `GET/PUT/PATCH/DELETE /api/sessions/:id` — load/save, rename, archive/restore,
  or permanently delete a session and its matching Mastra thread data.
- `GET /api/sessions/:id/hooks` — ordered hook audit runs for the session.
- `GET/POST /api/memories` and `PATCH/DELETE /api/memories/:id` — long-term
  memory management; retention and mutability semantics live in
  [AGENTS_MEMORY.md](AGENTS_MEMORY.md).
- `POST /api/chat` — AI SDK UI-message stream via `handleChatStream` from
  `@mastra/ai-sdk` + `createUIMessageStreamResponse` from `ai`. The route accepts
  only the session id, optional selected model, workspace, and newest UI message
  from the client; callers cannot inject Mastra tools or execution options. The
  chat id maps to the `popagent-user` memory thread. A typed `RequestContext`
  carries one database-backed runtime-settings snapshot through the supervisor
  and delegated specialists. The route also dispatches managed lifecycle hooks,
  injects native cross-session recall, and adds turn IDs to assistant metadata
  so the UI can correlate hook activity.
- `GET/PATCH /api/settings/memory` — read or update persisted Observational
  Memory compaction settings; changes apply on the next agent turn.
- `GET/PATCH /api/settings/agent-runtime` — read or update the default model,
  execution/delegation bounds and feedback, and background-task controls.
- `GET/PATCH /api/settings/autonomy` — read or update the persisted evolution,
  skill-creation, and contained self-update controls.
- `GET /api/autonomy/signals`, `POST /api/autonomy/signals/:id/ignore`,
  `POST /api/autonomy/signals/:id/retry`, `GET /api/autonomy/revisions`, and
  `POST /api/autonomy/revisions/:id/revert` — inspect bounded recent evidence,
  ignore pending/dead-letter evidence, retry dead-letter evidence from attempt zero,
  inspect append-only learned changes, or append a compensating revert.
- `GET /api/agents` and `PATCH /api/agents/:id` — list persisted system agents
  and update their names, delegation descriptions, instructions, workspace and
  browser access, delegation eligibility, and tool membership.
- `GET/PATCH /api/settings/browser` — read or toggle the global persisted browser
  capability; changes apply immediately without restarting.
- `GET/POST /api/agents/:id/skills` and
  `PATCH/DELETE /api/agents/:id/skills/:skillId` — manage database-backed Agent
  Skills.
- `GET/POST /api/tasks` and `POST /api/tasks/:id/cancel` — manage persistent
  background agent work; `GET /api/tasks/:id/workflow` returns the bounded
  ordered self-update phase detail and evidence projection.
- `GET/POST /api/schedules` and `PATCH/DELETE /api/schedules/:id` — manage
  recurring schedules. Clients submit cron expressions; timezone is
  server-controlled and is not part of the writable API.


Wire DTOs shared with the UI live in `src/api-types.ts`; do not redeclare those
response shapes in React code.

## Decisions

- One user-facing agent (`orchistrator`) supervises the coding specialists
  `researcher`, `implementer`, and `reviewer`, plus governed AkurAI Build
  specialists `build-maintainer`, `build-release-manager`, and
  `community-steward`, using Mastra's documented `Agent.agents` supervisor
  pattern. The preferred coding delivery sequence is research, implementation,
  then independent review; Build delegation is role-specific and never weakens
  the coding specialists' workspace or browser boundaries. Specialists do not
  delegate directly to one another and are not exposed as chat agents.
  Autonomous self-update execution additionally uses a deterministic Mastra
  workflow that owns the ordered `prepare`, `research`, `decide`, `implement`,
  `inspect-change`, `verify`, `review`, and `commit` phases. Typed handoffs and
  the accepted-review gate prevent later phases from running after a failure;
  a no-op decision marks later phases skipped.
- All persisted system agents have display names, delegation descriptions,
  complete role instructions, source URLs, workspace/browser access,
  delegation eligibility, and tool membership in `popagent_agents`; `agent.ts`
  contains no role-prompt or capability fallback. `src/agent-role-seed.sql`
  applies versioned initial role data once, preserving pre-existing guidance,
  after which PostgreSQL is canonical and edits apply on the next run. Custom
  tool groups and their contract tests live under `src/tools/`; tightly coupled
  secret operations intentionally share one module.
- Each persisted agent profile has an optional model override. The read-only
  Researcher and Reviewer default to the proven low-latency
  `cx/gpt-5.3-codex-spark`; the Orchestrator and Implementer inherit the
  request-selected or persisted default model. Agent Settings applies model
  edits on the next run.
- Model access uses Mastra's provider interface; no direct vendor SDK imports
  appear in app code, so the provider stays swappable. Each execution records
  the selected route in Mastra `RequestContext`; delegated specialists resolve
  that route dynamically instead of falling back to the server default.
- `popagent_agent_runtime_settings` is the canonical singleton for the model
  source and effective default model, supervisor/specialist step ceilings,
  concurrent tool-call limit, processor retries, final-response feedback,
  delegation context/result bounds, delegation failure/truncation text, and
  background-task concurrency/timing.
  One settings snapshot is carried through each execution. The final step is
  reserved for a response, specialist calls cannot raise their configured step
  ceiling, and suspension-capable calls remain sequential under Mastra's
  `called` concurrency policy.
- Browser, memory-compaction, agent-runtime, and autonomy singleton rows are
  seeded exactly once by versioned SQL migrations recorded in
  `popagent_data_migrations`. Their configurable columns have no database
  defaults, and startup never recreates a row after its migration marker exists.
  PostgreSQL is the only runtime source; a missing row is a configuration error,
  not a signal to restore code defaults.
- Delegation returns only specialist text to the supervisor: nested tool
  payloads and run metadata stay out of model context. Context slicing, result
  truncation, and failure feedback use the execution's persisted settings
  snapshot before hook audit and supervisor synthesis.
- `AgentWorkspaceStore` persists repository definitions in
  `popagent_agent_workspaces`. Repository paths are relative to
  `POPAGENT_WORKSPACE_ROOT` (default `workspace/`) and must resolve inside that
  root. The default registration points at the ignored `workspace/popagent` Git
  checkout; `deploy.sh` clones the public repository when that checkout is absent.
  Every session, background task, and schedule carries a `workspace_id`.
  Request context resolves the supervisor and implementer to a contained writable
  Mastra Workspace for that repository; researcher and reviewer resolve a
  read-only view of the same path with no shell tools. Every role receives
  Mastra's native `mastra_workspace_lsp_inspect` tool for TypeScript/JavaScript
  hover, diagnostics, definitions, and implementations. Agent tool inputs remain
  repository-relative. A contained `LocalFilesystem.exists` adapter translates
  only Mastra's absolute internal LSP marker probes back beneath the repository;
  other absolute and escaping paths remain rejected. LSP servers use a host
  process manager rooted at the contained repository because bubblewrap cannot
  expose host absolute URIs to the language server; agent shell commands remain
  in the network-disabled `LocalSandbox`. The application pins
  `typescript-language-server`.
  Writable file tools require a current tool-level read before overwriting or
  editing an existing file. `LocalFilesystem` containment and the bubblewrap
  boundary prevent access outside the selected repository.
- `repoBrief` is a server-native, read-only deterministic repository-inspection tool. It resolves the selected contained workspace, uses fixed Git inspection plus TypeScript parsing, and returns only bounded path, export-symbol, adjacent-test, and change-state evidence; it never calls a model, embeddings, network, or request-derived shell. `AgentExecutionRuntime` injects a successful brief before the first model call and sets a request-context flag that withholds Researcher for that turn only when the brief is ready for implementation. Self-update execution does not preflight and retains its fixed research policy. `GET /api/workspaces/:id/repo-brief?q=` and `popagent repo-brief <workspace> <query>` expose the same service to operators; agents must use the native tool rather than the HTTP CLI because their sandbox is network-disabled.
- Failed, cancelled, and dead-lettered tasks can be operator-resolved through `POST /api/tasks/:id/resolve` with a bounded evidence summary. Resolution preserves the task record, changes it to completed, and clears stale failure/retry fields; queued, running, and already-completed tasks reject resolution.
- PostgreSQL-backed project-folder Markdown documentation and workspace-filtered RAG use Mastra `MDocument`, AI SDK embeddings, `PgVector`, and local-model reranking; source, indexing, and retrieval contracts live in [AGENTS_DOCUMENTATION.md](AGENTS_DOCUMENTATION.md).
- Web search uses the local SearXNG container at `SEARXNG_URL` (default
  `http://127.0.0.1:8889`). The `webSearch` runtime tool (`createTool` ID
  `web-search`) requests SearXNG's JSON API
  and is available to the supervisor and every subagent. Keep search
  credential-free and self-hosted; do not add a hosted search API dependency.
- Browser automation, network policy, profiles, recordings, session lifecycle, and verification contracts live in [AGENTS_BROWSER.md](AGENTS_BROWSER.md).
- Agent Skills are inline Mastra skills resolved dynamically from
  `popagent_agent_skills` for the supervisor and each specialist on every run.
  The role data migration seeds two source-attributed, role-appropriate skills
  per agent; all skill instructions, references, provenance, and enablement live
  in PostgreSQL. Enabled skills provide Mastra's `skill`, `skill_read`, and
  `skill_search` tools; disabled skills remain editable but are excluded from
  agent execution. Skill names follow the Agent Skills lowercase-hyphen
  specification, references are JSONB, and ownership is exposed as
  `evolutionManaged` in the Agent Skill DTO. Every ordinary user/API update
  clears `evolution_managed`; once edited, a learned skill is user-owned and
  evolution cannot reclaim or revert it.
- Autonomous learning persists in `popagent_evolution_signals`,
  `popagent_learned_overlays`, and append-only `popagent_evolution_revisions`.
  Each signal retains the real turn trace ID plus bounded, secret-redacted
  Goal/Outcome or Goal/Failure evidence with a failure classification.
  Signals are deduplicated before a periodic worker claims due batches with
  `FOR UPDATE SKIP LOCKED`; every run first recovers abandoned claims, and
  concurrent worker start/stop transitions are serialized.
- Reflection uses a tool-free Mastra agent and the configured 9Router model.
  It consumes Mastra's streaming structured-output contract because 9Router
  returns SSE for this route. The model can select only a closed, code-owned
  strategy enum; it cannot author prompt or skill text. With automatic fact
  retention enabled, the `retain-fact` strategy may author only a bounded
  operational fact and semantic key established by cited evidence for the same
  agent. Application locks and rereads the autonomy singleton and rechecks
  cancellation in the same transaction as learned mutation and signal
  completion. A disabled global or target policy wins before mutation: claimed
  signals become ignored and the claim attempt is released rather than retried
  or dead-lettered. Other failures use capped exponential backoff and terminal
  dead-lettering.
- Learned overlays are stored separately from `popagent_agents.instructions`
  and composed after the immutable user-managed base on every agent resolution.
  Automatic skill changes are limited to rows marked `evolution_managed`; a
  user-managed skill with the same name is never overwritten. Evolution-managed
  facts use `evolution.<agent-id>.<semantic-key>` keys in the fixed
  `popagent-user` resource and retain their cited signal's session provenance.
  Automatic learning cannot change tool membership, workspace/browser access,
  secrets, hooks, authentication, execution bounds, or user-managed
  instructions. Secret-bearing and protected-boundary proposals fail without
  partial mutation. Reverts append a compensating revision with
  `revertsRevisionId`; a partial unique index permits at most one compensation
  per source revision. Revert restores the prior effective overlay, skill, or
  fact only when that source is still the latest target version and the current
  managed target still matches it.
- Background tasks and schedules persist in `popagent_agent_tasks` and
  `popagent_agent_schedules`. Both carry immutable `source: 'user' |
  'self-update'` ownership and their originating workspace; materialized tasks
  inherit both fields, so retry and recovery never change provenance or the
  repository boundary.
  Tasks persist `attempt_count`, `max_attempts` (default three),
  `next_attempt_at`, `last_error_class`, and `dead_lettered_at`. Due
  queued claims atomically increment attempts. Only contained `self-update`
  tasks may automatically retry timeout, provider/rate-limit, browser, database,
  or infrastructure failures: every attempt starts from a fresh clone and only
  a verified commit can escape it. User tasks fail after an execution error
  rather than replaying unknown side effects. Hook denial, cancellation/abort,
  validation, permission/containment, missing configuration, and unknown
  failures never retry; exhausted self-update work becomes terminal
  `dead-letter` with durable explanatory progress.
- Each task persistently retains its latest 256 iteration activity records: agent identity,
  step bounds, finish state, model text, and structured tool calls with arguments,
  results, errors, and execution status. Plaintext-secret tool arguments and results
  are redacted before persistence. This bounded transcript survives polling, retry,
  terminal review, and restart.
- Task polling rediscovers delayed work. Restart recovery preserves the prompt,
  model, session, schedule, and workspace; it requeues only non-stale running
  work below `maxAttempts`, dead-letters non-stale exhausted work, fails stale
  work, and finalizes cancelling work as cancelled. Runtime start/stop operations
  are serialized; a stop requested during startup lets initialization and one
  recovery pass finish but prevents polling or execution from being resurrected.
- Each claimed attempt retains its executor promise. Timeout, cancellation, and
  shutdown abort the signal, then wait a bounded acknowledgement interval while
  retaining the occupied execution slot. Cancellation becomes durable only after
  executor settlement; timeout becomes retry-eligible only after settlement. An
  executor that ignores abort remains non-runnable in `running` or `cancelling`,
  emits `interrupted`, holds its slot, and blocks same-runtime restart recovery
  rather than overlapping a retry.
- Durable completion, failure, dead-letter, and cancellation transitions precede
  terminal lifecycle delivery. Every attempt uses one correlated turn and trace
  ID; completed events carry the outcome and failures carry their classified
  error. `TaskStop`, channel, hook, browser-close, and evolution observers are
  best effort after the transition: observer failure is logged and cannot emit a
  contradictory failed terminal state. Intermediate retries still emit failed
  `TaskStop` before a later `TaskStart`; final exhaustion emits `dead-letter`.
- `DELETE /api/tasks/:id` removes only terminal `completed`, `failed`,
  `cancelled`, or `dead-letter` rows. It returns conflict for active work and
  not-found for missing rows; task removal never cancels execution or rolls
  back repository changes.
- Schedules persist `maxAttempts`; create/update preserve it and transactional
  materialization copies it into each task under row locking, with enqueue only
  after commit. The retry/schema migration holds a PostgreSQL advisory
  transaction lock, rechecks its version marker after locking, and records the
  marker only after the schema and one-time timezone/next-run backfill succeed.
  Ordinary initialization never recalculates an existing schedule after that
  marker. `src/agent-execution.ts` gives ordinary task work the same prompt,
  delegation/tool hooks, browser activation, recall, lifecycle processors,
  execution bounds, final-response guard, and episode retention as chat turns.
  Self-update tasks use `src/self-update-workflow.ts` instead: the deterministic
  Mastra workflow owns typed Researcher, decision, Implementer, inspection,
  verification, Reviewer, and trusted local commit handoffs under one trace,
  while each specialist iteration streams into the task activity transcript.
  Typed workflow handoffs use the specialist model for tool-driven work, then
  the persisted orchestrator model—without the schedule model override—for
  streaming structured-output conversion. Handoffs recover the first complete
  JSON object from surrounding stream text, normalize keyed Researcher evidence
  into bounded entries, and retry one transient transport or provider JSON-parse failure. Repository research permits at most five inspection/tool iterations followed by one final handoff iteration; implementation, verification, and review remain separate later phases.
  Contained absolute workspace paths are normalized to repository-relative
  paths before policy validation; escaping paths remain denied. Researchers may
  read repository `AGENTS*.md` and `.agents/skills/*/SKILL.md` instructions, but
  cannot modify them. Any containment-denial evidence is a failed inspection,
  never a verified no-op.
  Schedule calculations use the server timezone; keep UI history limits separate
  from the unbounded queued-work recovery query. Transient task failures remain
  queued with bounded exponential backoff until their dependency recovers; the
  durable attempt ceiling advances one attempt at a time instead of creating a
  dead-letter requiring operator action. Restart-interrupted work follows the same
  rule. Permanent failures retain their evidence and enqueue one deduplicated,
  workspace-scoped remediation task keyed by parent task ID. Remediation uses the
  current default model, retries every failure until it completes, and never
  recursively creates another remediation task. Its prompt embeds the original
  request and workflow evidence and requires exact lookup through
  `popagent task <id>`; a successful remediation report does not automatically
  resolve the parent. The remediation must explicitly run `task-resolve` only
  after concrete evidence proves the original request is satisfied.
- `SelfUpdateScheduler` serially creates one default `source='self-update'`
  schedule per eligible registered workspace. A PostgreSQL partial unique index
  owns that `(workspace_id)` slot. Reconciliation never overwrites an existing
  schedule's timing, enabled state, prompt, model, or attempt limit. Schedule
  PATCH permits timing and enabled-state changes for system rows while preserving
  their code-owned identity and execution payload; only user rows may be deleted.
  `POST /api/autonomy/reconcile` and `./popagent automation-reconcile` create
  missing defaults and enforce workspace eligibility without resetting operator
  timing choices.
  Scheduler lifecycle operations share one serialized subscription/reconcile/timer
  state; a stop requested during initial reconciliation prevents later timer
  resurrection, and a subsequent explicit start can initialize again.
- Scheduling requires enabled global/self-update policy, read-write supervisor
  and implementer, delegation-eligible implementer and reviewer, reviewer
  repository access, and a resolved writable Git repository. Both scheduled and
  immediate materialization take the same PostgreSQL transaction advisory lock,
  reject any globally active self-update task, and select at most one due
  workspace, so only one autonomous task can be queued, running, or cancelling
  across the application. Materialization copies source and `maxAttempts` while
  atomically rereading enabled autonomy policy; disabling self-update therefore
  blocks new work before reconciliation. Trusted self-update tasks propagate
  `executionSource='self-update'`.
- Execution first requires a clean registered checkout and captures its HEAD,
  branch, and remote digest, then works in a secret-free, network-disabled local
  clone on a unique `autonomous/self-update/*` branch. Fixed runtime policy,
  independent of prompts and managed hooks, removes memory mutation and
  `useBrowserSecret`; denies hidden/credential/protected paths and remote,
  publish, deploy, restart, or unrestricted shell operations; and keeps reviewer
  workspace access read-only. The registered checkout is never the agent's
  writable surface.
- Non-Git verification runs only through the autonomous workspace's
  network-disabled bubblewrap sandbox with minimal environment, abort
  propagation, a bounded timeout, and hard stdout/stderr retention limits; Git
  inspection remains fixed-option and byte-bounded.
- Completion requires successful Researcher evidence collection. A clean no-op
  may complete immediately without Implementer or Reviewer delegation; changed
  results require successful Implementer execution and Reviewer verification.
  Inspect produces the bounded actual patch (including deletions) and its
  SHA-256 digest. Reviewer approval must echo that digest, and commit/import
  re-check the digest immediately before mutation. Acceptance permits only a
  clean no-op or clean descendant commit with unchanged remotes and an allowed
  diff. Continuous contained execution uses explicitly selected workspaces and
  its fixed local coding-model policy; PostgreSQL schedule state plus global
  active-task state gates one bounded cycle at a time.
- The model never receives credentials or direct publish/deploy tools. Accepted
  commits are imported as local refs. With explicit autonomous deployment
  enabled, fixed server policy may fast-forward the reviewed commit, invoke the
  repository's maintained `deploy.sh publish`, then launch `deploy.sh deploy`;
  failures remain visible and never force-push.
- Durable memory uses `@mastra/memory` with `@mastra/pg` `PostgresStoreVNext`
  (`src/storage.ts`). `DATABASE_URL` is required for application and memory data;
  `OBSERVABILITY_DATABASE_URL` is required for the isolated observability domain.
  Production application PostgreSQL is `popagent-postgres` on `127.0.0.1:5433`;
  observability PostgreSQL is `popagent-observability-postgres` on
  `127.0.0.1:5435`. Never route observability writes into the application store.
- Repository schema initialization shares concurrent work, retains successful
  completion, and resets after a failed attempt so transient database outages do
  not poison a store until process restart.
- Session display metadata and AI SDK messages live in PostgreSQL. Sessions have
  explicit archive timestamps and a title lock so a user rename survives later
  message saves. Session rows retain their repository workspace; list APIs can
  filter by it. Mastra also persists each conversation as a memory thread. Do not
  reintroduce SQLite.
- Long conversations use Mastra Observational Memory in thread scope. The
  Observer compacts old messages and tool results into observations and the
  Reflector bounds the observation log; original messages remain in PostgreSQL.
  Settings in `popagent_memory_settings` control enablement; observation and
  reflection thresholds; raw-history retention; async observation/reflection
  buffering and safety thresholds; idle/provider-change activation; shared
  budgets; previous-observation context limits; temporal markers; attachment
  forwarding; exact-history retrieval scope; and custom Observer/Reflector
  guidance. Shared budgets disable async buffering as required by Mastra.
  Each request sends only its newest UI message because Mastra reconstructs
  history from the thread. Observations remain thread-scoped; retrieval can be
  restricted to that thread or browse all threads for `popagent-user`. Observer
  and Reflector calls use the selected 9Router model rather than Mastra's hosted
  default. Configuration changes apply on the next turn.
- Delegated specialist messages use isolated generated resource/thread IDs. The pinned `@mastra/core` patch backfills those IDs onto provider response messages before persistence; removing it reintroduces `Thread ID is required` observability errors. `src/agent.test.ts` exercises the real delegated streaming path.
- Permanent chat deletion uses Mastra's `Memory.deleteThread()` before removing
  thread state and the session row, so messages and vector artifacts follow the
  framework lifecycle. Workspace-wide deletion runs that same lifecycle for
  every active and archived session, including SessionEnd hooks, browser closure,
  and hook-audit cleanup. Cross-session retention semantics live in
  [AGENTS_MEMORY.md](AGENTS_MEMORY.md).
- Administrative direct-database cleanup requires a data-only backup of
  `popagent_sessions`, `mastra_threads`, `mastra_messages`, and
  `mastra_thread_state`. Reconcile orphaned `popagent-user` Mastra threads that
  have no `popagent_sessions` row; test chat requests can create such threads
  independently of UI sessions. Verify active/archived API lists and preserve
  `popagent_memories`.
- Bootstrap configuration comes from the ignored `.env`: `DATABASE_URL` for the
  Bun process, `POSTGRES_PASSWORD` for Docker Compose, and a base64-encoded
  32-byte `POPAGENT_SECRET_KEY` for application-managed secret encryption.
- Application-managed secrets live encrypted in `popagent_secrets`
  (`src/secrets.ts`), scoped by resource and exact name. AES-256-GCM authenticates
  the resource ID and name as additional data. `popagent_secret_bindings` stores
  metadata-only exact HTTPS origin bindings. The authenticated `/api/secrets`
  interface provisions, rotates, binds, lists, and deletes credentials without
  returning values.
- Agents have no plaintext secret create, update, or recall tools.
  `useBrowserSecret` resolves a bound value server-side and fills only a browser
  element whose page and frame origins both equal the configured origin. Its
  model-visible result contains only the secret name, destination origin, and
  success status.
- The chat stream merges specialist iteration telemetry as transient AI SDK
  `data-agentActivity` parts beside the supervisor response. Persist those parts
  with the assistant message so reopened sessions retain delegation evidence.
  Usage and trace correlation metadata are defined in [AGENTS_OBSERVABILITY.md](AGENTS_OBSERVABILITY.md).
- Observability storage isolation, correlation, redaction, APIs, and retention live in [AGENTS_OBSERVABILITY.md](AGENTS_OBSERVABILITY.md).
- API authentication is controlled by `POPAGENT_API_KEY`. When set, every `/api/*` request must include the matching `x-popagent-key`; the root page and static assets remain public. Retired plaintext secret tool arguments/results are centrally redacted at chat ingress, session and Mastra-message persistence, hook dispatch, and transcript rendering.

- Session writes use optimistic revision locking: clients send `revision`; each successful save increments it atomically, and stale writes return `409 { error: "revision conflict", session }`. Session lists are capped at 200 rows.
- Mutation routes validate bounded JSON bodies and reject unknown models against the cached 9Router catalog; catalog outages fail open with a warning.
- Native long-term memory is separate from Mastra thread history. See
  [AGENTS_MEMORY.md](AGENTS_MEMORY.md). Hook protocol and policy live in
  [AGENTS_HOOKS.md](AGENTS_HOOKS.md).

## Contract-test exceptions

- `src/server.test.ts` owns the slow-stream idle-timeout configuration.
- Storage contracts require disposable local PostgreSQL.
- `src/agent.test.ts` and chat-stream scenarios require live 9Router models on
  Titan.


## Live BifrOSt tool

Agents with persisted browser access receive `bifrostNavigator`, a single typed gateway to the local visible BifrOSt Navigator MCP catalog. The tool accepts an exact `browser_*` operation plus its catalog arguments, preserves MCP text/image/resource content, enforces `interactive` versus `read-only` role access server-side, and is absent during trusted self-update. BifrOSt remains a user-owned desktop process; Popagent connects only to its protected Unix socket and does not launch or stop it.

## AkurAI Build roles

Persisted `build-maintainer`, `build-release-manager`, and `community-steward`
profiles are governed Mastra specialists. Build Maintainer has contained
read-write workspace access and no browser access; Release Manager and
Community Steward have no workspace or browser access. They use only their
role-specific typed tools in `src/tools/akurai-build.ts` for Build operations.
Build Maintainer can inspect and maintain repositories, runs, workers, cache,
audit, pipeline validation, metrics, and community content but cannot promote
protected environments. Build Release Manager owns explicit release and
protected-promotion operations and is never eligible for autonomous scheduling.
Community Steward is limited to public repository and issue/comment operations
and has no pipeline or deployment mutation. `AgentSettingsStore` rejects Build
tools on general roles and enforces each Build role allowlist.

The AkurAI Build client uses JSON-RPC MCP at `AKURAI_BUILD_URL` (default
`https://akurai-build.olibuijr.com`) and reads its bearer only from
`AKURAI_BUILD_API_KEY`. It initializes lazily, bounds request/response bodies
and timeouts, propagates aborts, strictly parses MCP results, and reports
disabled status when credentials are absent without logging credentials.

Idle `build-maintenance` work is persisted as a distinct task/schedule source
and uses a process-local autonomous lease in addition to the database gate, so
interactive and autonomous work cannot overlap accidentally. It runs only in
the writable `AkurAI-Build` workspace, delegates to Build Maintainer plus the
independent Reviewer, queues CI for the exact reviewed branch/ref, and records
branch/run/issue/comment evidence. The contained completion policy imports only
an allowed descendant commit after immutable checkout evidence; it never
promotes or deploys production for Build maintenance. Managed deployment reuses
immutable CI verification only when the exact 40-character revision and UTC
timestamp evidence are supplied.