popagent

Latest change 7fe5fe2acff014eed98cc59694a021837a6713c6 - Serialize code-intelligence sessions per repository and self-heal corrupt or locked indexes by AkurAI Build

# Agent backend

Framework: **Mastra** (`@mastra/core`). Docs: https://mastra.ai/docs

## Model provider

Gateway, catalog, model source, local admission control, and the model routing
policy live in [AGENTS_MODELS.md](AGENTS_MODELS.md).

## HTTP server (`src/server.ts`)

`Bun.serve` with `routes`:

- `/` — the chat UI (`src/ui/index.html`, bundled by Bun at serve time).
- `GET /api/models` — `{ models, contextWindows, defaultModel }` from 9Router.
- `GET/POST /api/sessions` — list active/archived or create sessions.
- `DELETE /api/workspaces/:id/sessions` — permanently deletes every active and
  archived chat in one workspace through the normal per-session lifecycle.
- Workspace registration, code-context/symbol-context/change-impact inspection, and documentation CRUD/index/search routes are workspace-contained; documentation contracts are owned by [AGENTS_DOCUMENTATION.md](AGENTS_DOCUMENTATION.md).
- Browser settings, health, sessions, takeover, profiles, recordings, and screencast routes are owned by [AGENTS_BROWSER.md](AGENTS_BROWSER.md).
- Native channel settings, history, event, post, and cancellation routes are owned by [AGENTS_CHANNELS.md](AGENTS_CHANNELS.md).
- Observability overview, traces, logs, health, and feedback routes are owned by [AGENTS_OBSERVABILITY.md](AGENTS_OBSERVABILITY.md).
- `GET /api/health` — protected deployment readiness for application and observability PostgreSQL. Model, search, and browser outages are intentionally not deployment blockers.
- `GET/PUT/PATCH/DELETE /api/sessions/:id` — load/save, rename, archive/restore,
  or permanently delete a session and its matching Mastra thread data.
- `GET /api/sessions/:id/hooks` — ordered hook audit runs for the session.
- `GET/POST /api/memories` and `PATCH/DELETE /api/memories/:id` — long-term
  memory management; retention and mutability semantics live in
  [AGENTS_MEMORY.md](AGENTS_MEMORY.md).
- `POST /api/chat` — AI SDK UI-message stream via `handleChatStream` from
  `@mastra/ai-sdk` + `createUIMessageStreamResponse` from `ai`. The route accepts
  only the session id, optional selected model, workspace, and newest UI message
  from the client; callers cannot inject Mastra tools or execution options. The
  chat id maps to the `popagent-user` memory thread. A typed `RequestContext`
  carries one database-backed runtime-settings snapshot through the supervisor
  and delegated specialists. The route also dispatches managed lifecycle hooks,
  injects native cross-session recall, and adds turn IDs to assistant metadata
  so the UI can correlate hook activity.
- `GET/PATCH /api/settings/memory` — read or update persisted Observational
  Memory compaction settings; changes apply on the next agent turn.
- `GET/PATCH /api/settings/agent-runtime` — read or update the default model,
  execution/delegation bounds and feedback, and background-task controls.
- `GET/PATCH /api/settings/autonomy` — read or update the persisted evolution,
  skill-creation, and contained self-update controls.
- `GET /api/autonomy/signals`, `POST /api/autonomy/signals/:id/ignore`,
  `POST /api/autonomy/signals/:id/retry`, `GET /api/autonomy/revisions`, and
  `POST /api/autonomy/revisions/:id/revert` — inspect bounded recent evidence,
  ignore pending/dead-letter evidence, retry dead-letter evidence from attempt zero,
  inspect append-only learned changes, or append a compensating revert.
- `GET /api/company` — return the fixed organizational roster recovered from
  Midget, grouped by department. Company members are display and mailbox
  identities; they do not add Mastra agents or consume model iterations.
- `GET /api/agents` and `PATCH /api/agents/:id` — list persisted system agents
  and update their names, delegation descriptions, instructions, workspace and
  browser access, delegation eligibility, and tool membership.
- `GET /api/agents/health` — deterministically validates persisted agent/tool
  assignments and enabled skills; the `popagent agent-health` CLI is the
  maintained operator and autonomous-agent surface.
- `GET/PATCH /api/settings/browser` — read or toggle the global persisted browser
  capability; changes apply immediately without restarting.
- `GET/POST /api/agents/:id/skills` and
  `PATCH/DELETE /api/agents/:id/skills/:skillId` — manage database-backed Agent
  Skills.
- `GET/POST /api/tasks` and `POST /api/tasks/:id/cancel` — manage persistent
  background agent work. The list projection deliberately omits `activity`;
  transcripts run to megabytes per row and the board polls this route every two
  seconds, so only `GET /api/tasks/:id` carries the activity records.
  `GET /api/tasks/:id/workflow` returns the bounded ordered self-update phase
  detail and evidence projection.
- `GET/POST /api/schedules` and `PATCH/DELETE /api/schedules/:id` — manage
  recurring schedules. Clients submit cron expressions; timezone is
  server-controlled and is not part of the writable API.

Wire DTOs shared with the UI live in `src/api-types.ts`; do not redeclare those
response shapes in React code.

## Decisions

- One user-facing agent (`orchistrator`) supervises the coding specialists
  `researcher`, `implementer`, and `reviewer`, plus governed AkurAI Build
  specialists `build-maintainer`, `build-release-manager`, and
  `community-steward`, using Mastra's documented `Agent.agents` supervisor
  pattern. The preferred coding delivery sequence is research, implementation,
  then independent review; Build delegation is role-specific and never weakens
  the coding specialists' workspace or browser boundaries. Specialists do not
  delegate directly to one another and are not exposed as chat agents.
  Autonomous self-update execution additionally uses a deterministic Mastra
  workflow that owns the ordered `prepare`, `research`, `decide`, `implement`,
  `inspect-change`, `verify`, `review`, and `commit` phases. Typed handoffs and
  the accepted-review gate prevent later phases from running after a failure;
  a no-op decision marks later phases skipped.
- Each claimed background task owns one process-local
  `TaskAgentCommunicationSession`, carried by reference through Mastra
  `RequestContext` into the supervisor and every governed participant. The
  automatically bound `agentCommunication` tool lists task peers and supports
  addressed sends, broadcast, inbox, bounded wait, and replies without changing
  any profile's persisted capability membership. Mailboxes retain at most 100
  messages and drop the oldest deterministically. `delivered` means an active
  waiter consumed the message; `woken` means Popagent scheduled the addressed
  idle participant's own forked Mastra execution with the same contained task
  context; `queued` means an already-active recipient has not consumed it yet.
  Woken runs are tracked through settlement and must reply from the recipient's
  identity with the original message ID; blank terminal output receives a
  deterministic “No additional findings” reply so an otherwise successful task
  is not failed by protocol bookkeeping. Replies are queued for their addressed
  recipient but never wake another run, preventing acknowledgement loops. The
  bus rejects sender impersonation before persistence. Up to eight queued messages
  enter a participant's instructions at its next safe model boundary. Task abort
  or executor settlement closes the session, rejects waiters, clears mailboxes,
  and makes all later sends fail.
- All persisted system agents have display names, delegation descriptions,
  complete role instructions, source URLs, workspace/browser access,
  delegation eligibility, and tool membership in `popagent_agents`; `agent.ts`
  contains no role-prompt or capability fallback. `src/agent-role-seed.sql`
  applies versioned initial role data once, preserving pre-existing guidance,
  after which PostgreSQL is canonical and edits apply on the next run. Custom
  tool groups and their contract tests live under `src/tools/`; tightly coupled
  secret operations intentionally share one module.
- Each persisted agent profile has an optional model override. The read-only
  Researcher and Reviewer default to the proven low-latency
  `cx/gpt-5.3-codex-spark`; the Orchestrator and Implementer inherit the
  request-selected or persisted default model. Agent Settings applies model
  edits on the next run.
- Model access uses Mastra's provider interface; no direct vendor SDK imports
  appear in app code, so the provider stays swappable. Each execution records
  the selected route in Mastra `RequestContext`; delegated specialists resolve
  that route dynamically instead of falling back to the server default.
- `popagent_agent_runtime_settings` is the canonical singleton for the model
  source and effective default model, supervisor/specialist step ceilings,
  concurrent tool-call limit, processor retries, final-response feedback,
  delegation context/result bounds, delegation failure/truncation text, and
  background-task concurrency/timing.
  One settings snapshot is carried through each execution. The final step is
  reserved for a response, specialist calls cannot raise their configured step
  ceiling, and suspension-capable calls remain sequential under Mastra's
  `called` concurrency policy.
- A step ceiling is passed as Mastra's `maxSteps`, which `@mastra/core` turns
  into `stopWhen: [stepCountIs(maxSteps)]`; one step is one model call plus the
  tool calls it batches, so parallel tool calls cost one step. Both ceilings
  accept 1–128 and apply per turn — an exhausted budget ends the turn with a
  forced summary and is never resumed, so coding-shaped work needs ceilings in
  the tens, not the low teens. `createFinalResponseGuard` consumes one of them,
  and Mastra caps a delegation's requested `maxSteps` at the sub-agent's own
  configured ceiling, so `specialistMaxSteps` bounds every delegated run.
  Raising a ceiling for background work usually requires raising `taskTimeoutMs`
  too, since that wall-clock kill is independent of steps. Change either with
  `popagent settings-patch agent-runtime '{"supervisorMaxSteps":64}'`.
- Browser, memory-compaction, agent-runtime, and autonomy singleton rows are
  seeded exactly once by versioned SQL migrations recorded in
  `popagent_data_migrations`. Their configurable columns have no database
  defaults, and startup never recreates a row after its migration marker exists.
  PostgreSQL is the only runtime source; a missing row is a configuration error,
  not a signal to restore code defaults.
- Delegation returns only specialist text to the supervisor: nested tool
  payloads and run metadata stay out of model context. Context slicing, result
  truncation, and failure feedback use the execution's persisted settings
  snapshot before hook audit and supervisor synthesis. The context slice
  (`messageFilter` in `src/hook-lifecycle.ts`) trims trailing assistant-role
  messages after the raw tail slice so a delegated subagent never receives
  context ending on an unresolved assistant turn; strict Claude-family models
  reject that as invalid assistant-message prefill.
- `AgentWorkspaceStore` persists repository definitions beneath
  `POPAGENT_WORKSPACE_ROOT` (default `workspace/`); registrations are existing
  relative directories contained there. Every session, task, and schedule carries
  a `workspace_id`. Request context gives supervisor/implementer a contained
  writable Mastra Workspace and researcher/reviewer a read-only view. Agent paths
  stay repository-relative; `LocalFilesystem`, bubblewrap, and host LSP marker
  translation reject escapes. Shell remains network-disabled, writes require a
  current read, and `typescript-language-server` supplies semantic inspection.
- `opencode-codebase-index@0.23.0` supplies local Tree-sitter parsing,
  branch-aware incremental indexing, BM25/vector retrieval, and call graphs.
  Each checkout owns an ignored `.codebase-index/`; a secret-free external config
  selects the LAN embedding model. Queries refresh without watchers or hosted fallback.
  A cold full index takes minutes on the local CPU embedder, so
  `OpenCodebaseIndexBackend` warms indexes in the background (serialized, 45 min
  budget, readiness marker under `data/code-intelligence/ready/`) and answers a
  cold or warming lookup immediately with an "index warming — use grep/read and
  retry" error instead of blocking the agent's step budget. Warm-ups start at
  server boot for the default workspace only (`server.ts`; other workspaces
  warm lazily on first lookup so the CPU embedder is not saturated for hours) and in the self-update
  `prepare` phase for the task checkout (`warmCodeIndex` dependency); the
  Researcher `codebase-investigation` skill queries the index first and reads
  files only to confirm lines. Index sessions are serialized per repository
  (`withRepository`): the native vector store is plain files, and concurrent
  refresh+query from two tasks in one workspace produced "Vector store
  structure mismatch: vectors=549, ids=548" for the reader. A corruption
  message (`isIndexCorruption`) discards `.codebase-index/` and the readiness
  marker, restarts the warm-up, and answers "index was corrupt and is being
  rebuilt"; an `INDEX_BUSY` whose lock owner PID is dead on this host has its
  `indexing.lock`/recovery markers removed (`clearStaleLock`) and the warm-up
  retried once. Mastra's "Delegation maxSteps exceeds sub-agent default,
  capping" is expected (the Orchistrator asks for more than the specialist
  budget) and is logged at info by `PopagentLogger`.
- `codeContext` is bounded conceptual discovery; `symbolContext` returns a known
  symbol's definitions and direct callers/callees; `changeImpact` combines branch
  and working-tree evidence. LSP owns typed references/renames; grep owns exhaustive
  text. Autonomous tools bind the trusted task checkout. Matching workspace APIs
  and `popagent` commands never expose repository paths or index management.
- `repoBrief` remains the deterministic Git/TypeScript preflight injected before
  ordinary turns; it controls Researcher withholding and never calls the index.
- Failed, cancelled, and dead-lettered tasks can be operator-resolved through `POST /api/tasks/:id/resolve` with a bounded evidence summary. Resolution preserves the task record, changes it to completed, and clears stale failure/retry fields; queued, running, and already-completed tasks reject resolution.
- PostgreSQL-backed project-folder Markdown documentation and workspace-filtered RAG use Mastra `MDocument`, AI SDK embeddings, `PgVector`, and local-model reranking; source, indexing, and retrieval contracts live in [AGENTS_DOCUMENTATION.md](AGENTS_DOCUMENTATION.md).
- Web search uses the local SearXNG container at `SEARXNG_URL` (default
  `http://127.0.0.1:8889`). The `webSearch` runtime tool (`createTool` ID
  `web-search`) requests SearXNG's JSON API
  and is available to the supervisor and every subagent. Keep search
  credential-free and self-hosted; do not add a hosted search API dependency.
- Browser automation, network policy, profiles, recordings, session lifecycle, and verification contracts live in [AGENTS_BROWSER.md](AGENTS_BROWSER.md).
- Agent Skills are inline Mastra skills resolved dynamically from
  `popagent_agent_skills` for every execution agent on every run. The company
  roster is the specialist-agent catalog, but role identity and common
  evidence/verification behavior live in agent instructions rather than
  duplicated `*-practice` skills. Add a skill only for a concrete,
  task-triggered procedure with distinct domain steps or boundaries. Company
  staff remain delegable specialists under the Orchestrator; established
  research, implementation, review, Build, and community identities retain
  their governed runtime duties. All skill instructions, references,
  provenance, and enablement live
  in PostgreSQL. Enabled skills provide Mastra's `skill`, `skill_read`, and
  `skill_search` tools; disabled skills remain editable but are excluded from
  agent execution. Skill names follow the Agent Skills lowercase-hyphen
  specification, references are JSONB, and ownership is exposed as
  `evolutionManaged` in the Agent Skill DTO. Every ordinary user/API update
  clears `evolution_managed`; once edited, a learned skill is user-owned and
  evolution cannot reclaim or revert it.
- Autonomous learning persists in `popagent_evolution_signals`,
  `popagent_learned_overlays`, and append-only `popagent_evolution_revisions`.
  Each signal retains the real turn trace ID plus bounded, secret-redacted
  Goal/Outcome or Goal/Failure evidence with a failure classification.
  Signals are deduplicated before a periodic worker claims due batches with
  `FOR UPDATE SKIP LOCKED`; every run first recovers abandoned claims, and
  concurrent worker start/stop transitions are serialized.
- Reflection uses a tool-free Mastra agent and the configured 9Router model.
  It consumes Mastra's streaming structured-output contract because 9Router
  returns SSE for this route. The model can select only a closed, code-owned
  strategy enum; it cannot author prompt or skill text. With automatic fact
  retention enabled, the `retain-fact` strategy may author only a bounded
  operational fact and semantic key established by cited evidence for the same
  agent. Application locks and rereads the autonomy singleton and rechecks
  cancellation in the same transaction as learned mutation and signal
  completion. A disabled global or target policy wins before mutation: claimed
  signals become ignored and the claim attempt is released rather than retried
  or dead-lettered. Other failures use capped exponential backoff and terminal
  dead-lettering.
- Learned overlays are stored separately from `popagent_agents.instructions`
  and composed after the immutable user-managed base on every agent resolution.
  Automatic skill changes are limited to rows marked `evolution_managed`; a
  user-managed skill with the same name is never overwritten. Evolution-managed
  facts use `evolution.<agent-id>.<semantic-key>` keys in the fixed
  `popagent-user` resource and retain their cited signal's session provenance.
  Automatic learning cannot change tool membership, workspace/browser access,
  secrets, hooks, authentication, execution bounds, or user-managed
  instructions. Secret-bearing and protected-boundary proposals fail without
  partial mutation. Reverts append a compensating revision with
  `revertsRevisionId`; a partial unique index permits at most one compensation
  per source revision. Revert restores the prior effective overlay, skill, or
  fact only when that source is still the latest target version and the current
  managed target still matches it.
- Background tasks and schedules persist in `popagent_agent_tasks` and
  `popagent_agent_schedules`. Both carry immutable `source: 'user' |
  'self-update'` ownership and their originating workspace; materialized tasks
  inherit both fields, so retry and recovery never change provenance or the
  repository boundary.
  Tasks persist `attempt_count`, `max_attempts` (default three),
  `next_attempt_at`, `last_error_class`, and `dead_lettered_at`. Due
  queued claims atomically increment attempts. Only contained `self-update`
  tasks may automatically retry timeout, provider/rate-limit, browser, database,
  or infrastructure failures, including upstream response-header timeouts:
  every attempt starts from a fresh clone and only
  a verified commit can escape it. User tasks fail after an execution error
  rather than replaying unknown side effects. Hook denial, cancellation/abort,
  validation, permission/containment, missing configuration, and unknown
  failures never retry; exhausted self-update work becomes terminal
  `dead-letter` with durable explanatory progress.
- Each task persistently retains its latest 256 iteration activity records: agent identity,
  step bounds, finish state, model text, and structured tool calls with arguments,
  results, errors, and execution status. Plaintext-secret tool arguments and results
  are redacted before persistence. This bounded transcript survives polling, retry,
  terminal review, and restart. It is read through single-task queries only; list
  queries use a column set without it.
- Mastra stream chunks (`step-start`, `text-delta`, `tool-call`) forward into the
  transcript as they happen so a run shows work before its first iteration ends.
  Tool calls report immediately; text deltas are throttled so a long answer cannot
  storm the task store. This refines `onIterationComplete` reporting rather than
  replacing it, so a model result carrying no chunk stream is skipped instead of
  failing the run.
- Task polling rediscovers delayed work. Restart recovery preserves the prompt,
  model, session, schedule, and workspace; it requeues only non-stale running
  work below `maxAttempts`, dead-letters non-stale exhausted work, fails stale
  work, and finalizes cancelling work as cancelled. Runtime start/stop operations
  are serialized; a stop requested during startup lets initialization and one
  recovery pass finish but prevents polling or execution from being resurrected.
- Each claimed attempt retains its executor promise. Timeout, cancellation, and
  shutdown abort the signal, then wait a bounded acknowledgement interval while
  retaining the occupied execution slot. Cancellation becomes durable only after
  executor settlement; timeout becomes retry-eligible only after settlement. An
  executor that ignores abort remains non-runnable in `running` or `cancelling`,
  emits `interrupted`, holds its slot, and blocks same-runtime restart recovery
  rather than overlapping a retry.
- Durable completion, failure, dead-letter, and cancellation transitions precede
  terminal lifecycle delivery. Every attempt uses one correlated turn and trace
  ID; completed events carry the outcome and failures carry their classified
  error. `TaskStop`, channel, hook, browser-close, and evolution observers are
  best effort after the transition: observer failure is logged and cannot emit a
  contradictory failed terminal state. Intermediate retries still emit failed
  `TaskStop` before a later `TaskStart`; final exhaustion emits `dead-letter`.
- `DELETE /api/tasks/:id` removes only terminal `completed`, `failed`,
  `cancelled`, or `dead-letter` rows. It returns conflict for active work and
  not-found for missing rows; task removal never cancels execution or rolls
  back repository changes.
- Schedules persist `maxAttempts`; create/update preserve it and transactional
  materialization copies it into each task under row locking, with enqueue only
  after commit. The retry/schema migration holds a PostgreSQL advisory
  transaction lock, rechecks its version marker after locking, and records the
  marker only after the schema and one-time timezone/next-run backfill succeed.
  Ordinary initialization never recalculates an existing schedule after that
  marker. `src/agent-execution.ts` gives ordinary task work the same prompt,
  delegation/tool hooks, browser activation, recall, lifecycle processors,
  execution bounds, final-response guard, and episode retention as chat turns.
  A task that exhausts its step budget mid-tool-call yields blank text; the
  executor then issues one tool-free synthesis call (`toolChoice: "none"`,
  `maxSteps: 1`) so a run never completes with empty output, trimming the
  unresolved assistant tail first because unmatched `tool_use` blocks are
  rejected by the provider.
  Self-update tasks use `src/self-update-workflow.ts` instead: the deterministic
  Mastra workflow owns typed Researcher, decision, Implementer, inspection,
  verification, Reviewer, and trusted local commit handoffs under one trace,
  while each specialist iteration streams into the task activity transcript.
  Typed workflow handoffs use the specialist model for tool-driven work, then
  the persisted orchestrator model—without the schedule model override—for
  streaming structured-output conversion. Handoffs recover the first complete
  JSON object from surrounding stream text, normalize keyed Researcher evidence,
  truncate each evidence entry to its persisted 500-character boundary, and
  retry up to two transient transport or provider JSON-parse failures. A
  specialist that exhausts its step budget on tool calls with no text gets one
  tool-free structured synthesis call over its gathered evidence before the
  handoff is treated as malformed.
  Repository research uses the configured specialist iteration limit;
  implementation, verification, and review remain separate later phases. A
  clean checkout is baseline evidence only: a no-op additionally
  requires inspected paths and symbols, deterministic evidence, concrete
  candidates, and rejection reasons.
  Operator-requested repository changes use `POST /api/tasks` with
  `workflow=true` or `./popagent workflow-create <workspace> <prompt> [model]`
  to enter that same governed specialist workflow. Ordinary `task-create`
  remains for non-repository background work and its final model response is
  not a substitute for specialist source-change evidence.
  Contained absolute workspace paths are normalized to repository-relative
  paths before policy validation; escaping paths remain denied. Researchers may
  read repository `AGENTS*.md`, `.agents/skills/*/SKILL.md`, protected source,
  and package metadata, but cannot modify protected paths. Any containment-denial
  evidence is a failed inspection, never a verified no-op.
  Schedule calculations use the server timezone; keep UI history limits separate
  from the unbounded queued-work recovery query. Transient task failures remain
  queued with bounded exponential backoff until their dependency recovers; the
  durable attempt ceiling advances one attempt at a time instead of creating a
  dead-letter requiring operator action. Restart-interrupted work follows the same
  rule. Permanent failures retain their evidence and enqueue one deduplicated,
  workspace-scoped remediation task keyed by parent task ID. Remediation uses the
  current default model, retries every failure until it completes, and never
  recursively creates another remediation task. Its prompt embeds the original
  request, error, and workflow evidence and tells the agent the sandbox has no
  `./popagent` binary or API access, so it must not hunt for the CLI. A
  successful remediation report never resolves the parent (`task-resolve` stays
  an operator/CLI action); instead the runtime requeues the failed parent for
  one more attempt while `attempt_count < max_attempts`, and a later failure
  of that parent (whose single remediation already exists) is retried directly
  under the same ceiling. Once attempts are exhausted the parent moves to
  `dead-letter`, so a `failed` task never waits on an operator while retries
  remain.
- `SelfUpdateScheduler` serially creates one default `source='self-update'`
  schedule per eligible registered workspace. A PostgreSQL partial unique index
  owns that `(workspace_id)` slot. Reconciliation updates the system-owned name,
  prompt, model, and attempt limit so policy repairs reach existing schedules.
  System schedule rows are execution-policy records, not wall-clock triggers:
  task polling materializes one whenever the process is idle, no user schedule is
  due, and no autonomous task is queued, running, or cancelling. Their legacy
  cron and next-run fields are ignored. Operators may pause or resume system rows;
  only user rows expose editable timing and deletion. `POST
  /api/autonomy/reconcile` and `./popagent automation-reconcile` create missing
  defaults, refresh code-owned execution policy, and enforce workspace
  eligibility.
  Scheduler lifecycle operations share one serialized subscription/reconcile/timer
  state; a stop requested during initial reconciliation prevents later timer
  resurrection, and a subsequent explicit start can initialize again.
- Scheduling requires enabled global/self-update policy, read-write supervisor
  and implementer, delegation-eligible implementer and reviewer, reviewer
  repository access, and a resolved writable Git repository. Both scheduled and
  immediate materialization take the same PostgreSQL transaction advisory lock,
  reject any globally active self-update task, and select at most one idle-work
  workspace, so only one autonomous task can be queued, running, or cancelling
  across the application. Materialization copies source and `maxAttempts` while
  atomically rereading enabled autonomy policy; disabling self-update therefore
  blocks new work before reconciliation. Trusted self-update tasks propagate
  `executionSource='self-update'`.
- Execution first requires a clean registered checkout and captures its HEAD,
  branch, and remote digest, then works in a secret-free, network-disabled local
  clone on a unique `autonomous/self-update/*` branch. Fixed runtime policy,
  independent of prompts and managed hooks, removes memory mutation and
  `useBrowserSecret`; denies hidden/credential/protected paths and remote,
  publish, deploy, restart, or unrestricted shell operations; and keeps reviewer
  workspace access read-only. The registered checkout is never the agent's
  writable surface.
- Non-Git verification runs only through the autonomous workspace's
  network-disabled bubblewrap sandbox with minimal environment, abort
  propagation, a bounded timeout, and hard stdout/stderr retention limits; Git
  inspection remains fixed-option and byte-bounded.
- Completion requires successful Researcher evidence collection. A clean no-op
  may complete immediately without Implementer or Reviewer delegation; changed
  results require successful Implementer execution and Reviewer verification.
  Inspect produces the bounded actual patch (including deletions) and its
  SHA-256 digest. Reviewer approval must echo that digest, and commit/import
  re-check the digest immediately before mutation. Acceptance permits only a
  clean no-op or clean descendant commit with unchanged remotes and an allowed
  diff. Continuous contained execution uses explicitly selected workspaces and
  its fixed local coding-model policy; PostgreSQL schedule state plus global
  active-task state gates one bounded cycle at a time.
- The model never receives credentials or direct publish/deploy tools. Accepted
  commits are imported as local refs. With explicit autonomous deployment
  enabled, fixed server policy may fast-forward the reviewed commit, invoke the
  repository's maintained `deploy.sh publish`, then launch `deploy.sh deploy`;
  failures remain visible and never force-push.
- Durable memory uses `@mastra/memory` with `@mastra/pg` `PostgresStoreVNext`
  (`src/storage.ts`). `DATABASE_URL` is required for application and memory data;
  `OBSERVABILITY_DATABASE_URL` is required for the isolated observability domain.
  Production application PostgreSQL is `popagent-postgres` on `127.0.0.1:5433`;
  observability PostgreSQL is `popagent-observability-postgres` on
  `127.0.0.1:5435`. Never route observability writes into the application store.
- Repository schema initialization shares concurrent work, retains successful
  completion, and resets after a failed attempt so transient database outages do
  not poison a store until process restart.
- Session display metadata and AI SDK messages live in PostgreSQL. Sessions have
  explicit archive timestamps and a title lock so a user rename survives later
  message saves. Session rows retain their repository workspace; list APIs can
  filter by it. Mastra also persists each conversation as a memory thread. Do not
  reintroduce SQLite.
- Long conversations use Mastra Observational Memory in thread scope. The
  Observer compacts old messages and tool results into observations and the
  Reflector bounds the observation log; original messages remain in PostgreSQL.
  Settings in `popagent_memory_settings` control enablement; observation and
  reflection thresholds; raw-history retention; async observation/reflection
  buffering and safety thresholds; idle/provider-change activation; shared
  budgets; previous-observation context limits; temporal markers; attachment
  forwarding; exact-history retrieval scope; and custom Observer/Reflector
  guidance. Shared budgets disable async buffering as required by Mastra.
  Each request sends only its newest UI message because Mastra reconstructs
  history from the thread. Observations remain thread-scoped; retrieval can be
  restricted to that thread or browse all threads for `popagent-user`. Observer
  and Reflector calls use the selected 9Router model rather than Mastra's hosted
  default. Configuration changes apply on the next turn.
- Delegated specialist messages use isolated generated resource/thread IDs. The pinned `@mastra/core` patch backfills those IDs onto provider response messages before persistence; removing it reintroduces `Thread ID is required` observability errors. `src/agent.test.ts` exercises the real delegated streaming path.
- Permanent chat deletion uses Mastra's `Memory.deleteThread()` before removing
  thread state and the session row, so messages and vector artifacts follow the
  framework lifecycle. Workspace-wide deletion runs that same lifecycle for
  every active and archived session, including SessionEnd hooks, browser closure,
  and hook-audit cleanup. Cross-session retention semantics live in
  [AGENTS_MEMORY.md](AGENTS_MEMORY.md).
- Administrative direct-database cleanup requires a data-only backup of
  `popagent_sessions`, `mastra_threads`, `mastra_messages`, and
  `mastra_thread_state`. Reconcile orphaned `popagent-user` Mastra threads that
  have no `popagent_sessions` row; test chat requests can create such threads
  independently of UI sessions. Verify active/archived API lists and preserve
  `popagent_memories`.
- Bootstrap configuration comes from the ignored `.env`: `DATABASE_URL` for the
  Bun process, `POSTGRES_PASSWORD` for Docker Compose, and a base64-encoded
  32-byte `POPAGENT_SECRET_KEY` for application-managed secret encryption.
- Application-managed secrets live encrypted in `popagent_secrets`
  (`src/secrets.ts`), scoped by resource and exact name. AES-256-GCM authenticates
  the resource ID and name as additional data. `popagent_secret_bindings` stores
  metadata-only exact HTTPS origin bindings. The authenticated `/api/secrets`
  interface provisions, rotates, binds, lists, and deletes credentials without
  returning values.
- Agents have no plaintext secret create, update, or recall tools.
  `useBrowserSecret` resolves a bound value server-side and fills only a browser
  element whose page and frame origins both equal the configured origin. Its
  model-visible result contains only the secret name, destination origin, and
  success status.
- The chat stream merges specialist iteration telemetry as transient AI SDK
  `data-agentActivity` parts beside the supervisor response. Persist those parts
  with the assistant message so reopened sessions retain delegation evidence.
  Usage and trace correlation metadata are defined in [AGENTS_OBSERVABILITY.md](AGENTS_OBSERVABILITY.md).
- Observability storage isolation, correlation, redaction, APIs, and retention live in [AGENTS_OBSERVABILITY.md](AGENTS_OBSERVABILITY.md).
- API authentication is controlled by `POPAGENT_API_KEY`. When set, every `/api/*` request must include the matching `x-popagent-key`; the root page and static assets remain public. Retired plaintext secret tool arguments/results are centrally redacted at chat ingress, session and Mastra-message persistence, hook dispatch, and transcript rendering.

- Session writes use optimistic revision locking: clients send `revision`; each successful save increments it atomically, and stale writes return `409 { error: "revision conflict", session }`. Session lists are capped at 200 rows.
- Mutation routes validate bounded JSON bodies and reject unknown models against the cached 9Router catalog; catalog outages fail open with a warning.
- Native long-term memory is separate from Mastra thread history. See
  [AGENTS_MEMORY.md](AGENTS_MEMORY.md). Hook protocol and policy live in
  [AGENTS_HOOKS.md](AGENTS_HOOKS.md).

## Contract-test exceptions

- `src/server.test.ts` owns the slow-stream idle-timeout configuration.
- Storage contracts require disposable local PostgreSQL.
- `src/agent.test.ts` and chat-stream scenarios require live 9Router models on
  Titan.

## Follow-up tasks and live company state

Autonomous follow-up planning for scheduled runs and the `companyStatus`
reporting tool live in [AGENTS_TASKS.md](AGENTS_TASKS.md).

## Live BifrOSt tool

Agents with persisted browser access receive `bifrostNavigator`, a single typed gateway to the local visible BifrOSt Navigator MCP catalog. The tool accepts an exact `browser_*` operation plus its catalog arguments, preserves MCP text/image/resource content, enforces `interactive` versus `read-only` role access server-side, and is absent during trusted self-update. BifrOSt remains a user-owned desktop process; Popagent connects only to its protected Unix socket and does not launch or stop it.

## AkurAI Build roles

Build specialist roles, tool allowlists, the Build MCP client, and idle
Build maintenance live in [AGENTS_BUILD.md](AGENTS_BUILD.md).