popagent

public

Latest change 74d33974c7d2e99d005596af3bce84f97c7b5f56 - Fix autonomous scheduling and runtime panel behavior by AkurAI Build

# Agent backend

Framework: **Mastra** (`@mastra/core`). Docs: https://mastra.ai/docs

## Model provider

- LLM gateway: local **9Router** (`http://127.0.0.1:20128/v1`, OpenAI-compatible),
via `createOpenAICompatible` from `@ai-sdk/openai-compatible`.
Env: `NINEROUTER_URL`, `NINEROUTER_API_KEY` (see `.env.example`).
- The default model is persisted in the singleton
`popagent_agent_runtime_settings` row. `GET /api/models` merges that value with
the gateway's current model IDs and context-window metadata; do not hard-code
the catalog, default, or context limits in UI code.
- Per-request selection: `POST /api/chat` accepts an optional `model` body field;
when omitted, the server reads the persisted default.
- Titan's persisted production default and `orchistrator` profile use
`cx/gpt-5.6-luna-max` for intensive orchestration, complex reasoning, and
generating delegated instructions. The `researcher`, `implementer`, and
`reviewer` profiles use `titan/ornith-1.0-9b-mtp-q4_k_m` for token-heavy local
coding work. Ornith runs with a 131,072-token llama.cpp context and a
128,000-token application limit advertised by 9Router. A changed llama.cpp
alias becomes selectable only after OmniRoute imports the upstream `/models`
catalog.
- Bun's maximum 255-second idle timeout leaves enough time for slow local models
to emit their first stream event.

## HTTP server (`src/server.ts`)

`Bun.serve` with `routes`:

- `/` — the chat UI (`src/ui/index.html`, bundled by Bun at serve time).
- `GET /api/models` — `{ models, contextWindows, defaultModel }` from 9Router.
- `GET/POST /api/sessions` — list active/archived or create sessions.
- `DELETE /api/workspaces/:id/sessions` — permanently deletes every active and
archived chat in one workspace through the normal per-session lifecycle.
- Workspace registration and documentation CRUD/index/search routes are owned by [AGENTS_DOCUMENTATION.md](AGENTS_DOCUMENTATION.md).
- Browser settings, health, sessions, takeover, profiles, recordings, and screencast routes are owned by [AGENTS_BROWSER.md](AGENTS_BROWSER.md).
- Native channel settings, history, event, post, and cancellation routes are owned by [AGENTS_CHANNELS.md](AGENTS_CHANNELS.md).
- Observability overview, traces, logs, health, and feedback routes are owned by [AGENTS_OBSERVABILITY.md](AGENTS_OBSERVABILITY.md).
- `GET /api/health` — protected deployment readiness for application and observability PostgreSQL. Model, search, and browser outages are intentionally not deployment blockers.
- `GET/PUT/PATCH/DELETE /api/sessions/:id` — load/save, rename, archive/restore,
or permanently delete a session and its matching Mastra thread data.
- `GET /api/sessions/:id/hooks` — ordered hook audit runs for the session.
- `GET/POST /api/memories` and `PATCH/DELETE /api/memories/:id` — long-term
memory management; retention and mutability semantics live in
[AGENTS_MEMORY.md](AGENTS_MEMORY.md).
- `POST /api/chat` — AI SDK UI-message stream via `handleChatStream` from
`@mastra/ai-sdk` + `createUIMessageStreamResponse` from `ai`. The route accepts
only the session id, optional selected model, workspace, and newest UI message
from the client; callers cannot inject Mastra tools or execution options. The
chat id maps to the `popagent-user` memory thread. A typed `RequestContext`
carries one database-backed runtime-settings snapshot through the supervisor
and delegated specialists. The route also dispatches managed lifecycle hooks,
injects native cross-session recall, and adds turn IDs to assistant metadata
so the UI can correlate hook activity.
- `GET/PATCH /api/settings/memory` — read or update persisted Observational
Memory compaction settings; changes apply on the next agent turn.
- `GET/PATCH /api/settings/agent-runtime` — read or update the default model,
execution/delegation bounds and feedback, and background-task controls.
- `GET/PATCH /api/settings/autonomy` — read or update the persisted evolution,
skill-creation, and contained self-update controls.
- `GET /api/autonomy/signals`, `POST /api/autonomy/signals/:id/ignore`,
`POST /api/autonomy/signals/:id/retry`, `GET /api/autonomy/revisions`, and
`POST /api/autonomy/revisions/:id/revert` — inspect bounded recent evidence,
ignore pending/dead-letter evidence, retry dead-letter evidence from attempt zero,
inspect append-only learned changes, or append a compensating revert.
- `GET /api/agents` and `PATCH /api/agents/:id` — list persisted system agents
and update their names, delegation descriptions, instructions, workspace and
browser access, delegation eligibility, and tool membership.
- `GET/PATCH /api/settings/browser` — read or toggle the global persisted browser
capability; changes apply immediately without restarting.
- `GET/POST /api/agents/:id/skills` and
`PATCH/DELETE /api/agents/:id/skills/:skillId` — manage database-backed Agent
Skills.
- `GET/POST /api/tasks` and `POST /api/tasks/:id/cancel` — manage persistent
background agent work.
- `GET/POST /api/schedules` and `PATCH/DELETE /api/schedules/:id` — manage
recurring schedules. Clients submit cron expressions; timezone is
server-controlled and is not part of the writable API.

Wire DTOs shared with the UI live in `src/api-types.ts`; do not redeclare those
response shapes in React code.

## Decisions

- One user-facing agent (`orchistrator`) supervises three internal Mastra subagents:
`researcher`, `implementer`, and `reviewer`, using Mastra's documented
`Agent.agents` supervisor pattern. The preferred delivery sequence is research,
implementation, then independent review, but the supervisor retains ownership,
may skip stages that do not apply, and synthesizes every result; specialists do
not delegate directly to one another and are not exposed as chat agents.
All four agents have persisted display names, delegation descriptions,
complete role instructions, source URLs, workspace/browser access, delegation
eligibility, and tool membership in `popagent_agents`; `agent.ts` contains no
role-prompt or capability fallback. `src/agent-role-seed.sql` applies the
versioned initial role data from agency-agents once, preserving pre-existing
guidance, after which PostgreSQL is canonical and edits apply on the next
run. Custom tool groups and their contract tests live under `src/tools/`;
tightly coupled secret operations intentionally share one module. No teams or
workflows.
- Each persisted agent profile has an optional model override. The read-only
Researcher and Reviewer default to the proven low-latency
`cx/gpt-5.3-codex-spark`; the Orchestrator and Implementer inherit the
request-selected or persisted default model. Agent Settings applies model
edits on the next run.
- Model access uses Mastra's provider interface; no direct vendor SDK imports
appear in app code, so the provider stays swappable. Each execution records
the selected route in Mastra `RequestContext`; delegated specialists resolve
that route dynamically instead of falling back to the server default.
- `popagent_agent_runtime_settings` is the canonical singleton for the default
model, supervisor/specialist step ceilings, concurrent tool-call limit,
processor retries, final-response feedback, delegation context/result bounds,
delegation failure/truncation text, and background-task concurrency/timing.
One settings snapshot is carried through each execution. The final step is
reserved for a response, specialist calls cannot raise their configured step
ceiling, and suspension-capable calls remain sequential under Mastra's
`called` concurrency policy.
- Browser, memory-compaction, agent-runtime, and autonomy singleton rows are
seeded exactly once by versioned SQL migrations recorded in
`popagent_data_migrations`. Their configurable columns have no database
defaults, and startup never recreates a row after its migration marker exists.
PostgreSQL is the only runtime source; a missing row is a configuration error,
not a signal to restore code defaults.
- Delegation returns only specialist text to the supervisor: nested tool
payloads and run metadata stay out of model context. Context slicing, result
truncation, and failure feedback use the execution's persisted settings
snapshot before hook audit and supervisor synthesis.
- `AgentWorkspaceStore` persists repository definitions in
`popagent_agent_workspaces`. Repository paths are relative to
`POPAGENT_WORKSPACE_ROOT` (default `workspace/`) and must resolve inside that
root. The default registration points at the ignored `workspace/popagent` Git
checkout; `deploy.sh` clones the public repository when that checkout is absent.
Every session, background task, and schedule carries a `workspace_id`.
Request context resolves the supervisor and implementer to a contained writable
Mastra Workspace for that repository; researcher and reviewer resolve a
read-only view of the same path with no shell tools. Every role receives
Mastra's native `mastra_workspace_lsp_inspect` tool for TypeScript/JavaScript
hover, diagnostics, definitions, and implementations. Agent tool inputs remain
repository-relative. A contained `LocalFilesystem.exists` adapter translates
only Mastra's absolute internal LSP marker probes back beneath the repository;
other absolute and escaping paths remain rejected. LSP servers use a host
process manager rooted at the contained repository because bubblewrap cannot
expose host absolute URIs to the language server; agent shell commands remain
in the network-disabled `LocalSandbox`. The application pins
`typescript-language-server`.
Writable file tools require a current tool-level read before overwriting or
editing an existing file. `LocalFilesystem` containment and the bubblewrap
boundary prevent access outside the selected repository.
- Failed, cancelled, and dead-lettered tasks can be operator-resolved through `POST /api/tasks/:id/resolve` with a bounded evidence summary. Resolution preserves the task record, changes it to completed, and clears stale failure/retry fields; queued, running, and already-completed tasks reject resolution.
- PostgreSQL-backed project-folder Markdown documentation and workspace-filtered RAG use Mastra `MDocument`, AI SDK embeddings, `PgVector`, and local-model reranking; source, indexing, and retrieval contracts live in [AGENTS_DOCUMENTATION.md](AGENTS_DOCUMENTATION.md).
- Web search uses the local SearXNG container at `SEARXNG_URL` (default
`http://127.0.0.1:8889`). The `webSearch` runtime tool (`createTool` ID
`web-search`) requests SearXNG's JSON API
and is available to the supervisor and every subagent. Keep search
credential-free and self-hosted; do not add a hosted search API dependency.
- Browser automation, network policy, profiles, recordings, session lifecycle, and verification contracts live in [AGENTS_BROWSER.md](AGENTS_BROWSER.md).
- Agent Skills are inline Mastra skills resolved dynamically from
`popagent_agent_skills` for the supervisor and each specialist on every run.
The role data migration seeds two source-attributed, role-appropriate skills
per agent; all skill instructions, references, provenance, and enablement live
in PostgreSQL. Enabled skills provide Mastra's `skill`, `skill_read`, and
`skill_search` tools; disabled skills remain editable but are excluded from
agent execution. Skill names follow the Agent Skills lowercase-hyphen
specification, references are JSONB, and ownership is exposed as
`evolutionManaged` in the Agent Skill DTO. Every ordinary user/API update
clears `evolution_managed`; once edited, a learned skill is user-owned and
evolution cannot reclaim or revert it.
- Autonomous learning persists in `popagent_evolution_signals`,
`popagent_learned_overlays`, and append-only `popagent_evolution_revisions`.
Each signal retains the real turn trace ID plus bounded, secret-redacted
Goal/Outcome or Goal/Failure evidence with a failure classification.
Signals are deduplicated before a periodic worker claims due batches with
`FOR UPDATE SKIP LOCKED`; every run first recovers abandoned claims, and
concurrent worker start/stop transitions are serialized.
- Reflection uses a tool-free Mastra agent and the configured 9Router model.
It consumes Mastra's streaming structured-output contract because 9Router
returns SSE for this route. The model can select only a closed, code-owned
strategy enum; it cannot author
prompt or skill text. Every selection must cite claimed evidence for the same
agent. Application locks and rereads the autonomy singleton and
rechecks cancellation in the same transaction as learned mutation and signal
completion. A disabled global or target policy wins before mutation: claimed
signals become ignored and the claim attempt is released rather than retried
or dead-lettered. Other failures use capped exponential backoff and terminal
dead-lettering.
- Learned overlays are stored separately from `popagent_agents.instructions`
and composed after the immutable user-managed base on every agent resolution.
Automatic skill changes are limited to rows marked `evolution_managed`; a
user-managed skill with the same name is never overwritten. Automatic learning
cannot change tool membership, workspace/browser access, secrets, hooks,
authentication, execution bounds, or user-managed instructions. Secret-bearing
and protected-boundary proposals fail without partial mutation. Reverts append
a compensating revision with `revertsRevisionId`; a partial unique index permits
at most one compensation per source revision. Revert restores the prior
effective overlay or skill only when that source is still the latest target
version and the current managed target still matches it.
- Background tasks and schedules persist in `popagent_agent_tasks` and
`popagent_agent_schedules`. Both carry immutable `source: 'user' |
'self-update'` ownership and their originating workspace; materialized tasks
inherit both fields, so retry and recovery never change provenance or the
repository boundary.
Tasks persist `attempt_count`, `max_attempts` (default three),
`next_attempt_at`, `last_error_class`, and `dead_lettered_at`. Due
queued claims atomically increment attempts. Only contained `self-update`
tasks may automatically retry timeout, provider/rate-limit, browser, database,
or infrastructure failures: every attempt starts from a fresh clone and only
a verified commit can escape it. User tasks fail after an execution error
rather than replaying unknown side effects. Hook denial, cancellation/abort,
validation, permission/containment, missing configuration, and unknown
failures never retry; exhausted self-update work becomes terminal
`dead-letter` with durable explanatory progress.
- Each task persistently retains its latest 256 iteration activity records: agent identity,
step bounds, finish state, model text, and invoked tool names. This bounded transcript
survives polling, retry, terminal review, and restart without storing tool arguments or
results.
- Task polling rediscovers delayed work. Restart recovery preserves the prompt,
model, session, schedule, and workspace; it requeues only non-stale running
work below `maxAttempts`, dead-letters non-stale exhausted work, fails stale
work, and finalizes cancelling work as cancelled. Runtime start/stop operations
are serialized; a stop requested during startup lets initialization and one
recovery pass finish but prevents polling or execution from being resurrected.
- Each claimed attempt retains its executor promise. Timeout, cancellation, and
shutdown abort the signal, then wait a bounded acknowledgement interval while
retaining the occupied execution slot. Cancellation becomes durable only after
executor settlement; timeout becomes retry-eligible only after settlement. An
executor that ignores abort remains non-runnable in `running` or `cancelling`,
emits `interrupted`, holds its slot, and blocks same-runtime restart recovery
rather than overlapping a retry.
- Durable completion, failure, dead-letter, and cancellation transitions precede
terminal lifecycle delivery. Every attempt uses one correlated turn and trace
ID; completed events carry the outcome and failures carry their classified
error. `TaskStop`, channel, hook, browser-close, and evolution observers are
best effort after the transition: observer failure is logged and cannot emit a
contradictory failed terminal state. Intermediate retries still emit failed
`TaskStop` before a later `TaskStart`; final exhaustion emits `dead-letter`.
- `DELETE /api/tasks/:id` removes only terminal `completed`, `failed`,
`cancelled`, or `dead-letter` rows. It returns conflict for active work and
not-found for missing rows; task removal never cancels execution or rolls
back repository changes.
- Schedules persist `maxAttempts`; create/update preserve it and transactional
materialization copies it into each task under row locking, with enqueue only
after commit. The retry/schema migration holds a PostgreSQL advisory
transaction lock, rechecks its version marker after locking, and records the
marker only after the schema and one-time timezone/next-run backfill succeed.
Ordinary initialization never recalculates an existing schedule after that
marker. `src/agent-execution.ts` gives task work the same prompt,
delegation/tool hooks, browser activation, recall, lifecycle processors,
execution bounds, final-response guard, and episode retention as chat turns.
Schedule calculations use the server timezone; keep UI history limits separate
from the unbounded queued-work recovery query.
- `SelfUpdateScheduler` serially reconciles one `source='self-update'` schedule
per registered workspace and cadence. A PostgreSQL partial unique index owns
that `(workspace_id)` slot; ordinary schedule create is always `source='user'`
and ordinary update/delete match only user rows, so display names cannot claim
or mutate system schedules.
Scheduler lifecycle operations share one serialized subscription/reconcile/timer
state; a stop requested during initial reconciliation prevents later timer
resurrection, and a subsequent explicit start can initialize again.
- Scheduling requires enabled global/self-update policy, read-write supervisor
and implementer, delegation-eligible implementer and reviewer, reviewer
repository access, and a resolved writable Git repository. Both scheduled and
immediate materialization take the same PostgreSQL transaction advisory lock,
reject any globally active self-update task, and select at most one due
workspace, so only one autonomous task can be queued, running, or cancelling
across the application. Materialization copies source and `maxAttempts` while
atomically rereading enabled autonomy policy; disabling self-update therefore
blocks new work before reconciliation. Trusted self-update tasks propagate
`executionSource='self-update'`.
- Execution first requires a clean registered checkout and captures its HEAD,
branch, and remote digest, then works in a secret-free, network-disabled local
clone on a unique `autonomous/self-update/*` branch. Fixed runtime policy,
independent of prompts and managed hooks, removes memory mutation and
`useBrowserSecret`; denies hidden/credential/protected paths and remote,
publish, deploy, restart, or unrestricted shell operations; and keeps reviewer
workspace access read-only. The registered checkout is never the agent's
writable surface.
- Completion requires successful Researcher evidence collection. A clean no-op may complete immediately without Implementer or Reviewer delegation; changed results require successful Implementer execution and Reviewer verification. Acceptance permits only a clean no-op or clean descendant commit with unchanged remotes and an allowed diff. Continuous contained execution uses explicitly selected workspaces and its fixed local coding-model policy; PostgreSQL schedule state plus global active-task state gates one bounded cycle at a time.
- The model never receives credentials or direct publish/deploy tools. Accepted commits are imported as local refs. With explicit autonomous deployment enabled, fixed server policy may fast-forward the reviewed commit, invoke the repository's maintained `deploy.sh publish`, then launch `deploy.sh deploy`; failures remain visible and never force-push.
- Durable memory uses `@mastra/memory` with `@mastra/pg` `PostgresStoreVNext`
(`src/storage.ts`). `DATABASE_URL` is required for application and memory data;
`OBSERVABILITY_DATABASE_URL` is required for the isolated observability domain.
Production application PostgreSQL is `popagent-postgres` on `127.0.0.1:5433`;
observability PostgreSQL is `popagent-observability-postgres` on
`127.0.0.1:5435`. Never route observability writes into the application store.
- Repository schema initialization shares concurrent work, retains successful
completion, and resets after a failed attempt so transient database outages do
not poison a store until process restart.
- Session display metadata and AI SDK messages live in PostgreSQL. Sessions have
explicit archive timestamps and a title lock so a user rename survives later
message saves. Session rows retain their repository workspace; list APIs can
filter by it. Mastra also persists each conversation as a memory thread. Do not
reintroduce SQLite.
- Long conversations use Mastra Observational Memory in thread scope. The
Observer compacts old messages and tool results into observations and the
Reflector bounds the observation log; original messages remain in PostgreSQL.
Settings in `popagent_memory_settings` control enablement; observation and
reflection thresholds; raw-history retention; async observation/reflection
buffering and safety thresholds; idle/provider-change activation; shared
budgets; previous-observation context limits; temporal markers; attachment
forwarding; exact-history retrieval scope; and custom Observer/Reflector
guidance. Shared budgets disable async buffering as required by Mastra.
Each request sends only its newest UI message because Mastra reconstructs
history from the thread. Observations remain thread-scoped; retrieval can be
restricted to that thread or browse all threads for `popagent-user`. Observer
and Reflector calls use the selected 9Router model rather than Mastra's hosted
default. Configuration changes apply on the next turn.
- Delegated specialist messages use isolated generated resource/thread IDs. The pinned `@mastra/core` patch backfills those IDs onto provider response messages before persistence; removing it reintroduces `Thread ID is required` observability errors. `src/agent.test.ts` exercises the real delegated streaming path.
- Permanent chat deletion uses Mastra's `Memory.deleteThread()` before removing
thread state and the session row, so messages and vector artifacts follow the
framework lifecycle. Workspace-wide deletion runs that same lifecycle for
every active and archived session, including SessionEnd hooks, browser closure,
and hook-audit cleanup. Cross-session retention semantics live in
[AGENTS_MEMORY.md](AGENTS_MEMORY.md).
- Administrative direct-database cleanup requires a data-only backup of
`popagent_sessions`, `mastra_threads`, `mastra_messages`, and
`mastra_thread_state`. Reconcile orphaned `popagent-user` Mastra threads that
have no `popagent_sessions` row; test chat requests can create such threads
independently of UI sessions. Verify active/archived API lists and preserve
`popagent_memories`.
- Bootstrap configuration comes from the ignored `.env`: `DATABASE_URL` for the
Bun process, `POSTGRES_PASSWORD` for Docker Compose, and a base64-encoded
32-byte `POPAGENT_SECRET_KEY` for application-managed secret encryption.
- Application-managed secrets live encrypted in `popagent_secrets`
(`src/secrets.ts`), scoped by resource and exact name. AES-256-GCM authenticates
the resource ID and name as additional data. `popagent_secret_bindings` stores
metadata-only exact HTTPS origin bindings. The authenticated `/api/secrets`
interface provisions, rotates, binds, lists, and deletes credentials without
returning values.
- Agents have no plaintext secret create, update, or recall tools.
`useBrowserSecret` resolves a bound value server-side and fills only a browser
element whose page and frame origins both equal the configured origin. Its
model-visible result contains only the secret name, destination origin, and
success status.
- The chat stream merges specialist iteration telemetry as transient AI SDK
`data-agentActivity` parts beside the supervisor response. Persist those parts
with the assistant message so reopened sessions retain delegation evidence.
Usage and trace correlation metadata are defined in [AGENTS_OBSERVABILITY.md](AGENTS_OBSERVABILITY.md).
- Observability storage isolation, correlation, redaction, APIs, and retention live in [AGENTS_OBSERVABILITY.md](AGENTS_OBSERVABILITY.md).
- API authentication is controlled by `POPAGENT_API_KEY`. When set, every `/api/*` request must include the matching `x-popagent-key`; the root page and static assets remain public. Retired plaintext secret tool arguments/results are centrally redacted at chat ingress, session and Mastra-message persistence, hook dispatch, and transcript rendering.

- Session writes use optimistic revision locking: clients send `revision`; each successful save increments it atomically, and stale writes return `409 { error: "revision conflict", session }`. Session lists are capped at 200 rows.
- Mutation routes validate bounded JSON bodies and reject unknown models against the cached 9Router catalog; catalog outages fail open with a warning.
- Native long-term memory is separate from Mastra thread history. See
[AGENTS_MEMORY.md](AGENTS_MEMORY.md). Hook protocol and policy live in
[AGENTS_HOOKS.md](AGENTS_HOOKS.md).

## Contract-test exceptions

- `src/server.test.ts` owns the slow-stream idle-timeout configuration.
- Storage contracts require disposable local PostgreSQL.
- `src/agent.test.ts` and chat-stream scenarios require live 9Router models on
Titan.

## Live BifrOSt tool

Agents with persisted browser access receive `bifrostNavigator`, a single typed gateway to the local visible BifrOSt Navigator MCP catalog. The tool accepts an exact `browser_*` operation plus its catalog arguments, preserves MCP text/image/resource content, enforces `interactive` versus `read-only` role access server-side, and is absent during trusted self-update. BifrOSt remains a user-owned desktop process; Popagent connects only to its protected Unix socket and does not launch or stop it.