popagent

public

Latest change d1541efe163924a2d4eed6a64451b69c409dad80 - Refresh documentation storage and embedding facts by Ólafur Búi Ólafsson

# Project documentation and RAG

PostgreSQL is the canonical source for private project documentation. Page bodies are not written into registered Git repositories.

## Workspaces, folders, files, and UI

- `GET/POST /api/workspaces` and `PATCH/DELETE /api/workspaces/:id` manage repository registrations beneath `POPAGENT_WORKSPACE_ROOT`. The non-deletable default is seeded once. Deletion is blocked while sessions, tasks, schedules, channel history, or documentation reference a workspace.
- Each workspace owns a PostgreSQL-backed project-folder tree and Markdown files in `popagent_document_folders` and `popagent_document_pages`.
- Folder and file paths reject absolute paths, traversal, hidden path segments, empty segments, and non-Markdown files. Pages over 1,000,000 characters are rejected.
- The documentation workspace is routed under `/workspaces/:workspaceId/docs/*`. It provides a collapsible folder tree, Markdown editor, GFM preview, semantic search, deletion, and manual reindexing.
- Writes use the SHA-256 content revision returned by the read API. Stale updates and deletes return a revision conflict. Saves reindex the page.
- Documentation is private application data backed up with PostgreSQL. Popagent does not commit documentation automatically. On 2026-08-14, 200 active AkurAI Notes were imported into the default workspace; `source_note_id` makes the migration idempotent, and archived Notes were intentionally excluded.

## Indexing

- `DocumentationIndex` uses `MDocument.fromMarkdown()` from `@mastra/rag`, Markdown-aware chunks, AI SDK embeddings through an OpenAI-compatible LAN service, `PgVector` from `@mastra/pg`, and a final relevance pass through the current database-configured local language model.
- `POPAGENT_EMBEDDING_URL` and `POPAGENT_EMBEDDING_API_KEY` may select a dedicated embedding service; otherwise they inherit 9Router configuration. Production uses the LAN endpoint `http://192.168.1.10:8081/v1`, never Tailnet `100.x` addresses, with `text-embedding-3-small` at 1024 dimensions. The code defaults to 1536 dimensions when `POPAGENT_EMBEDDING_DIMENSION` is unset. On dimension change, initialization replaces the derived vector index and clears source-index metadata so every page is reindexed. Reranking uses the current `popagent_agent_runtime_settings.default_model` and falls back to vector order if the model is unavailable.
- The application PostgreSQL image includes pgvector. Vectors remain in the application database, never the isolated observability database.
- `popagent_document_sources` records workspace/path hashes, model, dimension, chunk counts, and indexing timestamps. Unchanged hashes are skipped; replaced and removed pages delete stale vectors.
- Vector metadata always contains the fixed resource, workspace ID, page path, title, heading, content hash, and chunk text. Workspace and resource filters are server-controlled.

## Retrieval

- `GET /api/docs/search` serves explicit UI search and may search one workspace or all registered workspaces.
- The `searchDocumentation` Mastra tool searches only the current workspace from trusted `RequestContext`; agents cannot override that boundary.
- Retrieved documentation text is untrusted reference material, not instructions. Agent answers should cite the returned workspace, path, and heading and state when documentation is missing or contradictory.
- Documentation retrieval is separate from Mastra thread memory and `popagent_memories`. Retrieved chunks are not automatically retained as long-term memory.

## Verification

- Workspace route contracts in `src/server.test.ts` own registration containment, mutation, and deletion references.
- `src/documentation-files.test.ts` owns database-backed folder/file operations, revision conflicts, deletion, and path validation.
- `src/ui/routes.test.ts` owns durable documentation URLs.
- Storage/index integration requires disposable PostgreSQL with pgvector and a live compatible embedding route on Titan.
- Documentation UI changes require desktop and mobile browser verification under the repository UI workflow.