AkurAI-Framework / AGENTS_DATA.md

Latest change 4ed55d17e0979479dc70291811217113b6062fa0 - docs: split AGENTS.md into domain-indexed subfiles + add breaking-changes guide by Ólafur Búi Ólafsson

# AGENTS_DATA.md — the data engine

> How the embedded database is built out of small crates, what the shared B+tree
> keyspace looks like, and how the `?search` / embedding lifecycle works. Read
> [`AGENTS_CRATES.md`](./AGENTS_CRATES.md) for each crate's file list; this file
> is about how they **compose**.

## The stack

Everything durable sits on one crash-safe B+tree file. Two independent data
models share it, plus the semantic-search layer on top:

```
                         ┌────────── crates/cli ──────────┐
                         │ wires HTTP edge + ?search glue  │
                         └───────┬──────────────┬──────────┘
              SQL path           │              │     records path
   ┌─────────────────────────────┘              └────────────────────────────┐
   │                                                                          │
crates/sql   →  crates/sqlexec                          crates/toml  →  crates/collections
(text→AST)      (AST→rows)                              (parse        (schema + CRUD +
                    │                                     collections.toml)   substring search)
                    │                                                          │   │
                    └──────────────┬───────────────────────────────┬──────────┘   │
                                   ▼                                ▼               ▼
                            crates/storage (single-file CoW B+tree)        crates/vector
                                                                   (embeddings + cosine/rank)
```

- **`crates/storage`** — the substrate. A single-file, copy-on-write, crash-safe
  B+tree: `get`/`insert`/`delete`/`range`, atomic `commit()`. Nothing is durable
  until `commit()`; a torn commit rolls back to the last good root via the
  double-buffered checksummed meta page.
- **`crates/sql` + `crates/sqlexec`** — the relational query path. `sql` parses
  text to an AST (parser-only, never panics). `sqlexec` runs that AST over the
  B+tree: a catalog, a typed row codec, a WHERE evaluator, and the `Executor`
  (CREATE / INSERT / SELECT end-to-end).
- **`crates/collections`** — the auto-API record path. A declarative schema
  (`Collection` of typed `Field`s) yields CRUD + substring search + a `/_meta`
  manifest, with the engine owning `id`/`created`. This is the PocketBase-style
  surface that becomes REST.
- **`crates/vector`** — the semantic-search primitives layered on the record
  path: an f32 codec to store embeddings as B+tree blobs, `cosine`/`rank`, and a
  plain-HTTP embedding client.
- **`crates/toml`** — parses `backend/collections.toml`, the declarative source
  of truth (principle #5), into the schema the CLI hands to `collections`.

The two data models are **deliberately separate** today: SQL tables and
auto-API collections are different surfaces over the same tree, kept apart by
their key prefixes (below). The CLI is the only place they meet.

## The shared keyspace

All of it lives in one B+tree, partitioned by key prefix. Three reserved zones:

| Zone | Key shape | Owner | Notes |
|------|-----------|-------|-------|
| **Collection records** | `coll:<name>:<id_be>` (8-byte big-endian id) | `crates/collections` | sorts in numeric/insertion order; `id` auto-increments |
| **Collection counter** | `coll:<name>:_seq` (BE u64) | `crates/collections` | last id used; shares the `coll:<name>:` prefix but differs in length |
| **SQL catalog** | `\x00cat:<table>` | `crates/sqlexec` | table schema bytes; survives reopen |
| **SQL rowid counter** | `\x00seq:<table>` (BE u64) | `crates/sqlexec` | seeded at 0 on `CREATE TABLE` |
| **SQL row data** | `<table>:<rowid_be>` | `crates/sqlexec` | the table's own rows |

Two invariants make this collision-free:

1. **The `\0` (`\x00`) prefix is reserved for SQL metadata.** SQL identifiers are
   never allowed to begin with `\x00`, so `\x00cat:` / `\x00seq:` can never
   collide with table data keys (`<table>:<rowid>`).
2. **The `coll:` prefix is reserved for the auto-API record engine.** Within a
   collection, record keys are *exactly* `len("coll:<name>:") + 8` bytes. The
   `_seq` counter (and any future per-collection meta key) shares the prefix but
   has a different length, so a `range` scan filters strictly on length and never
   confuses records with the counter. Big-endian ids mean a prefix range scan
   yields records in id (insertion) order for free.

When adding a new keyspace, pick a prefix that can't collide with these and
document it **here** plus in the owning crate's module docs.

## Collection record shape

A stored record is a JSON object whose first two keys are always the
engine-assigned `id` (int) and `created` (unix seconds, int), followed by the
supplied schema fields in declaration order. The engine **owns** `id`/`created`:
input may not set them, and `update` may not patch them. Field validation and
coercion (e.g. `Int` accepted where `Float` is declared) happen on `create`;
`update` re-validates only the patched keys. `list` returns newest-first;
`search` is a case-insensitive substring scan over `Text` fields only.

The `/_meta` manifest (`collections::meta`) describes every collection's fields
(`name`/`type`/`required`/`embed`) and the REST endpoints the CLI exposes:
`list` / `create` / `get` / `update` / `delete` / `search` under
`/api/collections/<name>/records`. The CLI merges this into the framework-wide
`GET /api/_meta` (principle #6).

## The embedding / `?search` lifecycle

Semantic search is built-in and free on every collection, but the **model runs
externally** (an OpenAI-compatible `/v1/embeddings` endpoint, e.g. Titan
EmbeddingGemma at `:8081`). The framework owns storage + ranking in pure Rust;
it never embeds in-process.

Write path (on `create`, for fields marked `embed`):

1. `collections` stores the record under `coll:<name>:<id_be>` and exposes the
   `embed` flag in `/_meta` — but **embeds nothing itself**.
2. The CLI's vector layer reads the flag, calls `vector::embed(endpoint, model,
   text)` (which POSTs to `<AKURAI_EMBED_URL>/v1/embeddings`), and stores the
   returned vector as a little-endian f32 blob (`vector::encode`) in the B+tree
   alongside the record.

Read path (`GET …/records?search=<query>`):

1. With `AKURAI_EMBED_URL` set: embed the query, `vector::decode` each stored
   embedding, `vector::cosine` against it, and `vector::rank` the top-k.
2. With **no** endpoint configured: fall back to `collections`' substring
   `search`. **That fallback lives in the CLI, not in `vector`** — the vector
   crate is config-free, endpoint/model are always parameters.

Honest constraints (don't fake around them):

- **No TLS.** `vector`'s HTTP client is plain HTTP only; an `https://` endpoint
  returns `EmbedError::TlsUnsupported` (std has no TLS; TLS terminates at the
  edge, principle #2). Point it at a plain-HTTP embedding endpoint behind your
  own edge.
- **Brute-force cosine first.** Fine at small/medium scale; HNSW is the later
  scale optimization (queued in [`BATCHES.md`](./BATCHES.md)), droppable behind
  the same `rank` API.
- `cosine` returns `0.0` for mismatched lengths or a zero-norm vector, and
  `decode` returns `None` for a non-multiple-of-4 blob — the search path never
  panics on bad data.

## Reference docs

User-facing docs for this domain (keep them updated with the code, principle #10
— see [`AGENTS_DOCS.md`](./AGENTS_DOCS.md)):
`site/content/docs/{storage,sql,sql-executor,collections,semantic-search,toml}.md`.