AkurAI-Framework / AGENTS_CRATES.md

Latest change 4ed55d17e0979479dc70291811217113b6062fa0 - docs: split AGENTS.md into domain-indexed subfiles + add breaking-changes guide by Ólafur Búi Ólafsson

# AGENTS_CRATES.md — the crate map

> The authoritative per-crate index. The list of workspace members lives in
> `Cargo.toml` `members` — keep this file in lockstep with it. Every crate is
> pure `std` plus, at most, other **workspace** crates; nothing here pulls an
> external runtime dependency (constitutional principle #1).

Crates inherit `version.workspace = true` from `Cargo.toml`'s `[workspace.package]`,
so `deploy.sh` bumps one number and every crate moves in lockstep
([`AGENTS_DEPLOY.md`](./AGENTS_DEPLOY.md)).

## Crate map

| Crate | Owns | Status |
|-------|------|--------|
| `crates/http` | HTTP/1.1 server, request/response, thread pool, connection upgrade, SSE, middleware — pure std | 🟢 working |
| `crates/ws` | WebSocket server (RFC 6455): handshake, framing, sha1+base64, conn — pure std | 🟢 working |
| `crates/router` | URL pattern matcher: static/`:param`/`*wildcard` → params, specificity-ranked — pure std; wired into `serve` via `backend/routes.json` | 🟢 working |
| `crates/json` | std-only JSON parse/serialize/value (int/float preserved, order-preserving objects) | 🟢 working |
| `crates/sql` | lexer + recursive-descent parser (SQL text → AST) for SELECT/INSERT/CREATE TABLE + WHERE expr tree — **parser-only** | 🟡 parser-only |
| `crates/sqlexec` | SQL executor: runs the `akurai-sql` AST over the `akurai-storage` B+tree — catalog, typed row codec, WHERE/ORDER BY/LIMIT (CREATE/INSERT/SELECT end-to-end) | 🟢 working |
| `crates/storage` | single-file pager, node codec, CoW B+tree + atomic meta + free list; the durable substrate every data crate sits on | 🟢 working |
| `crates/collections` | declarative collection schema + auto-CRUD records + substring search + `/_meta` manifest over `akurai-storage` | 🟢 working |
| `crates/vector` | embedding client (plain-HTTP OpenAI-compatible) + cosine/top-k rank + f32 codec — semantic-search primitives | 🟢 working |
| `crates/toml` | pure-std TOML parser (the slice needed for `backend/collections.toml`) | 🟢 working |
| `crates/template` | std-only SSR template engine (vars, raw, if/for/include, comments) | 🟢 working |
| `crates/markdown` | std-only Markdown → HTML (docs + changelog) incl. the live `preview` fence | 🟢 working |
| `crates/css` | from-scratch atomic/utility CSS engine (Tailwind-class): scan classes → generate only-used utilities | 🟢 working |
| `crates/assets` | static-asset pipeline: content-hash fingerprinting + conservative minify | 🟢 working |
| `crates/cli` | `akurai new \| serve \| dev \| version \| update` — bootstrap + lean static/SSR server; wires every crate above | 🟢 working |

The data-engine crates (`storage`, `sql`, `sqlexec`, `collections`, `vector`,
`toml`) compose into one query/record engine — see [`AGENTS_DATA.md`](./AGENTS_DATA.md)
for how they fit together and the shared keyspace contract.

---

## `crates/http`

The HTTP/1.1 server: TCP accept loop, request parsing, response building, a
std thread pool, the connection-hijack seam for upgrades (WebSocket), SSE, and an
ordered middleware stack. Pure std.

| File | Responsibility |
|------|----------------|
| `src/lib.rs` | crate root, module exports |
| `src/request.rs` | parse the HTTP request head + `body`/`content_length`/`body_str` |
| `src/form.rs` | `application/x-www-form-urlencoded` parse + percent-decode |
| `src/response.rs` | build/write HTTP responses (`to_bytes` + head-only `to_head_bytes`) |
| `src/reply.rs` | `Reply` (`Response \| Upgrade`) — the connection-hijack seam |
| `src/server.rs` | `TcpListener` accept loop; upgrades go to a dedicated thread; `run` / `run_with(stack, handler)` |
| `src/pool.rs` | std thread pool (`mpsc` + workers) |
| `src/sse.rs` | Server-Sent Events stream (`text/event-stream`, spec framing) |
| `src/middleware.rs` | `Middleware` trait + ordered `MiddlewareStack` (first pushed = outermost) |
| `src/middleware_builtins.rs` | built-ins: `SecurityHeaders`, `RequestLogger`, `Timing` |
| `tests/{body,middleware,sse}.rs` | end-to-end over a real socket |

**Gotcha:** an upgrade (WS) hijacks the connection on its own thread — don't
assume request→response is the only shape. Middleware order matters: first
pushed wraps outermost.

## `crates/ws`

WebSocket server (RFC 6455), built entirely from std — including SHA-1 and
base64 for the handshake, which exist here only to avoid a dependency.

| File | Responsibility |
|------|----------------|
| `src/lib.rs` | crate root + `upgrade(req, on_conn)` entry (101 or 400) |
| `src/handshake.rs` | recognise WS upgrade + compute `Sec-WebSocket-Accept` |
| `src/sha1.rs` | from-scratch SHA-1 (handshake only) + FIPS test vectors |
| `src/base64.rs` | from-scratch base64 encode (RFC 4648) |
| `src/frame.rs` | RFC 6455 §5 frame read/write, masking, length, limits |
| `src/message.rs` | `Message` enum + RFC 6455 §7.4.1 close codes |
| `src/conn.rs` | `WsConn`: reassembly, auto ping/pong, close handshake |
| `tests/echo.rs` | end-to-end: real handshake + masked echo + close |

**Gotcha:** `sha1.rs`/`base64.rs` are handshake-scoped — don't grow them into a
general crypto utility; that's scope creep against principle #9.

## `crates/router`

Transport-agnostic URL pattern matcher. Knows nothing about HTTP, so the same
matcher can back a future client-side router unchanged.

| File | Responsibility |
|------|----------------|
| `src/lib.rs` | `Router<T>` + `Match<'a, T>` (matched value + captured params) |
| `src/pattern.rs` | `Pattern`: parse `/posts/:id` / `/files/*path`, match + capture |

**Conventions:** static > param > wildcard specificity, so a general fallback
never shadows an exact route. Matching is linear over the table (brute force
first; a radix trie is the later scale optimization behind the same API). Wired
into `serve` via `backend/routes.json` (see `crates/cli/src/routes.rs`).

## `crates/json`

Order-preserving JSON value, parser, and serializer. The lingua franca between
crates (collections records, `/_meta`, template context, embedding bodies).

| File | Responsibility |
|------|----------------|
| `src/value.rs` | `Value` enum (order-preserving objects) + serializer (`to_json`) |
| `src/parse.rs` | recursive-descent parser (escapes, unicode, int/float) |
| `src/error.rs` | parse error with character position |

**Convention:** objects preserve insertion order — relied on across the codebase
(e.g. collection records put `id`/`created` first; embedding bodies emit `model`
then `input`). Integers and floats are distinct `Value` variants; don't collapse
them.

## `crates/sql`

SQL front end: text → AST. **Parser-only** — execution lives in `sqlexec`.

| File | Responsibility |
|------|----------------|
| `src/lib.rs` | crate root + `parse(sql) -> Result<Statement, SqlError>` + AST re-exports |
| `src/token.rs` | `Token` + `Keyword` table (case-insensitive) + position-tagged `SpannedToken` |
| `src/lexer.rs` | `tokenize` — SQL text → token stream; strings (`''` escape), numbers, operators, positions |
| `src/ast.rs` | AST types: `Statement`, `SelectStmt`, `InsertStmt`, `CreateTableStmt`, `Expr`, `Literal`, `ColumnDef`, `DataType`, `Direction`, … |
| `src/parser.rs` | recursive-descent parser; WHERE expr tree with `AND` > `OR` precedence climbing |
| `src/error.rs` | `SqlError { message, position }` — positioned, never panics |

**Convention:** the parser never panics — every malformed input returns a
positioned `SqlError`. It's a pure transform; it never touches storage.

## `crates/sqlexec`

The query back end: runs an `akurai-sql` AST over the `akurai-storage` B+tree.
Owns the catalog, a typed row codec, a WHERE evaluator, and the `Executor`.

| File | Responsibility |
|------|----------------|
| `src/lib.rs` | crate root; re-exports `Executor`, `Outcome`, `Value`; defines `ResultSet { columns, rows }` |
| `src/catalog.rs` | table schemas + their persistence; reserved key prefixes `\x00cat:` / `\x00seq:`; byte codec + `Reader` cursor |
| `src/value.rs` | the executor's runtime `Value` (Integer/Text/Real/Null) |
| `src/row.rs` | typed row encode/decode against a `Schema` |
| `src/eval.rs` | WHERE expression evaluation over a decoded row |
| `src/exec.rs` | `Executor`: `execute(sql)` → CREATE / INSERT / SELECT; data key `<table>:<rowid_be>` |
| `src/error.rs` | `ExecError` (TableExists, ColumnCountMismatch, Corrupt, …) |
| `tests/end_to_end.rs` | CREATE → INSERT → SELECT over a real DB file |

**Conventions & gotchas:** the catalog is *just more rows in the same tree* under
the reserved `\x00`-prefixed keyspace, which table data (SQL identifiers, never
starting `\x00`) can't collide with. Rowids auto-increment from a per-table
`\x00seq:` counter seeded at 0 on `CREATE TABLE`. `UPDATE`/`DELETE`, joins,
indexes, and transactions are deliberately out of this slice (queued in
[`BATCHES.md`](./BATCHES.md)). See [`AGENTS_DATA.md`](./AGENTS_DATA.md) for how
the SQL keyspace coexists with the `coll:` records keyspace.

## `crates/storage`

The durable substrate: a single-file, crash-safe, copy-on-write B+tree. Every
data crate (`sqlexec`, `collections`) stores its bytes here.

| File | Responsibility |
|------|----------------|
| `src/lib.rs` | crate root; re-exports `BTree` |
| `src/pager.rs` | single-file fixed-size page read/write/allocate + `sync` |
| `src/node.rs` | B+tree node byte layout: leaf + interior encode/decode |
| `src/meta.rs` | double-buffered checksummed meta page + inline free list |
| `src/tree.rs` | CoW `BTree`: `get`/`insert`/`delete`/`range`, splits, atomic `commit` |
| `tests/{pager,tree}.rs` | round-trip, persistence, durability, torn-commit recovery, range |

**Conventions & gotchas:** writes are buffered; **nothing is durable until
`commit()`**. Recovery is via the double-buffered meta page (a torn commit
rolls back to the last good root). `range(start, end)` is half-open `[start, end)`
— callers build prefix upper bounds themselves. Underfull-node merge/rebalance,
overflow-page blobs, and MVCC are queued, not done.

## `crates/collections`

The auto-generated-API engine: declare a `Collection` (typed `Field`s) and get
CRUD, substring search, and a `/_meta` manifest over the B+tree — no hand-written
endpoints. HTTP-agnostic; the CLI maps it onto REST.

| File | Responsibility |
|------|----------------|
| `src/lib.rs` | crate root; re-exports `Collection`, `Field`, `FieldKind`, `Store`, `meta`, `CollError` |
| `src/schema.rs` | `Collection` / `Field` / `FieldKind` (Text/Int/Float/Bool) + builders (`required()`, `embed()`) |
| `src/store.rs` | `Store`: `create`/`get`/`list`/`update`/`delete`/`search` over one `BTree`; key layout + validation/coercion |
| `src/meta.rs` | `meta(&[Collection])` → JSON manifest of fields + REST endpoints |
| `src/error.rs` | `CollError` (validation, corrupt, storage) |
| `tests/crud.rs` | create/get/list/update/delete/search round-trips |

**Conventions & gotchas:** the engine **owns and assigns** `id` (auto-increment
u64) and `created` (unix seconds) on every record — input may not set them, and
they're stored first, in order. Records live under `coll:<name>:<id_be>` with a
sibling `coll:<name>:_seq` counter; record keys are exactly `prefix + 8` bytes so
scans can filter `_seq` and future meta keys by length. `list` is newest-first;
`search` is a case-insensitive substring baseline over `Text` fields only — the
`embed` flag is stored but this crate embeds nothing (the CLI's vector layer
does). See [`AGENTS_DATA.md`](./AGENTS_DATA.md).

## `crates/vector`

Semantic-search primitives, two dependency-free halves: vector math + codec, and
a minimal plain-HTTP embedding client. Config-free — endpoint and model are
always parameters.

| File | Responsibility |
|------|----------------|
| `src/lib.rs` | crate root; re-exports `cosine`, `rank`, `encode`, `decode`, `embed`, `embed_many`, `Endpoint`, `EmbedInput`, `EmbedError` |
| `src/math.rs` | `cosine(a, b)` (clamped `[-1, 1]`, never panics) + top-k `rank` (stable sort, descending) |
| `src/codec.rs` | little-endian `f32` `encode`/`decode` so embeddings store as B+tree byte blobs |
| `src/http.rs` | minimal HTTP/1.1 client over `TcpStream`; `parse_endpoint`/`build_request`/`split_response`; **plain HTTP only** |
| `src/client.rs` | `embed`/`embed_many` → POST `/v1/embeddings`; `EmbedInput`; response parsing |
| `src/error.rs` | `EmbedError` (incl. `TlsUnsupported`, `InvalidEndpoint`) |

**Conventions & gotchas:** `cosine` returns `0.0` for mismatched lengths or a
zero-norm vector — safe, non-panicking defaults for a search path. An `https://`
endpoint is **rejected** with `EmbedError::TlsUnsupported` (std has no TLS; TLS
terminates at the edge, principle #2). Pure parts (endpoint parse, request build,
response split) are unit-tested against strings; the socket round-trip is not
(tests never touch the network). This crate wires no HTTP routes and never
touches collections — the CLI composes the two.

## `crates/toml`

A small, correct, pure-std TOML parser for the slice the framework needs to
declare collections in `backend/collections.toml`. Produces an order-preserving
`Value` mirroring `akurai-json`'s shape.

| File | Responsibility |
|------|----------------|
| `src/lib.rs` | crate root + `parse` + `Value` + `TomlError` re-exports; documents the supported subset |
| `src/lexer.rs` | tokenize TOML text |
| `src/parser.rs` | `parse(text) -> Result<Value, TomlError>` |
| `src/value.rs` | order-preserving `Value` |
| `src/error.rs` | `TomlError` |
| `tests/collections.rs` | parse a real `collections.toml` shape |

**Supported subset:** `key = value` (bare + dotted keys), basic strings,
integers/floats/booleans, inline arrays, `[table]` + `[[array of tables]]`
headers, `#` comments. **Out of scope (errors cleanly, never panics):** multi-line
`"""` strings, literal `'...'` strings, datetimes, inline `{...}` tables, and
non-decimal integers. Don't grow it past `collections.toml`'s needs.

## `crates/template`

The SSR/dynamic-content layer: a small server-side HTML templating language that
renders against a JSON `Value` context.

| File | Responsibility |
|------|----------------|
| `src/lib.rs` | `Engine`: `register(name, src)` + `render(name, ctx)`; named templates can `include` each other |
| `src/ast.rs` | `Node` AST |
| `src/parse.rs` | template source → `Vec<Node>` |
| `src/render.rs` | `Renderer`: walk nodes against the `Value` context |
| `src/error.rs` | `Error` |

**Syntax:** `{{ x }}` (HTML-escaped), `{{{ x }}}` (raw — e.g. the layout
`content` slot), `{% if x %}…{% else %}…{% endif %}`, `{% for v in xs %}…
{% endfor %}`, `{% include "name" %}`, `{# comment #}`. Paths are dotted
(`user.name`); loop variables shadow root. **Default to escaped output** — only
use `{{{ }}}` for trusted, already-safe HTML.

## `crates/markdown`

Pragmatic CommonMark subset → HTML for docs and changelogs.

| File | Responsibility |
|------|----------------|
| `src/lib.rs` | `to_html(md)`: headings, fenced code, lists, blockquotes, rules, paragraphs, **+ the `preview` fence** |
| `src/inline.rs` | inline code / bold / italic / links |

**The `preview` fence:** a ` ```preview ` fenced block renders its body **twice** —
once as raw, *unescaped* live HTML inside `.preview-demo`, and once as an escaped,
copyable code block inside `.preview-code`, wrapped in `.preview` with a
`.preview-label`. This powers the design-system live previews
([`AGENTS_FRONTEND.md`](./AGENTS_FRONTEND.md)). A normal fence still escapes its
body. Not yet supported: tables, nested lists, images.

## `crates/css`

From-scratch atomic/utility CSS engine (Tailwind-class): scan class names out of
HTML, emit only the utilities actually used.

| File | Responsibility |
|------|----------------|
| `src/lib.rs` | `build(sources: &[&str]) -> String` (scan all → emit only-used utilities, first-seen order) |
| `src/scan.rs` | extract class tokens from `class="…"`/`class='…'`; word-boundary, dedup, first-seen order |
| `src/utilities.rs` | utility grammar: class → CSS declaration body (keyword table + parametric 4px spacing) |
| `src/generate.rs` | assemble compact `.class{decl}` rules for recognised utilities, preserving order |
| `tests/build.rs` | HTML snippet → exactly the used rules; cross-source dedup |

**Status note:** the engine works as a library; wiring it into `serve`/`dev` to
emit `utilities.css` (with a dev watch) is a queued batch item
([`BATCHES.md`](./BATCHES.md)).

## `crates/assets`

Static-asset pipeline: content-hash fingerprinting for immutable caching, plus
conservative minification.

| File | Responsibility |
|------|----------------|
| `src/lib.rs` | re-exports `fingerprint`, `hashed_name`, `minify_css`, `minify_js` |
| `src/fingerprint.rs` | 64-bit FNV-1a content hash → lowercase hex; `hashed_name` inserts the hash before the extension |
| `src/minify.rs` | conservative `minify_css` (strips comments/whitespace, preserves strings + `url()`) and `minify_js` (strips comments/blank lines, preserves strings + templates) |
| `tests/pipeline.rs` | minify → fingerprint → hashed name; determinism + sensitivity |

**Convention:** minification is *conservative* — it never risks breaking
semantics (preserves strings, `url()`, template literals). Like `css`, the
library exists but the `serve`/`dev` wiring (fingerprint + minify + immutable
cache headers) is a queued batch item.

## `crates/cli`

The single shipped binary, `akurai`. Bootstraps a project, serves it (static +
SSR + route layouts + data loaders + dev live-reload), and self-updates. This is
the crate that **wires every other crate together** at the HTTP edge.

| File | Responsibility |
|------|----------------|
| `src/main.rs` | hand-rolled arg parsing + dispatch (`new` / `serve` / `dev` / `version` / `update`); `--help` |
| `src/cmd_new.rs` | bootstrap a project from embedded templates |
| `src/cmd_serve.rs` | static server + `/api/health`, `/api/_meta`, SPA fallback; route layouts (`{{{ content }}}` shell); per-route data loaders (merge `backend/<data>.json`, traversal-safe) |
| `src/cmd_update.rs` | `akurai update` self-updater; `asset_name()` resolves `akurai-<os>-<arch>` from the latest GitHub release (shared with `install.sh`) |
| `src/routes.rs` | load `backend/routes.json` → `akurai-router` of `RouteDef { template, layout, data }`; inject params; optional per-route `layout` + `data` |
| `src/seo.rs` | `robots.txt` + `sitemap.xml` (static routes) + per-route `<head>` meta; `site_url` from `page.json` |
| `src/store.rs` | append/list record store on the B+tree (`data/store.db`), JSON values, u64 keys |
| `src/guestbook.rs` | form-actions demo: POST validate + persist to B+tree + PRG redirect |
| `src/auth.rs` | auth helpers for the demo site |
| `src/dev_reload.rs` | dev-only SSE live-reload: file watcher + broadcast + script inject |
| `src/mime.rs` | file-extension → MIME mapping |
| `assets/*` | embedded starter page + config/collection templates |

**⚠️ Ownership:** `crates/cli/**` is owned by the orchestrator / a dedicated
agent. Worktree agents may **read** it but must not modify it without owning the
batch — at most one cli-touching feature per batch
([`AGENTS_TESTING.md`](./AGENTS_TESTING.md)).