AkurAI-Framework / AGENTS.md

Latest change dbd1e2529b1a8875ac975415d54ef3e98e046457 - merge: worktree agent a89bbaec936462de5 by Ólafur Búi Ólafsson

# AGENTS.md — AkurAI-Framework

> The map. Read this first. Every subsystem is a small, single-responsibility
> crate; every crate is many small files. Land in the right file via the tables
> below — don't grep blind.

## What this is

A **single-binary, pure-Rust, zero-runtime-dependency** full-stack web framework
in the spirit of PocketBase: embedded relational database, auto-generated REST
API from a declarative schema, built-in semantic search, and a from-scratch
client-first frontend toolchain. Copy one binary to a box, run it, done.

## Constitutional principles (do not violate)

1. **Zero runtime dependencies.** The shipped binary links NO external crates —
   `std` only. Internal workspace crates are fine. Dev/test/bench tooling
   (proptest, criterion, Playwright) is allowed; it never enters the binary.
2. **TLS terminates at the edge** (Caddy/nginx). The server speaks plain HTTP on
   localhost. std cannot do TLS and we will not pull a crate for it.
3. **Client-first.** The browser does rendering, routing, and state. The server
   serves the data API and does SSR only for SEO / dynamic content that needs it.
   Keep the server lean.
4. **No-build frontend.** Native ES modules + import maps + web components +
   server-rendered HTML. NO JS bundler/transpiler. Win DX/memory by subtraction.
5. **Declarative source of truth.** `backend/collections.toml` defines the data
   model; the API, validation, migrations, and embeddings all derive from it.
6. **Self-describing.** `GET /api/_meta` emits a machine-readable manifest of
   every generated route. Agents discover the whole surface in one call.
7. **TDD, always.** Failing test first, every phase. No subsystem advances
   without its test layer (unit → property → integration → e2e/visual).
8. **Performance is a gate, not a hope.** Budgets (cold start, idle memory,
   hot-reload latency, throughput) are asserted; regressions fail the release.
9. **Small scoped files.** One responsibility per file. If a file grows two
   concerns, split it and index it here.
10. **Docs are first-class.** Maintain a `/docs` documentation site in a
    Next.js/Vue-style layout (sidebar nav, sectioned guides + API reference) and
    keep it updated as the framework evolves — every new crate or feature ships
    with its docs in the same commit.
11. **Data updates without a rebuild — essential.** All application data lives
    in the runtime data store (crash-safe, WAL-style durability) and editable
    context files (e.g. `backend/page.json`), never compiled into the binary.
    Changing data must NEVER require recompiling or redeploying the binary.

## Crate map

| Crate | Owns | Status |
|-------|------|--------|
| `crates/http` | HTTP/1.1 server, request/response, thread pool, connection upgrade, SSE — pure std | 🟢 working |
| `crates/ws` | WebSocket server (RFC 6455): handshake, framing, sha1+base64, conn — pure std | 🟢 working |
| `crates/router` | URL pattern matcher: static/`:param`/`*wildcard` → params, specificity-ranked — pure std; wired into `serve` via `backend/routes.json` | 🟢 working |
| `crates/cli` | `akurai new \| serve \| dev` — bootstrap + lean static server | 🟢 working |
| `crates/json` | std-only JSON parse/serialize/value (int/float preserved) | 🟢 working |
| `crates/storage` | single-file pager ✅, node codec ✅, CoW B+tree + atomic meta + free list ✅; **wired into `serve`** via cli `store.rs`; blobs/rebalance next | 🟢 working |
| `crates/sql` | lexer ✅ + recursive-descent parser ✅ (SQL text → AST) for SELECT/INSERT/CREATE TABLE + WHERE expr tree; **parser-only** — planner/executor/catalog/types/MVCC next | 🟡 parser-only |
| `crates/vector` | embedding client, vector store, cosine + HNSW semantic search | ⬜ planned |
| `crates/api` | schema, auto-router, filter/rules DSL, auth, records, `/_meta` | ⬜ planned |
| `crates/template` | std-only SSR template engine (vars, if/for/include) | 🟢 working |
| `crates/markdown` | std-only Markdown → HTML (docs + changelog) | 🟢 working |
| `crates/css` | from-scratch atomic/utility CSS engine (Tailwind-class): scan classes → generate only-used utilities via `build()` — pure std | 🟢 working |
| `crates/assets` | static-asset pipeline: content-hash fingerprinting (`fingerprint`/`hashed_name`) + conservative `minify_css`/`minify_js` — pure std | 🟢 working |
| `crates/server` | (folded into `crates/cli` for now) SSR-on-demand later | ⬜ planned |
| `bench` | criterion micro-benchmarks + whole-stack load gen + perf budgets | ⬜ planned |

### `crates/http` files

| File | Responsibility | Status |
|------|----------------|--------|
| `src/request.rs` | parse the HTTP request head + `body`/`content_length`/`body_str` | ✅ |
| `src/form.rs` | `application/x-www-form-urlencoded` parse + percent-decode | ✅ |
| `src/lib.rs` | crate root, module exports | ✅ |
| `src/response.rs` | build/write HTTP responses (`to_bytes` + head-only `to_head_bytes`) | ✅ |
| `src/reply.rs` | `Reply` (Response \| Upgrade) — the connection-hijack seam | ✅ |
| `src/server.rs` | `TcpListener` accept loop; upgrades go to a dedicated thread; `run`/`run_with(stack, handler)` | ✅ |
| `src/pool.rs` | std thread pool (`mpsc` + workers) | ✅ |
| `src/sse.rs` | Server-Sent Events stream (`text/event-stream`, spec framing) | ✅ |
| `src/middleware.rs` | `Middleware` trait + ordered `MiddlewareStack` (first pushed = outermost) | ✅ |
| `src/middleware_builtins.rs` | built-ins: `SecurityHeaders`, `RequestLogger`, `Timing` | ✅ |
| `tests/middleware.rs` | end-to-end: `run_with` stack adds headers + short-circuits over a real socket | ✅ |

### `crates/ws` files

| File | Responsibility | Status |
|------|----------------|--------|
| `src/lib.rs` | crate root + `upgrade(req, on_conn)` entry (101 or 400) | ✅ |
| `src/handshake.rs` | recognise WS upgrade + compute `Sec-WebSocket-Accept` | ✅ |
| `src/sha1.rs` | from-scratch SHA-1 (handshake only) + FIPS test vectors | ✅ |
| `src/base64.rs` | from-scratch base64 encode (RFC 4648) | ✅ |
| `src/frame.rs` | RFC 6455 §5 frame read/write, masking, length, limits | ✅ |
| `src/message.rs` | `Message` enum + RFC 6455 §7.4.1 close codes | ✅ |
| `src/conn.rs` | `WsConn`: reassembly, auto ping/pong, close handshake | ✅ |
| `tests/echo.rs` | end-to-end: real handshake + masked echo + close | ✅ |

### `crates/json` files

| File | Responsibility | Status |
|------|----------------|--------|
| `src/value.rs` | `Value` enum (order-preserving objects) + serializer | ✅ |
| `src/parse.rs` | recursive-descent parser (escapes, unicode, int/float) | ✅ |
| `src/error.rs` | parse error with character position | ✅ |

### `crates/sql` files

| File | Responsibility | Status |
|------|----------------|--------|
| `src/lib.rs` | crate root + `parse(sql) -> Result<Statement, SqlError>` + AST re-exports | ✅ |
| `src/token.rs` | `Token` + `Keyword` table (case-insensitive) + position-tagged `SpannedToken` | ✅ |
| `src/lexer.rs` | `tokenize` — SQL text → token stream; strings (`''` escape), numbers, operators, positions | ✅ |
| `src/ast.rs` | AST types: `Statement`, `SelectStmt`, `InsertStmt`, `CreateTableStmt`, `Expr`, `Literal`, … | ✅ |
| `src/parser.rs` | recursive-descent parser; WHERE expr tree with `AND`>`OR` precedence climbing | ✅ |
| `src/error.rs` | `SqlError { message, position }` — positioned, never panics | ✅ |
| _next_ | planner + executor over the storage B+tree, catalog, type checking, txn/MVCC | ⬜ |

### `crates/storage` files

| File | Responsibility | Status |
|------|----------------|--------|
| `src/pager.rs` | single-file fixed-size page read/write/allocate + `sync` | ✅ |
| `src/node.rs` | B+tree node byte layout: leaf + interior encode/decode | ✅ |
| `src/meta.rs` | double-buffered checksummed meta page + inline free list | ✅ |
| `src/tree.rs` | CoW `BTree`: get/insert/delete/range, splits, atomic commit | ✅ |
| `tests/pager.rs` | integration: round-trip, persistence, durability, bounds | ✅ |
| `tests/tree.rs` | integration: torn-commit recovery, bulk reopen, range | ✅ |
| _next_ | underfull-node merge/rebalance, blobs (overflow pages), MVCC | ⬜ |

### `crates/cli` files

| File | Responsibility | Status |
|------|----------------|--------|
| `src/main.rs` | arg parsing + command dispatch (`new`/`serve`/`dev`/`version`) | ✅ |
| `src/cmd_new.rs` | bootstrap a project from embedded templates | ✅ |
| `src/cmd_serve.rs` | static server + `/api/health`, `/api/_meta`, SPA fallback; route layouts (wrap a page in a shared shell via `{{{ content }}}`); per-route data loaders (merge `backend/<data>.json` into the render context, traversal-safe) | ✅ |
| `src/mime.rs` | file-extension → MIME mapping | ✅ |
| `src/dev_reload.rs` | dev-only SSE live-reload: file watcher + broadcast + script inject | ✅ |
| `src/routes.rs` | load `backend/routes.json` → `akurai-router` of `RouteDef { template, layout, data }`; inject params into context; optional per-route `layout` + `data` loader file | ✅ |
| `src/seo.rs` | `robots.txt` + `sitemap.xml` (static routes only) + per-route `<head>` meta (`meta_title`/`meta_description`); `site_url` from `page.json` | ✅ |
| `src/store.rs` | append/list record store on the B+tree (`data/store.db`), JSON values, u64 keys | ✅ |
| `src/guestbook.rs` | form-actions demo: POST validate + **persist to B+tree** + PRG redirect | ✅ |
| `assets/*` | embedded starter page + config/collection templates | ✅ |

### `crates/css` files

| File | Responsibility | Status |
|------|----------------|--------|
| `src/lib.rs` | crate root + `build(sources: &[&str]) -> String` (scan all → emit only-used utilities, first-seen order) | ✅ |
| `src/scan.rs` | extract class tokens from `class="…"`/`class='…'` attrs; word-boundary match, dedup, first-seen order | ✅ |
| `src/utilities.rs` | utility grammar: class → CSS declaration body (keyword table + parametric 4px spacing) | ✅ |
| `src/generate.rs` | assemble compact `.class{decl}` rules for recognised utilities, preserving order | ✅ |
| `tests/build.rs` | end-to-end: HTML snippet → exactly the used rules; cross-source dedup | ✅ |

### `crates/assets` files

| File | Responsibility | Status |
|------|----------------|--------|
| `src/lib.rs` | crate root + re-exports (`fingerprint`, `hashed_name`, `minify_css`, `minify_js`) | ✅ |
| `src/fingerprint.rs` | 64-bit FNV-1a content hash → lowercase hex; `hashed_name` inserts hash before the extension | ✅ |
| `src/minify.rs` | conservative `minify_css` (comments/whitespace, preserves strings + `url()`) and `minify_js` (comment/blank-line strip, preserves strings + templates) | ✅ |
| `tests/pipeline.rs` | end-to-end: minify → fingerprint → hashed name; determinism + sensitivity | ✅ |

## Directory layout

```
AkurAI-Framework/
├── AGENTS.md            ← this file (the index)
├── CHANGELOG.md         ← Keep-a-Changelog; deploy.sh cuts releases
├── VERSION              ← starts 0.0.1; deploy.sh bumps lockstep
├── deploy.sh            ← gate → bump → changelog → tag → push (creates repo)
├── Cargo.toml           ← workspace root
├── crates/              ← one subsystem per crate, small files
├── frontend/            ← native-ESM SPA + admin UI (embedded at build)
├── backend/             ← collections.toml, rules/, migrations/ (declarative SSOT)
├── data/                ← store.db + blobs/ (gitignored)
├── site/                ← the framework's OWN website (the framework serves it):
│                          frontend/ (landing, docs.html, components.html,
│                          signals.js = no-build reactivity runtime + reactivity.html,
│                          spa.js = opt-in client-side SPA router + spa.html) +
│                          content/ (docs/*.md, changelog.md) + backend/page.json.
│                          Deployed to akurai-framework.olibuijr.com via deploy.sh ec2.
└── tests/e2e/           ← Playwright visual scripts + screenshot snapshots
```

## Semantic search (built-in, all collections)

Embedding model runs externally (OpenAI-compatible endpoint, e.g. Titan
EmbeddingGemma at `:8081`). The framework owns storage + search in pure Rust:
auto-embed configured text fields on write → store vector → rank by similarity.
Brute-force cosine first (fine at small/medium scale), HNSW later for scale.
Every collection gets `?search=<text>` for free.

## Testing & release

- **Unit:** inline `#[cfg(test)]` per module.
- **Property/fuzz:** proptest against SQL parser + B-tree invariants (dev-dep).
- **Integration:** `tests/` per crate (real DB files, real HTTP).
- **E2E/visual:** Playwright in `tests/e2e/` with screenshot snapshots; locally
  also verify with Interceptor (real Chrome).
- **Perf:** `bench/` with criterion + a load generator; budgets enforced in `deploy.sh`.
- **Release:** `./deploy.sh [patch|minor|major]` runs the gate, bumps VERSION +
  crate manifests, cuts the CHANGELOG, tags `vX.Y.Z`, pushes (creating the GitHub
  repo on first run).

### Changelog policy

**Every feature ships its changelog entry in the same commit** — the same rule
the docs follow (principle #10). Two files, always together:

- `CHANGELOG.md` — developer-facing, detailed; under `## [Unreleased]`.
- `site/content/changelog.md` — visitor-facing, curated; under `## Unreleased`.

A non-blocking `pre-commit` hook nudges you when a commit touches `crates/**` or
`site/frontend/**` without staging `CHANGELOG.md`. It only warns (always exits
0) so docs-only / refactor / WIP commits are never blocked. Enable it once per
clone:

```
git config core.hooksPath scripts/hooks
```

`deploy.sh` then cuts both `## [Unreleased]` / `## Unreleased` sections into a
dated, versioned release in lockstep.

## Roadmap (build order — each gated by its tests)

1. HTTP core + JSON
2. Storage engine (CoW B+tree, crash-safe) — the bedrock
3. SQL-A: single-table SELECT/INSERT/UPDATE/DELETE, WHERE, ORDER BY, LIMIT
4. SQL-B: indexes + index selection, JOIN, GROUP BY/aggregates
5. SQL-C: optimizer, hash joins, subqueries, txn/MVCC hardening
6. Vector/semantic search
7. Auto-API layer (schema → routes → `/_meta`)
8. Auth + relations + SSE realtime + uploads
9. CSS engine (dev watch + build) + frontend + admin UI
10. SSR-on-demand + perf-budget enforcement