AkurAI-Framework / site/content/docs/collections.md

Latest change d88533200d3e5293cc2264fb91be84f0c0c5bcaa - feat(cli): wire relations, file uploads, and expand into the auto-API by Ólafur Búi Ólafsson

# Collections — the auto-generated API engine

Declare a **collection** — a name plus a set of typed fields — and the framework
gives you a full REST resource over the embedded database: create, read, list,
update, delete, and a baseline search. No hand-written endpoints, no ORM, no
external dependencies. The engine is the `akurai-collections` crate.

## The schema model

A collection is a `name` and an ordered list of `fields`. Each field declares:

| Property   | Meaning                                                              |
|------------|---------------------------------------------------------------------|
| `name`     | The field's key in the JSON record. Must be unique in the collection. |
| `kind`     | One of `text`, `int`, `float`, `bool`, `relation`, `file`.          |
| `required` | When true, the field must be present and non-null on create.        |
| `embed`    | Marks a `text` field for **later** semantic indexing (see below).   |

The CLI builds these schemas from your `collections.toml` at startup. The engine
itself is decoupled from TOML — it consumes plain schema values.

### Field types

- **`text`** — a UTF-8 string. Only `text` fields participate in search.
- **`int`** — a 64-bit signed integer.
- **`float`** — a 64-bit float. Integer inputs are accepted and coerced to float.
- **`bool`** — `true` / `false`.
- **`relation`** — a reference to another collection's record by its `id` (see
  [Relations](#relations) below). The stored value is the referenced record's
  integer id; the `collection` property names the target.
- **`file`** — an uploaded file. The stored value is a descriptor object
  `{ blob, filename, content_type, size }` (or `null` when unset); the bytes
  live in the content-addressed blob store. Files arrive over
  `multipart/form-data`, not in a JSON body — see [Uploads](/docs/uploads) and
  the [auto-API](/docs/auto-api) download endpoint.

## Reserved fields the engine owns

Every stored record carries two engine-assigned keys that your input may **not**
set:

- **`id`** — an auto-incrementing `u64`, unique per collection, never reused
  (even after deletes).
- **`created`** — the creation time in unix seconds.

A stored record is a JSON object whose first two keys are always `id` and
`created`, followed by the schema fields you supplied, in declaration order.
Optional fields you omit are left out of the record entirely (not stored as
`null`).

## Record key layout

Records live in the B+tree under a per-collection prefix:

```text
coll:<name>:<id>     <id> is a big-endian u64   →  the record (JSON bytes)
coll:<name>:_seq     the auto-increment counter (big-endian u64, last id used)
```

Because ids are fixed 8-byte big-endian, they sort numerically, so a range scan
over the prefix returns records in insertion order. The `_seq` counter shares
the prefix but has a different key length, so scans filter on the exact record
key length and never confuse the two. Each collection has its own counter, so
ids in `posts` and `tags` are independent.

## The REST endpoints

For a collection named `posts`, the framework exposes:

| Method & path                              | Action                          |
|--------------------------------------------|---------------------------------|
| `GET /api/collections/posts/records`       | List records (newest first).    |
| `POST /api/collections/posts/records`      | Create a record from a JSON body. |
| `GET /api/collections/posts/records/<id>`  | Fetch one record by id.         |
| `PATCH /api/collections/posts/records/<id>`| Partially update a record.      |
| `DELETE /api/collections/posts/records/<id>`| Delete a record.               |
| `GET /api/collections/posts/records?search=<query>` | Substring search. |

`list` and `search` return newest-first (highest id first) and accept an
optional `limit`.

These same descriptors are emitted by the engine's manifest and merged into the
framework-wide `/api/_meta` document, so clients can discover every collection,
its fields, and its endpoints at runtime.

## Validation rules

The engine never panics on bad input — it returns a validation error.

**On create:**

- Every `required` field must be present and non-null.
- Every supplied value must match its field's declared type (`int` is accepted
  where a `float` is expected and coerced).
- Unknown fields (not in the schema) are rejected.
- The reserved keys `id` and `created` may not be supplied.
- The body must be a JSON object.

**On update (PATCH):** the patch is partial — only the keys you send change.

- Each changed field is re-validated against the schema (type checked).
- Unknown fields and the reserved keys are rejected.
- Setting a `required` field to `null` is rejected.
- Setting an *optional* field to `null` clears (removes) it from the record.
- Updating a non-existent id returns "not found" (no error).

## Search

`?search=<query>` is an **honest baseline**: a case-insensitive substring match
across the collection's `text` fields. A record matches if any of its `text`
fields contains the query. Non-text fields are never searched. An empty query
matches everything. Results are newest-first and honor `limit`.

This is a deliberate, predictable baseline — **semantic ranking is layered on
top** by the framework's vector layer, not by this engine.

## Relations

A **relation** field references a record in another collection by its `id`. The
field stores a plain integer (the referenced record's id); the schema records
which collection it points at.

### Declaring a relation

In `collections.toml`, a relation field is `type = "relation"` plus a
`collection` key naming the target:

```toml
[[collections]]
name = "authors"
[[collections.fields]]
name = "name"
type = "text"
required = true

[[collections]]
name = "posts"
[[collections.fields]]
name = "title"
type = "text"
required = true
[[collections.fields]]
name = "author"
type = "relation"
collection = "authors"   # the target collection
```

This maps to `FieldKind::Relation("authors")` in the engine. In `/api/_meta`,
the field reports `"type": "relation"` and adds `"collection": "authors"`:

```json
{ "name": "author", "type": "relation", "required": false,
  "embed": false, "collection": "authors" }
```

### Validation

- A relation value must be an **integer** id (the same type rules as `int`);
  anything else is a validation error. A relation field can be `required` and is
  cleared with `null` like any other optional field.
- **Referential existence is opt-in.** The plain `create` / `update` path only
  type-checks the id — it does *not* verify the referenced record exists, so a
  dangling id is accepted. To enforce existence, the caller uses the *checked*
  path (`create_checked` / `update_checked`), passing the set of known
  collections. If the relation's target collection is in that set, the
  referenced id must exist (else a validation error); if the target is **not**
  in the set, the id is accepted without an existence check. This lets the CLI
  enforce integrity for collections it knows while staying lenient about
  cross-store or not-yet-loaded targets.

### Expansion — `?expand=`

By default a record carries only the relation **id**. Pass `?expand=<field>` to
inline the referenced record alongside it. For each named relation field, the
engine looks up the referenced record in its target collection and adds a
sibling key **`<field>_expanded`** holding the full target record (the original
id key is left untouched):

```text
GET /api/collections/posts/records/1?expand=author
```

```json
{
  "id": 1,
  "created": 1750000000,
  "title": "Hello",
  "author": 3,
  "author_expanded": { "id": 3, "created": 1749990000, "name": "Ada" }
}
```

Rules:

- A reference that is **absent**, points at a **missing** record, or whose
  target collection isn't known expands to `"<field>_expanded": null`.
- Names in `expand` that are **not** relation fields are silently ignored (no
  `_expanded` key is added for them).
- Expansion is **opt-in** — the plain `GET`/list responses are unchanged unless
  `?expand=` is supplied. Multiple fields can be expanded
  (`?expand=author&expand=editor`).

## The `embed` flag

Setting `embed` on a `text` field marks it for semantic indexing. The
collections engine **stores and exposes** the flag (it appears in `/api/_meta`)
but does not embed anything itself. The CLI's vector crate reads the flag and
builds the semantic index on top of the records this engine stores. So `embed`
is a contract between your schema and the semantic layer — the substring
`?search=` above works regardless.