AkurAI-Notes / .plan/epics/E04-notes-core-and-storage.md

Latest change f1d9b8cc6ff48113d891a2d95fd79b2c49f1f4ff - Initial plan and .plan directory by Claude Code

# E04 — notes-core + storage + collections schema

## Goal

The data model: `notes`, `notebooks`, `tags`. Storage is the Framework's `BTree`
via `akurai-storage`. Schema is declarative in
`site/backend/collections.toml` (Framework principle #5). Auto-REST API mounts
`/api/notes`, `/api/notebooks`, `/api/tags` via `akurai-collections`.

## Reference sources

- `~/Projects/AkurAI-Framework/AGENTS_DATA.md` — the B+tree key layout, the `coll:`
  / `\0`-prefix reserved keyspaces, and the embedding / `?search` lifecycle.
- `~/Projects/akurai-crm/crates/crm-core/src/{entities,metadata,types,error.rs}` —
  full entity-metadata pattern to copy 1:1.
- `~/Projects/AkurAI-Framework/site/backend/collections.toml` — Framework's
  canonical demo collection (one collection, fields with `embed = true`).
- **Oblivion references:** read the data-model files in
  `.references/oblivion/` after cloning (E01 made `.references/` a dir).

## Pre-flight

Read `~/Projects/AkurAI-Framework/crates/collections/src/lib.rs` and
`AGENTS_DATA.md` to confirm the keyspace prefix the auto-API uses for each
collection. The CRM keyspace is `people:<id>`, `companies:<id>`, etc. — Notes
will use `note:<id>`, `notebook:<id>`, `tag:<id>`. Confirm by reading
`~/Projects/AkurAI-Framework/crates/collections/src/<the file that constructs
the prefix>` (likely `store.rs` or `lib.rs`).

Also, clone the references for UX inspiration (no build imports):

```bash
cd ~/Projects/AkurAI-Notes/.references
git clone --depth 1 https://github.com/elie222/oblivion.git oblivion 2>&1 | tail -5

# Top 4 open-source note apps (from memory / Ólafur's preferences):
git clone --depth 1 https://github.com/logseq/logseq.git        logseq
git clone --depth 1 https://github.com/silverbulletmd/silverbullet.git silverbullet
git clone --depth 1 https://github.com/usememos/memos.git       memos
git clone --depth 1 https://github.com/obsidianmd/obsidian-sample.git obsidian-sample 2>/dev/null || echo "obsidian missing — skip"

# These are refs only — they will never be imported into our build, only read.
# Add a `.gitignore` rule so they're not committed to our repo:
printf '\n/.references/\n' >> ~/Projects/AkurAI-Notes/.gitignore
```

(If `oblivion` is private/missing, fall back to `elie222/oblivion-docs` or note
the clone failure in the PR and proceed — it's reference material, not a build
dependency.)

## Data model

### `notes`

```toml
[[collection]]
name = "notes"

  [[collection.field]]
  name = "title"
  type = "text"
  required = true
  embed = true              # Framework auto-API: this field is embedded on insert

  [[collection.field]]
  name = "body"
  type = "text"
  embed = true              # body also embedded (title + body concat = search corpus)

  [[collection.field]]
  name = "format"
  type = "text"             # "markdown" | "plain" — v0.0.1 supports markdown only

  [[collection.field]]
  name = "notebookId"
  type = "int"
  index = true               # filter by notebook quickly

  [[collection.field]]
  name = "ownerSub"
  type = "text"
  index = true               # every query is scoped to the current user

  [[collection.field]]
  name = "tagIds"
  type = "text"              # comma-separated ints for v0.0.1 simplicity

  [[collection.field]]
  name = "createdAt"
  type = "int"
  index = true

  [[collection.field]]
  name = "updatedAt"
  type = "int"
  index = true

  [[collection.field]]
  name = "archived"
  type = "bool"
```

### `notebooks`

```toml
[[collection]]
name = "notebooks"
  [[collection.field]]   name = "name",       type = "text", required = true, embed = true
  [[collection.field]]   name = "description", type = "text"
  [[collection.field]]   name = "ownerSub",    type = "text", index = true
  [[collection.field]]   name = "color",       type = "text"       # hex token for the chip
  [[collection.field]]   name = "createdAt",   type = "int", index = true
```

### `tags`

```toml
[[collection]]
name = "tags"
  [[collection.field]]   name = "name",       type = "text", required = true, embed = true
  [[collection.field]]   name = "ownerSub",    type = "text", index = true
  [[collection.field]]   name = "color",       type = "text"
  [[collection.field]]   name = "createdAt",   type = "int"
```

## Steps

### 1. Write `crates/notes-core/src/entities/`

One Rust file per entity, each defining a plain struct with `pub` fields. Copy
the CRM pattern exactly (`~/Projects/AkurAI-crm/crates/crm-core/src/entities/`).
The fields are informational — the storage layout comes from `collections.toml`,
but having a typed Rust struct lets handlers pass strongly-typed data around
instead of `Value` everywhere.

`notes-core/src/entities/note.rs`:
```rust
use akurai_json::Value;

pub struct Note {
    pub id: i64,
    pub title: String,
    pub body: String,
    pub format: String,
    pub notebook_id: i64,
    pub owner_sub: String,
    pub tag_ids: Vec<i64>,
    pub created_at: i64,
    pub updated_at: i64,
    pub archived: bool,
}

impl Note {
    pub fn from_json(v: &Value) -> Option<Self> { /* read fields via v.get(…) */ }
    pub fn to_json(&self) -> Value { /* build Value::Object */ }
    pub fn embed_text(&self) -> String {
        // The text feed to the embedding model: title + body concat.
        format!("{} {}", self.title, self.body)
    }
}
```

`notebook.rs` and `tag.rs` are smaller versions of the same.

`entities/mod.rs` re-exports them: `pub use note::*; pub use notebook::*; pub use tag::*;`

### 2. Write `crates/notes-core/src/metadata/`

One `ObjectMetadata` struct per entity. The metadata crate at
`~/Projects/AkurAI-crm/crates/crm-core/src/metadata/` defines
`ObjectMetadata::standard_objects()` which returns the three CRM entities.
Notes does the same, returning the three Notes objects. The metadata is used by
the `GET /api/_meta` route to self-describe the API.

### 3. Persist the TOML schema in `site/backend/collections.toml`

Concatenate the three collections above into the file
`~/Projects/AkurAI-Notes/site/backend/collections.toml`. Framework cli loads it
on `serve`. The auto-API mounts at `/api/notes`, `/api/notebooks`, `/api/tags`
with full CRUD + `?search`.

### 4. Write `notes-api/src/handlers/crud.rs`

For v0.0.1 we mostly lean on the Framework's auto-collections-API helper (see
`CODE-PATTERNS.md §6`). The two notes-specific concerns:

- **Owner scoping.** Every list/get/update/delete must verify `owner_sub ==
  session.user.sub`. The auto-API doesn't do this — wrap it.
- **Embedding on insert.** After an `INSERT` on `notes`, schedule an embedding
  POST to the embedding URL and persist the f32 byte vec under
  `note_emb:<id>`. The Framework's auto-API knows about `embed = true` already
  so this may be automatic if `AKURAI_NOTES_EMBED_URL` is set — verify in
  `~/Projects/AkurAI-Framework/crates/cli/src/collections_api.rs` and `crates/
  vector/client.rs`. **E05 owns this** — E04 only writes the handler scaffolding
  and leaves the embed call as a `// TODO: see E05` placeholder.

A `crud.rs` handler shape (port of CRM's `handlers/crud.rs`):

```rust
pub fn list_route(state: Arc<Mutex<NotesState>>, coll: &'static str) -> RouteHandler {
    Box::new(move |req: &Request| -> Response {
        let user = match require_session(&state, req) { Some(u) => u, None => return Response::unauthorized() };
        let s = state.lock();
        let prefix = format!("{}:", coll);
        let mut out = Vec::new();
        for (k, v) in s.db.range_prefix(prefix.as_bytes()) {
            if let Ok(rec) = akurai_json::parse(&String::from_utf8_lossy(&v)) {
                if rec.get("ownerSub").and_then(|v| v.as_str()) == Some(&user.sub) {
                    out.push(rec);
                }
            }
        }
        json_response(Value::Array(out))
    })
}
```

Use the CRM `handlers/crud.rs` as the port source — the per-collection dispatch
shape is identical; only `ownerSub` filtering is Notes-specific.

### 5. Write `notes-api/src/handlers/meta.rs` (port of CRM's `meta.rs`)

Lists all routes + their inputs/outputs as JSON. Read the CRM version (read it on
disk; it's <80 lines) and port with Notes routes + Fields.

### 6. Wire owner-scoped routes into `router_setup.rs`

Update `router_setup.rs` to mount:
```
GET   /api/notes  + /api/notes/:id
POST  /api/notes
PUT   /api/notes/:id
DELETE /api/notes/:id
… same for notebooks and tags …
GET  /api/timeline   (recent edits, owner-scoped)
GET  /api/search     (E05; stub it returning [] for now)
```

### 7. Write `crates/notes-cli/src/cmd_seed.rs` (port of CRM's `cmd_seed.rs`)

Copy `~/Projects/AkurAI-crm/crates/crm-cli/src/cmd_seed.rs` verbatim as a
template, write the Notes seed data: 3 notebooks, 12 notes spanning all 3
notebooks, 6 tags, all owned by `olibuijr@olibuijr.com` so the live UI has
demo data on first deploy.

The confirm helper from `crates/notes-cli/src/cmd_seed.rs` (the closure
`|entity, id, fields: Vec<(&str, Value)>| -> Result<()> {\n … db.insert(format!("{}:{}", entity, id).as_bytes(), json.as_bytes()) ...}`)
is exactly what Notes needs — copy verbatim, swap the seed values.

### 8. Write tests

`crates/notes-api/tests/api_test.rs` — port CRM's `api_test.rs` skeleton and
test:
- `list_notes_returns_only_owned_notes` — insert notes with two different
  `ownerSub`, list as one owner, assert the other owner's notes don't appear.
- `create_note_inserts_with_current_user_as_owner`
- `update_note_rejects_cross_user` (silver-bullet test for the v0.0.1
  multi-user concern)
- `delete_note_rejects_cross_user`

## Files touched

```
crates/notes-core/src/lib.rs
crates/notes-core/src/entities/{mod.rs,note.rs,notebook.rs,tag.rs,embedding.rs}
crates/notes-core/src/metadata/mod.rs           ← ObjectMetadata for the three entities
crates/notes-core/src/types.rs
crates/notes-core/src/error.rs
crates/notes-api/src/handlers/{mod.rs,crud.rs,meta.rs,search.rs,timeline.rs,static_files.rs}
site/backend/collections.toml
crates/notes-cli/src/cmd_seed.rs
crates/notes-api/tests/api_test.rs
.references/                                    ← gitignored clones of Oblivion + 4 apps
```

## Acceptance

```bash
cd ~/Projects/AkurAI-Notes
cargo test --workspace
cargo clippy --all-targets --all-features -- -D warnings
cargo fmt --all -- --check

# Auto-API mounts the 3 collections:
AKURAI_NOTES_DB=/tmp/notes-e2e.db akurai-notes seed
akurai-notes serve --db /tmp/notes-e2e.db --frontend-dir site/frontend --port 9811 &
curl -fsS http://127.0.0.1:9811/api/_meta | jq '."collections" | keys'   # → ["notes","notebooks","tags"]
kill %1

# Owner scoping holds:
cargo test -p notes-api --test api_test
```

## Time

≈ 10 minutes (porting heavy + writing CRUD handlers + seed).

## Depends on

E01 (workspace scaffold).