AkurAI-Notes / docs/adr/004-chunking-and-embedding-versioning.md

Latest change b5edafd099fbb715485f63f0c36354159da099e2 - OLI-114: ADRs 002-005 (FTS vs ParadeDB, zoekt, chunking/embedding versioning, memory schema) by Dev Lead

# ADR 004: Chunking, content-hash incremental index, embedding versioning

Status: Proposed (Dev Lead, 2026-09-30) · Epic OLI-112 · Task OLI-114

## Decisions
1. **Chunking:** tree-sitter at symbol granularity (Rust, TS/TSX, TOML, YAML, Markdown sections). Oversized containers (impl/trait/mod/class/section > 60 lines) are split into members; anything else > 120 lines, unknown languages and inter-symbol gaps use a sliding window (60 lines, 10 overlap), so every line is covered. Parse failure falls back to windows.
2. **Identity:** `chunk.content_hash = sha256(path \0 symbol \0 text)`; `file.blob_sha = sha256(text)`. Chunks are stored once per content hash and referenced by (branch, path), so identical content on several branches shares rows and vectors.
3. **Incremental:** a push diffs `before..after` (`git diff --name-status`), re-chunks only changed files, skips unchanged blob sha, and embeds only chunk hashes not already stored. A zero `before` (new branch) indexes the full tree. Unreferenced chunks are garbage-collected.
4. **Tombstones:** deleted files leave a tombstone `(branch, path) -> blob_sha` and are excluded from search immediately; in Postgres the tombstone row is kept for audit and vector rows are removed.
5. **Exclusions before chunking:** lockfiles, key/env files, `.git`, `node_modules`, `target`, blobs > 512 KiB or non-UTF-8 (ADR 006 adds the gitleaks gate in front of embed).
6. **Embedding versioning:** every vector stores `(model_id, model_version, dim)`. A model change adds a new `embedding_version`; queries dual-read (old version until backfill is complete), a backfill job re-embeds by content hash, then the pointer flips and old vectors are dropped. Default model Qwen3-Embedding-0.6B (Q8) via llama.cpp, loopback.

## Consequences
Content-hash sharing keeps embed cost proportional to real change. Path is in the hash, so a pure rename re-embeds (accepted; cheap, and path is a ranking signal). Verified by `akurai-core` tests (symbol names, window overlap, skip/tombstone, branch sharing).