AkurAI-Notes / docs/adr/002-bm25-fts-vs-paradedb.md

Latest change b5edafd099fbb715485f63f0c36354159da099e2 - OLI-114: ADRs 002-005 (FTS vs ParadeDB, zoekt, chunking/embedding versioning, memory schema) by Dev Lead

# ADR 002: Lexical ranking via Postgres FTS vs ParadeDB BM25

Status: Proposed (Dev Lead, 2026-09-30) · Epic OLI-112 · Task OLI-114

## Context
Hybrid retrieval needs a lexical leg. Identifiers dominate code queries (`parseHttpRequest`, `snake_case`), which stock `tsvector` stemming handles badly. Postgres 17 + pgvector is fixed by ADR 001; adding an extension on the critical store raises restore-drill and upgrade risk (hindsight lesson).

## Decision
1. Phase 1 uses stock Postgres FTS (`tsvector` + GIN, `ts_rank_cd`) plus `pg_trgm` for identifier substrings, over a **pre-tokenised** column: identifiers are split on case/underscore boundaries (the same `tokenize` as `akurai-core::rank`) and the original lowercase token is kept. Symbol name and path are appended (symbol x3) so a name match outranks a body mention.
2. ParadeDB `pg_search` (true BM25) is deferred. Revisit when the QA baseline (ADR 007) shows lexical-leg recall@10 is the bottleneck, or corpus > ~500k chunks.
3. Exact/regex search is not FTS's job; that is zoekt (ADR 003).
4. Until Postgres is available the in-process `akurai-core::Index` implements BM25 (k1=1.2, b=0.75) with the same tokenizer, so eval numbers are comparable when the store swaps.

## Consequences
No new extension in the store; ranking is weaker than BM25 on length normalisation; fused via RRF (k=60) so absolute score scale does not matter. Re-evaluate against eval set, not by opinion.