AkurAI-Notes / docs/adr/007-retrieval-eval-and-ci-gate.md

Latest change 192f8cfdcab73e201cf87f7af385107b5b8719f2 - OLI-116: ADR 007 retrieval eval and CI gate by QA Engineer

# ADR 007: Retrieval eval set and CI regression gate

Status: accepted (Phase 1) | Owner: QA Engineer | Issue: OLI-116 (epic OLI-112)

## Context
akurai-index (code) and akurai-memory (facts) are only useful if retrieval is good, and "good" drifts silently
when chunking, embedding model, fusion or rerank change. Without a gated number, the pipeline would manufacture
confidence (green-by-default).

## Decision
1. Repo `akurai-index-eval` (stdlib Python, no deps). Backend-agnostic: any command speaking line-delimited JSON
   `{kind,query,k} -> {ids}`. indexd/memoryd ship a thin adapter; the harness never imports their code.
2. Three sets, all in git: `code` (120 queries; 80 docstring-derived semantic + 40 identifier lookups, ground truth
   = file at a pinned commit of 5 repos, read from git objects), `memory` (55 queries over 50 facts, incl. 10 deliberate
   paraphrases so repeated facts must be recalled together: sandboxed HOME, VPN deferred, forge loopback-only...),
   `dedupe` (24 queries over 112 issue titles; ground truth OLI-21/22/23, 52/54, 56/66, 109/110, 58-61, 81-84, 92-95, 46-48).
3. Metrics: recall@5, recall@10, MRR, nDCG@10 (binary relevance), latency p50/p95/max per query (client-side, end to end).
4. Gate (`.akurai.yml` job `retrieval-gate`): exit 1 if recall@10 drops > 2.0 pts vs `baselines/current.json`, if p95 exceeds
   `eval/budgets.json` (300 ms, from design section 2), or if the query count differs from baseline (prevents "fixing" a
   regression by deleting queries). Re-baselining is an explicit commit.
5. Floor: a pure-Python BM25 reference backend is baselined first. Real backends must beat it; the gate then moves to them.

## Baseline (phase1-bm25, 2026-09-30)
| set | n | recall@5 | recall@10 | MRR | nDCG@10 | p95 ms |
|---|---|---|---|---|---|---|
| code | 120 | 0.9083 | 0.9417 | 0.8070 | 0.8400 | 0.78 |
| memory | 55 | 0.9394 | 0.9576 | 0.8894 | 0.8846 | 0.08 |
| dedupe | 24 | 0.9167 | 0.9583 | 0.9213 | 0.9292 | 0.19 |

## Consequences / known limits (read before trusting a number)
- Code queries are derived from doc comments, not real agent questions; lexical overlap with the target is high, so
  BM25 scores well and the set has little headroom for semantic gains. Real agent questions must be appended
  (`source: "agent"`); until then a dense/hybrid win is under-measured.
- Memory/dedupe sets are small and near-saturated for BM25 (paraphrases share vocabulary). Expect the gate to catch
  regressions, not to prove improvements. Harder paraphrases are a Phase 2 task.
- Latency budget is against the harness's client-side timing on the same host; the BM25 floor (<1 ms) says nothing
  about GPU rerank cost. Budgets need re-measurement against indexd + rerank.
- Gate measures recall@10 only for regression; MRR/nDCG are reported, not gated.