AkurAI-Build

Latest change d9b2b1e262d3265c1661d308de806d70a84d802a - test+docs: real disposable-data evidence for binary/schema rollback by Ólafur Búi Ólafsson

# PR queue recovery: binary rollback vs migration rollback

Scope: what actually happens today when a production deploy needs to be
walked back, and what is still only a plan. Owned by the
`feature/pr-ci-safety` slice (`.akurai.yml`, this doc, and
`tests/test_pr_pipeline.py`); it does not touch `src/`, `migrations/`, or
Cargo files.

Revision note: an earlier version of this doc labeled binary rollback
"TESTED" and claimed it was schema-independent. Independent review (comment
76 on parent t_2578e75e) found both claims false. A later revision
corrected the wording but left the schema-compatibility question
unverified; this revision adds real runtime evidence for it (section 3)
using disposable data, per the same review thread's follow-up request. See
"What is NOT tested" below for what still has no runtime evidence.

## 1. Deploy-agent binary rollback — IMPLEMENTED, NOT RUNTIME-TESTED

Path: `deploy/service-deploy.sh` (in-sandbox CI job) hands a checksummed
candidate to the root-owned host agent `deploy/host/akurai-deploy-agent.sh`,
which:

1. Verifies the staged sha256 against the request file.
2. Copies the current `akurai` binary to `akurai.previous` before installing
   the candidate (`cp "$bin" "$bin.previous"`).
3. Restarts the service and polls `GET /api/health` up to 30 times
   (2s interval, 60s budget).
4. On health-gate failure: restores `akurai.previous` over the live binary,
   restarts, and reports failure — but does NOT re-poll `/api/health` on the
   restored binary before reporting (`akurai-deploy-agent.sh:67-69`). A
   restore that itself fails to come up healthy would still report "rolled
   back" without confirming it.
5. On health-gate success: deletes `akurai.previous`, `processing`, and the
   candidate (`akurai-deploy-agent.sh:57`). The backup is temporary — it
   exists only for the duration of one health-gate window, not retained as
   a standing rollback target. A second bad deploy after a first succeeded
   has no `.previous` to restore from.

Status: this is a description of what the script does, verified by reading
`deploy/host/akurai-deploy-agent.sh` end to end. It is NOT runtime-tested:
no test in this repository (`tests/`, `src/`) exercises this script, forces
a candidate health failure, or observes the restore actually happen and the
restored service actually come back healthy. Do not call it "TESTED" until
that runtime evidence exists (forced failure -> observed restore -> observed
post-restore health-check pass, plus a test verifying step 4's blind spot is
closed).

## 2. EC2 rollback (`src/ec2/deploy.rs::rollback`) — SEPARATE, unrelated path

`src/ec2/deploy.rs::rollback` is a distinct, manually-invoked `akurai-ec2
rollback <name>` verb operated against `AkurAI-EC2`-managed services. It is
NOT invoked by `.akurai.yml`'s `deploy-production` job, which calls
`deploy/service-deploy.sh` only. The two rollback mechanisms share the same
shape (back up to `.prev`/`.previous` before overwrite, require the backup
to exist to roll back) but are independent code paths for independent
deployment targets. Treat them separately; do not assume test or runtime
evidence for one applies to the other.

## 3. Migration rollback — PLANNED, not yet implemented

`src/db.rs::migrate` applies `MIGRATIONS` (currently 001-010) forward inside
one transaction per boot, tracked by a ledger table and `PRAGMA
user_version`. There is no down-migration, no reverse-SQL table, and no CLI
verb that reverts a specific migration number. Every migration file
inspected (001-010) is additive (`ALTER TABLE ...ADD COLUMN`, `CREATE
TABLE`, `CREATE INDEX`) with no destructive change to existing
columns/rows.

Because binary rollback (section 1) puts an older binary in front of the
current (possibly newer) schema, "rolling back" a bad deploy that shipped
alongside a migration means running an old binary against a new, additive
schema.

RUNTIME EVIDENCE (2026-09-06, disposable data, repo-local `.tmp/`
scratch): built the `v1.4.6` binary (schema version 4 in its own
`validate_schema` check) and the current worktree's binary (schema version
10) from source, then against one throwaway SQLite database:

1. `v1.4.6 akurai migrate` on a fresh DB → applies migrations 1-4,
   succeeds.
2. current `akurai migrate` on that same DB → applies migrations 5-10,
   succeeds (schema now v10).
3. `v1.4.6 akurai migrate` again on the now-v10 DB (the actual rollback
   scenario: operator reinstalls the old binary, schema was never rolled
   back) → **fails closed**: exit 1,
   `{"error":{"code":"failed","message":"database schema is not AkurAI Build v4"}}`.
   The database file is left untouched (verified: the current binary reads
   it fine immediately afterward with 0 migrations pending).

Conclusion drawn from this evidence, not assumption: `validate_schema`
performs an *exact* version-equality check (`version == N` for the
binary's own expected N), not "version >= N" or an additive-tolerant
check. An old binary rolled back onto an upgraded schema does not run in a
degraded/best-effort mode against unknown columns — it refuses to start at
all. This is safer than silent incompatibility (no old binary writing
against a schema it doesn't understand) but means binary rollback alone
does NOT restore service once a migration has shipped; the schema also has
to be reverted for the old binary to run again, and no down-migration path
exists (this section, below). Binary rollback is schema-locked, not
schema-tolerant.

Reproduce this evidence:

    AKURAI_OLD_BINARY=/path/to/old/target/release/akurai \
    AKURAI_NEW_BINARY=/path/to/new/target/release/akurai \
        python3 tests/test_pr_pipeline.py -v BinaryRollbackSchemaCompatibility

(`tests/test_pr_pipeline.py`'s `BinaryRollbackSchemaCompatibility` class;
skipped by default since it needs two prebuilt release binaries and is not
part of the default `verify` run — see that class's docstring.)

If a future migration needs to be destructive (drops a column, rewrites
data) this exact-version-lock behavior still applies (old binary still
refuses to start), but there would be no way to get back to a working old
binary+schema pair at all without a tested down-path; this doc does not
claim such a path exists.

Coordination note: this slice does not add migration 011 or PR persistence
— that is owned by the primary application-integration builder (t_2578e75e
/ t_2667e5e8 parent). If migration 011 stays additive, section 3's
reasoning continues to hold without further doc changes; if it is not
additive, the primary builder should update this section rather than
assume it.

## 4. What is NOT tested

- Deploy-agent rollback under a forced candidate health failure (no test
  drives `akurai-deploy-agent.sh` through a failure and observes restore).
- Health verification of the restored binary after a rollback restore
  (`akurai-deploy-agent.sh` does not re-check health post-restore; nothing
  papers over that gap).
- Recovering from two consecutive bad production deploys (only one
  `.previous` generation exists, and only until the first deploy's health
  gate succeeds — after success it is deleted).
- Reverting a specific already-applied migration (no down-SQL exists) —
  this is now the actual blocker for full recovery after a migration ships
  with a bad deploy: binary rollback alone (section 1) restarts an old
  binary against an unreverted, newer schema, which section 3's runtime
  evidence shows fails closed rather than degrading, so restoring service
  after such a deploy requires either forward-fixing the new binary or a
  down-migration path that does not exist yet.
- `src/ec2/deploy.rs::rollback` runtime behavior (separate mechanism per
  section 2; not exercised by this repository's `deploy-production` job or
  by any test referenced in this doc).