AkurAI-Build

Latest change 266a1aa21e346770b48e96f008ab6c2a5a603961 - Complete PR interfaces, isolated CI and crash-safe merge recovery by Ólafur Búi Ólafsson

# PR queue recovery: binary rollback vs migration rollback

Scope: what actually happens today when a production deploy needs to be
walked back, and what is still only a plan. Owned by the
`feature/pr-ci-safety` slice (`.akurai.yml`, this doc, and
`tests/test_pr_pipeline.py`); it does not touch `src/`, `migrations/`, or
Cargo files.

Revision note: an earlier version of this doc labeled binary rollback
"TESTED" and claimed it was schema-independent. Independent review (comment
76 on parent t_2578e75e) found both claims false. A later revision
corrected the wording but left the schema-compatibility question
unverified; this revision adds real runtime evidence for it (section 3)
using disposable data, per the same review thread's follow-up request. See
"What is NOT tested" below for what still has no runtime evidence.

## 1. Deploy-agent binary rollback — IMPLEMENTED, NOT RUNTIME-TESTED

Path: `deploy/service-deploy.sh` (in-sandbox CI job) hands a checksummed
candidate to the root-owned host agent `deploy/host/akurai-deploy-agent.sh`,
which:

1. Verifies the staged sha256 against the request file.
2. Copies the current `akurai` binary to `akurai.previous` before installing
   the candidate (`cp "$bin" "$bin.previous"`).
3. Restarts the service and polls `GET /api/health` up to 30 times
   (2s interval, 60s budget).
4. On health-gate failure: restores `akurai.previous` over the live binary,
   restarts, and reports failure — but does NOT re-poll `/api/health` on the
   restored binary before reporting (`akurai-deploy-agent.sh:67-69`). A
   restore that itself fails to come up healthy would still report "rolled
   back" without confirming it.
5. On health-gate success: deletes `akurai.previous`, `processing`, and the
   candidate (`akurai-deploy-agent.sh:57`). The backup is temporary — it
   exists only for the duration of one health-gate window, not retained as
   a standing rollback target. A second bad deploy after a first succeeded
   has no `.previous` to restore from.

Status: this is a description of what the script does, verified by reading
`deploy/host/akurai-deploy-agent.sh` end to end. It is NOT runtime-tested:
no test in this repository (`tests/`, `src/`) exercises this script, forces
a candidate health failure, or observes the restore actually happen and the
restored service actually come back healthy. Do not call it "TESTED" until
that runtime evidence exists (forced failure -> observed restore -> observed
post-restore health-check pass, plus a test verifying step 4's blind spot is
closed).

## 2. EC2 rollback (`src/ec2/deploy.rs::rollback`) — SEPARATE, unrelated path

`src/ec2/deploy.rs::rollback` is a distinct, manually-invoked `akurai-ec2
rollback <name>` verb operated against `AkurAI-EC2`-managed services. It is
NOT invoked by `.akurai.yml`'s `deploy-production` job, which calls
`deploy/service-deploy.sh` only. The two rollback mechanisms share the same
shape (back up to `.prev`/`.previous` before overwrite, require the backup
to exist to roll back) but are independent code paths for independent
deployment targets. Treat them separately; do not assume test or runtime
evidence for one applies to the other.

## 3. Migration rollback — PLANNED, not yet implemented

`src/db.rs::migrate` applies `MIGRATIONS` (currently 001-010) forward inside
one transaction per boot, tracked by a ledger table and `PRAGMA
user_version`. There is no down-migration, no reverse-SQL table, and no CLI
verb that reverts a specific migration number. Every migration file
inspected (001-010) is additive (`ALTER TABLE ...ADD COLUMN`, `CREATE
TABLE`, `CREATE INDEX`) with no destructive change to existing
columns/rows.

Because binary rollback (section 1) puts an older binary in front of the
current (possibly newer) schema, "rolling back" a bad deploy that shipped
alongside a migration means running an old binary against a new, additive
schema.

RUNTIME EVIDENCE (2026-09-06, disposable data, repo-local `.tmp/`
scratch): built the `v1.4.6` binary (schema version 4 in its own
`validate_schema` check) and the current worktree's binary (schema version
10) from source, then against one throwaway SQLite database:

1. `v1.4.6 akurai migrate` on a fresh DB → applies migrations 1-4,
   succeeds.
2. current `akurai migrate` on that same DB → applies migrations 5-10,
   succeeds (schema now v10).
3. `v1.4.6 akurai migrate` again on the now-v10 DB (the actual rollback
   scenario: operator reinstalls the old binary, schema was never rolled
   back) → **fails closed**: exit 1,
   `{"error":{"code":"failed","message":"database schema is not AkurAI Build v4"}}`.
   The database file is left untouched (verified: the current binary reads
   it fine immediately afterward with 0 migrations pending).

Conclusion drawn from this evidence, not assumption: `validate_schema`
performs an *exact* version-equality check (`version == N` for the
binary's own expected N), not "version >= N" or an additive-tolerant
check. An old binary rolled back onto an upgraded schema does not run in a
degraded/best-effort mode against unknown columns — it refuses to start at
all. This is safer than silent incompatibility (no old binary writing
against a schema it doesn't understand) but means binary rollback alone
does NOT restore service once a migration has shipped; the schema also has
to be reverted for the old binary to run again, and no down-migration path
exists (this section, below). Binary rollback is schema-locked, not
schema-tolerant.

Reproduce this evidence:

    AKURAI_OLD_BINARY=/path/to/old/target/release/akurai \
    AKURAI_NEW_BINARY=/path/to/new/target/release/akurai \
        python3 tests/test_pr_pipeline.py -v BinaryRollbackSchemaCompatibility

(`tests/test_pr_pipeline.py`'s `BinaryRollbackSchemaCompatibility` class;
skipped by default since it needs two prebuilt release binaries and is not
part of the default `verify` run — see that class's docstring.)

If a future migration needs to be destructive (drops a column, rewrites
data) this exact-version-lock behavior still applies (old binary still
refuses to start), but there would be no way to get back to a working old
binary+schema pair at all without a tested down-path; this doc does not
claim such a path exists.

Coordination note: this slice does not add migration 011 or PR persistence
— that is owned by the primary application-integration builder (t_2578e75e
/ t_2667e5e8 parent). If migration 011 stays additive, section 3's
reasoning continues to hold without further doc changes; if it is not
additive, the primary builder should update this section rather than
assume it.

## 4. What is NOT tested

- Deploy-agent rollback under a forced candidate health failure (no test
  drives `akurai-deploy-agent.sh` through a failure and observes restore).
- Health verification of the restored binary after a rollback restore
  (`akurai-deploy-agent.sh` does not re-check health post-restore; nothing
  papers over that gap).
- Recovering from two consecutive bad production deploys (only one
  `.previous` generation exists, and only until the first deploy's health
  gate succeeds — after success it is deleted).
- Reverting a specific already-applied migration (no down-SQL exists) —
  this is now the actual blocker for full recovery after a migration ships
  with a bad deploy: binary rollback alone (section 1) restarts an old
  binary against an unreverted, newer schema, which section 3's runtime
  evidence shows fails closed rather than degrading, so restoring service
  after such a deploy requires either forward-fixing the new binary or a
  down-migration path that does not exist yet.
- `src/ec2/deploy.rs::rollback` runtime behavior (separate mechanism per
  section 2; not exercised by this repository's `deploy-production` job or
  by any test referenced in this doc).

## Integrated recovery (1.5.0)

The runner now restricts native execution to trusted default refs. Build PR
and merge refs use the pinned verify-pr Docker job without shared caches or
secrets. Queue CI identity is persisted before execution. After a crash past
the final push, recovery verifies successful CI, exact merge parents and target
ancestry before marking the PR merged. Retried entries remove only their own
exact disposable ref using compare-and-swap deletion.