AkurAI-Build

Latest change 941c1e29871d5e7548906ba3b75ddd925cc9326e - ci: fail closed on PR branches, fix false rollback claims, drop PyYAML by Ólafur Búi Ólafsson

# PR queue recovery: binary rollback vs migration rollback

Scope: what actually happens today when a production deploy needs to be
walked back, and what is still only a plan. Owned by the
`feature/pr-ci-safety` slice (`.akurai.yml`, this doc, and
`tests/test_pr_pipeline.py`); it does not touch `src/`, `migrations/`, or
Cargo files.

Revision note: an earlier version of this doc labeled binary rollback
"TESTED" and claimed it was schema-independent. Independent review (comment
76 on parent t_2578e75e) found both claims false. This revision corrects
them; see "What is NOT tested" below for the exact gaps.

## 1. Deploy-agent binary rollback — IMPLEMENTED, NOT RUNTIME-TESTED

Path: `deploy/service-deploy.sh` (in-sandbox CI job) hands a checksummed
candidate to the root-owned host agent `deploy/host/akurai-deploy-agent.sh`,
which:

1. Verifies the staged sha256 against the request file.
2. Copies the current `akurai` binary to `akurai.previous` before installing
   the candidate (`cp "$bin" "$bin.previous"`).
3. Restarts the service and polls `GET /api/health` up to 30 times
   (2s interval, 60s budget).
4. On health-gate failure: restores `akurai.previous` over the live binary,
   restarts, and reports failure — but does NOT re-poll `/api/health` on the
   restored binary before reporting (`akurai-deploy-agent.sh:67-69`). A
   restore that itself fails to come up healthy would still report "rolled
   back" without confirming it.
5. On health-gate success: deletes `akurai.previous`, `processing`, and the
   candidate (`akurai-deploy-agent.sh:57`). The backup is temporary — it
   exists only for the duration of one health-gate window, not retained as
   a standing rollback target. A second bad deploy after a first succeeded
   has no `.previous` to restore from.

Status: this is a description of what the script does, verified by reading
`deploy/host/akurai-deploy-agent.sh` end to end. It is NOT runtime-tested:
no test in this repository (`tests/`, `src/`) exercises this script, forces
a candidate health failure, or observes the restore actually happen and the
restored service actually come back healthy. Do not call it "TESTED" until
that runtime evidence exists (forced failure -> observed restore -> observed
post-restore health-check pass, plus a test verifying step 4's blind spot is
closed).

## 2. EC2 rollback (`src/ec2/deploy.rs::rollback`) — SEPARATE, unrelated path

`src/ec2/deploy.rs::rollback` is a distinct, manually-invoked `akurai-ec2
rollback <name>` verb operated against `AkurAI-EC2`-managed services. It is
NOT invoked by `.akurai.yml`'s `deploy-production` job, which calls
`deploy/service-deploy.sh` only. The two rollback mechanisms share the same
shape (back up to `.prev`/`.previous` before overwrite, require the backup
to exist to roll back) but are independent code paths for independent
deployment targets. Treat them separately; do not assume test or runtime
evidence for one applies to the other.

## 3. Migration rollback — PLANNED, not yet implemented

`src/db.rs::migrate` applies `MIGRATIONS` (currently 001-010) forward inside
one transaction per boot, tracked by a ledger table and `PRAGMA
user_version`. There is no down-migration, no reverse-SQL table, and no CLI
verb that reverts a specific migration number. Every migration file
inspected (001-010) is additive (`ALTER TABLE ...ADD COLUMN`, `CREATE
TABLE`, `CREATE INDEX`) with no destructive change to existing
columns/rows.

Because binary rollback (section 1) puts an older binary in front of the
current (possibly newer) schema, "rolling back" a bad deploy that shipped
alongside a migration means running an old binary against a new, additive
schema. Whether an older binary genuinely tolerates a newer additive schema
at runtime is NOT verified by any automated test today — it is an assumption
resting on the migration authors' additive discipline, not a proven runtime
property. Binary rollback is therefore NOT schema-independent in general;
it is conditioned on that untested compatibility assumption holding, and
production rollback after a migration should be treated as unverified until
an integration test exercises old-binary-against-new-schema.

If a future migration needs to be destructive (drops a column, rewrites
data) the additive assumption breaks entirely and a tested down-path would
be required before that migration ships; this doc does not claim such a
path exists.

Coordination note: this slice does not add migration 011 or PR persistence
— that is owned by the primary application-integration builder (t_2578e75e
/ t_2667e5e8 parent). If migration 011 stays additive, section 3's
reasoning continues to hold without further doc changes; if it is not
additive, the primary builder should update this section rather than
assume it.

## 4. What is NOT tested

- Deploy-agent rollback under a forced candidate health failure (no test
  drives `akurai-deploy-agent.sh` through a failure and observes restore).
- Health verification of the restored binary after a rollback restore
  (`akurai-deploy-agent.sh` does not re-check health post-restore; nothing
  papers over that gap).
- Recovering from two consecutive bad production deploys (only one
  `.previous` generation exists, and only until the first deploy's health
  gate succeeds — after success it is deleted).
- Reverting a specific already-applied migration (no down-SQL exists).
- Old-binary-against-newer-additive-schema runtime compatibility (assumed
  from migration authors' discipline, not verified by an integration test).
- `src/ec2/deploy.rs::rollback` runtime behavior (separate mechanism per
  section 2; not exercised by this repository's `deploy-production` job or
  by any test referenced in this doc).