Menu
Public repository / c3ef99b092caa55fb6a3eed9d3e776734cc99e97
AkurAI-Mail / scripts/ops/README.md
Latest change e0753e49bfea7ef672aabf8df9a3a27b8bdd9dc7 - Recover EC2-only ops scripts, checks and live divergent copies by Ólafur Búi Ólafsson
# Mail EC2 anti-spam observability
Install (run on `akurai-ec2` as root/sudo):
sudo install -m 0755 antispam-metrics.sh /usr/local/sbin/
sudo install -m 0644 antispam-metrics.service antispam-metrics.timer \
/etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now antispam-metrics.timer
## What it does
`antispam-metrics.sh` does not re-run the anti-spam sweep
(`akurai-mail-antispam.service`, every 10 min). It reads that unit's journal
(each run already emits one JSON line: `scanned`, `moved`, per-mailbox
breakdown), and:
- Appends `{ts, scanned, moved, ratio, ok, mailboxes}` per run to a durable
JSONL series at
`/var/lib/akurai-mail-ops/antispam-metrics/daily.jsonl` (trend, not just
last value). `mailboxes` is the cumulative per-mailbox Junk (spam) count
from `antiSpam.mailboxes[]` in the sweep journal line, filtered to
non-zero entries.
- Alerts (mailed to `postmaster@olibuijr.com` via the local Postfix relay,
deduped per alert kind per calendar day via
`/var/lib/akurai-mail-ops/antispam-metrics/alerts.state`):
- **sweep-errors** — sweep unit exec/exit failures seen in the journal
window since the last check.
- **zero-scans** — trailing window (default 1h) has recorded runs but
total `scanned` across them is 0 (silent-failure detector: unit exits 0
but isn't actually reading mailboxes).
- **move-ratio-spike** — latest run's move ratio exceeds
`baseline x ANTISPAM_SPIKE_FACTOR` (default 3x), where baseline is the
mean ratio of the trailing `ANTISPAM_BASELINE_N` (default 20) runs, and
`moved >= ANTISPAM_MIN_MOVED_FOR_SPIKE` (default 5) to ignore noise on
tiny counts.
Run every 5 min by `antispam-metrics.timer` (`OnUnitActiveSec=5min`).
Known limitation: `alert_once` dedupes one alert per kind per calendar day —
a second distinct incident of the same kind on the same day will not
re-alert until the next day.
## Verification performed for this change (round 2 — addressing review round 1)
Per-mailbox Junk counts now persisted (BLOCKER 1 fix). Fresh record on the
mail VM after the fix:
$ ssh akurai-ec2 sudo tail -1 /var/lib/akurai-mail-ops/antispam-metrics/daily.jsonl
{"ts":1788802673,"scanned":2702,"moved":1,"ratio":0.000370096,"ok":true,"mailboxes":[{"email":"olibuijr@olibuijr.com","spam":892}]}
Real units installed at `/etc/systemd/system/` (BLOCKER 2 fix — no more
`/tmp` symlinks):
$ ssh akurai-ec2 readlink -f /etc/systemd/system/antispam-metrics.service /etc/systemd/system/antispam-metrics.timer
/etc/systemd/system/antispam-metrics.service
/etc/systemd/system/antispam-metrics.timer
$ ssh akurai-ec2 systemctl status antispam-metrics.timer --no-pager
● antispam-metrics.timer - Run antispam-metrics.service every 5 minutes
Loaded: loaded (/etc/systemd/system/antispam-metrics.timer; enabled; vendor preset: enabled)
Active: active (waiting)
All three alerts re-fired, scratch under this task's `.tmp/tmp` locally and
`~/asm-test-scratch` on the VM (not `/tmp`):
=== zero-scans induced ===
[DRY_RUN] Subject: [mail VM] anti-spam sweep scanned 0 messages in the last 60m
Records in window: 1, total scanned: 0.
=== move-ratio-spike induced ===
[DRY_RUN] Subject: [mail VM] anti-spam move-ratio spike: 0.4 vs baseline 0.010000000000000002
Latest run: moved=40 ratio=0.4
Trailing baseline (last 20 runs): 0.010000000000000002
Spike threshold (baseline x 3): 0.03
=== sweep-errors induced ===
[DRY_RUN] Subject: [mail VM] anti-spam sweep failed 1x since last check
Unit: akurai-mail-antispam.service
Failures observed: 1
antispam-metrics: appended 0 record(s), 1 failure(s) in journal window
sweep-errors=2026-09-07:sweep-errors
## Verification performed (round 1, retained for real-fire evidence)
Real recorded data (from the mail VM, `/var/lib/akurai-mail-ops/antispam-metrics/daily.jsonl`):
{"ts":1788794333,"scanned":2715,"moved":15,"ratio":0.00552486,"ok":true}
{"ts":1788794845,"scanned":2701,"moved":0,"ratio":0,"ok":true}
(window collected: two sweep cycles at 15:18 and 15:27 UTC on 2026-09-07,
confirming the series accumulates across real runs; the timer keeps
appending every 10 min going forward.)
sweep-errors fired for real (pre-existing broken deploy of
`/usr/local/sbin/akurai-mail-server` caused 10 EXEC failures before this
task's own `ship-tools` re-deploy fixed it):
$ journalctl -u antispam-metrics.service --no-pager -n 4
Sep 07 15:25:53 mail.olibuijr.com antispam-metrics.sh[1365833]: antispam-metrics: appended 0 record(s), 10 failure(s) in journal window
$ cat /var/lib/akurai-mail-ops/antispam-metrics/alerts.state
sweep-errors=2026-09-07:sweep-errors
$ sudo tail -2 /var/log/mail.log
Sep 7 15:25:53 mail dovecot: lmtp(postmaster@olibuijr.com)<...>: saved mail to INBOX
Sep 7 15:25:53 mail postfix/lmtp[...]: ...: to=<postmaster@olibuijr.com>, ... status=sent (250 2.0.0 ... Saved)
Alert destination: `postmaster@olibuijr.com`, confirmed reachable (real
sweep-errors alert delivered and saved to that mailbox above).
## Rollback (antispam-metrics)
sudo systemctl disable --now antispam-metrics.timer
sudo rm -f /etc/systemd/system/antispam-metrics.{service,timer} \
/usr/local/sbin/antispam-metrics.sh
sudo systemctl daemon-reload
# optional: sudo rm -rf /var/lib/akurai-mail-ops/antispam-metrics
No changes were made to the sweep itself, postfix, sudoers, or DNS — this is
purely additive read-only observability on top of the existing sweep unit.
## Installed state (antispam-metrics, this task)
On `akurai-ec2`, the service/timer are real files at
`/etc/systemd/system/antispam-metrics.{service,timer}` (copied from
`~/ops-units/` via `sudo cp`, replacing an earlier `/tmp`-backed
`systemctl link` symlink that would not have survived a reboot since
`/tmp` is wiped by `systemd-tmpfiles`). Confirmed via `readlink -f` showing
the real `/etc/systemd/system/...` paths (no `/tmp`), see verification
section above.
# Mail EC2 ops hygiene
Install (run on `akurai-ec2` as root/sudo):
sudo install -m 0755 disk-alert.sh cert-age-alert.sh /usr/local/sbin/
sudo install -m 0644 disk-alert.service disk-alert.timer \
cert-age-alert.service cert-age-alert.timer /etc/systemd/system/
sudo install -m 0644 logrotate-syslog-mail.conf /etc/logrotate.d/rsyslog
sudo install -m 0755 certbot-reload-mail.sh \
/etc/letsencrypt/renewal-hooks/deploy/reload-mail.sh
sudo systemctl daemon-reload
sudo systemctl enable --now disk-alert.timer cert-age-alert.timer
## What each piece does
- `disk-alert.sh` + `.timer` — checks root `df` usage every 15 min, mails
`postmaster@olibuijr.com` via the local Postfix relay (Brevo) the first
time each of 75/80/85% is crossed per day (state file dedupes repeats).
- `cert-age-alert.sh` + `.timer` — checks `mail.olibuijr.com` cert expiry
daily, independent of certbot's own renewal, and alerts at 21/14/7 days
remaining the same way.
- `certbot-reload-mail.sh` — certbot deploy hook. Certbot only runs deploy
hooks for lineages it actually renewed (`$RENEWED_LINEAGE`); reloads
postfix + dovecot so TLS changes take effect without a manual step.
- `logrotate-syslog-mail.conf` — replaces `/etc/logrotate.d/rsyslog`'s
`weekly` policy for `/var/log/syslog` with `daily` + `size 50M`, so a
noisy day (DNS-abuse traffic, mail floods) can't grow one log file past
the retained total before rotation kicks in.
## Verification performed for this change (disk/cert alerting)
# dry-run renewal + reload hook
sudo certbot renew --dry-run --cert-name mail.olibuijr.com
# → "Congratulations, all simulated renewals succeeded"
# (dry-run does not fire deploy hooks; hook logic verified by invoking
# it directly with RENEWED_LINEAGE set, see PR description)
# disk alert threshold simulation
DISK_ALERT_STATE=/tmp/x THRESHOLDS_TEST=1 bash -c '
used=76; echo "$used" # simulate df output path manually, see PR notes'
# cert alert
sudo /usr/local/sbin/cert-age-alert.sh # no mail sent, cert has >21d left
# autoconfig / autodiscover / SRV (already live, no changes needed)
curl -s 'https://autoconfig.olibuijr.com/mail/config-v1.1.xml?emailaddress=x@olibuijr.com'
curl -s -X POST https://autodiscover.olibuijr.com/autodiscover/autodiscover.xml -d '<Autodiscover/>'
dig +short SRV _imaps._tcp.olibuijr.com _submission._tcp.olibuijr.com