Fix what a retroactive review found in alerting, the scripts and the units

[baz8080]

Slice 4 of the retroactive review, following #48, #51, #52 and #53. It covers a whole-file review of esb_outages/alert.py, esb_outages/__main__.py, the three shell scripts and the four systemd units.

The history is 18 commits, one per fix. The follow-ups from several review rounds of this PR are folded into the fix they belong to. Every commit passes lint and the full test suite on its own.

The common thread: several failures on the Pi ended silently, or with only the dead-man's monitor noticing hours later with no cause.

Collector

1. Alerts are delivered best effort and never print their URLs.
   • A webhook URL missing its scheme used to crash the run it was reporting on.
   • A failed delivery's warning quoted the URL, or its path, which is the secret. It now names only the kind of error.
2. A run that fails partway now alerts.
   • The pre-run probe passes on a full disk.
   • An OSError, or a SQLite error whose code means disk or permissions, is the storage alert (exit 6).
   • Anything else is a new crash alert (exit 1). It names esb rebuild, and says that a rebuild failing the same way, or the next run crashing again, means the code needs a fix.
   • Neither pings.
   • Recorded in notes/alerting.md, and the README's exit table gains exit 1.
3. A negative or non-numeric poll delay is refused, with one parser shared by the flag and the environment.
4. An overlapping run that removed the .write-test probe no longer raises a false storage alarm.
5. Only a held lock reads as held. Any other flock error used to be a silent skip. A poll also waits up to two minutes for the lock, counted inside its budget, and the messages name every possible holder.
6. Tests for alert paths nothing held: which runs ping, that the ping is a GET, ntfy versus JSON bodies, and every outcome of test-alert.
7. Stale wording is fixed, including "hourly" and the hardcoded chown path.
8. Error text is capped: a response body at 200 characters, an alert at 1,900 (Discord rejects anything over 2,000).
9. The run budget is 22 minutes (was 24) and the backstop 26 (was 25).
   • After the budget come the last fetch, the webhook and the heartbeat, each able to time out on two addresses and then the read.
   • The timer's jitter plus the backstop still ends before the next trigger.
   • Recorded in notes/storms.md, and TestTheBackstop holds it.
10. esb rebuild and esb compact work beside a malformed esb.db. Neither opens it first, and Store.open() keeps its connection only once the file has opened.

Scripts and units

11. Every backup failure is announced, through an EXIT trap.
12. The backup's git errors go into variables, not fixed file names in /tmp, and the unit gets PrivateTmp.
13. The backup takes the poll lock for add, commit and merge.
    • It fetches first, outside the lock.
    • Git under the lock closes the lock's file descriptor, so a detached auto-gc can't keep holding it.
    • A push rejected after a long wait gets one more fetch, commit and merge.
    • It waits up to 30 minutes, which is longer than a poll's backstop plus its stop timeout.
14. A stalled backup times out and says so. The unit covers two lock waits and four network steps, ssh has connect and keepalive timeouts, and a TERM trap reports the timeout.
15. The esb wrapper passes secrets to sudo in the environment, not on the command line, which sudo logs.
16. One interpreter, /usr/bin/python3, is used by the installer's gate, the wrapper and the unit.
17. An empty host key scan stops the install, and the unit pins the key (StrictHostKeyChecking=yes).
18. The backup setup docs are fixed: git runs as the service user, the deploy key path and mode are documented, and the installer names the backup timer.

Tried and dropped

Building a rebuild into a side file and renaming it in. It was about 60 lines, and four review rounds kept finding holes in it, all to guard esb.db, which the repo treats as disposable. A rebuild that fails now leaves a partial esb.db, and the next successful one replaces it; the raw logs are untouched either way. Recorded in notes/alerting.md.

Not taken

• The backup unit's timeout bounds dead connections, not a slow live push. It is a backstop, and the TERM trap announces a push it stops.
• A SIGTERM during the two-minute lock wait ends the process by default. Nothing has been written by then, and a stop is the operator's intent.
• ntfy detection by substring: it fits the one webhook in use.
• Reporting only the partial banner when a run is also drifted: the next run reports the drift.
• An env-file URL containing &: no configured URL has one.
• argparse's usage-error exit code, 2, is the same number as EXIT_AUTH: only interactive runs can hit it.

Deploying

These changes reach the Pi only when sudo sh scripts/install-native.sh is re-run there. The backup unit now pins GitHub's host key, and the installer re-seeds an empty known_hosts file and fails loudly if the scan fails.

Checks

• ruff check: clean, on every commit.
• unittest discover: passes on every commit. The final commit, with ESB_DATA_DIR set, runs 400 tests OK.
• tests/test_scripts.py runs the backup against throwaway git repos, using hooks to stand in for auto-gc and a racing host, and runs the wrapper against stand-in sudo and id.
• dash -n passes on all three scripts.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UqWuH6frirpbQDKnF5rEdT