[abrichr]

Found while testing #1918 (on-prem offline install).

What this fixes

The on-prem audit log is a hash chain. audit-log.sh serializes writers with a lock, and the release updater (lib-release.sh) uses a similar lock so that two updates or rollbacks can't run at once. Both locks treated a zero exit status from mkdir LOCKDIR as ownership.

That isn't safe on Ubuntu 26.04, which is the image GitHub's runner used for #1918. Its /usr/bin/mkdir is uutils (the Rust coreutils). In the version it ships, when two processes race to create the same directory, the loser's EEXIST is swallowed and both exit 0. A plain mkdir on a directory that already exists still fails, so this only shows up under contention. Others have reported the same behavior (gaia-harness/gaia#3084 measured double success in 236 of 300 two-way races on ubuntu:26.04 with GNU mkdir at 0 of 300; cahenesy/throughline#148). As a result:

• Concurrent audit writers could all "hold" the lock, chain to the same parent and break the chain. That is the "concurrent audit writers preserve one hash chain" failure on on-prem: install, update and roll back the real engine offline in CI #1918's CI, which recurred after the first commit here ("audit chain broken at line 20").
• A second update could proceed while another update or rollback was live.

The change

• Ownership is now the exclusive creation of the pid file inside the lock directory, done by bash itself with noclobber (set -C, which opens with O_CREAT|O_EXCL). The mkdir -p of the lock directory is no longer what grants the lock. The on-disk layout (LOCK/pid) is unchanged, so an existing holder is still recognized.
• Stale locks (owner process gone) are broken only under a second exclusive lock that serializes breakers, and only if the lock still names the same dead owner when re-read. This was the first commit here; it closes a separate race where a breaker could delete a lock a live writer had just taken. The breaker lock is now also a noclobber file (audit.log.lock-breaker, .update.lock-breaker). The update lock gets the same compare-before-remove rule; previously it did an unserialized rmdir and mkdir.
• A symlinked update lock is refused.

Tests (test-update.sh)

• A mkdir shim that never reports EEXIST models the uutils behavior on any OS.
  • With a live owner holding the update lock, an update under the shim is refused and the owner's lock is untouched.
  • 12 concurrent audit writers under the shim leave an intact 12-record chain.
• A shimmed cat hands the lock from a dead owner to a live process right after the writer reads it; the live writer's lock must survive (first commit).
• When a chain check fails, the test prints the verifier output.

Testing

• macOS: bash deploy/on-prem/bin/test-update.sh, 49 passed, 0 failed (3 runs).
• Linux (Debian, bash 5.2): 51 passed, 0 failed, 3 runs with GNU coreutils and 3 runs with uutils 0.12.0 first in PATH. Some checks are platform-gated, so the totals differ by OS.
• Control: the new test file with the previous audit-log.sh and lib-release.sh fails all 3 new checks. The update under the shim goes through, which also fails 2 later rollback checks (44 passed, 5 failed).
• Stress on Linux pinned to 2 CPUs, 12 concurrent writers per round with the EEXIST shim: the previous code broke the chain in 5 of 5 rounds and this code in 0 of 40.
• A lock left by a dead owner is still reclaimed at once, for both locks, and both lock directories are removed on release.
• shellcheck is clean on the changed files apart from an existing SC1091 info.

Not tested

• I did not run on an actual Ubuntu 26.04 machine. uutils 0.12.0 (the newest release) was atomic in my race test (0 of 200), so the shim stands in for the older uutils on the runner. The real check is CI on ubuntu-26.04. With this commit, the offline job that runs test-update.sh passed on on-prem: install, update and roll back the real engine offline in CI #1918 (run 41940061341), and on the stacked on-prem: build the runner image from a signed release with no network, test it in CI #1923 test-update.sh passed 51 of 51 on two consecutive runs (41940100303, 41941053864). Before this commit that check had failed on 3 of 5 runs.
• No workflow on main runs test-update.sh. on-prem: install, update and roll back the real engine offline in CI #1918 adds the first CI job that does.
• If a breaker is killed while holding the breaker lock, stale locks are no longer broken automatically. Audit writers then wait up to 10s and fail loudly, and updates refuse, instead of corrupting the chain or running concurrently. Removing the leftover .lock-breaker file clears it.