feat(audit): detect chain truncation and add buzz-admin audit verify

[SeanGearin]

Problem

buzz-audit implements a real per-community SHA-256 hash chain, but the
"tamper-evident" claim was a property of the schema, not of operations:

1. verify_chain had no operational caller. It is invoked only from
   tests. The conformance notes
   (crates/buzz-test-client/tests/conformance_multitenant.rs, audit gate)
   describe audit reads as "operator-internal (consumed by buzz-admin)" —
   but buzz-admin never actually consumed them (it declares the
   buzz-audit dependency and never uses it). Nothing deployed could verify
   a chain.

2. The verifier accepted truncated chains.

   • Prefix truncation: verify_chain(c, 1, N) seeded
     expected_prev = None, so the first row was never checked to be the
     genesis. DELETE ... WHERE seq <= k left a suffix that verified clean.
   • Unanchored segments: verify_chain(c, 5, 10) never compared row 5's
     prev_hash against row 4's stored hash — the left edge of any
     mid-chain range was unchecked.
   • Tail truncation: nothing reported what the walk covered versus what
     should exist. Deleting the newest rows is invisible to any walk over
     stored rows, because the head is defined by what is stored.
   • Interior deletions surfaced as a generic ChainViolation
     (misdiagnosed as mutation rather than deletion).

Fix

buzz-audit — boundary hardening, no schema or wire changes:

• verify_entries (new, pub, pure): the chain's seq-contiguity pass.
  A single uniform walk that enumerates entries in seq order and asserts,
  per entry, that (1) seq is contiguous and (2) prev_hash equals the
  previous entry's hash, then recomputes each stored hash. Prefix
  truncation (front edge), interior deletion (mid-run), and — via
  verify_full_chain's head check — suffix truncation (the sequence ending
  short) all fail as breaks in that one pass, the same way tampered entries
  do, with no per-case detection path. Being pure, it is unit-tested without
  Postgres and usable by callers that fetched entries through another read
  path. This framing — seq-contiguity as one explicit named pass, so both
  truncation ends surface as seq gaps — is the cleaner one @SaravananJaichandar
  named on buzz-audit: hash chain is tamper-evident in schema but not operationally verifiable; verify_chain accepts truncated chains #2620; adopting it here.
• verify_chain: same signature, now anchored. from_seq <= 1 asserts
  genesis linkage; from_seq > 1 requires the from_seq - 1 entry to
  exist (append-only chains have no legitimate holes), match its own stored
  hash, and be linked to by the first entry in range.
• verify_full_chain (new): the operational entry point. Pages through the
  whole chain and returns ChainVerification { entries_verified, head_seq, head_hash }. Passing a previously recorded head as expected_head_seq
  applies the same "seq must reach where it should" rule to the tail,
  closing the tail-truncation hole — the one deletion class a stored-rows
  walk cannot see by construction.
• New AuditError variants — MissingGenesis, MissingAnchor,
  SequenceGap, TruncatedTail — name which edge of the contiguity pass
  broke. They are diagnostic labels over the one check (not separate
  detection paths), carry per-community seq values only, and are added to
  the existing error-text sanitization test (no community_id/constraint
  leakage, same fence as before).

buzz-admin — the operator surface that makes the claim operational:

• buzz-admin audit verify [--expect-head N] resolves the deployment's
  community exactly like every other buzz-admin command (RELAY_URL host
  → durable host map, fail-closed) and verifies the full chain. JSON report
  on stdout:

  {
    "ok": true,
    "community": "3f0f…",
    "entries_verified": 12873,
    "head_seq": 12873,
    "head_hash": "9a41…"
  }

  Exit codes: 0 chain verified, 1 integrity failure (JSON verdict with
  the reason), 5 operational error. Record head_seq/head_hash
  externally after each run and pass the seq back via --expect-head on
  the next run to make tail truncation detectable.

• Deliberately not a new HTTP endpoint: the conformance notes treat the
  absence of an audit wire surface as an isolation property ("the oracle's
  surface does not exist"), and CONTRIBUTING discourages new
  endpoint-specific JSON APIs. The operator command reads Postgres directly
  on the same plane as the rest of buzz-admin, constructing a small
  dedicated audit pool the same way buzz-relay's main.rs does.

CI and docs wiring: just test-unit and the scripts/run-tests.sh
fallback gain a buzz-audit --lib step so the new pure tests execute in
CI rather than merely compiling (details under Test evidence), and
ARCHITECTURE.md's buzz-admin subcommand table documents
audit verify.

Test evidence

All commands below were run this session against HEAD
(3c14b6a), hermit toolchain, scoped to the two crates.

$ cargo fmt -p buzz-audit -p buzz-admin -- --check
# clean (exit 0)

$ cargo clippy -p buzz-audit --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s)   # 0 warnings, exit 0
$ cargo clippy -p buzz-admin --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s)   # 0 warnings, exit 0

cargo-nextest is not installed in this dev sandbox (error: no such command: nextest), so the unit gate was run with the documented
cargo test fallback. CI runs it under nextest via the just test-unit
recipe added here.

$ cargo test -p buzz-audit -p buzz-admin
# buzz-admin (binary, no unit tests):
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
# buzz-audit (src/lib.rs):
test service::tests::verify_entries_accepts_valid_genesis_run ... ok
test service::tests::verify_entries_detects_prefix_truncation ... ok
test service::tests::verify_entries_rejects_regrafted_genesis ... ok
test service::tests::verify_entries_detects_interior_gap ... ok
test service::tests::verify_entries_detects_tampering ... ok
test service::tests::verify_entries_anchored_run_checks_left_edge ... ok
test service::tests::verify_entries_suffix_truncation_needs_the_recorded_head ... ok
test error::tests::audit_error_text_carries_no_community_id_or_constraint ... ok
test result: ok. 15 passed; 0 failed; 10 ignored; 0 measured; 0 filtered out

The seven verify_entries_* tests are the first coverage of the verifier
that runs without Postgres — each builds a real hash chain in memory and
exercises one edge of the seq-contiguity pass: a valid genesis run, prefix
truncation (front edge), a re-rooted genesis, an interior gap, content
tamper (with and without hash recomputation), anchored mid-chain segments,
and suffix truncation (back edge — a deleted tail leaves a shorter but
still-contiguous run, so it passes verify_entries and needs the recorded
head, which is what motivates verify_full_chain's --expect-head check).

Ten #[ignore = "requires Postgres"] tests are counted as ignored above;
four of them are new and drive the same detections through the service
against real DELETEs: verify_detects_deleted_genesis_prefix,
verify_anchors_mid_chain_segments, verify_detects_interior_deletion,
and full_chain_report_and_tail_truncation (which also does the
report/--expect-head round trip: record the head, delete the tail,
re-verify against the recorded seq → TruncatedTail). They compile under
clippy --all-targets but were not executed here (no Docker/Postgres in the
sandbox). Run them like the existing ones:
DATABASE_URL=... cargo test -p buzz-audit -- --ignored.

buzz-audit was absent from the just test-unit crate list, so the new
pure tests would have compiled in CI (workspace clippy) but never executed.
This PR adds cargo nextest run -p buzz-audit --lib to the test-unit
recipe (and the matching cargo test -p buzz-audit --lib step to the
scripts/run-tests.sh fallback, keeping the two lists in sync) — same shape
as the existing buzz-db --lib gate: the Postgres-backed tests are
#[ignore]d, so --lib stays infra-free.

CLI smoke (debug binary built this session, exit codes captured):

$ buzz-admin audit verify --help
Verify this relay's audit hash chain, genesis to head.
...
      --expect-head <EXPECT_HEAD>
          Fail with a tail-truncation error unless the verified head has
          reached at least this seq (from a previously recorded report)
# renders; exit 0

$ RELAY_URL=https://relay.example.com \
  DATABASE_URL=postgres://nobody:nope@127.0.0.1:59999/none \
    buzz-admin audit verify
error: database error: pool timed out while waiting for an open connection
$ echo $?
5

Compatibility

• verify_chain signature unchanged; all in-repo callers verify
  genesis-rooted chains and keep passing.
• AuditError gains variants; there are no exhaustive matches outside
  tests.
• No migrations, no new /api/* routes, no wire-format changes, no new
  workspace dependencies (buzz-admin already declared buzz-audit).
• Compatible with relay: batch audit-log appends per community (T2a write-amp) #2126 (batched audit appends): batching changes how
  entries are written, not the chain invariants this verifies.
• Independent of the created_at hashing-precision determinism bug tracked
  in buzz-audit: every hash chain fails verification — created_at is hashed at nanosecond precision but stored at microsecond precision #2637 (fixes in flight: fix(audit): hash created_at at the precision Postgres stores #2638, fix(buzz-audit): truncate created_at to microseconds before hashing #2678). This PR does not touch
  compute_hash; it adds the verifier and its operator surface. Note the
  interaction: audit verify recomputes each stored hash, so its usefulness
  against real Postgres rows depends on compute_hash being deterministic
  against the stored created_at — the property buzz-audit: every hash chain fails verification — created_at is hashed at nanosecond precision but stored at microsecond precision #2637 reports as currently
  broken and fix(audit): hash created_at at the precision Postgres stores #2638/fix(buzz-audit): truncate created_at to microseconds before hashing #2678 fix. The pure verify_entries tests build in-memory
  chains and are unaffected; the #[ignore]d Postgres tests were not run
  here, so this PR makes no claim about the round-trip until that fix lands.

Closes #2620

[SaravananJaichandar]

Just saw #2715 — adopting verify_entries as the seq-contiguity pass is the right cut. One thing worth checking if it is already covered: concurrent-writer race producing adjacent entries that share the same prev_hash. Both are individually well-formed, both would pass a naive walk, but the chain link between them is broken. Silent until the verifier looks for it explicitly.

Surfaced 113 of these on one of our own chains this week when a streaming offline verifier caught up on chain history that predated our fix. Trigger was HTTP webhook writes hitting append in parallel with active sessions. Fix on our side: BEGIN IMMEDIATE on the append transaction. Test vector shape: two entries at seq N and N+1 both pointing at the same N-1 prev_hash.

[SeanGearin]

Good call to check — it's covered, though not by the sequence pass (you're right that a seq-only walk would wave it through). verify_entries also asserts prev-hash linkage: each entry's prev_hash has to equal the recomputed hash of the entry immediately before it in seq order, independent of the contiguity check.

So for the shape you described — seq N and N+1 both carrying N-1's prev_hash — the walk clears N (its prev_hash does equal hash(N-1)), then trips at N+1: its predecessor is now N, so its prev_hash has to be hash(N), but it's still hash(N-1). That surfaces as a broken hash link rather than a SequenceGap — exactly the "silent until you look for it explicitly" case, made loud.

The 113-on-a-live-chain number is great validation that it's worth catching — and nice call on BEGIN IMMEDIATE. Sweeping historical forks with an offline verifier that caught up on pre-fix history is precisely the use case this is built for. There's already a prev-hash-linkage assertion in the pure in-memory tests; happy to add a dedicated vector in exactly your shape (two entries at N/N+1 sharing N-1's prev_hash) if you'd like it pinned explicitly.

[SeanGearin]

#2638 landing fixes the created_at precision split that made every chain fail verification at its first entry. @shani-singh1 called that dependency in #2637 and was right — this needed that baseline before its own tests meant anything against a real database.

The interaction is a little sharper than "the Postgres tests would have failed": verify_entries recomputes each stored hash as it walks in seq order, so on a real chain it stopped at seq 1 and never advanced. The variants this PR adds — MissingGenesis, SequenceGap, TruncatedTail — exist to tell an operator which edge of the chain broke, and none of them were reachable while the preimage split was there. Verification failed, but the diagnosis failed first. With #2638 landed those paths are live.

@SaravananJaichandar — one correction to something I wrote on #2620: this CLI's exit codes are 0 verified / 1 integrity failure / 5 operational error, and the external anchor is --expect-head N rather than a signed manifest. I was describing your setup there rather than mine.