Repository navigation
v3.1.0 chore: consolidate release for main - #783
Merged
Merged
Conversation
* docs: design record for WindowRef and Ledger::read_window (#264) Add docs/prism-coordinator-refactor/window-ref.md, the design record that #265 (migration 007) and #273 (migration 008) implement against, and link it from the directory README. The record specifies: - WindowRef { anchor_ms, prior_balances_digest, shares: Option<ShareRange> } - the six window_* columns and their three-state CHECK - Ledger::read_window: one REPEATABLE READ READ ONLY transaction, mandatory ascending keyset paging, a streaming digest, typed errors and never a partial window - per-caller revision fences that keep today's observe_candidate rule for claims - the non-window bundle inputs a rebuild stores instead of reading config - the upgrade procedure, retention floor and expected cost at 400k shares - the borrowed qbit-prism parts API merged in #296 It went through two adversarial reviews. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: hex-serialize WindowRef digests and clamp the resume timeout (#264) Two fixes to the WindowRef design record after review: - A derived Serialize writes a [u8; 32] as a JSON array of 32 integers, not the lowercase 64-hex string the payload and column contract requires. Both digest fields now carry #[serde(with = "hex32")], a small adapter in ledger/window.rs that writes lowercase hex and reads exactly 64 lowercase hex characters. The server's hex dependency has no serde feature. - The proposed resume timeout, initial_job_timeout_seconds minus 5 s, is zero or negative for any setting at or below 5 s, which validation allows. It is now outer - min(5 s, outer / 2), computed from the Duration the caller already builds: 25 s at the 30 s default, always positive, strictly shorter than the outer timeout, and free of Duration underflow. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: bootstrap resumes use stored policy inputs (#264) An empty-window resume rebuilt each miner's bootstrap bundle through Coordinator::build_bundle. That function reads config.ctv_enabled, payout_policy, ctv_direct_floor and ctv_config, so after a config or fingerprint reset a resumed job could get a different signed bundle under the same job ID. That is the drift the stored-inputs invariant forbids. The record now has the per-miner bootstrap rebuild call the borrowing builders directly. It takes its own build_slots permit after the single-flight entry's permit is released. Its inputs are the synthetic share build_bundle fabricates today, the template, anchor, payout_policy and ctv stored in StoredPrepared, and the balances and revision read_window returned. #273's must-not row now forbids build_bundle for any resume rebuild. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: readable imported audits and a race-free upgrade order (#264) Two fixes to the WindowRef design record after review. Imported legacy audits: #265 stores canonical_audit_bytes and no inline body. Only audit_canonical_bytes reads those bytes today. Ledger::audit_bundle returns just the JSON column, so backfill-ctv would stop on imported rows, and the bundle endpoint's fallback reads only body_uri. The record now requires the same PR to make both readers decode the stored canonical bytes first, digest-checked and under spawn_blocking. Leaving an imported audit readable only through body_uri is now forbidden. Upgrade order: a zero pending count taken while the pre-007 frontends still run proves nothing, because a block can be found before they stop. The count is now taken after every frontend is stopped. A non-zero count means restarting the pre-007 frontends to drain and repeating, and 007's refusal remains the backstop. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: one owned window per Prepared and the established lock order (#264) Two fixes to the WindowRef design record after review. Window ownership: Prepared and JobContext would have held an owned AuditBundle beside the Snapshot. into_bundle takes the share Vec by value, and two Arcs don't share storage, so every resume would keep a second window: about 0.75 GB per 400k rebuild instead of 0.5 GB. Both structs now hold Arc<Snapshot>, the only owned window, next to Arc<AuditBundleBody>. Canonical bytes and verification come from the parts functions, and an owned bundle is assembled only at claim. Lock order: the D6 prune now takes SETTLEMENT_LOCK and then ORDER_LOCK, the order every existing dual-lock site uses, so it can't form an advisory-lock cycle with a refresh snapshot or a candidate finalization. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: version the audit builder behind durable references (#264) Stored inputs reproduce a bundle only under the builder that first built it. If a release changes builder output, a pending candidate that survived the upgrade would rebuild into a different bundle than its coinbase commits to, and fail on every retry or land changed audit data. qbit-prism gains AUDIT_BUILDER_VERSION, which both the 007 candidate and 008's StoredPrepared store. A rebuild checks it before read_window: a mismatched candidate is retried with an alert and never rebuilt with the current builder, and a mismatched job is a cache miss. A release that bumps the constant ships a migration step that refuses while pending outbox rows carry another version; they drain through the 007 procedure. Keeping frozen old builders is recorded as the rejected alternative. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: pin the signing keys behind durable references (#264) The bundle embeds both public keys and their signatures, so a rebuild reproduces the original bytes only under the original keys. Rotating a key takes an operator resetting the cluster fingerprint, and a candidate still pending across that reset would be re-signed into bytes its coinbase does not commit to. Both documents now store signer_keys, the two public keys the building frontend signed with. A rebuild checks them next to the builder version, before read_window, and never re-signs a mismatch with the local keys. #265 extends configure to refuse a new fingerprint while a pending outbox row stores other keys, so the operator drains with the old keys first. Keeping retired seeds online to reproduce old signatures is recorded as the rejected alternative. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: reuse the refresh-time WindowRef at submit (#264) Submit built the candidate's reference from the prepared Snapshot, which re-hashes and re-serializes the whole 233 to 260 MB window whenever a block is found. That puts 0.7 to 1.4 s of digest CPU on the latency-critical share path, on top of the once-per-refresh cost the cost table budgets. refresh_once already computes the identical reference, so Prepared now keeps it as window: WindowRef. Submit clones that small value into the candidate, a resume takes it from StoredPrepared, and re-digesting at submit is a must-not for #265. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: sort rebuilt prior balances canonically (#264) prior_balances_digest sorts internally, but AuditBundle.prior_balances serializes in vector order. read_prior_balances has no outer ORDER BY, and SQL doesn't guarantee the function's internal order through a table expression. Balances returned in a different order at claim or resume would pass the digest check and still change the canonical bundle. read_prior_balances now sorts with the digest's comparator (order_key, recipient_id, p2mr_program_hex, byte order). Refresh and every rebuild share that function, so the builder gets the same vector. A SQL ORDER BY alone would follow the database collation, not the digest's byte order. A #265 unit test requires permutations to give identical bytes. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: finish recovered claims from landed digest columns (#264) A claim that finds its block's audit already landed finished through materialize_audit_row. That function reads the whole range unpaged, serializes and clones it on the async runtime, and builds an owned AuditBundle: two share arrays at 400k on the submit loop, which the cooperative 60 s deadline cannot interrupt. The recovery branch now checks the landed row's columns alone, as today's landing does for an existing audit. share_snapshot_sha256 must equal window_snapshot_sha256, found_block_bits must match the header, and the claim then finishes. materialize_audit_row on the submit loop is a #265 must-not. If that path ever needs the body, #267's paged, parts-based reader becomes a prerequisite. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: re-pin the WindowRef record to 3.x.x after #283 (#264) #283 landed through #300: the 2.x.x tip merge and the split of ledger.rs into candidates, jobs, window and connect submodules. The record's code citations were pinned to 1398bbc, and many pointed into the old ledger.rs. Every citation now resolves at 3.x.x 41a2afd: 76 of 252 changed, and all 247 cited line numbers were checked against that tree. "Where the code lives" now describes what #300 landed. ledger/window.rs exists under workstream B and already holds share_from_row and read_prior_balances, and SELECT_SHARE stays in the module root. The only question left for Anatolie is whether read_range moves out of A-owned ledger/audit.rs. Two review fixes in the landed-audit recovery path. A retried claim authenticates the landed audit against the block it found: the landed coinbase must match block_hex, and the root of the landed commitment leaves must equal the coinbase's witness reserved value. For an empty window, the snapshot comparison uses the digest of [bootstrap_share], because window_snapshot_sha256 is NULL there. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: keep pool connections free during window rebuilds (#264) build_slots bounds concurrent rebuilds at build_workers, but that doesn't protect the database pool. PRISM_DATABASE_MAX_CONNECTIONS may be as low as 4, build_workers may exceed it, and share appends and candidate-lease heartbeats share the pool with a 15 s acquire timeout. Rebuilds of distinct jobs could hold every connection for their whole multi-page reads, and a heartbeat that fails to get a connection drops recoverable claim work. read_window now also takes a permit from a window_reads semaphore sized clamp(database_max_connections - 2, 1, build_workers). The caller takes it after build_slots and releases it before the rebuild, so two connections always stay free. Calling read_window without that permit is a must-not for #265 and #273. A dedicated pool was rejected because it would exceed the configured connection budget. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: continue to submission after recognizing a landed audit (#264) The recovery path finished a claim as soon as it authenticated a landed audit. But an inactive block lands before the lease renewal and submitblock, so an earlier claim can land and lose its lease before it submits. Finishing then would abandon a valid found block that was never submitted. A recognized landed audit now means landing is done. The claim skips read_window, the rebuild and land_candidate, and continues through today's observe, renew and submitblock flow, finishing only where today's code does: an active block, or an inactive one whose revision or parent moved. Finishing a claim only because its audit landed is a #265 must-not. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: reserve in-flight windows and scope the claim deadline (#264) Two review fixes. The retention floor covered written references only. Ledger::snapshot releases its locks when its first transaction commits, and the scan, the bundle and save_job follow with no row referencing the window yet. A prune in that gap, after a fall in difficulty, could delete rows of the window being built. D6 now adds a reservation. The snapshot's locked first transaction inserts it, the refresh fills in the window start after the scan, and save_job deletes it (publishing nothing if it expired or the rows are gone). The prune skips while any reservation is unresolved and otherwise keeps every reserved window. An age-based horizon was rejected because a window's reach in time depends on hashrate. The claim's 60 s timeout covered the whole call, including submitblock, whose RPC timeout may be up to 86,400 s. It now covers read_window plus the rebuild only, and the later steps keep their existing deadlines. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: answer djh58's four B contract points (#264) djh58's review on #273 (comment 5624891010) raised four points against the resume contract. The record now: - lets B's published-identity and lease check decide resume eligibility instead of revision equality, and rebuilds eligible work with its as-issued balances from an immutable qbit_prism_balance_snapshots row keyed by prior_balances_digest; - keeps reconstruction failures (Incomplete, digest mismatch, decode, database, expiry) distinct from missing work, under one end-to-end deadline, the caller's existing initial_job_timeout_seconds, with no inner timeout; - names both share digests, the native snapshot_sha256 and #274's canonical_digest_hex, amends #274's reuse instruction, and adds a cost model for a sliding window; - moves the template out of the prepared payload into an immutable qbit_prism_templates row, keeping #273's 1 MB criterion whole-value, and moves #265's block_hex into a bytea column. It also records the permit-ownership and coalesced-waiter rules from the same review. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: add the balance source to read_window and authenticate block bytes (#264) The resume contract needs as-issued balances, but read_window's declared API had no way to ask for them, and the read algorithm always used the current set. Implemented literally, it would have returned PriorBalancesChanged for the published work it was meant to rebuild. read_window now takes a BalanceSource. Current reads the current view as before. AsIssued reads the qbit_prism_balance_snapshots row keyed by the reference's digest, sorts it with the digest's comparator and verifies it. A missing row is the new BalanceSnapshotMissing, and a mismatching one is Decode. Claims pass Current, and resumes of published work pass AsIssued. Moving block_hex to a bytea column took it out of candidate_sha256. The candidate JSON now keeps block_sha256. Claim decode checks it and that the block header hashes to block_hash before any witness derivation, landing or submission. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: prune externalized templates and balance snapshots with their jobs (#264) Moving each full template into an immutable qbit_prism_templates row left no cleanup for those rows. Templates change on every new transaction set and at every reanchor, so each is a new, possibly multi-megabyte row, and prune_expired_jobs deletes only job rows. The balance snapshots had the same gap. prune_expired_jobs becomes one transaction under SETTLEMENT_LOCK, the lock save_job holds while it inserts the template, the balance snapshot and the job row. It deletes the expired batch, then the template and balance rows that batch referenced and no remaining job row references, in two statements, because a data-modifying CTE's NOT EXISTS would still see the deleted jobs. The job row gains an indexed template_sha256. Balance snapshots follow job rows only, since claims read the current balances. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: keep the issued revision on resume and ratchet D6's horizon (#264) Two review fixes. A resumed Snapshot now keeps StoredPrepared.payout_revision, the revision the job was issued with. read_window's current revision is only B's eligibility input and the claim's probe hint. The job wire and any candidate take their revision from the snapshot, so a block found on R0 work carries R0, and observe_candidate takes the superseded path instead of retrying PriorBalancesChanged forever. A D6 horizon computed from today's difficulty fails when difficulty rises after a prune: Ledger::snapshot stops when rows run out, so it would publish an underweight window. D6 now needs a ratcheted horizon kept at a durable maximum difficulty in qbit_prism_cluster. It must also record the share_seq it deleted below, so snapshot fails when its scan reaches that floor before the requested weight. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: drop rebuilt windows from resumed jobs once the coinbase is built (#264) build_workers bounds how many rebuilds run at once, not how many rebuilt windows stay resident. Each resumed JobContext kept its Prepared, with Arc<Snapshot> and Arc<AuditBundleBody>, and a session keeps up to 64 jobs per connection. Sequential resumes of different jobs could therefore pin about 0.5 GB each, and the stated k x 0.5 GB peak didn't hold. Once the rebuild has produced the job's coinbase, a resumed Prepared keeps only what submit reads: the WindowRef, the snapshot scalars, the template, the fee and the body's non-window fields. Submit reads nothing else, and the whole-bundle clone into the candidate goes with #265. Snapshot.shares and reward_manifest.shares are dropped before the job reaches the session, so windows exist only inside rebuilds. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: stop old signers before a reset and fence writes on the fingerprint (#264) configure's one-time pending-row check could pass while an old-key frontend was still running. That frontend could then find a block and enqueue a candidate with the old signer_keys, which no new-key frontend can rebuild, and enqueue is not fenced by the fingerprint today. A rotation now follows the 007 order: drain, stop every old-key frontend, then check that nothing is pending, reset and start the new frontends. #265 and #273 also re-read config_fingerprint FOR SHARE inside every enqueue and save_job transaction and refuse writes under a foreign fingerprint. FOR SHARE conflicts with configure's FOR UPDATE, so a write and a reset cannot interleave, and a procedure violation fails loudly instead of leaving rows nobody can rebuild. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: keep every stored input in slim resumed jobs (#264) The slim resumed Prepared kept the WindowRef, the snapshot scalars, the template, the fee and the body's non-window fields, but not the stored payout_policy and nested ctv. A candidate found on that job must persist those exact inputs, and neither the fee nor the coinbase manifest can recover them, so submit would have had to read local configuration. The slim form now keeps the decoded StoredPrepared itself: payout_policy, the whole ctv, audit_builder_version, signer_keys, coinbase_suffix, template_sha256, the WindowRef and the snapshot scalars, all O(1). For an empty window it also keeps the miner's own bootstrap_share. Only the window is dropped. Dropping a stored input before enqueue is a #273 must-not. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: let the published-work lease reach submission (#264) Keeping R0 on a resumed snapshot doesn't help if its found block never reaches observe_candidate. Today submit rejects work whose frontend or job revision differs from the ledger's, apart from parent-stale work inside the stale grace. That grace work is credited as a share, but a candidate is enqueued only when the work isn't stale, so its block is dropped. B8's lease now decides those submit checks the way it decides resume eligibility. Work the lease covers is not stale because the revision changed, and a block found on it is enqueued as a candidate carrying R0, with its as-issued window and inputs. observe_candidate then decides at claim whether it lands. Work outside the lease keeps today's handling. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: submit leased candidates first and clear terminal block bytes (#264) Two review fixes. A candidate enqueued from leased R0 work still met the unchanged claim fence, which would finish it as superseded before submitblock and discard the block. The candidate now records a leased flag. A leased candidate skips the pre-submit supersession checks and the pre-submit landing, renews its lease, calls submitblock, and only then meets observe_candidate. Whether an active leased block lands is B8's rule with #289. Moving block_hex into the block_bytes bytea column meant the terminal UPDATE no longer removed the block. It now NULLs block_bytes too, so the outbox doesn't keep every finished block forever. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: correct claim-flow line citations to 3.x.x 41a2afd (#264) The landed-audit continuation and leased-candidate paragraphs cited coordinator.rs claim-flow lines read from a later 3.x.x, where #308 had shifted them by five. They now point at the record's pin, 41a2afd: the post-rebuild observe at :1039, the active branch at :1040-1048, the inactive supersession at :1050-1063, the landing at :1067-1069, the post-land check at :1070-1080, the lease renewal and submitblock at :1084-1092, and the post-submit observe at :1095. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: submit leased candidates before any rebuild (#264) A leased candidate skipped the pre-submit fences, but the #265 claim rebuild still called read_window with BalanceSource::Current before them. For an R0 candidate that returns PriorBalancesChanged and retries forever, so the block never reached submitblock. A leased candidate now submits before any rebuild, because submitblock needs only the stored block. If the block is active, its audit rebuild uses BalanceSource::AsIssued. A non-terminal leased outbox row therefore keeps its balance snapshot. The prune checks for such rows and takes SETTLEMENT_LOCK then ORDER_LOCK, so it cannot interleave with a leased enqueue. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: release rebuilt windows off the runtime; hold the lease to unchanged balances (#264) Two review fixes. Freeing a completed rebuild's last Arc<Snapshot> and Arc<AuditBundleBody> drops about 0.5 GB on whichever Tokio worker releases it, which can stall lease heartbeats and share processing. The rebuild's spawn_blocking closure now builds the coinbase and the slim resumed form and drops the window there. Every other release of a rebuilt window goes through spawn_blocking(drop). An active leased block rebuilt with R0's balances could not land, because landing requires the current balances. Until B8 and #289 define how such a block lands, the lease covers only revision changes that leave prior_balances_digest unchanged, and a balance change ends it at once. An active leased block then lands through today's path, and the open question asks djh58 and #289 for the full rule. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: land first-time claims from parts; sweep orphaned balance snapshots (#264) Two review fixes. A first-time landing assembled the rebuilt window into an AuditBundle, then cloned it, serialized it whole for the stored body and the canonical length, and re-read the range unpaged, all on a Tokio worker. At 400k shares that is several half-gigabyte representations on the runtime. CandidateClaim now holds the parts. Landing verifies, serializes and digests them in one spawn_blocking closure before its transaction opens. It keeps the range fence, but as a paged re-read. #267 is not a prerequisite. The balance-snapshot prune only considered digests from the expired batch, so a snapshot kept alive by a leased candidate could never be reclaimed once its job had gone. Snapshots now get their own statement over every row with no remaining reference. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: keep the bootstrap share in local jobs; correct local window residency (#264) Three review fixes. An empty-window local job held its only full AcceptedShare in JobContext.bundle. Once JobContext holds a body, that share would be gone and submit could not fill Candidate.bootstrap_share. JobContext now keeps the synthetic share beside the body. The record claimed the local frontend keeps one full window. Retired local jobs keep their generation's Prepared for Stratum's retention period, as they do today. The record now says so. Bounding that is logged as a #273 acceptance note. The landing bind still copies the stored body into SQLx's argument buffer on the runtime. The record now names that copy. Measuring it is logged as a #265 acceptance note. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: keep null-response leased candidates pending; carry empty-window balances (#264) Two review fixes. The record finished an inactive leased candidate as superseded after submission. Today's wrapper treats a null submitblock result on an inactive block as unresolved, because it can still be a known side-chain block, and retries with the candidate intact. Leased candidates now keep that outcome, so a block that becomes active later still has its audit inputs. For an empty window, resume waiters build their own bootstrap bundles from the as-issued balances. The single-flight entry now carries those balances until every waiter has built, instead of only the rebuilding caller holding them. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: land a submitted leased candidate before any terminal outcome (#264) After a null submitblock result on an inactive leased block, the retry resubmitted before landing. A node can then answer "duplicate", which the string branch treats as terminal, clearing the unlanded audit inputs. Today's flow avoids that by landing an inactive block before its RPC, so a late acceptance stays reconcilable. A leased candidate still submits first, so delivery never waits on a rebuild. It then rebuilds with AsIssued and lands at the observed revision before any terminal outcome, whether or not the block is active, and only then takes today's three outcomes. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: re-establish candidate references inside the enqueue transaction (#264) Taking locks in order stops a prune and an enqueue from interleaving, but it doesn't decide which runs first. A submission that passed its expiry check could lose to a prune that deleted the job row and its balance snapshot, and then enqueue a leased candidate pointing at missing data. Enqueue now re-inserts a leased candidate's balance snapshot from the as-issued balances the job carries (the slim resumed form keeps them), and probes the window's first share row, both inside its ORDER_LOCK transaction. Shares can't be written back, so an expired job row now holds the retention floor until the prune deletes it, and the prune waits a grace past expiry. A failed probe still publishes the candidate, with an alert. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(prism): reserve session extranonces safely across sequence wrap * fix(prism): fence stopped session owners by process incarnation * fix(prism): drain session guards before shutdown cleanup * fix(prism): scope migration readiness to mining startup * test(prism): build complete legacy schema for read-only connect * fix(tests): route integration databases through test gate * test(prism): register all gated session coverage * fix(tests): adapt fixtures to shared session backend
…320) * docs(prism): generate metric inventory and migrate deployed alerts * fix(prism): preserve the network-wide connected client alert floor * fix(prism): evaluate standby lag from observed durable WAL progress * fix(prism): delete provisioned native rules when gates disable them * fix(prism): alert on invalid synchronous standby configuration * docs(prism): record local review evidence and prepare PR draft * fix(prism): reject stale alert and metric metadata * fix(prism): alert when public scrape fails * fix(prism): preserve native failure and paging alerts * fix(prism): preserve native alert tuning overrides * Revert "fix(prism): preserve native alert tuning overrides" This reverts commit 31b2681. * fix(prism): preserve configurable alert tuning * fix(prism): render tuning dwells as JSON * fix(prism): validate rendered alert dwell overrides
* docs(prism): pin intentional legacy health differences * docs(prism): correct legacy max blocks semantics * style(prism): format health snapshot assertions
* fix: preserve PRISM issued-work and stale-grace semantics Separate observed and published tip authority, retain original issued worker context through replacement and reconnects, and preserve bounded delivery-based grace without renewing expiry after awaited work. Exercise real Coordinator decisions and socket interleavings, including constant node-RPC load across 1, 32 and 128 clients. Refs #280 * docs: reference PRISM miner parity settings Co-Authored-By: OpenAI Codex <noreply@openai.com> * fix(prism): bind share admission to one published payout view * fix(prism): verify changed block notifications with fresh chain state * docs(prism): record B8 review regressions and rebased qualification * test(prism): adapt merged admission fixture to stale grace contract * fix(prism): classify retained capacity by current published tip * fix(prism): retain prepared dependencies for issued work * test(prism): verify issued job dependency persistence * fix(prism): keep sessions alive while retained work is usable * docs(prism): record timer coverage and final qualification procedure * docs: point D2 credit explanation to miner submit module * test: align candidate lease tip fixture with coordinator state * fix: integrate miner parity with native metrics registry * fix: integrate miner parity with metrics registry API * ci: run issued job dependency acceptance explicitly * test: keep miner parity fixture publication reachable * test: complete JSONB gate fixture config * docs: specify prepared window reference integration plan * test: restore D2 coordinator coverage * feat(prism): add window reference schema and authenticated reader Add the independently approved B273 schema and reader foundation on PR313, with typed reconstruction errors, immutable blob tables, per-version migrations, and PostgreSQL qualification. Runtime prepared/resume, candidate, atomic repair and GC integration remain gated on A265 and shared owner agreement. * feat(prism): add coherent payout state for lease eligibility * Keep cancelled window cleanup off runtime workers and qualify current base * Qualify landed 008 and 009 migration orders on current base * Preserve legacy balance order and reject fatal window reads * Prove legacy balance order preservation in PostgreSQL * Run all gated suites before manifest proof * Revert "Run all gated suites before manifest proof" This reverts commit b11cbbb. --------- Co-authored-by: OpenAI Codex <noreply@openai.com>
…m on read (#325) * 3.x.x: store imported legacy audits as canonical bytes and decode them on read The legacy-audit import wrote the parsed bundle back into the audit_bundle JSONB column. That value holds two copies of the payout window (shares and reward_manifest.shares), so at 400k shares it crosses PostgreSQL's JSONB container limit. The import now stores only canonical_audit_bytes and the non-share metadata. It keeps its idempotency and optimistic guards, verifies off the runtime, and drops the parsed window before the write transaction opens. Rows that already hold an inline body keep it unchanged, because the body-present CHECK needs one of audit_bundle and body_uri. found_block_bits stays as stored: no audit body carries the compact bits. A shared decode_canonical_audit_body helper checks the stored bytes against audit_bundle_sha256 and parses them with parse_audit_bundle_value. It returns the logical body the import used to store inline, and all of that work runs under spawn_blocking. Ledger::audit_bundle serves native snapshot rows first, then imported canonical bytes, then inline bodies, so backfill-ctv works on imported rows. The bundle endpoint serves imported bytes before any body_uri fallback, on the same public pool. A digest mismatch or undecodable bytes refuses the row, even when the legacy file still exists. The settlement-artifacts audit link now also accepts stored canonical bytes, so it does not decode a whole body just to show a link. The JSONB ceiling gate drops its import-phase known violation, and the measurement and capacity-readiness docs now count three window-carrying writes. Part of #265. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * 3.x.x: load only the served audit body and bound imported-audit decodes The bundle queries now project no inline audit_bundle for a row whose canonical bytes supersede it, the same rule Ledger::audit_bundle uses, and report whether canonical bytes exist so bundle() skips the bytes query for rows without them. The flag is removed before responding. Imported-audit decodes run after their read connection is released, so ApiState now carries a semaphore sized from the same read concurrency as the public read pool. bundle() takes an owned permit inside the request's existing deadline and moves it into the blocking decode, so a dropped request cannot free it before the decode ends. Ledger::audit_bundle takes no permit and releases its fetched row before decoding. The decode refuses body-ref and v2 envelope schemas, since canonical bytes are always a flat bundle. The storage-sizing doc describes the canonical-bytes import, and the measurement doc gains a post-#265 note. Tests pin the pre-change row shape (inline body, canonical bytes and body_uri, no snapshot), bound the decode across a dropped request, and refuse a digest-valid body-ref envelope. Part of #265. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test: add deterministic HA readiness probe harness * test: qualify readiness probe timing * test: preserve probe state in jsonl harness * test: harden readiness probe input boundaries * test: reject overlapping readiness probes * docs: use generic HA readiness contract wording * test: classify malformed probe durations as failures
#333) * ledger: let append callers veto COMMIT at the last moment append_at_revision_gated takes a non-optional pre-commit hook. It runs at most once, after every statement including persist_candidate has succeeded, while ORDER_LOCK is held and immediately before COMMIT. A false return rolls the transaction back explicitly and fails with the typed CommitGateClosed error, so COMMIT is never sent. append and append_at_revision keep their signatures and pass no hook. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * metrics: count late share confirmations and unknown outcomes A share whose commit is confirmed after the acknowledgement deadline is still accepted, and a commit whose outcome cannot be established is no longer reported as a clean rejection. Both need their own signal. Add the late_confirmed_shares_total family, recorded through record_late_confirmation and zero-registered like the other counters, and the ledger-outcome-unknown reject reason. Without the new reason, an unknown outcome would fold into internal-error. PrismShareAppendFailures paged on any ledger-confirmation-failed, so it now matches the unknown reason too and keeps its low-volume total-write- failure coverage. The deployment patch is regenerated from the reviewed snapshot, and a scenario covers the new reason alone. Document the narrowed meaning of ledger-confirmation-failed, the new reason, the statement-timeout guard, and the block-only acknowledgement bound. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * coordinator: reconcile share-commit deadlines instead of dropping the append (#324) submit wrapped the whole persistence future in a timeout and dropped it when the deadline fired, answering ledger-confirmation-failed. If COMMIT had already been flushed, sqlx drained the reply and reused the connection, so the share was committed and credited without an acknowledgement. A load harness reproduced that: 164 such rejections, two of them committed rows. The block-only path committed its candidate first, so its timeout always led to a later credit with no acknowledgement. Run the append in a task behind a one-shot commit gate. At the deadline the acknowledgement path either closes the gate, which proves COMMIT was never sent and aborts the doomed append, or finds the commit already in flight and awaits its reply for share_commit_grace. Only a server error of severity ERROR is a definite rejection; every other error, and a commit that outlasts the grace, is answered ledger-outcome-unknown and logged with its eventual outcome. A confirmation whose commit took at least the effective statement_timeout is also unknown, because a cancelled synchronous-replication wait commits only locally. An append carrying a found-block candidate is never refused, since that would discard the block; it waits for block_only_ack_timeout instead. Block-only proofs always finish their enqueue and then follow the candidate's disposition within the same bound, and poll errors after the enqueue never become a rejection. The new bounds are Config fields rather than environment variables, and statement_timeout is read from the session the ledger pool actually uses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * coordinator: bound the block-only enqueue wait and time COMMIT in the task Review feedback on PR #333. enqueue_candidate_once was awaited directly, outside the bound that covers every other step of a block-only submission, so a degraded database could hold the acknowledgement open well past block_only_ack_timeout. Wait for it in a task until the bound instead. The enqueue still runs to completion, because cutting it off could strand a COMMIT that was already sent, but the miner is answered ledger-outcome-unknown on time and the enqueue's eventual outcome is logged. The synchronous-replication guard compared statement_timeout against a COMMIT duration measured when the coordinator classified the finished task. Scheduler delay after a successful COMMIT therefore counted as database time and could turn a durable accepted share into an unknown one. Measure the duration inside the append task, where COMMIT actually finishes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * metrics: count a late confirmation from when the append finished Review feedback on PR #333. late_confirmed_shares_total was decided by reading the clock while the acknowledgement was being processed, which can be arbitrarily later than the append itself. A coordinator task scheduled late therefore counted an on-time confirmation as a late one. Decide it from the instant the append finished, which the task already reports, and record it where that instant is known. The help text also claimed every counted share was confirmed within the grace period. That was never true of a share-pass submission carrying a found-block candidate: those are deliberately never refused, so they can be confirmed later still, up to the block-only bound, and they are counted. Say what the counter measures instead, and spell the found-block case out in the metrics documentation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test(prism): add bounded PostgreSQL WAL measurement helper * fix(prism): harden WAL qualification failure semantics * fix(prism): harden WAL helper validation and outcomes * fix(prism): sanitize WAL helper connection inputs * test(prism): cover WAL helper outcomes and scope * fix(prism): preserve WAL operation outcome uncertainty * fix(prism): bound WAL measurement subprocesses * fix(prism): complete WAL helper diagnostics and test coverage * test(prism): repair WAL helper test layout * test(prism): stabilize WAL diagnostic mocks
…323) * 3.x.x: measure the share-append throughput floor against PostgreSQL test/test-prism-postgres-throughput.sh measured `cargo run -- benchmark`, a CPU-only run of the audit builder with no database at all, and the 2.x.x report it replaced measured one bulk INSERT ... SELECT ... FROM generate_series. Neither says anything about the path a share actually takes. Every native share append is one transaction that takes the cluster-wide ORDER_LOCK advisory lock, checks writability, bumps the single-row ledger clock and writes two rows, so share appends are serialized cluster-wide, and nobody had measured that with more than one writer. tests/throughput_floor.rs seeds a payout window with the #264 fixture generator and then appends through Ledger::append at 1, 2 and 4 concurrent appenders, each appender holding its own Ledger with its own pool and its own instance_id, as separate production frontends do. A connection outside every appender pool samples pg_locks for backends queued behind ORDER_LOCK while a level runs, and the report says how many waiters it saw and for how long -- labelled throughout as sampled estimates, with the episode count a lower bound, because a wait shorter than the 10 ms interval can be missed entirely. A sampler that fails reports null and the error, never zero. The report is qbit.prism.postgres-throughput.v2, a deliberately new schema name: its numbers must never be read against v1's bulk-insert rate. It is written before the floor is asserted, so a run that fails the floor still leaves its evidence behind. Every QBIT_PRISM_* input is validated at entry, before a connection is opened: nan, inf, a negative or an empty floor are configuration errors rather than extreme floors, and a window size that does not divide the fixture's window weight would not produce the window it claims to. CI_MIN_SHARES_PER_SEC is provisional -- 25% of the slowest level rate measured in a debug build on the writer's host. Debug is the right base because CI runs this test in debug, and the wide margin is honest about the fact that nothing has calibrated it on the 2-vCPU runner yet. Part of #271 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: point the throughput script and target at the share-append floor test-prism-postgres-throughput ran `cargo run -- benchmark`, which measures the audit builder's CPU cost and never opens a database, so the target named after PostgreSQL throughput was not measuring PostgreSQL at all. It now runs the full-size share-append floor: the ignored throughput_floor test, in release, against PRISM_TEST_DATABASE_URL when one is set and otherwise through prism-native-tests.sh, which initdb's a disposable cluster on a random loopback port and removes it on exit. The builder benchmark has not gone anywhere and is still worth running; the script says so, and gives the exact command, because it answers a different question rather than a worse version of this one. `set -e` is suspended around the test so the report path is printed on a red run too -- the test writes the report before it asserts the floor, so a run that fails the floor is exactly the run whose report someone wants to read -- and the test's exit status is re-raised immediately afterwards. Every QBIT_PRISM_* variable reaches the test process unchanged; the script reads none of them except to say where the report landed. Part of #271 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: resolve pg_stat_statements by schema, not by search path The pg_stat_statements probe was `to_regclass('pg_stat_statements')`, and the read and the reset named the view unqualified. Every connection in this run carries `options=-csearch_path=<run schema>`, which replaces the search path rather than prepending to it, so `public` is not on it. On a server with the extension installed in `public` the probe therefore returned NULL and the report said "unavailable" about a view that was sitting right there -- an EP-OBSERVABILITY failure of exactly the kind the rest of this file is careful about, because it reads as "this server cannot tell us" when the truth was "this query looked in the wrong place". The extension's schema is now resolved from pg_extension and the server's own quote_ident, and both the reset and the read qualify with it. Verified against PostgreSQL 16.15 with `shared_preload_libraries=pg_stat_statements` and the extension created in public: the block now reports 2000 calls per level, and its total_exec_time corroborates the sampler closely -- 4254 ms against a sampled 3.97-4.31 waiter-seconds at two appenders, and 17387 ms against a sampled 16.9-17.3 at four. Two independent methods agreeing to within a few percent is worth more than either alone, which is the whole reason the block is there. Part of #271 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: attribute ORDER_LOCK waits to the run that caused them The adversarial review found that the per-run schema isolates the tables but not the lock. ORDER_LOCK is a PostgreSQL advisory lock and an advisory lock is scoped to the database, so any other backend appending shares into the same database serialises against this run's appenders, and the sampler -- filtered only by database -- counted those foreign waits as this level's. Measured on a quiet server against the same server with one other appending process, the rates fell 40-57% and nothing in the report said why. A run like that either fails a calibrated floor for a reason that is not the append path, or, once the margin narrows, passes on a number describing two workloads. Every connection this run opens now carries a per-run application_name, `throughput-<uuid>`, pushed through the URL -- which is the only thing Ledger::connect is handed, so the seeding ledger, every appender ledger and the sampler all announce themselves without any production code knowing this test exists. sqlx 0.8 parses the parameter out of the URL and sends it in the startup packet, but the run does not take that on trust: it reads current_setting('application_name') back on its first connection and refuses to measure anything if the server disagrees, because a silently ignored parameter would make every foreign backend look like ours and every contaminated run look clean. The sampler then reports this run's waiters as before, and foreign_waiters_max and foreign_backends_seen beside them. A foreign backend counts whether it was waiting on the lock or holding it: either way it is serialising against these appends. A backend whose pg_stat_activity row this role may not read counts as foreign, which is the safe direction. A level during which any foreign backend was seen is marked contaminated, in the level, at the top of the report and in the floor's failure message; it is still measured and still gated, because the rate is what the appends really achieved, but the report now says plainly that the number describes a shared database and must not be carried forward. contaminated is null, not false, when the sampler could not look. "Nobody checked" is not "nothing was there", and the same reasoning fixes the smaller version of that bug next door: a level that ended before the sampler completed a poll used to emit mean_waiters 0.0, max_waiters 0 and estimated_waiter_seconds 0.0, which reads as "nobody ever waited". Those are null now, samples stays 0, and the error says the level ended first. The sampler's shutdown and its interval are fixed together. It tested the stop flag only at the top of its loop and then blocked in sleep(interval), so every level burned up to a whole interval of dead time after its appends had finished -- 65 seconds of wall clock for 0.43 seconds of measurement at a 20-second interval. The flag is now paired with a Notify signalled alongside it, so the sleep is interruptible and a level ends when the appends end; notify_one stores the permit, so the signal cannot be lost in the window between a poll and a sleep. The interval itself is bounded to 1..=1000 ms at entry, before any connection, with a message saying that an interval coarser than a second cannot resolve a lock wait that lasts milliseconds. Two other gaps the review left open. A run that dies connecting, seeding or appending now writes a report anyway -- identity, configuration, the full error chain, passed_minimum false and an empty results array -- so the newest file on disk belongs to this run rather than to some earlier one; a failure to write it is attached to the original error as context and never replaces it. And the default report path is per test, target/prism-postgres-throughput-<test>.json, with a "test" field in the JSON to match, because the two floor tests run in parallel threads under --include-ignored and the survivor of a shared path was whichever finished last, unlabelled. QBIT_PRISM_THROUGHPUT_REPORT is still the explicit override. Cancellation is documented rather than fixed: SIGINT kills the process before the schema drop runs, and handling it properly needs a signal handler, which means a dependency this test may not add. The module docs now say so and give the query that finds and drops the leftovers, next to the standing requirement that this run wants a database nothing else is appending to. Part of #271 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: make the throughput script point at a report that exists Two things the script got wrong, both of which send a reader looking for evidence that is not there. It printed the report path unconditionally, after the status had been captured. On the fallback path with PRISM_TEST_DATABASE_URL unset and no local server binaries, the run dies before the test starts and writes nothing, yet the script still announced a file. It now prints the path only when the file is actually on disk, and otherwise says that no report was written and where it would have been. The exit status is unchanged either way. It also pointed at the old shared default path. The test writes target/prism-postgres-throughput-<test name>.json now, one per test, so the script derives its path from the test it runs rather than from a constant that has moved. The header gains the two warnings the test itself carries. ORDER_LOCK is database-wide, not schema-wide, so this must run against a database nothing else is appending to; a run that shares one is marked contaminated and its number describes two workloads. And a cancelled run leaves its prism_throughput_<uuid> schema behind, so the header gives the query that finds the leftovers and the statement that drops them -- unnecessary on a disposable cluster, which is thrown away whole, and that is the reason to prefer one. Part of #271 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: print the floor's per-level rates where CI can see them The floor test prints its rates with println!, and libtest throws away the captured output of a passing test. CI keeps no report artifact either, so a green prism-native-postgres run recorded nothing about how far above the floor it was, and the provisional CI_MIN_SHARES_PER_SEC could not be calibrated from CI at all. After the report is written, the test now emits one summary line through std::io::stderr() directly, which the test harness does not capture: the test name, build profile, window, shares per level, the rate at each appender count, the floor, the verdict and the contamination flag. Every CI log then carries the margin the floor is set from. A failed write is ignored, because the report file already holds everything the line summarises. Part of #271 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * 3.x.x: move the throughput floor onto the shared integration gate #322 replaced every test's private copy of the database guard with one shared gate, qbit-prism-test-gate, and made the prism-native-postgres job prove that each gated test executed against test/prism-gated-tests.txt. The floor still carried its own copy of the old guard, which the new check_gate_env_reads lint rejects, and it was not on the list, so the proof step would have failed the moment the test ran. The floor now asks the gate for its database URL, so it follows the same decision table as every other gated test and records its decision in the execution manifest. share_append_throughput_floor joins the list; the full-size test is not selected in CI and stays off it. Part of #271 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * 3.x.x: set the CI throughput floor from the CI runner's own numbers CI_MIN_SHARES_PER_SEC was a provisional 71 shares/s, a quarter of what an 8-vCPU devbox managed, because nothing had measured the runner that enforces it. Four runs of prism-native-postgres on blacksmith-2vcpu-ubuntu-2404, read from the summary line the test now writes, put the slowest level of each run at 1,290, 847, 1,242 and 1,200 shares/s (debug build, the job's PostgreSQL 16 service, a 20k-share window and 2,000 appends per level). The floor is now 250, about 30% of the slowest level seen. Runs on that runner differ by about a third, so the margin keeps a noisy run green, while a 3-4x collapse of the append path still fails the build. The report's source label no longer calls the constant provisional. Part of #271 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * 3.x.x: run the floor the disposable-cluster path was skipping Four findings from the review of this PR, all in the paths that are hardest to exercise by hand. The wrapper script passed cargo_args beginning with 'test' to prism-native-tests.sh, which runs 'cargo test "$@"' itself, so the second word became a test-name filter. With --exact alongside it the disposable-cluster path selected nothing, printed nothing and exited 0: the script that exists to measure the floor measured nothing whenever PRISM_TEST_DATABASE_URL was unset. The array now carries only the arguments, and the direct call adds 'test' itself. The full-size test is selected by name, so a missing database must fail it rather than skip it; it now takes the gate's required entry point, and the CI-sized test keeps the skipping one. Resetting pg_stat_statements cleared the whole server's counters, including other databases; each level now reports the difference of two reads instead, and says so where the numbers are. The level clock started after the barrier opened, so an appender the runtime resumed first could work outside the measured window and inflate the rate; it starts before. Part of #271 * 3.x.x: scope the lock statistics to this database and stop announcing stale reports Two findings from review, both about a number that looks right and is not. `pg_stat_statements` is server-wide. The level's advisory-lock figures were the difference of two reads filtered only by query text, so an advisory lock taken in any other database on the same server landed in the level's count. A lock in another database cannot contend with this run -- an advisory lock is scoped to its database -- so the level was correctly marked uncontaminated while its lock figures described someone else's work. Filter on `dbid`, and say so in the report's caveat. Measured on a PostgreSQL 16 with `pg_stat_statements` loaded, over a level during which another database took 3,000 advisory locks: the old query attributed 3,000 calls to the level, the filtered one attributes 0. With the fix and a concurrent foreign load of about 24,000 locks, each level reports exactly 300 calls for 300 appended shares -- one lock per share, which is what the append path takes. The wrapper announced any file at the report path as this run's evidence, so a run that died during compilation or cluster setup pointed the reader at a previous run's report while returning a failure. The earlier file is now moved aside before the run, which makes the report's presence afterwards proof that this run wrote it, and it is put back if this run wrote nothing. Deleting it instead would have destroyed the previous run's data on a run that produced none of its own. Timestamps were tried first and rejected: a warm build and a small window can finish inside one filesystem timestamp tick, and a fresh report then compares as no newer than the moment the run started, which reports a good run as stale. Verified end to end through the wrapper against a real PostgreSQL 16, and for all three announcement cases: a failed run with an earlier report present, a successful run that writes one, and no report at all. Part of #271 --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
) `3.x.x` has been red since #344 landed. Two pull requests collided semantically while each was green on its own: - #333 (e13051c) added `ledger-outcome-unknown` to `RejectReason`, so `/metrics` now emits one more `qbit_prism_rejections_total` series; - #344 (037b04e) added `event_exposition_matches_the_pre_allocation_change_byte_for_byte`, whose fixture was captured at 2914a62, before that reason existed. Neither branch could see the problem, because it only appears in the merged tree. The whole difference is one line, so the fixture gains it. The comment above the assertion is refreshed at the same time: it claimed provenance the fixture no longer had, and it now says what the pin implies -- adding a value to a closed label set has to be recorded here. Verified on this branch: the four metrics_allocation tests pass, and the failure reproduces on a6fe3fd without this change. Part of #338
* fix: record cancelled collector pool acquisitions * docs: align collector cancellation timing contract * docs: qualify cancelled collector waits in alert contracts * test(metrics): include approved unknown ledger outcome in fixture PR #333 intentionally added the ledger-outcome-unknown rejection label and a producer contract test. Add exactly that sample to the older exposition fixture from #344, retaining the byte-for-byte equality assertion and leaving runtime behavior unchanged. (cherry picked from commit 4e5b4c4)
rustls 0.23.44 accepted TLS 1.3 handshake messages sent at the wrong encryption level when they followed a key-changing message in the same record, instead of terminating the connection as RFC 8446 section 5.1 requires. The transcript stays authenticated, so a network-position attacker cannot alter or complete a handshake; the effect is that a peer can send handshake messages in plaintext that should have been encrypted without rustls rejecting the connection. It is the same bug as Go's CVE-2025-61730. The advisory names 0.23.45 as the fix. This is a lockfile-only change: `cargo update -p rustls`, no manifest or source change. The advisory database picked this up today, so the dependency audit now fails on 3.x.x and on every open pull request.
…ndow (#353) * 3.x.x: add PrismRewardManifestHeader and a checked reward-manifest rebuild (#267) A native audit row is about to stop storing `reward_manifest.shares`. The counted window is a pure fold over `(shares, found_block)`, so the row only needs the manifest header; `restore_reward_manifest` reruns the fold and returns its window only after proving the rebuilt header is identical to the stored one, failing as the existing `reward_manifest` audit mismatch otherwise. There is no unchecked variant. `PrismRewardManifestHeader` denies unknown fields, so a legacy manifest that still carries `shares` can never be read as a header; and the full manifest still requires `shares`, so an older reader meeting a normalized row fails explicitly instead of serving an empty window. Tests prove the rebuilt bundle is byte-identical to the original (both schemas, credit_policy included), that one changed field in a rebuilt share moves the canonical sha256, that a tampered header is refused by name, and that neither shape can be mistaken for the other. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LXvnfUC8CvPxskAXLgd1xx * 3.x.x: store native audit bodies without reward_manifest.shares and verify the range before the settlement lock (#267) The landed native audit body kept `reward_manifest.shares`, one record per counted share, so it grew with the payout window: about 235 MB in one JSONB value at the production window, 88% of PostgreSQL's container limit. The landing write now removes that copy alongside the top-level `shares`; the stored body is O(distinct miners and payout accounts). Nothing else is added or removed, `audit_body_byte_len` stays the full canonical artifact length, and the GIN-indexed leaf arrays stay in the body. Reads branch on the stored shape. A body whose `reward_manifest` has no `shares` key is decoded as `PrismRewardManifestHeader` and its window rebuilt through `restore_reward_manifest`, which refuses a header the fold does not reproduce; a body that still has the key (landed before this change) is served exactly as before. Both arms then run the unchanged canonical digest check, so a rebuilt body is proven byte-identical, not merely equivalent. Imported rows never enter this path. The snapshot digest, the rebuild and the canonical hash run on one blocking thread, and the public API hands that job the same permit imported decodes hold, so a dropped request cannot free the limit before the rebuild ends. The durable-range proof no longer runs inside the settlement transaction. `verify_durable_range` runs before the lock, page by page (4096 rows, the window reader's page), decoding and comparing each page on a blocking thread against the same positions of the candidate window: same predicate, same ordering, same full equality, same bootstrap exemption, same error text, and the first mismatched page fails the landing. An already-landed block skips it and takes the existing-row digest path as before. Under the lock the snapshot now only counts the same anchored predicate and fails the landing if the count differs from the window. The JSONB ceiling gate's landing entry is deleted: that write no longer crosses the threshold. Gated tests prove the stored shape and identical served bytes, that a tampered stored header is refused by name, that pre-change native rows still serve identical bytes, and that imported rows serve from their canonical bytes even with a poisoned inline body. An ignored harness measures the stored body, the settlement-lock hold and the landing peak RSS at a configurable window size. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LXvnfUC8CvPxskAXLgd1xx * 3.x.x: document the measured native audit body before and after normalization (#267) State the measured stored bytes per block at 20,000 and 100,000 shares before and after `reward_manifest.shares` left the stored body, say that the residual is O(distinct miners and payout accounts) rather than a fixed bound, and separate `audit_body_byte_len` (the full canonical artifact) from `sum(pg_column_size(audit_bundle))` (the stored, compressed non-share body). Also record the measured cost of rebuilding a native audit on read. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LXvnfUC8CvPxskAXLgd1xx * 3.x.x: correct three statements the audit-body normalization made untrue (#267) The JSONB ceiling gate's module doc and its landing phase note both still described the landing write as embedding the payout window, and the gate printed that note on every run. The payout-artifact measurement guide counted three native window-carrying writes. After normalization there are two, and landing is flat in the window. `d2_bootstrap_tests` cited `persist_audit_snapshot` re-reading the share range inside the transaction as one of the three closed exits for an empty payout window beside a non-zero carry. The exit is still closed, but by `verify_durable_range` page by page before the settlement lock plus the in-transaction count of the same anchored range, so the citation now names both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * 3.x.x: give the paged durable-range proof negative multi-page coverage (#267) `verify_durable_range` had no negative coverage: its body could be replaced with `Ok(())`, together with the in-transaction count guard, and every reachable test still passed. The gated tests only ever drove it with a one-share window, so no test had executed a second page of the loop. A new gated test loads a 10,000-share window, three pages of the proof's 4096 rows, and lands four candidates that disagree with the ledger from the claim side, since the share ledger is immutable by trigger. Each window is built and signed as it is, so the audit signature check that precedes the proof accepts it and the proof is what refuses it, asserted by name: one `ntime` differs on the first share of the second page, one on the last share of the final partial page, one share is missing from the final page so the durable page outruns the window slice, and one share the ledger accepted after the anchor is claimed, so every page matches and the final count refuses it. A refused landing writes nothing. The unaltered window then lands and serves identical bytes, so the same fixture proves the loop accepts a correct multi-page window. The in-transaction count guard has no independent coverage. The anchored set is frozen once the anchor is issued and the ledger is append-only, so a window that passes the paged proof cannot count differently under the lock without a writer that defeats the immutability trigger between the two reads; it is defence in depth, and the test names the proof's message, not the guard's. `land_small_block` now appends a second share after landing, so present-day ledger state differs from the block's anchored snapshot in every test that uses it. A read that rebuilt the window from current ledger state instead of the snapshot range passed all of them before; it now fails the snapshot digest. Two comments are corrected without a behaviour change: the negative block in `reward_manifest_header.rs` now mutates the restored window it claimed to, and the permit comment in `read_models.rs` says the permit also spans the snapshot lookup and window read that precede the blocking job. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016P2UWk2rh2R1W8yqTdLZ4k --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* feat(prism): record ledger advisory-lock and pool-acquire waits The native registry declares two database wait families, but nothing in the ledger observed either one: advisory-lock waits had no observations at all, and pool acquisition counted only the metrics collector's own. ORDER_LOCK contention, which serializes every share acknowledgement, was therefore invisible in production and in qualification. Give Ledger an optional metrics handle, supplied by the new connect_with_metrics; connect keeps its signature and passes None, so the tools and tests that connect without a registry are unchanged and record nothing. The coordinator passes its own handle. Route every advisory lock through the existing lock() helper, now timing the pg_advisory_xact_lock statement alone, and every ledger transaction through a new begin() helper that times pool acquisition alone. begin() is Pool::begin split in two around the timer, down to its sqlx::Error return type, so both helpers are drop-in replacements and every call site's `?` converts exactly as before. Leaving the BEGIN round trip outside the timer keeps the pool series meaning what the collector already records. Errors, including a lock_timeout expiry, propagate unchanged and record as failure. A drop guard records a wait abandoned by cancellation as a failure with its elapsed time, so a cancelled wait stays distinct from one that never happened. Completing the wait takes the handle out of the guard, so a wait is recorded exactly once. The metrics mutex is taken only after the wait has ended, never across an await, so rendering cannot queue behind a contended lock; a guard held across the await would also make the future !Send and fail to compile in the spawned share path. The CPFP funding lock has no LockKind value and stays unobserved; an unmapped key trips a debug assertion rather than borrowing another lock's label. Refs #328. * test(prism): cover ledger wait metrics by lock and outcome Two gated PostgreSQL tests and one source invariant. The first attaches a registry and drives each outcome against a real database: the migration lock from schema creation, a share append taking ORDER_LOCK once, a second session holding ORDER_LOCK past a 200 ms lock_timeout so the append fails with SQLSTATE 55P03 and records one failure of at least that duration, an append cancelled mid-wait recording its partial wait and no more, the settlement lock through save_job, five transactions each observing one pool acquisition plus the collector's own unchanged, a pool acquisition cancelled while every connection is held, and a closed pool recording an acquisition failure. The second runs the same sequence on a ledger with no handle and asserts that results and error codes, 55P03 included, are what they are at the base. It deliberately makes no assertion about a registry: one no ledger can reach has nothing to record into and could not fail the test. The third walks the ledger sources recursively and fails if a transaction begins outside the timed helper or a second pg_advisory_xact_lock statement appears, so a new call site, including one in a nested module directory, cannot silently stop being observed. Refs #328. * docs(prism): describe the recorded database wait metrics The two database wait families now say what they measure and who records them: pool acquisition excludes transaction BEGIN and the queries that follow, and advisory-lock timing covers one pg_advisory_xact_lock round trip for the migration, order and settlement locks. Both record a wait abandoned by cancellation as a failure, except the collector's own acquisitions, which keep reporting cancellation through collector status rather than inventing a completed wait. The follow-up ownership and alert-migration notes no longer describe either family as unpopulated, and record that PrismDatabasePoolWaitHigh now evaluates ledger acquisitions while its description still predates this change. No alert rule or digest input changes. Refs #328. * docs(prism): record the pool alert's collector-availability gate PrismDatabasePoolWaitHigh is gated on collector_available for the database collector. That collector acquires from the same pool as the ledger, so sustained pool exhaustion times out its own acquisition, zeroes the gauge and suppresses the rule exactly while the ledger histogram is recording the waits that should fire it. The gate cost nothing before this branch, because the family carried no samples during such an outage. It does now. The rule itself cannot change here: its file feeds the alert specification digest embedded in the deployment patch, which can only be regenerated from the frozen template. #336 tracks that work; this records the interaction where operators read it. Refs #328.
* docs: correct pool-acquire alert measurement boundaries Describe metrics collection and instrumented ledger transaction pool acquisition, retaining query/advisory-lock/read-gate exclusions and the five no-replacement dispositions. Regenerate migration tables and deployment patch from the verified frozen snapshot; all alert behavior is unchanged. * test(metrics): include approved unknown ledger outcome in fixture PR #333 intentionally added the ledger-outcome-unknown rejection label and a producer contract test. Add exactly that sample to the older exposition fixture from #344, retaining the byte-for-byte equality assertion and leaving runtime behavior unchanged. * feat(prism): record ledger advisory-lock and pool-acquire waits The native registry declares two database wait families, but nothing in the ledger observed either one: advisory-lock waits had no observations at all, and pool acquisition counted only the metrics collector's own. ORDER_LOCK contention, which serializes every share acknowledgement, was therefore invisible in production and in qualification. Give Ledger an optional metrics handle, supplied by the new connect_with_metrics; connect keeps its signature and passes None, so the tools and tests that connect without a registry are unchanged and record nothing. The coordinator passes its own handle. Route every advisory lock through the existing lock() helper, now timing the pg_advisory_xact_lock statement alone, and every ledger transaction through a new begin() helper that times pool acquisition alone. begin() is Pool::begin split in two around the timer, down to its sqlx::Error return type, so both helpers are drop-in replacements and every call site's `?` converts exactly as before. Leaving the BEGIN round trip outside the timer keeps the pool series meaning what the collector already records. Errors, including a lock_timeout expiry, propagate unchanged and record as failure. A drop guard records a wait abandoned by cancellation as a failure with its elapsed time, so a cancelled wait stays distinct from one that never happened. Completing the wait takes the handle out of the guard, so a wait is recorded exactly once. The metrics mutex is taken only after the wait has ended, never across an await, so rendering cannot queue behind a contended lock; a guard held across the await would also make the future !Send and fail to compile in the spawned share path. The CPFP funding lock has no LockKind value and stays unobserved; an unmapped key trips a debug assertion rather than borrowing another lock's label. Refs #328. * test(prism): cover ledger wait metrics by lock and outcome Two gated PostgreSQL tests and one source invariant. The first attaches a registry and drives each outcome against a real database: the migration lock from schema creation, a share append taking ORDER_LOCK once, a second session holding ORDER_LOCK past a 200 ms lock_timeout so the append fails with SQLSTATE 55P03 and records one failure of at least that duration, an append cancelled mid-wait recording its partial wait and no more, the settlement lock through save_job, five transactions each observing one pool acquisition plus the collector's own unchanged, a pool acquisition cancelled while every connection is held, and a closed pool recording an acquisition failure. The second runs the same sequence on a ledger with no handle and asserts that results and error codes, 55P03 included, are what they are at the base. It deliberately makes no assertion about a registry: one no ledger can reach has nothing to record into and could not fail the test. The third walks the ledger sources recursively and fails if a transaction begins outside the timed helper or a second pg_advisory_xact_lock statement appears, so a new call site, including one in a nested module directory, cannot silently stop being observed. Refs #328. * docs(prism): describe the recorded database wait metrics The two database wait families now say what they measure and who records them: pool acquisition excludes transaction BEGIN and the queries that follow, and advisory-lock timing covers one pg_advisory_xact_lock round trip for the migration, order and settlement locks. Both record a wait abandoned by cancellation as a failure, except the collector's own acquisitions, which keep reporting cancellation through collector status rather than inventing a completed wait. The follow-up ownership and alert-migration notes no longer describe either family as unpopulated, and record that PrismDatabasePoolWaitHigh now evaluates ledger acquisitions while its description still predates this change. No alert rule or digest input changes. Refs #328. * docs(prism): record the pool alert's collector-availability gate PrismDatabasePoolWaitHigh is gated on collector_available for the database collector. That collector acquires from the same pool as the ledger, so sustained pool exhaustion times out its own acquisition, zeroes the gauge and suppresses the rule exactly while the ledger histogram is recording the waits that should fire it. The gate cost nothing before this branch, because the family carried no samples during such an outage. It does now. The rule itself cannot change here: its file feeds the alert specification digest embedded in the deployment patch, which can only be regenerated from the frozen template. #336 tracks that work; this records the interaction where operators read it. Refs #328. * feat(prism): record ledger advisory-lock and pool-acquire waits The native registry declares two database wait families, but nothing in the ledger observed either one: advisory-lock waits had no observations at all, and pool acquisition counted only the metrics collector's own. ORDER_LOCK contention, which serializes every share acknowledgement, was therefore invisible in production and in qualification. Give Ledger an optional metrics handle, supplied by the new connect_with_metrics; connect keeps its signature and passes None, so the tools and tests that connect without a registry are unchanged and record nothing. The coordinator passes its own handle. Route every advisory lock through the existing lock() helper, now timing the pg_advisory_xact_lock statement alone, and every ledger transaction through a new begin() helper that times pool acquisition alone. begin() is Pool::begin split in two around the timer, down to its sqlx::Error return type, so both helpers are drop-in replacements and every call site's `?` converts exactly as before. Leaving the BEGIN round trip outside the timer keeps the pool series meaning what the collector already records. Errors, including a lock_timeout expiry, propagate unchanged and record as failure. A drop guard records a wait abandoned by cancellation as a failure with its elapsed time, so a cancelled wait stays distinct from one that never happened. Completing the wait takes the handle out of the guard, so a wait is recorded exactly once. The metrics mutex is taken only after the wait has ended, never across an await, so rendering cannot queue behind a contended lock; a guard held across the await would also make the future !Send and fail to compile in the spawned share path. The CPFP funding lock has no LockKind value and stays unobserved; an unmapped key trips a debug assertion rather than borrowing another lock's label. Refs #328. * test(prism): cover ledger wait metrics by lock and outcome Two gated PostgreSQL tests and one source invariant. The first attaches a registry and drives each outcome against a real database: the migration lock from schema creation, a share append taking ORDER_LOCK once, a second session holding ORDER_LOCK past a 200 ms lock_timeout so the append fails with SQLSTATE 55P03 and records one failure of at least that duration, an append cancelled mid-wait recording its partial wait and no more, the settlement lock through save_job, five transactions each observing one pool acquisition plus the collector's own unchanged, a pool acquisition cancelled while every connection is held, and a closed pool recording an acquisition failure. The second runs the same sequence on a ledger with no handle and asserts that results and error codes, 55P03 included, are what they are at the base. It deliberately makes no assertion about a registry: one no ledger can reach has nothing to record into and could not fail the test. The third walks the ledger sources recursively and fails if a transaction begins outside the timed helper or a second pg_advisory_xact_lock statement appears, so a new call site, including one in a nested module directory, cannot silently stop being observed. Refs #328. * docs(prism): describe the recorded database wait metrics The two database wait families now say what they measure and who records them: pool acquisition excludes transaction BEGIN and the queries that follow, and advisory-lock timing covers one pg_advisory_xact_lock round trip for the migration, order and settlement locks. Both record a wait abandoned by cancellation as a failure, except the collector's own acquisitions, which keep reporting cancellation through collector status rather than inventing a completed wait. The follow-up ownership and alert-migration notes no longer describe either family as unpopulated, and record that PrismDatabasePoolWaitHigh now evaluates ledger acquisitions while its description still predates this change. No alert rule or digest input changes. Refs #328. * docs(prism): record the pool alert's collector-availability gate PrismDatabasePoolWaitHigh is gated on collector_available for the database collector. That collector acquires from the same pool as the ledger, so sustained pool exhaustion times out its own acquisition, zeroes the gauge and suppresses the rule exactly while the ledger histogram is recording the waits that should fire it. The gate cost nothing before this branch, because the family carried no samples during such an outage. It does now. The rule itself cannot change here: its file feeds the alert specification digest embedded in the deployment patch, which can only be regenerated from the frozen template. #336 tracks that work; this records the interaction where operators read it. Refs #328. --------- Co-authored-by: Alex <alex@swaplabs.xyz>
* test: cover native ledger deadlines and execution safety Add 14 PostgreSQL regressions for late issued-job expiry, atomic publication, migration restart, incomplete snapshots, and claim-fenced execution. Use a test-only protocol observer to detect sensitive write replay after timeouts or lost acknowledgements, and document the native retry and snapshot contracts. * test: make candidate backoff eligibility deterministic Verify retry scheduling before controlling eligibility through fixture timestamps. Use future and past infinity to exercise claim fencing without racing the one-second retry deadline or waiting for it to elapse. Addresses the timing-race review comment on #355. * test: remove fanout backoff timing race and clarify locks Compare the fanout retry schedule with its durable write timestamp, then control claim eligibility through fixture timestamps. Document settlement and order locking and the scope of the sibling-row independence test. Addresses the two follow-up review comments on #355. * test: isolate ledger deadline and fanout eligibility checks Freeze all three completed fanouts before the global pre-deadline claim assertion. Use a dedicated fixture pool without competing server timeouts for the issued-job absolute-deadline regression, retaining bounded barriers and cleanup. Production behavior and other fixture defaults are unchanged. Addresses the two follow-up review comments on #355.
* docs: correct pool-acquire alert measurement boundaries Describe metrics collection and instrumented ledger transaction pool acquisition, retaining query/advisory-lock/read-gate exclusions and the five no-replacement dispositions. Regenerate migration tables and deployment patch from the verified frozen snapshot; all alert behavior is unchanged. * test(metrics): include approved unknown ledger outcome in fixture PR #333 intentionally added the ledger-outcome-unknown rejection label and a producer contract test. Add exactly that sample to the older exposition fixture from #344, retaining the byte-for-byte equality assertion and leaving runtime behavior unchanged. * fix: keep pool-wait alert active when database collector fails Remove only the collector-availability gate; preserve snapshot freshness, sample sufficiency, thresholds, duration and labels. Add bounded PromQL scenarios with per-instance assertions, recovery and the old-expression negative control. Refs #336. Landing depends on #334 and reconciliation with #343 and #345. * feat(prism): record ledger advisory-lock and pool-acquire waits The native registry declares two database wait families, but nothing in the ledger observed either one: advisory-lock waits had no observations at all, and pool acquisition counted only the metrics collector's own. ORDER_LOCK contention, which serializes every share acknowledgement, was therefore invisible in production and in qualification. Give Ledger an optional metrics handle, supplied by the new connect_with_metrics; connect keeps its signature and passes None, so the tools and tests that connect without a registry are unchanged and record nothing. The coordinator passes its own handle. Route every advisory lock through the existing lock() helper, now timing the pg_advisory_xact_lock statement alone, and every ledger transaction through a new begin() helper that times pool acquisition alone. begin() is Pool::begin split in two around the timer, down to its sqlx::Error return type, so both helpers are drop-in replacements and every call site's `?` converts exactly as before. Leaving the BEGIN round trip outside the timer keeps the pool series meaning what the collector already records. Errors, including a lock_timeout expiry, propagate unchanged and record as failure. A drop guard records a wait abandoned by cancellation as a failure with its elapsed time, so a cancelled wait stays distinct from one that never happened. Completing the wait takes the handle out of the guard, so a wait is recorded exactly once. The metrics mutex is taken only after the wait has ended, never across an await, so rendering cannot queue behind a contended lock; a guard held across the await would also make the future !Send and fail to compile in the spawned share path. The CPFP funding lock has no LockKind value and stays unobserved; an unmapped key trips a debug assertion rather than borrowing another lock's label. Refs #328. * test(prism): cover ledger wait metrics by lock and outcome Two gated PostgreSQL tests and one source invariant. The first attaches a registry and drives each outcome against a real database: the migration lock from schema creation, a share append taking ORDER_LOCK once, a second session holding ORDER_LOCK past a 200 ms lock_timeout so the append fails with SQLSTATE 55P03 and records one failure of at least that duration, an append cancelled mid-wait recording its partial wait and no more, the settlement lock through save_job, five transactions each observing one pool acquisition plus the collector's own unchanged, a pool acquisition cancelled while every connection is held, and a closed pool recording an acquisition failure. The second runs the same sequence on a ledger with no handle and asserts that results and error codes, 55P03 included, are what they are at the base. It deliberately makes no assertion about a registry: one no ledger can reach has nothing to record into and could not fail the test. The third walks the ledger sources recursively and fails if a transaction begins outside the timed helper or a second pg_advisory_xact_lock statement appears, so a new call site, including one in a nested module directory, cannot silently stop being observed. Refs #328. * docs(prism): describe the recorded database wait metrics The two database wait families now say what they measure and who records them: pool acquisition excludes transaction BEGIN and the queries that follow, and advisory-lock timing covers one pg_advisory_xact_lock round trip for the migration, order and settlement locks. Both record a wait abandoned by cancellation as a failure, except the collector's own acquisitions, which keep reporting cancellation through collector status rather than inventing a completed wait. The follow-up ownership and alert-migration notes no longer describe either family as unpopulated, and record that PrismDatabasePoolWaitHigh now evaluates ledger acquisitions while its description still predates this change. No alert rule or digest input changes. Refs #328. * docs(prism): record the pool alert's collector-availability gate PrismDatabasePoolWaitHigh is gated on collector_available for the database collector. That collector acquires from the same pool as the ledger, so sustained pool exhaustion times out its own acquisition, zeroes the gauge and suppresses the rule exactly while the ledger histogram is recording the waits that should fire it. The gate cost nothing before this branch, because the family carried no samples during such an outage. It does now. The rule itself cannot change here: its file feeds the alert specification digest embedded in the deployment patch, which can only be regenerated from the frozen template. #336 tracks that work; this records the interaction where operators read it. Refs #328. * docs: distinguish pool-wait guard from database gauge guards * feat(prism): record ledger advisory-lock and pool-acquire waits The native registry declares two database wait families, but nothing in the ledger observed either one: advisory-lock waits had no observations at all, and pool acquisition counted only the metrics collector's own. ORDER_LOCK contention, which serializes every share acknowledgement, was therefore invisible in production and in qualification. Give Ledger an optional metrics handle, supplied by the new connect_with_metrics; connect keeps its signature and passes None, so the tools and tests that connect without a registry are unchanged and record nothing. The coordinator passes its own handle. Route every advisory lock through the existing lock() helper, now timing the pg_advisory_xact_lock statement alone, and every ledger transaction through a new begin() helper that times pool acquisition alone. begin() is Pool::begin split in two around the timer, down to its sqlx::Error return type, so both helpers are drop-in replacements and every call site's `?` converts exactly as before. Leaving the BEGIN round trip outside the timer keeps the pool series meaning what the collector already records. Errors, including a lock_timeout expiry, propagate unchanged and record as failure. A drop guard records a wait abandoned by cancellation as a failure with its elapsed time, so a cancelled wait stays distinct from one that never happened. Completing the wait takes the handle out of the guard, so a wait is recorded exactly once. The metrics mutex is taken only after the wait has ended, never across an await, so rendering cannot queue behind a contended lock; a guard held across the await would also make the future !Send and fail to compile in the spawned share path. The CPFP funding lock has no LockKind value and stays unobserved; an unmapped key trips a debug assertion rather than borrowing another lock's label. Refs #328. * test(prism): cover ledger wait metrics by lock and outcome Two gated PostgreSQL tests and one source invariant. The first attaches a registry and drives each outcome against a real database: the migration lock from schema creation, a share append taking ORDER_LOCK once, a second session holding ORDER_LOCK past a 200 ms lock_timeout so the append fails with SQLSTATE 55P03 and records one failure of at least that duration, an append cancelled mid-wait recording its partial wait and no more, the settlement lock through save_job, five transactions each observing one pool acquisition plus the collector's own unchanged, a pool acquisition cancelled while every connection is held, and a closed pool recording an acquisition failure. The second runs the same sequence on a ledger with no handle and asserts that results and error codes, 55P03 included, are what they are at the base. It deliberately makes no assertion about a registry: one no ledger can reach has nothing to record into and could not fail the test. The third walks the ledger sources recursively and fails if a transaction begins outside the timed helper or a second pg_advisory_xact_lock statement appears, so a new call site, including one in a nested module directory, cannot silently stop being observed. Refs #328. * docs(prism): describe the recorded database wait metrics The two database wait families now say what they measure and who records them: pool acquisition excludes transaction BEGIN and the queries that follow, and advisory-lock timing covers one pg_advisory_xact_lock round trip for the migration, order and settlement locks. Both record a wait abandoned by cancellation as a failure, except the collector's own acquisitions, which keep reporting cancellation through collector status rather than inventing a completed wait. The follow-up ownership and alert-migration notes no longer describe either family as unpopulated, and record that PrismDatabasePoolWaitHigh now evaluates ledger acquisitions while its description still predates this change. No alert rule or digest input changes. Refs #328. * docs(prism): record the pool alert's collector-availability gate PrismDatabasePoolWaitHigh is gated on collector_available for the database collector. That collector acquires from the same pool as the ledger, so sustained pool exhaustion times out its own acquisition, zeroes the gauge and suppresses the rule exactly while the ledger histogram is recording the waits that should fire it. The gate cost nothing before this branch, because the family carried no samples during such an outage. It does now. The rule itself cannot change here: its file feeds the alert specification digest embedded in the deployment patch, which can only be regenerated from the frozen template. #336 tracks that work; this records the interaction where operators read it. Refs #328. * Expose pool acquisition waits independently of health publication --------- Co-authored-by: Alex <alex@swaplabs.xyz>
* docs: add PRISM fatal-state recovery runbook (#290) Document the operator procedure for a shared fatal state: evidence first, graceful stop of every frontend with stale-versus-stopped instance guidance, separately reviewed reconciliation before clearing, the audited fatal-state clear command, and restart verification. Includes read-only, bounded SELECT examples only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013RXEBLAaU5LEGU9Px59q19 * Add transactional PRISM fatal-state recovery and regression tests (#290) --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* v3.0.0 fix: harden native PRISM configuration and containers * fix: align heartbeat freshness with health cadence * fix: decouple job pruning from health publication cadence
…ns, gate startup (#321) * 3.x.x: pin the migration source schema and gate startup on it The migrator accepted any database that had a share ledger, tested the outbox with a JSONB predicate that a #258 v2 row slips past, never read qbit_prism_schema_capabilities, and skipped the schema version entirely when PRISM_POSTGRES_INIT_SCHEMA=0, the native default. Its "frozen 2.x.x schema" in the upgrade tests was include_str! of the live 001, which #244 already edits. Freeze the 2.x.x release SQL as byte-exact fixtures taken from the release commits (001 from v2.0.2, identical at v2.0.1 and v2.0.0; 002 from v2.0.2), pin their digests, pin the live 001's digest, and compare the live 001 to the release statement for statement, since #244 changed only its comments. Editing either live file fails the test with the instruction the editor needs. Migration 006 pins the accepted source states as data: fresh, pre-#258 (001 only), #258 applied (candidate_storage_version = 2 and every 002 object), partial 002 (refused, naming the missing object) and newer (refused before any DDL). The inventory comes from the catalog, so the drain predicate is built only from outbox columns that exist: the storage_version and body_id clauses on a #258 source, the v1 form on a pre-#258 one, where a fixed predicate errored instead of refusing. The check reads rows, never the capability row, which only proves 002 ran. The refusal names the blocking rows and the 2.x.x drain: the pinned coordinator's submitter, or lab.prism.recover_pending_blocks from the 2.x.x image. Every start now reads qbit_prism_schema_migrations and the capability rows, with or without initialize, and refuses a schema other than version 6 or a capability the binary does not understand, naming the required version. A successful migration records the accepted source state, its 2.x.x release and commit, the declared capability value and the prior schema version in qbit_prism_migration_source; migrate prints it, every start logs it, and a repeated migrate never rewrites it. 006 also declares candidate_storage_version = 1 on sources that had no capability row and adds the outbox storage_version column so one claim statement serves every source; the chunk tables are not carried. The API database test hand-applies migrations up to 003 and then connected without initialize, which the gate now refuses; it initializes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AauTYgQEqxzDE13xLjYTR9 * 3.x.x: park candidates the claim lane cannot decode The claim lane returned the outbox row's candidate column alone. A #258 v2 row returns NULL with the claim token set, no CandidateClaim can be built, the submit loop logs "candidate polling failed", and the row re-enters the due lane at every lease expiry, forever. Return storage_version and an Option<Value> candidate. A version 1 row with a body is decoded as before. Any other row, a v2 body this server does not import or a version it does not know, is parked inside the claim transaction: the claim is released, last_error says why, and next_attempt_at becomes infinity, so no lease expiry offers it again and ordinary v1 work keeps flowing. The row stays pending with its body untouched; resetting next_attempt_at re-offers it, which is how an operator hands it back after draining it with the 2.x.x image. The edit is deliberately small, since #265 reshapes candidate storage. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AauTYgQEqxzDE13xLjYTR9 * Document the supported 2.x.x source schemas and the drain requirement The migration guide names the minimum supported 2.x.x release (v2.0.1, whose offline recovery command is part of the drain; v2.0.0 has the same schema), the newest (v2.0.2), #258's rollback floor, the source-state table the migrator pins, the drain requirement for v1 and v2 rows with the 2.x.x commands, what qbit_prism_migration_source records, the startup gate, and how a parked row is found and re-offered. The server README points at it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AauTYgQEqxzDE13xLjYTR9 * 3.x.x: accept a newer schema at start and let capabilities fence formats The startup gate refused any schema version other than the one this binary migrates to. That silently withdrew the rollout the guide promises, where compatible Rust releases drain and replace one frontend at a time, because a release that adds a migration would have kept every frontend on the previous release from starting once its migrate ran. Refuse only a schema below the required version, as the issue asks. A newer schema is accepted with a warning that names both versions: native migrations are additive, and a release whose format an older binary must not touch declares a capability, which require_known_capabilities still refuses exactly as before. The guide's rollout paragraph is restored with the order that makes it hold: the new release's migrate runs first, frontends on the previous release keep starting while its migrations are additive and it declares no capability they do not understand, and a release that declares a new capability needs every frontend stopped first. The startup-gate test now asserts that a start on the next schema version succeeds and rewrites nothing. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AauTYgQEqxzDE13xLjYTR9 * Record the migrator and schema freeze in the 3.x.x development-line note The note listed how 002_candidate_bodies.sql is handled as frozen by a pending issue. It is now carried byte-identical to v2.0.2 and digest-pinned, never applied, and a #258 source is accepted after the drain check while a partial or newer source is refused before any DDL. The matching "not done yet" bullet is gone. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AauTYgQEqxzDE13xLjYTR9 * 3.x.x: verify the frozen 001 schema after it has run, refusing drift transactionally A database with no 002 objects was accepted as the pre-#258 source on the strength of qbit_share_ledger existing. 001 is idempotent, but its IF NOT EXISTS leaves an existing table, column, index or named constraint as it is, so an older, partially applied or locally modified schema passed the pin, was recorded as v2.0.1, and its incompatible definitions survived a successful migration. The migrator now applies 001 to the source as before and then, before 002_multi_instance.sql or any later native DDL touches those tables, checks that what 001 left alone is the release definition. The expected definitions are not a stored fingerprint: under a savepoint in the migration transaction it creates a scratch schema, applies the same 001 there (plus 002_candidate_bodies.sql for a #258 source), reads every table, column, constraint, index, trigger and function that produced, and rolls the savepoint back, so the comparison is exact for the running PostgreSQL version and nothing from the scratch apply survives. Each object then needs an equivalent in the schema the migrator runs in: column type, NOT NULL, default, identity, generated status and collation; constraints per table by definition; index definitions and validity; trigger definitions and enabled state; function arguments, result, language, body, volatility and settings. Schema qualification, column order, comments, auto-generated constraint names and the NOT VALID mark 001 itself leaves on an upgraded table are ignored; extra objects are kept and logged; a missing or different one fails the migration, which rolls back whole, naming each object and what differs. A migrate role without CREATE on the database is refused with the grant it needs. The source-state table gains the "drifted 001" row in SOURCE_STATES, migration 006's comment and the migration guide; the migration record is unchanged. Tests: a pre-#258 source with a dropped foreign key, a dropped NOT NULL, a changed column type and a same-named index with a different definition is refused naming all four and left untouched; a #258 source with a replaced 002 function body and a disabled 002 trigger is refused; an extra table, column and index, a 001 column moved to a different physical position with its data, and a NOT VALID credit-policy CHECK still migrate; a role without CREATE on the database is told the grant. Replays of the real 0.1.2 -> 1.x -> 2.x history (every 001 revision in order, then the frozen v2.0.2 file, with and without 002) all migrate. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EVxRdWsnzBhrsgiEX2kvKR * 3.x.x: register the source-schema gated tests in the gate manifest The branch was rebased onto #322, which fails the prism-native-postgres job for any gated test that executes without being listed in test/prism-gated-tests.txt. None of the ten Postgres tests this PR adds in tests/support/ledger_2x.rs were listed, so the job would have failed on its manifest check. List them, sorted, and update the count the gate guide quotes. The other #322 checks (formatting, clippy with -D warnings, the gate env-read lint, the toolchain pin, version skew) already pass on this branch and need no change. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019mWgeEnsyHLZSiGaAj8bZi * 3.x.x: compare release sequences in the source-schema check The post-001 fingerprint read tables, columns, constraints, indexes, triggers and functions, but never the sequences behind 001's bigserial columns or its explicit qbit_audit_publication_sequence_seq. CREATE TABLE IF NOT EXISTS leaves an existing sequence exactly as it is, so a lowered MAXVALUE or a changed INCREMENT on qbit_share_ledger_share_seq_seq survived the idempotent apply, compared equal to the frozen release, and migrated; the next share inserts would then fail or take unexpected identifiers. The fingerprint now records every sequence of the schema from pg_sequence: data type, start, increment, minimum, maximum, cache and cycle. The value a sequence has reached (last_value, is_called) is the source's data and is not read, so an advanced ledger still migrates and keeps its position. A release sequence that is missing or whose structure differs is drift, named in the refusal the way columns and functions are, for example sequence qbit_share_ledger_share_seq_seq differs: maximum expected 9223372036854775807, found 1000000 and an extra sequence is logged with the other extras. Tests: the pre-#258 drift test also lowers the share sequence's maximum and doubles the payout entry sequence's increment, and the refusal names both and rolls back; the tolerated-differences test advances the share sequence with setval and the first native share takes the next value; a unit test covers the comparison. The real 0.1.2 -> 1.x -> 2.x histories (every 001 revision in order, then the frozen v2.0.2 001, with and without 002) all still migrate, and each leaves the same sequence structure as a fresh apply. The migration guide and the 006 comment list what is compared. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019mWgeEnsyHLZSiGaAj8bZi * 3.x.x: treat a database as fresh only when no release object exists A database without qbit_share_ledger and without any 002 object was classified as fresh on that evidence alone, and the post-001 comparison was skipped for it. A selective restore or a hand-installed piece of the schema, qbit_pool_blocks on its own for instance, therefore passed: 001's CREATE ... IF NOT EXISTS kept whatever those objects held, the migration recorded success, and the accounting code ran over a schema nobody had checked. The release definitions are now taken once, before any DDL touches the source: the frozen release SQL (001, plus 002 for a #258 source) is applied to a scratch schema under a savepoint and rolled back, and that one fingerprint serves both checks. A database without qbit_share_ledger is fresh only if it has no table, sequence, index, trigger or function the release 001 creates. Otherwise it is the new source state "partial 001", refused before any DDL and naming the objects present, capped as the drift report is: refusing to migrate a partial 001 source before any DDL: the database has no qbit_share_ledger but holds 4 object(s) that the 2.x.x release's 001_share_ledger.sql creates (table qbit_pool_blocks; index ...), so it is neither an empty database nor a 2.x.x ledger, and 001's IF NOT EXISTS would keep those objects whatever they hold. Nothing was changed. Restore the full pre-migration backup, or migrate into an empty database Objects the release does not create, an operator's own table, do not disqualify a fresh database and are logged as extras. The post-001 comparison now runs for every source state, fresh included, where it passes trivially: 001 has just created everything. SOURCE_STATES gains the partial 001 row after fresh. The rows are cited by name through constants and looked up by name, never by position, and a unit test checks every cited name is exactly one row, so reordering the table cannot silently change a refusal again. The pinned-data test, the 006 comment, the guide's table and the development-line note carry the new row. Tests (Postgres): a leftover 001 table, sequence or function alone, each created from the frozen release definition, is refused as partial 001 naming it, with no native table, no scratch schema and the leftover untouched; an empty database with only an operator's table still migrates as fresh and keeps the table; a truly empty database migrates as fresh. The real 0.1.2 -> 1.x -> 2.x history replays, with and without 002, still migrate. Both new gated tests are in the manifest. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019mWgeEnsyHLZSiGaAj8bZi * 3.x.x: recheck the outbox before 006 on a native schema an earlier build migrated A database an earlier 3.x.x build brought to native schema 3, 4 or 5 went straight to 004, 005 and 006 without the drain check, and that build's v1-only predicate never counted a v2 row (`candidate ?& ...` is NULL for a NULL body). A pending v2 candidate could therefore be carried into version 6 and parked by the claim lane instead of refusing the cutover, leaving a possibly valid found block unsubmitted. `migrate_schema` now runs the column-aware inventory and `refuse_undrained_outbox` on a recorded version 3, 4 or 5 before 004, 005 and 006 run, so a refusal on that path is before any DDL as well. Native pending rows carry the native payout_revision, bundle and block_hash fields and are not counted; only a v2 body, a body_id, a storage_version other than 1, or a v1 body without those fields is refused. The refusal has its own wording: it names the native schema version, names the rows the same way as the 2.x.x refusal, and gives the remedy the guide backs, restoring the pre-migration 2.x.x backup and draining there with the pinned image. The 2.x.x release is not supported against a native schema (its revert script refuses one), so the message says not to point it at the database. Postgres tests build the earlier build's schema 5 faithfully: the frozen 001 (and 002), the current migration, then 006 undone, deleting the version 6 row and dropping qbit_prism_migration_source, and on a pre-#258 source also the storage_version column and the capability table 006 created. A pending v2 row on the #258 simulation is refused naming the row, with max(version) still 5 and no 006 object; a native pending candidate on either simulation migrates to 6 and is still claimable; an undrained 2.x.x v1 row on the pre-#258 simulation exercises the v1-only form of the predicate. The guide's drain text and the 006 comment describe the check; the gated-test manifest and its documented count are updated. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QuEYH8xHQRo7C4SnexM3cP * 3.x.x: compare table and sequence persistence in the source-schema check The release-schema fingerprint recorded a table by name and compared its columns, constraints and indexes, so a release table an operator had changed to UNLOGGED compared equal to the logged table the scratch apply creates, and the migration accepted a ledger or outbox whose rows PostgreSQL truncates after a crash. The fingerprint now records pg_class.relpersistence for every table and every sequence (PostgreSQL 15 and later have unlogged sequences, and a sequence owned by an unlogged table is unlogged with it). A mismatch is drift, named the way the other differences are: `table qbit_share_ledger differs: expected logged, found UNLOGGED`, and for a sequence the persistence joins its other properties. Extra objects are unaffected. A Postgres test makes qbit_payout_carry_forward UNLOGGED on a pre-#258 source: it holds the carried balances and has no foreign key in either direction, so SET UNLOGGED is accepted as it is, and the sequence behind its bigserial column follows the table. The migration is refused naming the table and its sequence, rolls back with the relations still unlogged and no native table, and accepts the same source once it is logged again. (qbit_audit_publication_sequence_seq is not used for this: 001's own guard already refuses that sequence when it is not logged.) A unit test covers the comparison for tables, sequences and a temporary table. Replays of the real 0.1.2 -> 1.x -> 2.x schema history, with and without 002, still migrate: no revision creates an unlogged relation. The guide's release-schema paragraph, its drifted 001 row and the comment in migration 006 name persistence; the gated-test manifest and its count are updated. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QuEYH8xHQRo7C4SnexM3cP * 3.x.x: compare constraint validation, exempting only the release's own NOT VALID check The source-schema check stripped NOT VALID from every constraint definition before comparing, so a foreign key or CHECK an operator had dropped and re-added NOT VALID compared equal to the validated release constraint, and a source whose rows were never checked against it, orphaned outbox rows or invalid accounting rows for instance, still migrated. The blanket strip is replaced by a pinned exemption list, NOT_VALID_EXEMPT: the constraints the frozen release 001 itself adds NOT VALID on an upgraded table and never validates. Today that is qbit_share_ledger_credit_policy_check alone (001 adds it NOT VALID to a qbit_share_ledger upgraded from before the column existed; a fresh apply creates it validated inside CREATE TABLE). An exempt constraint is accepted in either state on either side. Every other constraint is compared by pg_constraint.convalidated as well as by definition: a release constraint that is NOT VALID in the source is drift, `constraint qbit_block_candidate_outbox_share_id_fkey on qbit_block_candidate_outbox is NOT VALID; the release validates it`. The definition without its NOT VALID suffix is still the lookup key, so an unvalidated constraint finds its release counterpart and is named. A test parses the frozen 001 and 002 fixtures for constraints added NOT VALID and not validated later in the same file (plain statements and the format() form alike) and asserts the set equals NOT_VALID_EXEMPT, so the list cannot drift from the release. A Postgres test drops and re-adds the outbox share_id foreign key and a carry-forward CHECK NOT VALID: the migration is refused naming both, rolls back with nothing validated, and accepts the source once they are validated. The existing tolerated- differences case, with the credit-policy check NOT VALID, still migrates. Unit tests cover the comparison and the exemption. Replays of the real 0.1.2 -> 1.x -> 2.x schema history, with and without 002, still migrate: the only constraint a genuine history leaves NOT VALID is the exempt one. The guide's release-schema paragraph, its drifted 001 row and the comment in migration 006 describe the rule; the gated-test manifest and its count are updated. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QuEYH8xHQRo7C4SnexM3cP * 3.x.x: require every schema migration at start, 009 included and may land after 009, and the rebase moved its 009 step into migrate_schema with the rest of the chain. The startup gate still compared max(version) with 6, which a database at 2, 3, 4, 5 and 9 passes without 006, and its "newer than required" warning would have fired on every start once 009 was in the chain. REQUIRED_SCHEMA_VERSIONS names 2, 3, 4, 5, 6 and 9. A start refuses a database missing any of them, naming the gap, and warns only about migrations this binary does not know. The pre-006 outbox refusal names the applied set instead of a high-water mark; prior_schema_version in the source record stays the highest migration recorded. Tests: a database at 2, 3, 4, 5 and 9 applies 006 on the next migrate and leaves 009 as it found it; a start without PRISM_POSTGRES_INIT_SCHEMA on that database is refused naming 6; the newest-migration case now undoes 009. #319's read-only pre-009 connect test is removed, since the gate refuses that database by design, and the gate manifest follows. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * 3.x.x: refuse unknown capabilities before 004 to 006 run on a native schema On a database an earlier 3.x.x build migrated to native schema 3, 4 or 5, migrate_schema ran the drain check and then applied 004, 005 and 006 without reading the capability rows, so a database a newer release wrote (candidate_storage_version = 3, or a capability this binary does not know) was altered and recorded at 6, and only Ledger::connect refused it, after the commit. Every native path now takes the inventory first and refuses those rows before any DDL, on a database already at 6 too, before 009; the 006 record reuses the same rows instead of reading them again, and the recorded versions are read in order so the refusal names them deterministically. A pre-006 test on a #258 source covers both refusals and the migration once the row is gone; the migration guide and the 006 comment describe the order. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * 3.x.x: compare row-level security and policies in the source-schema check The release-schema fingerprint read a table's name, persistence and columns only, so a release table with row-level security enabled or forced, or a policy on it, compared equal to the frozen release. For a migrate role that does not bypass row-level security, a policy hiding pending rows (or forced security with no policy) makes the drain check see an empty outbox, so the migration commits with legacy candidates stranded where the native claim lane never sees them. The fingerprint now reads relrowsecurity and relforcerowsecurity from pg_class and every policy from pg_policy (command, permissive or restrictive, roles, USING and WITH CHECK, rendered without schema qualification), both for the scratch release apply and the source, and the comparison refuses a release table whose flags differ or that carries a policy the release does not have, naming the table, the flags found and each policy. Policies on tables the release does not create are not reported. A Postgres test migrates as a non-superuser owner role and shows the row hidden, the refusal, the rollback, and the migration once security is off and the row drained; the docs and the 006 header say what is compared and why. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Quote the pre-006 outbox refusal as the migrator words it since 009 joined the chain The guide's example still read "a native schema 5 database", the wording from before the applied migrations were tracked as a set; the migrator names the recorded set ("a native database at schema migrations 2, 3, 4, 5, 9") since the startup gate stopped comparing a high-water mark. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * 3.x.x: refuse a 2.x.x or empty source that already holds a native object or column Codex P1 on #321: a source that already contains an object a native migration creates (a qbit_prism_cluster(singleton boolean PRIMARY KEY), for instance) passed the fresh and release checks as a harmless extra. 002's CREATE TABLE IF NOT EXISTS then kept it as it was, the migration recorded its versions, and only after commit did writable() fail on the missing fatal_error column, with every later migrate skipping the recorded version. A column a native migration adds with ADD COLUMN IF NOT EXISTS (002's claim columns on the outbox and the fanout artifacts, 003's canonical_audit_bytes and inactive_since, 006's storage_version on a pre-#258 outbox) is the same bug: a source release table that already has a column of that name keeps it whatever its type or default. The scratch apply release_fingerprint runs now continues, after the release SQL, with the native migrations in the order migrate_schema applies them (NATIVE_MIGRATIONS, pinned to REQUIRED_SCHEMA_VERSIONS by a unit test) and fingerprints the scratch schema again. The second reading minus the first is the reserved set (ReservedObjects): every table, sequence, index, trigger and function a native migration creates and the release does not, and, per release table, every column the native chain adds to it. qbit_prism_schema_migrations, the migrator's own, is never in it, and an object or column in both readings (the capability table on a #258 source, the outbox's storage_version there, which the release 002 added; on a pre-#258 source 006 adds it, so it is reserved) stays governed by the release checks. Reserved columns are kept in their own field: objects_present also serves the partial-001 check, where a present table already refuses and listing its columns would be noise, so only require_no_native_collision looks at them. On the 2.x.x path, before any DDL, the source is fingerprinted once, require_fresh_source (now synchronous, on that reading) runs on a fresh source, and a source that already has a reserved object or column is refused as the new "native collision" state, naming each (capped by named_objects; a column as column <table>.<column>), saying nothing was changed, and telling the operator to restore the backup or check and remove the objects; nothing is dropped and no DROP is suggested. release_objects_present is renamed objects_present. The migrator applies each native migration from NATIVE_MIGRATIONS, so the scratch apply and the real one cannot drift apart. SOURCE_STATES, its pinned-table test, the comment in 006 and the guide's table carry the new row; the guide shows the refusal and names the column case. Tests, against PostgreSQL 16: an empty database with a stray qbit_prism_cluster is refused naming it, with no migrator table and the stray table unaltered, and migrates as fresh once the table is removed; a pre-#258 and a #258 source with a stray native table, function and sequence are refused naming all three, unchanged, and migrate once they are removed; a pre-#258 and a #258 source with a stray claim_token integer on the outbox are refused naming column qbit_block_candidate_outbox.claim_token, unchanged (the column keeps its type), and migrate once it is dropped, after which 002 adds the text column it wanted. The unit test on the reserved set pins the column derivation, including storage_version reserved on a pre-#258 outbox and not on a #258 one. The api_database fixture, which hand-applies 001 to 003 and then lets connect bring the schema forward, now records versions 2 and 3 the way the migrator does: native objects with no record are the collision this commit refuses. The genuine 0.1.2 -> 1.x -> 2.x history replays (the 12 distinct 001 revisions reachable from the release tags, main, 2.x.x and 3.x.x, applied in order into one database and each on its own, then the frozen v2.0.2 001, with and without 002, then migrate: 26 runs) are all accepted, so no real history contains a native object or column. The new Postgres tests are registered in test/prism-gated-tests.txt, and the "N today" count in docs/prism-integration-test-gate.md, which had fallen behind the manifest at an earlier rebase, is set to the manifest's id count, 114. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * 3.x.x: fold migration 008 into the native chain after rebasing on 3.x.x #327 added 008_prepared_window_reference.sql to the runner in connect.rs, which this branch had moved into migrate_schema. Put 008 in NATIVE_MIGRATIONS, so the scratch apply reserves what it creates, and in REQUIRED_SCHEMA_VERSIONS, so the startup gate requires it; apply it after 006 and before 009. 007 is the only reserved number now. The pre-006 fixtures the tests undo from the current chain therefore model the database today's 3.x.x leaves, at 2, 3, 4, 5, 8 and 9, and the messages and docs name that set. * 3.x.x: refuse a native record with 3 and not 2 before any DDL Every native build has recorded 2 and 3 in the one transaction that applied them, so a record with 3 and not 2 was edited or restored selectively. migrate_schema's native path applied the rest of the chain above it and only the startup gate then refused the gap, at every start, and no later migrate could repair it: 002 is not re-run on a database at 3, and recording it unseen would vouch for objects the run never checked. Refuse the record before any DDL, naming the remedy; a start without initialize keeps refusing the gap. * 3.x.x: require the capability declaration once migration 006 is recorded require_known_capabilities accepted a database at 6 whose capability table or candidate_storage_version row was gone, as if absence were the legacy state it was before 006; after a selective restore or a deletion the database could no longer say which release wrote it and the connect gate let it through. 006 created both and nothing native removes them, so require them at connect once require_schema_version has established 6, naming the remedy, and refuse the same database on the migrate path before 008 or 009 run above the missing declaration. * 3.x.x: compare the enforcement triggers of release constraints The trigger reading skipped every internal trigger and the constraint comparison looked at the definition and convalidated only, so a release foreign key whose referential triggers a superuser had disabled (ALTER TABLE ... DISABLE TRIGGER ALL during a bulk load, say) compared equal: no new row was checked against it, 001 never re-enables a trigger it did not create, and the native ledger could go on accumulating orphaned rows. Read the enabled state of each internal trigger that enforces a constraint into its definition, on both tables of a foreign key, and refuse a release constraint whose states differ from the scratch apply, naming them. * 3.x.x: refuse inheritance involving a release table The table reading never looked at pg_inherits, so a child created with INHERITS (qbit_share_ledger) was an extra ordinary table, accepted with a warning, while PostgreSQL includes its rows in every query of the ledger, the unqualified share reads included, without the release constraints ever checking them. Read each table's parents and children into its definition and refuse a release table that has either, naming them; a release table attached as a partition of or inheriting from another table is drift the same way, and inheritance among the operator's own tables stays theirs. * 3.x.x: count a reserved name held by a relation of any kind as a collision The index reading left every index backing a primary, unique or exclusion constraint to the constraint comparison, and the collision check looked up reserved index names in that reading only, so an operator table whose unique constraint was named qbit_prism_candidate_claim_idx owned that name unseen: 002's CREATE INDEX IF NOT EXISTS would skip the outbox index it meant to create, record the migration anyway, and leave candidate polling to scan the outbox. IF NOT EXISTS looks at the name alone, for a view or a composite type as much as for such an index, so read every relation of the kinds no other map models into the fingerprint, reserve the native ones the release lacks, and count a reserved table, sequence or index name held by any of them, naming what holds it. An index backing a constraint of an expected table goes with that table and is not named twice. * 3.x.x: refuse extra user triggers on release tables The release-schema check classified a user trigger the release does not create on a release table as a tolerated extra, so require_release_schema logged it and migrated over it. Such a trigger can be valid for the legacy writer and incompatible with the native one: a guard that rejects writer_epoch = 0 lets the 2.x.x writer's leased epochs through and fails every native share insert, which writes epoch 0, and a trigger can just as well rewrite what the migrator and the native writers put in the accounting tables. Report a found user trigger on a release table whose (table, name) the release lacks as drift, naming the trigger, its table and its definition, and whether the release has no trigger on that table or simply does not create this one, so the migration refuses transactionally and the database is unchanged. Triggers on tables the release does not create stay the operator's own; the expected-trigger definition and enabled-state comparison and the enforcement-trigger check of release constraints are unchanged. Regression: a comparator unit test for both reasons, a missing release trigger and an operator table's trigger; and a two_x PostgreSQL test on the frozen pre-#258 and #258 sources that shows the guard accepting a legacy epoch and refusing epoch 0, the refusal naming the trigger and rolling back with the trigger, the legacy rows and no native table or version in place, then the same source migrating once the operator drops the guard, with the operator table's trigger kept and firing and the native writers inserting their epoch-0 shares. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N2if7arQVKaZ1yMupJiprr * fix: reject extra constraints on migration source tables * fix: reject unsatisfied extra migration source columns * fix: reject extra indexes that affect native writes * fix: update rustls to address RUSTSEC-2026-0285 * fix: reject executable expressions on extra source columns * fix: fingerprint table rewrite rules before migration * fix: reject incoming foreign keys from extra source tables * fix: validate storage version before native migration 006 * fix: reject domain-typed extra migration columns * fix: reject migration sources resolved outside the current schema * fix: reject all extra indexes on release tables * fix: validate source metadata before later native migrations * fix: reject preexisting source metadata before migration 006 * fix: reject row security on capability metadata * fix: separate legacy source versions from native capabilities * fix: reject outbox row security before native migration drain * fix: validate migration history before trusting version records * fix: require an ordinary table for capability declarations * fix: bind the capability declaration to the current schema The connect-time gate probed qbit_prism_schema_capabilities with an unqualified to_regclass, which follows search_path. When the migrated schema had lost its capability table, the probe could resolve a later schema's table, and its version-1 row cleared a database whose own declaration was gone, while require_migration_history had bound the history to current_schema(). Migrate was already safe through require_source_schema_resolution; connect was not. read_capabilities now looks the visible relation up in the catalog and refuses one outside current_schema(), naming both schemas, before any row is read. Both callers share the path: the connect gate drops its separate probe and the migrate inventory reads through the same function. A new gated test drops the migrated schema's declaration, declares version 1 in a later schema on search_path, and asserts that connect and migrate both refuse it, that the loss is still named as a loss without the later schema, and that the restored declaration is read with the later schema still present. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: bind migration provenance to the verified schema * fix: reject object collisions in native migration gaps --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…write off the ordering lock (#350) * 3.x.x: add migration 007 and refuse pre-007 candidate outbox rows Migration 007 mirrors 008 onto qbit_block_candidate_outbox: the same six window-reference columns, and a three-state CHECK whose body is identical to 008's apart from the table and constraint names. It also adds block_bytes, the nullable bytea the block moves into so the candidate document stays small, and a partial index on window_prior_balances_sha256 for the balance-snapshot GC. Nothing writes any of them yet; this is additive schema only. The runner applies 007 by membership, in its own block between 5 and 8, so it assumes neither 006 nor 008 has run. Before any DDL it refuses while the outbox still holds a pending row in a pre-007 shape: a native inline candidate, a NULL candidate, or storage_version other than 1 on the 2.x.x schemas that have that column, which is checked for first so the predicate does not error on a native schema. The refusal names every offending block_hash and the drain procedure, and the whole migration transaction rolls back, so a refused start leaves the schema untouched. The applied set is now [2, 3, 4, 5, 7, 8, 9], and the four exact migration assertions move with it rather than being weakened to a superset. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: retain the cluster fingerprint configure pinned or verified The candidate enqueue and save_job must fence against a fingerprint reset by re-reading qbit_prism_cluster.config_fingerprint FOR SHARE in their own transaction and refusing a write that is not this frontend's own. That needs the value configure settled at startup, so Ledger keeps it in a shared OnceLock, set only once the pin or the match has committed, and exposes it read-only through config_fingerprint(). configure keeps its signature and its mismatch refusal, and a refused call retains nothing. The rotation refusal in configure, and the writer fence itself, belong to the enqueue change and are not here. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: share the window reference, balance writer and page primitives WindowRef::from_snapshot is the synchronous reference builder both writers need: anchor_ms copied, the semantic balances digest, and for a non-empty snapshot the first and last share_seq, the length, and the share array's SHA-256 streamed through the existing DigestWriter, so the 233 to 260 MB serialized window at 400,000 shares is never built and the digest equals the one qbit_prism_audit_snapshots already stores. An empty snapshot gives the empty-window reference. Callers run it inside their own spawn_blocking. put_balance_snapshot is the one canonical writer of qbit_prism_balance_snapshots, on the caller's transaction: the sort by (order_key, recipient_id, p2mr_program_hex), the semantic digest that keys the row, and compact JSON of the sorted set, all off the runtime, then INSERT ... ON CONFLICT DO NOTHING. On a conflict it reads the stored bytes back and compares them with its own, because these rows are immutable and can only disagree through corruption. The enqueue and save_job and its repair all write through it. read_prior_balances and the current-balance read still keep their SQL order; the writer's sort defines the stored encoding, which is the order read_window(..., AsIssued) returns. probe_share_rows and read_range_paged are read_window's endpoint probe and page loop, lifted onto a caller's connection so the enqueue probe and the landing re-read share one implementation. The predicate, the ascending keyset order, the 4096-row page bound and the per-page count bound are unchanged, and read_window now calls them, with its signature and behaviour unchanged. The paged reader takes the caller's state and a per-page consumer, both of which travel to the blocking thread with the rows, so the landing re-read can compare each page with its slice of a vector it already holds without any whole-window work on the runtime; a dropped read still releases the accumulated state on a blocking thread. read_window_with_permit holds the caller's window_reads permit across the whole read. The permit is moved into a BlockingDrop declared before the read starts, so a cancellation releases it on a blocking thread after the page state queued its own release there, and never with the future; a return releases it inline, before the caller's rebuild. WindowError::TaskFailed separates a cancelled or panicking blocking hand-off from corruption. Every hand-off maps a JoinError to it instead of to Decode, so a cancellation is a retryable failure with its own alert and never takes the abandon path. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: derive the witness merkle leaves from an assembled block A claim rebuilds from stored inputs, and the witness leaves are the one builder input that scales with the block, so they are re-derived rather than stored: one 64-hex string per transaction beside the candidate would grow the document with the block. witness_merkle_leaves_from_block skips the 80-byte header, reads the compact-size transaction count, walks past the coinbase and returns double_sha256 of every remaining transaction's full serialized bytes. That is sound because assemble_submission appends the template's transactions verbatim after the coinbase, so the result equals witness_merkle_leaves_hex(&transactions_from_template(template)) for the block built from that template, with or without segwit transactions. The segwit-aware walk is strip_witness_transaction's own, factored into a helper that also reports where the transaction ends, which is what a block walker needs; strip_witness_transaction keeps its trailing-byte check and its output. A truncated block, trailing bytes, a zero count and a count that disagrees with the transactions present are all errors, never a partial list of leaves. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: version the audit builders and key the CC5 goldens by it A stored window reference reproduces its bundle only under the builder that first built it. The schema strings do not capture that, because the share shape chooses them rather than the builder logic, so qbit-prism gains AUDIT_BUILDER_VERSION, starting at 1. A candidate and a prepared job store it, and a rebuild refuses a reference at another version instead of rebuilding it with the current builder. The bump rule is on the constant: any change that alters canonical_audit_bundle_bytes for the same inputs bumps it in the same commit. The CC5 golden test now selects its expected digests from a table keyed by the constant, and fails when no entry exists, so a new output is recorded as a new entry beside the old one and review sees both. Version 1 holds the existing literals, unchanged. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: cover 007, the fingerprint and the window primitives on PostgreSQL Ten gated database cases, all listed in test/prism-gated-tests.txt. They go in a new file rather than the shared 2.x.x support module, which another open change appends to. Migration 007: a frozen 2.x.x schema carrying only terminal outbox rows upgrades with every row and field intact and the seven new columns NULL; each of the three pre-007 pending shapes is refused by name, with nothing from 007 applied and no version 7 row, and the same binary starts once the row is drained; a database at [2, 3, 4, 5, 8, 9] applies 007 and only 007, leaving every other version's applied_at untouched; the CHECK accepts exactly the three states across all 64 null combinations and rejects a short or uppercase digest, a zero or out-of-range count and a zero sequence; and the retention index is the partial one on window_prior_balances_sha256. config_fingerprint() returns what configure pinned or verified, is shared by clones, and stays empty after a refused configure. from_snapshot's streamed digest equals sha256(serde_json::to_vec) and the snapshot_sha256 landing writes for the same window. The balance writer stores the canonical sort, is idempotent under any permutation, errors rather than accepting a different encoding of the same digest, and read_window(..., AsIssued) hands back the stored order. The probe misses a missing endpoint, the paged reader compares 4,999 rows page by page with a vector held inside the caller's transaction, and a row the predicate skips inside a range is Incomplete with no partial window. The permit case builds a runtime with one blocking thread, occupies it, and cancels a reader parked mid-page on a database gate: no permit is available while the cleanup is queued, and one is once it has run. A failed blocking hand-off surfaces as TaskFailed, never as Decode. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: carry the window reference on prepared work, behind a read permit `Prepared` gains `window: WindowRef`, computed once per non-cached refresh beside `build_bundle` under `tokio::join!`, so submit can clone it instead of re-digesting the window on the share path when a block is found. An empty snapshot gets the empty-window reference on the runtime thread; a resume computes it inside the blocking decode it already runs. `Coordinator` gains the `window_reads` semaphore, sized `clamp(database_max_connections - 2, 1, build_workers)`, and the one entry point callers use: it takes an owned permit after the caller's `build_slots` permit and hands it to `Ledger::read_window_with_permit`, which holds it across the whole read, including the off-runtime cleanup a cancellation leaves behind, and releases it before the rebuild. `Ledger::read_window` is awaited on the runtime, never wrapped in `spawn_blocking`; the test with a single blocking thread pins that. The four gated `window_ref_tests` are listed in the manifest. Part of #265 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * 3.x.x: switch the candidate outbox to the window reference `Candidate` drops the embedded `AuditBundle` for `window: WindowRef` and every builder input except the window: `found_block`, `payout_policy`, the nested `ctv` triple (whose presence selects the CTV builder), the stored `bootstrap_share` of an empty window, `audit_builder_version`, `signer_keys`, a `leased` flag and the now-required `coinbase_suffix_hex`. The block leaves the document for the `block_bytes` column, authenticated by `block_sha256`, which `candidate_sha256` covers. The write serializes, digests and checks the candidate before the append or enqueue transaction opens; under `ORDER_LOCK` the transaction re-reads `config_fingerprint FOR SHARE` and refuses a value the writer did not pin, re-inserts a `leased` candidate's as-issued balance snapshot through the shared writer, probes the window's first row (an absent row alerts and still publishes), and inserts only the prepared bytes with the six typed columns. Submit builds the slim candidate from `Prepared.window`, the `BundleInputs` captured where `build_bundle` read configuration, and the job's own bootstrap share; it clones no bundle and reads no config. The claim's `UPDATE … RETURNING` returns the columns and the block, and the decode, off the runtime, checks the document against them, the digest, `sha256(block_bytes)`, the header hash, the required fields and the two reference invariants. A pending row with a NULL anchor is a hard error naming the pre-007 frontend; an inline document is never decoded. Before any read the claim retries, with an alert naming both values, a stored builder version or signer pair that is not this binary's; a landed audit is authenticated against the block (coinbase, audit root as the witness reserved value, share snapshot, bits) and the claim then skips the rebuild and landing and continues to observe, renew and submit. Otherwise it takes `build_slots`, then `window_reads` inside one 60 s deadline around the awaited `read_window(Current)` and the borrowing builder run directly in `spawn_blocking`, and keeps `Arc<AuditBundleBody>` and the shares as `ClaimParts`. `Incomplete`, `SnapshotDigestMismatch`, `Decode`, `BalanceSnapshotMissing`, `TaskFailed` and the deadline retry with their own alerts; `PriorBalancesChanged` finishes only a block `observe_candidate` proves inactive and superseded; `Database` propagates. A `leased` candidate submits from `block_bytes` first, then rebuilds `AsIssued` and lands before any terminal outcome. Rebuilt parts are released off the runtime on every path. Landing takes the parts and does all whole-window work in one blocking closure: `verify_audit_parts`, the raw body, the counted canonical bytes, the leaves and accounts, and the share snapshot digest, which must equal the reference's. The balance fence compares the current set and the parts' through the semantic digest, so an as-issued set compares equal in any order. `persist_audit_snapshot` re-reads the range with the paged reader inside the landing transaction, page against slice on the blocking thread; `read_range` stays for the API. The terminal UPDATE NULLs the six window columns and `block_bytes` with `candidate`. `configure` refuses to pin a fingerprint onto a reset one while a pending row stores other signer keys, and the startup gate requires 007 with 009. The JSONB gate drops its enqueue entry, and every helper that built a candidate builds the slim one, handing the ledger-level landings the bundle's parts through `CandidateClaim::with_bundle`. Part of #265 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * 3.x.x: cover the candidate switch on PostgreSQL Ledger-level, in `candidate_window_switch`: a two-page window enqueues a document with no array and no share row, under 1 MB, whose six typed columns equal its reference and whose block is stored as bytes; the writer fence refuses a reset or different fingerprint in the same transaction, share included; a missing window prefix row alerts and still publishes, and its read then meets `Incomplete`; a `leased` enqueue re-inserts the as-issued balance snapshot in the writer's canonical order, writes none when not leased, and refuses balances that do not hash to the reference; every preparation failure is refused without waiting for `ORDER_LOCK`; every column, digest, block, header, NULL-anchor, missing-field, invariant and inline-document disagreement is refused at claim; landing refuses parts that are not the referenced window and a claim without parts; the terminal UPDATE NULLs the six columns and the block for submitted and abandoned rows; the landed-audit authentication rejects a forged coinbase, root, snapshot or bits; and `configure` refuses a rotation over a foreign-signed pending candidate. Coordinator-level, in `window_switch_tests` behind a fake node: a stored builder version or signer pair that is not this binary's retries, naming both values, while the build slot is held; a landed audit is authenticated and the claim continues to `submitblock` without a build slot, and a forged landed coinbase or root is refused with the row left pending; `Incomplete`, a digest mismatch, changed balances on a current block and the rebuild deadline all retry with an alert and release their permits; `classify_window_error` keeps `TaskFailed` apart from corruption and propagates `Database`; the read, rebuild and landing complete while `ORDER_LOCK` is held, with one build worker; a `leased` candidate submits first, then lands from `AsIssued`, and finishes submitted, abandoned with the node's reason, or pending when its balance snapshot is missing; and `OffRuntime` releases on a blocking thread, on cancellation too. The fifteen gated tests are listed in the manifest. Part of #265 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * 3.x.x: migrate the imported-audit tests to the slim candidate Slice 1 (#325) merged after this stack branched, and its tests in tests/support/ledger_2x.rs still built the pre-007 candidate: they read block.bundle from a Candidate and called candidate_with_bundle without a window reference or settlement inputs. The rebase merged them cleanly and the tree then failed to compile. The three helpers now take the TestCandidate the constructors return, so they reach the bundle beside the slim candidate, and land_confirmed attaches the claim's parts the way the other ledger tests do. The three constructor calls pass the window reference the bundle was built on, and the settlement inputs for the CTV case. No assertion changes. Part of #265. * 3.x.x: qualify lease renewal and the claim's canonical bytes at scale Incident 1: a candidate whose window holds N shares is claimed and processed under the short lease, and every renewal is read back from claim_expires_at. The largest gap stays below interval + 2 x timeout, the lease never lapses, and a successor ledger cannot take the claim. The bundle a claim rebuilds through the window-read entry point and the borrowing builder has the canonical bytes refresh built for the same job, and the landed audit carries that bundle's digest. Both run at PRISM_WINDOW_QUALIFY_SHARES (20,000 by default); the 400,000-share variants are ignored and run on a larger host. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: measure ACK latency and ORDER_LOCK hold across window sizes Incident 2: two frontends on one database submit ordinary shares at a fixed rate while one of them solves a block on an N-share window and its own submit loop claims, rebuilds, lands and submits it. The solving share's ACK and the bystander's p99 stay within max(2 x baseline p99 + 50 ms, 250 ms), every ACK is well under the share-commit timeout, and the block is enqueued exactly once. Latency is the in-process MiningBackend::submit wall clock; the socket-level harness (#271) does not exist yet. A pg_locks sampler on its own connection measures how long a solving append holds ORDER_LOCK at two window sizes with no other traffic: the larger size stays within a fixed slack of the smaller, and the candidate document is the same size at both and under 1 MiB. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: hold the JSONB gate's enqueue to a window-free row under 1 MiB At every measured size the enqueue must be accepted and every JSONB value it writes, the outbox candidate document included, must stay under 1,048,576 bytes. At the reduced sizes the enqueue, claim and landing phases must all run. The ratchet and its expected values are unchanged. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: make the window-read permit test actually wait before it looks The delay that gives an unpermitted read time to finish was an empty async block, so the check that the read had not finished ran at once and passed whether or not the entry point took a permit. It now sleeps 500 ms inside a 5 s bound, and fails when the permit acquisition is removed from the entry point. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: rebuild a CTV candidate from its stored settlement inputs No host-runnable test reached the CTV branch of the claim rebuild. A CTV-enabled frontend now claims a candidate whose stored direct floor differs from its configuration, and builds a different coinbase than the configured floor would, and the claim must land that candidate's own audit and submit it. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: pin the SQL order of the current-balance reads Nothing defended the ruling that the current-balance path keeps its SQL order: every fixture's balances already arrived in byte-wise canonical order. The carry table's text columns now take an ICU collation, under which the SQL order and the canonical sort disagree, and the snapshot and the current-balance window read must both return the SQL order. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: record what incident 2 must also assert once #324 is in the base After #324 a slow solve is acknowledged late rather than refused, so acceptance proves nothing and the latency percentiles stay the assertion. The doc comment and a TODO at the outcome checks name the counter and outcome to assert at rebase. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: assert incident 2's late-confirm counter now that #324 is in the base W3c wrote the qualification tests against a base that predated #324 and left a marked place for the assertions that only make sense once the commit gate is there. The base now carries it, so apply them: incident 2 asserts `qbit_prism_late_confirmed_shares_total` is zero on both frontends and that no acknowledgement, the block-solving share included, is `ledger-outcome-unknown`. Acceptance alone is no longer proof. #324 gives a share-pass append carrying a candidate its own `block_only_ack_timeout`, so a slow solve is never refused at the share deadline; it is answered late, or answered unknown. Either would have satisfied the acceptance checks with the serialization stall still present. Also set #324's two new `Config` fields in the qualification fixture. * 3.x.x: pin a refused candidate as a definite rejection, not an unknown outcome #324 decides what a miner is told when a block-only enqueue fails: `enqueue_failed_before_commit` answers a provably uncommitted failure with `ledger-confirmation-failed`, and anything else with `ledger-outcome-unknown`, which asks an operator to go and follow the outbox row. Slice 3 added the checks that guard the reference: the block's length, its header hash, its digest against `block_sha256`, the coinbase suffix, the reference invariants, and a leased candidate's balances. All of them are raised by `prepare_candidate` before the enqueue takes a connection, so none carries an `sqlx::Error` and all classify as definite today. That is a property of where the checks sit, not something the classifier knows about them, so it is pinned here rather than left to hold by accident: a corrupt candidate must never send a miner chasing a row that was never written. Verified by mutation: classifying a non-sqlx failure as unknown fails this test. * 3.x.x: say why the full-size qualification variants cannot reach 400k yet Their ignore reasons blamed RAM and the host. The real limit is #273: the job payload still holds three copies of the window (2 x AcceptedShare + 1 x CountedShare), so publishing a job for a 400,000-share window needs about 670 MiB and PostgreSQL refuses the write above its 256 MiB jsonb object limit. Every full-size variant fails at fixture set-up, before it measures anything. The count must also divide the window weight exactly, so the largest usable size today is 125,000, where all four variants pass in release: incident 1 50 renewals, max gap 118.25 ms against a 700 ms bound incident 2 phase-A p99 2.60 ms, phase-B p99 95.79 ms, bound 250 ms lock hold median 0.75 ms at 5,000 against 1.32 ms at 125,000 canonical refresh and claim rebuild both 139,040,610 B, same sha256 The candidate document stays flat at 1103 B and 1107 B, which is the property this slice exists to establish. * 3.x.x: migrate the audit-body normalization tests to the slim candidate #267 landed its own test file against the candidate that still carried an assembled bundle. This slice replaced that with a window reference, so the file keeps a `TestCandidate` pairing the stored candidate with the bundle a test still needs, as `tests/ledger_postgres.rs` does, and attaches the claim's parts through `with_bundle`. One case needed more than a shape change. The four negative cases hand the claim a window the ledger does not hold and require the durable-range proof to be what refuses it. A candidate now commits to a snapshot digest, so a reference taken from the unaltered snapshot would be refused earlier, at the reference check, and the proof under test would never run. `window_ref_for` therefore derives each altered window's reference from that window, leaving the proof as the only thing that can notice, which is what #267 wrote them to prove. * 3.x.x: pin the durable-range proof outside the settlement lock #267 moved the full-equality proof of a candidate's window out of the settlement transaction so the lock is not held for a window-sized read. This slice made that read paged, so it does not occupy a runtime thread either. Reconciling the two is the step that can quietly undo the first: keep this slice's `persist_audit_snapshot` through the rebase and the full read goes back under the lock. The landing stays correct, every other test still passes, and only the lock hold grows, so nothing in the suite can tell. Measured at 100,000 shares on #267's branch that was a 34% longer hold. The guard is deterministic and needs no large window. A second connection holds `SETTLEMENT_LOCK`, so the landing blocks on it once its proof has run. While it waits, one share's `ntime` is altered: full equality sees that, a count over the same anchored predicate cannot. The landing must then succeed, which it can only do if it had already read and compared the shares before taking the lock. Accepted history is immutable and a trigger enforces it, so the test lifts that trigger for the single statement, inside its own disposable schema, and puts it straight back. Verified by mutation: restoring the in-transaction range read fails this test with "the landing read share payloads under the settlement lock". * 3.x.x: serialise the coordinator test modules that share the ledger's locks The ledger's advisory locks are cluster-wide constants, not schema-scoped, so two coordinator test modules running in parallel contend for ORDER_LOCK even though each works in its own throwaway schema. Every module held a lock of its own, which serialised it against itself and left cross-module waits to `lock_timeout`, 5 s by default. #324's reconciliation tests hold ORDER_LOCK for up to about 1.5 s, and this slice adds three more modules that open a ledger. Once enough of them ran at once those waits stacked past the timeout: a full `--lib` run failed with "canceling statement due to lock timeout" in `candidate_lease_tests` and `window_switch_tests`, and passed on the next run. A suite that fails for a reason unrelated to what it tests is worse than a slow one. All seven modules now take one shared lock. The run goes from about 35 s to about 62 s, and three consecutive runs pass where the flake had reproduced. * 3.x.x: pass the signer keys #334's wait-metrics tests now need `Ledger::configure` takes the frontend's signing keys since this slice, so the fingerprint contention tests #334 added call it with a fixed pair. That suite measures advisory-lock and pool-acquire waits, not signing, so the only thing the keys have to be is the same on every call. * 3.x.x: resume a job's bundle only while the current inputs still describe it A resumed job reuses the bundle it was issued with, but captures its inputs from this frontend's configuration now. Before this slice that was harmless: the candidate carried the assembled bundle, and landing used it directly. Now the candidate stores the inputs and the claim rebuilds the audit from them, so an old bundle labelled with today's configuration rebuilds to a coinbase the block does not commit to. The claim fails its comparison before `submitblock` and retries until the job expires, which can lose a found block. The stored bundle is now reused only when the captured inputs still describe it, comparing what the bundle itself records about how it was built: the payout policy, and the public keys that actually signed its coinbase manifest and its ledger attestation. A difference is a cache miss, so the bundle is rebuilt rather than mislabelled. Both directions are covered, since over-tightening would silently defeat the job cache and rebuild on every resume: unchanged inputs must still describe the bundle, and a payout-policy change, a full signer rotation and a rotation of either key alone must each refuse it. Known residue, recorded in the doc comment: a bundle records neither the CTV settlement inputs nor `audit_builder_version`, so a change to either between issue and resume is still undetectable here. Closing that needs the issued job to store its inputs, which is #273's format. * 3.x.x: migrate #355's ledger tests to the slim candidate #355 landed deadline and single-execution coverage against the candidate that still carried an assembled bundle, and against `configure` before it took the frontend's signing keys. Adapted, the same way #325's and #267's suites were: a `TestCandidate` pairs the stored candidate with the bundle a landing still needs, claims attach their parts through `with_bundle`, and the conflicting claim that is never landed carries none. `ledger_deadline_atomicity` builds its candidate from a bundle rather than a snapshot, so its window reference is derived from that bundle: the anchor and prior-balances digest it already records, and the range and digest of the shares it carries. Its signer keys come from the public keys that actually signed it, rather than from a local pair, so the stored inputs describe the bundle. * 3.x.x: make the landed-audit forgery always alter the row, and migrate #360's fixture The forged-coinbase case placed a literal '0' at a fixed offset. Whenever the digit already held that value the forgery changed nothing, so the claim was right to accept it and the case failed only by luck. The coinbase varies per run, because the bundle's anchor comes from the ledger clock, so this passed repeatedly and then failed once as "the forgery was accepted" — an alarming message for a case that was in fact a no-op. A security check that can quietly become vacuous is worse than one that is absent. The digit is now flipped to a value it does not currently hold, and the row is read back either side of the statement so a forgery that altered nothing fails the case for that reason instead. Also passes the signer keys #360's `tests/support/fatal_state.rs` now needs, since `configure` takes them in this slice. * 3.x.x: drop the pre-#362 path for the heartbeat SQL gated test #362 moved `live_instance_tests` from `tools` to `ledger::instances`. Merging the gated-test manifest during the rebase kept both paths, because a rename upstream looks like one id removed and another added, and a union keeps the removed one. The old `tools::` path then never executed and the gate refused the run: "194 distinct gated tests executed, below the minimum of 195". The manifest is a superset of the base's ids, and every id this branch adds over the base resolves to a test function that exists in the tree. * v3.0.0 feat: add compact prepared storage and immutable template blobs (#359) * 3.x.x: add migration 007 and refuse pre-007 candidate outbox rows Migration 007 mirrors 008 onto qbit_block_candidate_outbox: the same six window-reference columns, and a three-state CHECK whose body is identical to 008's apart from the table and constraint names. It also adds block_bytes, the nullable bytea the block moves into so the candidate document stays small, and a partial index on window_prior_balances_sha256 for the balance-snapshot GC. Nothing writes any of them yet; this is additive schema only. The runner applies 007 by membership, in its own block between 5 and 8, so it assumes neither 006 nor 008 has run. Before any DDL it refuses while the outbox still holds a pending row in a pre-007 shape: a native inline candidate, a NULL candidate, or storage_version other than 1 on the 2.x.x schemas that have that column, which is checked for first so the predicate does not error on a native schema. The refusal names every offending block_hash and the drain procedure, and the whole migration transaction rolls back, so a refused start leaves the schema untouched. The applied set is now [2, 3, 4, 5, 7, 8, 9], and the four exact migration assertions move with it rather than being weakened to a superset. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: retain the cluster fingerprint configure pinned or verified The candidate enqueue and save_job must fence against a fingerprint reset by re-reading qbit_prism_cluster.config_fingerprint FOR SHARE in their own transaction and refusing a write that is not this frontend's own. That needs the value configure settled at startup, so Ledger keeps it in a shared OnceLock, set only once the pin or the match has committed, and exposes it read-only through config_fingerprint(). configure keeps its signature and its mismatch refusal, and a refused call retains nothing. The rotation refusal in configure, and the writer fence itself, belong to the enqueue change and are not here. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: share the window reference, balance writer and page primitives WindowRef::from_snapshot is the synchronous reference builder both writers need: anchor_ms copied, the semantic balances digest, and for a non-empty snapshot the first and last share_seq, the length, and the share array's SHA-256 streamed through the existing DigestWriter, so the 233 to 260 MB serialized window at 400,000 shares is never built and the digest equals the one qbit_prism_audit_snapshots already stores. An empty snapshot gives the empty-window reference. Callers run it inside their own spawn_blocking. put_balance_snapshot is the one canonical writer of qbit_prism_balance_snapshots, on the caller's transaction: the sort by (order_key, recipient_id, p2mr_program_hex), the semantic digest that keys the row, and compact JSON of the sorted set, all off the runtime, then INSERT ... ON CONFLICT DO NOTHING. On a conflict it reads the stored bytes back and compares them with its own, because these rows are immutable and can only disagree through corruption. The enqueue and save_job and its repair all write through it. read_prior_balances and the current-balance read still keep their SQL order; the writer's sort defines the stored encoding, which is the order read_window(..., AsIssued) returns. probe_share_rows and read_range_paged are read_window's endpoint probe and page loop, lifted onto a caller's connection so the enqueue probe and the landing re-read share one implementation. The predicate, the ascending keyset order, the 4096-row page bound and the per-page count bound are unchanged, and read_window now calls them, with its signature and behaviour unchanged. The paged reader takes the caller's state and a per-page consumer, both of which travel to the blocking thread with the rows, so the landing re-read can compare each page with its slice of a vector it already holds without any whole-window work on the runtime; a dropped read still releases the accumulated state on a blocking thread. read_window_with_permit holds the caller's window_reads permit across the whole read. The permit is moved into a BlockingDrop declared before the read starts, so a cancellation releases it on a blocking thread after the page state queued its own release there, and never with the future; a return releases it inline, before the caller's rebuild. WindowError::TaskFailed separates a cancelled or panicking blocking hand-off from corruption. Every hand-off maps a JoinError to it instead of to Decode, so a cancellation is a retryable failure with its own alert and never takes the abandon path. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: derive the witness merkle leaves from an assembled block A claim rebuilds from stored inputs, and the witness leaves are the one builder input that scales with the block, so they are re-derived rather than stored: one 64-hex string per transaction beside the candidate would grow the document with the block. witness_merkle_leaves_from_block skips the 80-byte header, reads the compact-size transaction count, walks past the coinbase and returns double_sha256 of every remaining transaction's full serialized bytes. That is sound because assemble_submission appends the template's transactions verbatim after the coinbase, so the result equals witness_merkle_leaves_hex(&transactions_from_template(template)) for the block built from that template, with or without segwit transactions. The segwit-aware walk is strip_witness_transaction's own, factored into a helper that also reports where the transaction ends, which is what a block walker needs; strip_witness_transaction keeps its trailing-byte check and its output. A truncated block, trailing bytes, a zero count and a count that disagrees with the transactions present are all errors, never a partial list of leaves. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: version the audit builders and key the CC5 goldens by it A stored window reference reproduces its bundle only under the builder that first built it. The schema strings do not capture that, because the share shape chooses them rather than the builder logic, so qbit-prism gains AUDIT_BUILDER_VERSION, starting at 1. A candidate and a prepared job store it, and a rebuild refuses a reference at another version instead of rebuilding it with the current builder. The bump rule is on the constant: any change that alters canonical_audit_bundle_bytes for the same inputs bumps it in the same commit. The CC5 golden test now selects its expected digests from a table keyed by the constant, and fails when no entry exists, so a new output is recorded as a new entry beside the old one and review sees both. Version 1 holds the existing literals, unchanged. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: cover 007, the fingerprint and the window primitives on PostgreSQL Ten gated database cases, all listed in test/prism-gated-tests.txt. They go in a new file rather than the shared 2.x.x support module, which another open change appends to. Migration 007: a frozen 2.x.x schema carrying only terminal outbox rows upgrades with every row and field intact and the seven new columns NULL; each of the three pre-007 pending shapes is refused by name, with nothing from 007 applied and no version 7 row, and the same binary starts once the row is drained; a database at [2, 3, 4, 5, 8, 9] applies 007 and only 007, leaving every other version's applied_at untouched; the CHECK accepts exactly the three states across all 64 null combinations and rejects a short or uppercase digest, a zero or out-of-range count and a zero sequence; and the retention index is the partial one on window_prior_balances_sha256. config_fingerprint() returns what configure pinned or verified, is shared by clones, and stays empty after a refused configure. from_snapshot's streamed digest equals sha256(serde_json::to_vec) and the snapshot_sha256 landing writes for the same window. The balance writer stores the canonical sort, is idempotent under any permutation, errors rather than accepting a different encoding of the same digest, and read_window(..., AsIssued) hands back the stored order. The probe misses a missing endpoint, the paged reader compares 4,999 rows page by page with a vector held inside the caller's transaction, and a row the predicate skips inside a range is Incomplete with no partial window. The permit case builds a runtime with one blocking thread, occupies it, and cancels a reader parked mid-page on a database gate: no permit is available while the cleanup is queued, and one is once it has run. A failed blocking hand-off surfaces as TaskFailed, never as Decode. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: carry the window reference on prepared work, behind a read permit `Prepared` gains `window: WindowRef`, computed once per non-cached refresh beside `build_bundle` under `tokio::join!`, so submit can clone it instead of re-digesting the window on the share path when a block is found. An empty snapshot gets the empty-window reference on the runtime thread; a resume computes it inside the blocking decode it already runs. `Coordinator` gains the `window_reads` semaphore, sized `clamp(database_max_connections - 2, 1, build_workers)`, and the one entry point callers use: it takes an owned permit after the caller's `build_slots` permit and hands it to `Ledger::read_window_with_permit`, which holds it across the whole read, including the off-runtime cleanup a cancellation leaves behind, and releases it before the rebuild. `Ledger::read_window` is awaited on the runtime, never wrapped in `spawn_blocking`; the test with a single blocking thread pins that. The four gated `window_ref_tests` are listed in the manifest. Part of #265 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * 3.x.x: switch the candidate outbox to the window reference `Candidate` drops the embedded `AuditBundle` for `window: WindowRef` and every builder input except the window: `found_block`, `payout_policy`, the nested `ctv` triple (whose presence selects the CTV builder), the stored `bootstrap_share` of an empty window, `audit_builder_version`, `signer_keys`, a `leased` flag and the now-required `coinbase_suffix_hex`. The block leaves the document for the `block_bytes` column, authenticated by `block_sha256`, which `candidate_sha256` covers. The write serializes, digests and checks the candidate before the append or enqueue transaction opens; under `ORDER_LOCK` the transaction re-reads `config_fingerprint FOR SHARE` and refuses a value the writer did not pin, re-inserts a `leased` candidate's as-issued balance snapshot through the shared writer, probes the window's first row (an absent row alerts and still publishes), and inserts only the prepared bytes with the six typed columns. Submit builds the slim candidate from `Prepared.window`, the `BundleInputs` captured where `build_bundle` read configuration, and the job's own bootstrap share; it clones no bundle and reads no config. The claim's `UPDATE … RETURNING` returns the columns and the block, and the decode, off the runtime, checks the document against them, the digest, `sha256(block_bytes)`, the header hash, the required fields and the two reference invariants. A pending row with a NULL anchor is a hard error naming the pre-007 frontend; an inline document is never decoded. Before any read the claim retries, with an alert naming both values, a stored builder version or signer pair that is not this binary's; a landed audit is authenticated against the block (coinbase, audit root as the witness reserved value, share snapshot, bits) and the claim then skips the rebuild and landing and continues to observe, renew and submit. Otherwise it takes `build_slots`, then `window_reads` inside one 60 s deadline around the awaited `read_window(Current)` and the borrowing builder run directly in `spawn_blocking`, and keeps `Arc<AuditBundleBody>` and the shares as `ClaimParts`. `Incomplete`, `SnapshotDigestMismatch`, `Decode`, `BalanceSnapshotMissing`, `TaskFailed` and the deadline retry with their own alerts; `PriorBalancesChanged` finishes only a block `observe_candidate` proves inactive and superseded; `Database` propagates. A `leased` candidate submits from `block_bytes` first, then rebuilds `AsIssued` and lands before any terminal outcome. Rebuilt parts are released off the runtime on every path. Landing takes the parts and does all whole-window work in one blocking closure: `verify_audit_parts`, the raw body, the counted canonical bytes, the leaves and accounts, and the share snapshot digest, which must equal the reference's. The balance fence compares the current set and the parts' through the semantic digest, so an as-issued set compares equal in any order. `persist_audit_snapshot` re-reads the range with the paged reader inside the landing transaction, page against slice on the blocking thread; `read_range` stays for the API. The terminal UPDATE NULLs the six window columns and `block_bytes` with `candidate`. `configure` refuses to pin a fingerprint onto a reset one while a pending row stores other signer keys, and the startup gate requires 007 with 009. The JSONB gate drops its enqueue entry, and every helper that built a candidate builds the slim one, handing the ledger-level landings the bundle's parts through `CandidateClaim::with_bundle`. Part of #265 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * 3.x.x: cover the candidate switch on PostgreSQL Ledger-level, in `candidate_window_switch`: a two-page window enqueues a document with no array and no share row, under 1 MB, whose six typed columns equal its reference and whose block is stored as bytes; the writer fence refuses a reset or different fingerprint in the same transaction, share included; a missing window prefix row alerts and still publishes, and its read then meets `Incomplete`; a `leased` enqueue re-inserts the as-issued balance snapshot in the writer's canonical order, writes none when not leased, and refuses balances that do not hash to the reference; every preparation failure is refused without waiting for `ORDER_LOCK`; every column, digest, block, header, NULL-anchor, missing-field, invariant and inline-document disagreement is refused at claim; landing refuses parts that are not the referenced window and a claim without parts; the terminal UPDATE NULLs the six columns and the block for submitted and abandoned rows; the landed-audit authentication rejects a forged coinbase, root, snapshot or bits; and `configure` refuses a rotation over a foreign-signed pending candidate. Coordinator-level, in `window_switch_tests` behind a fake node: a stored builder version or signer pair that is not this binary's retries, naming both values, while the build slot is held; a landed audit is authenticated and the claim continues to `submitblock` without a build slot, and a forged landed coinbase or root is refused with the row left pending; `Incomplete`, a digest mismatch, changed balances on a current block and the rebuild deadline all retry with an alert and release their permits; `classify_window_error` keeps `TaskFailed` apart from corruption and propagates `Database`; the read, rebuild and landing complete while `ORDER_LOCK` is held, with one build worker; a `leased` candidate submits first, then lands from `AsIssued`, and finishes submitted, abandoned with the node's reason, or pending when its balance snapshot is missing; and `OffRuntime` releases on a blocking thread, on cancellation too. The fifteen gated tests are listed in the manifest. Part of #265 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * 3.x.x: migrate the imported-audit tests to the slim candidate Slice 1 (#325) merged after this stack branched, and its tests in tests/support/ledger_2x.rs still built the pre-007 candidate: they read block.bundle from a Candidate and called candidate_with_bundle without a window reference or settlement inputs. The rebase merged them cleanly and the tree then failed to compile. The three helpers now take the TestCandidate the constructors return, so they reach the bundle beside the slim candidate, and land_confirmed attaches the claim's parts the way the other ledger tests do. The three constructor calls pass the window reference the bundle was built on, and the settlement inputs for the CTV case. No assertion changes. Part of #265. * 3.x.x: qualify lease renewal and the claim's canonical bytes at scale Incident 1: a candidate whose window holds N shares is claimed and processed under the short lease, and every renewal is read back from claim_expires_at. The largest gap stays below interval + 2 x timeout, the lease never lapses, and a successor ledger cannot take the claim. The bundle a claim rebuilds through the window-read entry point and the borrowing builder has the canonical bytes refresh built for the same job, and the landed audit carries that bundle's digest. Both run at PRISM_WINDOW_QUALIFY_SHARES (20,000 by default); the 400,000-share variants are ignored and run on a larger host. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: measure ACK latency and ORDER_LOCK hold across window sizes Incident 2: two frontends on one database submit ordinary shares at a fixed rate while one of them solves a block on an N-share window and its own submit loop claims, rebuilds, lands and submits it. The solving share's ACK and the bystander's p99 stay within max(2 x baseline p99 + 50 ms, 250 ms), every ACK is well under the share-commit timeout, and the block is enqueued exactly once. Latency is the in-process MiningBackend::submit wall clock; the socket-level harness (#271) does not exist yet. A pg_locks sampler on its own connection measures how long a solving append holds ORDER_LOCK at two window sizes with no other traffic: the larger size stays within a fixed slack of the smaller, and the candidate document is the same size at both and under 1 MiB. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: hold the JSONB gate's enqueue to a window-free row under 1 MiB At every measured size the enqueue must be accepted and every JSONB value it writes, the outbox candidate document included, must stay under 1,048,576 bytes. At the reduced sizes the enqueue, claim and landing phases must all run. The ratchet and its expected values are unchanged. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: make the window-read permit test actually wait before it looks The delay that gives an unpermitted read time to finish was an empty async block, so the check that the read had not finished ran at once and passed whether or not the entry point took a permit. It now sleeps 500 ms inside a 5 s bound, and fails when the permit acquisition is removed from the entry point. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: rebuild a CTV candidate from its stored settlement inputs No host-runnable test reached the CTV branch of the claim rebuild. A CTV-enabled frontend now claims a candidate whose stored direct floor differs from its configuration, and builds a different coinbase than the configured floor would, and the claim must land that candidate's own audit and submit it. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: pin the SQL order of the current-balance reads Nothing defended the ruling that the current-balance path keeps its SQL order: every fixture's balances already arrived in byte-wise canonical order. The carry table's text columns now take an ICU collation, under which the SQL order and the canonical sort disagree, and the snapshot and the current-balance window read must both return the SQL order. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: record what incident 2 must also assert once #324 is in the base After #324 a slow solve is acknowledged late rather than refused, so acceptance proves nothing and the latency percentiles stay the assertion. The doc comment and a TODO at the outcome checks name the counter and outcome to assert at rebase. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: assert incident 2's late-confirm counter now that #324 is in the base W3c wrote the qualification tests against a base that predated #324 and left a marked place for the assertions that only make sense once the commit gate is there. The base now carries it, so apply them: incident 2 asserts `qbit_prism_late_confirmed_shares_total` is zero on both frontends and that no acknowledgement, the block-solving share included, is `ledger-outcome-unknown`. Acceptance alone is no longer proof. #324 gives a share-pass append carrying a candidate its own `block_only_ack_timeout`, so a slow solve is never refused at the share deadline; it is answered late, or answered unknown. Either would have satisfied the acceptance checks with the serialization stall still present. Also set #324's two new `Config` fields in the qualification fixture. * 3.x.x: pin a refused candidate as a definite rejection, not an unknown outcome #324 decides what a miner is told when a block-only enqueue fails: `enqueue_failed_before_commit` answers a provably uncommitted failure with `ledger-confirmation-failed`, and anything else with `ledger-outcome-unknown`, which asks an operator to go and follow the outbox row. Slice 3 added the checks that guard the reference: the block's length, its header hash, its digest against `block_sha256`, the coinbase suffix, the reference invariants, and a leased candidate's balances. All of them are raised by `prepare_candidate` before the enqueue takes a connection, so none carries an `sqlx::Error` and all classify as definite today. That is a property of where the checks sit, not something the classifier knows about them, so it is pinned here rather than left to hold by accident: a corrupt candidate must never send a miner chasing a row that was never written. Verified by mutation: classifying a non-sqlx failure as unknown fails this test. * 3.x.x: say why the full-size qualification variants cannot reach 400k yet Their ignore reasons blamed RAM and the host. The real limit is #273: the job payload still holds three copies of the window (2 x AcceptedShare + 1 x CountedShare), so publishing a job for a 400,000-share window needs about 670 MiB and PostgreSQL refuses the write above its 256 MiB jsonb object limit. Every full-size variant fails at fixture set-up, before it measures anything. The count must also divide the window weight exactly, so the largest usable size today is 125,000, where all four variants pass in release: incident 1 50 renewals, max gap 118.25 ms against a 700 ms bound incident 2 phase-A p99 2.60 ms, phase-B p99 95.79 ms, bound 250 ms lock hold median 0.75 ms at 5,000 against 1.32 ms at 125,000 canonical refresh and claim rebuild both 139,040,610 B, same sha256 The candidate document stays flat at 1103 B and 1107 B, which is the property this slice exists to establish. * 3.x.x: migrate the audit-body normalization tests to the slim candidate #267 landed its own test file against the candidate that still carried an assembled bundle. This slice replaced that with a window reference, so the file keeps a `TestCandidate` pairing the stored candidate with the bundle a test still needs, as `tests/ledger_postgres.rs` does, and attaches the claim's parts through `with_bundle`. One case needed more than a shape change. The four negative cases hand the claim a window the ledger does not hold and require the durable-range proof to be what refuses it. A candidate now commits to a snapshot digest, so a reference taken from the unaltered snapshot would be refused earlier, at the reference check, and the proof under test would never run. `window_ref_for` therefore derives each altered window's reference from that window, leaving the proof as the only thing that can notice, which is what #267 wrote them to prove. * 3.x.x: pin the durable-range proof outside the settlement lock #267 moved the full-equality proof of a candidate's window out of the settlement transaction so the lock is not held for a window-sized read. This slice made that read paged, so it does not occupy a runtime thread either. Reconciling the two is the step that can quietly undo the first: keep this slice's `persist_audit_snapshot` through the rebase and the full read goes back under the lock. The landing stays correct, every other test still passes, and only the lock hold grows, so nothing in the suite can tell. Measured at 100,000 shares on #267's branch that was a 34% longer hold. The guard is deterministic and needs no large window. A second connection holds `SETTLEMENT_LOCK`, so the landing blocks on it once its proof has run. While it waits, one share's `ntime` is altered: full equality sees that, a count over the same anchored predicate cannot. The landing must then succeed, which it can only do if it had already read and compared the shares before taking the lock. Accepted history is immutable and a trigger enforces it, so the test lifts that trigger for the single statement, inside its own disposable schema, and puts it straight back. Verified by mutation: restoring the in-transaction range read fails this test with "the landing read share payloads under the settlement lock". * 3.x.x: serialise the coordinator test modules that share the ledger's locks The ledger's advisory locks are cluster-wide constants, not schema-scoped, so two coordinator test modules running in parallel contend for ORDER_LOCK even though each works in its own throwaway schema. Every module held a lock of its own, which serialised it against itself and left cross-module waits to `lock_timeout`, 5 s by default. #324's reconciliation tests hold ORDER_LOCK for up to about 1.5 s, and this slice adds three more modules that open a ledger. Once enough of them ran at once those waits stacked past the timeout: a full `--lib` run failed with "canceling statement due to lock timeout" in `candidate_lease_tests` and `window_switch_tests`, and passed on the next run. A suite that fails for a reason unrelated to what it tests is worse than a slow one. All seven modules now take one shared lock. The run goes from about 35 s to about 62 s, and three consecutive runs pass where the flake had reproduced. * 3.x.x: add migration 007 and refuse pre-007 candidate outbox rows Migration 007 mirrors 008 onto qbit_block_candidate_outbox: the same six window-reference columns, and a three-state CHECK whose body is identical to 008's apart from the table and constraint names. It also adds block_bytes, the nullable bytea the block moves into so the candidate document stays small, and a partial index on window_prior_balances_sha256 for the balance-snapshot GC. Nothing writes any of them yet; this is additive schema only. The runner applies 007 by membership, in its own block between 5 and 8, so it assumes neither 006 nor 008 has run. Before any DDL it refuses while the outbox still holds a pending row in a pre-007 shape: a native inline candidate, a NULL candidate, or storage_version other than 1 on the 2.x.x schemas that have that column, which is checked for first so the predicate does not error on a native schema. The refusal names every offending block_hash and the drain procedure, and the whole migration transaction rolls back, so a refused start leaves the schema untouched. The applied set is now [2, 3, 4, 5, 7, 8, 9], and the four exact migration assertions move with it rather than being weakened to a superset. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: retain the cluster fingerprint configure pinned or verified The candidate enqueue and save_job must fence against a fingerprint reset by re-reading qbit_prism_cluster.config_fingerprint FOR SHARE in their own transaction and refusing a write that is not this frontend's own. That needs the value configure settled at startup, so Ledger keeps it in a shared OnceLock, set only once the pin or the match has committed, and exposes it read-only through config_fingerprint(). configure keeps its signature and its mismatch refusal, and a refused call retains nothing. The rotation refusal in configure, and the writer fence itself, belong to the enqueue change and are not here. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: share the window reference, balance writer and page primitives WindowRef::from_snapshot is the synchronous reference builder both writers need: anchor_ms copied, the semantic balances digest, and for a non-empty snapshot the first and last share_seq, the length, and the share array's SHA-256 streamed through the existing DigestWriter, so the 233 to 260 MB serialized window at 400,000 shares is never built and the digest equals the one qbit_prism_audit_snapshots already stores. An empty snapshot gives the empty-window reference. Callers run it inside their own spawn_blocking. put_balance_snapshot is the one canonical writer of qbit_prism_balance_snapshots, on the caller's transaction: the sort by (order_key, recipient_id, p2mr_program_hex), the semantic digest that keys the row, and compact JSON of the sorted set, all off the runtime, then INSERT ... ON CONFLICT DO NOTHING. On a conflict it reads the stored bytes back and compares them with its own, because these rows are immutable and can only disagree through corruption. The enqueue and save_job and its repair all write through it. read_prior_balances and the current-balance read still keep their SQL order; the writer's sort defines the stored encoding, which is the order read_window(..., AsIssued) returns. probe_share_rows and read_range_paged are read_window's endpoint probe and page loop, lifted onto a caller's connection so the enqueue probe and the landing re-read share one implementation. The predicate, the ascending keyset order, the 4096-row page bound and the per-page count bound are unchanged, and read_window now calls them, with its signature and behaviour unchanged. The paged reader takes the caller's state and a per-page consumer, both of which travel to the blocking thread with the rows, so the landing re-read can compare each page with its slice of a vector it already holds without any whole-window work on the runtime; a dropped read still releases the accumulated state on a blocking thread. read_window_with_permit holds the caller's window_reads permit across the whole read. The permit is moved into a BlockingDrop declared before the read starts, so a cancellation releases it on a blocking thread after the page state queued its own release there, and never with the future; a return releases it inline, before the caller's rebuild. WindowError::TaskFailed separates a cancelled or panicking blocking hand-off from corruption. Every hand-off maps a JoinError to it instead of to Decode, so a cancellation is a retryable failure with its own alert and never takes the abandon path. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: derive the witness merkle leaves from an assembled block A claim rebuilds from stored inputs, and the witness leaves are the one builder input that scales with the block, so they are re-derived rather than stored: one 64-hex string per transaction beside the candidate would grow the document with the block. witness_merkle_leaves_from_block skips the 80-byte header, reads the compact-size transaction count, walks past the coinbase and returns double_sha256 of every remaining transaction's full serialized bytes. That is sound because assemble_submission appends the template's transactions verbatim after the coinbase, so the result equals witness_merkle_leaves_hex(&transactions_from_template(template)) for the block built from that template, with or without segwit transactions. The segwit-aware walk is strip_witness_transaction's own, factored into a helper that also reports where the transaction ends, which is what a block walker needs; strip_witness_transaction keeps its trailing-byte check and its output. A truncated block, trailing bytes, a zero count and a count that disagrees with the transactions present are all errors, never a partial list of leaves. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: version the audit builders and key the CC5 goldens by it A stored window reference reproduces its bundle only under the builder that first built it. The schema strings do not capture that, because the share shape chooses them rather than the builder logic, so qbit-prism gains AUDIT_BUILDER_VERSION, starting at 1. A candidate and a prepared job store it, and a rebuild refuses a reference at another version instead of rebuilding it with the current builder. The bump rule is on the constant: any change that alters canonical_audit_bundle_bytes for the same inputs bumps it in the same commit. The CC5 golden test now selects its expected digests from a table keyed by the constant, and fails when no entry exists, so a new output is recorded as a new entry beside the old one and review sees both. Version 1 holds the existing literals, unchanged. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: cover 007, the fingerprint and the window primitives on PostgreSQL Ten gated database cases, all listed in test/prism-gated-tests.txt. They go in a new file rather than the shared 2.x.x support module, which another open change appends to. Migration 007: a frozen 2.x.x schema carrying only terminal outbox rows upgrades with every row and field intact and the seven new columns NULL; each of the three pre-007 pending shapes is refused by name, with nothing from 007 applied and no version 7 row, and the same binary starts once the row is drained; a database at [2, 3, 4, 5, 8, 9] applies 007 and only 007, leaving every other version's applied_at untouched; the CHECK accepts exactly the three states across all 64 null combinations and rejects a short or uppercase digest, a zero or out-of-range count and a zero sequence; and the retention index is the partial one on window_prior_balances_sha256. config_fingerprint() returns what configure pinned or verified, is shared by clones,…
* 3.x.x: add migration 007 and refuse pre-007 candidate outbox rows Migration 007 mirrors 008 onto qbit_block_candidate_outbox: the same six window-reference columns, and a three-state CHECK whose body is identical to 008's apart from the table and constraint names. It also adds block_bytes, the nullable bytea the block moves into so the candidate document stays small, and a partial index on window_prior_balances_sha256 for the balance-snapshot GC. Nothing writes any of them yet; this is additive schema only. The runner applies 007 by membership, in its own block between 5 and 8, so it assumes neither 006 nor 008 has run. Before any DDL it refuses while the outbox still holds a pending row in a pre-007 shape: a native inline candidate, a NULL candidate, or storage_version other than 1 on the 2.x.x schemas that have that column, which is checked for first so the predicate does not error on a native schema. The refusal names every offending block_hash and the drain procedure, and the whole migration transaction rolls back, so a refused start leaves the schema untouched. The applied set is now [2, 3, 4, 5, 7, 8, 9], and the four exact migration assertions move with it rather than being weakened to a superset. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: retain the cluster fingerprint configure pinned or verified The candidate enqueue and save_job must fence against a fingerprint reset by re-reading qbit_prism_cluster.config_fingerprint FOR SHARE in their own transaction and refusing a write that is not this frontend's own. That needs the value configure settled at startup, so Ledger keeps it in a shared OnceLock, set only once the pin or the match has committed, and exposes it read-only through config_fingerprint(). configure keeps its signature and its mismatch refusal, and a refused call retains nothing. The rotation refusal in configure, and the writer fence itself, belong to the enqueue change and are not here. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: share the window reference, balance writer and page primitives WindowRef::from_snapshot is the synchronous reference builder both writers need: anchor_ms copied, the semantic balances digest, and for a non-empty snapshot the first and last share_seq, the length, and the share array's SHA-256 streamed through the existing DigestWriter, so the 233 to 260 MB serialized window at 400,000 shares is never built and the digest equals the one qbit_prism_audit_snapshots already stores. An empty snapshot gives the empty-window reference. Callers run it inside their own spawn_blocking. put_balance_snapshot is the one canonical writer of qbit_prism_balance_snapshots, on the caller's transaction: the sort by (order_key, recipient_id, p2mr_program_hex), the semantic digest that keys the row, and compact JSON of the sorted set, all off the runtime, then INSERT ... ON CONFLICT DO NOTHING. On a conflict it reads the stored bytes back and compares them with its own, because these rows are immutable and can only disagree through corruption. The enqueue and save_job and its repair all write through it. read_prior_balances and the current-balance read still keep their SQL order; the writer's sort defines the stored encoding, which is the order read_window(..., AsIssued) returns. probe_share_rows and read_range_paged are read_window's endpoint probe and page loop, lifted onto a caller's connection so the enqueue probe and the landing re-read share one implementation. The predicate, the ascending keyset order, the 4096-row page bound and the per-page count bound are unchanged, and read_window now calls them, with its signature and behaviour unchanged. The paged reader takes the caller's state and a per-page consumer, both of which travel to the blocking thread with the rows, so the landing re-read can compare each page with its slice of a vector it already holds without any whole-window work on the runtime; a dropped read still releases the accumulated state on a blocking thread. read_window_with_permit holds the caller's window_reads permit across the whole read. The permit is moved into a BlockingDrop declared before the read starts, so a cancellation releases it on a blocking thread after the page state queued its own release there, and never with the future; a return releases it inline, before the caller's rebuild. WindowError::TaskFailed separates a cancelled or panicking blocking hand-off from corruption. Every hand-off maps a JoinError to it instead of to Decode, so a cancellation is a retryable failure with its own alert and never takes the abandon path. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: derive the witness merkle leaves from an assembled block A claim rebuilds from stored inputs, and the witness leaves are the one builder input that scales with the block, so they are re-derived rather than stored: one 64-hex string per transaction beside the candidate would grow the document with the block. witness_merkle_leaves_from_block skips the 80-byte header, reads the compact-size transaction count, walks past the coinbase and returns double_sha256 of every remaining transaction's full serialized bytes. That is sound because assemble_submission appends the template's transactions verbatim after the coinbase, so the result equals witness_merkle_leaves_hex(&transactions_from_template(template)) for the block built from that template, with or without segwit transactions. The segwit-aware walk is strip_witness_transaction's own, factored into a helper that also reports where the transaction ends, which is what a block walker needs; strip_witness_transaction keeps its trailing-byte check and its output. A truncated block, trailing bytes, a zero count and a count that disagrees with the transactions present are all errors, never a partial list of leaves. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: version the audit builders and key the CC5 goldens by it A stored window reference reproduces its bundle only under the builder that first built it. The schema strings do not capture that, because the share shape chooses them rather than the builder logic, so qbit-prism gains AUDIT_BUILDER_VERSION, starting at 1. A candidate and a prepared job store it, and a rebuild refuses a reference at another version instead of rebuilding it with the current builder. The bump rule is on the constant: any change that alters canonical_audit_bundle_bytes for the same inputs bumps it in the same commit. The CC5 golden test now selects its expected digests from a table keyed by the constant, and fails when no entry exists, so a new output is recorded as a new entry beside the old one and review sees both. Version 1 holds the existing literals, unchanged. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: cover 007, the fingerprint and the window primitives on PostgreSQL Ten gated database cases, all listed in test/prism-gated-tests.txt. They go in a new file rather than the shared 2.x.x support module, which another open change appends to. Migration 007: a frozen 2.x.x schema carrying only terminal outbox rows upgrades with every row and field intact and the seven new columns NULL; each of the three pre-007 pending shapes is refused by name, with nothing from 007 applied and no version 7 row, and the same binary starts once the row is drained; a database at [2, 3, 4, 5, 8, 9] applies 007 and only 007, leaving every other version's applied_at untouched; the CHECK accepts exactly the three states across all 64 null combinations and rejects a short or uppercase digest, a zero or out-of-range count and a zero sequence; and the retention index is the partial one on window_prior_balances_sha256. config_fingerprint() returns what configure pinned or verified, is shared by clones, and stays empty after a refused configure. from_snapshot's streamed digest equals sha256(serde_json::to_vec) and the snapshot_sha256 landing writes for the same window. The balance writer stores the canonical sort, is idempotent under any permutation, errors rather than accepting a different encoding of the same digest, and read_window(..., AsIssued) hands back the stored order. The probe misses a missing endpoint, the paged reader compares 4,999 rows page by page with a vector held inside the caller's transaction, and a row the predicate skips inside a range is Incomplete with no partial window. The permit case builds a runtime with one blocking thread, occupies it, and cancels a reader parked mid-page on a database gate: no permit is available while the cleanup is queued, and one is once it has run. A failed blocking hand-off surfaces as TaskFailed, never as Decode. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: carry the window reference on prepared work, behind a read permit `Prepared` gains `window: WindowRef`, computed once per non-cached refresh beside `build_bundle` under `tokio::join!`, so submit can clone it instead of re-digesting the window on the share path when a block is found. An empty snapshot gets the empty-window reference on the runtime thread; a resume computes it inside the blocking decode it already runs. `Coordinator` gains the `window_reads` semaphore, sized `clamp(database_max_connections - 2, 1, build_workers)`, and the one entry point callers use: it takes an owned permit after the caller's `build_slots` permit and hands it to `Ledger::read_window_with_permit`, which holds it across the whole read, including the off-runtime cleanup a cancellation leaves behind, and releases it before the rebuild. `Ledger::read_window` is awaited on the runtime, never wrapped in `spawn_blocking`; the test with a single blocking thread pins that. The four gated `window_ref_tests` are listed in the manifest. Part of #265 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * 3.x.x: switch the candidate outbox to the window reference `Candidate` drops the embedded `AuditBundle` for `window: WindowRef` and every builder input except the window: `found_block`, `payout_policy`, the nested `ctv` triple (whose presence selects the CTV builder), the stored `bootstrap_share` of an empty window, `audit_builder_version`, `signer_keys`, a `leased` flag and the now-required `coinbase_suffix_hex`. The block leaves the document for the `block_bytes` column, authenticated by `block_sha256`, which `candidate_sha256` covers. The write serializes, digests and checks the candidate before the append or enqueue transaction opens; under `ORDER_LOCK` the transaction re-reads `config_fingerprint FOR SHARE` and refuses a value the writer did not pin, re-inserts a `leased` candidate's as-issued balance snapshot through the shared writer, probes the window's first row (an absent row alerts and still publishes), and inserts only the prepared bytes with the six typed columns. Submit builds the slim candidate from `Prepared.window`, the `BundleInputs` captured where `build_bundle` read configuration, and the job's own bootstrap share; it clones no bundle and reads no config. The claim's `UPDATE … RETURNING` returns the columns and the block, and the decode, off the runtime, checks the document against them, the digest, `sha256(block_bytes)`, the header hash, the required fields and the two reference invariants. A pending row with a NULL anchor is a hard error naming the pre-007 frontend; an inline document is never decoded. Before any read the claim retries, with an alert naming both values, a stored builder version or signer pair that is not this binary's; a landed audit is authenticated against the block (coinbase, audit root as the witness reserved value, share snapshot, bits) and the claim then skips the rebuild and landing and continues to observe, renew and submit. Otherwise it takes `build_slots`, then `window_reads` inside one 60 s deadline around the awaited `read_window(Current)` and the borrowing builder run directly in `spawn_blocking`, and keeps `Arc<AuditBundleBody>` and the shares as `ClaimParts`. `Incomplete`, `SnapshotDigestMismatch`, `Decode`, `BalanceSnapshotMissing`, `TaskFailed` and the deadline retry with their own alerts; `PriorBalancesChanged` finishes only a block `observe_candidate` proves inactive and superseded; `Database` propagates. A `leased` candidate submits from `block_bytes` first, then rebuilds `AsIssued` and lands before any terminal outcome. Rebuilt parts are released off the runtime on every path. Landing takes the parts and does all whole-window work in one blocking closure: `verify_audit_parts`, the raw body, the counted canonical bytes, the leaves and accounts, and the share snapshot digest, which must equal the reference's. The balance fence compares the current set and the parts' through the semantic digest, so an as-issued set compares equal in any order. `persist_audit_snapshot` re-reads the range with the paged reader inside the landing transaction, page against slice on the blocking thread; `read_range` stays for the API. The terminal UPDATE NULLs the six window columns and `block_bytes` with `candidate`. `configure` refuses to pin a fingerprint onto a reset one while a pending row stores other signer keys, and the startup gate requires 007 with 009. The JSONB gate drops its enqueue entry, and every helper that built a candidate builds the slim one, handing the ledger-level landings the bundle's parts through `CandidateClaim::with_bundle`. Part of #265 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * 3.x.x: cover the candidate switch on PostgreSQL Ledger-level, in `candidate_window_switch`: a two-page window enqueues a document with no array and no share row, under 1 MB, whose six typed columns equal its reference and whose block is stored as bytes; the writer fence refuses a reset or different fingerprint in the same transaction, share included; a missing window prefix row alerts and still publishes, and its read then meets `Incomplete`; a `leased` enqueue re-inserts the as-issued balance snapshot in the writer's canonical order, writes none when not leased, and refuses balances that do not hash to the reference; every preparation failure is refused without waiting for `ORDER_LOCK`; every column, digest, block, header, NULL-anchor, missing-field, invariant and inline-document disagreement is refused at claim; landing refuses parts that are not the referenced window and a claim without parts; the terminal UPDATE NULLs the six columns and the block for submitted and abandoned rows; the landed-audit authentication rejects a forged coinbase, root, snapshot or bits; and `configure` refuses a rotation over a foreign-signed pending candidate. Coordinator-level, in `window_switch_tests` behind a fake node: a stored builder version or signer pair that is not this binary's retries, naming both values, while the build slot is held; a landed audit is authenticated and the claim continues to `submitblock` without a build slot, and a forged landed coinbase or root is refused with the row left pending; `Incomplete`, a digest mismatch, changed balances on a current block and the rebuild deadline all retry with an alert and release their permits; `classify_window_error` keeps `TaskFailed` apart from corruption and propagates `Database`; the read, rebuild and landing complete while `ORDER_LOCK` is held, with one build worker; a `leased` candidate submits first, then lands from `AsIssued`, and finishes submitted, abandoned with the node's reason, or pending when its balance snapshot is missing; and `OffRuntime` releases on a blocking thread, on cancellation too. The fifteen gated tests are listed in the manifest. Part of #265 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * 3.x.x: migrate the imported-audit tests to the slim candidate Slice 1 (#325) merged after this stack branched, and its tests in tests/support/ledger_2x.rs still built the pre-007 candidate: they read block.bundle from a Candidate and called candidate_with_bundle without a window reference or settlement inputs. The rebase merged them cleanly and the tree then failed to compile. The three helpers now take the TestCandidate the constructors return, so they reach the bundle beside the slim candidate, and land_confirmed attaches the claim's parts the way the other ledger tests do. The three constructor calls pass the window reference the bundle was built on, and the settlement inputs for the CTV case. No assertion changes. Part of #265. * 3.x.x: qualify lease renewal and the claim's canonical bytes at scale Incident 1: a candidate whose window holds N shares is claimed and processed under the short lease, and every renewal is read back from claim_expires_at. The largest gap stays below interval + 2 x timeout, the lease never lapses, and a successor ledger cannot take the claim. The bundle a claim rebuilds through the window-read entry point and the borrowing builder has the canonical bytes refresh built for the same job, and the landed audit carries that bundle's digest. Both run at PRISM_WINDOW_QUALIFY_SHARES (20,000 by default); the 400,000-share variants are ignored and run on a larger host. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: measure ACK latency and ORDER_LOCK hold across window sizes Incident 2: two frontends on one database submit ordinary shares at a fixed rate while one of them solves a block on an N-share window and its own submit loop claims, rebuilds, lands and submits it. The solving share's ACK and the bystander's p99 stay within max(2 x baseline p99 + 50 ms, 250 ms), every ACK is well under the share-commit timeout, and the block is enqueued exactly once. Latency is the in-process MiningBackend::submit wall clock; the socket-level harness (#271) does not exist yet. A pg_locks sampler on its own connection measures how long a solving append holds ORDER_LOCK at two window sizes with no other traffic: the larger size stays within a fixed slack of the smaller, and the candidate document is the same size at both and under 1 MiB. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: hold the JSONB gate's enqueue to a window-free row under 1 MiB At every measured size the enqueue must be accepted and every JSONB value it writes, the outbox candidate document included, must stay under 1,048,576 bytes. At the reduced sizes the enqueue, claim and landing phases must all run. The ratchet and its expected values are unchanged. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: make the window-read permit test actually wait before it looks The delay that gives an unpermitted read time to finish was an empty async block, so the check that the read had not finished ran at once and passed whether or not the entry point took a permit. It now sleeps 500 ms inside a 5 s bound, and fails when the permit acquisition is removed from the entry point. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: rebuild a CTV candidate from its stored settlement inputs No host-runnable test reached the CTV branch of the claim rebuild. A CTV-enabled frontend now claims a candidate whose stored direct floor differs from its configuration, and builds a different coinbase than the configured floor would, and the claim must land that candidate's own audit and submit it. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: pin the SQL order of the current-balance reads Nothing defended the ruling that the current-balance path keeps its SQL order: every fixture's balances already arrived in byte-wise canonical order. The carry table's text columns now take an ICU collation, under which the SQL order and the canonical sort disagree, and the snapshot and the current-balance window read must both return the SQL order. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: record what incident 2 must also assert once #324 is in the base After #324 a slow solve is acknowledged late rather than refused, so acceptance proves nothing and the latency percentiles stay the assertion. The doc comment and a TODO at the outcome checks name the counter and outcome to assert at rebase. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: assert incident 2's late-confirm counter now that #324 is in the base W3c wrote the qualification tests against a base that predated #324 and left a marked place for the assertions that only make sense once the commit gate is there. The base now carries it, so apply them: incident 2 asserts `qbit_prism_late_confirmed_shares_total` is zero on both frontends and that no acknowledgement, the block-solving share included, is `ledger-outcome-unknown`. Acceptance alone is no longer proof. #324 gives a share-pass append carrying a candidate its own `block_only_ack_timeout`, so a slow solve is never refused at the share deadline; it is answered late, or answered unknown. Either would have satisfied the acceptance checks with the serialization stall still present. Also set #324's two new `Config` fields in the qualification fixture. * 3.x.x: pin a refused candidate as a definite rejection, not an unknown outcome #324 decides what a miner is told when a block-only enqueue fails: `enqueue_failed_before_commit` answers a provably uncommitted failure with `ledger-confirmation-failed`, and anything else with `ledger-outcome-unknown`, which asks an operator to go and follow the outbox row. Slice 3 added the checks that guard the reference: the block's length, its header hash, its digest against `block_sha256`, the coinbase suffix, the reference invariants, and a leased candidate's balances. All of them are raised by `prepare_candidate` before the enqueue takes a connection, so none carries an `sqlx::Error` and all classify as definite today. That is a property of where the checks sit, not something the classifier knows about them, so it is pinned here rather than left to hold by accident: a corrupt candidate must never send a miner chasing a row that was never written. Verified by mutation: classifying a non-sqlx failure as unknown fails this test. * 3.x.x: say why the full-size qualification variants cannot reach 400k yet Their ignore reasons blamed RAM and the host. The real limit is #273: the job payload still holds three copies of the window (2 x AcceptedShare + 1 x CountedShare), so publishing a job for a 400,000-share window needs about 670 MiB and PostgreSQL refuses the write above its 256 MiB jsonb object limit. Every full-size variant fails at fixture set-up, before it measures anything. The count must also divide the window weight exactly, so the largest usable size today is 125,000, where all four variants pass in release: incident 1 50 renewals, max gap 118.25 ms against a 700 ms bound incident 2 phase-A p99 2.60 ms, phase-B p99 95.79 ms, bound 250 ms lock hold median 0.75 ms at 5,000 against 1.32 ms at 125,000 canonical refresh and claim rebuild both 139,040,610 B, same sha256 The candidate document stays flat at 1103 B and 1107 B, which is the property this slice exists to establish. * 3.x.x: migrate the audit-body normalization tests to the slim candidate #267 landed its own test file against the candidate that still carried an assembled bundle. This slice replaced that with a window reference, so the file keeps a `TestCandidate` pairing the stored candidate with the bundle a test still needs, as `tests/ledger_postgres.rs` does, and attaches the claim's parts through `with_bundle`. One case needed more than a shape change. The four negative cases hand the claim a window the ledger does not hold and require the durable-range proof to be what refuses it. A candidate now commits to a snapshot digest, so a reference taken from the unaltered snapshot would be refused earlier, at the reference check, and the proof under test would never run. `window_ref_for` therefore derives each altered window's reference from that window, leaving the proof as the only thing that can notice, which is what #267 wrote them to prove. * 3.x.x: pin the durable-range proof outside the settlement lock #267 moved the full-equality proof of a candidate's window out of the settlement transaction so the lock is not held for a window-sized read. This slice made that read paged, so it does not occupy a runtime thread either. Reconciling the two is the step that can quietly undo the first: keep this slice's `persist_audit_snapshot` through the rebase and the full read goes back under the lock. The landing stays correct, every other test still passes, and only the lock hold grows, so nothing in the suite can tell. Measured at 100,000 shares on #267's branch that was a 34% longer hold. The guard is deterministic and needs no large window. A second connection holds `SETTLEMENT_LOCK`, so the landing blocks on it once its proof has run. While it waits, one share's `ntime` is altered: full equality sees that, a count over the same anchored predicate cannot. The landing must then succeed, which it can only do if it had already read and compared the shares before taking the lock. Accepted history is immutable and a trigger enforces it, so the test lifts that trigger for the single statement, inside its own disposable schema, and puts it straight back. Verified by mutation: restoring the in-transaction range read fails this test with "the landing read share payloads under the settlement lock". * 3.x.x: serialise the coordinator test modules that share the ledger's locks The ledger's advisory locks are cluster-wide constants, not schema-scoped, so two coordinator test modules running in parallel contend for ORDER_LOCK even though each works in its own throwaway schema. Every module held a lock of its own, which serialised it against itself and left cross-module waits to `lock_timeout`, 5 s by default. #324's reconciliation tests hold ORDER_LOCK for up to about 1.5 s, and this slice adds three more modules that open a ledger. Once enough of them ran at once those waits stacked past the timeout: a full `--lib` run failed with "canceling statement due to lock timeout" in `candidate_lease_tests` and `window_switch_tests`, and passed on the next run. A suite that fails for a reason unrelated to what it tests is worse than a slow one. All seven modules now take one shared lock. The run goes from about 35 s to about 62 s, and three consecutive runs pass where the flake had reproduced. * 3.x.x: add migration 007 and refuse pre-007 candidate outbox rows Migration 007 mirrors 008 onto qbit_block_candidate_outbox: the same six window-reference columns, and a three-state CHECK whose body is identical to 008's apart from the table and constraint names. It also adds block_bytes, the nullable bytea the block moves into so the candidate document stays small, and a partial index on window_prior_balances_sha256 for the balance-snapshot GC. Nothing writes any of them yet; this is additive schema only. The runner applies 007 by membership, in its own block between 5 and 8, so it assumes neither 006 nor 008 has run. Before any DDL it refuses while the outbox still holds a pending row in a pre-007 shape: a native inline candidate, a NULL candidate, or storage_version other than 1 on the 2.x.x schemas that have that column, which is checked for first so the predicate does not error on a native schema. The refusal names every offending block_hash and the drain procedure, and the whole migration transaction rolls back, so a refused start leaves the schema untouched. The applied set is now [2, 3, 4, 5, 7, 8, 9], and the four exact migration assertions move with it rather than being weakened to a superset. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: retain the cluster fingerprint configure pinned or verified The candidate enqueue and save_job must fence against a fingerprint reset by re-reading qbit_prism_cluster.config_fingerprint FOR SHARE in their own transaction and refusing a write that is not this frontend's own. That needs the value configure settled at startup, so Ledger keeps it in a shared OnceLock, set only once the pin or the match has committed, and exposes it read-only through config_fingerprint(). configure keeps its signature and its mismatch refusal, and a refused call retains nothing. The rotation refusal in configure, and the writer fence itself, belong to the enqueue change and are not here. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: share the window reference, balance writer and page primitives WindowRef::from_snapshot is the synchronous reference builder both writers need: anchor_ms copied, the semantic balances digest, and for a non-empty snapshot the first and last share_seq, the length, and the share array's SHA-256 streamed through the existing DigestWriter, so the 233 to 260 MB serialized window at 400,000 shares is never built and the digest equals the one qbit_prism_audit_snapshots already stores. An empty snapshot gives the empty-window reference. Callers run it inside their own spawn_blocking. put_balance_snapshot is the one canonical writer of qbit_prism_balance_snapshots, on the caller's transaction: the sort by (order_key, recipient_id, p2mr_program_hex), the semantic digest that keys the row, and compact JSON of the sorted set, all off the runtime, then INSERT ... ON CONFLICT DO NOTHING. On a conflict it reads the stored bytes back and compares them with its own, because these rows are immutable and can only disagree through corruption. The enqueue and save_job and its repair all write through it. read_prior_balances and the current-balance read still keep their SQL order; the writer's sort defines the stored encoding, which is the order read_window(..., AsIssued) returns. probe_share_rows and read_range_paged are read_window's endpoint probe and page loop, lifted onto a caller's connection so the enqueue probe and the landing re-read share one implementation. The predicate, the ascending keyset order, the 4096-row page bound and the per-page count bound are unchanged, and read_window now calls them, with its signature and behaviour unchanged. The paged reader takes the caller's state and a per-page consumer, both of which travel to the blocking thread with the rows, so the landing re-read can compare each page with its slice of a vector it already holds without any whole-window work on the runtime; a dropped read still releases the accumulated state on a blocking thread. read_window_with_permit holds the caller's window_reads permit across the whole read. The permit is moved into a BlockingDrop declared before the read starts, so a cancellation releases it on a blocking thread after the page state queued its own release there, and never with the future; a return releases it inline, before the caller's rebuild. WindowError::TaskFailed separates a cancelled or panicking blocking hand-off from corruption. Every hand-off maps a JoinError to it instead of to Decode, so a cancellation is a retryable failure with its own alert and never takes the abandon path. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: derive the witness merkle leaves from an assembled block A claim rebuilds from stored inputs, and the witness leaves are the one builder input that scales with the block, so they are re-derived rather than stored: one 64-hex string per transaction beside the candidate would grow the document with the block. witness_merkle_leaves_from_block skips the 80-byte header, reads the compact-size transaction count, walks past the coinbase and returns double_sha256 of every remaining transaction's full serialized bytes. That is sound because assemble_submission appends the template's transactions verbatim after the coinbase, so the result equals witness_merkle_leaves_hex(&transactions_from_template(template)) for the block built from that template, with or without segwit transactions. The segwit-aware walk is strip_witness_transaction's own, factored into a helper that also reports where the transaction ends, which is what a block walker needs; strip_witness_transaction keeps its trailing-byte check and its output. A truncated block, trailing bytes, a zero count and a count that disagrees with the transactions present are all errors, never a partial list of leaves. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: version the audit builders and key the CC5 goldens by it A stored window reference reproduces its bundle only under the builder that first built it. The schema strings do not capture that, because the share shape chooses them rather than the builder logic, so qbit-prism gains AUDIT_BUILDER_VERSION, starting at 1. A candidate and a prepared job store it, and a rebuild refuses a reference at another version instead of rebuilding it with the current builder. The bump rule is on the constant: any change that alters canonical_audit_bundle_bytes for the same inputs bumps it in the same commit. The CC5 golden test now selects its expected digests from a table keyed by the constant, and fails when no entry exists, so a new output is recorded as a new entry beside the old one and review sees both. Version 1 holds the existing literals, unchanged. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: cover 007, the fingerprint and the window primitives on PostgreSQL Ten gated database cases, all listed in test/prism-gated-tests.txt. They go in a new file rather than the shared 2.x.x support module, which another open change appends to. Migration 007: a frozen 2.x.x schema carrying only terminal outbox rows upgrades with every row and field intact and the seven new columns NULL; each of the three pre-007 pending shapes is refused by name, with nothing from 007 applied and no version 7 row, and the same binary starts once the row is drained; a database at [2, 3, 4, 5, 8, 9] applies 007 and only 007, leaving every other version's applied_at untouched; the CHECK accepts exactly the three states across all 64 null combinations and rejects a short or uppercase digest, a zero or out-of-range count and a zero sequence; and the retention index is the partial one on window_prior_balances_sha256. config_fingerprint() returns what configure pinned or verified, is shared by clones, and stays empty after a refused configure. from_snapshot's streamed digest equals sha256(serde_json::to_vec) and the snapshot_sha256 landing writes for the same window. The balance writer stores the canonical sort, is idempotent under any permutation, errors rather than accepting a different encoding of the same digest, and read_window(..., AsIssued) hands back the stored order. The probe misses a missing endpoint, the paged reader compares 4,999 rows page by page with a vector held inside the caller's transaction, and a row the predicate skips inside a range is Incomplete with no partial window. The permit case builds a runtime with one blocking thread, occupies it, and cancels a reader parked mid-page on a database gate: no permit is available while the cleanup is queued, and one is once it has run. A failed blocking hand-off surfaces as TaskFailed, never as Decode. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: carry the window reference on prepared work, behind a read permit `Prepared` gains `window: WindowRef`, computed once per non-cached refresh beside `build_bundle` under `tokio::join!`, so submit can clone it instead of re-digesting the window on the share path when a block is found. An empty snapshot gets the empty-window reference on the runtime thread; a resume computes it inside the blocking decode it already runs. `Coordinator` gains the `window_reads` semaphore, sized `clamp(database_max_connections - 2, 1, build_workers)`, and the one entry point callers use: it takes an owned permit after the caller's `build_slots` permit and hands it to `Ledger::read_window_with_permit`, which holds it across the whole read, including the off-runtime cleanup a cancellation leaves behind, and releases it before the rebuild. `Ledger::read_window` is awaited on the runtime, never wrapped in `spawn_blocking`; the test with a single blocking thread pins that. The four gated `window_ref_tests` are listed in the manifest. Part of #265 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * 3.x.x: switch the candidate outbox to the window reference `Candidate` drops the embedded `AuditBundle` for `window: WindowRef` and every builder input except the window: `found_block`, `payout_policy`, the nested `ctv` triple (whose presence selects the CTV builder), the stored `bootstrap_share` of an empty window, `audit_builder_version`, `signer_keys`, a `leased` flag and the now-required `coinbase_suffix_hex`. The block leaves the document for the `block_bytes` column, authenticated by `block_sha256`, which `candidate_sha256` covers. The write serializes, digests and checks the candidate before the append or enqueue transaction opens; under `ORDER_LOCK` the transaction re-reads `config_fingerprint FOR SHARE` and refuses a value the writer did not pin, re-inserts a `leased` candidate's as-issued balance snapshot through the shared writer, probes the window's first row (an absent row alerts and still publishes), and inserts only the prepared bytes with the six typed columns. Submit builds the slim candidate from `Prepared.window`, the `BundleInputs` captured where `build_bundle` read configuration, and the job's own bootstrap share; it clones no bundle and reads no config. The claim's `UPDATE … RETURNING` returns the columns and the block, and the decode, off the runtime, checks the document against them, the digest, `sha256(block_bytes)`, the header hash, the required fields and the two reference invariants. A pending row with a NULL anchor is a hard error naming the pre-007 frontend; an inline document is never decoded. Before any read the claim retries, with an alert naming both values, a stored builder version or signer pair that is not this binary's; a landed audit is authenticated against the block (coinbase, audit root as the witness reserved value, share snapshot, bits) and the claim then skips the rebuild and landing and continues to observe, renew and submit. Otherwise it takes `build_slots`, then `window_reads` inside one 60 s deadline around the awaited `read_window(Current)` and the borrowing builder run directly in `spawn_blocking`, and keeps `Arc<AuditBundleBody>` and the shares as `ClaimParts`. `Incomplete`, `SnapshotDigestMismatch`, `Decode`, `BalanceSnapshotMissing`, `TaskFailed` and the deadline retry with their own alerts; `PriorBalancesChanged` finishes only a block `observe_candidate` proves inactive and superseded; `Database` propagates. A `leased` candidate submits from `block_bytes` first, then rebuilds `AsIssued` and lands before any terminal outcome. Rebuilt parts are released off the runtime on every path. Landing takes the parts and does all whole-window work in one blocking closure: `verify_audit_parts`, the raw body, the counted canonical bytes, the leaves and accounts, and the share snapshot digest, which must equal the reference's. The balance fence compares the current set and the parts' through the semantic digest, so an as-issued set compares equal in any order. `persist_audit_snapshot` re-reads the range with the paged reader inside the landing transaction, page against slice on the blocking thread; `read_range` stays for the API. The terminal UPDATE NULLs the six window columns and `block_bytes` with `candidate`. `configure` refuses to pin a fingerprint onto a reset one while a pending row stores other signer keys, and the startup gate requires 007 with 009. The JSONB gate drops its enqueue entry, and every helper that built a candidate builds the slim one, handing the ledger-level landings the bundle's parts through `CandidateClaim::with_bundle`. Part of #265 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * 3.x.x: cover the candidate switch on PostgreSQL Ledger-level, in `candidate_window_switch`: a two-page window enqueues a document with no array and no share row, under 1 MB, whose six typed columns equal its reference and whose block is stored as bytes; the writer fence refuses a reset or different fingerprint in the same transaction, share included; a missing window prefix row alerts and still publishes, and its read then meets `Incomplete`; a `leased` enqueue re-inserts the as-issued balance snapshot in the writer's canonical order, writes none when not leased, and refuses balances that do not hash to the reference; every preparation failure is refused without waiting for `ORDER_LOCK`; every column, digest, block, header, NULL-anchor, missing-field, invariant and inline-document disagreement is refused at claim; landing refuses parts that are not the referenced window and a claim without parts; the terminal UPDATE NULLs the six columns and the block for submitted and abandoned rows; the landed-audit authentication rejects a forged coinbase, root, snapshot or bits; and `configure` refuses a rotation over a foreign-signed pending candidate. Coordinator-level, in `window_switch_tests` behind a fake node: a stored builder version or signer pair that is not this binary's retries, naming both values, while the build slot is held; a landed audit is authenticated and the claim continues to `submitblock` without a build slot, and a forged landed coinbase or root is refused with the row left pending; `Incomplete`, a digest mismatch, changed balances on a current block and the rebuild deadline all retry with an alert and release their permits; `classify_window_error` keeps `TaskFailed` apart from corruption and propagates `Database`; the read, rebuild and landing complete while `ORDER_LOCK` is held, with one build worker; a `leased` candidate submits first, then lands from `AsIssued`, and finishes submitted, abandoned with the node's reason, or pending when its balance snapshot is missing; and `OffRuntime` releases on a blocking thread, on cancellation too. The fifteen gated tests are listed in the manifest. Part of #265 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * 3.x.x: migrate the imported-audit tests to the slim candidate Slice 1 (#325) merged after this stack branched, and its tests in tests/support/ledger_2x.rs still built the pre-007 candidate: they read block.bundle from a Candidate and called candidate_with_bundle without a window reference or settlement inputs. The rebase merged them cleanly and the tree then failed to compile. The three helpers now take the TestCandidate the constructors return, so they reach the bundle beside the slim candidate, and land_confirmed attaches the claim's parts the way the other ledger tests do. The three constructor calls pass the window reference the bundle was built on, and the settlement inputs for the CTV case. No assertion changes. Part of #265. * 3.x.x: qualify lease renewal and the claim's canonical bytes at scale Incident 1: a candidate whose window holds N shares is claimed and processed under the short lease, and every renewal is read back from claim_expires_at. The largest gap stays below interval + 2 x timeout, the lease never lapses, and a successor ledger cannot take the claim. The bundle a claim rebuilds through the window-read entry point and the borrowing builder has the canonical bytes refresh built for the same job, and the landed audit carries that bundle's digest. Both run at PRISM_WINDOW_QUALIFY_SHARES (20,000 by default); the 400,000-share variants are ignored and run on a larger host. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: measure ACK latency and ORDER_LOCK hold across window sizes Incident 2: two frontends on one database submit ordinary shares at a fixed rate while one of them solves a block on an N-share window and its own submit loop claims, rebuilds, lands and submits it. The solving share's ACK and the bystander's p99 stay within max(2 x baseline p99 + 50 ms, 250 ms), every ACK is well under the share-commit timeout, and the block is enqueued exactly once. Latency is the in-process MiningBackend::submit wall clock; the socket-level harness (#271) does not exist yet. A pg_locks sampler on its own connection measures how long a solving append holds ORDER_LOCK at two window sizes with no other traffic: the larger size stays within a fixed slack of the smaller, and the candidate document is the same size at both and under 1 MiB. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: hold the JSONB gate's enqueue to a window-free row under 1 MiB At every measured size the enqueue must be accepted and every JSONB value it writes, the outbox candidate document included, must stay under 1,048,576 bytes. At the reduced sizes the enqueue, claim and landing phases must all run. The ratchet and its expected values are unchanged. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: make the window-read permit test actually wait before it looks The delay that gives an unpermitted read time to finish was an empty async block, so the check that the read had not finished ran at once and passed whether or not the entry point took a permit. It now sleeps 500 ms inside a 5 s bound, and fails when the permit acquisition is removed from the entry point. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: rebuild a CTV candidate from its stored settlement inputs No host-runnable test reached the CTV branch of the claim rebuild. A CTV-enabled frontend now claims a candidate whose stored direct floor differs from its configuration, and builds a different coinbase than the configured floor would, and the claim must land that candidate's own audit and submit it. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: pin the SQL order of the current-balance reads Nothing defended the ruling that the current-balance path keeps its SQL order: every fixture's balances already arrived in byte-wise canonical order. The carry table's text columns now take an ICU collation, under which the SQL order and the canonical sort disagree, and the snapshot and the current-balance window read must both return the SQL order. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: record what incident 2 must also assert once #324 is in the base After #324 a slow solve is acknowledged late rather than refused, so acceptance proves nothing and the latency percentiles stay the assertion. The doc comment and a TODO at the outcome checks name the counter and outcome to assert at rebase. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: assert incident 2's late-confirm counter now that #324 is in the base W3c wrote the qualification tests against a base that predated #324 and left a marked place for the assertions that only make sense once the commit gate is there. The base now carries it, so apply them: incident 2 asserts `qbit_prism_late_confirmed_shares_total` is zero on both frontends and that no acknowledgement, the block-solving share included, is `ledger-outcome-unknown`. Acceptance alone is no longer proof. #324 gives a share-pass append carrying a candidate its own `block_only_ack_timeout`, so a slow solve is never refused at the share deadline; it is answered late, or answered unknown. Either would have satisfied the acceptance checks with the serialization stall still present. Also set #324's two new `Config` fields in the qualification fixture. * 3.x.x: pin a refused candidate as a definite rejection, not an unknown outcome #324 decides what a miner is told when a block-only enqueue fails: `enqueue_failed_before_commit` answers a provably uncommitted failure with `ledger-confirmation-failed`, and anything else with `ledger-outcome-unknown`, which asks an operator to go and follow the outbox row. Slice 3 added the checks that guard the reference: the block's length, its header hash, its digest against `block_sha256`, the coinbase suffix, the reference invariants, and a leased candidate's balances. All of them are raised by `prepare_candidate` before the enqueue takes a connection, so none carries an `sqlx::Error` and all classify as definite today. That is a property of where the checks sit, not something the classifier knows about them, so it is pinned here rather than left to hold by accident: a corrupt candidate must never send a miner chasing a row that was never written. Verified by mutation: classifying a non-sqlx failure as unknown fails this test. * 3.x.x: say why the full-size qualification variants cannot reach 400k yet Their ignore reasons blamed RAM and the host. The real limit is #273: the job payload still holds three copies of the window (2 x AcceptedShare + 1 x CountedShare), so publishing a job for a 400,000-share window needs about 670 MiB and PostgreSQL refuses the write above its 256 MiB jsonb object limit. Every full-size variant fails at fixture set-up, before it measures anything. The count must also divide the window weight exactly, so the largest usable size today is 125,000, where all four variants pass in release: incident 1 50 renewals, max gap 118.25 ms against a 700 ms bound incident 2 phase-A p99 2.60 ms, phase-B p99 95.79 ms, bound 250 ms lock hold median 0.75 ms at 5,000 against 1.32 ms at 125,000 canonical refresh and claim rebuild both 139,040,610 B, same sha256 The candidate document stays flat at 1103 B and 1107 B, which is the property this slice exists to establish. * 3.x.x: migrate the audit-body normalization tests to the slim candidate #267 landed its own test file against the candidate that still carried an assembled bundle. This slice replaced that with a window reference, so the file keeps a `TestCandidate` pairing the stored candidate with the bundle a test still needs, as `tests/ledger_postgres.rs` does, and attaches the claim's parts through `with_bundle`. One case needed more than a shape change. The four negative cases hand the claim a window the ledger does not hold and require the durable-range proof to be what refuses it. A candidate now commits to a snapshot digest, so a reference taken from the unaltered snapshot would be refused earlier, at the reference check, and the proof under test would never run. `window_ref_for` therefore derives each altered window's reference from that window, leaving the proof as the only thing that can notice, which is what #267 wrote them to prove. * 3.x.x: pin the durable-range proof outside the settlement lock #267 moved the full-equality proof of a candidate's window out of the settlement transaction so the lock is not held for a window-sized read. This slice made that read paged, so it does not occupy a runtime thread either. Reconciling the two is the step that can quietly undo the first: keep this slice's `persist_audit_snapshot` through the rebase and the full read goes back under the lock. The landing stays correct, every other test still passes, and only the lock hold grows, so nothing in the suite can tell. Measured at 100,000 shares on #267's branch that was a 34% longer hold. The guard is deterministic and needs no large window. A second connection holds `SETTLEMENT_LOCK`, so the landing blocks on it once its proof has run. While it waits, one share's `ntime` is altered: full equality sees that, a count over the same anchored predicate cannot. The landing must then succeed, which it can only do if it had already read and compared the shares before taking the lock. Accepted history is immutable and a trigger enforces it, so the test lifts that trigger for the single statement, inside its own disposable schema, and puts it straight back. Verified by mutation: restoring the in-transaction range read fails this test with "the landing read share payloads under the settlement lock". * 3.x.x: serialise the coordinator test modules that share the ledger's locks The ledger's advisory locks are cluster-wide constants, not schema-scoped, so two coordinator test modules running in parallel contend for ORDER_LOCK even though each works in its own throwaway schema. Every module held a lock of its own, which serialised it against itself and left cross-module waits to `lock_timeout`, 5 s by default. #324's reconciliation tests hold ORDER_LOCK for up to about 1.5 s, and this slice adds three more modules that open a ledger. Once enough of them ran at once those waits stacked past the timeout: a full `--lib` run failed with "canceling statement due to lock timeout" in `candidate_lease_tests` and `window_switch_tests`, and passed on the next run. A suite that fails for a reason unrelated to what it tests is worse than a slow one. All seven modules now take one shared lock. The run goes from about 35 s to about 62 s, and three consecutive runs pass where the flake had reproduced. * 3.x.x: pass the signer keys #334's wait-metrics tests now need `Ledger::configure` takes the frontend's signing keys since this slice, so the fingerprint contention tests #334 added call it with a fixed pair. That suite measures advisory-lock and pool-acquire waits, not signing, so the only thing the keys have to be is the same on every call. * feat: add compact prepared storage and immutable template blobs * Validate compact prepared endpoints and document caller contracts * 3.x.x: add migration 007 and refuse pre-007 candidate outbox rows Migration 007 mirrors 008 onto qbit_block_candidate_outbox: the same six window-reference columns, and a three-state CHECK whose body is identical to 008's apart from the table and constraint names. It also adds block_bytes, the nullable bytea the block moves into so the candidate document stays small, and a partial index on window_prior_balances_sha256 for the balance-snapshot GC. Nothing writes any of them yet; this is additive schema only. The runner applies 007 by membership, in its own block between 5 and 8, so it assumes neither 006 nor 008 has run. Before any DDL it refuses while the outbox still holds a pending row in a pre-007 shape: a native inline candidate, a NULL candidate, or storage_version other than 1 on the 2.x.x schemas that have that column, which is checked for first so the predicate does not error on a native schema. The refusal names every offending block_hash and the drain procedure, and the whole migration transaction rolls back, so a refused start leaves the schema untouched. The applied set is now [2, 3, 4, 5, 7, 8, 9], and the four exact migration assertions move with it rather than being weakened to a superset. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: retain the cluster fingerprint configure pinned or verified The candidate enqueue and save_job must fence against a fingerprint reset by re-reading qbit_prism_cluster.config_fingerprint FOR SHARE in their own transaction and refusing a write that is not this frontend's own. That needs the value configure settled at startup, so Ledger keeps it in a shared OnceLock, set only once the pin or the match has committed, and exposes it read-only through config_fingerprint(). configure keeps its signature and its mismatch refusal, and a refused call retains nothing. The rotation refusal in configure, and the writer fence itself, belong to the enqueue change and are not here. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: share the window reference, balance writer and page primitives WindowRef::from_snapshot is the synchronous reference builder both writers need: anchor_ms copied, the semantic balances digest, and for a non-empty snapshot the first and last share_seq, the length, and the share array's SHA-256 streamed through the existing DigestWriter, so the 233 to 260 MB serialized window at 400,000 shares is never built and the digest equals the one qbit_prism_audit_snapshots already stores. An empty snapshot gives the empty-window reference. Callers run it inside their own spawn_blocking. put_balance_snapshot is the one canonical writer of qbit_prism_balance_snapshots, on the caller's transaction: the sort by (order_key, recipient_id, p2mr_program_hex), the semantic digest that keys the row, and compact JSON of the sorted set, all off the runtime, then INSERT ... ON CONFLICT DO NOTHING. On a conflict it reads the stored bytes back and compares them with its own, because these rows are immutable and can only disagree through corruption. The enqueue and save_job and its repair all write through it. read_prior_balances and the current-balance read still keep their SQL order; the writer's sort defines the stored encoding, which is the order read_window(..., AsIssued) returns. probe_share_rows and read_range_paged are read_window's endpoint probe and page loop, lifted onto a caller's connection so the enqueue probe and the landing re-read share one implementation. The predicate, the ascending keyset order, the 4096-row page bound and the per-page count bound are unchanged, and read_window now calls them, with its signature and behaviour unchanged. The paged reader takes the caller's state and a per-page consumer, both of which travel to the blocking thread with the rows, so the landing re-read can compare each page with its slice of a vector it already holds without any whole-window work on the runtime; a dropped read still releases the accumulated state on a blocking thread. read_window_with_permit holds the caller's window_reads permit across the whole read. The permit is moved into a BlockingDrop declared before the read starts, so a cancellation releases it on a blocking thread after the page state queued its own release there, and never with the future; a return releases it inline, before the caller's rebuild. WindowError::TaskFailed separates a cancelled or panicking blocking hand-off from corruption. Every hand-off maps a JoinError to it instead of to Decode, so a cancellation is a retryable failure with its own alert and never takes the abandon path. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: derive the witness merkle leaves from an assembled block A claim rebuilds from stored inputs, and the witness leaves are the one builder input that scales with the block, so they are re-derived rather than stored: one 64-hex string per transaction beside the candidate would grow the document with the block. witness_merkle_leaves_from_block skips the 80-byte header, reads the compact-size transaction count, walks past the coinbase and returns double_sha256 of every remaining transaction's full serialized bytes. That is sound because assemble_submission appends the template's transactions verbatim after the coinbase, so the result equals witness_merkle_leaves_hex(&transactions_from_template(template)) for the block built from that template, with or without segwit transactions. The segwit-aware walk is strip_witness_transaction's own, factored into a helper that also reports where the transaction ends, which is what a block walker needs; strip_witness_transaction keeps its trailing-byte check and its output. A truncated block, trailing bytes, a zero count and a count that disagrees with the transactions present are all errors, never a partial list of leaves. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: version the audit builders and key the CC5 goldens by it A stored window reference reproduces its bundle only under the builder that first built it. The schema strings do not capture that, because the share shape chooses them rather than the builder logic, so qbit-prism gains AUDIT_BUILDER_VERSION, starting at 1. A candidate and a prepared job store it, and a rebuild refuses a reference at another version instead of rebuilding it with the current builder. The bump rule is on the constant: any change that alters canonical_audit_bundle_bytes for the same inputs bumps it in the same commit. The CC5 golden test now selects its expected digests from a table keyed by the constant, and fails when no entry exists, so a new output is recorded as a new entry beside the old one and review sees both. Version 1 holds the existing literals, unchanged. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: cover 007, the fingerprint and the window primitives on PostgreSQL Ten gated database cases, all listed in test/prism-gated-tests.txt. They go in a new file rather than the shared 2.x.x support module, which another open change appends to. Migration 007: a frozen 2.x.x schema carrying only terminal outbox rows upgrades with every row and field intact and the seven new columns NULL; each of the three pre-007 pending shapes is refused by name, with nothing from 007 applied and no version 7 row, and the same binary starts once the row is drained; a database at [2, 3, 4, 5, 8, 9] applies 007 and only 007, leaving every other version's applied_at untouched; the CHECK accepts exactly the three states across all 64 null combinations and rejects a short or uppercase digest, a zero or out-of-range count and a zero sequence; and the retention index is the partial one on window_prior_balances_sha256. config_fingerprint() returns what configure pinned or verified, is shared by clones, and stays empty after a refused configure. from_snapshot's streamed digest equals sha256(serde_json::to_vec) and the snapshot_sha256 landing writes for the same window. The balance writer stores the canonical sort, is idempotent under any permutation, errors rather than accepting a different encoding of the same digest, and read_window(..., AsIssued) hands back the stored order. The probe misses a missing endpoint, the paged reader compares 4,999 rows page by page with a vector held inside the caller's transaction, and a row the predicate skips inside a range is Incomplete with no partial window. The permit case builds a runtime with one blocking thread, occupies it, and cancels a reader parked mid-page on a database gate: no permit is available while the cleanup is queued, and one is once it has run. A failed blocking hand-off surfaces as TaskFailed, never as Decode. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: carry the window reference on prepared work, behind a read permit `Prepared` gains `window: WindowRef`, computed once per non-cached refresh beside `build_bundle` under `tokio::join!`, so submit can clone it instead of re-digesting the window on the share path when a block is found. An empty snapshot gets the empty-window reference on the runtime thread; a resume computes it inside the blocking decode it already runs. `Coordinator` gains the `window_reads` semaphore, sized `clamp(database_max_connections - 2, 1, build_workers)`, and the one entry point callers use: it takes an owned permit after the caller's `build_slots` permit and hands it to `Ledger::read_window_with_permit`, which holds it across the whole read, including the off-runtime cleanup a cancellation leaves behind, and releases it before the rebuild. `Ledger::read_window` is awaited on the runtime, never wrapped in `spawn_blocking`; the test with a single blocking thread pins that. The four gated `window_ref_tests` are listed in the manifest. Part of #265 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * 3.x.x: switch the candidate outbox to the window reference `Candidate` drops the embedded `AuditBundle` for `window: WindowRef` and every builder input except the window: `found_block`, `payout_policy`, the nested `ctv` triple (whose presence selects the CTV builder), the stored `bootstrap_share` of an empty window, `audit_builder_version`, `signer_keys`, a `leased` flag and the now-required `coinbase_suffix_hex`. The block leaves the document for the `block_bytes` column, authenticated by `block_sha256`, which `candidate_sha256` covers. The write serializes, digests and checks the candidate before the append or enqueue transaction opens; under `ORDER_LOCK` the transaction re-reads `config_fingerprint FOR SHARE` and refuses a value the writer did not pin, re-inserts a `leased` candidate's as-issued balance snapshot through the shared writer, probes the window's first row (an absent row alerts and still publishes), and inserts only the prepared bytes with the six typed columns. Submit builds the slim candidate from `Prepared.window`, the `BundleInputs` captured where `build_bundle` read configuration, and the job's own bootstrap share; it clones no bundle and reads no config. The claim's `UPDATE … RETURNING` returns the columns and the block, and the decode, off the runtime, checks the document against them, the digest, `sha256(block_bytes)`, the header hash, the required fields and the two reference invariants. A pending row with a NULL anchor is a hard error naming the pre-007 frontend; an inline document is never decoded. Before any read the claim retries, with an alert naming both values, a stored builder version or signer pair that is not this binary's; a landed audit is authenticated against the block (coinbase, audit root as the witness reserved value, share snapshot, …
* 3.x.x: add migration 007 and refuse pre-007 candidate outbox rows Migration 007 mirrors 008 onto qbit_block_candidate_outbox: the same six window-reference columns, and a three-state CHECK whose body is identical to 008's apart from the table and constraint names. It also adds block_bytes, the nullable bytea the block moves into so the candidate document stays small, and a partial index on window_prior_balances_sha256 for the balance-snapshot GC. Nothing writes any of them yet; this is additive schema only. The runner applies 007 by membership, in its own block between 5 and 8, so it assumes neither 006 nor 008 has run. Before any DDL it refuses while the outbox still holds a pending row in a pre-007 shape: a native inline candidate, a NULL candidate, or storage_version other than 1 on the 2.x.x schemas that have that column, which is checked for first so the predicate does not error on a native schema. The refusal names every offending block_hash and the drain procedure, and the whole migration transaction rolls back, so a refused start leaves the schema untouched. The applied set is now [2, 3, 4, 5, 7, 8, 9], and the four exact migration assertions move with it rather than being weakened to a superset. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: retain the cluster fingerprint configure pinned or verified The candidate enqueue and save_job must fence against a fingerprint reset by re-reading qbit_prism_cluster.config_fingerprint FOR SHARE in their own transaction and refusing a write that is not this frontend's own. That needs the value configure settled at startup, so Ledger keeps it in a shared OnceLock, set only once the pin or the match has committed, and exposes it read-only through config_fingerprint(). configure keeps its signature and its mismatch refusal, and a refused call retains nothing. The rotation refusal in configure, and the writer fence itself, belong to the enqueue change and are not here. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: share the window reference, balance writer and page primitives WindowRef::from_snapshot is the synchronous reference builder both writers need: anchor_ms copied, the semantic balances digest, and for a non-empty snapshot the first and last share_seq, the length, and the share array's SHA-256 streamed through the existing DigestWriter, so the 233 to 260 MB serialized window at 400,000 shares is never built and the digest equals the one qbit_prism_audit_snapshots already stores. An empty snapshot gives the empty-window reference. Callers run it inside their own spawn_blocking. put_balance_snapshot is the one canonical writer of qbit_prism_balance_snapshots, on the caller's transaction: the sort by (order_key, recipient_id, p2mr_program_hex), the semantic digest that keys the row, and compact JSON of the sorted set, all off the runtime, then INSERT ... ON CONFLICT DO NOTHING. On a conflict it reads the stored bytes back and compares them with its own, because these rows are immutable and can only disagree through corruption. The enqueue and save_job and its repair all write through it. read_prior_balances and the current-balance read still keep their SQL order; the writer's sort defines the stored encoding, which is the order read_window(..., AsIssued) returns. probe_share_rows and read_range_paged are read_window's endpoint probe and page loop, lifted onto a caller's connection so the enqueue probe and the landing re-read share one implementation. The predicate, the ascending keyset order, the 4096-row page bound and the per-page count bound are unchanged, and read_window now calls them, with its signature and behaviour unchanged. The paged reader takes the caller's state and a per-page consumer, both of which travel to the blocking thread with the rows, so the landing re-read can compare each page with its slice of a vector it already holds without any whole-window work on the runtime; a dropped read still releases the accumulated state on a blocking thread. read_window_with_permit holds the caller's window_reads permit across the whole read. The permit is moved into a BlockingDrop declared before the read starts, so a cancellation releases it on a blocking thread after the page state queued its own release there, and never with the future; a return releases it inline, before the caller's rebuild. WindowError::TaskFailed separates a cancelled or panicking blocking hand-off from corruption. Every hand-off maps a JoinError to it instead of to Decode, so a cancellation is a retryable failure with its own alert and never takes the abandon path. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: derive the witness merkle leaves from an assembled block A claim rebuilds from stored inputs, and the witness leaves are the one builder input that scales with the block, so they are re-derived rather than stored: one 64-hex string per transaction beside the candidate would grow the document with the block. witness_merkle_leaves_from_block skips the 80-byte header, reads the compact-size transaction count, walks past the coinbase and returns double_sha256 of every remaining transaction's full serialized bytes. That is sound because assemble_submission appends the template's transactions verbatim after the coinbase, so the result equals witness_merkle_leaves_hex(&transactions_from_template(template)) for the block built from that template, with or without segwit transactions. The segwit-aware walk is strip_witness_transaction's own, factored into a helper that also reports where the transaction ends, which is what a block walker needs; strip_witness_transaction keeps its trailing-byte check and its output. A truncated block, trailing bytes, a zero count and a count that disagrees with the transactions present are all errors, never a partial list of leaves. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: version the audit builders and key the CC5 goldens by it A stored window reference reproduces its bundle only under the builder that first built it. The schema strings do not capture that, because the share shape chooses them rather than the builder logic, so qbit-prism gains AUDIT_BUILDER_VERSION, starting at 1. A candidate and a prepared job store it, and a rebuild refuses a reference at another version instead of rebuilding it with the current builder. The bump rule is on the constant: any change that alters canonical_audit_bundle_bytes for the same inputs bumps it in the same commit. The CC5 golden test now selects its expected digests from a table keyed by the constant, and fails when no entry exists, so a new output is recorded as a new entry beside the old one and review sees both. Version 1 holds the existing literals, unchanged. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: cover 007, the fingerprint and the window primitives on PostgreSQL Ten gated database cases, all listed in test/prism-gated-tests.txt. They go in a new file rather than the shared 2.x.x support module, which another open change appends to. Migration 007: a frozen 2.x.x schema carrying only terminal outbox rows upgrades with every row and field intact and the seven new columns NULL; each of the three pre-007 pending shapes is refused by name, with nothing from 007 applied and no version 7 row, and the same binary starts once the row is drained; a database at [2, 3, 4, 5, 8, 9] applies 007 and only 007, leaving every other version's applied_at untouched; the CHECK accepts exactly the three states across all 64 null combinations and rejects a short or uppercase digest, a zero or out-of-range count and a zero sequence; and the retention index is the partial one on window_prior_balances_sha256. config_fingerprint() returns what configure pinned or verified, is shared by clones, and stays empty after a refused configure. from_snapshot's streamed digest equals sha256(serde_json::to_vec) and the snapshot_sha256 landing writes for the same window. The balance writer stores the canonical sort, is idempotent under any permutation, errors rather than accepting a different encoding of the same digest, and read_window(..., AsIssued) hands back the stored order. The probe misses a missing endpoint, the paged reader compares 4,999 rows page by page with a vector held inside the caller's transaction, and a row the predicate skips inside a range is Incomplete with no partial window. The permit case builds a runtime with one blocking thread, occupies it, and cancels a reader parked mid-page on a database gate: no permit is available while the cleanup is queued, and one is once it has run. A failed blocking hand-off surfaces as TaskFailed, never as Decode. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: carry the window reference on prepared work, behind a read permit `Prepared` gains `window: WindowRef`, computed once per non-cached refresh beside `build_bundle` under `tokio::join!`, so submit can clone it instead of re-digesting the window on the share path when a block is found. An empty snapshot gets the empty-window reference on the runtime thread; a resume computes it inside the blocking decode it already runs. `Coordinator` gains the `window_reads` semaphore, sized `clamp(database_max_connections - 2, 1, build_workers)`, and the one entry point callers use: it takes an owned permit after the caller's `build_slots` permit and hands it to `Ledger::read_window_with_permit`, which holds it across the whole read, including the off-runtime cleanup a cancellation leaves behind, and releases it before the rebuild. `Ledger::read_window` is awaited on the runtime, never wrapped in `spawn_blocking`; the test with a single blocking thread pins that. The four gated `window_ref_tests` are listed in the manifest. Part of #265 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * 3.x.x: switch the candidate outbox to the window reference `Candidate` drops the embedded `AuditBundle` for `window: WindowRef` and every builder input except the window: `found_block`, `payout_policy`, the nested `ctv` triple (whose presence selects the CTV builder), the stored `bootstrap_share` of an empty window, `audit_builder_version`, `signer_keys`, a `leased` flag and the now-required `coinbase_suffix_hex`. The block leaves the document for the `block_bytes` column, authenticated by `block_sha256`, which `candidate_sha256` covers. The write serializes, digests and checks the candidate before the append or enqueue transaction opens; under `ORDER_LOCK` the transaction re-reads `config_fingerprint FOR SHARE` and refuses a value the writer did not pin, re-inserts a `leased` candidate's as-issued balance snapshot through the shared writer, probes the window's first row (an absent row alerts and still publishes), and inserts only the prepared bytes with the six typed columns. Submit builds the slim candidate from `Prepared.window`, the `BundleInputs` captured where `build_bundle` read configuration, and the job's own bootstrap share; it clones no bundle and reads no config. The claim's `UPDATE … RETURNING` returns the columns and the block, and the decode, off the runtime, checks the document against them, the digest, `sha256(block_bytes)`, the header hash, the required fields and the two reference invariants. A pending row with a NULL anchor is a hard error naming the pre-007 frontend; an inline document is never decoded. Before any read the claim retries, with an alert naming both values, a stored builder version or signer pair that is not this binary's; a landed audit is authenticated against the block (coinbase, audit root as the witness reserved value, share snapshot, bits) and the claim then skips the rebuild and landing and continues to observe, renew and submit. Otherwise it takes `build_slots`, then `window_reads` inside one 60 s deadline around the awaited `read_window(Current)` and the borrowing builder run directly in `spawn_blocking`, and keeps `Arc<AuditBundleBody>` and the shares as `ClaimParts`. `Incomplete`, `SnapshotDigestMismatch`, `Decode`, `BalanceSnapshotMissing`, `TaskFailed` and the deadline retry with their own alerts; `PriorBalancesChanged` finishes only a block `observe_candidate` proves inactive and superseded; `Database` propagates. A `leased` candidate submits from `block_bytes` first, then rebuilds `AsIssued` and lands before any terminal outcome. Rebuilt parts are released off the runtime on every path. Landing takes the parts and does all whole-window work in one blocking closure: `verify_audit_parts`, the raw body, the counted canonical bytes, the leaves and accounts, and the share snapshot digest, which must equal the reference's. The balance fence compares the current set and the parts' through the semantic digest, so an as-issued set compares equal in any order. `persist_audit_snapshot` re-reads the range with the paged reader inside the landing transaction, page against slice on the blocking thread; `read_range` stays for the API. The terminal UPDATE NULLs the six window columns and `block_bytes` with `candidate`. `configure` refuses to pin a fingerprint onto a reset one while a pending row stores other signer keys, and the startup gate requires 007 with 009. The JSONB gate drops its enqueue entry, and every helper that built a candidate builds the slim one, handing the ledger-level landings the bundle's parts through `CandidateClaim::with_bundle`. Part of #265 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * 3.x.x: cover the candidate switch on PostgreSQL Ledger-level, in `candidate_window_switch`: a two-page window enqueues a document with no array and no share row, under 1 MB, whose six typed columns equal its reference and whose block is stored as bytes; the writer fence refuses a reset or different fingerprint in the same transaction, share included; a missing window prefix row alerts and still publishes, and its read then meets `Incomplete`; a `leased` enqueue re-inserts the as-issued balance snapshot in the writer's canonical order, writes none when not leased, and refuses balances that do not hash to the reference; every preparation failure is refused without waiting for `ORDER_LOCK`; every column, digest, block, header, NULL-anchor, missing-field, invariant and inline-document disagreement is refused at claim; landing refuses parts that are not the referenced window and a claim without parts; the terminal UPDATE NULLs the six columns and the block for submitted and abandoned rows; the landed-audit authentication rejects a forged coinbase, root, snapshot or bits; and `configure` refuses a rotation over a foreign-signed pending candidate. Coordinator-level, in `window_switch_tests` behind a fake node: a stored builder version or signer pair that is not this binary's retries, naming both values, while the build slot is held; a landed audit is authenticated and the claim continues to `submitblock` without a build slot, and a forged landed coinbase or root is refused with the row left pending; `Incomplete`, a digest mismatch, changed balances on a current block and the rebuild deadline all retry with an alert and release their permits; `classify_window_error` keeps `TaskFailed` apart from corruption and propagates `Database`; the read, rebuild and landing complete while `ORDER_LOCK` is held, with one build worker; a `leased` candidate submits first, then lands from `AsIssued`, and finishes submitted, abandoned with the node's reason, or pending when its balance snapshot is missing; and `OffRuntime` releases on a blocking thread, on cancellation too. The fifteen gated tests are listed in the manifest. Part of #265 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * 3.x.x: migrate the imported-audit tests to the slim candidate Slice 1 (#325) merged after this stack branched, and its tests in tests/support/ledger_2x.rs still built the pre-007 candidate: they read block.bundle from a Candidate and called candidate_with_bundle without a window reference or settlement inputs. The rebase merged them cleanly and the tree then failed to compile. The three helpers now take the TestCandidate the constructors return, so they reach the bundle beside the slim candidate, and land_confirmed attaches the claim's parts the way the other ledger tests do. The three constructor calls pass the window reference the bundle was built on, and the settlement inputs for the CTV case. No assertion changes. Part of #265. * 3.x.x: qualify lease renewal and the claim's canonical bytes at scale Incident 1: a candidate whose window holds N shares is claimed and processed under the short lease, and every renewal is read back from claim_expires_at. The largest gap stays below interval + 2 x timeout, the lease never lapses, and a successor ledger cannot take the claim. The bundle a claim rebuilds through the window-read entry point and the borrowing builder has the canonical bytes refresh built for the same job, and the landed audit carries that bundle's digest. Both run at PRISM_WINDOW_QUALIFY_SHARES (20,000 by default); the 400,000-share variants are ignored and run on a larger host. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: measure ACK latency and ORDER_LOCK hold across window sizes Incident 2: two frontends on one database submit ordinary shares at a fixed rate while one of them solves a block on an N-share window and its own submit loop claims, rebuilds, lands and submits it. The solving share's ACK and the bystander's p99 stay within max(2 x baseline p99 + 50 ms, 250 ms), every ACK is well under the share-commit timeout, and the block is enqueued exactly once. Latency is the in-process MiningBackend::submit wall clock; the socket-level harness (#271) does not exist yet. A pg_locks sampler on its own connection measures how long a solving append holds ORDER_LOCK at two window sizes with no other traffic: the larger size stays within a fixed slack of the smaller, and the candidate document is the same size at both and under 1 MiB. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: hold the JSONB gate's enqueue to a window-free row under 1 MiB At every measured size the enqueue must be accepted and every JSONB value it writes, the outbox candidate document included, must stay under 1,048,576 bytes. At the reduced sizes the enqueue, claim and landing phases must all run. The ratchet and its expected values are unchanged. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: make the window-read permit test actually wait before it looks The delay that gives an unpermitted read time to finish was an empty async block, so the check that the read had not finished ran at once and passed whether or not the entry point took a permit. It now sleeps 500 ms inside a 5 s bound, and fails when the permit acquisition is removed from the entry point. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: rebuild a CTV candidate from its stored settlement inputs No host-runnable test reached the CTV branch of the claim rebuild. A CTV-enabled frontend now claims a candidate whose stored direct floor differs from its configuration, and builds a different coinbase than the configured floor would, and the claim must land that candidate's own audit and submit it. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: pin the SQL order of the current-balance reads Nothing defended the ruling that the current-balance path keeps its SQL order: every fixture's balances already arrived in byte-wise canonical order. The carry table's text columns now take an ICU collation, under which the SQL order and the canonical sort disagree, and the snapshot and the current-balance window read must both return the SQL order. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: record what incident 2 must also assert once #324 is in the base After #324 a slow solve is acknowledged late rather than refused, so acceptance proves nothing and the latency percentiles stay the assertion. The doc comment and a TODO at the outcome checks name the counter and outcome to assert at rebase. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: assert incident 2's late-confirm counter now that #324 is in the base W3c wrote the qualification tests against a base that predated #324 and left a marked place for the assertions that only make sense once the commit gate is there. The base now carries it, so apply them: incident 2 asserts `qbit_prism_late_confirmed_shares_total` is zero on both frontends and that no acknowledgement, the block-solving share included, is `ledger-outcome-unknown`. Acceptance alone is no longer proof. #324 gives a share-pass append carrying a candidate its own `block_only_ack_timeout`, so a slow solve is never refused at the share deadline; it is answered late, or answered unknown. Either would have satisfied the acceptance checks with the serialization stall still present. Also set #324's two new `Config` fields in the qualification fixture. * 3.x.x: pin a refused candidate as a definite rejection, not an unknown outcome #324 decides what a miner is told when a block-only enqueue fails: `enqueue_failed_before_commit` answers a provably uncommitted failure with `ledger-confirmation-failed`, and anything else with `ledger-outcome-unknown`, which asks an operator to go and follow the outbox row. Slice 3 added the checks that guard the reference: the block's length, its header hash, its digest against `block_sha256`, the coinbase suffix, the reference invariants, and a leased candidate's balances. All of them are raised by `prepare_candidate` before the enqueue takes a connection, so none carries an `sqlx::Error` and all classify as definite today. That is a property of where the checks sit, not something the classifier knows about them, so it is pinned here rather than left to hold by accident: a corrupt candidate must never send a miner chasing a row that was never written. Verified by mutation: classifying a non-sqlx failure as unknown fails this test. * 3.x.x: say why the full-size qualification variants cannot reach 400k yet Their ignore reasons blamed RAM and the host. The real limit is #273: the job payload still holds three copies of the window (2 x AcceptedShare + 1 x CountedShare), so publishing a job for a 400,000-share window needs about 670 MiB and PostgreSQL refuses the write above its 256 MiB jsonb object limit. Every full-size variant fails at fixture set-up, before it measures anything. The count must also divide the window weight exactly, so the largest usable size today is 125,000, where all four variants pass in release: incident 1 50 renewals, max gap 118.25 ms against a 700 ms bound incident 2 phase-A p99 2.60 ms, phase-B p99 95.79 ms, bound 250 ms lock hold median 0.75 ms at 5,000 against 1.32 ms at 125,000 canonical refresh and claim rebuild both 139,040,610 B, same sha256 The candidate document stays flat at 1103 B and 1107 B, which is the property this slice exists to establish. * 3.x.x: migrate the audit-body normalization tests to the slim candidate #267 landed its own test file against the candidate that still carried an assembled bundle. This slice replaced that with a window reference, so the file keeps a `TestCandidate` pairing the stored candidate with the bundle a test still needs, as `tests/ledger_postgres.rs` does, and attaches the claim's parts through `with_bundle`. One case needed more than a shape change. The four negative cases hand the claim a window the ledger does not hold and require the durable-range proof to be what refuses it. A candidate now commits to a snapshot digest, so a reference taken from the unaltered snapshot would be refused earlier, at the reference check, and the proof under test would never run. `window_ref_for` therefore derives each altered window's reference from that window, leaving the proof as the only thing that can notice, which is what #267 wrote them to prove. * 3.x.x: pin the durable-range proof outside the settlement lock #267 moved the full-equality proof of a candidate's window out of the settlement transaction so the lock is not held for a window-sized read. This slice made that read paged, so it does not occupy a runtime thread either. Reconciling the two is the step that can quietly undo the first: keep this slice's `persist_audit_snapshot` through the rebase and the full read goes back under the lock. The landing stays correct, every other test still passes, and only the lock hold grows, so nothing in the suite can tell. Measured at 100,000 shares on #267's branch that was a 34% longer hold. The guard is deterministic and needs no large window. A second connection holds `SETTLEMENT_LOCK`, so the landing blocks on it once its proof has run. While it waits, one share's `ntime` is altered: full equality sees that, a count over the same anchored predicate cannot. The landing must then succeed, which it can only do if it had already read and compared the shares before taking the lock. Accepted history is immutable and a trigger enforces it, so the test lifts that trigger for the single statement, inside its own disposable schema, and puts it straight back. Verified by mutation: restoring the in-transaction range read fails this test with "the landing read share payloads under the settlement lock". * 3.x.x: serialise the coordinator test modules that share the ledger's locks The ledger's advisory locks are cluster-wide constants, not schema-scoped, so two coordinator test modules running in parallel contend for ORDER_LOCK even though each works in its own throwaway schema. Every module held a lock of its own, which serialised it against itself and left cross-module waits to `lock_timeout`, 5 s by default. #324's reconciliation tests hold ORDER_LOCK for up to about 1.5 s, and this slice adds three more modules that open a ledger. Once enough of them ran at once those waits stacked past the timeout: a full `--lib` run failed with "canceling statement due to lock timeout" in `candidate_lease_tests` and `window_switch_tests`, and passed on the next run. A suite that fails for a reason unrelated to what it tests is worse than a slow one. All seven modules now take one shared lock. The run goes from about 35 s to about 62 s, and three consecutive runs pass where the flake had reproduced. * 3.x.x: pass the signer keys #334's wait-metrics tests now need `Ledger::configure` takes the frontend's signing keys since this slice, so the fingerprint contention tests #334 added call it with a fixed pair. That suite measures advisory-lock and pool-acquire waits, not signing, so the only thing the keys have to be is the same on every call. * 3.x.x: resume a job's bundle only while the current inputs still describe it A resumed job reuses the bundle it was issued with, but captures its inputs from this frontend's configuration now. Before this slice that was harmless: the candidate carried the assembled bundle, and landing used it directly. Now the candidate stores the inputs and the claim rebuilds the audit from them, so an old bundle labelled with today's configuration rebuilds to a coinbase the block does not commit to. The claim fails its comparison before `submitblock` and retries until the job expires, which can lose a found block. The stored bundle is now reused only when the captured inputs still describe it, comparing what the bundle itself records about how it was built: the payout policy, and the public keys that actually signed its coinbase manifest and its ledger attestation. A difference is a cache miss, so the bundle is rebuilt rather than mislabelled. Both directions are covered, since over-tightening would silently defeat the job cache and rebuild on every resume: unchanged inputs must still describe the bundle, and a payout-policy change, a full signer rotation and a rotation of either key alone must each refuse it. Known residue, recorded in the doc comment: a bundle records neither the CTV settlement inputs nor `audit_builder_version`, so a change to either between issue and resume is still undetectable here. Closing that needs the issued job to store its inputs, which is #273's format. * 3.x.x: migrate #355's ledger tests to the slim candidate #355 landed deadline and single-execution coverage against the candidate that still carried an assembled bundle, and against `configure` before it took the frontend's signing keys. Adapted, the same way #325's and #267's suites were: a `TestCandidate` pairs the stored candidate with the bundle a landing still needs, claims attach their parts through `with_bundle`, and the conflicting claim that is never landed carries none. `ledger_deadline_atomicity` builds its candidate from a bundle rather than a snapshot, so its window reference is derived from that bundle: the anchor and prior-balances digest it already records, and the range and digest of the shares it carries. Its signer keys come from the public keys that actually signed it, rather than from a local pair, so the stored inputs describe the bundle. * 3.x.x: make the landed-audit forgery always alter the row, and migrate #360's fixture The forged-coinbase case placed a literal '0' at a fixed offset. Whenever the digit already held that value the forgery changed nothing, so the claim was right to accept it and the case failed only by luck. The coinbase varies per run, because the bundle's anchor comes from the ledger clock, so this passed repeatedly and then failed once as "the forgery was accepted" — an alarming message for a case that was in fact a no-op. A security check that can quietly become vacuous is worse than one that is absent. The digit is now flipped to a value it does not currently hold, and the row is read back either side of the statement so a forgery that altered nothing fails the case for that reason instead. Also passes the signer keys #360's `tests/support/fatal_state.rs` now needs, since `configure` takes them in this slice. * 3.x.x: drop the pre-#362 path for the heartbeat SQL gated test #362 moved `live_instance_tests` from `tools` to `ledger::instances`. Merging the gated-test manifest during the rebase kept both paths, because a rename upstream looks like one id removed and another added, and a union keeps the removed one. The old `tools::` path then never executed and the gate refused the run: "194 distinct gated tests executed, below the minimum of 195". The manifest is a superset of the base's ids, and every id this branch adds over the base resolves to a test function that exists in the tree. * v3.0.0 feat: add compact prepared storage and immutable template blobs (#359) * 3.x.x: add migration 007 and refuse pre-007 candidate outbox rows Migration 007 mirrors 008 onto qbit_block_candidate_outbox: the same six window-reference columns, and a three-state CHECK whose body is identical to 008's apart from the table and constraint names. It also adds block_bytes, the nullable bytea the block moves into so the candidate document stays small, and a partial index on window_prior_balances_sha256 for the balance-snapshot GC. Nothing writes any of them yet; this is additive schema only. The runner applies 007 by membership, in its own block between 5 and 8, so it assumes neither 006 nor 008 has run. Before any DDL it refuses while the outbox still holds a pending row in a pre-007 shape: a native inline candidate, a NULL candidate, or storage_version other than 1 on the 2.x.x schemas that have that column, which is checked for first so the predicate does not error on a native schema. The refusal names every offending block_hash and the drain procedure, and the whole migration transaction rolls back, so a refused start leaves the schema untouched. The applied set is now [2, 3, 4, 5, 7, 8, 9], and the four exact migration assertions move with it rather than being weakened to a superset. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: retain the cluster fingerprint configure pinned or verified The candidate enqueue and save_job must fence against a fingerprint reset by re-reading qbit_prism_cluster.config_fingerprint FOR SHARE in their own transaction and refusing a write that is not this frontend's own. That needs the value configure settled at startup, so Ledger keeps it in a shared OnceLock, set only once the pin or the match has committed, and exposes it read-only through config_fingerprint(). configure keeps its signature and its mismatch refusal, and a refused call retains nothing. The rotation refusal in configure, and the writer fence itself, belong to the enqueue change and are not here. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: share the window reference, balance writer and page primitives WindowRef::from_snapshot is the synchronous reference builder both writers need: anchor_ms copied, the semantic balances digest, and for a non-empty snapshot the first and last share_seq, the length, and the share array's SHA-256 streamed through the existing DigestWriter, so the 233 to 260 MB serialized window at 400,000 shares is never built and the digest equals the one qbit_prism_audit_snapshots already stores. An empty snapshot gives the empty-window reference. Callers run it inside their own spawn_blocking. put_balance_snapshot is the one canonical writer of qbit_prism_balance_snapshots, on the caller's transaction: the sort by (order_key, recipient_id, p2mr_program_hex), the semantic digest that keys the row, and compact JSON of the sorted set, all off the runtime, then INSERT ... ON CONFLICT DO NOTHING. On a conflict it reads the stored bytes back and compares them with its own, because these rows are immutable and can only disagree through corruption. The enqueue and save_job and its repair all write through it. read_prior_balances and the current-balance read still keep their SQL order; the writer's sort defines the stored encoding, which is the order read_window(..., AsIssued) returns. probe_share_rows and read_range_paged are read_window's endpoint probe and page loop, lifted onto a caller's connection so the enqueue probe and the landing re-read share one implementation. The predicate, the ascending keyset order, the 4096-row page bound and the per-page count bound are unchanged, and read_window now calls them, with its signature and behaviour unchanged. The paged reader takes the caller's state and a per-page consumer, both of which travel to the blocking thread with the rows, so the landing re-read can compare each page with its slice of a vector it already holds without any whole-window work on the runtime; a dropped read still releases the accumulated state on a blocking thread. read_window_with_permit holds the caller's window_reads permit across the whole read. The permit is moved into a BlockingDrop declared before the read starts, so a cancellation releases it on a blocking thread after the page state queued its own release there, and never with the future; a return releases it inline, before the caller's rebuild. WindowError::TaskFailed separates a cancelled or panicking blocking hand-off from corruption. Every hand-off maps a JoinError to it instead of to Decode, so a cancellation is a retryable failure with its own alert and never takes the abandon path. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: derive the witness merkle leaves from an assembled block A claim rebuilds from stored inputs, and the witness leaves are the one builder input that scales with the block, so they are re-derived rather than stored: one 64-hex string per transaction beside the candidate would grow the document with the block. witness_merkle_leaves_from_block skips the 80-byte header, reads the compact-size transaction count, walks past the coinbase and returns double_sha256 of every remaining transaction's full serialized bytes. That is sound because assemble_submission appends the template's transactions verbatim after the coinbase, so the result equals witness_merkle_leaves_hex(&transactions_from_template(template)) for the block built from that template, with or without segwit transactions. The segwit-aware walk is strip_witness_transaction's own, factored into a helper that also reports where the transaction ends, which is what a block walker needs; strip_witness_transaction keeps its trailing-byte check and its output. A truncated block, trailing bytes, a zero count and a count that disagrees with the transactions present are all errors, never a partial list of leaves. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: version the audit builders and key the CC5 goldens by it A stored window reference reproduces its bundle only under the builder that first built it. The schema strings do not capture that, because the share shape chooses them rather than the builder logic, so qbit-prism gains AUDIT_BUILDER_VERSION, starting at 1. A candidate and a prepared job store it, and a rebuild refuses a reference at another version instead of rebuilding it with the current builder. The bump rule is on the constant: any change that alters canonical_audit_bundle_bytes for the same inputs bumps it in the same commit. The CC5 golden test now selects its expected digests from a table keyed by the constant, and fails when no entry exists, so a new output is recorded as a new entry beside the old one and review sees both. Version 1 holds the existing literals, unchanged. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: cover 007, the fingerprint and the window primitives on PostgreSQL Ten gated database cases, all listed in test/prism-gated-tests.txt. They go in a new file rather than the shared 2.x.x support module, which another open change appends to. Migration 007: a frozen 2.x.x schema carrying only terminal outbox rows upgrades with every row and field intact and the seven new columns NULL; each of the three pre-007 pending shapes is refused by name, with nothing from 007 applied and no version 7 row, and the same binary starts once the row is drained; a database at [2, 3, 4, 5, 8, 9] applies 007 and only 007, leaving every other version's applied_at untouched; the CHECK accepts exactly the three states across all 64 null combinations and rejects a short or uppercase digest, a zero or out-of-range count and a zero sequence; and the retention index is the partial one on window_prior_balances_sha256. config_fingerprint() returns what configure pinned or verified, is shared by clones, and stays empty after a refused configure. from_snapshot's streamed digest equals sha256(serde_json::to_vec) and the snapshot_sha256 landing writes for the same window. The balance writer stores the canonical sort, is idempotent under any permutation, errors rather than accepting a different encoding of the same digest, and read_window(..., AsIssued) hands back the stored order. The probe misses a missing endpoint, the paged reader compares 4,999 rows page by page with a vector held inside the caller's transaction, and a row the predicate skips inside a range is Incomplete with no partial window. The permit case builds a runtime with one blocking thread, occupies it, and cancels a reader parked mid-page on a database gate: no permit is available while the cleanup is queued, and one is once it has run. A failed blocking hand-off surfaces as TaskFailed, never as Decode. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: carry the window reference on prepared work, behind a read permit `Prepared` gains `window: WindowRef`, computed once per non-cached refresh beside `build_bundle` under `tokio::join!`, so submit can clone it instead of re-digesting the window on the share path when a block is found. An empty snapshot gets the empty-window reference on the runtime thread; a resume computes it inside the blocking decode it already runs. `Coordinator` gains the `window_reads` semaphore, sized `clamp(database_max_connections - 2, 1, build_workers)`, and the one entry point callers use: it takes an owned permit after the caller's `build_slots` permit and hands it to `Ledger::read_window_with_permit`, which holds it across the whole read, including the off-runtime cleanup a cancellation leaves behind, and releases it before the rebuild. `Ledger::read_window` is awaited on the runtime, never wrapped in `spawn_blocking`; the test with a single blocking thread pins that. The four gated `window_ref_tests` are listed in the manifest. Part of #265 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * 3.x.x: switch the candidate outbox to the window reference `Candidate` drops the embedded `AuditBundle` for `window: WindowRef` and every builder input except the window: `found_block`, `payout_policy`, the nested `ctv` triple (whose presence selects the CTV builder), the stored `bootstrap_share` of an empty window, `audit_builder_version`, `signer_keys`, a `leased` flag and the now-required `coinbase_suffix_hex`. The block leaves the document for the `block_bytes` column, authenticated by `block_sha256`, which `candidate_sha256` covers. The write serializes, digests and checks the candidate before the append or enqueue transaction opens; under `ORDER_LOCK` the transaction re-reads `config_fingerprint FOR SHARE` and refuses a value the writer did not pin, re-inserts a `leased` candidate's as-issued balance snapshot through the shared writer, probes the window's first row (an absent row alerts and still publishes), and inserts only the prepared bytes with the six typed columns. Submit builds the slim candidate from `Prepared.window`, the `BundleInputs` captured where `build_bundle` read configuration, and the job's own bootstrap share; it clones no bundle and reads no config. The claim's `UPDATE … RETURNING` returns the columns and the block, and the decode, off the runtime, checks the document against them, the digest, `sha256(block_bytes)`, the header hash, the required fields and the two reference invariants. A pending row with a NULL anchor is a hard error naming the pre-007 frontend; an inline document is never decoded. Before any read the claim retries, with an alert naming both values, a stored builder version or signer pair that is not this binary's; a landed audit is authenticated against the block (coinbase, audit root as the witness reserved value, share snapshot, bits) and the claim then skips the rebuild and landing and continues to observe, renew and submit. Otherwise it takes `build_slots`, then `window_reads` inside one 60 s deadline around the awaited `read_window(Current)` and the borrowing builder run directly in `spawn_blocking`, and keeps `Arc<AuditBundleBody>` and the shares as `ClaimParts`. `Incomplete`, `SnapshotDigestMismatch`, `Decode`, `BalanceSnapshotMissing`, `TaskFailed` and the deadline retry with their own alerts; `PriorBalancesChanged` finishes only a block `observe_candidate` proves inactive and superseded; `Database` propagates. A `leased` candidate submits from `block_bytes` first, then rebuilds `AsIssued` and lands before any terminal outcome. Rebuilt parts are released off the runtime on every path. Landing takes the parts and does all whole-window work in one blocking closure: `verify_audit_parts`, the raw body, the counted canonical bytes, the leaves and accounts, and the share snapshot digest, which must equal the reference's. The balance fence compares the current set and the parts' through the semantic digest, so an as-issued set compares equal in any order. `persist_audit_snapshot` re-reads the range with the paged reader inside the landing transaction, page against slice on the blocking thread; `read_range` stays for the API. The terminal UPDATE NULLs the six window columns and `block_bytes` with `candidate`. `configure` refuses to pin a fingerprint onto a reset one while a pending row stores other signer keys, and the startup gate requires 007 with 009. The JSONB gate drops its enqueue entry, and every helper that built a candidate builds the slim one, handing the ledger-level landings the bundle's parts through `CandidateClaim::with_bundle`. Part of #265 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * 3.x.x: cover the candidate switch on PostgreSQL Ledger-level, in `candidate_window_switch`: a two-page window enqueues a document with no array and no share row, under 1 MB, whose six typed columns equal its reference and whose block is stored as bytes; the writer fence refuses a reset or different fingerprint in the same transaction, share included; a missing window prefix row alerts and still publishes, and its read then meets `Incomplete`; a `leased` enqueue re-inserts the as-issued balance snapshot in the writer's canonical order, writes none when not leased, and refuses balances that do not hash to the reference; every preparation failure is refused without waiting for `ORDER_LOCK`; every column, digest, block, header, NULL-anchor, missing-field, invariant and inline-document disagreement is refused at claim; landing refuses parts that are not the referenced window and a claim without parts; the terminal UPDATE NULLs the six columns and the block for submitted and abandoned rows; the landed-audit authentication rejects a forged coinbase, root, snapshot or bits; and `configure` refuses a rotation over a foreign-signed pending candidate. Coordinator-level, in `window_switch_tests` behind a fake node: a stored builder version or signer pair that is not this binary's retries, naming both values, while the build slot is held; a landed audit is authenticated and the claim continues to `submitblock` without a build slot, and a forged landed coinbase or root is refused with the row left pending; `Incomplete`, a digest mismatch, changed balances on a current block and the rebuild deadline all retry with an alert and release their permits; `classify_window_error` keeps `TaskFailed` apart from corruption and propagates `Database`; the read, rebuild and landing complete while `ORDER_LOCK` is held, with one build worker; a `leased` candidate submits first, then lands from `AsIssued`, and finishes submitted, abandoned with the node's reason, or pending when its balance snapshot is missing; and `OffRuntime` releases on a blocking thread, on cancellation too. The fifteen gated tests are listed in the manifest. Part of #265 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * 3.x.x: migrate the imported-audit tests to the slim candidate Slice 1 (#325) merged after this stack branched, and its tests in tests/support/ledger_2x.rs still built the pre-007 candidate: they read block.bundle from a Candidate and called candidate_with_bundle without a window reference or settlement inputs. The rebase merged them cleanly and the tree then failed to compile. The three helpers now take the TestCandidate the constructors return, so they reach the bundle beside the slim candidate, and land_confirmed attaches the claim's parts the way the other ledger tests do. The three constructor calls pass the window reference the bundle was built on, and the settlement inputs for the CTV case. No assertion changes. Part of #265. * 3.x.x: qualify lease renewal and the claim's canonical bytes at scale Incident 1: a candidate whose window holds N shares is claimed and processed under the short lease, and every renewal is read back from claim_expires_at. The largest gap stays below interval + 2 x timeout, the lease never lapses, and a successor ledger cannot take the claim. The bundle a claim rebuilds through the window-read entry point and the borrowing builder has the canonical bytes refresh built for the same job, and the landed audit carries that bundle's digest. Both run at PRISM_WINDOW_QUALIFY_SHARES (20,000 by default); the 400,000-share variants are ignored and run on a larger host. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: measure ACK latency and ORDER_LOCK hold across window sizes Incident 2: two frontends on one database submit ordinary shares at a fixed rate while one of them solves a block on an N-share window and its own submit loop claims, rebuilds, lands and submits it. The solving share's ACK and the bystander's p99 stay within max(2 x baseline p99 + 50 ms, 250 ms), every ACK is well under the share-commit timeout, and the block is enqueued exactly once. Latency is the in-process MiningBackend::submit wall clock; the socket-level harness (#271) does not exist yet. A pg_locks sampler on its own connection measures how long a solving append holds ORDER_LOCK at two window sizes with no other traffic: the larger size stays within a fixed slack of the smaller, and the candidate document is the same size at both and under 1 MiB. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: hold the JSONB gate's enqueue to a window-free row under 1 MiB At every measured size the enqueue must be accepted and every JSONB value it writes, the outbox candidate document included, must stay under 1,048,576 bytes. At the reduced sizes the enqueue, claim and landing phases must all run. The ratchet and its expected values are unchanged. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: make the window-read permit test actually wait before it looks The delay that gives an unpermitted read time to finish was an empty async block, so the check that the read had not finished ran at once and passed whether or not the entry point took a permit. It now sleeps 500 ms inside a 5 s bound, and fails when the permit acquisition is removed from the entry point. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: rebuild a CTV candidate from its stored settlement inputs No host-runnable test reached the CTV branch of the claim rebuild. A CTV-enabled frontend now claims a candidate whose stored direct floor differs from its configuration, and builds a different coinbase than the configured floor would, and the claim must land that candidate's own audit and submit it. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: pin the SQL order of the current-balance reads Nothing defended the ruling that the current-balance path keeps its SQL order: every fixture's balances already arrived in byte-wise canonical order. The carry table's text columns now take an ICU collation, under which the SQL order and the canonical sort disagree, and the snapshot and the current-balance window read must both return the SQL order. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: record what incident 2 must also assert once #324 is in the base After #324 a slow solve is acknowledged late rather than refused, so acceptance proves nothing and the latency percentiles stay the assertion. The doc comment and a TODO at the outcome checks name the counter and outcome to assert at rebase. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: assert incident 2's late-confirm counter now that #324 is in the base W3c wrote the qualification tests against a base that predated #324 and left a marked place for the assertions that only make sense once the commit gate is there. The base now carries it, so apply them: incident 2 asserts `qbit_prism_late_confirmed_shares_total` is zero on both frontends and that no acknowledgement, the block-solving share included, is `ledger-outcome-unknown`. Acceptance alone is no longer proof. #324 gives a share-pass append carrying a candidate its own `block_only_ack_timeout`, so a slow solve is never refused at the share deadline; it is answered late, or answered unknown. Either would have satisfied the acceptance checks with the serialization stall still present. Also set #324's two new `Config` fields in the qualification fixture. * 3.x.x: pin a refused candidate as a definite rejection, not an unknown outcome #324 decides what a miner is told when a block-only enqueue fails: `enqueue_failed_before_commit` answers a provably uncommitted failure with `ledger-confirmation-failed`, and anything else with `ledger-outcome-unknown`, which asks an operator to go and follow the outbox row. Slice 3 added the checks that guard the reference: the block's length, its header hash, its digest against `block_sha256`, the coinbase suffix, the reference invariants, and a leased candidate's balances. All of them are raised by `prepare_candidate` before the enqueue takes a connection, so none carries an `sqlx::Error` and all classify as definite today. That is a property of where the checks sit, not something the classifier knows about them, so it is pinned here rather than left to hold by accident: a corrupt candidate must never send a miner chasing a row that was never written. Verified by mutation: classifying a non-sqlx failure as unknown fails this test. * 3.x.x: say why the full-size qualification variants cannot reach 400k yet Their ignore reasons blamed RAM and the host. The real limit is #273: the job payload still holds three copies of the window (2 x AcceptedShare + 1 x CountedShare), so publishing a job for a 400,000-share window needs about 670 MiB and PostgreSQL refuses the write above its 256 MiB jsonb object limit. Every full-size variant fails at fixture set-up, before it measures anything. The count must also divide the window weight exactly, so the largest usable size today is 125,000, where all four variants pass in release: incident 1 50 renewals, max gap 118.25 ms against a 700 ms bound incident 2 phase-A p99 2.60 ms, phase-B p99 95.79 ms, bound 250 ms lock hold median 0.75 ms at 5,000 against 1.32 ms at 125,000 canonical refresh and claim rebuild both 139,040,610 B, same sha256 The candidate document stays flat at 1103 B and 1107 B, which is the property this slice exists to establish. * 3.x.x: migrate the audit-body normalization tests to the slim candidate #267 landed its own test file against the candidate that still carried an assembled bundle. This slice replaced that with a window reference, so the file keeps a `TestCandidate` pairing the stored candidate with the bundle a test still needs, as `tests/ledger_postgres.rs` does, and attaches the claim's parts through `with_bundle`. One case needed more than a shape change. The four negative cases hand the claim a window the ledger does not hold and require the durable-range proof to be what refuses it. A candidate now commits to a snapshot digest, so a reference taken from the unaltered snapshot would be refused earlier, at the reference check, and the proof under test would never run. `window_ref_for` therefore derives each altered window's reference from that window, leaving the proof as the only thing that can notice, which is what #267 wrote them to prove. * 3.x.x: pin the durable-range proof outside the settlement lock #267 moved the full-equality proof of a candidate's window out of the settlement transaction so the lock is not held for a window-sized read. This slice made that read paged, so it does not occupy a runtime thread either. Reconciling the two is the step that can quietly undo the first: keep this slice's `persist_audit_snapshot` through the rebase and the full read goes back under the lock. The landing stays correct, every other test still passes, and only the lock hold grows, so nothing in the suite can tell. Measured at 100,000 shares on #267's branch that was a 34% longer hold. The guard is deterministic and needs no large window. A second connection holds `SETTLEMENT_LOCK`, so the landing blocks on it once its proof has run. While it waits, one share's `ntime` is altered: full equality sees that, a count over the same anchored predicate cannot. The landing must then succeed, which it can only do if it had already read and compared the shares before taking the lock. Accepted history is immutable and a trigger enforces it, so the test lifts that trigger for the single statement, inside its own disposable schema, and puts it straight back. Verified by mutation: restoring the in-transaction range read fails this test with "the landing read share payloads under the settlement lock". * 3.x.x: serialise the coordinator test modules that share the ledger's locks The ledger's advisory locks are cluster-wide constants, not schema-scoped, so two coordinator test modules running in parallel contend for ORDER_LOCK even though each works in its own throwaway schema. Every module held a lock of its own, which serialised it against itself and left cross-module waits to `lock_timeout`, 5 s by default. #324's reconciliation tests hold ORDER_LOCK for up to about 1.5 s, and this slice adds three more modules that open a ledger. Once enough of them ran at once those waits stacked past the timeout: a full `--lib` run failed with "canceling statement due to lock timeout" in `candidate_lease_tests` and `window_switch_tests`, and passed on the next run. A suite that fails for a reason unrelated to what it tests is worse than a slow one. All seven modules now take one shared lock. The run goes from about 35 s to about 62 s, and three consecutive runs pass where the flake had reproduced. * 3.x.x: add migration 007 and refuse pre-007 candidate outbox rows Migration 007 mirrors 008 onto qbit_block_candidate_outbox: the same six window-reference columns, and a three-state CHECK whose body is identical to 008's apart from the table and constraint names. It also adds block_bytes, the nullable bytea the block moves into so the candidate document stays small, and a partial index on window_prior_balances_sha256 for the balance-snapshot GC. Nothing writes any of them yet; this is additive schema only. The runner applies 007 by membership, in its own block between 5 and 8, so it assumes neither 006 nor 008 has run. Before any DDL it refuses while the outbox still holds a pending row in a pre-007 shape: a native inline candidate, a NULL candidate, or storage_version other than 1 on the 2.x.x schemas that have that column, which is checked for first so the predicate does not error on a native schema. The refusal names every offending block_hash and the drain procedure, and the whole migration transaction rolls back, so a refused start leaves the schema untouched. The applied set is now [2, 3, 4, 5, 7, 8, 9], and the four exact migration assertions move with it rather than being weakened to a superset. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: retain the cluster fingerprint configure pinned or verified The candidate enqueue and save_job must fence against a fingerprint reset by re-reading qbit_prism_cluster.config_fingerprint FOR SHARE in their own transaction and refusing a write that is not this frontend's own. That needs the value configure settled at startup, so Ledger keeps it in a shared OnceLock, set only once the pin or the match has committed, and exposes it read-only through config_fingerprint(). configure keeps its signature and its mismatch refusal, and a refused call retains nothing. The rotation refusal in configure, and the writer fence itself, belong to the enqueue change and are not here. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: share the window reference, balance writer and page primitives WindowRef::from_snapshot is the synchronous reference builder both writers need: anchor_ms copied, the semantic balances digest, and for a non-empty snapshot the first and last share_seq, the length, and the share array's SHA-256 streamed through the existing DigestWriter, so the 233 to 260 MB serialized window at 400,000 shares is never built and the digest equals the one qbit_prism_audit_snapshots already stores. An empty snapshot gives the empty-window reference. Callers run it inside their own spawn_blocking. put_balance_snapshot is the one canonical writer of qbit_prism_balance_snapshots, on the caller's transaction: the sort by (order_key, recipient_id, p2mr_program_hex), the semantic digest that keys the row, and compact JSON of the sorted set, all off the runtime, then INSERT ... ON CONFLICT DO NOTHING. On a conflict it reads the stored bytes back and compares them with its own, because these rows are immutable and can only disagree through corruption. The enqueue and save_job and its repair all write through it. read_prior_balances and the current-balance read still keep their SQL order; the writer's sort defines the stored encoding, which is the order read_window(..., AsIssued) returns. probe_share_rows and read_range_paged are read_window's endpoint probe and page loop, lifted onto a caller's connection so the enqueue probe and the landing re-read share one implementation. The predicate, the ascending keyset order, the 4096-row page bound and the per-page count bound are unchanged, and read_window now calls them, with its signature and behaviour unchanged. The paged reader takes the caller's state and a per-page consumer, both of which travel to the blocking thread with the rows, so the landing re-read can compare each page with its slice of a vector it already holds without any whole-window work on the runtime; a dropped read still releases the accumulated state on a blocking thread. read_window_with_permit holds the caller's window_reads permit across the whole read. The permit is moved into a BlockingDrop declared before the read starts, so a cancellation releases it on a blocking thread after the page state queued its own release there, and never with the future; a return releases it inline, before the caller's rebuild. WindowError::TaskFailed separates a cancelled or panicking blocking hand-off from corruption. Every hand-off maps a JoinError to it instead of to Decode, so a cancellation is a retryable failure with its own alert and never takes the abandon path. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: derive the witness merkle leaves from an assembled block A claim rebuilds from stored inputs, and the witness leaves are the one builder input that scales with the block, so they are re-derived rather than stored: one 64-hex string per transaction beside the candidate would grow the document with the block. witness_merkle_leaves_from_block skips the 80-byte header, reads the compact-size transaction count, walks past the coinbase and returns double_sha256 of every remaining transaction's full serialized bytes. That is sound because assemble_submission appends the template's transactions verbatim after the coinbase, so the result equals witness_merkle_leaves_hex(&transactions_from_template(template)) for the block built from that template, with or without segwit transactions. The segwit-aware walk is strip_witness_transaction's own, factored into a helper that also reports where the transaction ends, which is what a block walker needs; strip_witness_transaction keeps its trailing-byte check and its output. A truncated block, trailing bytes, a zero count and a count that disagrees with the transactions present are all errors, never a partial list of leaves. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: version the audit builders and key the CC5 goldens by it A stored window reference reproduces its bundle only under the builder that first built it. The schema strings do not capture that, because the share shape chooses them rather than the builder logic, so qbit-prism gains AUDIT_BUILDER_VERSION, starting at 1. A candidate and a prepared job store it, and a rebuild refuses a reference at another version instead of rebuilding it with the current builder. The bump rule is on the constant: any change that alters canonical_audit_bundle_bytes for the same inputs bumps it in the same commit. The CC5 golden test now selects its expected digests from a table keyed by the constant, and fails when no entry exists, so a new output is recorded as a new entry beside the old one and review sees both. Version 1 holds the existing literals, unchanged. Part of #265 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: cover 007, the fingerprint and the window primitives on PostgreSQL Ten gated database cases, all listed in test/prism-gated-tests.txt. They go in a new file rather than the shared 2.x.x support module, which another open change appends to. Migration 007: a frozen 2.x.x schema carrying only terminal outbox rows upgrades with every row and field intact and the seven new columns NULL; each of the three pre-007 pending shapes is refused by name, with nothing from 007 applied and no version 7 row, and the same binary starts once the row is drained; a database at [2, 3, 4, 5, 8, 9] applies 007 and only 007, leaving every other version's applied_at untouched; the CHECK accepts exactly the three states across all 64 null combinations and rejects a short or uppercase digest, a zero or out-of-range count and a zero sequence; and the retention index is the partial one on window_prior_balances_sha256. config_fingerprint() returns what configure pinned or verified, is shared by clones, and stays empty after a refused conf…
* feat: extend pool acquisition timing to direct ledger queries Reuse the existing cancellation-safe wait guard for a shared Ledger::acquire boundary and eight direct query paths. Preserve SQL and transaction behavior, and qualify elapsed timing, cancellation, and checkout release against PostgreSQL. Refs #352. * test: include publisher acquisitions in live pool observations The HTTP exhaustion test observes the aggregate pool histogram, now including payout-revision and heartbeat waits. Require every explicit cancellation to remain visible while preserving exact per-attempt assertions in the isolated tests. * docs: describe partial direct-query pool timing coverage Refresh canonical metric and alert guidance and regenerate the inventory and review-only provisioning patch from the checksum-pinned source snapshot. Alert expressions, thresholds and role exports are unchanged. * docs: complete direct-query timing coverage guidance Co-Authored-By: OpenAI Codex <noreply@openai.com> * docs: align acquisition coverage in compatibility explanations --------- Co-authored-by: OpenAI Codex <noreply@openai.com>
* refactor: share pool acquisition observation Use one optional-metrics guard for ledger and collector checkout waits. Choose Tokio monotonic time explicitly while retaining the existing advisory-lock clock and all checkout, error, and cancellation boundaries. Add first-poll, unattached cleanup, and SQLx BEGIN boundary regressions. Refs #352 * refactor: place checkout timing beside metrics events Rename the async timing boundary to time_pool_acquire and expose it through metrics while leaving the recording hook name unchanged. Consolidate helper tests under events, retaining unique result, clock, cancellation cleanup and concurrency assertions, and remove the ledger free forwarding function. Preserve advisory-lock timing and every PostgreSQL checkout/BEGIN regression. Refs #352
…no plan walks the ledger (#754) The pool hashrate series serves fully rolled buckets from the rollup tables. It re-reads two kinds of rows from the ledger: the boundary (the current bucket and the leading partial bucket) and the tail (the shares past the rollup watermark). Neither read gave the planner an index range it could use. The boundary bounded accepted_at with an OR and a NULL guard on the range start. The tail bounded share_seq only from below, with the same guard on time and an OR on the miner. On a production-sized ledger (66M shares, rollups current), rc.5's default pool request walked all of p0's seq_walk index for the boundary, removing 66,090,385 rows to return 2,390, in 19-40 s. With the boundary fixed, the generic plan's tail still walked p0's accepted_recent_idx for no row, in 17-27 s. Pool hashrate-series requests therefore ran past the public read deadline: PRISM answered 503 read_timeout, and the proxy in front of it 504. A custom plan also walked a big miner's whole history. 2.x builds the same read with bounded predicates, so this is a 3.x regression. The boundary is now two disjoint slices, each a range on accepted_at bounded at both ends: - [GREATEST(current_bucket, started_at), ended_at]; - [started_at, LEAST(first_full_bucket, current_bucket)), which is empty without a range start. The tail takes its shares from a subquery bounded by share_seq on both sides, by the watermark and by max(share_seq), which is one backward probe per partition. OFFSET 0 fences the subquery, so the time and miner tests only filter its stream. The tail therefore reads only the shares past the watermark: none while the rollups are current, and the lag while they catch up. With no progress row it reads every share, as rc.5 did. On a production-sized ledger the fixed query took 9-23 ms for the pool and 2-15 ms for miners. That held in 20 of 20 runs, across generic plans, custom plans with the anchor pinned or NULL, and auto plans after six executions. Its output equalled rc.5's in 7 of 7 anchored requests, for which rc.5 took 15.8-25.2 s. On a 6M-share copy with a 600k-share miner, every request ran in 1-17 ms at random_page_cost 4 and 1.1. There the output matched rc.5's in 104 cases per watermark state, 312 in all. The only exceptions were unanchored requests whose range start slid while the old query ran, and they differed only in the leading bucket. Two tests in api_2x pin this. - chart_rollups_match_raw_for_every_range_bucket_and_watermark requires the rollup series to equal the raw-ledger series in 78 cases (every range and bucket pair, context, anchor and subject). It checks them before the rollups run, at a current watermark, and with shares past it. - chart_rollups_read_only_the_partial_buckets_under_any_plan runs the chart's plans over 20,000 rolled-up shares, half of them one miner's. It covers the pool at three ranges, a small miner and the big miner, in custom and generic modes, at random_page_cost 4 and 1.1, with the watermark current and again with 300 shares past it. Each plan may read at most 1,000 ledger rows plus the lag. Every ledger index scan outside a Limit probe must bound one column from both sides, and no sequential scan may read ledger rows. It fails against rc.5's query and against the boundary-only version. The change is read-only SQL in the public read path: no migration, no schema change, no write path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Set VERSION and [workspace.package].version to 3.0.0-rc.6: rc.5 plus #754 (the pool hashrate series bounds both of its ledger reads, the boundary buckets and the tail past the rollup watermark, at both ends, fixing a 3.x regression that timed it out on large ledgers), #750 (#748: a failed record of 2 is re-checked on a fresh connection when its own one was lost, and each attempt is bounded on the client's side), and #753, a test-only fix for #724. Both Cargo.lock files move only their workspace-member entries. doc/release-notes-3.0.0-rc.6.md covers: - the changes, with #754's figures from its pull request (19 to 40 s and a 504 on a 66M-share ledger before, and 17 to 27 s with only the boundary fixed; 1 to 17 ms on a 6M-share copy after, under every plan mode and random_page_cost tried, with its output matching rc.5's; and on a production-sized ledger, 9 to 23 ms for the pool and 2 to 15 ms for miners in 20 of 20 runs, against rc.5's 15.8 to 25.2 s, with identical output), #750's lost connections, fresh read, client-side deadlines and two new stops, and #753's wait for current work; - upgrading: rc.6 adds no migration, capability, setting or script, so an upgrade from rc.5 and a rollback to it replace the binary only, at fence 2 too, and a database that rc.5's migrate migrated is served by rc.6 unchanged, which a cutover that pinned rc.5 can rely on. An upgrade from rc.4 or earlier is as in rc.5's notes, which the section links: migrate for 026, fence 2 and the deferred backfill, #738's snapshot rule and fatal-state clear --timeout-seconds; - the same evidence set as rc.5's, with the pair's serving evidence carried; - the known issues: rc.5's, with #748 fixed and #724's drill race and new-tip shares fixed on the test side (#747, #753), plus #752 after the cutover, #738's fix now planned after the cutover, and #739 not in this candidate. The development-line notes name 3.0.0-rc.6, and they and the 3.0.0 notes link all six candidates' notes. docs/prism-rust-migration.md names rc.5 where it said "this release" of serving at fence 2 and finishing the backfill. That is in three places: that every build before rc.5 refuses fence 2, that the starts of rc.5 and later accept it, and that only rc.5 and later finish a pending backfill. In this candidate's tree "this release" would have excluded rc.5, which does both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ion after 20 s, below qbitd's 30 s idle close (#759) (#760) * 3.x.x: drop an idle node connection after 20 s, below qbitd's 30 s idle close, and send each relay call on a new connection (#759) The frontends' qbitd RPC client kept an idle keep-alive connection for 60 seconds, while qbitd closes one after its -rpcservertimeout, 30 seconds by default. A request written into a connection just as the node closes it is never read, and fails after the connection existed. That is the plain "transport failed" error, which nothing retries. On submitblock it is an unknown outcome, and the block is never offered again. A production deployment logs that error from the CTV broadcaster on about one pass in eight to fifteen, which fits this race: the broadcaster's 30 second interval runs in step with the 2 second refresh interval. The client now keeps an idle connection for at most 20 seconds (POOL_IDLE_TIMEOUT), so it drops one before the node can close it. A compile-time assert keeps the bound below QBITD_DEFAULT_RPCSERVERTIMEOUT. node_client_builder() carries the bound, and the public API's node client and the load harness's node clients are built from it. Rpc::with_pool_idle_timeout sets another bound, for a node with a shorter idle close and for tests. Every RELAY_METHODS call now goes through a second client that keeps no idle connection, so each relay opens a new one. A relay can then fail only to connect, which is provably unsent, or on the connection it just opened, which is a genuine unknown rather than this race. A node with a shorter idle close than the bound, or a proxy that drops idle connections, can no longer race a relay. A relay is rare, and the new connection costs one connect. tests/rpc_idle_close.rs runs the real client against a loopback node that drops a request arriving after its idle limit, as the race does, and counts the connections it accepts. With a shorter bound, the next call opens a new connection and is answered. With the old long bound, it reuses the stale one and fails with the production error, which is not RpcNotSentError. A relay call after the pause is answered on a new connection even with the long bound. The migration guide notes the node-side requirement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.x.x: keep relay calls on the bounded pool, and build Rpc's client from node_client_builder (review on #760) A new connection for each relay put its connect inside submitblock's 1 second deadline (PRISM_BLOCK_SUBMIT_RPC_TIMEOUT_SECONDS). A deadline that passes while connecting is a timeout, not a connect error, so the block would count as an unknown offer and never be offered again: a lost SYN or a slow accept would cost a block that never reached the node. A warm pooled connection within the 20 second bound is the safer path, so relays share it again, as they did before this PR. Rpc's client now starts from node_client_builder(), like the public API's and the load harness's. The pool_idle_timeout accessor and its test only restated the constant, so they are gone. The docs name a proxy's idle timeout beside qbitd's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…t-audits (#765) (#770) import-audits no longer holds SETTLEMENT_LOCK while it sends a legacy audit's UPDATE. Each row's parameters are encoded off the runtime before any transaction opens. The UPDATE then runs on a connection of its own, under transaction-local write timeouts (PRISM_IMPORT_AUDITS_WRITE_TIMEOUT_MS), holding only that row's lock. After that the importer polls SETTLEMENT_LOCK with pg_try_advisory_xact_lock for up to PRISM_IMPORT_AUDITS_LOCK_TIMEOUT_MS, and only the write guard and the COMMIT run under it. A SETTLEMENT_LOCK holder waiting on the row preempts it, and a row that exceeds a bound is rolled back and named in the error, so a re-run resumes at it. SIGTERM and SIGINT stop the command between rows, or roll back the row in progress. Each row logs its parameter sizes and phase times on stderr. Fixes #765. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: node identity foundation (settings, migration 027, node-identity set)
The foundation the 3.1 dual-writer stack builds on (CONTRACT D1, D-4,
D-9, D-12, D-14, D-17). With PRISM_DUAL_WRITER off, the default, no new
setting is read and every statement PRISM runs is 3.0's.
Settings (config/dual_writer.rs): PRISM_DUAL_WRITER, PRISM_NODE_INDEX,
PRISM_CARRY_OWNER, PRISM_PEER_DATABASE_URL and its _FALLBACK,
PRISM_PEER_SYNC_INTERVAL_MS (250) and PRISM_PEER_SYNC_BATCH_ROWS (5000),
parsed into Config::dual_writer. Each error names its setting and never
echoes a value; a peer DSN must not name this node's own database.
PRISM_DUAL_WRITER_DOWNGRADE lets a single writer start on a database
that has run as a dual-writer node, which it otherwise refuses (D-12).
Types: NodeIndex and NodeIdentity (node_identity.rs), PeerSyncStatus
with its watch-channel publisher and the copied-table list
(peer_sync.rs). The Ledger carries the frontend's configured identity,
set once by Coordinator::new.
Migration 027, additive, with no capability and no shutdown proof:
origin_node smallint NOT NULL DEFAULT 0 on the thirteen copied tables, a
catalog-only change even on the partitioned share ledger (no CHECK,
which would scan every partition under the lock); sync_seq on
qbit_pool_blocks and qbit_prism_jobs, drawn after the inserting
transaction's xid so a puller can wait out every older transaction; the
append-only carry-owner journal qbit_prism_node_roles; the never-copied
identity singleton qbit_prism_node_identity; qbit_prism_node_lineage
(own-row floors and the system identifier and timeline of the last
own-log verification); and the peer sync's cursors, conflicts and safe
peer mark, qbit_prism_peer_share_mark().
qbit-prism-server node-identity set --index N personalises a database
once, in the bootstrap or cutover: every copied table's origin_node
default becomes N, share_seq, payout_entry_seq and carry_forward_seq
step by 2 with the node's parity above every value held (A even, B
odd), and the session sequence cycles within the node's half of the
extranonce1 space. A frontend never writes the identity: in dual mode
it refuses a database whose personalisation drifted, and reports one
with no identity or the other node's.
Gated tests cover the migration, personalisation of two databases
beside an untouched single writer, and the downgrade refusal. Every
schema version pin gains 27, and ledger_2x's historical fixture undoes
027.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: describe the peer_sync ORDER_LOCK holder in the metrics metadata
docs/prism-native-metrics.md is generated from the registry and
docs/prism-metric-metadata.json; the holder's description now lives in
the metadata, and the inventory is regenerated from it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: the peer sync engine and the own-log latch
With PRISM_DUAL_WRITER on, each node pulls the rows its peer originated
in the copied tables, by identity, and never a WAL position (CONTRACT
§3, D-1 to D-15). One supervised task per frontend runs a pass every
PRISM_PEER_SYNC_INTERVAL_MS, or at once while a batch was full, over
PRISM_PEER_DATABASE_URL, moving to the fallback when a path fails and
back to the first path every 30 s. Every statement against the peer is
bounded, so a black-holed path fails a pass instead of stalling it.
Each pass checks that this database is personalised as this node
(D-9), that the peer is the other node and runs this cluster's
fingerprint (D-6), and that both carry the same columns of every copied
table; otherwise it syncs nothing and alerts. Then, in order (D-5):
- shares and their header mappings, in share_seq order, scanned over
the peer's whole ledger. Each node commits its own rows in share_seq
order under ORDER_LOCK and raises its own sequence above every peer
row, under ORDER_LOCK, before inserting it, so once a position is
visible on the peer no row of the peer's below it can commit later.
The cursor is therefore the safe peer mark,
qbit_prism_peer_share_mark(), written with the rows it covers (D-14).
Partitions are attached before a row beyond the lead is inserted. A
row whose header this node credits to another share, or that it holds
with other content, is refused whole as a conflict, and the stream
moves past it;
- the carry-owner journal (D-4);
- landed blocks, each whole in one transaction, block row first, with
only their immutable landing columns, derived ones at this node's
defaults (prepared, immature, no publication sequence,
awaiting_maturity), so they count only once this node's reconciler
confirms them. A block this node already holds, its own landing or an
adoption, is skipped whole and its facts compared (D-1, D-10, D-15);
- prepared jobs with their template and balance blobs, blobs first, in
one transaction (D-3, D-11, D-19).
The two sync_seq streams stop at the mark read at the peer's sync
barrier: qbit_prism_next_sync_seq() now takes an advisory lock shared
until its transaction ends, and the puller reads the sequence while it
holds it exclusively, taken with pg_try_advisory_lock so no writer ever
queues behind it. Unlike a cluster-wide xid horizon, a long transaction
in another database never holds the pull back.
Rows held under an identity with other content are never written over:
each is recorded in qbit_prism_peer_sync_conflicts and alerted.
The own-log latch (D-8, D-17): until it is set, a pass pulls back this
node's own rows the peer holds and this database lacks (shares under
ORDER_LOCK in ascending batches, blocks, prepared jobs and journal
rows), raises the own sequences above everything the peer has seen of
this node and the ledger clock above the recovered rows, and records the
system identifier and WAL timeline it verified on. Without the peer it
is set when the database is still on that server, so a plain restart
serves; a new identifier or timeline, or no record, keeps it unset and
alerts. A later loss of the peer never clears it; a change of
identifier or timeline while the frontend runs does. Until it is set,
the Stratum listeners, the refresh and the submit loop wait, so a
restored node never reuses a key the peer holds.
PeerSyncStatus is published through a watch channel, which the
Coordinator holds for health (D-9), and as qbit_prism_peer_sync_*
gauges and counters, declared without samples on a single writer.
Gated tests on two databases: steady sync both ways, out-of-order
commits of shares and landings, a restored node's own-log recovery, the
latch with and without the peer, conflicts and whole-block skips,
peer rows beyond the attached partitions, the fallback path, and the
identity and fingerprint refusals.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: wait for the peer to hold what adopting a found block needs (D-19)
A dual-writer node has no failover standby, so the pair leaves
PRISM_OFFER_STANDBY_APPLICATION_NAME and _FLUSH_WAIT_MS unset there and
nothing requires them. Its peer adopts a block it found if it dies (S8).
Before a found block's submitblock, the node now waits up to
PRISM_PEER_INGEST_WAIT_MS (default 250, 0 off, at most 10000) for the
peer to hold what adoption reads: this node's own shares through the
block's window, and the prepared record the block was built on, whose
template and balance blob the peer applies in the same transaction. Both
come from the peer's own cursors over this node's streams, read on a
small pool of the wait's own over the peer DSN and its fallback. The
wait never holds a block: at its bound, or with the peer unreachable,
the block is offered as before, and the outcome is counted in
qbit_prism_peer_sync_offer_waits_total{outcome}.
A gated test checks that the wait confirms once the cursors cover the
needs, times out within its bound when they do not, reports an
unreachable peer, and is off at 0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: rebuild a node from its peer's database (node-identity repersonalise, table inventory)
CONTRACT D-16 (S7). A node whose disk was replaced is rebuilt from a
promoted physical copy of its peer's database, which says it is the peer.
qbit-prism-server node-identity repersonalise --index N makes it node N's,
in one transaction under SETTLEMENT_LOCK and ORDER_LOCK.
The table inventory (ledger/table_inventory.rs) classifies every table of
the schema as copied (peer_sync::COPIED_TABLES, checked at compile time),
local (cleared, kept or rewritten) or pair-wide (kept), with the decision
beside each entry, and names the rows and columns of copied tables that
are a node's own. Re-personalising derives its reset from it:
- cleared, the peer's in-flight work, which this node must neither settle
nor credit a second time: the candidate outbox and deferred shares,
session reservations, instance rows, CPFP packages and retired funding,
sync conflicts, and issued jobs (prepared records stay); the peer's
fanout claims are handed back as a holder's release does;
- kept, what is derived from rows both nodes hold or is history this
database carries: block, payout and carry states and publication
ordinals, the balance summary, the cluster row (payout revision, chain
epoch, ledger clock, chain view, fingerprint, any halt), divergences,
broadcast history, rollups, vardiff hints, the submission hold and the
operator journals, the partition catalog and the schema record;
- rewritten, the identity, the keys (set's personalisation, every key
above every key held and every position the peer reached in this node's
old streams), the floors with no own-log verification, and the cursors:
shares from the peer's newest share in the copy (D-14 prefix), blocks
and prepared records from the peer's floor, since those commit in no
sync_seq order.
It refuses, changing nothing, unless the identity names the peer, while an
instance row neither drained nor stopped has a heartbeat younger than 60 s
(a copied row of the peer's frontends stops counting 60 s after the copy
stopped following the peer), on the server the peer last proved its own
log on (its own database, not a promoted copy), and while the schema
holds a table the inventory does not classify or lacks one it names.
Gated tests (tests/dual_writer_rebuild.rs) copy node A's database with
CREATE DATABASE ... TEMPLATE and check the result against A table by
table, every refusal, and that the inventory classifies every table of a
migrated schema. A frontend started on the other node's database now
names the command in its alert.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: never carry an audit bundle's sealed bytes, and keep the sync's columns and the inventory in step
canonical_audit_bytes is stored when a database seals a share
partition, so it follows that database's partitions: the table
inventory keeps it local, and the peer sync no longer copies it. A
gated test checks that every column of each copied table is either
carried by the sync or named local by the inventory, never both and
never neither, but a prepared job's expires_at, copied as the peer
held it and never compared, so a column a later migration adds to a
copied table must be classified.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: fix the engine review's findings (barrier hold, latch, snapshots, retries)
From the coordinator's review of cca56f14.
- The sync barrier is one statement, qbit_prism_sync_barrier() (027):
it tries the barrier with pg_try_advisory_xact_lock and reads the
sequence under it. The lock ends with that statement's own
transaction, which a cancel or a vanished client also ends, so a
frozen or partitioned puller never leaves the survivor's job saves
and landings waiting. Before, the puller held a session lock across
two round trips. The two fence comments that forbid advisory locks
now say why this one cannot form a cycle.
- A peer that accepts the connection but cannot be read is unreachable
for the latch (D-8): any error after connecting falls back to the
local rollback evidence, unless own-log recovery found the own log
diverged from the peer's copy.
- Every peer read is one autocommit statement: no REPEATABLE READ
transaction, so no snapshot or xid horizon outlives a statement or a
dead client (#738). The peer session runs with statement_timeout 8 s,
under the client's 10 s bound, lock_timeout 2 s,
idle_in_transaction_session_timeout 5 s, and server keepalives and a
10 s tcp_user_timeout.
- A peer block is skipped only when a row of it is held here under the
same identity with other content (a unique violation, or the D-15
snapshot compare). Any other failure rolls back the block and its
cursor and the next pass applies it again.
- Own-log recovery raises the own share sequence above the rows it
pulls back before attaching partitions for them.
- A pulled share_seq more than 64 partitions above what this node can
hold is refused as a conflict, never raised to.
- Origin reads take the shape only the (origin_node, share_seq) index
can serve: origin_node BETWEEN $1 AND $1, ordered by origin_node and
share_seq.
- The cached sync mark is dropped whenever the peer connection is
recreated, so a mark read from a peer before its restore is never
used after it.
- D-5 across passes: a peer block waits until the share mark covers its
window's last share_seq.
- 027 alters the share ledger and its header mappings last, so a
frontend still serving waits on them only from that step to the
commit.
Gated tests: a stalled puller holds no barrier and leaves no lock; a
peer that answers but cannot be read leaves the latch to the evidence;
a block whose insert fails for a passing reason is retried, not
skipped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: document the peer sync and restoring a node from its own backup
The ledger operations chapter gains how the peer sync runs: its paths and
pacing, the identity, fingerprint and column checks, the four streams in
D-5 order, the sync barrier, conflicts, the own-log latch and its D-17
evidence, the D-19 wait before a found block's offer, and the metrics to
watch. A restore of a node's own backup needs nothing extra, except a
restore that keeps the timeline while the peer is down, which must clear
the verification first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: a peer row that fails to apply here fails only its stream, and a run of failures is alerted
The rest of the coordinator's review item 3. A block, share batch,
journal or prepared batch that fails to apply locally (a lock or statement
timeout, say) used to fail the whole pass: the prepared stream behind a
stuck block stopped too, the peer was reported unreachable and the sync
switched paths, and nothing alerted however long it lasted. Now only that
stream fails. Its cursor stays, the next pass applies the rows again, the
other streams go on, and its lag grows; after 20 failed passes in a row an
ALERT line names it, repeated every minute while it lasts. A failed peer
read still fails the pass and moves to the other path.
The retry test now also checks that the pass and the share and prepared
streams go on while the block waits.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: read the peer's fingerprint with only the sync role's column grant, and test the sync as that role
D6's first pair start found every pass failing with "permission denied for
table qbit_prism_cluster": the peer's fingerprint read filtered on
singleton, but the sync role may read config_fingerprint only. The read
now has no predicate; the table holds one row.
A new gated test, the_sync_needs_only_the_peer_roles_documented_grants,
creates a role with exactly the documented grants on both databases and
runs every read on the peer as it: each node's pull, its own-log check and
the found-block wait. It failed with the same error before this fix. The
ledger operations doc now lists the role's grants and settings, which the
test pins.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: stop the hashrate rollups and the share archive at the safe peer mark
The rollup watermark and the archive's rollup condition both assume no row
commits below one already folded. That holds for a node's own shares
(ORDER_LOCK) but not for the peer's, which the sync inserts later, below
shares this node already holds: a late peer share was never rolled up, and
a partition could leave while the peer's rows could still land in it,
after which the share stream could not insert them at all.
- On a dual-writer frontend the rollup sweep's batch also stops at
qbit_prism_peer_share_mark(): no peer row arrives at or below it, and
nothing rolls up before the first pull. A single writer runs 3.0's
statement unchanged; a unit test pins that the dual statement is 3.0's
text with one clause inserted after the batch bound.
- share-archive plan, on a personalised database, blocks the rollup
condition of every partition the mark has not passed.
- For a long peer outage, when the peer will be rebuilt from this node, an
operator declares its unpulled rows lost with one documented UPDATE of
the new qbit_prism_node_lineage.peer_tail_lost_at (027, local state); both
then go on without the mark. repersonalise clears it on the rebuilt node.
Gated tests: a late peer share below the local maximum is still rolled up,
with the declaration lifting the stop; a partition the mark has not passed
stays blocked with this node's sequence past its bound, clears when the
tail is declared lost, and clears for good once a peer share above its
bound moves the mark past it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: pin the metric census with the peer sync's families
The cardinality test pins the families and series a frontend serves. The
eleven qbit_prism_peer_sync_* families are declared on every frontend
(80 to 91 families, at startup and populated), and the peer_sync holder of
the ORDER_LOCK hold histogram adds its 14 buckets, sum and count (965 to
981 populated series). A single writer publishes no peer sync sample, so
the startup series count is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: index the jobs by origin and expiry for the dual-mode expiry prune
D3's request. A dual writer keeps the peer's prepared records a day past
their expiry (D-3), and its expiry prune selects each node's expired rows
by that node's own (origin_node, expires_at) range. With only the expiry
index, every prune of one node's rows walked the other's retained rows.
027 now builds qbit_prism_jobs_origin_expiry_idx in its transaction: the
expiry prune keeps a single writer's table to its unexpired jobs, so the
build is short, and 027 holds the table's lock from its first ALTER of it
to the commit either way. undo_027 drops it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: fix the local review's findings (a recovery cut short, block pacing, the found-block wait)
From /code-review medium on the local commits.
- A recovery cut short no longer latches on stale evidence. Since the
engine review, a peer that answers but fails leaves the latch to the
D-17 evidence. After a restore that kept the timeline, that evidence
still matches, so a recovery that failed partway set the latch with own
rows missing, before the sequences were raised above the peer's
positions. Now, once recovery finds an own row missing, it first forgets
the stored verification; the latch then waits for the peer until
recovery completes, across a crash too. The restore test locks the
peer's journal to cut recovery short, and checks that the latch stays
down and the verification is gone.
- Peer blocks waiting for the share mark ask for the next pass at once
only while the mark moves. A failing share stream no longer spins the
pass loop.
- The D-19 wait's bound covers this node's own read of what adoption
needs, so a busy local pool cannot hold the offer past it. The wait
keeps one warm connection per path, tries the path that answered last
first, and gives each path an even share of the time left. A first path
that accepts connections and never answers no longer uses up the
fallback's time. A path that runs out of time is not counted as
unreachable. New cases: a slow local read, and a silent first path.
- A unit test without a database checks that the pull's column lists and
the table inventory agree on every copied column.
- Docs: the latch's forgotten verification, the wait's connections, how
many connections the sync role needs, and the archive after a downgrade.
Kept as designed: a block that fails to apply is retried and alerted,
never skipped (the coordinator's review item 3). A pulled share_seq beyond
the pull's headroom is refused as a conflict (item 5); only a jump in the
peer's own sequence reaches it, never a backlog, as the constant's comment
now explains.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual writer: migration 031, the share ledger's (origin_node, share_seq) index, built online
CONTRACT D-14: the peer sync's share pulls and the window cut read the
share ledger by node in share_seq order, and without an index on
(origin_node, share_seq) each is a walk of the primary key until it meets a
row of the node it wants; on a node that has written nothing lately the cut
walks the whole ledger under ORDER_LOCK. Migration 031 adds
qbit_share_ledger_origin_seq_idx. It is index-only and additive, with no
capability and no shutdown proof, like 024. Versions 28 to 30 stay
reserved for the sibling delegates; each version is checked on its own, so
31 is required without them.
031 is the first online index migration on the partitioned ledger, and
PostgreSQL builds no index of a partitioned table CONCURRENTLY. The online
runner (ledger/migration/online.rs) now builds such an index the way
PostgreSQL documents: one leaf per partition with CREATE INDEX
CONCURRENTLY, named <partition>_origin_seq_idx as
qbit_prism_share_partition_create and the share archive name every leaf,
while appends continue; then the index ON ONLY the parent (catalog work
under a SHARE lock on the ledger) and each leaf attached (catalog work under
an ACCESS EXCLUSIVE lock on that leaf index alone), both taken with 017's
2 s lock timeout and retries. PostgreSQL marks the index valid when the
last leaf is attached, and 31 is recorded then. A partition attached before
the parent exists gets its leaf in the next pass; one attached after it gets
its leaf from the parent's definition. A partition that left while the
parent waited for its leaf would leave it invalid for good, so the runner
holds the share archive's lifecycle lock for the whole build and refuses,
before any DDL, a partition still detaching, a leaf attached but invalid,
and anything else under a leaf's name. Interrupted, it resumes: a built
leaf is kept, an invalid one dropped and built again, the parent kept.
migrate --offline-indexes builds the leaves plainly in the transaction that
records 31. The scratch derivation declares an index of a partitioned table
by its parent alone, and refuses an online drop of one, which DROP INDEX
CONCURRENTLY cannot do. A fresh or empty source applies the file in the
migration transaction, which names the leaves the same way.
Registered in REQUIRED_SCHEMA_VERSIONS, NATIVE_MIGRATIONS,
ONLINE_MIGRATIONS and derive_online, and in the recovery evidence's
required versions. Gated tests: the build on a ledger at 27 with shares
waits for an open append on its first partition while later appends land,
a start refuses the database until 31 is recorded, once, and a second start
builds nothing; an interrupted run resumes (an invalid leaf rebuilt, a built
leaf, an attached leaf, the parent and a partition attached meanwhile kept)
after refusing a foreign index under a leaf's name before any DDL. The
offline tests assert 031's plain leaf builds. The undo helpers undo 031
before 017 and 027, every version pin gains 31, and the docs, the
--offline-indexes help and the cutover rehearsal's step names cover it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: refuse to sync without a valid 031 index, and guard the plans of the sync's origin reads
The coordinator's review item 6, and D2's request for 031.
- The sync refuses to run unless qbit_share_ledger_origin_seq_idx is a
valid btree on both databases. Here it is checked with the node
identity, every 30 s; on the peer, in the facts read every pass. The
refusal is an ALERT and qbit_prism_peer_sync_refused{reason="schema"}.
Without the index every share read by origin would walk the ledger.
- The sync's origin reads are built by functions the tests can plan:
HIGHEST_SHARE, shares_scanned_sql(), shares_of_sql() and SHARES_BEYOND.
- New gated test the_origin_share_reads_never_walk_the_other_nodes_rows.
It holds node A's rows under a long run of node B's, where a plain
origin_node = $1 read walks the primary key backward. It plans each
read prepared, at random_page_cost 4 and 1.1, custom and generic. It
requires no sequential scan that reads rows (the empty lead partitions
read none), the origin index for every origin read, and no more ledger
rows visited than the read returns or scans by design, plus a probe per
partition. With the index dropped it fails on every origin read.
A one-off check on 3,000,000 rows across three partitions found the
same shapes: 1 to 10 rows visited, under a millisecond.
- The guard also caught the header-mapping lookup of both share reads:
`share_id IN (SELECT ... FROM picked)` let the planner hash the picked
rows and scan the whole mapping table, a row per share in production.
The lookup now takes `share_id = ANY(ARRAY(SELECT ...))`, the repo's
idiom for this, and probes the share_id key. The guard bounds the
mappings each read visits too.
- The refusal test drops the index on the peer, then here.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual writer: account for 027 in the recovery evidence and the metrics census
Fallout of d8593724 in the gated suites.
Migration 027 gives qbit_share_ledger, qbit_payout_carry_forward and
qbit_pool_payout_entries an origin_node column, and the recovery evidence
export (scripts/prism-recovery-evidence.sql) records those three tables as
whole rows (to_jsonb). Every row of a migrated 2.x.x source therefore gained
"origin_node": 0, so its share, carry and payout digests no longer matched
the source's: the restore runbook's `cmp source.summary.json
migrated.summary.json` would fail after any 3.1 migrate, and so did
migration_rollback's evidence comparisons across a migrate
(frozen_2x_backup_restore_reconciles_before_ack_and_exposes_post_ack_loss,
the_evidence_export_defers_share_hashes_while_the_backfill_permits_serving,
and the cutover rehearsal's "recovery evidence unchanged" check).
origin_node is a row's provenance in a dual-writer pair, not accounting.
The export already leaves it out of every other table 027 gave it (blocks,
audit bundles, share hashes and audit snapshots are recorded by column
list), as the script's header says native metadata is never exported. The
three whole-row records now drop it (`to_jsonb(x) - 'origin_node'`), on a
2.x.x source a no-op, so a source and its migrated copy export the same
rows again. The parallel exporter's cut point follows the shares statement,
the test support's accounting_state drops it the same way, and the restore
runbook says so.
The observability census pins the populated series count: the new
peer_sync ORDER_LOCK holder adds one hold histogram, 16 series (13 buckets,
+Inf, sum and count), so 965 becomes 981.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: classify the 031 runner's and the index check's call sites in the checkout census
scripts/check_prism_pool_acquires.py found the sync's new index check
(origin_index_valid, a pooled read) and the 031 runner's describe() calls,
which only format refusal text, unclassified.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: fix the second local review's findings (peer read sizes, a silent wait, repersonalise's read, the 031 wait)
From the second /code-review medium, on the integrated branch.
- The sync's reads fit the peer's 8 s statement timeout: the lag count
stops at 100,000 rows (was 1,000,000), and PRISM_PEER_SYNC_BATCH_ROWS at
most 20,000 (was 100,000), since a batch is one statement on the peer.
- A peer URL the PostgreSQL client cannot use fails the settings check by
name, and the found-block wait's constructor returns the error instead
of turning the D-19 wait off silently.
- repersonalise reads the peer's newest share in the shape 031's index
serves (HIGHEST_SHARE), never a walk of this node's rows under the locks
it holds.
- The 031 build logs every 30 s while a share-archive command holds the
lifecycle lock it waits for.
- 027 says why the header mappings are altered before the ledger: an
append reads the mapping first. The review proposed the other order,
which would deadlock with an append waiting between the two.
- The rebuild runbook says repersonalise can tell the peer's own database
from a promoted copy only while the peer has an own-log verification.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: stop the frontend when its database is restored under it, recheck columns every pass, never confirm an unread dependency
Codex's review of 193e7fd8 on #766, a local /code-review medium of the
first fix, and CI.
- P1: a database restored or promoted under the running frontend (its
system identifier or WAL timeline changed since the latch set) now stops
the frontend. The peer sync ends with OwnLogLostWhileRunning, which the
server takes as a critical task exit. For this one it cancels every task
at once instead of draining them, so no open transaction commits another
own row. The process manager restarts it, and the restart's startup gates
serve nothing until own-log recovery completes. Until it has exited, the
ledger refuses every share append, a credited deferred share included
(OwnLogLost). The check runs before the local refusals, so a restore
that also trips one is still caught. Gating each writer instead (an
earlier draft) left holes: a pass that recovers before the latch drop
is seen, and appends already waiting for ORDER_LOCK.
- P2: both sides' copied columns are compared every pass, before any pull.
This database's are read before the peer is, so a local failure is never
taken for the peer's. A rolling migration that gives one node a copied
column first stops the sync before a cursor passes rows whose new column
this node would drop.
- P2: a failed local read of what adopting a found block needs ends the
D-19 wait at once, unconfirmed (counted unreachable, the error in the
warning), never confirmed, and never a wait for nothing.
- CI: the fixture copy (CREATE DATABASE ... TEMPLATE) retries for up to
10 s while its source shows a session whose client has just closed:
repersonalise's copy test met one on a busy runner.
Tests: the append refusal and its lifting in the latch test; a failed
local read in the offer-wait test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: stop the frontend when its database changes identity while it runs
Codex's review of 31bf958b on #766. Once the peer sync has seen the
database ready as this node, a later identity check that finds it
unidentified, the other node's, or drifted (an origin default or a key
sequence's parity changed) now stops the frontend, as a restore under it
already does: the sync ends with FrontendStop::IdentityChanged, the
server cancels every task at once, and the ledger refuses every share
until the process exits. The restart applies the startup checks: it
refuses a drifted database, and admits no miners on one that is not this
node's. Before, only the sync paused, and the frontend went on writing
rows under an identity that was no longer its own.
The stop error is now one type, FrontendStop, with a variant per cause
(OwnLogLost, IdentityChanged), and both causes take one path: it raises an
ALERT, refuses every share, drops the latch and publishes it, so health and
the gauges stop reporting the node ready while it exits. The identity
counts as seen ready on any ready check, whatever the 031 index's state.
Rows written between a drift and the next check, every 30 seconds, keep
what the database gave them; the restart refuses the drifted database
until node-identity set repairs it.
New gated test: a_frontend_whose_identity_changes_while_it_runs_stops
(a drifted origin default, and an identity row rewritten as the other
node's; the published status and the refused share).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: end the sessions left on a fixture's database before copying it
repersonalise's copy test failed twice in CI on "source database ... is
being accessed by other users". PostgreSQL's CREATE DATABASE ... TEMPLATE
already waits 5 s for other sessions, and the 10 s retry added in
31bf958b met the same session for 265 s: not a closing backend but a
session that stays. Locally none remains.
FixtureDatabase::copy now waits up to 10 s for every other session on
the template to end, then names what remains on stderr and ends it with
pg_terminate_backend. The caller closed its pools, so anything left is
background work it no longer owns. The retry around CREATE DATABASE is
gone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: the coordinator's review of 193e7fd8, Codex's third round, and zeroed counters
The coordinator's review (status/review-D1-193e7fd8.md):
- P2, rollups during a peer outage: the sweep's cap at the safe peer mark
froze a survivor's own rollups while reporting them caught up. It is
gone. A dual writer sweeps as a single writer does, after taking the
progress row. A peer share the sync inserts below the watermark is
folded into the rollup buckets by that pull, in its transaction, under
the same row lock: the same grains and buckets as rollups.sql, which a
unit test pins. The two serialise on that row, so a share is folded
once, by whichever saw it. The archive gate still stops at the mark, and
peer_tail_lost_at now lifts only that.
- P2, repersonalise on the peer's own database: a copy whose lineage
carries no own-log verification is refused unless --unverified-copy
says the operator checked it. When PRISM_PEER_DATABASE_URL is set, the
command first reads that server's system identifier and timeline and
refuses if they are this database's, or if it cannot be read, unless
--unverified-copy.
- P3: the column test pins every copied table's full column list, so a new
column needs a decision before it ships.
- P3: a pulled share whose share_id this node already credits at another
share_seq is refused as a conflict (the ledger is unique on share_id per
partition only).
- P3: own rows pulled back in own-log recovery are exempt from the pull
ceiling; the sequence is raised to the highest of them. A backup node
restored from an old backup no longer wedges on its own rows.
Codex on a732525e:
- P1: own-log recovery reads every own landed block and prepared job the
peer holds from the lineage's sync_seq floor, not from the highest one
held here: those streams commit out of sync_seq order, so a backup can
hold a block whose earlier-numbered sibling was still open. Missing rows
are counted before the verification is forgotten; held ones are
compared. New gated test:
recovery_finds_an_own_block_committed_after_a_later_numbered_one.
- P2: a found block whose prepared record is no longer held here (pruned
while it waited) is offered unconfirmed, like a failed read.
D5's paging rules: a dual-writer frontend's peer sync counters (rows and
conflicts per table, failures per path, offer waits per outcome) start at
0, so the first increase pages. A unit test pins it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: bound own-log recovery's reads, and the local review's other findings
The local review of 4786fb64:
- Own-log recovery read whole every own landed block the peer holds since
the lineage's floor, with its audit bundle and share snapshot, at every
start. It now reads their keys, a thousand at a time, and reads whole
only the blocks missing here.
- It also read own prepared jobs this node had pruned at expiry while the
peer kept its copy, as D-3's retention will. Each counted as a missing
own row: it raised the rollback ALERT, forgot the verification and was
put back. Recovery now reads only the unexpired ones. New gated test:
recovery_leaves_own_prepared_jobs_pruned_at_expiry.
- Own rows read back in recovery had no ceiling, so one damaged row far up
the peer's sequence would have raised this node's sequence to it and
attached every partition on the way. The ceiling now counts from the
peer's cursor over this node's shares, which every own row the peer
holds came through, so a backup node restored from an old backup still
takes its rows back. New gated test:
recovery_refuses_an_own_row_beyond_the_peers_cursor.
- A pulled share that broke two refusal rules was recorded and counted as
two conflicts. It is now one, with both reasons; the conflicts test
covers it.
- The late-share fold runs only when the pull inserted rows.
- Every frontend on a personalised database takes the rollup progress row
before its sweep, with PRISM_DUAL_WRITER on or off. So a single writer
running beside a dual writer (a rollback in progress, or a
misconfiguration) cannot pass a late peer share unfolded.
- node-identity repersonalise reads the peer's server within one 10-second
bound, where only the connect was bounded before. It tries
PRISM_PEER_DATABASE_URL_FALLBACK when the first path does not answer.
Kept as they are:
- A found block whose prepared record is gone is offered at once, counted
unreachable, without waiting for its shares. The record goes only at
expiry, so the block has waited at least a job's lifetime, and nothing
can confirm the record.
- The missing-row checks before each apply stay. They forget the
verification before any own row goes in, so a recovery cut short between
pages never leaves the old evidence trusted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: compare held own blocks in recovery, and the second local review's other findings
The local review of a481852d:
- Reading only the missing own blocks dropped the comparison of those held
here. The key read now carries a digest of each block's immutable facts
and audit digest, which is everything apply_block compares. Recovery
reads whole the blocks missing here and those held with another digest.
A held one with other facts is a conflict, and the own log has diverged.
New gated test: recovery_finds_an_own_block_held_with_other_facts.
- The own-row ceiling now counts from the higher of the peer's cursor over
this node's shares and the peer's lineage floor. That covers own rows a
peer held when it was personalised, as well as the ones it pulled.
- If the peer returns fewer of the listed blocks than were asked for,
recovery fails and starts again instead of going on without them.
- repersonalise reports why each path to the peer's server failed, not
only the last one.
- The docs say to stop every frontend on a database before personalising
it. A frontend reads at start whether its database is personalised, and
its rollup sweep depends on that.
Kept: the recovery's unexpired-only read of prepared jobs also leaves out
held ones that the peer counts as expired. A prepared record is immutable
apart from its expiry, so a difference there is damage, not a lost row,
and this node prunes such a record anyway.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: forget the own-log verification when recovery finds the own log diverged
The local review of 1116c09a: a conflict among this node's own rows in
own-log recovery marked the own log diverged in memory only. The
verification stayed recorded, so a restart without the peer would latch on
it and serve a diverged own log. Any such conflict now forgets the
verification as well. The docs promise the latch stays unset with or
without the peer until an operator decides, and this makes that hold.
recovery_finds_an_own_block_held_with_other_facts now checks that the
verification is gone and that a peerless restart stays down.
The held-block comparison also probes each key once (a left join) instead
of twice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: a found block is confirmed only below rows the peer holds, not rows it refused
Codex on 9fa64d1a (P2): a share or prepared job the peer refused as a
conflict still moves the stream's scanned_through past it, as it must or
the stream would stall. The found-block wait read that cursor as proof
that the peer holds every row adoption needs, so it could confirm a block
whose window held a share the peer refused.
For shares and prepared jobs, the cursor's ingested_through now means
this: NULL while this node holds every peer row it scanned. A pull that
refuses one sets it just below the first refused row, where it stays
until an operator resets it. For a prepared batch with a conflict, that is
below the batch's lowest job. The wait reads the position through which
the peer holds this node's rows, so a block above a refused row is never
confirmed; it runs to its bound and counts timed_out. Blocks keep no stop,
since nothing waits on them. Repersonalise starts the rebuilt node's share
cursor with nothing refused.
The runbook's reset moves the stream's cursor back to the stop and clears
the stop, naming the stream. The next pull then reads the refused row
again, instead of the wait trusting a row that never came. 027's comment
on the column says what it now means (text only; no database change).
Tests: the conflicts test gains a prepared job B holds under A's id with
another parent. It checks each stream's stop, that a later clean pull
keeps the stop, and that the wait confirms needs below each stop and
times out above. The wait's unit test covers held positions below the
scanned ones.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: quarantine refused peer rows for good, and clear the own-log verification with an own-row conflict
The coordinator and D2 on f5597cb5's reset: a refused peer row must never
be inserted later, and the safe peer mark must never step back. A row that
landed below a window cut a window had already used would break the
landings and audits built on it (contract section 4.4).
- A refused share or prepared job is quarantined for good: recorded,
never inserted, never read again. The pull moves past it and nothing
moves back. The rewinding reset is gone; there is nothing to reset.
- For shares and prepared jobs, ingested_through now holds the highest
refused key (GREATEST; NULL while none was). For a prepared batch with
a conflict, that is the batch's highest job.
- The found-block wait confirms only a window whose first share lies above
the highest refused share, and a prepared record above the highest
refused one. Confirmations resume by themselves once windows move past
a quarantined share. AdoptionNeeds gains first_share_seq, taken from the
candidate's window.
Codex on f5597cb5 (P2): an own-row conflict in recovery was committed
before the verification was forgotten, so a crash between the two could
leave a restart without the peer trusting the old evidence. Recording a
conflict on a row this node originated now clears the verification in
the same transaction.
Tests:
- The conflicts test checks that later pulls never lower the safe peer
mark, and never insert a peer share at or below a mark already passed.
- It also checks that the refused rows never arrive, that each stream's
highest refused key holds, and that the wait confirms only above it.
- The divergence test applies a differing own block directly and checks
the verification is gone with the conflict.
- The wait's unit test covers windows and records around refused keys.
The docs give the quarantine rules instead of the reset. 027's comment on
the column changes in text only; no database change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: the found-block wait checks the prepared record the block's job names
Codex on 7658fa27 (P2): the wait found the block's prepared record by its
window's anchor and balances, taking the newest. A template update over an
unchanged window prepares a second record with the same two, so the wait
could check a newer record the peer holds instead of the one the block was
built on. That one might have expired, or been quarantined there.
The candidate keeps the issued job it was found on, and every issued job
names its prepared record (payload.prepared_key; full jobs are saved only
in tests). The wait now reads that exact record's sync_seq:
- If the job or the record is gone, the offer is unconfirmed.
- A record the peer holds anyway is no need. That is a record that is the
peer's own, or one from before 027, which both databases of a pair
started from.
The offer-wait test prepares a newer record over the same window. It
checks the lookup returns the one the job names, that a missing job fails
it, and that a peer-origin or pre-027 record is no need.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: the first rollup sweep and a pull serialise before any progress row exists
Codex on 7fed050c (P2): before the first sweep there is no progress row,
so the dual sweep's lock and a pull's fold both locked nothing. The first
sweep could pass a late peer share that a still-open pull had inserted,
and nothing would ever count it.
- The sweep on a dual-writer database creates the row first, at 0, which
is what a sweep reads for a missing row and what its own statement would
create.
- A pull whose fold finds no row locks the progress table (SHARE ROW
EXCLUSIVE) until it commits, and folds again. So the first sweep waits
for it, and the pull waits for a first sweep that created the row
meanwhile and then folds against its watermark.
- A pull never writes the row, so share-archive plan's "the rollup sweep
has never run" still holds whenever there is no row.
- The fold now also reports whether a row was there.
New gated tests:
- the_first_sweep_waits_for_a_pull_that_found_no_progress_row: without
the sweep's creation step it counts 2 of 3 shares.
- a_pull_that_found_no_progress_row_folds_against_the_first_sweep: it
drives the real pull. Without the table lock, the pull does not wait.
Codex's other P2 on 7fed050c, that the found-block wait does not verify
the database behind its path, is deferred as a misconfiguration case. The
rationale is on the thread and in the PR body's Follow-ups.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: share archives carry each row's origin_node (format v2)
Codex on f59ac088 (P1): the share archive's rows, its read projection and
the restore's insert all left out origin_node (migration 027). A restored
partition would therefore attribute every row, the peer's included, to
the restoring database's own node.
- ArchiveRow gains origin_node, which is read with the row and restored
explicitly. It is written last, and only when it is not 0, so a v1 line
stays a valid line for a row of node A's.
- A new schema, qbit.prism.share-archive.v2, is written exactly when some
row has another origin. Every archive of a database that never ran a
dual writer stays v1, byte for byte, and a 3.0 binary still reads it.
An archive that carries the peer's rows is refused by name by a binary
that would restore them as its own.
- The reader accepts v1 and v2, and requires the declared schema to be the
one its rows imply. Spelling out a 0 origin is not canonical.
Tests:
- New gated test share_archive::a_v2_archive_restores_each_rows_origin:
archive, verify, seal, detach, drop, then restore a partition with 50
peer rows. It checks v2 and every row's origin.
- The restore round-trip test now checks that an all-node-A partition
stays v1.
- The row unit tests cover the v2 encoding.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Cargo.lock: rustls 0.23.45 -> 0.23.46 for RUSTSEC-2026-0346 to -0349
The Dependency audit (cargo deny) fails on every PR from 2026-10-16, on
four rustls advisories published since the last green run, each fixed in
0.23.46. Nothing else in the lockfile moves.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: an attached archive import maps each header under its share's origin
The local review of eb12d75e:
- An attached import (restore --attach into a database with no catalog row)
rebuilt the header mappings without origin_node. On a database
personalised as the other node, they took its default. Each mapping now
takes its share row's origin.
- An archive row's origin must be 0 or 1. Anything else is refused while
streaming, on archive, verify and restore alike.
- a_v2_archive_restores_each_rows_origin also imports the archive,
attached, into the database with node B's origin defaults. It checks
every row's origin and that no mapping differs from its share's.
- The ledger ops chapter describes v2, and says to archive and restore a
dual-writer database's partitions only with a 3.1 binary.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: the origin index check requires the index's definition, not only its name
Codex on 1fc5d3c6 (P2): the peer sync accepted any valid btree named
qbit_share_ledger_origin_seq_idx, even one on another table or on other
columns, for example after a failed manual repair. It would then sync
without the (origin_node, share_seq) access path its origin reads need.
The check now also requires the index to be on qbit_share_ledger, on
exactly (origin_node, share_seq), with no predicate or expression, on
this node and the peer alike. the_sync_refuses_wrong_identities_and_fingerprints
replaces the index with one of that name on share_seq, which is refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: refuse a pulled share whose ID is held or retained here as a rejected share
Codex on 1ed3365b (P2): a share both nodes rejected, each at its own
share_seq in another partition, passed the pull's checks. The ledger is
unique on share_id per partition only, and a rejected share has no header
mapping. The pull then inserted a second row under the same global ID, and
archiving the newer partition would later fail on the retained ID.
Such a row is now refused as a conflict when the ID is held here as a
rejected share at another share_seq, or retained in
qbit_prism_rejected_share_ids at another share_seq. A refused rejected
share is in no window, so it never raises the stream's quarantine key that
holds the found-block wait back. The conflicts test adds a share both
nodes rejected: it is refused once, and the key stays put.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: record the own-log verification only on the server recovery began on
Codex on 75e24918 (P1): each step of own-log recovery takes its own
pooled connection. So a failover of the local database partway through
could leave the recovered rows and the raised sequences on the old
primary, while the verification and the latch landed on the promoted one,
which lacks them.
Recovery now captures the system identifier and WAL timeline when it
begins. record_own_log_verified applies only while the database still
shows them, checked in the same UPDATE. Otherwise it refuses, the latch
stays down, and recovery runs again on the new server. The latch test
checks that a verification under evidence the server no longer shows is
refused and leaves the recorded one alone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* qbit-prism: a dual-writer window's per-node cut in the reward manifest
A new WindowCut type ({"0": share_seq|null, "1": share_seq|null}) names
the highest share_seq of each node's rows a dual-writer window may include.
The reward manifest and its stored header carry it as `cut`, absent when
None, so every existing bundle, fixture and vector keeps its bytes. The
coinbase commits the cut through the reward-manifest leaf.
The builders take the cut as an input, and the verifier rebuilds a manifest
under its own cut. Every window share must be at or below the cut's highest
entry (ShareOutsideWindowCut). Dropping, adding or altering a cut fails at
the commitment leaf. The build-audit-bundle binary accepts `window_cut`.
Adds a dual-writer fixture: both nodes' share_seq parities, pre-3.1
history, a gap, late node-1 rows outside the cut, and a partially counted
crossing share. Also adds the window_cut_bundles tests (library and binary
build, verify and canonicalize; tampering; parallel equals serial; a
randomized two-node reproduction model) and pinned golden digests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* qbit-prism: pin the dual-writer fixture's CTV build and refuse one share_id from both nodes
The CTV-settlement build of the dual-writer fixture is pinned beside the
coinbase-options build: the reward manifest, cut included, also feeds every
fan-out precommitment, and both builds commit to the same manifest. A
window holding one share_id from both nodes is refused by the fold in the
builder and every verifier, never credited twice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* qbit-prism-server: per-node window cuts for the dual writer, and migration 028
In dual-writer mode every payout window records a per-node cut: the
highest share_seq of each node's rows it may include. A row is eligible
when it is eligible as in 3.0 (accepted, and accepted and issued at or
before the anchor) and its share_seq is at most its node's entry.
This node's entry is its newest own row, read under ORDER_LOCK in the
cutoff's statement. The peer's entry is its newest row at or below the
peer sync's safe mark (qbit_prism_peer_share_mark(), D-14) that is stamped
at or before the anchor. A peer row that arrives late is above the peer's
entry, so it is outside every window built before it and joins later
ones.
The anchor rule stays on every row (D-13). Each node's stamps rise with
its share_seq, so the cut alone decides membership. Clock skew can only
delay a peer's newest shares, never make a window irreproducible or a
landing fail.
Every path that selects, re-reads or proves a window applies the cut: the
snapshot and full scan, read_window, verify_durable_range, the in-lock
count, audit persistence and reconstruction, the #619 holding probe and
the leaf witness. The incremental advance reads each node's new rows
between the old and fresh cuts and merges the peer's late rows in
share_seq order (advance_in_cut), proved by the existing leaf witness and
eligible-row count.
At a 200k-row window a merge of 1,000 late peer rows and 1,000 own appends
takes about 92 ms, against 1.5 to 1.6 s for the full scan.
The cut reads are shaped so only the (origin_node, share_seq) index can
serve them. A plain max() with an equality filter was measured walking the
primary key through the other node's whole run inside ORDER_LOCK. The
index ships as D1's migration 031; until a ledger has a valid one, a
dual-writer snapshot refuses before it takes any lock. The dual-writer
test fixtures create the same index meanwhile.
A cached window is not reused for a new template once the peer's mark has
moved, since the peer's newer rows can all lie below this node's cutoff.
The refresh probe reads the mark in dual-writer mode only.
WindowRef carries the cut, absent when None, inside the digest-checked
candidate document and prepared payload. Migration 028 adds cut_seq_0 and
cut_seq_1 to qbit_prism_audit_snapshots. Bundles commit the cut in
reward_manifest.cut, except a bootstrap window's. With dual-writer mode off
nothing has a cut, and every eligibility, snapshot, probe and dependency
statement is 3.0's.
Tests:
- a 512-case proptest of the merge against the full scan;
- gated tests with both nodes' rows in one ledger: the model, late rows
with a no-cut negative control, clock skew, reproduction on the other
node's database, a seeded advance walk, single-writer parity, the plan
shape, the refusal without the index, and the peer mark;
- gated landings with late peer rows;
- a measurement harness.
Every version pin gains 28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* qbit-prism-server: rebuild a dual-writer block's audit with its window's cut, so it lands
The audit rebuild that every post-offer landing and every reconciliation
retry runs (build_claim_parts) called the serial builders, which build
without a cut. The coinbase was built by bundle_build with the window's
cut. So every dual-writer block was accepted on chain, but its landing
failed with "audit parts' window cut None differs from the candidate's
window reference" and the row stayed in reconciliation. D6 found it in
its S1 and S5 scenarios.
The rebuild now applies the rule bundle_build applies. A dual-writer
window's audit carries the reference's cut, whether or not the peer's
entry is empty. A single writer's window, and the bootstrap window with
its one synthetic share, carry none, so those bytes are 3.0's.
A row the failed landing left in reconciliation lands on its next retry:
it keeps every input, and the retry rebuilds the audit from them.
Tests:
- three coordinator tests through process_candidate (offer, rebuild,
landing): an idle peer with no mark, under direct settlement; both
nodes' rows, under CTV settlement; and a block whose first landing
failed, which lands on its reconciliation retry without a second offer;
- the ledger-level landing tests now land the rebuilt audit, check it is
the one the coinbase committed to, and gain an idle-peer case.
Without the fix, the three coordinator tests and both non-bootstrap
ledger tests fail with D6's error. The window_switch fixture now builds
coinbases with the snapshot's cut. That cut is None for every
single-writer test, so their bytes are unchanged.
The test fixtures' stand-in index takes migration 031's name,
qbit_share_ledger_origin_seq_idx (CREATE INDEX IF NOT EXISTS). The
plan-shape guard pins its cut reads to that index or its partitions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* qbit-prism-server: alert once a minute on a dual-writer ledger without 031's index, with a gauge
Without migration 031's (origin_node, share_seq) index, a dual-writer
snapshot refuses before it takes any lock, and the refresh loop logged
that refusal on every tick and wake: 920 lines in three minutes per
frontend in D6's integration run.
The refusal is now typed (OriginIndexMissing). The refresh loop logs it
as an ALERT at most once a minute, and again at once after a successful
refresh. The snapshot sets a new gauge,
qbit_prism_dual_writer_origin_index_missing: 1 while it refuses, 0 once
it finds the index. A single writer declares the gauge without a sample.
The metric census pins move with it. They also take in the peer sync's
families, which the base had not counted.
The refusal test checks the type, the gauge in both states, and only its
own database's advisory locks; other tests share the cluster.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* qbit-prism-server: dual-writer review fixes for the own entry, the holding probe and bootstrap work
Fixes from the second local review and the coordinator's review of
a3e9854.
The cut entries:
- This node's entry is now its newest own row eligible at the anchor, so
both entries are rows the window's predicate admits, and the higher one
is the window's newest row. A rejected or future-stamped legacy row on
top no longer stops the incremental advance.
- The own-cut read inside ORDER_LOCK first probes at or above the
retained window's entry, which reads only the newest partitions, as
3.0's cutoff does. The probe of every partition runs only for a first
snapshot, or when that one finds nothing.
- The index check requires a btree whose two columns share one
direction.
The #619 holding probe of a dual-writer window also checks this node's
entry row, and that the peer sync's mark covers the peer's entry. A
restore can lose each log's newest rows while the window's last row
survives.
Bootstrap work: empty-window work is replaced as soon as the peer mark
moves, as a first share replaces it. The refresh trigger label and the
work's equivalence check now see the cut.
The incremental advance:
- the delta bound counts the union of the nodes' ranges, not their sum;
- the margin walk and the witness-and-count proof are shared with 3.0's
advance, and its statements keep 3.0's text.
Statements:
- The refresh probe reads the peer mark in its revision statement. A
single writer's statement is 3.0's.
- compact_issued's two dependency statements are built at compile time
from one text. A test pins the single writer's to 3.0's digest.
Tests:
- the holding probe under a rewound mark, a lost own entry and a pruned
prefix;
- the bounded own-cut read across four partitions;
- the plan guards with the new parameters;
- a BRIN index refused;
- bootstrap work replaced when the mark moves, while full work keeps its
cadence;
- the union bound;
- pinned statement texts.
audit_completeness's hand-made snapshot table gains 028's columns.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* qbit-prism-server: final local review: the peer entry under both marks, one margin step, wording
Fixes from the last local review before the PR:
- The peer's entry is read in the second transaction, after ORDER_LOCK.
It is now bounded by the mark read there as well as by the mark the
cutoff's statement read, so a mark rewound in between never admits a
peer row whose lower rows this database no longer holds. The cutoff
still bounds every admitted row.
- The dual-writer cutoff statement is planned with its values, as every
bounded ledger read here is, so the retained entry prunes partitions in
every plan.
- Both merges carry one Margin, whose take step is the only copy of the
saturating fold below the retained first row. The proptest uses it too.
- read_range_paged says it applies 3.0's predicate with no cut.
- The ops doc says a held window's last row matches the window's
predicate, whichever node wrote it. It also says bootstrap work is
replaced whenever the peer's mark moves until a window holds shares.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* qbit-prism-server: drop the dual-writer tests' stand-in for 031's index
The base now carries migration 031, which creates
qbit_share_ledger_origin_seq_idx on every fixture the tests migrate. The
fixtures' own CREATE INDEX IF NOT EXISTS is gone. The refusal test, which
drops the index, still restores it under 031's name.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* qbit-prism-server: pre-PR review: the peer entry in one statement, the readiness probe, evidence
Fixes from the last local review before the PR:
- The peer's entry is bounded by the mark now inside the peer-cut
statement itself, beside the mark the cutoff's statement read, so a
rewind between statements is covered too, with one round trip fewer.
The statement also returns the mark the entry used, and the snapshot
records that one for the refresh to compare. A rewound mark that comes
back then rebuilds the window.
- The dual-writer cutoff statement is cached again, as 3.0's cutoff is,
so no plan is made under ORDER_LOCK. A generic plan prunes the leaves
below the retained entry at executor start, which the bounded-probe
guard checks.
- The public read's schema-readiness probe requires 028's cut_seq_0 and
cut_seq_1, which audit reconstruction selects.
- The recovery-evidence tamper test also changes a snapshot's cut and
expects the evidence to see it.
- migration.rs and the migration guide say 028 is taken, and 29 and 30
stay reserved.
- The measurement harness drops 031's index for its no-index case. The
refusal test drops its BRIN index before it restores 031's. The
plan-shape guard sets a peer mark, as a synced ledger has one, so the
peer read's walk is bounded and planned on 031's index. The
dual-writer ops paragraphs are rewrapped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* scripts: classify the window-cut changes in the pool-acquire census
Three of this branch's checkout sites changed shape:
- audit reconstruction's read_range call now passes the stored cut;
- the snapshot's advance call now passes the cut;
- verify_durable_range's newer-row probe is now two timed checkouts, one
per branch, with only one running.
The census entries are updated to match. The timed count goes from four to
five, and the policies stay the same.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* qbit-prism: bump the build daemon's serve protocol to 3 for window_cut
A build request's window_cut, added for PRISM 3.1's dual-writer windows,
changes what the daemon signs. A protocol 2 daemon ignores the field and
signs a reward manifest without the cut, and the build summary cannot show
that. The handshake now announces 3, so a client that checks the version
exactly refuses a pre-cut daemon instead of distributing work that does
not commit to the cut it asked for.
The native server builds in process and never starts the daemon. The doc
comment now says so, and that the client which checked the version was the
retired Python coordinator.
A new test sends the dual-writer fixture's window, with its cut, through
--serve. The daemon's job summary must equal a one-shot build's with the
cut, and that differs from the build without it. The serve tests that pin
the handshake, and the deleted-test map, now say 3. From Codex's review of
#768.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* qbit-prism-server: compare a prepared row's window without its cut in every mode
The issued-job saves check that a prepared row's payload agrees with its
typed columns. In dual-writer mode the payload's window was compared
without its cut, which the columns never carry. A single writer issued
3.0's whole-window comparison. A single writer started under D-12's
downgrade would therefore refuse every live prepared row a dual-writer
frontend wrote, though the work is valid and lands under its stored cut.
The comparison now drops the cut key in every mode. On a 3.0 payload,
which has no cut, that changes nothing. The window must be an object, and
a cut in it an object too, so a malformed payload is a mismatch, as 3.0
reported it, and never a statement error. It is the statement's one
difference from 3.0's text: the second deliberate single-writer exception,
beside audit reconstruction's read of the cut columns, and for the same
D-12 reason.
A unit test pins the statement to 3.0's digest once that one expression is
swapped back. A gated test writes a dual-writer prepared row and checks it
on a ledger with no dual-writer identity. The row matches its columns
there, and 3.0's whole-window comparison refuses it. Another key, a null
or scalar cut, a scalar window or a changed column still fails. From Codex's review of #768.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: the cut fixtures fake a pull that refused nothing
D1's 7658fa2 made a cursor's ingested_through the highest peer row a
pull refused as a conflict, quarantined for good, NULL while none was.
The window-cut fixtures still set it to the mark, which now reads as a
pull that refused every peer row through it, a state the peer's
found-block wait would never confirm a block under. They now set only
scanned_through, as a clean pull leaves the cursor.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: pin the audit snapshot's cut columns in the sync, and the quarantine in D-14's wording
D1's the_sync_carries_every_column_the_inventory_does_not_keep_local pins
every column the peer sync copies, and failed on the merge: 028 added
cut_seq_0 and cut_seq_1 to qbit_prism_audit_snapshots. The sync carries
that table whole, so a synced block's snapshot already brings its cut;
both columns are now pinned. The snapshot conflict check still compares
only the content-determined columns, as D1 wrote it for windows that
share one share array.
Since D1's 7658fa2 a peer row the sync refused as a conflict is
quarantined for good, below a mark that moves past it. Windows taken here
leave it out and stay whole, but one whose range spans it reproduces only
on the node that took it. cut.rs and the ops doc said every peer row at or
below the mark is committed here; they now name the exception. The
window-cut integration helper's mark upsert also sets peer_node, as its
twin in cut/dual_writer_tests does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: a new template never reuses a window with a peer row pending its anchor
Codex on #768 (PRRT_kwDOS9jIOc6rEtvE): a peer row at or below the mark
but stamped after the window's anchor, by a peer clock running ahead,
joins a later anchor's window with no new row and no move of the mark.
The reuse rule compared only the mark and the accepted cutoff, so a new
template could keep the old window, and leave that share out of the
work, until the reanchor interval ran out.
The peer-cut statement now also returns the peer's newest accepted row
under the same bounds, whatever its stamps, by the same index walk on
031's index, which stops there. When that row is above the peer's entry,
the snapshot reports a peer row pending (PeerCut::pending, carried on
SnapshotCapture and RefreshWindow), and RefreshWindow::reusable refuses
the window, so the next template reads a fresh one. Same-template work
keeps 3.0's cadence, as before.
Tests: the gated clock-skew test asserts the flag while the rows wait,
its clearing once this node's clock passes them, and that a rejected row
above the entry is never pending. A unit test covers reusable, and a
coordinator test shows new templates at an unchanged revision reading a
snapshot while a row is pending and reusing the window once none is.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: the 2.x audit envelopes refuse a window cut by name
Codex on #768 (PRRT_kwDOS9jIOc6tByaz): the audit-body-ref.v1 and
audit-bundle.v2 envelopes check that their shares form one contiguous
share_seq run, which a dual-writer window with a cut need not be (a late
peer row outside the cut leaves a gap), so such an envelope failed as
non-contiguous.
These envelopes are 2.x storage artifacts: the cutover import, the
read-model API's legacy bodies and the verify and canonicalize CLIs read
them, no 3.x writer produces them, and 2.x never wrote a cut. Both
resolvers now refuse a bundle whose reward manifest has a cut, by name and
before any reconstruction, rather than half-checking it; such a bundle is
valid only as a flat logical bundle, which every 3.x path stores and
serves. A test wraps the gapped cut fixture in each envelope and checks
the named refusal, and that the flat bundle still parses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer (3/5): carry owner and peer settlement Two independent writers, one carry owner (CONTRACT §4, §6). With PRISM_DUAL_WRITER off nothing changes: every statement is 3.0's text. - Carry gate (ledger/carry.rs): non-owner work is carry-free (empty priors), every change fenced by one payout revision bump; work is also carry-free whenever this node's journal no longer claims ownership, so a release fences later jobs at its commit. - Owner guard (carry_owner.rs): pays only while the node identity, PRISM_CARRY_OWNER, this node's journal and a live peer read agree; after a peer release only a claim transfer vetted pays; rollbacks under a running frontend, conflicts and mismatches are carry-free and alert; seeding waits for the own-log latch and a live read; start-up states are quiet during start-up. - Peer settlement (ledger/blocks.rs, divergence.rs): peer blocks are local derived state for this node's reconciler; the finder (coinbase extranonce1 half) decides found counts, divergence records and fanout sponsorship. - Peer fanouts (broadcaster.rs, ledger/fanout.rs): the finder sponsors a zero-fee fanout; the other node takes it over once it is overdue and the finder silent, and finishes any package it holds. - S8 adoption (coordinator/adoption.rs) of pool blocks whose finder died before their rows arrived, from its prepared record. - qbit-prism-server carry-owner status|release|transfer, with the release height read under the lock, the scan bound, the journal tail copy and the pre-write re-read; docs/prism-ledger-ops.md. - Tests: invariant model, gated carry-owner postgres, guard and adoption suites, observability pins. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.1 dual-writer: drop the tests' stand-in for 031's index; bound the release's tip read; a release depth of at least 2 On D2's d04cdc4 (D1 193e7fd, with migration 031): - The carry-owner tests no longer create a stand-in for the (origin_node, share_seq) index: migration 031 creates it on every fixture. The adoption fixture turns D1's D-19 offer wait off; it has no peer. - carry-owner release bounds its tip read under SETTLEMENT_LOCK at 5 s, since the lock holds work building back meanwhile. - transfer waits for a release depth of at least 2 (MIN_RELEASE_DEPTH): carry-paying work built just before a release can still reach the tip after the one the release recorded. - Work reads the gate only through job_prior_balance_sql_in, which also reads the journal; the gate-only selector is private. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.1 dual-writer: say in the ops doc that a release fences work at its commit Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.1 dual-writer: a live peer claim counts even behind a newer synced row; classify the carry owner's checkout sites From Codex's review of cae3115 (PR #767): the guard took only the newest of the peer's live and synced rows, so a peer whose database was rolled back to an older claim, behind the release this node's copy holds, could act on that claim while this node kept paying. The guard is now carry-free on any live claim of the peer's, as transfer already refuses one. The pool-acquire census (scripts/check_prism_pool_acquires.py): - the guard and the carry-owner commands check out through Ledger::acquire; - the journal helpers run SQL on the caller's connection (&mut *connection); - the peer reads through PeerJournal get their own policy, peer-journal; - the remaining carry-owner, fanout, adoption and guard-spawn sites are classified. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.1 dual-writer: a live peer claim behind this node's copy is its own state, peer_journal_behind From the local review of a1d8c7b: a claim in the peer's live journal behind, or against, a newer row of the peer's this node holds (a rolled-back peer database, or a lagging path) stops payments, as Codex asked, but it is not two owners: the advice is to restart the peer so its own-log check recovers its rows, never to release this node. The guard reports it as peer_journal_behind (alerting), and the ops doc says so. The module doc states the rule; its unit test stands alone and covers a same-epoch disagreement. transfer's chain scan checks out through Ledger::acquire as well. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.1 dual-writer: read the peer's rows and latest claim in one snapshot; let transfer recognize earlier pool markers From Codex's review of 7a2ea81 (PR #767): - transfer read the peer's latest rows and its latest claim in two reads, which a primary and a lagging fallback URL could answer differently; a missing claim then left the scan unbounded. PeerJournal::view reads both in one REPEATABLE READ snapshot on one connection, and scan_start refuses a peer that released when no claim of its can be read. - The chain scan recognized pool blocks by the current coinbase tag and pool-fee program only. carry-owner transfer takes the earlier ones (--also-tag, --also-fee-program, repeatable), and the ops doc says when to pass them, and that S8 adoption recognizes only the current ones. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.1 dual-writer: adoption waits for the own-log latch, as the submit loop does An adopted block lands with this node as its origin (D-10), so its rows are this node's own. The Stratum listeners, the refresh and the submit loop wait for the own-log latch (D-8) before writing any; adoption did not. A node restored from a backup could then land an adopted block before own-log recovery had raised its key sequences and pulled back its lost landings, reusing a key the peer holds or meeting its own earlier landing of the same block with other facts. Recovery would then find an own row that differs from the peer's copy, record the own log as diverged and keep the node out of service until an operator decides. The loop now waits for the latch before its first pass, inside adoption_loop, so the gated refusal test pins it: its control runs the loop and sees nothing land in 3 s with the latch unset, then the block adopted from its own record once the latch is set, and adopting it again rewrites nothing. A plain restart with the peer unreachable still sets the latch at once, so a surviving node that restarts keeps adopting. The ops doc says that until the latch is set adoption reports nothing either: a block that leaves the 1,440-block lookback while no loop runs gets no last ALERT. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.1 dual-writer: the adoption fixture fakes a pull that refused nothing D1's 7658fa2 made a cursor's ingested_through the highest peer row a pull refused as a conflict, quarantined for good, NULL while none was. The adoption fixture still set it to the share mark, which now reads as a pull that refused every peer share through it. It now sets only scanned_through, as a clean pull leaves the cursor, like D2's f55e876 for the window-cut fixtures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.1 dual-writer: count the guard's and the broadcaster's peer connections in the sync role's budget The carry-owner guard and the broadcaster's check of a silent finder each read the peer's database as prism_peer_sync, one connection per path, and every carry-owner command does while it runs. The ops doc's budget still counted only the sync and the found-block wait (four per frontend), so a role limited to it could refuse the found-block wait or a sync pass. It now asks for five per frontend, eight with a fallback path, plus one per path for a running command. It also names the new unadoptable case from D1's quarantine: a window holding a peer share, or a prepared record, that this node's sync refused as a conflict. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.1 dual-writer: Codex's two P1s on 4a06a8b, transfer's peer path and adoption's extranonce2 width Transfer reads the peer over PRISM_PEER_DATABASE_URL only. With the first path down, it read the peer through the fallback, which may reach a copy that lags the peer's journal. There an older release would pass every check, and the pre-write re-read could pick the same stale copy, while the peer itself had acquired again: both nodes would then pay carry. Its checks, its one-snapshot view and the re-read now all go through the first path, so a transfer with that path down refuses on peer_answered. The ops doc says to point PRISM_PEER_DATABASE_URL at another path to the peer's own database, never a replica. The gated transfer test runs one with the first path unreachable and a fallback that answers. Adoption reads the issuing node's extranonce2 width. PRISM_STRATUM_EXTRANONCE2_SIZE is not in the pair's fingerprint, so the nodes may differ, and adoption checked a peer record's coinbase suffix with this node's width: it refused a valid record, so S8 could not land the block. The suffix is the issuing node's tag and a zeroed placeholder, and the tag is printable ASCII, so its trailing zeros are exactly the placeholder: adoption takes the width from there (ends_with_suffix, unit-tested over widths 1 to 32). The rebuilt commitment root and the byte-for-byte coinbase still decide. Finder attribution still reads extranonce1 at this node's width; the ops doc says to keep the width the same on both nodes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.1 dual-writer: Codex's two P1s on 476fdd9, the transfer's tip at the acquire and carry-paying offers behind a closed or unfenced gate transfer now checks, right before the acquire and under SETTLEMENT_LOCK, that the scanned tip is still on the active chain. A shallow reorganisation between the scan and the commit could bring in a branch holding a carry-paying block of the peer's that the scan never saw and that is not landed here. The acquire then refuses and asks to be run again. Blocks found on top of the scanned tip are above the release depth, so carry-free, and do not refuse. The read is bounded like release's tip read. The gated transfer test runs a confirmed transfer against a chain whose scanned tip is reorganised away: it is refused, nothing is acquired, and the copied release row stands. An offer no longer takes carry-paying work behind a closed gate for current work. set_carry_paying moves the in-memory gate before its payout revision bump. If the bump fails (a halted cluster, a lock timeout, a lost connection), work built under the old mode stays at the current revision until the guard's retry commits. A block found on it was then offered and landed as current work, not through the #478 capture bound or, with capture off, abandoned. After the gate closed on a competing owner, carry-paying work could then land here while the peer also paid. Ledger::reserve_offer_within, which reads the revision FOR SHARE, now also treats a block whose prior balances are not the carry-free ones as superseded while the dual-writer gate is closed, or while a change of it awaits its bump (a gate closed and reopened before either bump commits), whatever the revision says. Such a block takes the capture path. This needs no pending-fence state and puts no lock on the offer path. Carry-free work while the gate is open stays current: it pays no carried balance. Leased candidates are reserved unconditionally, as in 3.0. A unit test pins the predicate (carry_paying_not_current). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.1 dual-writer: recheck the carry gate at the offer's send, after the token renewal Codex P1 on 0d9d9dd. The reservation decides whether carry-paying work is current, but submit_block comes after the found-block wait (up to PRISM_PEER_INGEST_WAIT_MS) and the token renewal. A gate that moved in that interval let a block reserved as current be sent while the peer might also pay carry. Right after the strictly-live renewal, immediately before the send, the offer now rechecks the gate in memory. A block reserved as current whose work is carry-paying is not sent if the gate is now closed or unfenced, or has changed at all since the reservation. That includes a gate closed and reopened meanwhile, since the peer may have paid the same balances while it was closed; the gate now counts its changes for this check. Its reservation returns to pending through release_unmade_offer, the #522 release for a call never made. It is due again at once rather than after the pending backoff, and the next attempt's reservation takes the #478 capture path, or abandons the block with capture off. The check reads only memory, so the renewal stays the last database step before the send. Leased candidates, reserved without a revision check, are unchanged. The ops doc's offer_reserved row names the new release, and a unit test pins carry_paying_unsendable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.1 dual-writer: classify release_unsent's checkout in the pool-acquire census 454ccca moved release_unsent_offer's transaction into release_unsent, which release_unmade_offer shares, so the census saw an unclassified site and a stale one. It is the same ledger checkout, renamed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.1 dual-writer: a coinbase names its finder only where the tag sits before extranonce1 Codex P1 on 835963e. PRISM_STRATUM_EXTRANONCE2_SIZE is not in the pair's fingerprint. finder_node read extranonce1 at this node's width, so on a peer block issued with another width it read four other bytes, which could fall in this node's half: both nodes then took the block for their own, and both funded conflicting CPFP children of its zero-fee fanout, leaving the losing wallet's coin reserved. finder_node now reads extranonce1 at this node's width only where the coinbase tag sits right before it, as every coinbase this pool builds has it. A block issued with another width therefore names no node, and found_here falls back to its origin: a peer-landed block is never taken for this node's own. This is a check of one placement, not a search over widths, which a short tag or crafted extranonce bytes could fool. An empty tag checks nothing, as before. The ledger records the tag with the width (set_coinbase_layout), and unit tests cover a peer block issued with widths 4 and 12 against a local 8, another tag, and an empty one. The ops doc says to keep the width and a non-empty tag the same on both nodes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.1 dual-writer: adopt across extranonce2 widths, end to end This closes the extranonce2-width class. Only two paths read a peer's coinbase with this node's PRISM_STRATUM_EXTRANONCE2_SIZE: - adoption's suffix check, which reads the issuing width from the record since 476fdd9; - finder attribution, which since 18c18b5 reads extranonce1 only where the tag sits right before it. A block issued with another width names no finder, and its origin decides. Every other width use builds or serves this node's own work. The gated refusal test's control now adopts the block on a second frontend whose width is half the one the work was issued with. The adoption loop waits for the own-log latch, then adopts the block from its own record, and the rebuilt coinbase is byte for byte the block's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.1 dual-writer: a coinbase that names no node is not this node's find Codex P1 on fa9ca53. finder_node names no node for a block issued with another extranonce2 width, and found_here then fell back to the block's origin. An adopted block carries the adopter as its origin, so the adopter took a cross-width adopted block for its own find. It sponsored the zero-fee fanout at once, bypassing the takeover's overdue and silence checks, while a recovered finder, whose own layout names it, sponsored it too: conflicting CPFP children, and a wallet coin left reserved. Every coinbase a dual-writer node builds names it: the tag sits right before its own extranonce1. A coinbase that names no node was therefore issued elsewhere, and found_here no longer falls back to the origin for it. Such a block is sponsored here only by takeover, adopted or not. Without a recorded layout or a coinbase, the origin still decides. A node that changes its own tag or width while its blocks still await fanouts stops taking them for its own finds, which the ops doc already rules out. The gated refusal test now checks that, after the narrow-width frontend adopts the block, only the issuing layout names its finder. Unit tests cover an adopted cross-width block and an unparseable coinbase with this node's origin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.1 dual-writer: refuse to start without a coinbase tag of at least 4 bytes Codex P1 on a7acc04. finder_node reads extranonce1 only where the coinbase tag sits right before it, but PRISM_COINBASE_TAG may be empty, and an empty tag verifies nothing. With unequal extranonce2 widths, both nodes could then take a block for their own find and fund conflicting CPFP children. A tag of a byte or two could also match by chance. A dual writer now refuses to start unless PRISM_COINBASE_TAG has at least 4 bytes (the default /PRISM/ has 7). Single writers keep their choice, an empty tag included. A unit test covers the check, the README and the ops doc say so, and D1's dual_writer and config_cli suites still pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.1 dual-writer: a peer database with no work is not a silent finder until it stays so for the takeover delay Codex P1 on 1aba741. When the finder's database answered with no work of its node at all, finder_silent took that as silence on the first read. That happens just after the frontend starts, or once its expired work is pruned. A live finder could still sponsor its overdue fanout as its own while this node took it over: conflicting CPFP children. Such an answer is now unknown: the liveness state notes since when the checks have found no work, any check that finds work clears that, and the finder counts as silent only once there has been none for SPONSOR_TAKEOVER_AFTER. The gated takeover test runs a finder whose database holds no work: it is not taken over at once, and it is taken over once the checks have found none for the delay (assume_no_work_since). The ops doc says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.1 dual-writer: a finder keeps its own block across a coinbase layout change Codex P1 on 7c8c72b. finder_node reads extranonce1 at this node's current PRISM_STRATUM_EXTRANONCE2_SIZE and coinbase tag. A block this node found before either changed therefore named no node, and found_here took it for the peer's. The peer read no finder either, so neither node sponsored the zero-fee fanout as its finder. A coinbase that names no node now counts as this node's find when the block carries this node's origin and this node holds the block's outbox row. The outbox is local and never copied, and an adopted block has none, so a cross-width adoption is still sponsored here only by takeover (a7acc04). Without a recorded layout or the coinbase, the origin still decides. The fanout sponsor, reconcile and divergence reads each select the outbox flag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.1 dual-writer: adoption loads a parent's prepared records one at a time Codex P2 on 6264cd4. compact_prepared_on_parent fetched every prepared record on a block's parent, each joined to its template and balances, in one fetch_all. A parent that stays current for hours, as in a chain stall on a busy mempool, holds many such records from both writers, so a single adoption could hold all of them in memory at once on the survivor. compact_prepared_on_parent now returns only the keys, newest first, and compact_prepared_held loads and decodes one record at a time, expired or not. A record that is gone, does not decode or has lost its blobs is reported among the block's refusal reasons. The pool-acquire census classifies the new checkout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.1 dual-writer: no takeover on a short outage without the finder's work Codex P1 on 4d8897c. With no prepared record of the finder's held here (none synced yet, or its copies expired after the 24-hour retention), the fence found every matured fanout overdue at once. The broadcaster then took it over after its finder's database failed three checks over one minute, an outage that only means silence when the stale copy here already shows the takeover delay. A live finder behind an asymmetric database outage could have its fanout funded twice. The fence now reports such a fanout as Unwitnessed rather than Overdue, and the broadcaster takes it over on an outage only once the outage itself has lasted SPONSOR_TAKEOVER_AFTER. A database that answers is read as before. The cached verdict is re-evaluated against each caller's outage threshold, so one fanout's verdict never shortens another's wait. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.1 dual-writer: an empty peer path keeps the finder's work age unknown Codex P1 on a7ce81e. PeerJournal::work_age combined the peer's URLs by the youngest age and let any age override a path that answered with no work. When the current path holds none (a frontend just restarted) and a lagging fallback still holds an old row, the stale age won, and the broadcaster could take over a live finder's fanout once that row was 30 minutes old. youngest_work_age now lets work younger than the takeover delay on any path win, since it shows the finder alive. Otherwise a path with no work makes the age unknown, which waits on the broadcaster's 30-minute no-work timer. Only when every path shows stale work does the youngest decide. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.1 dual-writer: refuse a coinbase layout mismatch between the pair Codex P1 on 9c0b64f. Under two PRISM_STRATUM_EXTRANONCE2_SIZE widths, a miner can place the coinbase tag and a value from the other node's range inside its own extranonce2 at the narrower width's offset. Both nodes then read themselves as the block's finder and fund conflicting CPFP children. The tag-adjacency check only made accidental collisions rare, and the pair's fingerprint binds neither the width nor the tag. Each node now reads the peer's tag and width from the coinbase suffix of the peer's newest prepared record it has synced, so no schema or grant changes. While they differ from this node's: - the carry owner guard builds carry-free work on either node under a new alerting reason, layout_mismatch; - fanout_sponsor sponsors nothing as finder or by takeover, and only finishes a CPFP package it already holds; - adoption refuses each block, which the pass reports with its ALERT. carry-owner status prints the mismatch. The metric label, scrape contract, generated metrics doc, census and ops docs follow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.1 dual-writer: only the peer's primary confirms an owner or a seed Codex P1 on 57972c7. The guard's live read takes the first peer URL that answers. An owner restored from a backup older than its release, started while the peer's primary was out of reach, could read a lagging fallback that still showed the peer as non-owner and this node's old claim. A fresh guard then confirmed the peer and opened the carry gate while the peer, which had acquired, was paying too. PeerRead::Answered now says whether PRISM_PEER_DATABASE_URL answered. Only that answer confirms the peer or lets this node seed its journal, as carry-owner transfer already reads only the primary. A fallback answer still stops payments: a claim, a newer row of this node's, or an unseeded peer it shows counts. Once confirmed, an owner keeps paying through a fallback answer, as through an unreachable peer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.1 dual-writer: read the peer's layout from its primary when none is held Codex P1 on 02474ed. layout_mismatch reads the peer's coinbase layout from its newest prepared record synced here, and took the layout to match when none was held. While the jobs stream was refused or lagging, a pair with two layouts could therefore pay carry, and sponsor fanouts as their finder. With none of the peer's records held, the carry owner guard now reads the peer's newest coinbase suffix live over PRISM_PEER_DATABASE_URL only (PeerJournal::primary_suffix), and the ledger keeps it for the fanout and adoption checks. The layout is unknown, and taken to match, only while the primary shows no work or has never answered, so a peer whose records expired and whose database is gone does not hold the survivor carry-free. Held records decide and clear the live value. carry-owner status reads it the same way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.1 dual-writer: a fallback's stale work is an outage, not silence Codex P1 on d42ea51. With PRISM_PEER_DATABASE_URL out of reach, a lagging fallback that answered with work 30 minutes old or more gave work_age that stale age, and finder_silent took the finder for silent at once. That skipped the outage timing, including the 30 minutes an Unwitnessed fanout needs. A live finder whose primary this node cannot reach could have its fanout funded twice. PeerJournal::work_age now knows whether the primary answered. Without it, only work younger than the takeover delay counts, as the finder alive; a fallback's stale or empty answer is no answer, so the broadcaster times it as an outage against the fanout's threshold. With the primary answering, the rules are as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.1 dual-writer: a transfer's scan starts a maturity below the acquire Codex P1 on 1b87916. carry-owner transfer's scan_start let --from-height equal the tip the peer's last acquire recorded. If the chain reorganized after that acquire to a fork below its tip, the peer, already owner, could find carry-paying blocks at heights under it. The scan would miss them, the transfer would pass without their landing rows, and this node could pay the same carry again. The scan must now start SCAN_REORG_MARGIN below the recorded tip: the coinbase maturity, the depth PRISM already treats as final, where a mature pool block disconnected is a fatal state. The default --from-height 0 is unaffected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.1 dual-writer: the offer reads the journal before it sends carry work Codex P1 on 11765f5. The offer rechecked only this frontend's in-memory carry gate before submit_block. A carry-owner release, committed by the operator's own process, reaches the gate only at the guard's next check, so a carry-paying block reserved before the release could still be sent after it. The new owner's transfer scan might already have read that tip, and the block's carry could be paid twice. Right before the send, alongside the gate, a carry-paying block reserved as current work now also reads this node's latest journal row. If the row no longer claims ownership, nothing is sent, and the reservation returns to pending as for a moved gate. Carry-free work is unaffected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.1 dual-writer: keep a failed journal read from losing a found block; refuse equal-epoch claims Two Codex P1s on b4dbb57. The offer's journal read before submit_block returned its error with ?, after the reservation had committed. The failed attempt then kept the block offer_reserved, which recovery treats as a call that may have run, so a valid block would never be offered again. A read that fails is now taken as withdrawn: nothing is sent, and the reservation returns to pending as for a moved gate. carry-owner transfer took the newer of the peer's live and synced claims by epoch, and silently chose one when they differed at the same epoch, a sign that the peer's journal was rewritten. A different recorded tip could then let a later --from-height pass scan_start. Such a conflict now fails scan_start. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.1 dual-writer: a failed layout read closes the carry gate Codex P1 on e776ee9. The guard's check returned with ? when the peer layout read failed, after the live peer read and before Guard::decide. The run loop keeps the gate as it is on an error, so a claim the peer read had just shown went undecided, and an open gate stayed open while the peer might also be paying. A layout read that fails now counts as a layout mismatch, so the check decides carry-free and alerts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer: a two-node pair simulation, its payout-invariant checker and CI lanes
crates/qbit-prism-dual-sim runs a whole PRISM pair on one host: a PostgreSQL 16
cluster per node (the 3.0 pair as a primary and its streaming standby, the 3.1
pair as two primaries), a regtest qbitd per node plus one for the network
(ramped once to 1e7fffc0 and cached), the real qbit-prism-server frontends, a
balancer stand-in that routes Stratum sessions by each node's health (A
preferred, B backup, inter 2 s fall 3 rise 2, sessions closed on mark-down,
paced failback; /readyz with its token for the dual-writer pair), Stratum load
through it from qbit-prism-load's client, and a block finder per node.
The fault injector kills or freezes a frontend or a PostgreSQL, drops a node's
network or cuts the link between the databases (a reset or a blackhole that
delivers held bytes on heal), and takes and restores base backups.
The checker holds CONTRACT.md section 4 to the chain's census of pool blocks:
per-account debt never rises except a recorded #478 race, the non-owner's blocks
are carry-free, every pool block is landed and identical on both nodes and its
audit verifies with qbit-prism-audit-verify against the chain's coinbase, CTV
fanouts are byte-identical, every acknowledged share is present except a
documented tail, nothing is credited twice, and windows are unchanged and
reproducible on both nodes. A negative control proves it fails when it must.
S10 (the 3.0 pair through a kill -9 of A's frontend) and the control pass on
today's code; S1 to S5 are written and wait for the 3.1 stack. CI: a required
fast lane in ci.yml (prism-dual-writer) and the full matrix nightly, on
dispatch and on the run-dual-writer-e2e label (dual-writer-nightly), each with
its gate-manifest proof, artifacts and a matrix summary; both lists are
registered in test/e2e-scenarios.toml.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: the dual-writer bootstrap, the restore, rebuild, cutover and transfer scenarios, and contract v1.9's checks
- Bootstrap a dual-writer pair as the cutover does (D-9, D-4): migrate,
node-identity set, D1's exact peer-role grants and role settings, then start
B before A. The balancer checks the token-protected /readyz (D-7) in dual
mode and can switch probes mid-run.
- S6: plain restart with the peer unreachable serves, a restore onto a new
timeline stays unready, and the node holds every own row the peer holds at
its first ready reading (D-8, D-17). Restores recover the way pgBackRest
does: archive recovery to the backup's consistent point, promoted onto a
new timeline.
- S7: the replaced disk is a promoted physical copy of the peer,
re-personalised (D-16); only the dead node's measured tail is excused.
- S8: an RPC gate holds or withholds the dying node's submitblock; the
survivor's adoption, one landing per node and the D-19 wait on and off.
- S9: a live 3.0 to 3.1 cutover with ledger digests before and after on both
nodes, every pre-cutover row node 0, D-12's refusal of a single-writer start,
and the owner's first priors equal to the chain's sums.
- S11: D3's release and transfer drill, refused by the chain scan while an own
block is unknown and accepted once it lands.
- Checker: D-1 and D-10 immutable landing facts compared across nodes and with
the chain's coinbase txid, D-13 cut-aware window recompute, ownership epochs
by height, the D-14 late-row negative control, a reconciliation row whose
block has landed treated as bookkeeping, and block finding on settled,
freshly delivered work.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: D-2 local-state and D-5 landing-order checks
The checker holds each dual-writer node's never-copied state to its own rows (no candidate reserved by the peer's frontend, no divergence record for a peer-origin block, its own identity), and S1 and S5 sample every peer block as it arrives on a node, requiring its whole window there first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: provenance-aware D-2 check, an atomic D-5 sampler, S4's D-8 check
The D-2 check now looks only at rows written since the pair began writing
as two primaries (Sim::dual_since: the dual bootstrap or the cutover), so
the history a 3.0 cutover leaves on both databases no longer fails S9. A
copied candidate is one the peer's frontend claimed or reserved. A copied
divergence row is one with an offer decision for a block this node never
had as a candidate: only the offering frontend records one. The check no
longer flags divergence rows for peer blocks, which each node's reconciler
writes when it confirms them. After S7's rebuild the same check proves
D-16's re-personalise reset the local tables.
The D-5 sampler counts each peer block's window in the statement that
finds the block, so detection and count share one snapshot. It keeps
polling a block until its window record arrives, records failed polls
instead of swallowing them, and stops when dropped. S1 and S5 judge each
direction on its own and need at least one block counted complete as it
arrived. The lag sampler also counts failed polls and stops when dropped.
S4 adds D-8's "a later link loss never withdraws a serving node": the
balancer marks neither node down from the cut to the catch-up.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: D-2 copies by identical rows, D-10 torn landings, structured balancer records
D-2 now finds a copy however it was made. A candidate or an offer decision
that is the same row on both nodes (the same insert timestamp) is a copy:
each node finds its own blocks, and an adopting survivor's rows are made
on their own. A candidate from before a cutover that was reserved after it
means the drain left it unfinished, and the cutover now refuses to start
unless no candidate can still be offered. The candidate count in the
report covers every row, not only a node's own.
The D-5 sampler takes its baseline before the scenario goes on, so a block
that lands during early failed polls is sampled, not excluded. A block
first seen after a failed poll cannot show the order. A block seen without
its audit bundle or snapshot breaks D-10 (a peer block arrives with all
its child rows), so it fails instead of passing as unresolved. One D-13
eligibility predicate now serves the window checks and the sampler. Both
samplers share one poller that stops when dropped.
The balancer records each mark-up and mark-down and every failed check,
with whether its node was serving. S4's D-8 check uses them: both nodes
stay up from the cut until a full mark-down's worth of checks after the
catch-up, and failed checks while serving are counted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: a frozen peer never stalls the survivor (S2's freeze and a mid-pull variant)
A review of D1's engine found that the sync barrier could hold a lock on
the peer's database across a client round trip. Every job and landing
takes that barrier (sync_seq's default), so a puller frozen while holding
it would stall the survivor's writes. Two checks now guard it.
S2's freeze variant keeps A frozen for at least 60 s while B finds a block
every few seconds. Each block must land, and B must record new jobs after
each one before A thaws.
A new S2 variant, s02_a_frozen_again_and_again_mid_pull_never_stalls_b,
freezes A's frontend 20 times for 1 to 4 s with both nodes taking miners.
Each freeze is timed, where it can be seen within 3 s, to a moment A's
puller has a query, a transaction or an advisory lock open on B's
database. During each freeze a new tip arrives and B must record jobs on
it within 15 s; every fifth freeze B also finds a block, which must land.
The report counts the freezes that caught A mid-pull, and freezes.json
keeps the schedule's seed and every round. It is registered for the
nightly lane as not run yet: the balancer's checks need D4's readiness.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: freeze checks prove what they claim; S4's D-8 judges nodes serving at the cut
The long freeze now requires B to build work on each of its blocks (a
job whose parent is the block), in one grouped query, and A's frontend to
still be the stopped process it froze at the end of the hold. The old
"held at least 60 s" condition could not fail.
The mid-pull variant times B's new work from the mint's return, not from
the freeze. Landing rounds solve on work B handed its finder before the
freeze, so the freeze stays short. A superseded solve is reported, not
judged. A is thawed before any error in a round is raised, and the pull
watch polls every 20 ms rather than every 2 ms. find_block is now a
prepare step and one solve-and-land attempt, which those rounds reuse.
S4's D-8 check judges only the nodes serving at the cut. Its horizon
allows a check timeout longer than the interval:
fall x max(interval, timeout) + interval. The balancer report gains
up_at. process::stopped reads a pid's state from /proc. The manifest
credits the 60 s hold to the S2 entry that runs it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: checks for the three P1s of D1's engine review
The coordinator's review of D1's peer sync engine (cca56f14) found three
P1 failure modes. Each now has a nightly scenario, so the fixes stay fixed.
- S6, unreadable peer (P1 1). B's PostgreSQL accepts the peer role, but
every read of its share ledger hangs (an ACCESS EXCLUSIVE lock held on
B) or fails (SELECT revoked); a probe as the peer role shows it. A
restarts with its database untouched, so its D-17 evidence matches, and
must report ready and take miners within 120 s (D-8). B is then made
readable again and the pair catches up.
- S2, puller death (P1 2), at 250 offered shares/s. For 20 s B is sampled
every 10 ms and must never show A's peer backend idle in transaction:
no pull holds a transaction across a round trip. Then A's database link
is blackholed so B never sees the close, as when A's host dies, and A's
frontend is killed while its puller is busy on B. A's backends on B
must hold no snapshot or transaction beyond 10 s, and B's share answer
latency must stay flat across the next minute (#738).
- S4, transient apply failure (P1 3). A holds EXCLUSIVE on its audit
bundles, so landing inserts fail at A's 5 s lock_timeout while reads go
on, for 12 s after B lands a block. Once the lock clears, the block
must land and confirm on A, with no sync conflict recorded for it.
Each is registered for the nightly lane as not run yet until D1's fixes
and D4's readiness land.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: a discarding link for the puller death, evidence for the transient apply
S2's puller death no longer blackholes A's database link. A blackholed
relay stops reading, so a B backend still sending a result would block on
a full window and hold its snapshot even with single-statement reads. The
relay gains a Discard state: every byte is read and dropped and no close
is passed on, which is how a dead client looks to its server. Its flat
appends check now starts its early window 5 s after A is marked down, out
of the reconnect burst, and needs A to have no session left.
S4's transient apply now requires at least one of A's applies to be seen
waiting on the locked table (distinct backend and statement start in
pg_locks), so the check cannot pass without exercising the retry. S4's
D-8 check judges both nodes from the moment routing settled; the balancer
now records each mark before it publishes the state, so a node seen up
always has its mark-up on record.
S6's unreadable peer always makes B readable again before it raises an
error. The mid-pull rounds report landings apart from new tips. A landing
round is Landed or LandingFailed, never a placeholder. Load::accepted_on
counts in place instead of copying the share log. The manifest and README
say what each round does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: Discard resets spliced connections on heal; tighter evidence and error paths
The relay's Discard state now drains in every case it could stall. Bytes
read as the link turns to Discard are dropped instead of waiting for Open.
A write blocked on a stalled or dead client is abandoned and the sender
drained. A failed write while discarding keeps the drain going. Closes
are held, and a connection that lived through Discard is reset on both
sides when the link leaves it, as a dead host's restarted kernel answers
the old connection: its stream has a hole, and splicing it would
desynchronise the protocol. Two relay tests cover the stalled write and
the reset on heal.
S4's transient apply counts only statements of A's frontend (its role, its
database) waiting on the locked table, records their text, and releases
the lock before any error is raised. S4's D-8 check needs both nodes up
from routing settling to the cut, and judges only mark-downs after the
cut. S6 restores B before raising, and keeps the restart's error as the
root cause. S2's puller death looks for A's mark-down from the cut and
drops a sessions condition that could not fail. The mid-pull landing
time is named for what it measures, from the solve request to
confirmation. One ShareRecord::accepted_on predicate serves the scenarios
and the load's in-place count.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: cancel-safe relay writes, apply evidence limited to inserts, a routing check for the puller death
The relay forwards each chunk a piece at a time with the cancel-safe
write, re-checking the link between pieces. A blackhole now holds the
rest of a chunk it was writing instead of finishing it, Discard drops it,
and a reset ends it. One wait_state helper replaces four copies of the
same watch loop. The Discard docs say what the code does: every
connection that lived through it is reset when the link leaves it, as the
dead host's restarted kernel would. The stalled-write test returns early
on a host whose buffers can hold the whole flood.
S4's transient apply counts only INSERTs into the locked table. A finds
no block of its own meanwhile, so these are B's block's apply, while A's
own backfills of the table are UPDATEs. It keeps the first error when
the unlock fails too. S6 records B's restore on the timeline even when
the restart failed. S2's puller death checks that no session is routed
to A after its mark-down. count_accepted_on serves the scenarios and the
load.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: routing decisions dated by the balancer; S4's own-block guard; a stalled-write test that cannot pass blind
The balancer records each routed session when it chooses the node, so
S2's puller death counts sessions routed to A after A's recorded
mark-down from one source. A 100 ms grace covers a choice that read the
state the mark was about to change. It no longer compares a polled
baseline with a later count.
S4's transient apply now requires that A created no candidate of its own
while its audit bundles were locked. Otherwise its own landing's insert
could pass for B's block's apply. On any error after the lock it closes
A's pool, which ends a connection whose rollback failed and its lock.
The stalled-write test passes early only when an open link has delivered
every byte of the flood (a host whose buffers hold all of it), never when
bytes went missing. The Discard docs say exactly which connections it
resets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: exact routing dates; S4's own-block guard is an expectation over the whole dual run
The balancer dates each routed session before it reads the nodes' state,
and records a mark-down after publishing it, so every session routed to
a node is dated before that node's mark-down. A mark-up is still
recorded before it is published, so a node seen up always has its mark.
S2's puller death then counts sessions routed to A strictly after its
mark-down and before its next mark-up, with no grace. The routing log
stays out of the written report.
S4's transient apply now judges the "A has no candidate of its own"
condition as part of its expectation, not as an error that skips the
landing checks. It counts every candidate A created since dual mode
began, so a candidate made just before the lock is counted too. The lock
timestamp it no longer needs is gone, along with its unguarded read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: what the first runs on the stack showed: catch-up under load, partial reports, late sessions
The first runs against D1 and D2 surfaced three harness gaps.
- wait_synced required both databases to hold equal counts, unchanged a
second later. Under load the counts never hold still, so every
mid-scenario catch-up (S3, S4, S2's puller death, S6's unreadable peer)
would time out. It now waits until each node holds every share and
block its peer had originated when it was called. settle, which pauses
the load first, keeps the strict equality as wait_converged.
- A scenario that failed part-way lost every expectation it had recorded:
the driver only had the body a finished scenario returned. The driver
now owns the body and lends it to each scenario, and a failed report
carries its expectations, gaps and blocks.
- A session the balancer chose just before its node's mark-down could
register after the mark-down closed every session and stay on the dead
node. A session now closes itself if its node is down once it has
registered.
S2's puller death drops its routing condition, which held by
construction, together with the routing log it needed. S4's transient
apply reads dual_since before it takes its lock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: a robust catch-up wait, failed reports keep the checker's verdicts, late sessions never reach a dead node
wait_synced polls once a second and retries a failed read until its
bound. A timeout now names each node's sync conflicts, since a row the
peer refuses never arrives. Its pools close on every path. Its doc says
it does not prove that a restored node pulled its own rows back (S6 and
S7 check that).
A failed report keeps the invariant report when the checker ran before
the failure, so the per-check verdicts in its expectations can be traced.
A session the balancer chose just before its node's mark-down, and that
registered after it, is reset before any byte reaches the node and is
not counted as routed. S2's puller death again checks that every miner
leaves dead A: no session stays open to it at the end of the watch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: a bounded catch-up wait; the puller death judges only what P1 2 asks
wait_synced now retries its first reads too (each node's own rows and its
clock) and caps every read with the time left before its bound. A
stalled database therefore cannot carry it past the bound. The conflicts
a timeout names are those recorded during the wait, read with their own
short cap.
S2's puller death drops "every miner leaves dead A". The balancer's map
of open sessions empties on a mark-down whatever happens to the miners,
so the condition could not fail. P1 2 asks about B's backends and B's
appends; miners leaving a dead node is the other S2 variants' check.
A session reset because its node was marked down as it connected is now
counted as refused and logged on the timeline. The driver no longer
copies the invariant report to keep it for a failed report.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: the catch-up wait cannot hang on a stalled database; S2 checks B took every miner
wait_synced caps connecting by its deadline, and closing each pool to 5 s.
A read cancelled mid-query leaves its connection pinging a stalled
database, which an uncapped close would wait on. On a timeout it reports
each node's sync conflicts in total and those seen during the wait (a
row refused earlier stays missing just the same). A count it could not
read says so, instead of printing an error as if it were one.
S2's puller death checks that B took every miner: each session has
shares accepted on B's jobs 45 to 60 s after the death, the late window
whose latency it judges.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: precise catch-up diagnostics; S2 names every miner session
When the wait never read a node's clock, its count of conflicts seen
during the wait reads "unread" rather than falling back to the harness's
clock and reporting 0. A connect that outlives the bound says the bound
was spent connecting to both databases, rather than blaming whichever
node was last. S2's every-miner check requires each session index of the
load to appear among B's late accepted shares, not merely as many
sessions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: S2 names the miner sessions B did not take; the catch-up wait says when it read a clock late
S2's puller death lists the load sessions with no share accepted on B in
its late window, rather than a count that a finder session could round
up. When the catch-up wait first read a node's clock on a later pass, its
timeout says how far into the wait that was, so a count of conflicts
"seen during the wait" is never shorter than it claims.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: freeze A when its puller holds a lock on B, the moment a barrier bug shows
Two runs of the mid-pull scenario on the stack caught D1's sync barrier
held in 2 of 20 freezes, then in none: freezing at any busy moment makes
catching it a matter of timing. Both S2 freeze checks now wait first, for
two thirds of their wait, for a moment A's puller holds an advisory lock
on B's database. Only then do they fall back to any moment it is busy
there. Each freeze records which moment it got (lock held, busy, unseen),
and the mid-pull summary counts the freezes made at a held lock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: each S2 check aims at its own moment; a dead puller's leaked barrier is caught
The lock-first timing suited the mid-pull freezes, not the puller death.
P1 2 is about a puller dying with a transaction open, and a lock-held
moment usually has none. Now:
- S2's puller death is timed first to a moment A's puller has a
transaction open on B, then to any busy moment.
- Its watch after the death also counts advisory locks. A dead puller's
backend that keeps D1's session-level sync barrier would stall every
writer on B, and that now fails the check instead of passing it.
- The mid-pull new-tip rounds aim at D1's sync barrier itself (its key's
two halves in pg_locks), not at any advisory lock.
- Landing rounds aim briefly at a busy moment, so the finder work handed
out just before the freeze stays current.
- The pull state counts backends with a transaction open and the
barrier separately. The doc comments, README and manifest say which
moment each check aims at.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: link latency makes the barrier and open-transaction moments catchable
D1's puller holds its sync barrier, and an explicit transaction, across
client round trips. On loopback those last a few milliseconds, so a
20 ms poll caught them only by luck: 2 freezes in 20 in one run, none in
the next. The relay can now add one-way latency to every chunk on a link
(Relay::set_latency, reported in its stats). S2's freeze and puller-death
checks put 40 ms each way on A's link to B, a path between two sites.
That holds a barrier or a transaction spanning round trips open long
enough to catch, while a single-statement pull is not stretched by it.
The puller death now aims at an explicit transaction idle between its
statements (state 'idle in transaction'). Before, it aimed at any backend
with xact_start set, which includes every running statement. A wait whose
first aim is a busy moment no longer spends two thirds of its time
looking for nothing. The barrier key is D1's own constant, so it cannot
drift. A death that fell back to a busy moment says the puller was busy
(a query, a transaction or a lock), not only that a query ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: link latency per burst, never on discarded bytes, never cut short
The relay's latency now models a path rather than a throttle.
- A chunk the pump had to wait for starts a burst and waits the latency.
So does a connection's first chunk. A chunk already waiting to be read
is the rest of a burst and follows at once (a non-blocking read tells
the two apart). A round trip costs twice the latency, and bulk transfer
is no longer capped at one chunk per latency.
- Bytes being discarded never wait, so a dead client's server drains at
full speed and S2's puller death cannot see a lingering backend that
the relay itself held up.
- The latency sleep runs to its deadline while the link stays open. A
watch notification that does not change the state no longer cuts it
short.
- Relay stats report the highest latency the link carried, not the value
left after a scenario lifted it.
The latency test now also echoes 4 MiB through a 150 ms link, which must
finish within 20 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: the relay's pumps are delay lines
Per-burst latency was a heuristic, and it was wrong in both directions. A
reply the server streamed in flushes paid the latency on every flush:
that stretched D1's single statements, which could hold the sync barrier
longer and fail a check after D1's fix. Bytes queued behind a delayed
chunk, or behind a blackhole, went out with no latency at all.
Each pump is now a delay line. It reads continuously, stamps each chunk
when it is read, and writes it once the link's latency has passed since
then, so every byte waits the same latency and a stream keeps its rate. A
blackhole stops both reading and writing, and held chunks keep their
stamps. Discard drops the line. The line holds at most 1 MiB, so a slow
receiver pushes back as over TCP. Writes stay partial and cancel-safe. A
new test streams 100 pieces of 8 KiB 5 ms apart through a 150 ms link: it
must arrive one latency after it starts, at the sender's pace, not at
150 ms a piece.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: the delay line carries the close, never crosses a cut, and wakes on a new latency
- The sender's close is a line entry with its own stamp. It is passed on
only when due, and only while the link is open: it waits the latency like
the data, and no FIN crosses a blackhole.
- A change of state or of latency is acted on before anything is read or
written (a biased select, with both changes first). No chunk slips
across a link that has just been cut. Latency travels on a watch
channel, so lowering or lifting it re-dates what the line holds at once.
- A write that fails just as the link turns to Discard drains on, as a
dead client's server must, instead of ending the pump.
- Without latency the line holds one chunk, as a plain relay would, so a
stopped receiver pushes back at once. With latency it holds up to 1 MiB
and 4096 chunks.
- The streaming test checks that the first byte waits the latency, not
only that the whole stream keeps its pace.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: a reset crosses the relay as a reset, and the delay line keeps its dates
An open link now passes each side's close on as it came. A FIN stays a
FIN. A reset, or any socket error, becomes a reset that ends the
connection both ways. Before, a sender's reset reached the far side as an
orderly close.
The delay line, from the review of ea2559ec:
- Without latency it holds one read, as a plain relay does, so a stopped
receiver pushes back at once.
- Reads less than 1 ms apart share an entry, so a stream of small reads
is not capped by the entry limit.
- A new latency re-dates what the line holds, but not a chunk already
partly written, so a message is never split across the change.
- A write that fails just as the link turns to Discard marks the
connection, so it is reset even if the link opens again before the pump
looks.
- Setting an unchanged latency wakes no pump.
New tests: a close waits the latency, a reset crosses as a reset (this
one fails without the fix), a close held in the line never crosses a
blackhole, and lifting the latency releases what waited.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: S2 measures how fast A withdraws on its own when its PostgreSQL dies
With its PostgreSQL killed, A's frontend is still up and answering, so the
balancer's fall count starts only once A stops answering ready. S2 used to
wait the 12 s mark-down bound from the kill. That bound covers a node whose
checks fail at once, not one that has to withdraw itself, so the scenario
ended with an error and checked nothing else.
For this variant S2 now waits up to 35 s, past D4's 15 s admission
staleness backstop and the balancer's fall, and then the rest of the
scenario runs. A new expectation names the owner: A's first failed
readiness check must come within 10 s of the kill (D4's writer probe gives
up after 2 s and a writer unanswered for 4 s withdraws), and the
mark-down within 12 s of that.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: a write a side refuses resets the whole connection
This is the local review's finding on 882e3eb6. When the relay's write
toward a side meets that side's reset first, the write uses up the
error. The pump that reads the same socket then sees only an end of file,
and passes on a FIN. If that pump's line was full, it sees nothing at
all. Nobody reads the other socket after the writing pump ends, so the
other side's writes block for good and the connection never ends.
A write refused outside Discard now ends the connection both ways with a
reset at once, whichever direction meets the reset first. The new test
covers the stuck case: the server reads nothing, so the line toward it is
full, and the client resets while the relay is writing to it. The server
must be reset too. Without the fix its writes block until the test's
10 s timeout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: time A's own withdrawal from its readiness; S8's lost block may stay in reconciliation
S2 with A's PostgreSQL killed, from the local review of 6f33ac8d.
A sampler now checks A's /readyz every 200 ms, from just before the kill,
exactly as the balancer's check does: same path, token and 1 s timeout,
through a new Balancer::node_check handle. The expectation uses the start
of A's final not-ready stretch. Before, it used A's first failed balancer
check, which carries up to a 2 s check interval and can come from an
earlier failure that a pass later reset.
The bound is 12 s, to leave headroom over D4's own cadence. D4's
unanswered-writer rule withdraws 6 to 10 s after the kill: the probe
gives up after 2 s, waits out a second's reuse, and is judged at a later
probe. Its stale-decision backstop comes 14 to 15 s after the kill.
The mark-down wait is 60 s. A not-ready publication arriving late can open
a fresh grace and readmit A, so the backstop can take two rounds. The
expectation is recorded whether or not the wait succeeds, so a withdrawal
too slow even for that is still measured. The balancer's follow is timed
from the final not-ready stretch.
S8, lost variant. The block's submitblock never reached the node, and
nobody recorded its offer's outcome. PRISM keeps such a candidate for
reconciliation and never offers it again, as the scenario allows. The
checker's candidates-settled check failed the run on exactly that row. A
scenario can now name candidates that may stay in reconciliation with no
landing (CheckOptions::unresolved_candidates). S8 names its lost block
only after confirming the block never reached the chain. The check lists
the row instead of failing on it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: S8 waits out the dead frontend's candidate lease; its failures name the claim and B's prepared records
S8's accepted variants first ran on the stack at D3 78ca1a30. A's
restarted frontend had not landed its own block 60 s after its return:
the candidate was still offer_reserved. The dead frontend's claim holds
for PRISM's 120 s candidate lease from its last renewal. Only after that
does the restarted frontend recover the reservation as unknown, find the
block on the chain and land it. The lost variant's settle also took until
the lease ran out. The wait after A returns is now 180 s
(RECOVERED_LANDING_BOUND), and the timeline records how long the landing
took.
Diagnostics:
- A block not confirmed on a node now also shows its candidate's claim
holder, the seconds left on the claim, and the seconds to the next
attempt. A parked row's next attempt is 'infinity', and a query error is
reported as such, not as "no candidate row".
- When B does not adopt, the expectation says how many of A's prepared
records on the block's parent each node held when A died (D-11).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: S8's lost block keeps the prepared row its offer wrote
The rerun on the stack showed that the lost block is not unlanded. Its
offer wrote its qbit_pool_blocks row in the prepared state, and that row
never confirms. The excuse from 9d86fe03 required no row at all, so it
never applied. The excuse now covers a named candidate in reconciliation
whose block never confirmed, whether it has no row or the prepared one. A
confirmed block is still handled as bookkeeping awaiting its next attempt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: S9 waits for B's journal seed before it checks D-12
D-12 refuses a single-writer start on a database whose carry-owner journal
holds rows. Since D3 78ca1a30, a node seeds its journal only once its own
log is caught up (D-8) and its peer's journal has answered that it holds
none of the node's rows. Before, it seeded on its first start.
S9 probed B's database about a second into B's dual start, before B could
seed. The journal was empty, the single writer ran, and the check failed
without testing D-12. S9 now waits up to 60 s for B's own journal row, as
an expectation of its own (D-4), and only then makes the single-writer
attempt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: S2 samples A's readiness until the heal; A must not be readmitted while its database is dead
This is the local review's finding on 9d86fe03. The sampler stopped once
the balancer marked A down, so a later readmission went unseen.
In the 09:06Z run on the stack, A's PostgreSQL stayed dead until the heal
at 49.1 s. Meanwhile the balancer marked A down at 37.4 s, marked it up
again at 40.4 s, and failed 29 miner sessions back to A. Their shares
were rejected as backend database unavailable until the heal.
The sampler now runs from just before the kill until just before the
heal. The expectation requires three things:
- A's final not-ready stretch begins within 12 s of the kill, and so lasts
until the heal.
- The balancer marks A down within 12 s of the stretch the mark-down came
in. Before, it was timed from A's first not-ready answer, which blamed
the balancer for a flap.
- The balancer never marks A up again before the heal.
The detail lists A's marks since the kill. If the balancer never follows,
the expectation is still recorded at once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: S8 buries A's accepted block 60 deep before it waits for B's adoption
D3 adopts a dead node's block only once the block is at least
ADOPT_AFTER_CONFIRMATIONS (60) below the tip, so its finder has time to
land and sync it first. Adoption passes run every 30 s. S8 waited 90 s
with A's block at the tip, so B could never adopt it, and B had no
candidate row for it at the end.
S8's accepted variants now mint 60 external blocks on C on top of A's
block, after the chain's nodes agree on the tip, and only then wait the
90 s adoption bound: two passes and a landing. The lost variant has no
block on the chain to bury.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: temporary stand-ins for D1's pending 031 index and singleton grant
The PR now targets the whole 3.1 stack (D1 cca56f14, D2 c25bb19b, D3
78ca1a30, D4), and that stack still lacks two things D1's pending push
delivers. Without them, no dual-writer pair can run:
- Migration 031's (origin_node, share_seq) index. D2's window cut refuses
to run without it.
- SELECT on qbit_prism_cluster.singleton for the peer role. D1's engine
reads the fingerprint WHERE singleton, which D1's documented column
grant does not cover (status/D6.md, finding 4).
Until that push, the bootstrap applies both as TEMPORARY_STAND_INS, after
PEER_ROLE_GRANTS, which stays exactly D1's list. The index uses 031's own
name, as D4's readiness fixture does (e5db2eb1). Both are no-ops once the
stack carries them, and the list is then removed. This way the dual-writer
scenarios run in CI on the stack meanwhile, instead of failing for D1's
known reasons. The README says so.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: S1, S2 kill -9, S4 and S5 join the fast lane; the nightly matrix runs every scenario the stack can
The PR is retargeted onto the whole 3.1 stack, so the scenarios that
waited for it now run.
- Fast lane (test/prism-dual-writer-gated-tests.txt): S1, S2 kill -9, S4
link cut and S5 join the checker control and S10. All six pass locally
on D3 78ca1a30 + D4.
- Nightly matrix (test/prism-dual-writer-nightly-gated-tests.txt), 22
scenarios: the fast lane plus S2 PostgreSQL kill, freeze and network
drop; S2 frozen mid-pull and puller death; S3; S4 transient apply; S6
restore A and B and the unreadable peer, locked and refused; S8 lost and
accepted, with and without the D-19 wait; S9; and S11. Some fail today
for findings their owners hold (D1's barrier and P1s 1 to 3), which is
what the matrix report is for.
- S7 stays planned. D-16's node-identity repersonalise is not in the stack,
and both variants stop there.
- test/e2e-scenarios.toml marks each of these as running, with its tests.
- The nightly job's limit goes from 90 to 150 minutes. At about 3 minutes
a scenario, 22 take about 70, and a failing scenario can wait out its
bounds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: S8 times B's adoption from the block's burial, and says so
The adoption wait starts once A's block is 60 blocks deep, after the
mint, not when A died. Its detail said "after A died".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: the candidate diagnostic shows an unclaimed row as unclaimed, and the lease as PRISM takes it over
These come from the local review of the S8 commits.
- An unclaimed row's claim_expires_at is NULL. The 'infinity' guard
turned that into Infinity, so a row nobody holds read as claimed
forever. NULL now stays NULL.
- Since migration 021, a restarted frontend takes a dead claim over only
once it has watched the claim go unrenewed for the whole 120 s lease on
its own clock, not when claim_expires_at passes. The diagnostic now calls
the expiry the database's view, and RECOVERED_LANDING_BOUND's comment
describes the takeover as PRISM does it. The 180 s bound still covers
it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: S9 digests the divergence report it claims; the singleton stand-in goes by hand; S6 and S8 criteria say what they check
These come from the local review of the stand-ins and the lane change.
- S9's criteria and ledger_digest's doc said the cutover leaves the
divergence report unchanged, but the digest never read it. It now
includes qbit_prism_payout_divergence_report(), which holds only counts
and sums.
- D1's fix for the peer role's singleton read drops the WHERE singleton
predicate rather than granting the column. So the singleton stand-in
never becomes a no-op, and it would hide that fault coming back. The
comment and the README now say it goes with the list, by hand, at the
rebase onto D1's push.
- S6 unreadable allows 120 s to ready and then 60 s for the first share.
The criteria now say so.
- S8's criteria now describe the lost variant (never on the chain, never
paid, the candidate kept for reconciliation) apart from the accepted ones
(adopted once 60 deep, one landing per node).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: S8 checks the lost block is never offered again; S6 and S8 docs match the tests
These come from the local review of 1303a261.
- S8's lost-variant criteria said A never offers the block again, but
nothing checked it. A's RPC gate stays in front of its qbitd through A's
restart, so at the end of the scenario the gate must have seen exactly
one submitblock of the lost block: the call it held and discarded as A
died. A resubmission would land as a stale sibling outside the census,
and S8 would have passed it. Adds RpcGate::submissions_of.
- The README and S6's doc comment now give S6's timing as the test runs it
(ready within 120 s, then a share within 60 s). The README now says B's
adoption is required with the D-19 wait and only measured without it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: drop the temporary stand-ins; D1's push carries 031 and the grant fix
D1's 193e7fd8 delivers migration 031's (origin_node, share_seq) index. It
also reads the peer's fingerprint with only the sync role's documented
column grant (e3fb8cc5). Once the stack sits on that push, the bootstrap
applies exactly D1's peer role grants and nothing else, as it did before
the stand-ins. The singleton grant goes too, as D1's fix drops the read
that needed it, so a regression to that read fails here again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: S7 waits, as D1's runbook says, for the copied frontend rows before it repersonalises
The first S7 run against D1's node-identity repersonalise (193e7fd8)
failed: the command refused with "a frontend may be running on this
database: dual-sim-a (running, heartbeat 0.690 seconds old)".
The rebuilt node is a physical copy of its peer, so it carries the peer's
frontend instance rows. D1's runbook (docs/prism-ledger-ops.md,
"Rebuilding a node from its peer") says those rows stop counting 60 s
after the copy stopped following the peer, and until then the command
refuses without changing anything.
S7 now retries the command every 5 s, for up to 90 s, while that is the
refusal. Any other refusal fails at once, and the expectation reports how
long the wait took.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: S7 joins the nightly matrix; all 24 scenarios run on the stack
D1's 193e7fd8 delivers node-identity repersonalise (D-16). With the
runbook's wait for the copied frontend rows (942038e4), both S7 variants
pass on a local tree of D4 c22bd1f7 merged with D1 193e7fd8, with no
stand-ins:
- repersonalised after 60 s;
- only the dead node's unsynced tail lost (1 share);
- every invariant held, including D-2's reset after the rebuild.
S7 is now marked running in test/e2e-scenarios.toml, and its two tests are
in the nightly list, which now holds all 24 scenarios. The nightly job's
comment is updated; its 150 minute limit stays.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: S7's runbook wait is on the timeline and in the README's list of retries
This comes from the local review of 942038e4. The README's "Bounds and
retries" said that no scenario retries the behaviour under test, and it
listed every retry. S7 now retries node-identity repersonalise for up to
90 s, for the one refusal D1's runbook says to wait out. The README now
names that wait as the single exception, and the timeline records how long
the wait took and how it ended.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: free_port never hands out the same port twice in one process
The full local matrix lost one scenario to "bind primary Stratum
listener: Address already in use". free_port binds port 0, reads the port
and drops the listener, and the port's user binds it later. Nothing stops
the next free_port from returning the same port in the meantime, so two
parts of one scenario could be given the same port.
free_port now records every port it hands out and draws again, up to 64
times, when the kernel offers one it already gave out. That rules out a
collision within one scenario. A collision with another process still
fails the start loudly; CI runs one scenario at a time on its runner. Adds
a unit test: 200 ports in a row are all distinct.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: relays, the balancer and the RPC gates take their ports through the same registry
This comes from the local review of 403e76ce. The registry covered only
free_port's own draws. Relay::open, the balancer and the RPC gate bound
127.0.0.1:0 themselves and kept the listener. One of them, opened after a
draw but before the port's user bound it, could still get that port. For
example, Sim::start draws A's Stratum port, then opens the public relays
before any frontend binds. The reviewer replayed Sim::start's bind order
and found this path at least as likely as the one 403e76ce closed.
All three now bind through fresh_listener(). It binds port 0, keeps the
listener only if the port is new to the registry, otherwise binds again,
and hands back a non-blocking std listener for tokio's from_std. The unit
test now checks that 200 handed-out ports and 50 kept listeners never
share a port.
The README also stated S7's retry too strongly. repersonalise joins every
refusal into one message, so a refusal that names the copied frontend rows
beside other reasons is retried until those rows age out, then fails on
the rest. The text now says that.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: a failed bind of a relay, the balancer or a gate names what it was binding again
This comes from the local review of 11fc9378. In
from_std(fresh_listener()?).with_context(..), the ? returned
fresh_listener's error before the context was added. An EMFILE, or 64
draws without a new port, then reported no "binding relay public-stratum-a"
or "binding the balancer". The context now wraps both calls.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: S1's lag percentiles count shares that never arrived; the share fingerprint keeps origin_node
These are Codex's two P1s on bfc581f6.
1. S1's lag check dropped unresolved samples from its percentiles. One
share that synced quickly, among many that took tens of seconds, could
pass the 2 s p95 bound.
- lag_report now ranks every sample, an unresolved one above every
resolved lag. A percentile that falls on one is None, which fails the
check.
- So that this is fair, the sampler stops taking samples when told to,
but keeps polling the pending ones for LAG_DRAIN_MS (3 s). Every
sample has then had more than the bound to arrive.
- The report counts the unresolved samples, and S1's detail names them.
- The unit test checks that one quick sample among nineteen that never
arrived does not pass.
2. inv4-no-double-credit compared each header's row across nodes without
origin_node. A sync that copied a share with the wrong provenance would
have passed. origin_node is now part of the fingerprint.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: the share check compares every column across nodes and each header with its share's own hash
These are Codex's two P1s on 8c7ac95f. Both are gaps in
inv4-no-double-credit.
1. A header-to-share mapping was only required to point at some accepted
ledger row. A sync that swapped two valid mappings would have passed.
Each mapping's header must now equal its share id's hash
(lower(right(share_id, 64)), the server's own mapping), or the check
fails with "header mapped to another share".
2. The cross-node comparison used a hand-picked list of columns, which
left out payout_order_key, template_height, credit_policy and others.
The ledger is append-only and copied whole (D1's table inventory leaves
no column out). So each header's row is now compared on both nodes as
to_jsonb, every column, read under SET LOCAL TIME ZONE 'UTC' so that
timestamps render alike. A mismatch names each differing column with
both values. A unit test covers that naming.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: acknowledged shares are found by their full id, and the pair converges by content, not counts
These are Codex's two P1s on e21a428e.
1. inv4-acked-shares-present looked each acknowledged share up by its
header only. A committed row under the same header but with another
worker prefix would have passed for it. The check now binds the exact
share ids the miners were answered for, which the load client builds as
the server does ({username}:{header hash}). Each must be an accepted
ledger row with its mapping.
2. wait_converged, which settle uses for the dual-writer pair, compared
the share and block counts. A sync that dropped one A row and a
different B row would have left the counts equal and the ledgers
different. Each reading is now a count plus an md5 digest of every
share mapping (header=share_id) and every block hash, ordered under the
C collation so that both clusters sort alike. The pair converges only
when the two digests match and still match a second later.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: landings compare in every column the sync carries, and a peer block counts as arrived only whole
These are Codex's two P1s on ac54ef80.
1. inv3-landing-rows compared a hand-picked set of landing facts across
nodes. A block copied with the wrong origin_node, or with another
carried column wrong (sync_seq, found_at, the solver), would have
passed. Each node's landing is now one document holding every row of
it: the block, bundle, payout entries, carry rows, fanout set and
artifacts. Each row has exactly the columns D1's peer sync carries of
its table, read from that node's catalog with D1's CarriedColumns. The
snapshot is compared by its content-determined facts. Timestamps are
read in UTC. A block both nodes hold must have the same document on
both, and a difference names each differing column by its path, with
both values. Only the columns the sync leaves to each node are left
out: chain and maturity state, publication order and claims.
2. The D-10 sampler of S1 and S5 settled a peer block once its audit
bundle and snapshot were there. Payout, carry or fanout rows committed
in a later transaction would have gone unseen. A block now counts as
arrived only when its whole landing is there in the same statement:
the bundle, the snapshot, a payout entry per manifest account, a carry
row per miner account, and the fanout set and artifacts the bundle
names. That is the condition inv3-landing-rows holds a landed block to.
A block seen with part of its landing missing is torn and fails the
scenario.
S8's adoption is the one path on which both nodes could land one block
themselves. B adopts A's block while A is down, and A's restarted
frontend could land it too. A cannot do so before it has waited out the
120 s candidate lease, and by then it has pulled B's adoption, so it
skips its own. If it ever did land its own, the pair would hold two
landings of one block, and the checker now says so.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: S8's accepted block must be on the chain, and a missing share counts once
These are Codex's P1 and P2 on efc90dff.
1. S8's accepted variants never required the block to reach the chain.
The RPC gate records a submitblock as answered once its forward ends,
also when the forward failed. Without the D-19 wait, the adoption
expectation passes either way, and every landing check sits under
`if on_chain`. So the no-wait variant could pass without its premise:
that the node accepted the block before A died. Both variants now
expect the block on the chain. Only the adoption's timing stays
ungated without the wait. The gate's docs now say what "answered"
records.
2. inv4-acked-shares-present counted a missing acknowledged share once
per database that lacked it. S7's unsynced tail is missing from both
the survivor and the rebuilt copy, so the report's excused tail
doubled. A lost share missing from both nodes was likewise reported
twice, and the report counts those problems as shares lost. A pure
tally_missing now sorts the missing ids into excused shares (by
reason) and lost shares (with the nodes that lack each). Each share
counts once and gets one problem naming its nodes. A unit test covers
a share missing from both databases.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: S8 asks the network whether it knows the block, and only a "not found" means no
S8 read "the block reached the chain" as any successful getblockheader
on C, and anything else as "not there". A failed call to C (C
unreachable, a timeout) therefore passed the lost variant's "the lost
block never reached the chain" without proving it.
The chain node now has knows_block. It returns false only on the node's
own "block not found" answer (RPC error -5, as qbitd and the server's own
RPC code use it), and an error for any other failure. S8 propagates that
error, so an unreachable C fails the scenario instead of deciding it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: D-2 checks deferred shares, and S9's cutover digest covers every carried column
These are Codex's two P1s on 68a87ade.
1. d2-local-state-not-copied checked candidates and offer decisions, but
never qbit_prism_deferred_shares. D1's table inventory keeps that
table local and clears the peer's rows on re-personalise, because a
copied deferred solving share could be credited twice. A deferred
share's foreign key ties it to a candidate in its own node's outbox,
but a copied candidate with fresh timestamps and no claim would slip
past the candidate checks. So the check now also requires that no
deferred share whose candidate dates from the dual epoch is on both
nodes, by block and share digest. The counts per node are in the
check's data.
2. S9's before/after digest covered picked columns of some tables and
none of qbit_ctv_fanout_sets. A cutover that corrupted a fanout set,
or any unpicked column, identically on both promoted databases would
have passed it, and every later cross-node check too. Each copied
table is now digested whole over every column the peer sync carries,
read from the database's catalog with D1's CarriedColumns: shares,
share hashes, blocks, bundles, payout entries, carry rows, fanout sets
and artifacts. Snapshots are digested by their content-determined
facts. Timestamps are read in UTC. Only each node's own columns are
left out (chain and maturity states, claims, schedules). The cutover
never rewrites a carried column: origin_node and sync_seq are set
only at insert.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: copied state, scenario premises and fair report numbers, from a sweep of the families Codex keeps finding
Before pushing round 7, three read-only passes over the whole harness
looked for the classes Codex's rounds kept finding: never-copied and
copied state left unchecked, a variant's premise assumed rather than
measured, and report numbers that misstate an outcome. All harness-only.
Copied and never-copied state (D-1, D-2, D-11, D-16, D-17):
- copied-state-identical, a new check. For each origin node against its
peer: the carry-owner journal is identical. Every unexpired prepared
record the origin wrote, up to the peer's cursor, is on the peer, equal
in every carried column but expires_at (D1's own comparison). Every
unexpired copy is still on its origin. The templates and balance
snapshots they reference are byte-identical. The check retries for up
to 20 s for rows in flight.
- D-2 covers issued jobs: an issued job of the peer's, by origin or
instance, since the epoch, is a copy that resume_job could replay.
- S7 checks re-personalise's work before the rebuilt node starts, from
D1's TABLE_INVENTORY:
- every cleared table is empty, and no Rows or Reset predicate matches;
- the identity is the node's own, and the lineage is unverified;
- every cursor follows the peer from at or below its highest key.
- S6's "held every own row before it was ready" now includes unexpired
prepared records and journal rows.
- inv3-audits-verify reports a stopped frontend, unless its node is
listed as away.
- The checker control runs the D-2 check on A's database in both slots,
where it must name a candidate that is the same row in both outboxes.
Premises measured, not assumed:
- S2 and S10: every session that mined before the fault has a share
accepted on the survivor's work within 30 s, read before the heal. Before,
one share sufficed. S10's failback must route every session back to A.
- S2 (kill, freeze, network drop): A holds none of B's blocks before the
heal. A's tail is checked to be on B at the end, not hard-coded true.
- S4's transient apply needs at least two attempts, which shows the
retry.
- S6: right after the restore, the node lacks the block it found after
the backup and own shares the peer holds.
- S9's D-12 refusal must be D-12's own text.
- A frontend that reports a PRISM setting unread ("set but unread by
native PRISM") fails the scenario. This covers S8's no-wait override
and every other one.
- The checker control's expected failures must name their causes through
both databases: each block, fanout, share and window missing on the
other node.
Fair numbers:
- A scenario that panics or is stopped leaves a failed report, written
when its run starts, instead of vanishing from the matrix.
- S3 and S4 are judged by the longest stretch with no accepted share over
each whole window (report::longest_gap), not only the one across the
fault.
- S7 excuses only tail shares answered within the sync lag of the death.
Older ones fail as lost.
- The matrix summary shows "not checked" and "not measured" for a run
that stopped before its checker or before any fault.
- Unplaced offers are reported. The puller-death p95 counts refusals and
unanswered shares, and S10's failback window ends where the load
stopped.
Not changed: the checker control's owner-balances failure stays
conditional on B's block leaving a carry. Forcing one would need a payout
floor that template building rejects.
Unit tests cover the longest gap, per-session failover and the tail age
bound. Python tests cover the summary's rows for runs that stopped early.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: the D-2 control counts the copies it finds on both nodes, past the capped problem list
The checker control runs the D-2 check on A's database in both slots, and
expected a "same row in both outboxes" problem among those listed. The
check failed as it must, with 245 problems. But it lists at most 40, and
its cross-node findings come after every node's own: the candidates A's
frontend claimed, and A's issued jobs, about 240 of them. So the expected
words were never listed, and the control failed on a correct check.
The D-2 check now counts its cross-node findings in its data
(same_rows_on_both_nodes: candidates, offer decisions, deferred shares).
The control requires each way of finding a copy to fire: a candidate
claimed by the peer's frontend and an issued job of the peer's (both
listed first), and at least one candidate that is the same row on both
nodes (counted).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: copied-state-identical bounds the prepared records it requires by what the origin had written, not by the peer's cursor
copied-state-identical required on the peer only the origin's prepared
records at or below the peer's cursor over that stream. A stream that
stopped would then pass. When apply_prepared fails, the engine retries
every pass and leaves the cursor where it is, so nothing would be
compared, and the fast lane would stay green while the peer could adopt
nothing. Only S8 would notice.
The bound is now the origin's highest prepared sync_seq when the check
begins. Every unexpired record up to it must reach the peer within the
check's 20 s retry, which already absorbs rows still in flight, whatever
the cursor says. The cursor stays in the check's data for diagnosis,
beside the new highest_at_start.
From the local review of 99d45284.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: S9 keeps the prepared state whole through the cutover, and copied-state-identical fixes what it requires when it begins
These are Codex's P1 and P2 on b91dde24.
1. S9's cutover digest left out four copied tables: prepared records,
their templates and balance snapshots, and the carry-owner journal. A
promotion or bootstrap that lost or damaged them identically on both
databases would have passed, and new work would then replace the lost
records. A digest across the whole cutover cannot see this, because
prepared records live about as long as the jobs that use them, which is
shorter than a cutover. Instead, cutover_to_dual reads the four tables,
whole and in UTC, at two points:
- on A right after the drain;
- on each node after the promotion and the bootstrap, before either
frontend starts.
Nothing writes or prunes these tables between those reads. S9 requires
every row from the drain to survive identically on both nodes, and
prepared records to exist at all, so the check cannot pass empty.
2. copied-state-identical recomputed the origin's unexpired prepared
records on every retry. A missing record that drifted into the 5 s
expiry cushion during the 20 s retry dropped out, and the check passed.
Each origin's targets are now read once, when the check begins:
- its unexpired prepared records up to its highest sync_seq;
- its journal rows;
- the md5s of the blobs those records reference.
The retries then compare only the peer's copies against that fixed set.
Copies are kept 24 h past expiry, so a record that ages meanwhile stays
required. The rows the origin lost while its peer kept them are judged
once, at the start: unexpired copies, journal rows, and blobs its
records reference.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: copied-state-identical reads the peer before the origin when it looks for lost rows
copied-state-identical judges once, at the start, which rows the origin
lost while its peer kept them. It read the origin first and the peer
after. A prepared record or journal row could be written on the origin
just after that first read and pulled by the peer before the second.
That row then looked lost on the origin, and the check failed for good on
a correct pair: a rare flake, since the load-paused frontends still write
a prepared record about every second and the peer pulls every 250 ms.
The peer's unexpired copies and journal rows are now read first. Anything
the peer holds, the origin wrote before, so the origin's later read holds
it too, unless it really was lost there. An unexpired copy's original
cannot be pruned in between, since its expiry is the later one.
From the local review of ed500cb1.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.1 dual-writer E2E: S9's prepared state must be exactly what the drain left, nothing added
This is Codex's P1 on 1e5403e0. S9 compared the prepared state on each
node after the promotion and the bootstrap with A's right after the
drain, but only the rows from before. A promotion or bootstrap that
inserted a prepared record, a blob or a journal row would have passed,
and so would every later cross-node check once both databases held it.
Neither step writes these tables. node-identity set changes only column
defaults and seq…
… 031, and p0 is never rewritten (#773) * 3.1 pre-migrate: rc.7 keeps serving while 3.1's migrate applies 027, 028 and 031, and p0 is never rewritten The 3.0 -> 3.1 cutover can run 3.1's migrate against A's database before its window, while the 3.0 pair serves on 3.0.0-rc.7. Two gated dual-sim tests hold that to what the cutover relies on, in the fast lane and the nightly matrix: - R1: the 3.0 pair on rc.7 under load (database made by rc.7's migrate, every start with PRISM_POSTGRES_INIT_SCHEMA=0). 3.1's migrate runs once with A's frontend environment while a reader holds a snapshot, so 031's CREATE INDEX CONCURRENTLY is caught part-way. 027 and 028 commit with no share refused and none waiting the 5 s lock timeout, appends continue, a block lands, A restarts and serves with 31 unrecorded and the index invalid, then 31 is recorded, the index is valid on the writer and the standby, B restarts at 31, a last block lands, and the checker's invariants hold. - p0: on a populated ledger with an appender under the frontends' lock timeout, p0 and its TOAST table keep relfilenode, relfrozenxid and relminmxid and their vacuum and analyze counters, the statement log holds no VACUUM, ANALYZE, CLUSTER or REINDEX, and p0's only index build is CREATE INDEX CONCURRENTLY (qbit-tools #1345). A by-hand measurement runs the second at millions of rows. The gate gains PRISM_TEST_RC7_SERVER_BIN, and the prism-rc7-server action builds the pinned rc.7 commit (#772's head b7b6923) once per pin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.1 pre-migrate tests: catch every index build on p0, and anchor R1's restarts From a local review of #773: - An index build on p0 is any CREATE INDEX on it, named or not, or on the ledger without ONLY (which recurses into every partition); the scratch apply's statements, under the migrator's rolled-back savepoint, are left out, and maintenance commands are counted everywhere. - p0's TOAST table's vacuum and analyze counters are compared too (pg_stat_all_tables), and log timestamps are compared by date and time, so a run across midnight measures right. - The seed is refused unless every row stays in p0. - R1's restarts wait for the balancer to mark the node down before starting it, require a graceful stop, and count shares from when the balancer had the node up again; the one-off migrate and the appender are aborted, and the migrate killed, when a step fails first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.1 pre-migrate tests: the pair serves rc.6, not rc.7 rc.7 was skipped: the pair serves v3.0.0-rc.6 (81c592c) until the 3.1 repin, so the pre-migrate runs beside rc.6 frontends, and R1 and the p0 test run rc.6 (PRISM_TEST_RC6_SERVER_BIN, the prism-rc6-server action at the v3.0.0-rc.6 commit). rc.6's start gate, require_schema_version, its NATIVE_CAPABILITIES and its connect-time capability checks are identical to rc.7's (4c13491), as are the migration files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* 3.x.x: version 3.1.0-rc.1 VERSION, the workspace version and both lockfiles move to 3.1.0-rc.1. The fuzz lockfile also picks up hyper and hyper-util, which #769 added to the server without refreshing it. The 3.0.0 and development-line notes point at the new candidate; its own notes follow in this pull request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.x.x: 3.1.0-rc.1 release notes The notes cover everything since v3.0.0-rc.6: #770's import-audits fix (#765) and #760's idle-connection bound (#759), which would have shipped as rc.7, and the 3.1 dual writer (#766, #768, #767, #769, #763), off by default. They describe the new settings, migrations 027, 028 and 031 (029 and 030 reserved), the upgrade from rc.6 with the migrate run while rc.6 serves, the rollbacks, and the dual writer's known limits. The migration guide's import-audits paragraph names this candidate in place of the skipped rc.7. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.x.x: 3.1.0-rc.1 notes: #773's pre-migrate tests Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.x.x: 3.1.0-rc.1 notes: the rollback to one writer stops and syncs the peer first Codex on #774: DOWNGRADE fences nothing, so the peer must be stopped, and its tail pulled, before the single-writer start; declare the tail lost only for a peer that never returns with its own database. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…tls for six RustSec advisories (#777) cargo deny fails on RUSTSEC-2026-0382 (rustls-webpki < 0.103.16) and RUSTSEC-2026-0381 (webpki-roots < 1.0.10), and the fuzz lockfile still has rustls 0.23.45 (RUSTSEC-2026-0346, -0347, -0348 and -0349); the root lockfile already has 0.23.46. Move each to its first fixed patch release with cargo update --precise, as #354 did. No other dependency changes. The same fix for 3.0.0 is in #776. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* 3.x.x: version 3.1.0, and tokio 1.53.3 for RUSTSEC-2026-0386 VERSION, the workspace version and both lockfiles' workspace members move to 3.1.0. tokio moves from 1.53.1 to 1.53.3, its first fixed patch release for RUSTSEC-2026-0386, in the root and fuzz lockfiles, as main took it in #781. cargo update -p tokio --precise 1.53.3 also moved the windows-sys edges of four unrelated crates to 0.52.0, so, as in #781, only tokio's version and checksum change. Nothing in the workspace uses AsyncFd. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.x.x: 3.1.0 release notes, and 3.x.x as the production line doc/release-notes-3.1.0.md covers the changes since 3.1.0-rc.1 (#777's TLS updates and the tokio update), the upgrade from rc.1 (a binary swap) and from 3.0.0 or 3.0.0-rc.6 (#773's pre-migrate, the binary, then the optional dual-writer cutover), and the known issues carried from rc.1 plus #779's L3 failures and the acknowledgement bound's connection drop that main fixes in #781. The README banner, the development-line note and the migration guide now say 3.x.x is the production line, deployed from the v3.1.0 tag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.x.x: 3.1.0 notes: review fixes The migration guide and ledger operations cite the released 3.0.0 notes' D5 boundary at v3.0.0, not this branch's pre-release copy. The development-line note's header no longer calls the line "not a release". The 3.1.0 notes say a dual-writer-off frontend serves as 3.0's does once the migrations are applied, tell a 2.0.x upgrade that #765's guard is not needed, and list the required prism-dual-writer lane and rc.1's evidence set under Verification. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.x.x: 3.1.0 notes: second review round The #779 known issue counts the async preset's failed repeats (four of six) and names the sync preset rather than a repeat's job. The cutover step names node-identity repersonalise and defers the cutover itself to the pair's runbook, as rc.1's notes do. Verification says docs changed since rc.1 too. The development-line note says 3.0.x is not an ancestor of 3.1.0, and the migration guide no longer implies the 3.0.0 candidates were a supported production upgrade. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.x.x: 3.1.0 notes: rewrap two lines Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Merge 3.x.x at the v3.1.0 tag (59194d0) into main and take its tree, as #778 did for 3.0.0. Changes kept on top of the tag: - #781's disposition-poll change, which main took after 3.1.0's cutoff: each block-proof acknowledgement poll runs in its own task, so the bound never drops a connection partway through a checkout. Its two acquire tests, the pool-acquire metrics guide and the checkout census come with it. miner_submit.rs is identical in v3.0.0 and v3.1.0, so the change applies unchanged. tokio is 1.53.3 on both sides. - docs/releasing.md, which only main has, records 3.1.0's promotion. - The README banner names the version, not the branch. - doc/release-notes-3.0.0.md is the released v3.0.0 text, in place of 3.x.x's pre-release copy. The links that pointed at blob/v3.0.0/ for it (the 3.1.0 notes, the development-line note, the migration guide and ledger operations) point at the local file again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs/releasing.md says #781 is on main because 3.1.0 changed no runtime code since rc.1, not because it came after the cutoff; names only v2.0.0 as tagged on main; and requires the post-3.1.0 sync into 3.x.x, which lacks #781's fix and the released 3.0.0 notes. The development-line note names 3.x.x where it said this branch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 3.1.0 paragraph lists every doc edit the promotion keeps and says 3.1.0 took #781's tokio update but not its disposition-poll change. The tag example no longer claims v3.0.0 and v3.1.0 used its message. The sync step says neither 3.x.x nor 3.0.x has merged a released main yet and lists what each lacks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 3.1.0 paragraph says the links to the 3.0.0 notes already pointed at the v3.0.0 tag and now point at the local copy, that 3.1.0 made #781's tokio update itself in #782, and that #779 records 3.0.0's L3 failures. The banner names the version. 3.x.x is the development branch and production line. The sync step says 3.x.x lacks only the local copy of the released notes. Edited paragraphs are rewrapped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: aeed395f38
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![qbit-reviewer[bot]](https://avatars.githubusercontent.com/in/5446813?s=60&v=4){kind=small}
Member
Author
|
@codex review |
|
Codex Review: Didn't find any major issues. You're on a roll. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
kiwidream
changed the title
This was referenced Oct 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Promotes the released v3.1.0 to
main, followingdocs/releasing.mdand #778, which did the same for v3.0.0. The head mergesmain(b9579c67, #781) with thev3.1.0tag (59194d03, #782) and takes the tag's tree. The repository allows only squash and rebase merges, so this lands as one squashed commit.v3.1.0is already tagged and published on3.x.x, so merging this PR adds no tag.What
mainbecomesThe
v3.1.0tag's tree plus whatmainhas that3.x.xdoesn't. Every file that differs from the tag:Why
mainkeeps each one:main: update tokio for RUSTSEC-2026-0386, and keep the block-proof acknowledgement bound from dropping a disposition poll's connection #781's part 2: the disposition poll. These files make up main: update tokio for RUSTSEC-2026-0386, and keep the block-proof acknowledgement bound from dropping a disposition poll's connection #781's second part, which
maintook after 3.1.0's cutoff and 3.1.0 doesn't carry:miner_submit.rsandminer_submit_acquire_tests.rs: each poll under the block-proof acknowledgement bound runs in its own task, so the bound no longer drops a connection partway through a checkout.docs/prism-pool-acquire-metrics.mdandscripts/prism_pool_acquires.json: the matching acquire-metrics guide and checkout census.miner_submit.rsand its acquire tests are byte-identical inv3.0.0andv3.1.0, so main: update tokio for RUSTSEC-2026-0386, and keep the block-proof acknowledgement bound from dropping a disposition poll's connection #781's hunks apply unchanged. The guide and the census differ between the tags, and main: update tokio for RUSTSEC-2026-0386, and keep the block-proof acknowledgement bound from dropping a disposition poll's connection #781's hunks apply cleanly onto 3.1.0's versions;scripts/check_prism_pool_acquires.pypasses. main: update tokio for RUSTSEC-2026-0386, and keep the block-proof acknowledgement bound from dropping a disposition poll's connection #781's part 1, tokio 1.53.3, is already inv3.1.0(DO NOT MERGE without coordinator GO: 3.x.x: version 3.1.0 and its release notes #782), so the lockfiles are the tag's.docs/releasing.mdexists only onmainand is kept. It now records 3.1.0's promotion:3.x.xitself, with no release branch, and this promotion adds no tag;3.x.xnor3.0.xhas merged a releasedmainyet. Both lack this guide, the banner and main: update tokio for RUSTSEC-2026-0386, and keep the block-proof acknowledgement bound from dropping a disposition poll's connection #781's disposition-poll change;3.0.xalso lacks its tokio update, and3.x.xthe local copy of the released 3.0.0 notes.README banner: it names the version ("Version 3.1.0: the Rust PRISM server …, with the 3.1 dual writer, off by default"), not the branch, as v3.0.0 chore: consolidate release for main #778's did for 3.0.0.
The released 3.0.0 notes. This settles the "UNRELEASED" point raised on DO NOT MERGE without coordinator GO: 3.x.x: version 3.1.0 and its release notes #782.
doc/release-notes-3.0.0.mdis now exactlygit show v3.0.0:doc/release-notes-3.0.0.md. That is the textmainhad after v3.0.0 chore: consolidate release for main #778, in place of3.x.x's pre-release copy, the D5 contract under an "UNRELEASED. NOT PRODUCTION-READY." header.DO NOT MERGE without coordinator GO: 3.x.x: version 3.1.0 and its release notes #782 had linked four places to
blob/v3.0.0/because the local copy was the pre-release text. They now point at the local file again:docs/prism-rust-migration.mdanddocs/prism-ledger-ops.md.The development-line note's sentence saying the local copy "still records only the D5 recovery contract" is replaced by a link to the notes. The
#migration-and-recovery-d5anchor exists in the released text, and DO NOT MERGE without coordinator GO: 3.x.x: version 3.1.0 and its release notes #782's ledger-ops one-line change stays, now pointing at the local file.3.x.xkeeps the pre-release copy until the post-release sync below.So
doc/release-notes-3.1.0.mddiffers from the tag only in that link and the rewrapped sentence around it. Its known issue about main: update tokio for RUSTSEC-2026-0386, and keep the block-proof acknowledgement bound from dropping a disposition poll's connection #781 still describes 3.1.0 as released.main's own content is all keptgit diff v3.0.0 origin/main --statlists exactly 8 files:v3.1.0;README.mdanddocs/releasing.md, kept as above;No
maincommit since #778 touches anything else.main's workflows equalv3.0.0's, so takingv3.1.0's CI and load-harness workflows loses nothing.Against
maingit diff origin/main HEAD: 202 files, +54,745 / −777, 66 of them added. It is 3.1.0 over 3.0.0:crates/qbit-prism-dual-simand its CI lanes (3.1 dual-writer (5/5): dual-writer E2E simulation and CI #763), and the rc.6 pre-migrate tests (DO NOT MERGE without coordinator GO: 3.1 pre-migrate: rc.6 serves while 3.1's migrate applies 027, 028 and 031, and p0 is never rewritten #773);VERSIONand the workspace version: 3.1.0;qbit-prism-dual-simmember, and the server'shyperandhyper-utiledges in the fuzz lockfile (3.1 dual-writer (4/5): dual-writer readiness and Stratum gating #769, refreshed in DO NOT MERGE without coordinator GO: 3.x.x: version 3.1.0-rc.1 and its release notes #774). tokio 1.53.3 and 3.x.x: update rustls-webpki, webpki-roots and the fuzz lockfile's rustls for six RustSec advisories #777's TLS versions are the same on both sides;Checked locally
python3 scripts/check_version_skew.py:VERSION 3.1.0 matches every workspace crate.python3 scripts/check_prism_settings.py:PRISM setting inventories and operator guidance agree.python3 scripts/check_toolchain_pin.py:Rust 1.98.1 is pinned consistently ….python3 scripts/check_prism_pool_acquires.pypasses.cargo metadata --lockedpasses for the root and fuzz workspaces.cargo deny --locked check:advisories ok, bans ok, licenses ok, sources ok.cargo check --locked --workspace --all-targets,cargo clippy --locked --workspace --all-targets -- -D warningsandcargo fmt --all -- --checkpass./code-review mediumrounds ofgit diff v3.1.0 HEADran before pushing, and their findings were fixed. These were all wording indocs/releasing.mdand the development-line note, where "this branch" and "the branch head" now name3.x.x. One finding was left as it is: main: update tokio for RUSTSEC-2026-0386, and keep the block-proof acknowledgement bound from dropping a disposition poll's connection #781's mapping of a panicked poll task to a retriedpoll-error. That ismain's merged code, carried unchanged.L3
As on #778 and #782, the 400k two-frontend presets can fail on #716's commit-gate refusal. That is #779, and it is handled separately.
Follow-ups
3.x.x. The peer sync'sapply_preparedtakes no cluster-row fence against blob pruning, so a pulled job can commit while the pruner deletes an existing blob it reuses. The code isv3.1.0's, unchanged here; this PR carries the tag's tree and fixes nothing in it.After merge
maininto3.x.xand into3.0.x(docs/releasing.md, "Reconcile the development branch"). They bring main: update tokio for RUSTSEC-2026-0386, and keep the block-proof acknowledgement bound from dropping a disposition poll's connection #781's part 2 and the released 3.0.0 notes back to3.x.x. Don't reset or force-push either branch.mainneeds a second human approval; no admin bypass.🤖 Generated with Claude Code