Repository navigation
v3.0.0 chore: consolidate release for main - #778
Merged
Merged
Conversation
…nces (#302) * Freeze money-path vectors from 2.x.x and record D2 payout differences (#269) Export 48 money-path cases from 2.x.x (504846c) into crates/qbit-prism/fixtures/vectors/. Topics 1-7 (window clipping, fractional oldest share, carry-only recipients, pool fee, dust recompute, largest-remainder ties, CTV chunks) come from the 2.x.x qbit-prism entry points. The bootstrap transition and below-target credit come from the 2.x.x Python rule code, with each rule's payout computed by the same engine. The exporter source and its regeneration command sit under fixtures/vectors/exporter/, outside any cargo target. tests/money_path_vectors.rs replays every case through the 3.x.x engine and reports all mismatches at once. Topics 1-7 match 2.x.x exactly. The five differing cases carry expected_2xx, expected_3xx and a d2_entry whose anchor must exist in docs/prism-rust-migration.md. That doc now records D2a bootstrap pooling, D2b below-target credit, and D2c prior balances during bootstrap, the last pending the umbrella owner's approval. checked_in_hand_worked_fixtures_match_engine now also asserts the power-law-accrual fixture's floor, on-chain count and accrued recipients. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Pin the money-path vectors and freeze the zero-fee boundary (#269) The vector test now pins the sha256 of every vector file next to SOURCE_COMMIT, so a vector changes only through a re-export from 2.x.x in a reviewed commit that updates the pin. It also requires the set of d2_entry anchors used by the cases to equal the entries documented in the "Payout differences from 2.x.x (decision D2)" section, which must be exactly D2a, D2b and D2c. Deleting a D2 case, or rewriting one into a plain matching case, now fails. pool_fee.json gains three cases where a 1-bps fee on a 9999-sat coinbase floors to zero. With a 1000-sat fixed floor, 2.x.x records the pool-fee account at 0 sats and emits no fee output, under both canonical and pool-fee-first ordering; with the day-one floor the coinbase is below 14720 and the policy fails. 3.x.x reproduces all three. The exporter's implemented_at citations now point at the lines that implement each rule (coordinator.rs:747 and 759,773; job_bundle.py:3400; accepted_preview_telemetry.py:144). The D2c reason labels build_collection_bundle as 2.x.x code, and D2b states the confirmation timing assumption behind its next-block payout. A real 1160-recipient TRUC chunk is left as a documented gap: it would add about 1.6 MB. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Note the 2.x.x share-weight settings 3.x.x does not carry (#269) 2.x.x read PRISM_STRATUM_SHARE_WEIGHT and PRISM_STRATUM_SHARE_WEIGHTS_JSON. A per-worker entry in the JSON object credited max(1, override) instead of the share target's difficulty; the default weight only filled the job's delivered share_weight field. 3.x.x supports neither. With the 2.x.x defaults crediting is unchanged, so this is not a D2 payout difference, but a deployment that set either variable must remove it before cutover. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* Add a native SQL oracle for the payout-window read (#269) Item 3 of #269 asks for an independent proof that `Ledger::snapshot` returns exactly the eligible payout window. The 2.x.x oracle was a Python module written against the old per-row read; 3.x.x pages the window backwards through a different query, so this is a new oracle written natively in Rust. Each test gets a throwaway schema, drives the read through the public `Ledger` API only, and recomputes the expected window in one unpaged statement: an eligibility filter, a running-sum window function standing in for the subtract-until-zero loop, and a single `json_agg`. The oracle shares no SQL text, constants or private items with `ledger.rs`, so it survives the pending split of that module. It then compares membership, order and every field of every share, plus the captured cutoff. The anchor barrier is expressed with exact integer millisecond arithmetic rather than the read's `to_timestamp(double precision/1000)`, so the two sides are genuinely independent. Scenarios cover the 4096-row page boundary from either side and three pages deep, the row that straddles the remaining weight, rows sitting exactly on the anchor and one millisecond past it on each timestamp, rejected rows interleaved with accepted ones, difficulties at the top of the `u128` range, anchors ending in 999/000/001 near 2^41 and 2^43 milliseconds, and empty windows. Each read is given a ceiling so a cursor that stops advancing fails with a named error instead of hanging the run. The database URL comes from `PRISM_TEST_DATABASE_URL`. One guard decides the outcome: run when it is set, fail when it is missing while `PRISM_TEST_REQUIRE_INTEGRATION=1` or `GITHUB_JOB=prism-native-postgres` demands the integration suite, and otherwise print a skip line. Keying on `CI` would break `rust-tests`, which runs the whole workspace with no database. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Derive oracle anchors from the clock and assert the append path at the anchor (#269) Two follow-ups to the payout-window oracle. The suite pinned fixed anchors in 2030 and 2039. Because a pinned anchor has to sit above the wall clock, those were dated fuses: from January 2030 every test would have failed in CI for no reason in the code. Anchors at an ordinary magnitude are now derived from the database clock at run time -- a day of margin, rounded up to a round millisecond, plus the scenario's own `999`, `000` or `001` suffix -- so the millisecond coverage is unchanged and the date is gone. Fixed anchors are kept only for the far-future magnitude cases, and the 2^41 anchors move to 2^42. The second change turns a comment into an assertion. `Ledger::append` stamps `accepted_at` with `GREATEST(ledger_clock_ms, floor(now))` and requires `job_issued_at_ms <= accepted_at_ms`, so pinning the ledger clock lets the public API alone place a real share exactly on an anchor and another a millisecond past it. Every anchor in the barrier scenarios now does that, and asserts the first is credited and the second is not. Doing so surfaced why those two changes belong together. Past roughly 2^43 epoch-milliseconds a `double precision` no longer resolves a microsecond, so the read's `to_timestamp(double precision/1000)` barrier lands up to a microsecond either side of the true millisecond. A fixture storing the exact millisecond and a share written through the writer's own conversion then fall on opposite sides, and no single barrier makes both agree. The exact-time oracle therefore runs only where the conversion is exact -- the derived anchors and 2^42 -- with a runtime guard that refuses an anchor which is not, so the fixture cannot silently pick one. Behaviour past 2^43 gets its own test, in the integer milliseconds `AcceptedShare` carries and with no oracle and no direct inserts: three shares per anchor, all through `Ledger::append`. Membership follows the contract at every anchor. The read-back does not: `share_from_row` floors, so a millisecond whose conversion lands low reads back one millisecond early -- 480000 of every 2000001 milliseconds measured near 2^43, none near 2^42 or at any realistic date. Rather than skip that check, the test pins it from both sides: a millisecond converting at or above itself must round-trip unchanged, one converting below must be early by exactly one millisecond and no more. Nothing known-wrong is asserted as right, and a regression in either direction fails. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Bound the far-future read-back drift instead of pinning it (#269) `writer_rows_follow_integer_milliseconds_past_2_43` required the read-back drift to be exactly one millisecond wherever the stored timestamp converts below its own millisecond. That asserted a known defect as expected behaviour: the day `share_from_row` rounds instead of flooring, the drift becomes zero and the test would have failed the fix. The branch now bounds the drift rather than requiring it -- at most one millisecond early, never late -- and pushes to the printed `read_back_early` list only when a millisecond actually drifted. The defect stays visible in the per-scenario output, a silent change of behaviour is still obvious, and a fix passes instead of failing. The other side of the classification is unchanged: a millisecond the conversion represents at or above itself must still round-trip unchanged. Membership is still asserted exactly at every anchor, and both timestamps must still drift alike. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Tolerate the read-back convention for off-grid fixture rows and tidy the oracle (#269) The oracle held the read's millisecond fields to its own `floor(extract(epoch …)*1000)`. Every timestamp a writer can produce lands on a whole millisecond, so that was exact for real rows -- but scenarios c and e deliberately insert rows 400 microseconds either side of the anchor, to prove the barrier compares instants rather than milliseconds, and `AcceptedShare` cannot say which side of a millisecond such a row fell on. Flooring one is no more correct than rounding it, yet the oracle demanded the floor, so the separately-reported `share_from_row` rounding fix would have failed `c:anchor-barrier` and every `e:` sub-case on `accepted_at_ms` and `job_issued_at_ms`. The oracle now reports, per share and per timestamp, whether the stored value lies on a whole millisecond. On the grid the read-back must still be exact, which covers every row the writer can write. Off the grid the read may return the floor or the floor plus one and nothing else, so a read that lost a whole millisecond or gained two still fails. Each timestamp is judged on its own flag, so a read that treated the two fields differently is still caught, and membership, order and every other field stay exact. Every test now drops its throwaway schema on the failing path too, through a helper that runs the body and then always closes the ledger and drops the schema. The body's error is what a failing test reports; a cleanup failure is attached to it as context rather than replacing it, so a leaked schema can never hide the assertion that actually failed. That covers the per-sub-case schemas in `check_anchor_boundary` and in the 2^43 loop, which previously leaked one schema per sub-case whenever a run went red. Three comments were also wrong. The module header claimed the oracle shares no SQL text with `ledger.rs`, but `oracle_cutoff` repeats the read's cutoff query and `compare_conversion` embeds the read's own `to_timestamp`; both are deliberate and the header now says which, and why each is acceptable. The guard said the repository sets both "required" signals, when it sets only `PRISM_TEST_DATABASE_URL` -- GitHub sets `GITHUB_JOB`, and nothing sets `PRISM_TEST_REQUIRE_INTEGRATION`, the opt-in switch proposed by #286. And the 2^43 measurement spans 2000001 milliseconds, about 33 minutes, not two seconds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Three CI jobs install Ubuntu packages with a bare `apt-get update` and `apt-get install`. On 2026-09-11 the Ubuntu archive refused connections and served hash-sum mismatches to the runners. Each of those jobs failed with exit 100 before any test ran. Add `.github/scripts/apt-install.sh` and use it in all three jobs. apt retries each download (Acquire::Retries=5, with 30 s timeouts). If the update-and-install still fails, the helper clears the package lists, waits 30 s, then 60 s, then 90 s, and tries again, for up to four attempts. Every attempt but the last treats a partially failed `apt-get update` as a failure, instead of installing from stale indexes. The last keeps apt's default leniency, so the step is never stricter than before. Each failed attempt is annotated as a warning. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Decision D2 in #260 was approved on 2026-09-11: keep the 3.x.x behaviour for D2a (bootstrap pooling), D2b (below-target block credit) and D2c (prior balances during bootstrap). Mark D2c as approved, and point the configuration note at the operator release notes. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(metrics): add native cutover registry and producer hooks * fix(metrics): enforce registry ownership and live observation contracts * docs(metrics): align operator guidance and collector validation * test(metrics): pass registry to merged admission fixture * test(metrics): reset whole socket before failed-write assertions * fix(metrics): validate listeners before coordinator startup * fix(metrics): bound progress guard to health publication * fix(metrics): render registry before first publication
#312 made the three CI apt steps retry, which rides out short blips. An outage like the one on 2026-09-11, about 35 minutes of refused connections and hash-sum mismatches from Canonical's archive, still fails them. - Pin fallback mirrors in .github/apt/ubuntu-mirrors.txt: Azure's cloud mirror, kernel.org and Princeton, with Canonical's archive last. After the first failed attempt, apt-install.sh replaces the runner's Ubuntu sources with that list. Priority tags make apt try the mirrors in order, and retries skip apt's growing delay so a refused mirror doesn't hold up every download. Healthy runs keep the runner's own sources. - Cache the downloaded packages. The new composite action .github/actions/apt-install restores and saves them with actions/cache, keyed on the package set and the month. The helper downloads into its own directory, keeps exactly the packages a successful install used, seeds the next install from them, and if every online attempt fails, installs them directly with dpkg. It never reinstalls or downgrades anything the runner already has. - Stop retrying after a deadline (480 s by default), so the cache still gets its turn well inside the jobs' timeouts. The three CI apt steps now use the composite action. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…#298) * 3.x.x: ratchet the payout window against the PostgreSQL JSONB ceiling (#264) At the production window the native server stores the whole share window inside single JSONB documents. PostgreSQL refuses any container whose elements exceed 268,435,455 bytes. CI never saw this, because the largest native fixture had 800 shares. Add a deterministic production-shaped window fixture, an in-process fake node, and a gate that drives refresh, enqueue, claim, landing and import against PostgreSQL 16. After each phase the gate measures every JSONB column, and it fits bytes per share at two reduced sizes to project to 400,000 shares. Four known violations are checked in as a ratchet. The gate fails on a new crossing and on a listed write that stops crossing, and it never skips when CI is set without a database. It runs in the prism-native-postgres job; the database-free rust-tests job skips it by name. The full-size run and the baseline sweep are #[ignore]d and documented. test/prism-native-tests.sh gains a cargo-args mode, and the measurement and capacity docs record the baselines. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * 3.x.x: attribute refused JSONB writes and check the sweep's fit stability (#264) Two accepted review findings on the JSONB ceiling gate, both in test code and the measurement document. A refused write was attributed to a hard-coded column. PostgreSQL's ceiling error names neither the table nor the column, and the server adds no context naming the write, so each of the four `ceiling_rejection` sites was recording an assumption: a phase that ever wrote a second oversized JSONB value would have had the refusal blamed on the expected column, and the new offender would never have been seen. Every refusal now goes through `attribute_refusal`, which takes the refused phase, the refused share count and what that phase's accepted measurements project for each of its columns there. Exactly one column must project past `ATTRIBUTION_FLOOR`, 90% of the hard limit - seven times the 1.4% the measured fits disagree by - and it must be the write the call site expects; no candidate, two candidates, or the wrong one fails the gate with both names. The full-size run and the baseline sweep now run the reduced pair first for those projections; the reduced run attributes a refusal at `n2` from the `n1` measurement scaled to it, and a refusal at `n1` fails outright, since nothing smaller was accepted to attribute it from. The ratchet row says how it was attributed and still never shows a byte count. The baseline sweep never checked fit stability. Its comment claimed the CI-pair projection had to agree with the large-pair one, but it only computed the large pair, and with the default sizes refresh is refused there and has no projection at all. The sweep now also runs the reduced pair for the CI projection, skipping a size its list already has, and compares the two per write: the large pair is the two largest sweep sizes at which that write was accepted, since the writes are refused at different sizes and no single pair fits them all. A disagreement over `FIT_STABILITY_TOLERANCE` fails the sweep, naming the write, both projections and both pairs; a write accepted at fewer than two sweep sizes fails rather than going unchecked; a CI projection below 1 MiB is printed as not compared, because relative noise on a write that small says nothing about the window's growth. The table has the columns of the document's own. Unit tests cover attribution (unique, ambiguous, unexplained, another phase's column, and a candidate that is not the expected key) and fit stability (agreement, a 10% divergence, pair selection skipping refused sizes, fewer than two accepted sizes, and the 1 MiB floor), all without a database. Verified on PostgreSQL 16: the reduced gate ends with the ratchet holding at exactly the four known violations, and the sweep at 10000,20000,40000 prints the comparison table with every compared write inside 0.22%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * 3.x.x: keep the n1-scaled projection for a write refused at n2 (#264) A write PostgreSQL refuses at n2 gets a Rejected verdict in the reduced fit, so projections_at gave it no projection. With PRISM_JSONB_GATE_N2 equal to the target, the full-size run then met the same refusal with nothing to attribute it from, and failed as unexplained although the write is a known violation. The sweep had the same gap at larger sizes. reduced_projections keeps the projection that attributed the n2 refusal, the n1 measurement scaled to the size in question, for every write the reduced pair refused. The full-size run and the sweep now use it. A unit test covers the refused write, the fitted write it leaves alone, and the attribution at the target. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * 3.x.x: attribute refusals before any table names their column (#264) The full-size run, the reduced run's n2 half and the baseline sweep printed their measurement tables before attribute_refusals ran, and write_table_lines named each refusal's table and column as the call site recorded it. A moved or new oversized write would have been shown under the old column before the attribution check disputed it. Every path now attributes first and still prints its table whatever the outcome. write_table_lines shows an unattributed refusal by phase only, as (refused) (not attributed). attribute_refusals no longer empties the map before every refusal has attributed, so a failure keeps the refusals in the report. The sweep attributes through a new attribute_sweep before printing any size. A unit test covers both rows. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * 3.x.x: fail on an unreadable database URL instead of skipping the gate (#264) database_url read PRISM_TEST_DATABASE_URL and CI with .ok(), which turns VarError::NotUnicode into "unset". Outside CI, a deliberately configured but malformed URL therefore skipped the ratchet silently, and a non-Unicode CI value counted as unset too. Both reads now go through database_url_from, where an unreadable variable is an error, as every other gate setting already is. The existing rule is unchanged: with CI set and no URL, the gate fails. A unit test covers a set URL, unset and blank values, CI without a URL, and non-UTF-8 values of both variables on Unix. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…315) * 3.x.x: set one 3.0.0 release identity across VERSION and every crate After #283 the tree carried three version strings on the 2.0.x line: VERSION and the two library crates read 2.0.2 by inheritance from the 2.x.x merge, and the rewritten server still read 2.0.0 from #244. None of them named a 3.x.x release. Declare the version once, as [workspace.package].version = "3.0.0" in the root Cargo.toml, and have every crate inherit it with version.workspace = true. VERSION reads 3.0.0 to match. Cargo.lock only moves the three workspace-member entries; no third-party dependency changes, and `cargo metadata --locked` and `cargo check --workspace --locked` pass. The toolchain pin and builder-image digest stay with #286; the 3.0.0 release notes stay with #291. Part of #284. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YRVX7BZXPMWRNnbY9QLm1H * 3.x.x: gate the release identity with a version-skew check scripts/check_version_skew.py asserts that VERSION equals the version Cargo resolves for every workspace crate. It reads crate versions from `cargo metadata --no-deps` rather than the manifests, so a crate that later drops version.workspace = true for its own literal is still caught. It names each mismatching crate with both versions and exits non-zero on any mismatch, on a missing, empty, or malformed VERSION, and when cargo itself cannot describe the workspace. CI runs it as the first step of the rust-tests job, which already has a Rust toolchain and is in the required checks aggregator's needs list, so skew fails the merge gate without a new job. `make check-version-skew` runs the same command locally. tests/test_check_version_skew.py drives the script against temporary workspaces: matching versions pass; VERSION behind the workspace names every crate; one crate with a literal version is named alone; empty, missing, and malformed VERSION files fail; an unusable cargo is reported rather than passing. It also asserts the checked-in tree passes, that the CI step lives in a job the gate needs, and that the make target exists. The 3.x.x development-line note records the single source and points the real release notes at #291. Part of #284. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YRVX7BZXPMWRNnbY9QLm1H --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* Cover the D2b below-target block credit against PostgreSQL Decision D2 on #260 keeps 3.x.x's payout behaviour, and entry d2b-below-target-credit of the migration guide is the one that changes a miner's money: a proof that meets the network target but misses its assigned share target is credited at network difficulty, and only once the block it solved is observed on the active chain. 2.x.x credited the assigned share difficulty the moment the node accepted the block. Until now the only in-tree coverage of that shape ran against an in-memory stratum backend, which can see the frame being forwarded but neither the credited amount nor when it lands. Add coordinator::d2_below_target_tests, which drives the real Coordinator against a throwaway PostgreSQL schema and a scripted qbit JSON-RPC node, one test per case of the frozen below_target_credit vectors. Each test seeds the vector's ledger window, builds the found-block work, mines a proof of the shape the case names, and starts a real submission, then asserts: * the credit is published only inside the confirmation transaction. While the candidate is pending there is no credit row, exactly one deferred share, and no acknowledgement to the miner; * the credited difficulty is the proven network work and not the 2.x.x assigned difficulty, read from both sides of the vector rather than written here; * a lost submitblock reply ends in exactly one credit, published by reconciliation, and repeating that observation does not credit twice; * a reorg after publication leaves the credit intact while the block goes inactive; * a block the node refuses is abandoned, credits nothing, and fails the submission with reason ledger-confirmation-failed, with the internal reason the miner never sees captured from the log; * a share-passing proof is still credited its assigned difficulty at once, with nothing deferred; * the block after the solved one pays exactly what the vector projects, compared leaf by leaf, which is where a wrong credit would become a wrong payout. The assigned share target is reached through the stratum difficulty floor, because reassign clamps a target easier than the network target back to the network target and the floor is the only way to get a share target harder than the block target. The two difficulty units differ, so the floor is converted from the vector's scaled unit and stepped up until it is at least as hard as the vector's target; no f64 divides to that exact 256-bit value. The tests carry the same integration guard as the window read oracle: they run when PRISM_TEST_DATABASE_URL is set, fail by name when PRISM_TEST_REQUIRE_INTEGRATION=1 or GITHUB_JOB=prism-native-postgres demand the suite, and otherwise print a skip line, so a database outage in that job cannot pass silently. Each test takes its own schema, serialises against the cluster-wide advisory locks the other coordinator tests share, bounds every wait, and drops its schema and aborts its tasks on the error paths too. Also point the stratum protocol test at this module, so the boundary between the two is written down where a reader meets it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Pin the D2a bootstrap-pooling selection with coordinator tests Decision D2 on #260 keeps 3.x.x's payout behaviour during pool bootstrap, and nothing yet fails if that behaviour drifts back towards 2.x.x. This adds an in-crate test module that drives Coordinator::refresh_once against a fake qbit JSON-RPC node and asserts on the work it actually publishes. What is asserted, and why it is asserted this way: * D2a, bootstrap pooling. With two accepted shares from two miners -- below 2.x.x's readiness gate of three, above 3.x.x's rule of "any share at all" -- refresh_once must select the ledger window, and the payout must be the proportional split the bootstrap_transition vectors record for 3.x.x, and must not be the 2.x.x whole-coinbase-to-the-finder outcome. Restoring any readiness gate of three or more fails this test. * The one-share boundary the vectors do not cover: a single share from a miner who is not the solver takes the whole coinbase. A restored gate of two or more fails this test. * An empty ledger and no balances still pays the solver, through the one synthetic bootstrap-share the bootstrap path builds. Every expectation is read out of the vector file by case name, checked against the D2 anchor the case carries, so a renamed, re-anchored or weakened case fails here rather than passing quietly. The assertions read the Prepared work refresh_once published and the bundle build_job hands a solver; build_bundle is never called directly, because that would compute the answer the selection under test was supposed to make. The harness template's network difficulty is 1_000_000 rather than the vectors' 100, so payouts that turn on the synthetic share are compared in units of each side's own network difficulty, with exact divisibility asserted rather than rounded away. Seeded-share payouts are compared exactly: the 8x window counts every seeded share at either scale. The integration guard is the one from the payout-window oracle: it runs when PRISM_TEST_DATABASE_URL is set, fails and names the variable under PRISM_TEST_REQUIRE_INTEGRATION=1 or GITHUB_JOB=prism-native-postgres, and otherwise prints a skip line. Keying on CI would be wrong, because the no-database job sets it too. The D2c carry-only case is not covered yet: an empty payout window alongside a non-zero carry-forward balance is not reachable from 3.x.x's own write paths, and how to reach it is still being decided. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Pin D2c: a bootstrap block still pays carried-forward balances Decision D2 on #260 keeps carried-forward balances in the payout of a bootstrap block, where 2.x.x dropped them from its collection bundle and left the account unpaid. Nothing failed if that regressed. This adds the carry-only case from the bootstrap_transition vectors. With an empty share window and one carry-only account whose balance is above the day-one floor, refresh_once must publish no window bundle, the bundle build_job hands the solver must carry the ledger's prior balances, and its payout policy manifest must match the recorded 3.x.x outcome exactly -- the solver paid the coinbase less the carried account's settlement, and the carried account paid out of this block. It must also differ from the 2.x.x outcome, which pays the solver everything. Passing an empty prior-balance list on the bootstrap path fails this test and no other. Reaching that state takes an explanation, because 3.x.x cannot produce it. An empty window beside a non-zero carry is the migrated, or archived, state of a pool that inherited balances without the share history behind them, and every route to it from a running 3.x.x is closed: a block that accrues must carry real ledger shares, because the landing path re-reads the bundle's share range out of the ledger and requires a match; a bootstrap-shaped block cannot accrue at all, because its single entitlement always takes the whole miner reward and so always clears the floor that was just checked against it; and shares can never be removed afterwards, because the share ledger refuses every update, delete and truncate at the database level, and one accepted row is enough to keep the window non-empty forever. So the test records the carrying parent block's rows directly, and does it without inventing a single figure. The parent's payout is built by the engine and verified through the public audit verifier, the test asserts the engine accrued exactly the balance the vector expects before anything is written, and then only that manifest's own accounts are recorded -- through the same statements, the same account-type filter and the same order the landing path uses, the block row prepared first and flipped to confirmed separately so the summary triggers count its carry once. Nothing writes a summary table by hand. A schema change that moves the real landing path leaves this copy failing loudly rather than quietly seeding some other state. The fake node's tip at that height is that parent block, so reconciliation sees it on the active chain, and the block under test is its child. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Share the D2 test harness and apply the review findings Both decision-D2 coordinator test modules carried their own copy of the same harness. Move it into coordinator::d2_test_support: the integration guard, a throwaway schema that is removed on success, on failure and when a fixture fails to open, the coordinator configuration both run, the payout projection with its leaf-level diff, and the shared constants. Also apply what review of the two modules found: * the bootstrap tests' fake node served a fixed template time, so every one of them would have started failing as stale in January 2027; it now serves the wall clock, as the below-target node already did; * each test now reports its own failure ahead of any cleanup failure, instead of letting a cleanup error replace it; * the below-target fixture now drops its schema when setup fails; * assertions that could no longer fail once the exact 3.x.x match had passed now check that the vector itself still records a difference from 2.x.x, which is what they were meant to guard; * the carry-seeding comment now says exactly which landing statements it repeats verbatim and which parts of landing it leaves out. No production code changes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…iene jobs (#322) * Fix every clippy warning without changing behaviour `cargo clippy --locked --workspace --all-targets -- -D warnings` now passes on the 1.98.1 toolchain. Each change keeps the code's behaviour: - `(n + 1) / 2` becomes `n.div_ceil(2)` in both Merkle root loops. - The two `Default` impls that picked one variant become `#[derive(Default)]` with `#[default]` on that variant. - The eligible-share sort keys on `Reverse(share_seq)` instead of a comparator that spelled the same order. - `&[x.clone()]` becomes `slice::from_ref(&x)` in the CTV manifest test. - `seq % 4 == 0` becomes `seq.is_multiple_of(4)` in the window fixture. - `chunks_exact(_mut)` with a constant size becomes `as_chunks(_mut)`, which ignores the same remainder. - The adversarial settlement case that looped over four blocks and broke after the first now states that it checks block 0 only, which is what it always did. - `tools.rs` keeps its `live_instance_tests` module after the last item. - The two owning audit-bundle entry points allow `too_many_arguments` with the reason that they mirror the borrowed forms they forward to. `cargo fmt --all --check` already passed on the base commit, so there is no formatting-only change to land. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DzhiDBXX9hveszStKkyFt1 * Route every environment-gated test through one shared gate Every test whose execution depends on PRISM_TEST_DATABASE_URL, PRISM_TEST_PG_BIN_DIR or QBITD_BIN used to open with its own copy of a skip: read the variable, print a line libtest hides, return Ok. The new qbit-prism-test-gate crate holds the one decision table that window_read_oracle.rs already followed, generalised per input: present, run; missing while PRISM_TEST_REQUIRE_INTEGRATION=1 or GITHUB_JOB=prism-native-postgres, fail naming the variable and the test; missing otherwise, print one line with a fixed prefix past libtest's capture and return. A variable that is not valid Unicode is an error, never absent, and a switch value other than 1 or 0 is an error too. The table is a pure function over injected values with a unit test per row, and a child-process test drives the runtime path. Every site is converted: the ledger, 2.x upgrade, rollups, API, readiness, authorization, migration-rollback, failover, replica, live regtest, candidate-lease, public-service and live-instance tests, and the window oracle, which drops its private guard. The two explicitly selected #[ignore] tests use the gate's required entry point, which never skips. The JSONB ceiling gate moves off its CI keying (GitHub sets CI=true in rust-tests, which has no database), so rust-tests no longer skips it by name; its unreadable-variable test now exercises the shared table. The live fixture keeps the qbitd path it was opened with for node restarts. The crate is a dev-dependency of qbit-prism-server and qbit-prism, so a gated test under either crate's tests, or in a #[cfg(test)] module, uses the same gate. scripts/check_gate_env_reads.py, run in the lint job, fails when any Rust file outside the gate crate names one of the gate variables in a string literal, so no new test can read them directly. Measured gated tests: 74 in the tree, 72 of which the native job runs (the two JSONB #[ignore] measurement runs are gated but never selected in CI). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DzhiDBXX9hveszStKkyFt1 * Prove in CI that every gated test executed With PRISM_TEST_GATE_MANIFEST set, the gate appends one line per decision, executed, skipped or failed, followed by <package>::<binary>::<test path>, in a single O_APPEND write so parallel test threads and separate test binaries share the file. test/prism-gated-tests.txt is the checked-in list of the 72 gated tests the native job must execute, including the two #[ignore] runs it selects explicitly; its length is the minimum. scripts/check_gate_manifest.py fails when any expected test did not execute, any skipped or failed line appears, fewer distinct tests executed than the list holds, a test executed that the list does not name, or the log shows the gate's skip prefix or the wording of the skip messages the gate replaced. It prints the manifest to the log and to the step summary. The prism-native-postgres job keeps its id (the gate keys on it), sets PRISM_TEST_REQUIRE_INTEGRATION=1 and the manifest path, runs the whole workspace with --all-targets and --nocapture through tee, then the two explicit --ignored runs, then the proof, and uploads the manifest as an artifact. A gated test added under crates/qbit-prism is therefore covered by a required job. test/prism-native-tests.sh mirrors the job: its default mode runs the workspace, both explicit runs and the proof, and exports the switch once the PostgreSQL server tools and qbitd are present; the live, replica and cargo-args modes always export it, since they refuse to start without their inputs or name their own tests. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DzhiDBXX9hveszStKkyFt1 * Add fmt, clippy, cargo-deny and hadolint as required checks Four new jobs join the checks aggregator: cargo fmt --all --check, cargo clippy --locked --workspace --all-targets -- -D warnings, cargo deny --locked check, and hadolint over every Dockerfile. deny.toml checks advisories (nothing ignored), licenses (an allow-list of what the tree uses, plus CDLA-Permissive-2.0 for the CA bundle that webpki-roots ships), bans (duplicate versions warn; wildcard requirements are refused except the workspace's own path dependencies, which needs qbit-prism and qbit-prism-server to be publish = false, as they are in practice) and sources (crates.io only). cargo-deny 0.20.2 and hadolint 2.15.1 are installed by .github/scripts/install-pinned-tool.sh from their GitHub releases after a sha256 check. hadolint reads .hadolint.yaml, which ignores nothing globally: lab/prism/Dockerfile pins its two apt packages, and the other images, whose behaviour is not this workstream's to change, carry inline ignores with reasons (unpinned apt versions, the named qbit-src build context that DL3022 mistakes for a stage, the deliberate separate cmake layers, and the checked pipes in the Bitcoin Core download). .github/actionlint.yaml names the Blacksmith runner label so actionlint runs clean locally. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DzhiDBXX9hveszStKkyFt1 * Pin the Rust toolchain to 1.98.1 in one place rust-toolchain.toml pins the exact stable version with rustfmt and clippy under the minimal profile. [workspace.package].rust-version carries the same version and every crate inherits it. The lab/prism/Dockerfile builder is rust:1.98.1-bookworm at the multi-arch index digest Docker Hub serves for that tag (resolved through the registry API; both the 1.98.1-bookworm and 1.98-bookworm tags point at it today). scripts/check_toolchain_pin.py fails when the three disagree, when a crate stops inheriting rust-version, when the builder line lacks a digest, or, with --installed, when the active rustc or cargo is not the pin. The lint job runs the static check; the new .github/actions/rust-toolchain composite action replaces the floating dtolnay/rust-toolchain@stable in every Rust-using job, installing the toolchain rustup reads from the file and asserting the installed version. docker-builds builds the builder stage on its own first and asserts that its cargo and rustc report the pin. VERSION and scripts/check_version_skew.py from #315 are unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DzhiDBXX9hveszStKkyFt1 * Keep one make target per native test suite test-prism-postgres-ledger, test-prism-postgres-scale, test-prism-postgres-native-ledger and test-prism-postgres-seed-guard all ran test/prism-native-tests.sh; they become test-prism-postgres. test-prism-stratum-regtest-live, test-prism-stratum-postgres-regtest-live and test-prism-combined-regtest all ran the live mode that test-prism-regtest already names; they are removed. test-prism-public-read-replica keeps its name and calls the script's replica mode directly. The eight wrapper scripts under test/ that only exec'd prism-native-tests.sh are deleted; test-prism-postgres-throughput and its script are untouched (#271 owns the throughput floor). README, the coordinator-refactor validation notes and the overload alert spec now name the surviving targets; the historical release notes under doc/ keep the names they shipped with. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DzhiDBXX9hveszStKkyFt1 * Document the integration test gate and the deleted-test map docs/prism-integration-test-gate.md is the gate contract: the inputs, the decision table, the switch, the skip prefix, test identity, the manifest, the expected list and the proof, local runs, and how to add a gated test. docs/prism-deleted-test-map.md rebuilds the readiness review's map from git: every test file under tests/ and test/ on origin/2.x.x (504846c) absent on 3.x.x, 142 files grouped by area, each with its native replacement, or the open issue that closes the gap, or a needs-triage note where neither exists. Both are linked from docs/README.md, and the wrapper descriptions in PRISM.md, the server README and the ledger operations guide now describe required mode and the manifest. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DzhiDBXX9hveszStKkyFt1 * Route the D2 payout tests through the shared gate #316 landed nine PostgreSQL-gated coordinator tests with a hand-written copy of the old guard in d2_test_support.rs, which read PRISM_TEST_DATABASE_URL, PRISM_TEST_REQUIRE_INTEGRATION and GITHUB_JOB directly and so failed scripts/check_gate_env_reads.py. The guard now takes the URL through qbit_prism_test_gate like every other #[cfg(test)] site, the nine callers drop the test name the gate derives itself, and the tests join test/prism-gated-tests.txt, which the native job's manifest is checked against: 81 expected gated tests, 83 in the tree. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DzhiDBXX9hveszStKkyFt1 * Require every explicit #[ignore] run and write manifest lines in one write The two JSONB measurement runs, jsonb_ceiling_ratchet_at_full_size and jsonb_ceiling_baseline_sweep, are #[ignore] tests someone asks for by name, yet they reached the gate through the non-required entry point, so a manual --ignored run with no database printed a skip line and reported ok. They now use the gate's required entry point, as the observability_database and stratum_admission_postgres runs already did, and the contract says that every explicit #[ignore] run is required. They stay out of test/prism-gated-tests.txt because CI never selects them. The gate's record_to used write_all, which can split a line across two writes after a short write and let another thread's line land between them. It now hands the whole line to one write and treats a short write as an error, so each manifest line stays a single O_APPEND write; the skip line on stderr goes out the same way. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DzhiDBXX9hveszStKkyFt1 * Leave the runtime image's apt packages unpinned The PRISM runtime stage floats on debian:bookworm-slim, and Debian drops superseded package versions from its archive, so the exact ca-certificates and curl pins would have broken the build at the next point or security release, with no Docker here to build-test a bump. The stage carries the same inline DL3008 ignore and reason as the other images; pinning its base and packages belongs to #288. The builder stage keeps its digest pin. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DzhiDBXX9hveszStKkyFt1 * Re-check the deleted-test map against the D2 payout tests The map is now built against 8dccc74, the merge of #316, and pins both ends of its file list (504846c and 8dccc74), since origin/2.x.x has moved on and this branch removes wrapper scripts of its own. The nine D2 tests were read against every row they could touch: ten partial rows now cite them (below-target credit, lost-reply reconciliation, reorg survival, bootstrap selection, the retired readiness gate), and the settlement balance comparison row states why D2c does not cover it. No status changes; the counts stay 5 full, 67 partial, 8 open gaps, 7 needs triage, 55 retired. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DzhiDBXX9hveszStKkyFt1 --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
… semantics (#313) * fix: preserve PRISM issued-work and stale-grace semantics Separate observed and published tip authority, retain original issued worker context through replacement and reconnects, and preserve bounded delivery-based grace without renewing expiry after awaited work. Exercise real Coordinator decisions and socket interleavings, including constant node-RPC load across 1, 32 and 128 clients. Refs #280 * docs: reference PRISM miner parity settings Co-Authored-By: OpenAI Codex <noreply@openai.com> * fix(prism): bind share admission to one published payout view * fix(prism): verify changed block notifications with fresh chain state * docs(prism): record B8 review regressions and rebased qualification * test(prism): adapt merged admission fixture to stale grace contract * fix(prism): classify retained capacity by current published tip * fix(prism): retain prepared dependencies for issued work * test(prism): verify issued job dependency persistence * fix(prism): keep sessions alive while retained work is usable * docs(prism): record timer coverage and final qualification procedure * docs: point D2 credit explanation to miner submit module * test: align candidate lease tip fixture with coordinator state * fix: integrate miner parity with native metrics registry * fix: integrate miner parity with metrics registry API * ci: run issued job dependency acceptance explicitly * test: keep miner parity fixture publication reachable * test: complete JSONB gate fixture config * test: restore D2 coordinator coverage * fix: bound configurable tip freshness durations * test: retain tokio time controls under pinned toolchain * chore: satisfy pinned toolchain checks * test: route issued job dependency through integration gate --------- Co-authored-by: OpenAI Codex <noreply@openai.com>
* docs: design record for WindowRef and Ledger::read_window (#264) Add docs/prism-coordinator-refactor/window-ref.md, the design record that #265 (migration 007) and #273 (migration 008) implement against, and link it from the directory README. The record specifies: - WindowRef { anchor_ms, prior_balances_digest, shares: Option<ShareRange> } - the six window_* columns and their three-state CHECK - Ledger::read_window: one REPEATABLE READ READ ONLY transaction, mandatory ascending keyset paging, a streaming digest, typed errors and never a partial window - per-caller revision fences that keep today's observe_candidate rule for claims - the non-window bundle inputs a rebuild stores instead of reading config - the upgrade procedure, retention floor and expected cost at 400k shares - the borrowed qbit-prism parts API merged in #296 It went through two adversarial reviews. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: hex-serialize WindowRef digests and clamp the resume timeout (#264) Two fixes to the WindowRef design record after review: - A derived Serialize writes a [u8; 32] as a JSON array of 32 integers, not the lowercase 64-hex string the payload and column contract requires. Both digest fields now carry #[serde(with = "hex32")], a small adapter in ledger/window.rs that writes lowercase hex and reads exactly 64 lowercase hex characters. The server's hex dependency has no serde feature. - The proposed resume timeout, initial_job_timeout_seconds minus 5 s, is zero or negative for any setting at or below 5 s, which validation allows. It is now outer - min(5 s, outer / 2), computed from the Duration the caller already builds: 25 s at the 30 s default, always positive, strictly shorter than the outer timeout, and free of Duration underflow. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: bootstrap resumes use stored policy inputs (#264) An empty-window resume rebuilt each miner's bootstrap bundle through Coordinator::build_bundle. That function reads config.ctv_enabled, payout_policy, ctv_direct_floor and ctv_config, so after a config or fingerprint reset a resumed job could get a different signed bundle under the same job ID. That is the drift the stored-inputs invariant forbids. The record now has the per-miner bootstrap rebuild call the borrowing builders directly. It takes its own build_slots permit after the single-flight entry's permit is released. Its inputs are the synthetic share build_bundle fabricates today, the template, anchor, payout_policy and ctv stored in StoredPrepared, and the balances and revision read_window returned. #273's must-not row now forbids build_bundle for any resume rebuild. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: readable imported audits and a race-free upgrade order (#264) Two fixes to the WindowRef design record after review. Imported legacy audits: #265 stores canonical_audit_bytes and no inline body. Only audit_canonical_bytes reads those bytes today. Ledger::audit_bundle returns just the JSON column, so backfill-ctv would stop on imported rows, and the bundle endpoint's fallback reads only body_uri. The record now requires the same PR to make both readers decode the stored canonical bytes first, digest-checked and under spawn_blocking. Leaving an imported audit readable only through body_uri is now forbidden. Upgrade order: a zero pending count taken while the pre-007 frontends still run proves nothing, because a block can be found before they stop. The count is now taken after every frontend is stopped. A non-zero count means restarting the pre-007 frontends to drain and repeating, and 007's refusal remains the backstop. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: one owned window per Prepared and the established lock order (#264) Two fixes to the WindowRef design record after review. Window ownership: Prepared and JobContext would have held an owned AuditBundle beside the Snapshot. into_bundle takes the share Vec by value, and two Arcs don't share storage, so every resume would keep a second window: about 0.75 GB per 400k rebuild instead of 0.5 GB. Both structs now hold Arc<Snapshot>, the only owned window, next to Arc<AuditBundleBody>. Canonical bytes and verification come from the parts functions, and an owned bundle is assembled only at claim. Lock order: the D6 prune now takes SETTLEMENT_LOCK and then ORDER_LOCK, the order every existing dual-lock site uses, so it can't form an advisory-lock cycle with a refresh snapshot or a candidate finalization. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: version the audit builder behind durable references (#264) Stored inputs reproduce a bundle only under the builder that first built it. If a release changes builder output, a pending candidate that survived the upgrade would rebuild into a different bundle than its coinbase commits to, and fail on every retry or land changed audit data. qbit-prism gains AUDIT_BUILDER_VERSION, which both the 007 candidate and 008's StoredPrepared store. A rebuild checks it before read_window: a mismatched candidate is retried with an alert and never rebuilt with the current builder, and a mismatched job is a cache miss. A release that bumps the constant ships a migration step that refuses while pending outbox rows carry another version; they drain through the 007 procedure. Keeping frozen old builders is recorded as the rejected alternative. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: pin the signing keys behind durable references (#264) The bundle embeds both public keys and their signatures, so a rebuild reproduces the original bytes only under the original keys. Rotating a key takes an operator resetting the cluster fingerprint, and a candidate still pending across that reset would be re-signed into bytes its coinbase does not commit to. Both documents now store signer_keys, the two public keys the building frontend signed with. A rebuild checks them next to the builder version, before read_window, and never re-signs a mismatch with the local keys. #265 extends configure to refuse a new fingerprint while a pending outbox row stores other keys, so the operator drains with the old keys first. Keeping retired seeds online to reproduce old signatures is recorded as the rejected alternative. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: reuse the refresh-time WindowRef at submit (#264) Submit built the candidate's reference from the prepared Snapshot, which re-hashes and re-serializes the whole 233 to 260 MB window whenever a block is found. That puts 0.7 to 1.4 s of digest CPU on the latency-critical share path, on top of the once-per-refresh cost the cost table budgets. refresh_once already computes the identical reference, so Prepared now keeps it as window: WindowRef. Submit clones that small value into the candidate, a resume takes it from StoredPrepared, and re-digesting at submit is a must-not for #265. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: sort rebuilt prior balances canonically (#264) prior_balances_digest sorts internally, but AuditBundle.prior_balances serializes in vector order. read_prior_balances has no outer ORDER BY, and SQL doesn't guarantee the function's internal order through a table expression. Balances returned in a different order at claim or resume would pass the digest check and still change the canonical bundle. read_prior_balances now sorts with the digest's comparator (order_key, recipient_id, p2mr_program_hex, byte order). Refresh and every rebuild share that function, so the builder gets the same vector. A SQL ORDER BY alone would follow the database collation, not the digest's byte order. A #265 unit test requires permutations to give identical bytes. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: finish recovered claims from landed digest columns (#264) A claim that finds its block's audit already landed finished through materialize_audit_row. That function reads the whole range unpaged, serializes and clones it on the async runtime, and builds an owned AuditBundle: two share arrays at 400k on the submit loop, which the cooperative 60 s deadline cannot interrupt. The recovery branch now checks the landed row's columns alone, as today's landing does for an existing audit. share_snapshot_sha256 must equal window_snapshot_sha256, found_block_bits must match the header, and the claim then finishes. materialize_audit_row on the submit loop is a #265 must-not. If that path ever needs the body, #267's paged, parts-based reader becomes a prerequisite. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: re-pin the WindowRef record to 3.x.x after #283 (#264) #283 landed through #300: the 2.x.x tip merge and the split of ledger.rs into candidates, jobs, window and connect submodules. The record's code citations were pinned to 1398bbc, and many pointed into the old ledger.rs. Every citation now resolves at 3.x.x 41a2afd: 76 of 252 changed, and all 247 cited line numbers were checked against that tree. "Where the code lives" now describes what #300 landed. ledger/window.rs exists under workstream B and already holds share_from_row and read_prior_balances, and SELECT_SHARE stays in the module root. The only question left for Anatolie is whether read_range moves out of A-owned ledger/audit.rs. Two review fixes in the landed-audit recovery path. A retried claim authenticates the landed audit against the block it found: the landed coinbase must match block_hex, and the root of the landed commitment leaves must equal the coinbase's witness reserved value. For an empty window, the snapshot comparison uses the digest of [bootstrap_share], because window_snapshot_sha256 is NULL there. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: keep pool connections free during window rebuilds (#264) build_slots bounds concurrent rebuilds at build_workers, but that doesn't protect the database pool. PRISM_DATABASE_MAX_CONNECTIONS may be as low as 4, build_workers may exceed it, and share appends and candidate-lease heartbeats share the pool with a 15 s acquire timeout. Rebuilds of distinct jobs could hold every connection for their whole multi-page reads, and a heartbeat that fails to get a connection drops recoverable claim work. read_window now also takes a permit from a window_reads semaphore sized clamp(database_max_connections - 2, 1, build_workers). The caller takes it after build_slots and releases it before the rebuild, so two connections always stay free. Calling read_window without that permit is a must-not for #265 and #273. A dedicated pool was rejected because it would exceed the configured connection budget. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: continue to submission after recognizing a landed audit (#264) The recovery path finished a claim as soon as it authenticated a landed audit. But an inactive block lands before the lease renewal and submitblock, so an earlier claim can land and lose its lease before it submits. Finishing then would abandon a valid found block that was never submitted. A recognized landed audit now means landing is done. The claim skips read_window, the rebuild and land_candidate, and continues through today's observe, renew and submitblock flow, finishing only where today's code does: an active block, or an inactive one whose revision or parent moved. Finishing a claim only because its audit landed is a #265 must-not. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: reserve in-flight windows and scope the claim deadline (#264) Two review fixes. The retention floor covered written references only. Ledger::snapshot releases its locks when its first transaction commits, and the scan, the bundle and save_job follow with no row referencing the window yet. A prune in that gap, after a fall in difficulty, could delete rows of the window being built. D6 now adds a reservation. The snapshot's locked first transaction inserts it, the refresh fills in the window start after the scan, and save_job deletes it (publishing nothing if it expired or the rows are gone). The prune skips while any reservation is unresolved and otherwise keeps every reserved window. An age-based horizon was rejected because a window's reach in time depends on hashrate. The claim's 60 s timeout covered the whole call, including submitblock, whose RPC timeout may be up to 86,400 s. It now covers read_window plus the rebuild only, and the later steps keep their existing deadlines. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: answer djh58's four B contract points (#264) djh58's review on #273 (comment 5624891010) raised four points against the resume contract. The record now: - lets B's published-identity and lease check decide resume eligibility instead of revision equality, and rebuilds eligible work with its as-issued balances from an immutable qbit_prism_balance_snapshots row keyed by prior_balances_digest; - keeps reconstruction failures (Incomplete, digest mismatch, decode, database, expiry) distinct from missing work, under one end-to-end deadline, the caller's existing initial_job_timeout_seconds, with no inner timeout; - names both share digests, the native snapshot_sha256 and #274's canonical_digest_hex, amends #274's reuse instruction, and adds a cost model for a sliding window; - moves the template out of the prepared payload into an immutable qbit_prism_templates row, keeping #273's 1 MB criterion whole-value, and moves #265's block_hex into a bytea column. It also records the permit-ownership and coalesced-waiter rules from the same review. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: add the balance source to read_window and authenticate block bytes (#264) The resume contract needs as-issued balances, but read_window's declared API had no way to ask for them, and the read algorithm always used the current set. Implemented literally, it would have returned PriorBalancesChanged for the published work it was meant to rebuild. read_window now takes a BalanceSource. Current reads the current view as before. AsIssued reads the qbit_prism_balance_snapshots row keyed by the reference's digest, sorts it with the digest's comparator and verifies it. A missing row is the new BalanceSnapshotMissing, and a mismatching one is Decode. Claims pass Current, and resumes of published work pass AsIssued. Moving block_hex to a bytea column took it out of candidate_sha256. The candidate JSON now keeps block_sha256. Claim decode checks it and that the block header hashes to block_hash before any witness derivation, landing or submission. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: prune externalized templates and balance snapshots with their jobs (#264) Moving each full template into an immutable qbit_prism_templates row left no cleanup for those rows. Templates change on every new transaction set and at every reanchor, so each is a new, possibly multi-megabyte row, and prune_expired_jobs deletes only job rows. The balance snapshots had the same gap. prune_expired_jobs becomes one transaction under SETTLEMENT_LOCK, the lock save_job holds while it inserts the template, the balance snapshot and the job row. It deletes the expired batch, then the template and balance rows that batch referenced and no remaining job row references, in two statements, because a data-modifying CTE's NOT EXISTS would still see the deleted jobs. The job row gains an indexed template_sha256. Balance snapshots follow job rows only, since claims read the current balances. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: keep the issued revision on resume and ratchet D6's horizon (#264) Two review fixes. A resumed Snapshot now keeps StoredPrepared.payout_revision, the revision the job was issued with. read_window's current revision is only B's eligibility input and the claim's probe hint. The job wire and any candidate take their revision from the snapshot, so a block found on R0 work carries R0, and observe_candidate takes the superseded path instead of retrying PriorBalancesChanged forever. A D6 horizon computed from today's difficulty fails when difficulty rises after a prune: Ledger::snapshot stops when rows run out, so it would publish an underweight window. D6 now needs a ratcheted horizon kept at a durable maximum difficulty in qbit_prism_cluster. It must also record the share_seq it deleted below, so snapshot fails when its scan reaches that floor before the requested weight. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: drop rebuilt windows from resumed jobs once the coinbase is built (#264) build_workers bounds how many rebuilds run at once, not how many rebuilt windows stay resident. Each resumed JobContext kept its Prepared, with Arc<Snapshot> and Arc<AuditBundleBody>, and a session keeps up to 64 jobs per connection. Sequential resumes of different jobs could therefore pin about 0.5 GB each, and the stated k x 0.5 GB peak didn't hold. Once the rebuild has produced the job's coinbase, a resumed Prepared keeps only what submit reads: the WindowRef, the snapshot scalars, the template, the fee and the body's non-window fields. Submit reads nothing else, and the whole-bundle clone into the candidate goes with #265. Snapshot.shares and reward_manifest.shares are dropped before the job reaches the session, so windows exist only inside rebuilds. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: stop old signers before a reset and fence writes on the fingerprint (#264) configure's one-time pending-row check could pass while an old-key frontend was still running. That frontend could then find a block and enqueue a candidate with the old signer_keys, which no new-key frontend can rebuild, and enqueue is not fenced by the fingerprint today. A rotation now follows the 007 order: drain, stop every old-key frontend, then check that nothing is pending, reset and start the new frontends. #265 and #273 also re-read config_fingerprint FOR SHARE inside every enqueue and save_job transaction and refuse writes under a foreign fingerprint. FOR SHARE conflicts with configure's FOR UPDATE, so a write and a reset cannot interleave, and a procedure violation fails loudly instead of leaving rows nobody can rebuild. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: keep every stored input in slim resumed jobs (#264) The slim resumed Prepared kept the WindowRef, the snapshot scalars, the template, the fee and the body's non-window fields, but not the stored payout_policy and nested ctv. A candidate found on that job must persist those exact inputs, and neither the fee nor the coinbase manifest can recover them, so submit would have had to read local configuration. The slim form now keeps the decoded StoredPrepared itself: payout_policy, the whole ctv, audit_builder_version, signer_keys, coinbase_suffix, template_sha256, the WindowRef and the snapshot scalars, all O(1). For an empty window it also keeps the miner's own bootstrap_share. Only the window is dropped. Dropping a stored input before enqueue is a #273 must-not. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: let the published-work lease reach submission (#264) Keeping R0 on a resumed snapshot doesn't help if its found block never reaches observe_candidate. Today submit rejects work whose frontend or job revision differs from the ledger's, apart from parent-stale work inside the stale grace. That grace work is credited as a share, but a candidate is enqueued only when the work isn't stale, so its block is dropped. B8's lease now decides those submit checks the way it decides resume eligibility. Work the lease covers is not stale because the revision changed, and a block found on it is enqueued as a candidate carrying R0, with its as-issued window and inputs. observe_candidate then decides at claim whether it lands. Work outside the lease keeps today's handling. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: submit leased candidates first and clear terminal block bytes (#264) Two review fixes. A candidate enqueued from leased R0 work still met the unchanged claim fence, which would finish it as superseded before submitblock and discard the block. The candidate now records a leased flag. A leased candidate skips the pre-submit supersession checks and the pre-submit landing, renews its lease, calls submitblock, and only then meets observe_candidate. Whether an active leased block lands is B8's rule with #289. Moving block_hex into the block_bytes bytea column meant the terminal UPDATE no longer removed the block. It now NULLs block_bytes too, so the outbox doesn't keep every finished block forever. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: correct claim-flow line citations to 3.x.x 41a2afd (#264) The landed-audit continuation and leased-candidate paragraphs cited coordinator.rs claim-flow lines read from a later 3.x.x, where #308 had shifted them by five. They now point at the record's pin, 41a2afd: the post-rebuild observe at :1039, the active branch at :1040-1048, the inactive supersession at :1050-1063, the landing at :1067-1069, the post-land check at :1070-1080, the lease renewal and submitblock at :1084-1092, and the post-submit observe at :1095. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: submit leased candidates before any rebuild (#264) A leased candidate skipped the pre-submit fences, but the #265 claim rebuild still called read_window with BalanceSource::Current before them. For an R0 candidate that returns PriorBalancesChanged and retries forever, so the block never reached submitblock. A leased candidate now submits before any rebuild, because submitblock needs only the stored block. If the block is active, its audit rebuild uses BalanceSource::AsIssued. A non-terminal leased outbox row therefore keeps its balance snapshot. The prune checks for such rows and takes SETTLEMENT_LOCK then ORDER_LOCK, so it cannot interleave with a leased enqueue. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: release rebuilt windows off the runtime; hold the lease to unchanged balances (#264) Two review fixes. Freeing a completed rebuild's last Arc<Snapshot> and Arc<AuditBundleBody> drops about 0.5 GB on whichever Tokio worker releases it, which can stall lease heartbeats and share processing. The rebuild's spawn_blocking closure now builds the coinbase and the slim resumed form and drops the window there. Every other release of a rebuilt window goes through spawn_blocking(drop). An active leased block rebuilt with R0's balances could not land, because landing requires the current balances. Until B8 and #289 define how such a block lands, the lease covers only revision changes that leave prior_balances_digest unchanged, and a balance change ends it at once. An active leased block then lands through today's path, and the open question asks djh58 and #289 for the full rule. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: land first-time claims from parts; sweep orphaned balance snapshots (#264) Two review fixes. A first-time landing assembled the rebuilt window into an AuditBundle, then cloned it, serialized it whole for the stored body and the canonical length, and re-read the range unpaged, all on a Tokio worker. At 400k shares that is several half-gigabyte representations on the runtime. CandidateClaim now holds the parts. Landing verifies, serializes and digests them in one spawn_blocking closure before its transaction opens. It keeps the range fence, but as a paged re-read. #267 is not a prerequisite. The balance-snapshot prune only considered digests from the expired batch, so a snapshot kept alive by a leased candidate could never be reclaimed once its job had gone. Snapshots now get their own statement over every row with no remaining reference. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: keep the bootstrap share in local jobs; correct local window residency (#264) Three review fixes. An empty-window local job held its only full AcceptedShare in JobContext.bundle. Once JobContext holds a body, that share would be gone and submit could not fill Candidate.bootstrap_share. JobContext now keeps the synthetic share beside the body. The record claimed the local frontend keeps one full window. Retired local jobs keep their generation's Prepared for Stratum's retention period, as they do today. The record now says so. Bounding that is logged as a #273 acceptance note. The landing bind still copies the stored body into SQLx's argument buffer on the runtime. The record now names that copy. Measuring it is logged as a #265 acceptance note. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: keep null-response leased candidates pending; carry empty-window balances (#264) Two review fixes. The record finished an inactive leased candidate as superseded after submission. Today's wrapper treats a null submitblock result on an inactive block as unresolved, because it can still be a known side-chain block, and retries with the candidate intact. Leased candidates now keep that outcome, so a block that becomes active later still has its audit inputs. For an empty window, resume waiters build their own bootstrap bundles from the as-issued balances. The single-flight entry now carries those balances until every waiter has built, instead of only the rebuilding caller holding them. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: land a submitted leased candidate before any terminal outcome (#264) After a null submitblock result on an inactive leased block, the retry resubmitted before landing. A node can then answer "duplicate", which the string branch treats as terminal, clearing the unlanded audit inputs. Today's flow avoids that by landing an inactive block before its RPC, so a late acceptance stays reconcilable. A leased candidate still submits first, so delivery never waits on a rebuild. It then rebuilds with AsIssued and lands at the observed revision before any terminal outcome, whether or not the block is active, and only then takes today's three outcomes. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: re-establish candidate references inside the enqueue transaction (#264) Taking locks in order stops a prune and an enqueue from interleaving, but it doesn't decide which runs first. A submission that passed its expiry check could lose to a prune that deleted the job row and its balance snapshot, and then enqueue a leased candidate pointing at missing data. Enqueue now re-inserts a leased candidate's balance snapshot from the as-issued balances the job carries (the slim resumed form keeps them), and probes the window's first share row, both inside its ORDER_LOCK transaction. Shares can't be written back, so an expired job row now holds the retention floor until the prune deletes it, and the prune waits a grace past expiry. A failed probe still publishes the candidate, with an alert. Part of #261. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(prism): reserve session extranonces safely across sequence wrap * fix(prism): fence stopped session owners by process incarnation * fix(prism): drain session guards before shutdown cleanup * fix(prism): scope migration readiness to mining startup * test(prism): build complete legacy schema for read-only connect * fix(tests): route integration databases through test gate * test(prism): register all gated session coverage * fix(tests): adapt fixtures to shared session backend
…320) * docs(prism): generate metric inventory and migrate deployed alerts * fix(prism): preserve the network-wide connected client alert floor * fix(prism): evaluate standby lag from observed durable WAL progress * fix(prism): delete provisioned native rules when gates disable them * fix(prism): alert on invalid synchronous standby configuration * docs(prism): record local review evidence and prepare PR draft * fix(prism): reject stale alert and metric metadata * fix(prism): alert when public scrape fails * fix(prism): preserve native failure and paging alerts * fix(prism): preserve native alert tuning overrides * Revert "fix(prism): preserve native alert tuning overrides" This reverts commit 31b2681. * fix(prism): preserve configurable alert tuning * fix(prism): render tuning dwells as JSON * fix(prism): validate rendered alert dwell overrides
* docs(prism): pin intentional legacy health differences * docs(prism): correct legacy max blocks semantics * style(prism): format health snapshot assertions
* fix: preserve PRISM issued-work and stale-grace semantics Separate observed and published tip authority, retain original issued worker context through replacement and reconnects, and preserve bounded delivery-based grace without renewing expiry after awaited work. Exercise real Coordinator decisions and socket interleavings, including constant node-RPC load across 1, 32 and 128 clients. Refs #280 * docs: reference PRISM miner parity settings Co-Authored-By: OpenAI Codex <noreply@openai.com> * fix(prism): bind share admission to one published payout view * fix(prism): verify changed block notifications with fresh chain state * docs(prism): record B8 review regressions and rebased qualification * test(prism): adapt merged admission fixture to stale grace contract * fix(prism): classify retained capacity by current published tip * fix(prism): retain prepared dependencies for issued work * test(prism): verify issued job dependency persistence * fix(prism): keep sessions alive while retained work is usable * docs(prism): record timer coverage and final qualification procedure * docs: point D2 credit explanation to miner submit module * test: align candidate lease tip fixture with coordinator state * fix: integrate miner parity with native metrics registry * fix: integrate miner parity with metrics registry API * ci: run issued job dependency acceptance explicitly * test: keep miner parity fixture publication reachable * test: complete JSONB gate fixture config * docs: specify prepared window reference integration plan * test: restore D2 coordinator coverage * feat(prism): add window reference schema and authenticated reader Add the independently approved B273 schema and reader foundation on PR313, with typed reconstruction errors, immutable blob tables, per-version migrations, and PostgreSQL qualification. Runtime prepared/resume, candidate, atomic repair and GC integration remain gated on A265 and shared owner agreement. * feat(prism): add coherent payout state for lease eligibility * Keep cancelled window cleanup off runtime workers and qualify current base * Qualify landed 008 and 009 migration orders on current base * Preserve legacy balance order and reject fatal window reads * Prove legacy balance order preservation in PostgreSQL * Run all gated suites before manifest proof * Revert "Run all gated suites before manifest proof" This reverts commit b11cbbb. --------- Co-authored-by: OpenAI Codex <noreply@openai.com>
…m on read (#325) * 3.x.x: store imported legacy audits as canonical bytes and decode them on read The legacy-audit import wrote the parsed bundle back into the audit_bundle JSONB column. That value holds two copies of the payout window (shares and reward_manifest.shares), so at 400k shares it crosses PostgreSQL's JSONB container limit. The import now stores only canonical_audit_bytes and the non-share metadata. It keeps its idempotency and optimistic guards, verifies off the runtime, and drops the parsed window before the write transaction opens. Rows that already hold an inline body keep it unchanged, because the body-present CHECK needs one of audit_bundle and body_uri. found_block_bits stays as stored: no audit body carries the compact bits. A shared decode_canonical_audit_body helper checks the stored bytes against audit_bundle_sha256 and parses them with parse_audit_bundle_value. It returns the logical body the import used to store inline, and all of that work runs under spawn_blocking. Ledger::audit_bundle serves native snapshot rows first, then imported canonical bytes, then inline bodies, so backfill-ctv works on imported rows. The bundle endpoint serves imported bytes before any body_uri fallback, on the same public pool. A digest mismatch or undecodable bytes refuses the row, even when the legacy file still exists. The settlement-artifacts audit link now also accepts stored canonical bytes, so it does not decode a whole body just to show a link. The JSONB ceiling gate drops its import-phase known violation, and the measurement and capacity-readiness docs now count three window-carrying writes. Part of #265. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * 3.x.x: load only the served audit body and bound imported-audit decodes The bundle queries now project no inline audit_bundle for a row whose canonical bytes supersede it, the same rule Ledger::audit_bundle uses, and report whether canonical bytes exist so bundle() skips the bytes query for rows without them. The flag is removed before responding. Imported-audit decodes run after their read connection is released, so ApiState now carries a semaphore sized from the same read concurrency as the public read pool. bundle() takes an owned permit inside the request's existing deadline and moves it into the blocking decode, so a dropped request cannot free it before the decode ends. Ledger::audit_bundle takes no permit and releases its fetched row before decoding. The decode refuses body-ref and v2 envelope schemas, since canonical bytes are always a flat bundle. The storage-sizing doc describes the canonical-bytes import, and the measurement doc gains a post-#265 note. Tests pin the pre-change row shape (inline body, canonical bytes and body_uri, no snapshot), bound the decode across a dropped request, and refuse a digest-valid body-ref envelope. Part of #265. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test: add deterministic HA readiness probe harness * test: qualify readiness probe timing * test: preserve probe state in jsonl harness * test: harden readiness probe input boundaries * test: reject overlapping readiness probes * docs: use generic HA readiness contract wording * test: classify malformed probe durations as failures
#333) * ledger: let append callers veto COMMIT at the last moment append_at_revision_gated takes a non-optional pre-commit hook. It runs at most once, after every statement including persist_candidate has succeeded, while ORDER_LOCK is held and immediately before COMMIT. A false return rolls the transaction back explicitly and fails with the typed CommitGateClosed error, so COMMIT is never sent. append and append_at_revision keep their signatures and pass no hook. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * metrics: count late share confirmations and unknown outcomes A share whose commit is confirmed after the acknowledgement deadline is still accepted, and a commit whose outcome cannot be established is no longer reported as a clean rejection. Both need their own signal. Add the late_confirmed_shares_total family, recorded through record_late_confirmation and zero-registered like the other counters, and the ledger-outcome-unknown reject reason. Without the new reason, an unknown outcome would fold into internal-error. PrismShareAppendFailures paged on any ledger-confirmation-failed, so it now matches the unknown reason too and keeps its low-volume total-write- failure coverage. The deployment patch is regenerated from the reviewed snapshot, and a scenario covers the new reason alone. Document the narrowed meaning of ledger-confirmation-failed, the new reason, the statement-timeout guard, and the block-only acknowledgement bound. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * coordinator: reconcile share-commit deadlines instead of dropping the append (#324) submit wrapped the whole persistence future in a timeout and dropped it when the deadline fired, answering ledger-confirmation-failed. If COMMIT had already been flushed, sqlx drained the reply and reused the connection, so the share was committed and credited without an acknowledgement. A load harness reproduced that: 164 such rejections, two of them committed rows. The block-only path committed its candidate first, so its timeout always led to a later credit with no acknowledgement. Run the append in a task behind a one-shot commit gate. At the deadline the acknowledgement path either closes the gate, which proves COMMIT was never sent and aborts the doomed append, or finds the commit already in flight and awaits its reply for share_commit_grace. Only a server error of severity ERROR is a definite rejection; every other error, and a commit that outlasts the grace, is answered ledger-outcome-unknown and logged with its eventual outcome. A confirmation whose commit took at least the effective statement_timeout is also unknown, because a cancelled synchronous-replication wait commits only locally. An append carrying a found-block candidate is never refused, since that would discard the block; it waits for block_only_ack_timeout instead. Block-only proofs always finish their enqueue and then follow the candidate's disposition within the same bound, and poll errors after the enqueue never become a rejection. The new bounds are Config fields rather than environment variables, and statement_timeout is read from the session the ledger pool actually uses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * coordinator: bound the block-only enqueue wait and time COMMIT in the task Review feedback on PR #333. enqueue_candidate_once was awaited directly, outside the bound that covers every other step of a block-only submission, so a degraded database could hold the acknowledgement open well past block_only_ack_timeout. Wait for it in a task until the bound instead. The enqueue still runs to completion, because cutting it off could strand a COMMIT that was already sent, but the miner is answered ledger-outcome-unknown on time and the enqueue's eventual outcome is logged. The synchronous-replication guard compared statement_timeout against a COMMIT duration measured when the coordinator classified the finished task. Scheduler delay after a successful COMMIT therefore counted as database time and could turn a durable accepted share into an unknown one. Measure the duration inside the append task, where COMMIT actually finishes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * metrics: count a late confirmation from when the append finished Review feedback on PR #333. late_confirmed_shares_total was decided by reading the clock while the acknowledgement was being processed, which can be arbitrarily later than the append itself. A coordinator task scheduled late therefore counted an on-time confirmation as a late one. Decide it from the instant the append finished, which the task already reports, and record it where that instant is known. The help text also claimed every counted share was confirmed within the grace period. That was never true of a share-pass submission carrying a found-block candidate: those are deliberately never refused, so they can be confirmed later still, up to the block-only bound, and they are counted. Say what the counter measures instead, and spell the found-block case out in the metrics documentation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test(prism): add bounded PostgreSQL WAL measurement helper * fix(prism): harden WAL qualification failure semantics * fix(prism): harden WAL helper validation and outcomes * fix(prism): sanitize WAL helper connection inputs * test(prism): cover WAL helper outcomes and scope * fix(prism): preserve WAL operation outcome uncertainty * fix(prism): bound WAL measurement subprocesses * fix(prism): complete WAL helper diagnostics and test coverage * test(prism): repair WAL helper test layout * test(prism): stabilize WAL diagnostic mocks
…323) * 3.x.x: measure the share-append throughput floor against PostgreSQL test/test-prism-postgres-throughput.sh measured `cargo run -- benchmark`, a CPU-only run of the audit builder with no database at all, and the 2.x.x report it replaced measured one bulk INSERT ... SELECT ... FROM generate_series. Neither says anything about the path a share actually takes. Every native share append is one transaction that takes the cluster-wide ORDER_LOCK advisory lock, checks writability, bumps the single-row ledger clock and writes two rows, so share appends are serialized cluster-wide, and nobody had measured that with more than one writer. tests/throughput_floor.rs seeds a payout window with the #264 fixture generator and then appends through Ledger::append at 1, 2 and 4 concurrent appenders, each appender holding its own Ledger with its own pool and its own instance_id, as separate production frontends do. A connection outside every appender pool samples pg_locks for backends queued behind ORDER_LOCK while a level runs, and the report says how many waiters it saw and for how long -- labelled throughout as sampled estimates, with the episode count a lower bound, because a wait shorter than the 10 ms interval can be missed entirely. A sampler that fails reports null and the error, never zero. The report is qbit.prism.postgres-throughput.v2, a deliberately new schema name: its numbers must never be read against v1's bulk-insert rate. It is written before the floor is asserted, so a run that fails the floor still leaves its evidence behind. Every QBIT_PRISM_* input is validated at entry, before a connection is opened: nan, inf, a negative or an empty floor are configuration errors rather than extreme floors, and a window size that does not divide the fixture's window weight would not produce the window it claims to. CI_MIN_SHARES_PER_SEC is provisional -- 25% of the slowest level rate measured in a debug build on the writer's host. Debug is the right base because CI runs this test in debug, and the wide margin is honest about the fact that nothing has calibrated it on the 2-vCPU runner yet. Part of #271 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: point the throughput script and target at the share-append floor test-prism-postgres-throughput ran `cargo run -- benchmark`, which measures the audit builder's CPU cost and never opens a database, so the target named after PostgreSQL throughput was not measuring PostgreSQL at all. It now runs the full-size share-append floor: the ignored throughput_floor test, in release, against PRISM_TEST_DATABASE_URL when one is set and otherwise through prism-native-tests.sh, which initdb's a disposable cluster on a random loopback port and removes it on exit. The builder benchmark has not gone anywhere and is still worth running; the script says so, and gives the exact command, because it answers a different question rather than a worse version of this one. `set -e` is suspended around the test so the report path is printed on a red run too -- the test writes the report before it asserts the floor, so a run that fails the floor is exactly the run whose report someone wants to read -- and the test's exit status is re-raised immediately afterwards. Every QBIT_PRISM_* variable reaches the test process unchanged; the script reads none of them except to say where the report landed. Part of #271 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: resolve pg_stat_statements by schema, not by search path The pg_stat_statements probe was `to_regclass('pg_stat_statements')`, and the read and the reset named the view unqualified. Every connection in this run carries `options=-csearch_path=<run schema>`, which replaces the search path rather than prepending to it, so `public` is not on it. On a server with the extension installed in `public` the probe therefore returned NULL and the report said "unavailable" about a view that was sitting right there -- an EP-OBSERVABILITY failure of exactly the kind the rest of this file is careful about, because it reads as "this server cannot tell us" when the truth was "this query looked in the wrong place". The extension's schema is now resolved from pg_extension and the server's own quote_ident, and both the reset and the read qualify with it. Verified against PostgreSQL 16.15 with `shared_preload_libraries=pg_stat_statements` and the extension created in public: the block now reports 2000 calls per level, and its total_exec_time corroborates the sampler closely -- 4254 ms against a sampled 3.97-4.31 waiter-seconds at two appenders, and 17387 ms against a sampled 16.9-17.3 at four. Two independent methods agreeing to within a few percent is worth more than either alone, which is the whole reason the block is there. Part of #271 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: attribute ORDER_LOCK waits to the run that caused them The adversarial review found that the per-run schema isolates the tables but not the lock. ORDER_LOCK is a PostgreSQL advisory lock and an advisory lock is scoped to the database, so any other backend appending shares into the same database serialises against this run's appenders, and the sampler -- filtered only by database -- counted those foreign waits as this level's. Measured on a quiet server against the same server with one other appending process, the rates fell 40-57% and nothing in the report said why. A run like that either fails a calibrated floor for a reason that is not the append path, or, once the margin narrows, passes on a number describing two workloads. Every connection this run opens now carries a per-run application_name, `throughput-<uuid>`, pushed through the URL -- which is the only thing Ledger::connect is handed, so the seeding ledger, every appender ledger and the sampler all announce themselves without any production code knowing this test exists. sqlx 0.8 parses the parameter out of the URL and sends it in the startup packet, but the run does not take that on trust: it reads current_setting('application_name') back on its first connection and refuses to measure anything if the server disagrees, because a silently ignored parameter would make every foreign backend look like ours and every contaminated run look clean. The sampler then reports this run's waiters as before, and foreign_waiters_max and foreign_backends_seen beside them. A foreign backend counts whether it was waiting on the lock or holding it: either way it is serialising against these appends. A backend whose pg_stat_activity row this role may not read counts as foreign, which is the safe direction. A level during which any foreign backend was seen is marked contaminated, in the level, at the top of the report and in the floor's failure message; it is still measured and still gated, because the rate is what the appends really achieved, but the report now says plainly that the number describes a shared database and must not be carried forward. contaminated is null, not false, when the sampler could not look. "Nobody checked" is not "nothing was there", and the same reasoning fixes the smaller version of that bug next door: a level that ended before the sampler completed a poll used to emit mean_waiters 0.0, max_waiters 0 and estimated_waiter_seconds 0.0, which reads as "nobody ever waited". Those are null now, samples stays 0, and the error says the level ended first. The sampler's shutdown and its interval are fixed together. It tested the stop flag only at the top of its loop and then blocked in sleep(interval), so every level burned up to a whole interval of dead time after its appends had finished -- 65 seconds of wall clock for 0.43 seconds of measurement at a 20-second interval. The flag is now paired with a Notify signalled alongside it, so the sleep is interruptible and a level ends when the appends end; notify_one stores the permit, so the signal cannot be lost in the window between a poll and a sleep. The interval itself is bounded to 1..=1000 ms at entry, before any connection, with a message saying that an interval coarser than a second cannot resolve a lock wait that lasts milliseconds. Two other gaps the review left open. A run that dies connecting, seeding or appending now writes a report anyway -- identity, configuration, the full error chain, passed_minimum false and an empty results array -- so the newest file on disk belongs to this run rather than to some earlier one; a failure to write it is attached to the original error as context and never replaces it. And the default report path is per test, target/prism-postgres-throughput-<test>.json, with a "test" field in the JSON to match, because the two floor tests run in parallel threads under --include-ignored and the survivor of a shared path was whichever finished last, unlabelled. QBIT_PRISM_THROUGHPUT_REPORT is still the explicit override. Cancellation is documented rather than fixed: SIGINT kills the process before the schema drop runs, and handling it properly needs a signal handler, which means a dependency this test may not add. The module docs now say so and give the query that finds and drops the leftovers, next to the standing requirement that this run wants a database nothing else is appending to. Part of #271 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: make the throughput script point at a report that exists Two things the script got wrong, both of which send a reader looking for evidence that is not there. It printed the report path unconditionally, after the status had been captured. On the fallback path with PRISM_TEST_DATABASE_URL unset and no local server binaries, the run dies before the test starts and writes nothing, yet the script still announced a file. It now prints the path only when the file is actually on disk, and otherwise says that no report was written and where it would have been. The exit status is unchanged either way. It also pointed at the old shared default path. The test writes target/prism-postgres-throughput-<test name>.json now, one per test, so the script derives its path from the test it runs rather than from a constant that has moved. The header gains the two warnings the test itself carries. ORDER_LOCK is database-wide, not schema-wide, so this must run against a database nothing else is appending to; a run that shares one is marked contaminated and its number describes two workloads. And a cancelled run leaves its prism_throughput_<uuid> schema behind, so the header gives the query that finds the leftovers and the statement that drops them -- unnecessary on a disposable cluster, which is thrown away whole, and that is the reason to prefer one. Part of #271 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 3.x.x: print the floor's per-level rates where CI can see them The floor test prints its rates with println!, and libtest throws away the captured output of a passing test. CI keeps no report artifact either, so a green prism-native-postgres run recorded nothing about how far above the floor it was, and the provisional CI_MIN_SHARES_PER_SEC could not be calibrated from CI at all. After the report is written, the test now emits one summary line through std::io::stderr() directly, which the test harness does not capture: the test name, build profile, window, shares per level, the rate at each appender count, the floor, the verdict and the contamination flag. Every CI log then carries the margin the floor is set from. A failed write is ignored, because the report file already holds everything the line summarises. Part of #271 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * 3.x.x: move the throughput floor onto the shared integration gate #322 replaced every test's private copy of the database guard with one shared gate, qbit-prism-test-gate, and made the prism-native-postgres job prove that each gated test executed against test/prism-gated-tests.txt. The floor still carried its own copy of the old guard, which the new check_gate_env_reads lint rejects, and it was not on the list, so the proof step would have failed the moment the test ran. The floor now asks the gate for its database URL, so it follows the same decision table as every other gated test and records its decision in the execution manifest. share_append_throughput_floor joins the list; the full-size test is not selected in CI and stays off it. Part of #271 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * 3.x.x: set the CI throughput floor from the CI runner's own numbers CI_MIN_SHARES_PER_SEC was a provisional 71 shares/s, a quarter of what an 8-vCPU devbox managed, because nothing had measured the runner that enforces it. Four runs of prism-native-postgres on blacksmith-2vcpu-ubuntu-2404, read from the summary line the test now writes, put the slowest level of each run at 1,290, 847, 1,242 and 1,200 shares/s (debug build, the job's PostgreSQL 16 service, a 20k-share window and 2,000 appends per level). The floor is now 250, about 30% of the slowest level seen. Runs on that runner differ by about a third, so the margin keeps a noisy run green, while a 3-4x collapse of the append path still fails the build. The report's source label no longer calls the constant provisional. Part of #271 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * 3.x.x: run the floor the disposable-cluster path was skipping Four findings from the review of this PR, all in the paths that are hardest to exercise by hand. The wrapper script passed cargo_args beginning with 'test' to prism-native-tests.sh, which runs 'cargo test "$@"' itself, so the second word became a test-name filter. With --exact alongside it the disposable-cluster path selected nothing, printed nothing and exited 0: the script that exists to measure the floor measured nothing whenever PRISM_TEST_DATABASE_URL was unset. The array now carries only the arguments, and the direct call adds 'test' itself. The full-size test is selected by name, so a missing database must fail it rather than skip it; it now takes the gate's required entry point, and the CI-sized test keeps the skipping one. Resetting pg_stat_statements cleared the whole server's counters, including other databases; each level now reports the difference of two reads instead, and says so where the numbers are. The level clock started after the barrier opened, so an appender the runtime resumed first could work outside the measured window and inflate the rate; it starts before. Part of #271 * 3.x.x: scope the lock statistics to this database and stop announcing stale reports Two findings from review, both about a number that looks right and is not. `pg_stat_statements` is server-wide. The level's advisory-lock figures were the difference of two reads filtered only by query text, so an advisory lock taken in any other database on the same server landed in the level's count. A lock in another database cannot contend with this run -- an advisory lock is scoped to its database -- so the level was correctly marked uncontaminated while its lock figures described someone else's work. Filter on `dbid`, and say so in the report's caveat. Measured on a PostgreSQL 16 with `pg_stat_statements` loaded, over a level during which another database took 3,000 advisory locks: the old query attributed 3,000 calls to the level, the filtered one attributes 0. With the fix and a concurrent foreign load of about 24,000 locks, each level reports exactly 300 calls for 300 appended shares -- one lock per share, which is what the append path takes. The wrapper announced any file at the report path as this run's evidence, so a run that died during compilation or cluster setup pointed the reader at a previous run's report while returning a failure. The earlier file is now moved aside before the run, which makes the report's presence afterwards proof that this run wrote it, and it is put back if this run wrote nothing. Deleting it instead would have destroyed the previous run's data on a run that produced none of its own. Timestamps were tried first and rejected: a warm build and a small window can finish inside one filesystem timestamp tick, and a fresh report then compares as no newer than the moment the run started, which reports a good run as stale. Verified end to end through the wrapper against a real PostgreSQL 16, and for all three announcement cases: a failed run with an earlier report present, a successful run that writes one, and no report at all. Part of #271 --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
) `3.x.x` has been red since #344 landed. Two pull requests collided semantically while each was green on its own: - #333 (e13051c) added `ledger-outcome-unknown` to `RejectReason`, so `/metrics` now emits one more `qbit_prism_rejections_total` series; - #344 (037b04e) added `event_exposition_matches_the_pre_allocation_change_byte_for_byte`, whose fixture was captured at 2914a62, before that reason existed. Neither branch could see the problem, because it only appears in the merged tree. The whole difference is one line, so the fixture gains it. The comment above the assertion is refreshed at the same time: it claimed provenance the fixture no longer had, and it now says what the pin implies -- adding a value to a closed label set has to be recorded here. Verified on this branch: the four metrics_allocation tests pass, and the failure reproduces on a6fe3fd without this change. Part of #338
* fix: record cancelled collector pool acquisitions * docs: align collector cancellation timing contract * docs: qualify cancelled collector waits in alert contracts * test(metrics): include approved unknown ledger outcome in fixture PR #333 intentionally added the ledger-outcome-unknown rejection label and a producer contract test. Add exactly that sample to the older exposition fixture from #344, retaining the byte-for-byte equality assertion and leaving runtime behavior unchanged. (cherry picked from commit 4e5b4c4)
rustls 0.23.44 accepted TLS 1.3 handshake messages sent at the wrong encryption level when they followed a key-changing message in the same record, instead of terminating the connection as RFC 8446 section 5.1 requires. The transcript stays authenticated, so a network-position attacker cannot alter or complete a handshake; the effect is that a peer can send handshake messages in plaintext that should have been encrypted without rustls rejecting the connection. It is the same bug as Go's CVE-2025-61730. The advisory names 0.23.45 as the fix. This is a lockfile-only change: `cargo update -p rustls`, no manifest or source change. The advisory database picked this up today, so the dependency audit now fails on 3.x.x and on every open pull request.
…ndow (#353) * 3.x.x: add PrismRewardManifestHeader and a checked reward-manifest rebuild (#267) A native audit row is about to stop storing `reward_manifest.shares`. The counted window is a pure fold over `(shares, found_block)`, so the row only needs the manifest header; `restore_reward_manifest` reruns the fold and returns its window only after proving the rebuilt header is identical to the stored one, failing as the existing `reward_manifest` audit mismatch otherwise. There is no unchecked variant. `PrismRewardManifestHeader` denies unknown fields, so a legacy manifest that still carries `shares` can never be read as a header; and the full manifest still requires `shares`, so an older reader meeting a normalized row fails explicitly instead of serving an empty window. Tests prove the rebuilt bundle is byte-identical to the original (both schemas, credit_policy included), that one changed field in a rebuilt share moves the canonical sha256, that a tampered header is refused by name, and that neither shape can be mistaken for the other. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LXvnfUC8CvPxskAXLgd1xx * 3.x.x: store native audit bodies without reward_manifest.shares and verify the range before the settlement lock (#267) The landed native audit body kept `reward_manifest.shares`, one record per counted share, so it grew with the payout window: about 235 MB in one JSONB value at the production window, 88% of PostgreSQL's container limit. The landing write now removes that copy alongside the top-level `shares`; the stored body is O(distinct miners and payout accounts). Nothing else is added or removed, `audit_body_byte_len` stays the full canonical artifact length, and the GIN-indexed leaf arrays stay in the body. Reads branch on the stored shape. A body whose `reward_manifest` has no `shares` key is decoded as `PrismRewardManifestHeader` and its window rebuilt through `restore_reward_manifest`, which refuses a header the fold does not reproduce; a body that still has the key (landed before this change) is served exactly as before. Both arms then run the unchanged canonical digest check, so a rebuilt body is proven byte-identical, not merely equivalent. Imported rows never enter this path. The snapshot digest, the rebuild and the canonical hash run on one blocking thread, and the public API hands that job the same permit imported decodes hold, so a dropped request cannot free the limit before the rebuild ends. The durable-range proof no longer runs inside the settlement transaction. `verify_durable_range` runs before the lock, page by page (4096 rows, the window reader's page), decoding and comparing each page on a blocking thread against the same positions of the candidate window: same predicate, same ordering, same full equality, same bootstrap exemption, same error text, and the first mismatched page fails the landing. An already-landed block skips it and takes the existing-row digest path as before. Under the lock the snapshot now only counts the same anchored predicate and fails the landing if the count differs from the window. The JSONB ceiling gate's landing entry is deleted: that write no longer crosses the threshold. Gated tests prove the stored shape and identical served bytes, that a tampered stored header is refused by name, that pre-change native rows still serve identical bytes, and that imported rows serve from their canonical bytes even with a poisoned inline body. An ignored harness measures the stored body, the settlement-lock hold and the landing peak RSS at a configurable window size. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LXvnfUC8CvPxskAXLgd1xx * 3.x.x: document the measured native audit body before and after normalization (#267) State the measured stored bytes per block at 20,000 and 100,000 shares before and after `reward_manifest.shares` left the stored body, say that the residual is O(distinct miners and payout accounts) rather than a fixed bound, and separate `audit_body_byte_len` (the full canonical artifact) from `sum(pg_column_size(audit_bundle))` (the stored, compressed non-share body). Also record the measured cost of rebuilding a native audit on read. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LXvnfUC8CvPxskAXLgd1xx * 3.x.x: correct three statements the audit-body normalization made untrue (#267) The JSONB ceiling gate's module doc and its landing phase note both still described the landing write as embedding the payout window, and the gate printed that note on every run. The payout-artifact measurement guide counted three native window-carrying writes. After normalization there are two, and landing is flat in the window. `d2_bootstrap_tests` cited `persist_audit_snapshot` re-reading the share range inside the transaction as one of the three closed exits for an empty payout window beside a non-zero carry. The exit is still closed, but by `verify_durable_range` page by page before the settlement lock plus the in-transaction count of the same anchored range, so the citation now names both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * 3.x.x: give the paged durable-range proof negative multi-page coverage (#267) `verify_durable_range` had no negative coverage: its body could be replaced with `Ok(())`, together with the in-transaction count guard, and every reachable test still passed. The gated tests only ever drove it with a one-share window, so no test had executed a second page of the loop. A new gated test loads a 10,000-share window, three pages of the proof's 4096 rows, and lands four candidates that disagree with the ledger from the claim side, since the share ledger is immutable by trigger. Each window is built and signed as it is, so the audit signature check that precedes the proof accepts it and the proof is what refuses it, asserted by name: one `ntime` differs on the first share of the second page, one on the last share of the final partial page, one share is missing from the final page so the durable page outruns the window slice, and one share the ledger accepted after the anchor is claimed, so every page matches and the final count refuses it. A refused landing writes nothing. The unaltered window then lands and serves identical bytes, so the same fixture proves the loop accepts a correct multi-page window. The in-transaction count guard has no independent coverage. The anchored set is frozen once the anchor is issued and the ledger is append-only, so a window that passes the paged proof cannot count differently under the lock without a writer that defeats the immutability trigger between the two reads; it is defence in depth, and the test names the proof's message, not the guard's. `land_small_block` now appends a second share after landing, so present-day ledger state differs from the block's anchored snapshot in every test that uses it. A read that rebuilt the window from current ledger state instead of the snapshot range passed all of them before; it now fails the snapshot digest. Two comments are corrected without a behaviour change: the negative block in `reward_manifest_header.rs` now mutates the restored window it claimed to, and the permit comment in `read_models.rs` says the permit also spans the snapshot lookup and window read that precede the blocking job. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016P2UWk2rh2R1W8yqTdLZ4k --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* feat(prism): record ledger advisory-lock and pool-acquire waits The native registry declares two database wait families, but nothing in the ledger observed either one: advisory-lock waits had no observations at all, and pool acquisition counted only the metrics collector's own. ORDER_LOCK contention, which serializes every share acknowledgement, was therefore invisible in production and in qualification. Give Ledger an optional metrics handle, supplied by the new connect_with_metrics; connect keeps its signature and passes None, so the tools and tests that connect without a registry are unchanged and record nothing. The coordinator passes its own handle. Route every advisory lock through the existing lock() helper, now timing the pg_advisory_xact_lock statement alone, and every ledger transaction through a new begin() helper that times pool acquisition alone. begin() is Pool::begin split in two around the timer, down to its sqlx::Error return type, so both helpers are drop-in replacements and every call site's `?` converts exactly as before. Leaving the BEGIN round trip outside the timer keeps the pool series meaning what the collector already records. Errors, including a lock_timeout expiry, propagate unchanged and record as failure. A drop guard records a wait abandoned by cancellation as a failure with its elapsed time, so a cancelled wait stays distinct from one that never happened. Completing the wait takes the handle out of the guard, so a wait is recorded exactly once. The metrics mutex is taken only after the wait has ended, never across an await, so rendering cannot queue behind a contended lock; a guard held across the await would also make the future !Send and fail to compile in the spawned share path. The CPFP funding lock has no LockKind value and stays unobserved; an unmapped key trips a debug assertion rather than borrowing another lock's label. Refs #328. * test(prism): cover ledger wait metrics by lock and outcome Two gated PostgreSQL tests and one source invariant. The first attaches a registry and drives each outcome against a real database: the migration lock from schema creation, a share append taking ORDER_LOCK once, a second session holding ORDER_LOCK past a 200 ms lock_timeout so the append fails with SQLSTATE 55P03 and records one failure of at least that duration, an append cancelled mid-wait recording its partial wait and no more, the settlement lock through save_job, five transactions each observing one pool acquisition plus the collector's own unchanged, a pool acquisition cancelled while every connection is held, and a closed pool recording an acquisition failure. The second runs the same sequence on a ledger with no handle and asserts that results and error codes, 55P03 included, are what they are at the base. It deliberately makes no assertion about a registry: one no ledger can reach has nothing to record into and could not fail the test. The third walks the ledger sources recursively and fails if a transaction begins outside the timed helper or a second pg_advisory_xact_lock statement appears, so a new call site, including one in a nested module directory, cannot silently stop being observed. Refs #328. * docs(prism): describe the recorded database wait metrics The two database wait families now say what they measure and who records them: pool acquisition excludes transaction BEGIN and the queries that follow, and advisory-lock timing covers one pg_advisory_xact_lock round trip for the migration, order and settlement locks. Both record a wait abandoned by cancellation as a failure, except the collector's own acquisitions, which keep reporting cancellation through collector status rather than inventing a completed wait. The follow-up ownership and alert-migration notes no longer describe either family as unpopulated, and record that PrismDatabasePoolWaitHigh now evaluates ledger acquisitions while its description still predates this change. No alert rule or digest input changes. Refs #328. * docs(prism): record the pool alert's collector-availability gate PrismDatabasePoolWaitHigh is gated on collector_available for the database collector. That collector acquires from the same pool as the ledger, so sustained pool exhaustion times out its own acquisition, zeroes the gauge and suppresses the rule exactly while the ledger histogram is recording the waits that should fire it. The gate cost nothing before this branch, because the family carried no samples during such an outage. It does now. The rule itself cannot change here: its file feeds the alert specification digest embedded in the deployment patch, which can only be regenerated from the frozen template. #336 tracks that work; this records the interaction where operators read it. Refs #328.
* docs: correct pool-acquire alert measurement boundaries Describe metrics collection and instrumented ledger transaction pool acquisition, retaining query/advisory-lock/read-gate exclusions and the five no-replacement dispositions. Regenerate migration tables and deployment patch from the verified frozen snapshot; all alert behavior is unchanged. * test(metrics): include approved unknown ledger outcome in fixture PR #333 intentionally added the ledger-outcome-unknown rejection label and a producer contract test. Add exactly that sample to the older exposition fixture from #344, retaining the byte-for-byte equality assertion and leaving runtime behavior unchanged. * feat(prism): record ledger advisory-lock and pool-acquire waits The native registry declares two database wait families, but nothing in the ledger observed either one: advisory-lock waits had no observations at all, and pool acquisition counted only the metrics collector's own. ORDER_LOCK contention, which serializes every share acknowledgement, was therefore invisible in production and in qualification. Give Ledger an optional metrics handle, supplied by the new connect_with_metrics; connect keeps its signature and passes None, so the tools and tests that connect without a registry are unchanged and record nothing. The coordinator passes its own handle. Route every advisory lock through the existing lock() helper, now timing the pg_advisory_xact_lock statement alone, and every ledger transaction through a new begin() helper that times pool acquisition alone. begin() is Pool::begin split in two around the timer, down to its sqlx::Error return type, so both helpers are drop-in replacements and every call site's `?` converts exactly as before. Leaving the BEGIN round trip outside the timer keeps the pool series meaning what the collector already records. Errors, including a lock_timeout expiry, propagate unchanged and record as failure. A drop guard records a wait abandoned by cancellation as a failure with its elapsed time, so a cancelled wait stays distinct from one that never happened. Completing the wait takes the handle out of the guard, so a wait is recorded exactly once. The metrics mutex is taken only after the wait has ended, never across an await, so rendering cannot queue behind a contended lock; a guard held across the await would also make the future !Send and fail to compile in the spawned share path. The CPFP funding lock has no LockKind value and stays unobserved; an unmapped key trips a debug assertion rather than borrowing another lock's label. Refs #328. * test(prism): cover ledger wait metrics by lock and outcome Two gated PostgreSQL tests and one source invariant. The first attaches a registry and drives each outcome against a real database: the migration lock from schema creation, a share append taking ORDER_LOCK once, a second session holding ORDER_LOCK past a 200 ms lock_timeout so the append fails with SQLSTATE 55P03 and records one failure of at least that duration, an append cancelled mid-wait recording its partial wait and no more, the settlement lock through save_job, five transactions each observing one pool acquisition plus the collector's own unchanged, a pool acquisition cancelled while every connection is held, and a closed pool recording an acquisition failure. The second runs the same sequence on a ledger with no handle and asserts that results and error codes, 55P03 included, are what they are at the base. It deliberately makes no assertion about a registry: one no ledger can reach has nothing to record into and could not fail the test. The third walks the ledger sources recursively and fails if a transaction begins outside the timed helper or a second pg_advisory_xact_lock statement appears, so a new call site, including one in a nested module directory, cannot silently stop being observed. Refs #328. * docs(prism): describe the recorded database wait metrics The two database wait families now say what they measure and who records them: pool acquisition excludes transaction BEGIN and the queries that follow, and advisory-lock timing covers one pg_advisory_xact_lock round trip for the migration, order and settlement locks. Both record a wait abandoned by cancellation as a failure, except the collector's own acquisitions, which keep reporting cancellation through collector status rather than inventing a completed wait. The follow-up ownership and alert-migration notes no longer describe either family as unpopulated, and record that PrismDatabasePoolWaitHigh now evaluates ledger acquisitions while its description still predates this change. No alert rule or digest input changes. Refs #328. * docs(prism): record the pool alert's collector-availability gate PrismDatabasePoolWaitHigh is gated on collector_available for the database collector. That collector acquires from the same pool as the ledger, so sustained pool exhaustion times out its own acquisition, zeroes the gauge and suppresses the rule exactly while the ledger histogram is recording the waits that should fire it. The gate cost nothing before this branch, because the family carried no samples during such an outage. It does now. The rule itself cannot change here: its file feeds the alert specification digest embedded in the deployment patch, which can only be regenerated from the frozen template. #336 tracks that work; this records the interaction where operators read it. Refs #328. * feat(prism): record ledger advisory-lock and pool-acquire waits The native registry declares two database wait families, but nothing in the ledger observed either one: advisory-lock waits had no observations at all, and pool acquisition counted only the metrics collector's own. ORDER_LOCK contention, which serializes every share acknowledgement, was therefore invisible in production and in qualification. Give Ledger an optional metrics handle, supplied by the new connect_with_metrics; connect keeps its signature and passes None, so the tools and tests that connect without a registry are unchanged and record nothing. The coordinator passes its own handle. Route every advisory lock through the existing lock() helper, now timing the pg_advisory_xact_lock statement alone, and every ledger transaction through a new begin() helper that times pool acquisition alone. begin() is Pool::begin split in two around the timer, down to its sqlx::Error return type, so both helpers are drop-in replacements and every call site's `?` converts exactly as before. Leaving the BEGIN round trip outside the timer keeps the pool series meaning what the collector already records. Errors, including a lock_timeout expiry, propagate unchanged and record as failure. A drop guard records a wait abandoned by cancellation as a failure with its elapsed time, so a cancelled wait stays distinct from one that never happened. Completing the wait takes the handle out of the guard, so a wait is recorded exactly once. The metrics mutex is taken only after the wait has ended, never across an await, so rendering cannot queue behind a contended lock; a guard held across the await would also make the future !Send and fail to compile in the spawned share path. The CPFP funding lock has no LockKind value and stays unobserved; an unmapped key trips a debug assertion rather than borrowing another lock's label. Refs #328. * test(prism): cover ledger wait metrics by lock and outcome Two gated PostgreSQL tests and one source invariant. The first attaches a registry and drives each outcome against a real database: the migration lock from schema creation, a share append taking ORDER_LOCK once, a second session holding ORDER_LOCK past a 200 ms lock_timeout so the append fails with SQLSTATE 55P03 and records one failure of at least that duration, an append cancelled mid-wait recording its partial wait and no more, the settlement lock through save_job, five transactions each observing one pool acquisition plus the collector's own unchanged, a pool acquisition cancelled while every connection is held, and a closed pool recording an acquisition failure. The second runs the same sequence on a ledger with no handle and asserts that results and error codes, 55P03 included, are what they are at the base. It deliberately makes no assertion about a registry: one no ledger can reach has nothing to record into and could not fail the test. The third walks the ledger sources recursively and fails if a transaction begins outside the timed helper or a second pg_advisory_xact_lock statement appears, so a new call site, including one in a nested module directory, cannot silently stop being observed. Refs #328. * docs(prism): describe the recorded database wait metrics The two database wait families now say what they measure and who records them: pool acquisition excludes transaction BEGIN and the queries that follow, and advisory-lock timing covers one pg_advisory_xact_lock round trip for the migration, order and settlement locks. Both record a wait abandoned by cancellation as a failure, except the collector's own acquisitions, which keep reporting cancellation through collector status rather than inventing a completed wait. The follow-up ownership and alert-migration notes no longer describe either family as unpopulated, and record that PrismDatabasePoolWaitHigh now evaluates ledger acquisitions while its description still predates this change. No alert rule or digest input changes. Refs #328. * docs(prism): record the pool alert's collector-availability gate PrismDatabasePoolWaitHigh is gated on collector_available for the database collector. That collector acquires from the same pool as the ledger, so sustained pool exhaustion times out its own acquisition, zeroes the gauge and suppresses the rule exactly while the ledger histogram is recording the waits that should fire it. The gate cost nothing before this branch, because the family carried no samples during such an outage. It does now. The rule itself cannot change here: its file feeds the alert specification digest embedded in the deployment patch, which can only be regenerated from the frozen template. #336 tracks that work; this records the interaction where operators read it. Refs #328. --------- Co-authored-by: Alex <alex@swaplabs.xyz>
* test: cover native ledger deadlines and execution safety Add 14 PostgreSQL regressions for late issued-job expiry, atomic publication, migration restart, incomplete snapshots, and claim-fenced execution. Use a test-only protocol observer to detect sensitive write replay after timeouts or lost acknowledgements, and document the native retry and snapshot contracts. * test: make candidate backoff eligibility deterministic Verify retry scheduling before controlling eligibility through fixture timestamps. Use future and past infinity to exercise claim fencing without racing the one-second retry deadline or waiting for it to elapse. Addresses the timing-race review comment on #355. * test: remove fanout backoff timing race and clarify locks Compare the fanout retry schedule with its durable write timestamp, then control claim eligibility through fixture timestamps. Document settlement and order locking and the scope of the sibling-row independence test. Addresses the two follow-up review comments on #355. * test: isolate ledger deadline and fanout eligibility checks Freeze all three completed fanouts before the global pre-deadline claim assertion. Use a dedicated fixture pool without competing server timeouts for the issued-job absolute-deadline regression, retaining bounded barriers and cleanup. Production behavior and other fixture defaults are unchanged. Addresses the two follow-up review comments on #355.
* docs: correct pool-acquire alert measurement boundaries Describe metrics collection and instrumented ledger transaction pool acquisition, retaining query/advisory-lock/read-gate exclusions and the five no-replacement dispositions. Regenerate migration tables and deployment patch from the verified frozen snapshot; all alert behavior is unchanged. * test(metrics): include approved unknown ledger outcome in fixture PR #333 intentionally added the ledger-outcome-unknown rejection label and a producer contract test. Add exactly that sample to the older exposition fixture from #344, retaining the byte-for-byte equality assertion and leaving runtime behavior unchanged. * fix: keep pool-wait alert active when database collector fails Remove only the collector-availability gate; preserve snapshot freshness, sample sufficiency, thresholds, duration and labels. Add bounded PromQL scenarios with per-instance assertions, recovery and the old-expression negative control. Refs #336. Landing depends on #334 and reconciliation with #343 and #345. * feat(prism): record ledger advisory-lock and pool-acquire waits The native registry declares two database wait families, but nothing in the ledger observed either one: advisory-lock waits had no observations at all, and pool acquisition counted only the metrics collector's own. ORDER_LOCK contention, which serializes every share acknowledgement, was therefore invisible in production and in qualification. Give Ledger an optional metrics handle, supplied by the new connect_with_metrics; connect keeps its signature and passes None, so the tools and tests that connect without a registry are unchanged and record nothing. The coordinator passes its own handle. Route every advisory lock through the existing lock() helper, now timing the pg_advisory_xact_lock statement alone, and every ledger transaction through a new begin() helper that times pool acquisition alone. begin() is Pool::begin split in two around the timer, down to its sqlx::Error return type, so both helpers are drop-in replacements and every call site's `?` converts exactly as before. Leaving the BEGIN round trip outside the timer keeps the pool series meaning what the collector already records. Errors, including a lock_timeout expiry, propagate unchanged and record as failure. A drop guard records a wait abandoned by cancellation as a failure with its elapsed time, so a cancelled wait stays distinct from one that never happened. Completing the wait takes the handle out of the guard, so a wait is recorded exactly once. The metrics mutex is taken only after the wait has ended, never across an await, so rendering cannot queue behind a contended lock; a guard held across the await would also make the future !Send and fail to compile in the spawned share path. The CPFP funding lock has no LockKind value and stays unobserved; an unmapped key trips a debug assertion rather than borrowing another lock's label. Refs #328. * test(prism): cover ledger wait metrics by lock and outcome Two gated PostgreSQL tests and one source invariant. The first attaches a registry and drives each outcome against a real database: the migration lock from schema creation, a share append taking ORDER_LOCK once, a second session holding ORDER_LOCK past a 200 ms lock_timeout so the append fails with SQLSTATE 55P03 and records one failure of at least that duration, an append cancelled mid-wait recording its partial wait and no more, the settlement lock through save_job, five transactions each observing one pool acquisition plus the collector's own unchanged, a pool acquisition cancelled while every connection is held, and a closed pool recording an acquisition failure. The second runs the same sequence on a ledger with no handle and asserts that results and error codes, 55P03 included, are what they are at the base. It deliberately makes no assertion about a registry: one no ledger can reach has nothing to record into and could not fail the test. The third walks the ledger sources recursively and fails if a transaction begins outside the timed helper or a second pg_advisory_xact_lock statement appears, so a new call site, including one in a nested module directory, cannot silently stop being observed. Refs #328. * docs(prism): describe the recorded database wait metrics The two database wait families now say what they measure and who records them: pool acquisition excludes transaction BEGIN and the queries that follow, and advisory-lock timing covers one pg_advisory_xact_lock round trip for the migration, order and settlement locks. Both record a wait abandoned by cancellation as a failure, except the collector's own acquisitions, which keep reporting cancellation through collector status rather than inventing a completed wait. The follow-up ownership and alert-migration notes no longer describe either family as unpopulated, and record that PrismDatabasePoolWaitHigh now evaluates ledger acquisitions while its description still predates this change. No alert rule or digest input changes. Refs #328. * docs(prism): record the pool alert's collector-availability gate PrismDatabasePoolWaitHigh is gated on collector_available for the database collector. That collector acquires from the same pool as the ledger, so sustained pool exhaustion times out its own acquisition, zeroes the gauge and suppresses the rule exactly while the ledger histogram is recording the waits that should fire it. The gate cost nothing before this branch, because the family carried no samples during such an outage. It does now. The rule itself cannot change here: its file feeds the alert specification digest embedded in the deployment patch, which can only be regenerated from the frozen template. #336 tracks that work; this records the interaction where operators read it. Refs #328. * docs: distinguish pool-wait guard from database gauge guards * feat(prism): record ledger advisory-lock and pool-acquire waits The native registry declares two database wait families, but nothing in the ledger observed either one: advisory-lock waits had no observations at all, and pool acquisition counted only the metrics collector's own. ORDER_LOCK contention, which serializes every share acknowledgement, was therefore invisible in production and in qualification. Give Ledger an optional metrics handle, supplied by the new connect_with_metrics; connect keeps its signature and passes None, so the tools and tests that connect without a registry are unchanged and record nothing. The coordinator passes its own handle. Route every advisory lock through the existing lock() helper, now timing the pg_advisory_xact_lock statement alone, and every ledger transaction through a new begin() helper that times pool acquisition alone. begin() is Pool::begin split in two around the timer, down to its sqlx::Error return type, so both helpers are drop-in replacements and every call site's `?` converts exactly as before. Leaving the BEGIN round trip outside the timer keeps the pool series meaning what the collector already records. Errors, including a lock_timeout expiry, propagate unchanged and record as failure. A drop guard records a wait abandoned by cancellation as a failure with its elapsed time, so a cancelled wait stays distinct from one that never happened. Completing the wait takes the handle out of the guard, so a wait is recorded exactly once. The metrics mutex is taken only after the wait has ended, never across an await, so rendering cannot queue behind a contended lock; a guard held across the await would also make the future !Send and fail to compile in the spawned share path. The CPFP funding lock has no LockKind value and stays unobserved; an unmapped key trips a debug assertion rather than borrowing another lock's label. Refs #328. * test(prism): cover ledger wait metrics by lock and outcome Two gated PostgreSQL tests and one source invariant. The first attaches a registry and drives each outcome against a real database: the migration lock from schema creation, a share append taking ORDER_LOCK once, a second session holding ORDER_LOCK past a 200 ms lock_timeout so the append fails with SQLSTATE 55P03 and records one failure of at least that duration, an append cancelled mid-wait recording its partial wait and no more, the settlement lock through save_job, five transactions each observing one pool acquisition plus the collector's own unchanged, a pool acquisition cancelled while every connection is held, and a closed pool recording an acquisition failure. The second runs the same sequence on a ledger with no handle and asserts that results and error codes, 55P03 included, are what they are at the base. It deliberately makes no assertion about a registry: one no ledger can reach has nothing to record into and could not fail the test. The third walks the ledger sources recursively and fails if a transaction begins outside the timed helper or a second pg_advisory_xact_lock statement appears, so a new call site, including one in a nested module directory, cannot silently stop being observed. Refs #328. * docs(prism): describe the recorded database wait metrics The two database wait families now say what they measure and who records them: pool acquisition excludes transaction BEGIN and the queries that follow, and advisory-lock timing covers one pg_advisory_xact_lock round trip for the migration, order and settlement locks. Both record a wait abandoned by cancellation as a failure, except the collector's own acquisitions, which keep reporting cancellation through collector status rather than inventing a completed wait. The follow-up ownership and alert-migration notes no longer describe either family as unpopulated, and record that PrismDatabasePoolWaitHigh now evaluates ledger acquisitions while its description still predates this change. No alert rule or digest input changes. Refs #328. * docs(prism): record the pool alert's collector-availability gate PrismDatabasePoolWaitHigh is gated on collector_available for the database collector. That collector acquires from the same pool as the ledger, so sustained pool exhaustion times out its own acquisition, zeroes the gauge and suppresses the rule exactly while the ledger histogram is recording the waits that should fire it. The gate cost nothing before this branch, because the family carried no samples during such an outage. It does now. The rule itself cannot change here: its file feeds the alert specification digest embedded in the deployment patch, which can only be regenerated from the frozen template. #336 tracks that work; this records the interaction where operators read it. Refs #328. * Expose pool acquisition waits independently of health publication --------- Co-authored-by: Alex <alex@swaplabs.xyz>
…pshot, byte-identical to the serial export (#712) (#721) The recovery-evidence export took 85 minutes on the 65.9M-share union copy, mostly in the shares and share_hashes kinds, and psql's FETCH_COUNT cursor never gets a parallel plan. scripts/prism-recovery-evidence-parallel.py prints exactly the bytes that the serial psql -f scripts/prism-recovery-evidence.sql prints. It uses sessions that share one exported snapshot, so the summarizer and every runbook cmp stay unchanged. A coordinator session runs the script's preamble and calls pg_export_snapshot(). Each worker imports the snapshot, runs the preamble again and exports one part. A part is a share_seq range of the shares, a header-hash range of the share hashes (COLLATE "C", as the script orders them), or the kinds around them. The driver cuts the script itself at those two exports and adds only a WHERE, so no query is copied. It refuses to run if those statements change. Sampled range boundaries balance the work. The first and last ranges are open, so every row lands in exactly one part, and each range sorts under SET LOCAL work_mem. The export fails closed. With FETCH_COUNT set, psql 16 exits 0 when it can't write its output, so each part's rows go through a pipe that the driver writes, and a full disk fails the export. The completion marker is printed only after every session, the coordinator's commit included, has succeeded. Sessions never prompt for a password, and SIGTERM, SIGHUP or ^C stops every session and removes the parts. The only change to the SQL moves the has_* \gset before the shares export. It prints nothing. A gated test compares the parallel export with the serial one byte for byte, on a frozen 2.x source and on its migrated three-partition ledger, at 1, 3 and 8 jobs and at FETCH_COUNT 1. It checks that the serial output equals the script before this change, and that a broken integrity report leaves no parts and no marker. Python tests cover the cut, the predicates and the failure paths. Locally, 8 jobs took 24 s against 115 s serially on a 6M-share 2.x copy, and about 12 s against 57 s once migrated, every run byte-identical. Refs #712. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Set VERSION and [workspace.package].version to 3.0.0-rc.3: rc.2 plus #719 (#711 fix B: the share append holds ORDER_LOCK for 3 statements plus COMMIT, down from 9 client round trips) and #721 (#712: the parallel recovery-evidence export). Both Cargo.lock files move only their workspace-member entries. doc/release-notes-3.0.0-rc.3.md covers: - the changes since rc.2, with N2's pair measurements on a build whose runtime is #719's head, each figure saying what it measured; - upgrading, with no migration from rc.2 and the parallel export for a 2.0.x cutover; - the same evidence set as rc.2; - the known issues: #716's open part with its three measured rates, #711 still open for fix A, and #712's remaining parts. The development-line notes name 3.0.0-rc.3, and they and the 3.0.0 notes link all three candidates' notes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ead of mio's 128, so a reconnect burst doesn't overflow the accept queue (#723) tokio's TcpListener::bind listens with mio's fixed backlog of 128, the standard library's value, and every frontend listener used it. On the pair, 400 simultaneous Stratum connects through HAProxy overflowed the accept queue 175 times (TcpExtListenOverflows). HAProxy's observe layer4 then marked the frontend down, and 158 sessions moved to the remote backup frontend and stayed there. Miners that reconnect together, at a cutover, a frontend restart or an LB failover, can outrun the accept loop by more than 128. listen.rs's bind_listener binds with an explicit backlog. It keeps TcpListener::bind's semantics: resolved addresses are tried in order, the first that listens wins, SO_REUSEADDR is set and IPV6_V6ONLY keeps the system default. Both Stratum listeners take PRISM_STRATUM_LISTEN_BACKLOG, which is revived from the retired 2.x setting. It defaults to 4096, and the kernel caps it at net.core.somaxconn. The audit/operator and public HTTP listeners use 1024. The startup log reports the backlog and the kernel's cap, and warns when the cap is lower. A test sends 300 connects to a listener that never accepts. With mio's 128, 133 complete; with the helper and a backlog of 512, all 300 complete. config_cli pins the setting's range. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Set VERSION and [workspace.package].version to 3.0.0-rc.4: rc.3 plus #723 (the Stratum listeners listen with PRISM_STRATUM_LISTEN_BACKLOG, default 4096, instead of mio's fixed 128). Both Cargo.lock files move only their workspace-member entries. doc/release-notes-3.0.0-rc.4.md covers: - the change and the pair's C5 finding (175 accept-queue overflows in a 400-connect burst, frontend A marked down, 158 sessions spilled to the backup); - upgrading: the setting is read again (rc.1 to rc.3 did not read it), no migration, somaxconn at least as high as the backlog, and removing the always-passed setting before rolling a frontend back to rc.3 or earlier; - the same evidence set as rc.3, with the pair evidence rc.3's notes cite carried; - the known issues carried from rc.3 and the #724 flake. The development-line notes name 3.0.0-rc.4, and they and the 3.0.0 notes link all four candidates' notes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the serial summarizer (cutover window) (#728) The recovery-evidence summarizer parsed and canonicalized every row on one thread, at about 29 MB/s. #721 parallelised the export, but the summary of the 65.99M-share copy still took about 45 minutes, and on the migrated side that is on the cutover's critical path. prism-recovery-evidence.py --jobs N parses newline-aligned byte ranges of the export in N worker processes. Each worker returns the canonical bytes of each run of rows, plus the few fields the checks need. The parent replays the results in file order through the same summary object the serial path uses. Every digest, count and check is therefore the serial summarizer's: the completion marker, the integrity-report order, share sequence order and the carry-integrity chains. A range a worker cannot read exactly as a text-mode read would (invalid UTF-8, an unexpected exception, or a failed worker) is read serially from there on. A refusal near an undecodable byte falls back to a full serial pass, so the exit code, stdout and stderr always match --jobs 1, which is the serial summarizer. Memory is bounded by jobs + 2 chunks in flight. New tests compare exit code, stdout and stderr between --jobs 1 and --jobs 2-4. Range boundaries go down to every line, across every kind, refusal, encoding edge and fallback. The gated recovery harness summarizes its fixture export serially and in parallel and requires identical bytes. On a 3.25 GB synthetic export, 8 jobs ran 6.5x faster than serial, with identical output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dow-proof parts digest, and write their sidecars in shards (#731) (#732) 2.x wrote each audit-bundle.v2 body's window-proof share_parts_digest_hex over its share parts in insertion order. 3.x recomputes it with sorted keys and refuses every such body (#731). import-audits reads a row's canonical sidecar before its body, so a v2 body without one stops the import with "window proof share_parts_digest_hex mismatch". On union, the bodies from before 2026-08-24, about 9,500 of them, have no sidecar. scripts/prism_legacy_parts_digest_check.py lists those rows without reading any slot. It follows import's precedence, so a sidecar in the import root wins, and it recomputes a v2 body's digest both ways. It writes the refused rows in the sidecar tool's --rows format and can run in parallel. It exits 0 only when no row is refused for this digest and no body is missing, unreadable or for another digest. scripts/prism_legacy_range_sidecars_sharded.sh runs scripts/prism_legacy_range_sidecars.py --write over those rows in disjoint shards, each with its own CSV, work directory and report. It exits 0 only when every shard succeeds and no temporary file is left in the import root. The sidecar tool publishes atomically and never replaces a sidecar, so shards can't collide, and a rerun verifies instead of rewriting. The tests use a v2 body and its two slots, written by 2.x.x's own writer. The checker must call the body 2x-order-only. The real canonicalizer must refuse it, and accept it once its digest is recomputed with sorted keys. The runner must turn copies of it into sidecars that decompress to the committed bytes. CI runs these tests beside the sidecar tool's, and the migration guide documents the steps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dit completeness while import-audits runs (#734) (#736) In production, self-check stops at legacy audit completeness and skips its local checks until import-audits has stored every legacy body (#734). A cutover that admits traffic while the import still runs therefore has no self-check to gate on for hours. scripts/prism_legacy_import_admission_gate.sh checks every fact self-check checks, decided as self_check() and self_check_local() decide them, except legacy completeness. It reports that as pending, along with whether an import-audits session can be seen. Native audit rows that self-check would count as incomplete still fail. The checks: - check-config; - the live-instance census, classified as live_instances() does; - the submission hold; - carry-forward integrity; - durability; - the offer standby, when its flush wait is on; - healthcheck; - the highdiff floor probe, authorized with self-check's own username fallbacks. Every query runs in a read-only transaction, and the database password reaches psql through PGPASSWORD, never a command line. The tests run the gate with stand-ins for qbit-prism-server and psql: every pass and fail path, the census, the standby and highdiff switches, and a local Stratum listener. A gated test runs it against a freshly migrated database. The migration guide describes when to use it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t standby, or skip it, under statement and lock timeouts (#738) (#740) The gate's check 5 ran qbit_carry_forward_integrity_report() on the writer with no statement timeout. At production size the report takes about a minute, and a snapshot held that long on a primary with live miners piles up dead versions of the cluster row every share updates, which slows the share append's lock (#738). PRISM_INTEGRITY_REPORT_DATABASE_URL now runs the report on a hot standby instead. The gate refuses that URL when it is not in recovery, is not a replica of this writer's database (system identifier and database name), is on another timeline than the writer, replicates a writer that waits for apply, has hot_standby_feedback on, has replayed no transaction, or is stale: its last replayed transaction is older than PRISM_INTEGRITY_REPORT_MAX_AGE_SECONDS (default 300). The live frontends' heartbeats keep the writer committing, so the age always applies. WAL positions are recorded but never used to call a standby fresh, since after a failover they aren't comparable across timelines. The gate warns unless replay is paused or both conflict delays are -1. PRISM_GATE_INTEGRITY=skip skips the check with a WARN. Every query now runs with a 15 s statement timeout and a 5 s lock timeout. The report gets 120 s on the writer and 600 s on a standby. A failed check names a statement timeout, lock timeout or recovery conflict without echoing psql's stderr, which can name the host. The standby's credential reaches psql through a subshell's environment, and a blank URL counts as unset. The Python shards already ran the gate's hermetic tests. Its real-schema tests now run in PostgreSQL shard 3, where a missing input fails them as it fails the gated Rust tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ift once each, as migration 026 (#737) (#744) 001's qbit_carry_forward_integrity_report() counts qbit_carry_forward_integrity_mismatches() and qbit_carry_forward_current_drift() in two scalar subqueries and lists them in two more, so each runs twice. Both are CPU-bound. On a production-sized ledger, 2,753,849 active carry rows on a standby with replay paused, EXPLAIN ANALYZE put the validator at 20.29 s and the drift check at 9.14 s, and the report took 57.42 s. self-check, fatal-state clear and the import admission gate all wait for it, and a clear holds its transaction open that long while the chain tip may move. Migration 026 replaces the report in place. It keeps 001's subqueries word for word, the active row count and the two listings, and evaluates each once in a MATERIALIZED CTE, in 001's order: the count, the validator, then the drift check, so a run that fails stops where 001's does. Each set's count is its listing's length. Running each check once is estimated at about 29 s at production size, the sum of the two parts, not a timed run; on a fixture of 400,000 clean carry rows the report took 1.2 s instead of 2.3 s. The report feeds the recovery evidence's source and migrated summaries and the admission gate, so it must be evaluated as 001's is. Listing a materialized copy of each set, the first plan, is not. 001's mismatch order, block height then carry_forward_seq, ties for every finding without a carry row, whose seq is NULL, and a manifest that names an account twice repeats a seq. 001 lists tied findings in the order the validator's own sort leaves them, since its aggregate reads that sort as presorted. Rows read back from a CTE are sorted again, and once that sort spills it can reorder ties: with 2,000 findings at one height and work_mem at 64kB, that design and 001 differed at 1,405 of 2,000 positions. 001's own order of ties changes with work_mem, so no tiebreak is added. 026 is applied in the migration transaction with no capability and no shutdown proof, as 025 is, and an earlier binary accepts the unknown migration with a warning and reads the same report. migrate runs 026 again whenever it runs 001, as it runs 025 again after 011. 001 runs only without 3, and a recorded history without 3 is refused before any DDL, so for now that is only a fresh or 2.x.x source, where 26 is missing anyway. Two gated tests cover it. One checks the report against 001's, installed beside it, on empty, clean, legacy-break, as-issued and drift fixtures and on the 2,000 tied findings at the default and the smallest work_mem. It counts one run of each set per report, checks that a report whose sets both fail fails in the validator, as 001's does, and fails against the per-set CTE design. The other checks that 26 is recorded once, survives a migrate that runs 011 and 025 again and one with nothing to do, and is applied once to the record an earlier build left, and that a record with 26 but not 3 is refused with 026's report kept. ledger_2x's undo_006 restores 001's report with its version, and every version pin gains 26. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ackfill pending once it has mapped the recent range (#743) On a production-sized ledger the share-hash backfill of migration 002 is most of what migrate does, and in the W1 cutover it must not sit inside the outage. migrate --defer-share-hashes now maps only the recent range before anything serves: every accepted legacy share whose template height is within 1,000, the coinbase maturity, of the highest one, from the first such share to the cursor's end, in 002's batches restricted to that height. It then raises the backfill's fence, share_hash_backfill_pending, from 1 to 2 in one transaction under the migration lock, after checking that nothing appended, and records the range's bounds beside the cursor. The step runs in the slot the backfill takes otherwise, before 013 drops the index that serves its bounds; 013, 017 and 024 follow with the cursor present. Plain migrate keeps rc.4's behaviour: the whole backfill, then 2. Serving on the recent range alone is safe because a native share is credited only on a job this cluster issued, whose parent is the tip or, within stale grace, the tip's parent. A header commits to its parent, so every copy of a legacy header has the same template height, and a credited native share can repeat only a header in the range unless the chain reorganized deeper than coinbase maturity survives. Every such header is mapped before anything serves, so the append's existing header check stays exact, and the range holds every copy of each header it maps, so the earliest copy still wins. At 2 every start of this release accepts the database with 2 unrecorded and logs a warning naming the cursor; the one start gate serves frontends, tools and operator commands alike. rc.4 understands the fence only at 1, so it refuses the database at connect and at migrate. A fence without its cursor is refused at 1 or 2, naming the value, and a value past 2 is a newer release's. A pending backfill at 2 is the operator's to finish: frontends started with PRISM_POSTGRES_INIT_SCHEMA=1, tools and operator connects leave it alone, and only plain migrate, through the new Ledger::connect_migrate, maps the rest while frontends serve. It runs after any other pending online migration, and its end stays where the cursor was planned: every row past it is a native share, which mapped its own header as it was appended. Before it records 2 it checks that the end lies at or below the conversion bound, and refuses, naming the share, if any native share repeats the header of an accepted legacy share, leaving the cursor and the fence in place. That check probes 2.x's header suffix index and the share ID index once per native row. It reads the native shares from the end up to the highest share_seq at its start, in chunks sized like the backfill's batches, each one short statement in a transaction of its own under the pool's statement timeout, and halves a chunk that times out: one statement over days of native shares would outlast that timeout, or hold a snapshot on the serving primary (#738). A share appended after the check starts cannot repeat a legacy header, since every one is mapped by then. A deferred run refuses a backfill a build before #669 started, which carries no fence and is never upgraded, and share-archive restore, and detach and drop of a partition below the cursor's end, refuse while any backfill is pending. The deferred run claims the backfill before it maps anything, with a deferred_at column the code adds beside the cursor, in the migration transaction that creates the cursor or finds it at fence 1. A deferred run that stops before serving is permitted leaves the claim, and then every other connect that migrates, a compose frontend's included, refuses the database as every start does. It runs neither the whole backfill, hours while the operator's retry waits for the runners' lock, nor 013, which drops the index the retry's range reads. A connect that planned the backfill before the claim reads it again once it holds that lock. Only the operator's migrate takes it on: --defer-share-hashes again, or plain migrate, chosen explicitly, which maps everything before anything serves. Without a claim nothing changes at fence 1. The transaction that raises the fence also keeps every native share at or above the cursor's end. It finds the ledger's share_seq sequence by its owning column, and when its next value would be below the end, as explicit inserts or a restore that left it uncalled can leave it, moves it to hand out the end next and logs the old and new values. A sequence that does not lag is not written to, so the cutover's evidence, which compares its last_value, sees the source's. And a plain migrate planned at fence 1, ahead of 017, that finds the fence at 2 once it holds the runners' lock, because a deferred run raised it meanwhile and stopped before 017, refuses before it maps a batch rather than at its record for want of 017's conversion bound. The recent range's first share is found through a materialized CTE: a bare min(share_seq) was planned as a primary-key walk that read the whole legacy table before reaching the range. Fourteen gated tests cover the deferral, the frontends' restraint at fence 2 and after a deferred run that stopped early, a connect queued behind a claim, a finish under concurrent appends with a frozen end, the double-credit refusal for both legacy ID forms and in the last of many chunks, the archive refusals, 013, 017 and 024 with the cursor present, a lagging share_seq sequence and an untouched healthy one, a finish that finds the fence raised while it waited, a resume of a part-way backfill, and the refusal of an unfenced one. The tests that used 2 as a newer release's value now use 3. The migration guide documents serving with the backfill pending. While migration 2's share-hash backfill was pending, share-archive detach and drop refused only a partition whose lower bound lay below the cursor's end, which kept the release table attached for the legacy shares the backfill still had to map. A native partition above the end could still leave: had the deferred backfill stayed pending long enough for a post-cutover partition to become archive-eligible, it could have been detached or dropped. The double-credit check that plain migrate runs before it records 2 reads the native shares through the attached parent, so the shares in that partition would have escaped it, and 2 could have been recorded over a header credited twice there. Detach and drop now refuse every share-ledger partition while the cursor exists, at fence 1 or 2, whatever its bounds. The refusal names the pending backfill and the remedy: finish it with qbit-prism-server migrate, then archive. Restore was already refused at any fence. Plan, seal, archive and verify are unchanged, as is every other archive eligibility condition. The doc comment, the start gate's warning and the migration guide say so. A gated test makes a native partition above the backfill's end ready to leave in every other respect: archived after the release table, sealed, verified and below the online horizon. While the backfill is pending its detach is refused, and so is its drop once it has been taken off the parent by hand. After plain migrate records 2, the same partition detaches and drops. The test that let a lead partition through now expects it refused. With the lower-bound predicate restored, both tests fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ts run, and bound its integrity report by its own deadline (#737) (#742) fatal-state clear captures the node's tip once and judges every observation against it. It then refused, before the integrity report and again before the commit, whenever getbestblockhash no longer named that tip. At production size a clear takes about a minute, nearly all of it the report, and tips arrive about once a minute, so a drill cleared about one attempt in three. Each failure rolled back for a retry, although a chain that only grew had invalidated nothing. Both checks now accept a pure extension. When the best block differs, the clear walks its headers back to the captured height, where the block must be the captured tip. A block commits to its ancestry, so every block at or below the captured height is then the one the capture saw: every pool-block and deep-fanout observation and the reconciliation built on them still hold, and maturity, computed at the captured height, can only come late. A best block at or below the captured height, another block there and a header the node does not know refuse as a reorganization, and so does a tip more than 1,000 blocks up; any other RPC failure propagates. The one observation the node was never asked for is that a pool block above the captured height is inactive, which the new blocks could falsify, so a moved tip still refuses while such a row exists. The event's reconciliation adds final_tip_hash, which descends from the captured tip, and tip_extended. The observations hold only if the node answered them from a chain through the captured tip. The chain can go over to a fork during the observation loop and come back onto a longer chain through that tip by the first check, which the walk alone would pass with stale observations. So the first check, once the tip has moved, asks the node every observation again and refuses on any changed answer. The check before the commit only walks, since nothing is observed after the first. A chain back on exactly the captured tip by the first check, which takes the fork being invalidated, and a second flap during the re-read itself still pass; the first passed before #737 too. The clear's hard-coded 120 s bound becomes fatal-state clear --timeout-seconds, 10 to 3600 and 120 by default, which clap checks before anything is read or connected. Ledger::clear_fatal_state_within takes the bound, and clear_fatal_state keeps 120 s for its callers. The report no longer runs under the session's statement timeout, 15 s by default. integrity_report_bounded reads the session's statement_timeout and sets a transaction-local one for the report in one statement, and puts the session's value back after it, never RESET, which would leave the server default. The clear gives it the time left in its bound less 5 s and one RPC timeout, for the commit and the closing tip check's node calls, and refuses with under a second left, so PostgreSQL cancels a report that runs out of time before the client gives up and nothing outlives a failed clear. self-check runs it in a read-only transaction with 300 s. Neither needs PRISM_DATABASE_STATEMENT_TIMEOUT_MS raised any more. The fatal-state tests' RPC proxy can now, before a chosen tip check, extend the chain by two blocks, move it onto a fork tip whose own ancestry alone shows the fork, shorten it, run it 1,001 blocks ahead, answer an unknown header, or flap it to a fork during the observation loop; the fake node's header for its tip carries its height. New gated tests cover an extension accepted at either check and when the node grows its own chain, each reorganization refused at either check with the reconciliation rolled back, a flap refused over a deep fanout and over a pool block, an extension refused over a pool block above the captured height, the report's own timeouts in self-check and the clear through the CLI under a 1 s session timeout, and a report that outlasts the bound cancelled by the server before the deadline. A config_cli test checks the flag's range before connecting. The operator guide describes the extension rule, the refusals, the retry guidance, the flag, and self-check's MVCC caveat on a busy primary. The closing tip check runs after the integrity report, and when the tip has moved it walks the new best block's headers back to the captured height: one getbestblockhash plus up to 1,000 sequential getblockheader calls. Before the report the clear kept back 5 s plus one node RPC timeout for it and the commit. A node that answers each header slowly, but within its RPC timeout, could therefore walk into those 5 s and let the bound's deadline fall inside the COMMIT, whose outcome would then be unknown. The closing check now runs under a deadline of its own, the bound less the commit's 5 s. A walk that cannot finish by then refuses with nothing committed and the halt kept, so the UPDATE, the event INSERT and the COMMIT always start with their 5 s. A test proxy mode answers a tip 500 blocks up with a 50 ms delay per header: the clear refuses about 5 s before its 10 s bound. Without the new deadline the same test runs into the bound itself and fails. The clear bounded everything, its COMMIT included, with one outer timeout. The closing tip check now ends 5 s before the deadline, but that left a fixed 5 s for the UPDATE, the event INSERT and the COMMIT. Under synchronous_commit=remote_apply, which the ledger's connections keep when the server sets it, a healthy COMMIT waits on a standby's replay and can take longer. The outer timeout would then cancel it in flight and report a failure whose outcome is unknown. The bounded part now ends with the event inserted and returns the open transaction, with the ORDER_LOCK hold guard. The caller commits it outside the bound and waits for the server's answer. An expired bound is therefore always before the COMMIT: nothing was committed and the halt stays, which its message now says. Only a connection lost during the COMMIT still leaves the outcome to be read from the event table. A deferred constraint trigger that sleeps 12 s inside the COMMIT stands in for the standby: the clear outlasts its 10 s bound and returns the committed event. With the COMMIT back under the bound, that test fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…th a plain parallel CREATE INDEX in one transaction when no instance is live (#745) The W1 cutover's migrate step runs with every frontend stopped, yet 013 and 024 build their indexes the online way: CREATE INDEX CONCURRENTLY reads the share ledger twice and waits out every older snapshot, and DROP INDEX CONCURRENTLY runs statement by statement. The plan puts the gain of building them plainly at about 3.7 down to 1.5 to 2 minutes of outage, to be measured in the pair rehearsal. `qbit-prism-server migrate --offline-indexes [--index-build-workers N] [--index-build-memory SIZE]` runs each index migration's plan in one transaction instead. N defaults to 4, at most PostgreSQL's 1024. SIZE is a maintenance_work_mem as PostgreSQL writes it, checked before connecting; without it the run takes 2GB, or the server's setting when that is higher. A build's sort takes about that much in all, divided among its leader and workers. The transaction takes the migration lock and sets a 5 s lock timeout, the workers and the memory for itself. Before any DDL it refuses an instance that has not reported drained or stopped and a live legacy writer lease, naming plain migrate, which builds CONCURRENTLY and records without stopping frontends, as the way out. It then locks the migration's tables ACCESS EXCLUSIVE in name order, runs each rendered definition verbatim as a plain build, drops with a plain DROP INDEX, verifies the declared set and records the version as it commits. Only a lock timeout reads as a transaction still holding a table; any other failure, in the guards or the table lock, surfaces as itself. An interrupted migration rolls back whole, so no invalid index and no partial drop remain while the migrations recorded before it stay recorded, and a rerun without the flag builds concurrently. The switch is the operator CLI's alone. MigrateOptions carries it beside #743's share-hash backfill mode, so migrate takes --defer-share-hashes and --offline-indexes together, as the W1 cutover's migrate step runs them: the recent range is mapped first, before 013 drops the index that serves it, and 013 and 024 then build offline with the backfill pending. Ledger::connect_migrate takes the options, or a ShareHashBackfill alone as before, through #743's shared operator connect; every other connect passes the default, so frontends with PRISM_POSTGRES_INIT_SCHEMA=1 run the same loop and stay on CONCURRENTLY. 013's SQL and registration are unchanged and no migration number is taken. Both modes share the preflight plan, the step loop and the record, and the legacy lease refusal is factored out of migrate_schema. Six gated index_trim tests watch offline runs through event triggers: the rendered definitions run verbatim in the transaction that records the version while an append waits; a live instance or legacy lease is refused before any DDL; an open writer or reader ends the run with 55P03 and nothing changed; a terminated run rolls back whole, after which a start without the flag builds concurrently; the operator's build memory is taken even below the server's, which is otherwise never lowered; and a lock timeout inside the guards reads as itself. A gated share_hash_backfill test runs the cutover's migrate step, both flags in one run on a populated 2.x.x source: the recent range is mapped at fence 2, 013 and 024 build offline with nothing concurrent, and a frontend then starts and serves. Unit tests pin the statements, the lock failure classification and the memory parser, and config_cli pins the flags. The docs describe the mode, its worker and memory needs and starting frontends only after migrate returns, and the isolated-restore runbooks pass the flag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, not on the persisted prepared row (#724) (#747) The async-promotion drill opens a Stratum session inside the replication gap and requires its job's prepared window to reach past the last share the standby replicated. It waited for that by polling qbit_prism_jobs every 200 ms for a saved prepared:live-0 row past the replicated share, then opened the session. A new session is issued the work the frontend has published in memory, though, and a refresh saves the prepared row before it publishes the work: reserve_fresh_compact commits the row, the work is checked against the node and the ledger twice more, and only then does publish() install it. A session opened in between is issued the work published before. At the drill's 1 s reanchor a frontend keeps its published work while shares arrive, rebuilding at once only for the first share, so when the gap is reached within about a second of the first share that work is still the first share's window. Required CI failed this way three times, each with a window ending at share 1 or 2 against 22 to 24 replicated (jobs 112476793983, 113062547171 and 113070225231), and #697 lists the same failure from local runs. The drill keeps its wait for the saved row, which still fails first for a frontend that never prepares work over the gap. It then opens the gap session in a loop bounded at 5 s, a few reanchor intervals, while the miners keep submitting in the gap. Each pass is bounded by what is left of the 5 s. It connects, takes the id of the session's first job from its first notify through a new ShareClient::job_id accessor, and reads that job's prepared window. A session whose window does not reach past the replicated share is dropped, and another opens 200 ms later. A first job whose issued row or prepared record is absent fails the drill at once, since issuance saves both before it announces a job. The first session whose window does reach past it solves the gap share and the gap block, which must both be on that job. That check replaces the old re-query of the block's window, which the shared notify made redundant. A replaced session solves and submits nothing, and the book records only the four scripted miners' submissions and kept work, so it and every count built on it are unchanged. The replaced sessions' windows are reported in the drill's summary line, and the timeline marks when the gap session opened. To prove it, a scratch 300 ms sleep between the reservation and the publication, never committed, made the old drill fail 8 of 8 runs with CI's message (last share Some(1) against 22). With the same sleep the drill passed 8 of 8 runs in its first form and 4 of 4 as amended, each run replacing one or two sessions issued the first share's window. Without the sleep, the first form passed 10 of 10 runs unpinned and 10 of 10 pinned to two CPUs, and the amended drill 3 of 3 unpinned and 4 of 6 pinned. Both pinned failures were the other setup race #697 lists, a COMMIT held across the fence answered as acknowledged, which happens after the gap session has opened. Interleaved and pinned, the base, the first form and the amended drill then passed 8 of 8 runs each. This addresses only the gap-window failure; #724's stale-job and backend-rpc-unavailable answers and #697's other items are separate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n throttled batches, and let the evidence export and self-check read it pending (#746) migrate --defer-share-hashes leaves migration 2's share-hash backfill pending at share_hash_backfill_pending = 2 while frontends serve. The new qbit-prism-server backfill-share-hashes maps the rest and records 2. Plain migrate on such a database now takes the same path, at the default throttle, instead of rc.4's unthrottled batches, with or without #745's --offline-indexes, which changes only how 013 and 024 build. A Throttle bounds every batch: at most --max-batch share_seq values (5,000), one statement in a transaction of its own, cancelled at --statement-timeout-ms (2 s, at most 5 s) and retried at half its size, and a rest of took * (1 - d) / d after each, where d is --duty-cycle (0.5). Every native append updates the cluster row under ORDER_LOCK, and a snapshot held on the primary keeps that row's dead versions (#738), so no statement holds one longer than its timeout, and none is held between batches. Each bound is checked in one place, which the flags' parsers call before anything connects. The chunked double-credit check that runs before 2 is recorded keeps to the same throttle. A batch or chunk that times out at its smallest size says to run backfill-share-hashes with a smaller --max-batch, or else a larger --statement-timeout-ms up to the cap; plain migrate's error names that command, as migrate takes no throttle flags. While frontends serve, the transaction that records 2 waits at most 2 s for the migration lock or the cursor table's lock, and its statements keep to the throttle's timeout. An attempt that waits longer is rolled back, holding nothing, logged, and tried again after a backoff from 2 s doubling to 30 s, 20 times, about ten minutes in all. The run then stops, naming what held it and to run backfill-share-hashes again, which goes straight to the double-credit check and the record. The record of a backfill before anything serves is unchanged. The command refuses, before it maps anything, a backfill that does not permit serving, and it shares A1's checks under the runners' lock: the fence and the claim, 017's conversion bound, the frozen end and the double-credit refusal. On a database whose backfill has finished it maps nothing and succeeds, so a retry after a lost reply is safe. It prints what it did as JSON, already_complete included. The recovery evidence export now reads a database at fence 2, as every start does. It exports the cursor as one share_hashes_deferred record in place of the partial mapping, and the summarizer marks the share_hashes kind deferred with it and says so in a note. The preamble, which every parallel session runs, reads only the catalog, and the cursor is read last, after the integrity report, so an export holds the cursor table's lock only until it commits, right after. The refusal of a cursor at fence 1 names migrate --defer-share-hashes to serve and plain migrate to finish; one that a build before #669 started, with no fence, names plain migrate alone. A complete export's summary is unchanged, and the parallel export and summarizer stay byte-identical to the serial ones. The export checks each capability's value against the highest that startup accepts, from one declaration that a unit test compares with NATIVE_CAPABILITIES. self-check samples the backfill beside the heartbeats, in one read of the cursor and its fence under one 5 s deadline, and reports it in a share_hash_backfill field with a warning line, never as a failure: "pending" with the fence and the cursor, or "unknown" with why it could not be read, a connection failure's text withheld. The field is absent only when the read succeeded and nothing is pending. The start gate's warning, migrate --defer-share-hashes's output and the share-archive refusals while the backfill is pending name backfill-share-hashes as the way to finish it. The migration guide and the ledger runbook describe when and how to run it after go-live, its throttle, how to watch it, the record's retries, and how to compare evidence while it is pending. The recovery test helpers now refuse an export script edited after the test binary was built. The serial export runs the copy compiled in, while the parallel exporter and the summarizer read the file, so a full suite whose script changed under it compared two scripts' exports: the deferred export test once failed so, at the line the edit had moved. The record of 2 while frontends serve retried a failed attempt without asking whether it had committed. An attempt can commit and still fail: its COMMIT's reply lost with the connection, or cut short after the commit was written. A retry then met the cursor that commit had dropped, failed on the missing table, and reported a finished backfill as failed. Now every failed attempt is followed, before anything else, by a read of what it left, in one statement of a fresh transaction under the record's own timeouts: whether 2 is recorded, and whether the cursor table exists. With 2 recorded and the cursor gone the backfill is done: the run logs that the attempt's reply was lost or cancelled but 2 is recorded, and succeeds as any finished run does. With 2 unrecorded and the cursor in place nothing committed, and a wait past a lock or statement timeout is retried as before. Either other state was made by hand, since the one transaction that records 2 drops the cursor in the same commit, and is refused loudly rather than retried. A read that fails too stops the run, naming backfill-share-hashes, which finds 2 recorded or records it. The statement timeout still covers the COMMIT. Under synchronous replication a COMMIT waits for the standby with its xid running and its locks held, which #738 rules out for long, and a cancelled wait leaves the commit local, which PostgreSQL reports as done with a warning. The re-check covers every way an attempt can fail after its commit, and a comment by the record's transaction says why. A gated lib test injects the failure through a test-only fault hook, in the pattern of the candidate claims' faults: keyed by schema, it lets the first record attempt commit and then return PostgreSQL's 57014. The run reports the backfill done, with 2 recorded and the cursor and the fence gone. A unit test pins the four states the read can find. The migration guide and the ledger runbook say what the run does after a lost reply. Every read of the share-hash cursor took any 42P01 for the cursor dropped since its look-up, by the transaction that records 2, and read it as nothing pending. But the read can miss another relation it needs: self-check's reads the fence from qbit_prism_schema_capabilities, so a database with its cursor beside a missing capability table, a damaged one self-check should diagnose, read as one whose backfill had finished, and self-check omitted both the field and the warning. Now a 42P01 reads as nothing pending only if the cursor is gone when it is looked up again, in one short statement. Otherwise the read's own error goes on, and self-check reports the backfill as unknown with it. Both readers of the cursor, self-check's and the one every start and migrate makes, go through that one place. The record of 2 has no such shortcut: its re-check after a failed attempt reads the catalog and the migration history, and a failure there stops the run. self-check's backfill test in ledger_postgres renames the capability table beside a pending cursor and requires state unknown with the missing relation named, and the warning. A gated lib test drops the cursor between its look-up and its read, through a test-only fault the record test's hook now shares, and requires both readers to read nothing pending. The lib tests move to share_hashes/postgres_tests.rs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Set VERSION and [workspace.package].version to 3.0.0-rc.5: rc.4 plus #737's fixes in #742 (fatal-state clear accepts a tip that only grew, takes --timeout-seconds, and runs the integrity report under its own statement timeout) and #744 (migration 026: the report computes each set once); the deferred share-hash backfill in #743 and #746 (migrate --defer-share-hashes, then the throttled backfill-share-hashes after go-live); #745 (migrate --offline-indexes, which composes with --defer-share-hashes); the cutover tooling merged after rc.4's tag (#728, #732, #736, #740); and #747, a test-only fix for #724. Both Cargo.lock files move only their workspace-member entries. doc/release-notes-3.0.0-rc.5.md, dated 2026-10-08, covers: - the changes, with the report's estimated 57 to 29 s since 026, the migrate --defer-share-hashes command measured on the pair at 5 min 06 s (539d6c9, before #745), backfill-share-hashes' throttle, record of 2 and safe rerun, the evidence scripts' changes, and the offline index builds' estimated gain; - upgrading: migrate is needed again (a binary-only upgrade refuses "missing migration(s) 26", and this export refuses a database without 026), the cutover's migrate step with both flags, and finishing the backfill after go-live under #738's snapshot rule, with self-check only at low share rates. Rolling back to rc.4 at fence 2 goes only by finishing the backfill first or from the pre-migrate backup or physical copy, and each rc.4 refusal is quoted where it applies: a start meets the cursor first and says to run migrate, which rc.4's migrate refuses. The rules still in force: #737's statement timeout for rc.4's tools, 026 or not, #738's snapshot rule, and fatal-state clear --timeout-seconds, whose bound ends before the commit; - the same evidence set as rc.4's, with each PR's merge SHA, the pair's serving evidence carried, and the rehearsal's full measurements left to the cutover's go/no-go record; - the known issues: rc.4's, minus #737's F1, F2, F4 and F5, plus #737's open part, #738 (its fix planned for rc.6), #739, #741, #731 and #734, with #582 closed and #712 and #724 updated. docs/prism-rust-migration.md now says how earlier candidates refuse a database at fence 2: a start with the backfill's own refusal, whose advice to run migrate holds only for this release, and their migrate, or a start with PRISM_POSTGRES_INIT_SCHEMA=1, with the fence. The development-line notes name 3.0.0-rc.5, and they and the 3.0.0 notes link all five candidates' notes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…its own one dropped (#748) (#750) After a failed attempt to record 2 while frontends serve, the run reads whether the attempt recorded 2 all the same, before it retries. It read on the attempt's own connection. When that connection had dropped, its COMMIT's reply lost with it, the read failed too, so a run whose COMMIT had recorded 2 stopped with "run backfill-share-hashes again". That was safe, the rerun finding 2 recorded, but it reported a finished backfill as failed. Now the read runs on a fresh connection when the attempt's own one is lost: when the attempt or the read on it failed with a transport, TLS or protocol error, or the server ended the session (SQLSTATE class 08, 57P01 to 57P05, 25P03), or the attempt got no reply within its deadline. The fresh connection comes from the run's own connect options, the operator pool's or the migrating connect's: the same URL, search path and TLS. Its read keeps the record's 2 s lock timeout and the throttle's statement timeout. A lost session is most often a restart, a crash or a failover, so a connect or read that meets one is tried again after 250 ms, doubling to 2 s, within the throttle's statement timeout plus 2 s plus a 10 s connect allowance. Each fresh connection is closed again on every path, within a second, or dropped. With 2 recorded and the cursor gone the run succeeds. Only if the fresh read fails too does the run stop, with both failures named. Each attempt is also bounded on the client's side, by its ten statements' statement and lock timeouts plus 5 s. The server's own timeouts end a healthy attempt well within that, so an attempt still without a reply is on a half-open connection, a partition say, whose COMMIT would otherwise wait out TCP's retransmissions for some 15 minutes. It is abandoned and its connection taken for lost. The caller's connection is never replaced. The run reports whether it lost that connection, and the caller then drops it quietly rather than close it, so a lost connection's failing close cannot turn a finished run into an error, and the connection cannot become a session without the runners' lock or the pool's settings. A lost connection that had recorded nothing stops the run instead of a retry. The runners' lock, a session lock on that connection, went with it, and another runner may have taken it since. Retrying would mean taking it again and redoing the checks made under it, the fence, the frozen end and the double-credit check, which is what a rerun of backfill-share-hashes does. The stop says whether the attempt or the read after it lost the connection, keeping the attempt's own error and the read's, and that a COMMIT still in flight there can land moments later. Only an attempt whose own connection answered the read is retried, which one match makes plain. Seven gated lib tests drive the record against a real PostgreSQL, five through the crate's execution proxy. A statement trigger marks the record's DELETE of the fence, since the start gate refuses any trigger on the migration history. - A COMMIT that completes before its connection drops ends in success, with 2 recorded and the cursor and the fence gone. - One whose fresh connects are refused, then told 57P03, reads 2 recorded on the third. - One whose reply is held past the attempt's deadline is read afresh then. - A drop before the COMMIT stops the run with 2 unrecorded, and the rerun records 2. - A test-only fault fails an attempt with 55P03 and terminates the connection before the read: the stop keeps both errors. Unit tests pin which SQLSTATEs and errors read as a lost connection and the client-side deadlines. The migration guide's record paragraph and refusal table and the ledger runbook say what the run does after a lost connection, and that 2 was not recorded when checked. The record of 2 reported a lost connection only on success: the run returned RunConnection::Lost, and the caller dropped the connection rather than close it. When a record attempt lost its connection, or got no reply on it within its client-side deadline, and the fresh read then found 2 unrecorded, the run returned the rerun instruction as an error, and both callers, backfill_share_hashes and apply_online_migration, closed the connection the run had given up on. A close writes Terminate and shuts the socket down. On a half-open connection whose send buffer is full that write could wait on TCP, defeating the attempt's new bound instead of returning the instruction at once. Now the lost state travels beside the result. The run marks a RunConnection out-parameter lost the moment an attempt, or the read after it, loses the connection, before the fresh read and whatever the run then comes to. Both callers close the connection through close_unless_lost, which drops a lost one without a word to the server on success and failure alike. Nothing else closes a lost connection: the fresh read closes only its own connection, within a second, and the operator's pool never held the detached one. A gated lib test holds the reply to the record's DELETE, before any COMMIT, through the execution proxy. The attempt is abandoned at its client-side deadline, shortened to 3 s for its schema, as the held transaction holds the database's migration lock that the binary's other tests need. The run stops with the rerun instruction within a few seconds of that deadline, with 2 unrecorded and the cursor and the fence in place. The proxy, which now records the connections whose client sent Terminate and the relays that ended, sees the lost connection dropped, never closed. On loopback a close of that connection returns at once, so the Terminate is what tells a close from a drop. The record test of a connection lost while checking now also requires it marked lost. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t the payout revision the final block's confirmation set (#724) (#753) The async-promotion drill's last check sends a share from each surviving session on work for the tip the final block moved to, and requires it accepted and credited once. It took the first job built on the new tip as that work. The final block moves the payout revision twice on the same parent: once when a frontend first records the new tip (window.rs), and again when the landing confirms the block (blocks.rs). Work published on the new tip between the two carries the first revision. Once the confirmation commits, a frontend answers shares on that work stale-job for the payout revision until it republishes: the landing frontend at once, the others at their next refresh (#752). A session whose first new-tip job came from that work was answered stale-job. Required CI failed this way twice (jobs 112241378522 and 113174599773). Each session now waits until its latest job on the new tip comes from work at the ledger's payout revision. The drill reads the job's prepared record and the cluster's revision on the promoted primary, and reads past a job on superseded work to the session's next one through a new ShareClient::newer_work. A job without its issued row or prepared record is passed over the same way, as the final block's loop does (#619). If the revision moves past a share's work after that check, the refused share is waited out once its ledger row is checked absent. Any other refusal fails, naming the job's and the ledger's revisions. A session may meet work on one superseded revision, possibly on more than one job; a second would be a revision move this drill never makes, and fails. The original symptom stays as a check. The first job on superseded work the drill meets gets one share, which must be refused and leave no ledger row. It is answered stale-job, or unknown-job if the job was retired past PRISM_STRATUM_SAME_TIP_JOB_RETENTION_SECONDS (30 s) and pruned before the drill reached its session: a pruned job is resumed from its durable rows only while its frontend still publishes that payout revision. Every step of the phase (the tip read, the revision and ledger reads, both submissions, the retries and the wait for newer work) runs within what is left of one 30 s deadline, through a small PhaseDeadline helper. An expiry names the phase, the session, the job and how long the phase had run, instead of waiting out the client's 70 s answer timeout. With scratch server delays that reproduce CI's message on every run, the old drill failed 10 of 10 runs and the amended drill passed 6 of 6. Unpinned and pinned to two CPUs it passed 6 of 6 each, and after each Codex fix 3 of 3. One further scratch held the kept share 35 s past retention; it was answered unknown-job and accepted with no ledger row. Only the drill changes; #752 tracks the server's stale-job burst after each of the pool's own blocks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…no plan walks the ledger (#754) The pool hashrate series serves fully rolled buckets from the rollup tables. It re-reads two kinds of rows from the ledger: the boundary (the current bucket and the leading partial bucket) and the tail (the shares past the rollup watermark). Neither read gave the planner an index range it could use. The boundary bounded accepted_at with an OR and a NULL guard on the range start. The tail bounded share_seq only from below, with the same guard on time and an OR on the miner. On a production-sized ledger (66M shares, rollups current), rc.5's default pool request walked all of p0's seq_walk index for the boundary, removing 66,090,385 rows to return 2,390, in 19-40 s. With the boundary fixed, the generic plan's tail still walked p0's accepted_recent_idx for no row, in 17-27 s. Pool hashrate-series requests therefore ran past the public read deadline: PRISM answered 503 read_timeout, and the proxy in front of it 504. A custom plan also walked a big miner's whole history. 2.x builds the same read with bounded predicates, so this is a 3.x regression. The boundary is now two disjoint slices, each a range on accepted_at bounded at both ends: - [GREATEST(current_bucket, started_at), ended_at]; - [started_at, LEAST(first_full_bucket, current_bucket)), which is empty without a range start. The tail takes its shares from a subquery bounded by share_seq on both sides, by the watermark and by max(share_seq), which is one backward probe per partition. OFFSET 0 fences the subquery, so the time and miner tests only filter its stream. The tail therefore reads only the shares past the watermark: none while the rollups are current, and the lag while they catch up. With no progress row it reads every share, as rc.5 did. On a production-sized ledger the fixed query took 9-23 ms for the pool and 2-15 ms for miners. That held in 20 of 20 runs, across generic plans, custom plans with the anchor pinned or NULL, and auto plans after six executions. Its output equalled rc.5's in 7 of 7 anchored requests, for which rc.5 took 15.8-25.2 s. On a 6M-share copy with a 600k-share miner, every request ran in 1-17 ms at random_page_cost 4 and 1.1. There the output matched rc.5's in 104 cases per watermark state, 312 in all. The only exceptions were unanchored requests whose range start slid while the old query ran, and they differed only in the leading bucket. Two tests in api_2x pin this. - chart_rollups_match_raw_for_every_range_bucket_and_watermark requires the rollup series to equal the raw-ledger series in 78 cases (every range and bucket pair, context, anchor and subject). It checks them before the rollups run, at a current watermark, and with shares past it. - chart_rollups_read_only_the_partial_buckets_under_any_plan runs the chart's plans over 20,000 rolled-up shares, half of them one miner's. It covers the pool at three ranges, a small miner and the big miner, in custom and generic modes, at random_page_cost 4 and 1.1, with the watermark current and again with 300 shares past it. Each plan may read at most 1,000 ledger rows plus the lag. Every ledger index scan outside a Limit probe must bound one column from both sides, and no sequential scan may read ledger rows. It fails against rc.5's query and against the boundary-only version. The change is read-only SQL in the public read path: no migration, no schema change, no write path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Set VERSION and [workspace.package].version to 3.0.0-rc.6: rc.5 plus #754 (the pool hashrate series bounds both of its ledger reads, the boundary buckets and the tail past the rollup watermark, at both ends, fixing a 3.x regression that timed it out on large ledgers), #750 (#748: a failed record of 2 is re-checked on a fresh connection when its own one was lost, and each attempt is bounded on the client's side), and #753, a test-only fix for #724. Both Cargo.lock files move only their workspace-member entries. doc/release-notes-3.0.0-rc.6.md covers: - the changes, with #754's figures from its pull request (19 to 40 s and a 504 on a 66M-share ledger before, and 17 to 27 s with only the boundary fixed; 1 to 17 ms on a 6M-share copy after, under every plan mode and random_page_cost tried, with its output matching rc.5's; and on a production-sized ledger, 9 to 23 ms for the pool and 2 to 15 ms for miners in 20 of 20 runs, against rc.5's 15.8 to 25.2 s, with identical output), #750's lost connections, fresh read, client-side deadlines and two new stops, and #753's wait for current work; - upgrading: rc.6 adds no migration, capability, setting or script, so an upgrade from rc.5 and a rollback to it replace the binary only, at fence 2 too, and a database that rc.5's migrate migrated is served by rc.6 unchanged, which a cutover that pinned rc.5 can rely on. An upgrade from rc.4 or earlier is as in rc.5's notes, which the section links: migrate for 026, fence 2 and the deferred backfill, #738's snapshot rule and fatal-state clear --timeout-seconds; - the same evidence set as rc.5's, with the pair's serving evidence carried; - the known issues: rc.5's, with #748 fixed and #724's drill race and new-tip shares fixed on the test side (#747, #753), plus #752 after the cutover, #738's fix now planned after the cutover, and #739 not in this candidate. The development-line notes name 3.0.0-rc.6, and they and the 3.0.0 notes link all six candidates' notes. docs/prism-rust-migration.md names rc.5 where it said "this release" of serving at fence 2 and finishing the backfill. That is in three places: that every build before rc.5 refuses fence 2, that the starts of rc.5 and later accept it, and that only rc.5 and later finish a pending backfill. In this candidate's tree "this release" would have excluded rc.5, which does both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ion after 20 s, below qbitd's 30 s idle close (#759) (#760) * 3.x.x: drop an idle node connection after 20 s, below qbitd's 30 s idle close, and send each relay call on a new connection (#759) The frontends' qbitd RPC client kept an idle keep-alive connection for 60 seconds, while qbitd closes one after its -rpcservertimeout, 30 seconds by default. A request written into a connection just as the node closes it is never read, and fails after the connection existed. That is the plain "transport failed" error, which nothing retries. On submitblock it is an unknown outcome, and the block is never offered again. A production deployment logs that error from the CTV broadcaster on about one pass in eight to fifteen, which fits this race: the broadcaster's 30 second interval runs in step with the 2 second refresh interval. The client now keeps an idle connection for at most 20 seconds (POOL_IDLE_TIMEOUT), so it drops one before the node can close it. A compile-time assert keeps the bound below QBITD_DEFAULT_RPCSERVERTIMEOUT. node_client_builder() carries the bound, and the public API's node client and the load harness's node clients are built from it. Rpc::with_pool_idle_timeout sets another bound, for a node with a shorter idle close and for tests. Every RELAY_METHODS call now goes through a second client that keeps no idle connection, so each relay opens a new one. A relay can then fail only to connect, which is provably unsent, or on the connection it just opened, which is a genuine unknown rather than this race. A node with a shorter idle close than the bound, or a proxy that drops idle connections, can no longer race a relay. A relay is rare, and the new connection costs one connect. tests/rpc_idle_close.rs runs the real client against a loopback node that drops a request arriving after its idle limit, as the race does, and counts the connections it accepts. With a shorter bound, the next call opens a new connection and is answered. With the old long bound, it reuses the stale one and fails with the production error, which is not RpcNotSentError. A relay call after the pause is answered on a new connection even with the long bound. The migration guide notes the node-side requirement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.x.x: keep relay calls on the bounded pool, and build Rpc's client from node_client_builder (review on #760) A new connection for each relay put its connect inside submitblock's 1 second deadline (PRISM_BLOCK_SUBMIT_RPC_TIMEOUT_SECONDS). A deadline that passes while connecting is a timeout, not a connect error, so the block would count as an unknown offer and never be offered again: a lost SYN or a slow accept would cost a block that never reached the node. A warm pooled connection within the 20 second bound is the safer path, so relays share it again, as they did before this PR. Rpc's client now starts from node_client_builder(), like the public API's and the load harness's. The pool_idle_timeout accessor and its test only restated the constant, so they are gone. The docs name a proxy's idle timeout beside qbitd's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…t-audits (#765) (#770) import-audits no longer holds SETTLEMENT_LOCK while it sends a legacy audit's UPDATE. Each row's parameters are encoded off the runtime before any transaction opens. The UPDATE then runs on a connection of its own, under transaction-local write timeouts (PRISM_IMPORT_AUDITS_WRITE_TIMEOUT_MS), holding only that row's lock. After that the importer polls SETTLEMENT_LOCK with pg_try_advisory_xact_lock for up to PRISM_IMPORT_AUDITS_LOCK_TIMEOUT_MS, and only the write guard and the COMMIT run under it. A SETTLEMENT_LOCK holder waiting on the row preempts it, and a row that exceeds a bound is rolled back and named in the error, so a re-run resumes at it. SIGTERM and SIGINT stop the command between rows, or roll back the row in progress. Each row logs its parameter sizes and phase times on stderr. Fixes #765. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* 3.0.x: version 3.0.0 and its release notes (#557) 3.0.0 is 3.0.0-rc.6 plus #770 (#765) and #760 (#759), the two fixes #772 prepared as 3.0.0-rc.7, which was never tagged. The 3.0.x branch starts at #770's merge, 4c13491, before 3.x.x took the 3.1 dual writer. - VERSION, [workspace.package].version and the workspace-member entries of both Cargo.lock files move to 3.0.0. - doc/release-notes-3.0.0.md becomes the 3.0.0 notes: #772's rc.7 notes as the changes since rc.6, then the D5 recovery contract it held. - The development-line note and the migration guide name 3.0.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.0.x: update rustls, rustls-webpki and webpki-roots for six RustSec advisories cargo deny fails on rc.6's lockfile with RUSTSEC-2026-0346, -0347, -0348 and -0349 (rustls < 0.23.46), RUSTSEC-2026-0382 (rustls-webpki < 0.103.16) and RUSTSEC-2026-0381 (webpki-roots < 1.0.10). Move each to its first fixed patch release with cargo update --precise, in the root and fuzz lockfiles, as #354 did for RUSTSEC-2026-0285. No other dependency changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * 3.0.x: describe 3.0.0 as this line's production release The development-line note's status, not-done and upgrade sections, and the README banner, said no production release of the Rust server existed. With the coordinator's go-ahead for 3.0.0 they now point to 3.0.0 on 3.0.x and its notes, and say 3.x.x continues as 3.1. The 3.0.0 notes list the TLS dependency update. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
3 of 6 tasks
kiwidream
marked this pull request as ready for review
October 22, 2026 17:50
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
2 tasks
Merge 3.0.x at the v3.0.0 tag (aabe470) into main and take its tree, as #246 did for 2.0.0. main's own commits since 2.0.0 are workflow syncs from 3.x.x, which 3.0.x already carries. Two promotion changes: - Keep docs/releasing.md, which only main has. Update it for the 3.x branches, the workspace version, squash-only merges and the L3 tag run, and record that v3.0.0 was tagged on 3.0.x before promotion. - Make the README banner name the version, not the 3.0.x branch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kiwidream
force-pushed
the
release-3.0.0-main
branch
from
October 22, 2026 18:55
123b760 to
75116cd
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 75116cd328
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Promotes the released v3.0.0 to
main, followingdocs/releasing.mdand the 2.0.0 precedent (#246). The repository allows only squash and rebase merges, so this lands as one squashed commit.What
mainbecomesmaingets thev3.0.0tag's tree (3.0.xataabe470f, #776) plus two promotion changes:3.0.xrelease branch".docs/releasing.mdexists only onmainand is kept. It is updated for the 3.x branch layout (3.x.xdevelopment and 3.1,3.0.xrelease,2.x.xmaintenance), the workspace version and both lockfiles, squash-only promotion, a version-generic tag and release command, and the L3 tag run. It records thatv3.0.0was tagged on3.0.xbefore this promotion, so merging this PR adds no tag.The release notes, version and code are byte-identical to the tag.
main's own commits since v2.0.0 are not lost. They are the PRISM load-harness workflow syncs from3.x.x(#538, #567, #583, #595, #601), and the tag'sprism-load-nightly.ymlis byte-identical tomain's. The 2.x Python runtime files (lab/,tests/, the oldtest/test-prism-*.shscripts and so on) are removed, as on3.x.x.L3
l3-fullis red, as it is on thev3.0.0tag's own run, and both runs are recorded in #779:throughput-400k-window-2fe-async-3-blocksrefuses one valid share per run with 3.x.x: a share under a tip-change lease is refused 'commit gate closed' when the authority try_read meets a writer, which can drop a valid share's credit and fires PrismShareAppendFailures #716'sauthority-unavailablecommit-gate closure. This is open and known, and listed in the 3.0.0 notes.throughput-400k-window-2fe-sync-r3fell 479 offers short once.The matrix can't go green on 3.0.0's code until #716 is fixed. The tree here differs from the tag only in the README and this guide, so this PR's new L3 run will show the same result.
After merge
maininto3.x.xand into3.0.x(docs/releasing.md, "Reconcile the development branch"). Don't reset or force-push either branch.🤖 Generated with Claude Code