Repository navigation
v3.1.0 chore: sync released main into 3.x.x - #786
Merged
Merged
Conversation
The scale fixture step failed on main with:
psql: error: connection to server on socket
"/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
The Postgres image initialises a fresh data directory by running a
temporary server that listens on the unix socket only, then shuts it down
and starts the real one. `pg_isready` over that socket reports "accepting
connections" during initialisation, so the harness proceeded against the
throwaway server and the very next call raced its shutdown -- hitting
`database "qbit" does not exist`, `the database system is shutting down`,
or the missing socket once the file was removed.
The window is roughly a second, which is why the scale step was the one to
break: the preceding ledger step had already pulled the image, so its
container reached the temporary-server phase inside the wait loop's first
probe.
Gate on the TCP listener instead -- absent until the real server is up,
since the temporary one runs with listen_addresses='' -- and confirm with
a real query over the socket the tests actually use. The probe now lives
in test/prism-postgres-lib.sh and replaces the identical racy loop in all
six PRISM test scripts.
Measured against postgres:16-alpine, 30 psql calls issued immediately
after the gate: 19-21 failures per 30 with the old socket probe, 0 per 30
with this one.
* Shard PRISM coordinator tests by domain
Re-cut of the original test sharding against current 1.x.x (36267d6): the
two monolithic coordinator test files split into nine domain shard files
plus two shared no-environment support modules, with the current test
bodies as the source of truth. All 609 sharded test methods are conserved
exactly once, and dependent test files change import paths only.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Extract PRISM coordinator core owners
Re-cut of the original core-owner extraction against current 1.x.x
(36267d6). Fifteen owner modules leave prism_coordinator.py at 15,436
lines (from 31,041): coordinator_config (immutable snapshot construction,
all env reads), rpc (JsonRpc + transport retry policy), bounded_executor,
ctv_runtime, progress_health, template_artifacts, bundle_compiler,
job_bundle, job_delivery, tip_refresh, stratum_session, share_writer,
payout_state, background_services (incl. the ledger lease heartbeat
service), coordinator_shutdown. The old extraction commit served as the
placement and wiring map; every moved body is the current coordinator
implementation, so all upstream fixes through #120 are carried, including
the landing-class deadlines, degraded accepted-parent previews,
append/anchor fencing, tip-refresh epoch waves and lease fencing.
Coordinator compatibility facades, private-name aliases and descriptor-
routed legacy fields preserve every existing import path and test seam.
share_ledger gains typed exact/conflict share replay consumed by the
share writer; the Postgres ledger shell test asserts the new replay
semantics. Nine owner-boundary test files are added (116 tests) and
support modules gain typed-recovery fakes; no existing behavioral
assertion is weakened.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Extract PRISM audit artifact ownership
Re-cut of the original audit-artifact extraction against the rebuilt PR 74
layer. lab/prism/audit_artifacts.py owns every audit path and filesystem
mutation under one identity-checked root: candidate-artifact lifecycle with
compiler inode-transfer adoption, ordered publication (thread + lock-file
process fencing) keyed by a durable audit publication ordinal
(SQL sequence/column/backfill migration + ledger confirm/reactivate/floor,
with an in-memory pool-block state machine for the compatibility ledger),
legacy-evidence upgrade, and the #88 gap-recovery flow with the ledger
range backfill threaded in as a per-call callback. The coordinator and
PsqlShareLedger keep one-line delegating shims; verify_bundle honors the
configured bundle build timeout. Bodies for pre-existing behavior come from
the current tree, so all upstream fixes through #120 are preserved; the
Postgres ledger shell gate gains publication-ordinal assertions plus three
A1 gates (transition parity, migration, cross-process fencing), and the
canonicalizer round-trip contract is pinned by new crate regression tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Extract PRISM block candidate submission
Re-cut of the original B1 extraction against the rebuilt PR 75 layer.
lab/prism/block_candidates.py owns the durable candidate codec and the full
submit-before-accounting lane from the current tree: node-submission
workers with hard RPC deadlines, same-hash disposition leases and terminal
outcomes, the accounting queue/thread handoff, startup and outbox replay
with quarantine, acceptance evidence and chain probes, retry
merge/pacing/backoff, block-work heartbeats and the DB/RPC call workers
including the landing-class deadline split with per-block escalation and
the startup enumeration-owed gate. The coordinator keeps the node
submission finalization side, thin delegates, and descriptor-routed
compatibility state; replay adoption registers credit floors through the
share-writer owner before preview registration so one bad row cannot abort
the rest.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Extract PRISM share submission
Re-cut of the original share-submission extraction against the rebuilt
PR 76 layer. lab/prism/share_submission.py decomposes the submit hot path
into pure parse/validate/classify functions plus ShareSubmissionService,
with bodies from the current tree: the cross-connection retained-submit
authority (same-tip-only guard, retained version mask, original
extranonce), stale-grace block-solve drop accounting ahead of
low-difficulty classification, and the durable-outbox fast paths in both
block routes wired to the block-candidate owner's facades. The recent-
share duplicate window becomes RecentShareIndex with oldest-insertion
eviction, and the service owns share-ack histogram state behind the
existing coordinator routing seam. The coordinator keeps a one-line
decorated handle_submit, the rejection-constant re-exports, and
descriptor-routed compatibility state.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Extract PRISM vardiff service and block finalization
Re-cut of the original vardiff/finalization extraction against the rebuilt
PR 77 layer. lab/prism/block_finalization.py owns the post-offer B3 lane
from the current tree — admission with acceptance-evidence short-circuits,
landing under the append-epoch fences and window-proof decision, the
CTV/share-credit tail, ordered audit publication through the PR 75 store,
and accepted-block accounting — packaged in the six bounded finalization
phases with candidate-interarrival metrics. The block-candidate owner's
ports are untouched: its targets remain the coordinator facades, which now
delegate to the service, and landing round-trips through the coordinator
seam so existing monkeypatches intercept. lab/prism/vardiff_service.py
owns vardiff configuration, share windows, the idle retarget engine and
its executor state behind twelve adopted compatibility fields, keeping the
difficulty-direction clean-jobs rule. The B3 decision document is
refreshed for the node-first accounting-actor architecture.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Extract PRISM observability and audit HTTP
Re-cut of the original observability extraction against the rebuilt PR 78
layer. lab/prism/observability.py owns mining-delivery and semantic-
currency policy, the cached base-health snapshot with live progress
overlay, the starting-state 503 and warm-up loop, and a new cached
complete-metrics snapshot pipeline with bounded collector diagnostics.
lab/prism/audit_http.py owns the audit listener lifecycle and route
handler behind a typed port, with genuinely bounded shutdown: a stop
racing startup before the serve loop commits now cancels the startup
window and returns within its deadline, pinned by new socket-lifecycle
regressions. The coordinator keeps delegating shims, the health-attribute
compatibility properties, the renamed complete-metrics renderer, and the
listener-binds-before-warm-up startup ordering with its phase stamps; the
compose environment gains the metrics refresh interval alongside every
existing pass-through.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Complete PRISM coordinator ownership cleanup
Re-cut of the original terminal cleanup layer against the rebuilt PR 79
stack. lab/prism/metrics.py owns the complete Prometheus payload and the
extracted formatter bodies behind copied owner snapshots, proven
byte-identical to the previous renderer by a frozen-reference parity
test. lab/prism/reorg_reconciler.py owns reconcile flight admission, the
per-tip trusted memo, lookup counters and the bounded proving prefetch
executor around the current core state machine, while the coordinator
keeps the exact serialized writer/payout/landed-preview adapter and
compatibility descriptors. The watchdog moves into a WatchdogService with
the coordination-blocked streak, classification, bounded lease release
and unconditional hard exit behind the dynamic coordinator seam that the
lease heartbeat service also fires. BlockFinalizationService drops its
implicit attribute forwarding for an explicit typed port with mechanical
coverage, the interim health compatibility properties and forty-plus
compatibility re-exports are deleted with every caller repointed, and
white-box tests now target the owning services.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Document PRISM refactor review stack
Re-derived documentation layer: the review-stack guide, ownership map,
invariants, roadmap, A1 contract and validation evidence, updated for the
rebuilt stack against current 1.x.x — the node-first finalization split,
the re-measured structural audit, and validation results actually produced
on this tree (Docker-backed Postgres/A1 gates, Rust workspace, container
compile and image build, compose validation; live Stratum targets recorded
unavailable for the missing qbitd prerequisite). The ledger operations
guide keeps its current landing/replay text and gains the known-outcome
replay-resumption contract; the audit publication ordinal comment in the
finalization owner is refreshed for reactivation reuse.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add a deterministic concurrency harness for PRISM, proven against the writer-lease lifecycle (#139)
* Add a deterministic concurrency harness and prove it against #123
The concurrency defects this line keeps shipping share a shape: two owners
interleave around a PostgreSQL row, and the bad schedule is too rare to
reproduce by running the coordinator and hoping. Every one was found in
production or in review, never by a test. The nine-slice refactor removed the
structural reason for that, so this adds the missing tool: a scheduler that
makes an interleaving an input rather than an accident.
The harness has three parts. A baton scheduler runs real threads one at a
time, stopping only at named checkpoints placed at PostgreSQL statement
boundaries, so the controlling test decides who runs next and the resulting
schedule is order-stable rather than merely likely. A virtual clock backs
every interval the lease lifecycle measures about itself, so nothing sleeps
in wall-clock time. An in-memory PostgreSQL model covers the semantics the
lease lifecycle actually depends on: tuple locks held for a transaction's
lifetime, FOR NO KEY UPDATE SKIP LOCKED declining to queue, per-statement
READ COMMITTED snapshots, lock and statement timeouts, session-scoped
advisory locks, and pg_stat_activity locker attribution against a
snapshot-visible xmax.
Substitution runs through ports rather than monkeypatching. LedgerSqlPort and
LeaseGuardPort name the two seams PsqlShareLedger already wrote through, and
the constructor now takes sql_backend_factory, lease_guard_factory and
monotonic alongside the existing lease_retry_sleep. No bound method is
reassigned and no clock is patched, so the code under test stays the shipped
code: the real retry loop, the real CAS SQL, the real adoption arithmetic,
the real fail-closed branches.
FakePostgres answers statements by recognising them against fragments lifted
from the production SQL, and raises on anything else. Statements belonging to
the state machines deferred to follow-up PRs are recognised only well enough
to name the follow-up. A fake that silently answers an unrecognised statement
drifts from production without anyone noticing; this one fails instead.
Issue #123 is the proving ground. Its trigger sequence now runs without a
live Postgres: a coordinator vanishes in the gap between a deadline-scoped
statement completing and its COMMIT, leaving an open transaction holding the
lease tuple lock, and a failover twin parks on that lock inside
PsqlShareLedger.__init__ with no deadline anywhere in the system that can
release it. The schedule is identical across repeated runs.
One test in this commit fails, deliberately.
OrphanedLeaseLockFailoverBoundTests asserts the bound #123's fix must
establish, and #123 is not fixed on this line: the startup lease upsert runs
with no lock_timeout, so nothing bounds the successor's wait. That assertion
is written against the fixed behaviour rather than against today's, because
weakening it to green would delete the only executable statement of what #123
asks for. It turns green when the fix lands.
Refs #128, #123
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Cover the remaining writer-lease interleavings, and two model fidelity fixes
Completes the writer-lease lifecycle as the harness's proving ground. Three
scenario modules, each driving shipped code through the harness rather than
re-describing it:
Renewal against a competing expiry claim. The heartbeat renews uncontended;
an expired row whose skipped lock belongs to a different-identity claimant
fails closed, and the control proves it is the combination that raises rather
than either half; the documented attribution misfire — this coordinator's own
fenced write committing between the verification statement's MVCC snapshot and
its live pg_stat_activity probe — is driven at exactly that checkpoint and
resolved by exactly one recheck; a claim that commits fences the session.
Heartbeat during a long fenced statement. The verification crosses an
in-flight fenced write without ever queueing, skipping renewal and resuming it
the moment that write commits. The control runs the pre-fix locking spelling
into the same lock and watches it die on the guard's statement timeout, which
is what makes renewed_count == 0 mean "declined a lock that was really there"
rather than "nothing was attempted". An own fenced write outlasting the TTL
defers instead of self-fencing, and the same expired row locked by a foreign
application_name still fails closed.
Takeover and epoch fencing. The advisory guard keeps a same-identity successor
out of the lease upsert entirely; adoption silence is measured from the
successor's own guard acquisition even when the row is already far staler than
the interval; the CAS rewrites the session token and fences the predecessor;
a lost CAS re-observes after a fresh full interval instead of refusing the
token permanently; and a different epoch refuses an unexpired lease while
taking an expired one through the ordinary expiry CAS.
Two fidelity fixes to the model, both found while reviewing the above.
A cancelled lock wait now reports which bound fired. A caller deadline arrives
as SET LOCAL lock_timeout, while the guard session's bound is a
connection-level statement_timeout; PostgreSQL reports them with different
messages, and the whole point of the SKIP LOCKED heartbeat is that the guard's
bound is the fatal one, so blurring them would let a scenario assert the wrong
mechanism.
Vanishing a coordinator no longer fabricates an orphan on a path that cannot
produce one. An explicit transaction's COMMIT is a separate client message, so
a vanished client leaves it open forever; an autocommit statement has no such
message and PostgreSQL finishes and commits it. #123 is explicit that the
orphan shape did not exist before the deadline plumbing arrived, and a model
that offered it on the autocommit path would argue for a fix to a defect that
path cannot reach.
Determinism: eight assert_deterministic tests at 25 repeats each, every one
asserting an exact checkpoint schedule rather than merely that runs agree.
Twenty consecutive full runs of the five modules produce an identical result.
Refs #128
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Correct the PostgreSQL snapshot model and the #123 bound
An adversarial review pass verified the model's PostgreSQL claims against a
scratch PostgreSQL 16.15 cluster and found two defects that mattered.
The model re-read committed state after a tuple-lock wait. PostgreSQL takes
one snapshot per command in READ COMMITTED: a statement that blocks re-reads
only the conflicting tuple, through EvalPlanQual, to evaluate its write
predicate, while every other scan in the statement keeps the original
snapshot. So a queued acquire reports the pre-wait row and a
lease_age_seconds that has silently absorbed the entire wait, where the model
reported a fresh row and a small age. That age is the row edge of
_writer_lease_adoption_wait_seconds, so the model was hiding the very input
that makes the adoption CAS's updated_at predicate load-bearing: in the model
the observation was always fresh, and the CAS would have matched even with
that predicate deleted.
Evaluation now takes the snapshot before any lock wait, resolves write
predicates against the freshly locked version, and reports results from the
snapshot. Statements that evaluate to SQL NULL return None and go through
production's own parse_single_json_value, rather than being smoothed over.
That surfaced a real production behaviour the model had been concealing. When
two coordinators race the first ever acquire, the loser's DO UPDATE matches
nothing and its COALESCE fallback reads a snapshot in which no row exists yet,
so the whole statement returns NULL and PsqlShareLedger.__init__ dies with
"postgres query returned no JSON" — an error naming neither the lease nor the
coordinator holding it. The harness's own contention self-test was constructing
exactly that interleaving and turning it into a clean refusal. It now asserts
the real error, and a dedicated test pins the contrast against the ordinary
"lease is held by" refusal.
The deliberately failing #123 test asserted a bound the fix it names would not
have satisfied. It advanced the clock once and drained once, so a fix built
from lock_timeout plus a bounded retry — which is what #123 proposes — would
have left the successor parked on the first of several short deadlines with
the test still red, and the person landing that fix looking at a red test and
reaching for the weakening this test exists to refuse. It now drives the
successor through every deadline it reaches. Both candidate fixes were
simulated: lock_timeout alone and lock_timeout with a bounded retry each turn
it green, and it stays red on this line.
Four smaller model corrections. A non-matching UPDATE no longer queues on the
tuple lock, because PostgreSQL evaluates the predicate against the snapshot
version and never locks a row it excludes; latent today, but it would have
misrepresented any fencing argument, where "the predecessor is blocked" and
"the predecessor already knows it lost" are different states. Tuple-lock
waiters are granted in arrival order rather than whichever the controller
steps first. Pool exhaustion honours the caller's deadline instead of waiting
forever. The guard session's serialization is now pinned against
_NativePostgresLeaseGuard._query_lock itself, since deleting that lock left
every assertion in the harness-level test green.
The two new protocols had no executable contract: the repository runs no mypy,
they were not runtime-checkable, and the signature test compared parameter
names only. They are now @runtime_checkable, asserted against both the
production classes and the fakes, and checked with full signatures across
every method. LedgerSqlPort.pool_size is dropped, being surface nothing reads.
Two production tidies from the same pass. The lease clock is a declared class
attribute overridden per instance, rather than a getattr fallback that would
silently revert to wall-clock time under a rename. _pool_application_name is
injectable, removing the last uncontrolled nondeterminism inside the code
under test — lease_locked_by_this_process turns on that value, so a scenario
that surfaced it would have failed the determinism check for reasons nothing
in the harness explained.
Full suite: 2023 tests, failures limited to the five pre-existing macOS
os.splice tests and the one intentional #123 assertion. Twenty consecutive
runs of the five harness modules are identical.
Refs #128, #123
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Clean up the harness under a standard Python linter
The repository configures no Python linter — CI runs shellcheck, bash -n and
compileall — so this is a quality pass on the files this branch adds rather
than a fix for a failing check. Run with ruff in Docker.
Real findings, fixed: a stale noqa suppressing BLE001 on an `except
BaseException` the rule does not apply to; Callable imported from typing
rather than collections.abc; an unsorted __all__; re.S in place of re.DOTALL;
a three-line if/return that is just its own condition; __enter__ annotated
with the concrete class rather than Self, imported under TYPE_CHECKING so no
runtime version floor is introduced.
The guard serialization test caught a blind `except Exception: pass`, which
was hiding what the test actually proves. It now catches the AttributeError
the statement itself raises once past the query slot, and asserts it fired:
reaching the statement is the evidence, and any other outcome means run_json
did not get where the test claims.
Two findings are deliberately left. Every test module in this repository
carries a shebang without the executable bit, so EXE001 is the house
convention and departing from it in six files would be the anomaly. The
`noqa: E402` on the deferred share_ledger import is reported unused only
because E402 is not enabled in this ad-hoc invocation; the import genuinely
does follow a sys.path statement, so the suppression is correct and documents
why.
Also corrects a comment the review pass flagged as overstated: the baton
timeout cannot change what a run does, but it is not free of consequence, and
claiming it "cannot make a passing run flaky" overstated a real if remote
failure mode.
Refs #128
* Mark the #123 bound as an expected failure
The assertion is unchanged. What changes is who gets told about it, and when.
Left as a plain failure, it made every CI run on this branch red, which is a
poor way to communicate one known-open issue: a permanently red check is
indistinguishable from a broken one at a glance, and it teaches people to stop
reading CI. Three separate readers took the red "Lint, compile, and test" badge
for a lint failure, which is the failure mode in miniature.
unittest reports an unexpected success as a suite failure, so the marker is
self-removing: the day someone fixes #123 the suite goes red and hands them
this file, which is exactly when learning about it is useful. Verified rather
than assumed — a marked test that starts passing does set wasSuccessful False.
The marker is only defensible because the assertion genuinely goes green under
a real fix, which it did not before the drive-to-quiescence correction. Both
candidate fixes named in #123 were simulated again against this commit:
lock_timeout alone, and lock_timeout with a bounded retry, each turns it green
and each leaves 2.x.x red. An expectedFailure on a test that would stay red
after the fix would be strictly worse than a plain failure, and that is the
state this branch was in two commits ago.
Full suite: 2023 tests, one expected failure, and the five pre-existing macOS
os.splice failures that pass on Linux. Ten consecutive runs of the five
harness modules report OK with one expected failure every time.
Refs #128, #123
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Preserve the audit envelope when two found blocks land in one confirm-publish window (#136)
* Preserve the audit envelope on interleaved found-block landings
Two distinct found blocks can land inside one confirm->publish window.
The synchronous miner-connection-thread tail confirms block A under the
payout-balance lock and the publication order guard, then releases both
before it publishes; the accounting-actor tail lands block B in that gap,
allocates the next ordinal and publishes first. The per-hash disposition
lease does not serialise two distinct hashes, so nothing prevents the
interleaving.
Block A then reached publish_success already superseded and returned
without writing anything: its live audit envelope -- the block's only
published evidence pointer -- was lost permanently and silently. The
Postgres audit row, canonical bundle, body, accounting, share credit and
settlement all survived; only the public evidence trail was dropped.
The old code conflated two invariants behind one floor check. The
evidence path is a single mutable pointer that must advance monotonically
in sequence, and floor gating is correct for it. The live envelope is a
per-block immutable artifact on its own height+hash path, and a
superseded ordinal's envelope competes with nothing. Being behind the
floor means you may not become current; it does not mean your own
envelope is not yours to write.
Sequence-versus-floor alone cannot authorise that write, though. A
process whose writer lease has been taken over is also behind the floor,
and it must stay fenced out of the shared audit root entirely: its
in-memory report may be stale, and publishing a stale report as a block's
public evidence is worse than leaving the envelope missing. Only the
caller knows which it is. The write is therefore opt-in through a new
restore_superseded_envelope keyword that defaults to false, and only
_publish_finalization_evidence -- the live writer finalizing a block it
confirmed itself in this landing -- passes true. A deposed writer
replaying stale state never reaches that call site, so the A1 process
gate's C3 fencing invariants are preserved exactly.
When the write is authorised it routes through
_write_superseded_live_envelope_locked, which writes the envelope only
when it is absent and logs the event. The publication order guard and the
process flock are held throughout, and the write is rolled back if any
post-write identity assertion fails. A present envelope is returned
untouched, so a same-hash stale replay keeps its existing behaviour
exactly. The current evidence reference, the cached identity and
retention pruning are not touched, and the publication still reports
published=False. The diagnostic goes to stderr: this store is hosted
inside worker processes that reserve stdout for a strict JSON protocol
stream, and the A1 process gate drives this exact superseded path.
Add two regression tests, both failing before the fix at the
envelope-existence assertion: the distinct-hash interleaving across the
synchronous and actor tails, and the convergence path where the peer is
confirmed but not yet published so the earlier block is retried behind
the floor. Because the write is opt-in, no existing assertion changes --
the test diff is purely additive.
Fixes #133
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Lease-gate the superseded-envelope restore and pin its wiring
An adversarial review of the interleaved-landing fix for issue #133
returned four findings; this lands all four.
The restore's opt-in rested on "a deposed writer never reaches the call
site", which is false for a writer deposed inside its own
confirm->publish gap: it confirms with a valid lease, loses it during
the gap, and everything from there to the publish call is
lease-unfenced. Content divergence was independently impossible -- the
report and persistence digests bind to the durably confirmed ordinal --
so this was a coherence defect against the A1 process gate's C3
immobility invariant rather than a corruption path, but that invariant
is cheap to keep literally true. _publish_finalization_evidence now
asks the coordinator's bounded, non-blocking exact-session fence
(require_fresh_lease_for_external_side_effect) for restore authority
before entering the publication order guard, holding no store lock
across the verification. The asymmetry is deliberate: a clean proof
grants restore_superseded_envelope=True, while a lease failure
(WriterLeaseRenewalDeferred, ShutdownInProgress) withholds the restore,
logs the withholding with the component and exception type, and lets
the publication proceed unchanged. The restore is a repair of an
evidence pointer, not the publication itself; degrading to False is
exactly the previously shipped behaviour, whereas failing the
found-block path on a lease hiccup would be a live regression. The
fence stays a no-op for ledgers without the writer lease guard, never
waits on the lease row, and its internal hard-exit arming for a
genuinely dead lease is left intact.
The production opt-in was also untested: flipping the call site back to
False passed all 254 tests. A new coordinator-level regression test
drives the real two-tail topology -- handle_submit's synchronous
landing paused inside its confirm->publish gap while a second
distinct-hash candidate rides the submitter->accounting-actor handoff,
lands, and publishes -- and asserts both blocks' live envelopes exist
afterwards. It fails at the envelope-existence assertion with the call
site reverted to False. A sibling test pins the degrade path: a fence
that raises withholds the restore without failing either landing.
The present-file early return in _write_superseded_live_envelope_locked
accepted any bytes and fsynced nothing, so a truncated or corrupt
envelope was silently accepted and could never be repaired. It now
compares the existing envelope against the one this identity would
build, ignoring created_at exactly as the main publish path does -- the
comparison is extracted into a helper shared with that path -- raises
the same invalid/conflicts errors, and fsyncs the parent directory on a
match before returning.
The superseded-write diagnostic used to print while holding both the
store lock and the publication-order flock; an undrained stderr would
have blocked the found-block path with locks held. The store now
records the write on the returned publication
(AuditPublication.superseded_envelope_written, defaulted so every
existing construction site is unchanged) and
_publish_finalization_evidence emits the message -- still on stderr,
because worker stdout is a strict JSON-lines protocol -- after the
balance lock and the order guard have been released.
Verified: the three unittest suites (256 tests), the Postgres ledger
gate and the four A1 gates including C3-B-wins-late-A, and ruff
E4/E7/E9/F on the changed files.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* Persist per-session vardiff difficulty across miner reconnects (#138)
* feat(prism): persist per-session vardiff difficulty across reconnects
A coordinator restart or miner reconnect previously reset every session
to its lane's start difficulty; at 4x per 300s retarget a large miner
needed 15-25 minutes to re-climb while flooding the share path, feeding
the 2026-07-16 loop: coordinator slows -> watchdog exits -> every miner
reconnects -> every session resets to lane start -> share rate spikes
-> coordinator slows further.
Remove the amplifier by retaining each worker's last converged
difficulty in a bounded, TTL'd in-process store keyed by
(listener name, exact Stratum username). The value is recorded at every
committed retarget and on disconnect, and adopted on the first
mining.authorize of a new connection, before password parsing, so an
explicit d= still outranks it and an md=-only password clamps the
resumed value. Adoption is capped at min(lane max difficulty,
lane start * resume factor): a stale or absurd retained entry is
clamped instead of adopted, while a value below the lane start resumes
as-is (vardiff targets a share rate, so a converged session is on
target by definition).
New tunables, documented in .env.example:
- PRISM_STRATUM_VARDIFF_RESUME (default 1): master switch; 0 restores
lane-start reconnect behavior exactly.
- PRISM_STRATUM_VARDIFF_RESUME_TTL_SECONDS (default 900): how long a
converged difficulty stays adoptable; keep short where the idle
step-down sweep is disabled.
- PRISM_STRATUM_VARDIFF_RESUME_MAX_ENTRIES (default 8192): LRU bound on
retained state (2x the deployed connection cap).
- PRISM_STRATUM_VARDIFF_RESUME_MAX_START_FACTOR (default 1024, must be
>= 1): plausibility ceiling as a multiple of the lane start.
Also adds convergence observability -- sessions sitting at their
vardiff ceiling, per-lane accepted share counts and rates, resume
outcomes by bounded label, and a retained-sessions gauge -- mirrored
into the frozen metrics reference implementation.
* fix(prism): age retained vardiff difficulty from real convergence
SessionDifficultyStore.record re-stamped recorded_monotonic on every
write, and the disconnect seam records on every disconnect with no share
evidence. A session that resumed a retained difficulty, submitted
nothing and disconnected therefore refreshed that value's clock, so
reconnect cycles shorter than the TTL kept it adoptable indefinitely
(reproduced: ttl=900 survives past 4000s on 800s cycles). That defeats
the short TTL exactly where it matters: with the idle step-down sweep
disabled as deployed, a session resumed above its true hashrate produces
no accepted shares and cannot step itself down.
record now takes an explicit share_backed flag and refreshes the
timestamp only on a first record, on a value that actually moved, or on
an unchanged value backed by accepted shares; otherwise the entry keeps
ageing from its original convergence. The stored difficulty is always
updated and LRU recency still moves on every record, so recency and TTL
stay independent. ClientState gains vardiff_accepted_any, set on the
first accepted share of a connection and never reset within it; the
retarget commit point passes its own window's evidence
(accepted_shares > 0), and the disconnect seam passes the connection's
accumulated flag. An idle step-down carries no shares but lowers the
value, so it refreshes on the changed-value branch as before.
Also from the same review:
- Disconnect retains client.share_difficulty only. The pending value is
advertised with a future job, so a disconnect racing an in-flight
retarget was retaining a difficulty the miner never received; a
retarget whose paired send completed still records itself.
- qbit_prism_vardiff_resume_total gains outcome="overridden", counted
when an adopted resume is superseded by an explicit difficulty request
in the same authorize, so resumed + clamped - overridden is the number
that stuck. Routed through a coordinator seam like the adjacent ones.
- Document shared-username retention as last-writer-wins next to
PRISM_STRATUM_VARDIFF_RESUME, sharpen the per-lane rate gauge HELP to
say it is a since-start average and point at rate() over the counter,
and record the at-ceiling census cost at convergence_snapshot.
Tests cover the reconnect-cycle repro and its share-backed counterpart,
the undelivered-pending disconnect, the retarget commit-point recording,
the env -> config -> coordinator -> store sizing chain, and an authorize
whose username reservation fails. The frozen metrics reference mirrors
both rendered-byte changes.
* Cap the escalated landing budget below the watchdog tolerance (#135)
* Cap the escalated landing budget below the watchdog tolerance
A landing-class ledger step cost two independent budgets of heartbeat
silence, and the escalated budget could exceed what the watchdog would
wait for. Both halves are fixed here (issue #125).
Half A -- the ledger admission wait is no longer heartbeat-silent.
PsqlShareLedger._operation_gate waited for the writer lock or read
semaphore in a single blocking acquire sized by
_remaining_operation_timeout, which under statement_timeout is the full
constant budget. That wait is local admission, not database work: no
statement has been sent, so no server-side deadline bounds it and the
ledger reports nothing until the gate opens. A new operation_progress
thread-local scope -- shaped exactly like its operation_timeout and
statement_timeout siblings -- installs a callback plus a slice; when one
is installed the acquire is served in slices and the callback fires
between them. Without a hook the path is unchanged: one blocking acquire
and the same LedgerOperationTimeout message, so non-block-work callers
pay nothing. The caller's deadline stays authoritative, the loop always
attempts the acquire at least once so an expired budget still fails
where it always did, the callback never runs while this thread holds the
gate, and an exception from it propagates rather than being swallowed
inside a lock wait.
Both statement-timeout scopes in BlockCandidateService run on the
watchdog-monitored block-work owner thread, so they now install that
hook (duck-typed, like the statement_timeout/operation_timeout probes
beside them) with _record_block_submitter_wait as the callback and the
already-watchdog-clamped _block_work_wait_slice as the slice. Ledgers
predating the hook keep working unchanged.
Half B -- the landing cap is derived from the configured watchdog.
DEFAULT_BLOCK_LANDING_DB_TIMEOUT_MAX_SECONDS (120s) and the watchdog
default (120s) were independent literals, so an escalated landing budget
could legally outrun the tolerance. _block_landing_db_timeout now clamps
both the base and the escalation cap to
watchdog_timeout_seconds * BLOCK_LANDING_DB_TIMEOUT_WATCHDOG_FRACTION
(0.5), read from the configured coordinator rather than a second
hardcoded number, and DEFAULT_PRISM_WATCHDOG_TIMEOUT_SECONDS replaces
the literal in the env default so the two cannot drift apart.
Arithmetic. Before: admission wait (up to 120s) plus a fresh server-side
statement (120s) gave 240s of silence against a 120s watchdog; the
watchdog hard-exited mid-landing, the in-memory escalation counts died
with the process, and the restart replayed the same doomed attempt from
the base budget. After: the silent span of one landing step is a single
statement, and at the 120s default the ceiling is 60s with escalation
running 30s -> 60s -> 60s. At the 300s production tolerance the ceiling
is 150s, so the reviewed 120s cap is granted in full and deployments
using that override see no behavior change.
Tests. New coverage pins the escalated budget inside half the configured
watchdog (and pins the 300s override leaving the 120s cap intact), the
sliced admission wait stamping progress while blocked and still raising
at the caller's deadline, the unhooked gate path staying a single
acquire, hook validation/nesting/propagation, and both scopes installing
the hook while a ledger without operation_progress still works. Two
existing tests that assert budgets above half the default watchdog now
set watchdog_timeout_seconds explicitly rather than weakening their
assertions.
Refs #125
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Make the landing scope's one-statement heartbeat invariant true
The escalated landing budget is clamped to half the watchdog tolerance on
the grounds that the heartbeat-silent span of one landing step is a single
server-side statement. That fraction is only sound if the claim holds, and
in three places it did not.
Ledger operations that hold a gate across two statements. Three
_operation_gate bodies issue a second statement without releasing the lock:
confirm_accepted_block and reactivate_pool_block read the publication
ordinal their mutating statement's command snapshot could not see, and
carry_forward_integrity_report reads its audit head under the same lock so
the two reads observe one another. No admission slice runs between the
statements, so a landing caller's monitor was asked to sit through two
statement budgets of silence -- at the 120s default that is the whole
tolerance with no margin at all. PsqlShareLedger._note_operation_progress
reports the completed first round trip, following the same reasoning as
verify_writer_lease_guard_session's on_statement_progress callback. That
callback stays as it is: its callers pass it explicitly per call rather
than through the thread-local scope, and its statements never run under
_operation_gate.
Unstamped stretches between the current-tip RPC and the audit build. The
reorg reconciliation, the prior-balance checks and the tip-height RPC each
run inline on the block-work thread and stamped nothing. All three now
bracket with the file's phase-pair idiom. Bracketing the reorg walk alone
would not have been enough: one pass spends a getblockhash plus up to two
ledger statements per watched pool block, so its silent span grows with
the reorg window. The reconcile state machine takes a record_progress
port -- wired to the coordinator's block-work phase recorder, defaulted to
a sink -- and stamps per row.
The unfenced-append drain. Its condition wait was untimed and unstamped.
It stays unbounded, because bounding it could let a landing arm its epoch
fences ahead of an append that must drain first; instead it waits in
watchdog-sized slices from the existing _block_work_wait_slice helper and
stamps each slice that still finds the predicate true. The while loop's
re-check makes the timed wait semantically transparent, and the common
path never enters the loop.
Every stamper here no-ops off the registered block-work owner thread, so
client-thread dispositions and background reconciliation are unchanged.
The fraction's comment now states what is actually bracketed and calls the
result a margin rather than a global invariant: a network stall
mid-statement has no client-side socket-read deadline on the native
psycopg path and can outlast any budget, which is a genuinely wedged
process where a watchdog exit is the right outcome.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Fix the progress-hook race and the landing clamp's blind spots
Four follow-ups from the review of the landing-budget clamp.
The progress thread-local was built lazily. operation_progress did a
getattr-then-assign that is not atomic, so two block-work threads entering
their first-ever scope on a fresh ledger could each build a local() and
have one assignment win. The loser's hook then lived on an object the
ledger no longer referenced and its admission wait silently reverted to
the heartbeat-silent path -- exactly the failure the hook exists to remove.
PsqlShareLedger.__init__ now creates it alongside its two siblings; the
scope keeps its lazy fallback for ledgers built through __new__ in focused
tests.
The clamp was silent. When the watchdog-derived ceiling lands below the
configured base or cap the operator gets less than they asked for -- at
the 120s default tolerance a 120s cap becomes 60s -- and "escalation
exhausted at my cap" becomes indistinguishable from "escalation never
reached my cap". _block_landing_db_timeout runs on every landing attempt,
so the notice names the configured values, the granted values, the ceiling
and the watchdog it came from, and prints exactly once; the flag flips
under the coordinator lock so concurrent first landings cannot both print,
and the line itself is emitted outside that lock.
The clamp ignored watchdog_enabled. A deployment running with the watchdog
off has no hard-exit hazard to stay under, so halving its cap bought
nothing. The ceiling is now infinite in that case, which also keeps the
clamp notice silent (an infinite ceiling never compares below a finite
configured value) and cannot poison the escalation arithmetic, whose
inputs stay finite. The attribute is read defensively and defaults to
enabled, since clamping is the safe answer; a test pins the name on
LifecycleConfig so the defensive read cannot outlive the setting.
Nesting semantics did not match the siblings. operation_timeout and
statement_timeout merge by minimum, so nesting only ever tightens;
operation_progress replaced outright, letting an inner scope with a larger
slice widen heartbeat gaps its enclosing caller had already sized against
its own monitor. Slices now merge by minimum. The callback still replaces:
the innermost caller is the one whose liveness the wait belongs to.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Alex <alex@prismpool.io>
* Bound PRISM writer-lease acquisition and guard coordinator Postgres sessions (#137)
* Bound writer-lease acquisition and guard Postgres sessions
A coordinator client that vanishes without an RST (partition, SIGSTOP, VM
pause) leaves its backend idle-in-transaction, still holding the
qbit_ledger_writer_lease row lock taken by the landing CTE. The successor's
startup lease upsert ran with no statement or lock timeout, so it blocked
inside PsqlShareLedger.__init__ - before the watchdog arms - until TCP
keepalive teardown, a multi-hour full-pool outage (issue #123). Settlement
correctness was never at risk; the outbox row stays pending and replays.
Two independent halves:
- Every coordinator Postgres session (pooled native client, dedicated
lease-guard session, psql subprocess backend) now carries session guards:
idle_in_transaction_session_timeout reaps the orphaned transaction, and
the server-side tcp_keepalives_* GUCs bound dead-socket detection to
30 + 3x10 = 60s. DSN/PGOPTIONS values the operator set are merged, with
the coordinator fragments last so they win.
- The startup lease upsert and adoption CAS run through
_run_lease_acquisition_json: each attempt under the existing
statement_timeout machinery (SET LOCAL statement_timeout/lock_timeout,
or PGOPTIONS plus subprocess timeout), a diagnostic per timed-out
attempt, and a RuntimeError naming the locked lease row after the retry
budget. 5 attempts x 5s outlasts the 15s idle-in-transaction timeout, so
the orphan case self-heals; only a genuinely stuck lock exits the
process for the supervisor to restart. The outer _ensure_writer_lease
TTL wait is unchanged - waiting out a live holder is intended failover.
Six tunables (PRISM_POSTGRES_IDLE_IN_TRANSACTION_TIMEOUT_SECONDS,
PRISM_POSTGRES_TCP_KEEPALIVES_{IDLE_SECONDS,INTERVAL_SECONDS,COUNT},
PRISM_LEDGER_LEASE_ACQUIRE_{LOCK_TIMEOUT_SECONDS,ATTEMPTS}) flow through
LedgerConfig, the coordinator wiring, .env.example, and compose.yaml, with
fail-closed validation in both load_coordinator_config and
PsqlShareLedger.__init__ for the direct-construction daemons.
Regression tests cover the guard options on all three connection paths,
conninfo option merging, the bounded startup deadline, the bounded retry
with visible failure, recovery on a later attempt, and config
defaults/validation; each was observed failing against the unmodified tree.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Report lease-acquisition timeouts honestly and pin the bounds
The fatal path in _run_lease_acquisition_json diagnosed every
LedgerOperationTimeout as a lock conflict on the qbit_ledger_writer_lease
row. A connect timeout, an exhausted pool slot, a psql subprocess timeout,
and a healthy-but-overloaded server all raise the same exception, so an
unreachable Postgres was reported to the operator as a stuck transaction.
The RuntimeError was also raised without `from`, discarding the one
exception that said what actually failed.
Both the per-attempt line and the fatal error now name the lock conflict
as the most common cause without asserting it, quote the underlying
exception text, and the fatal error chains the originating timeout.
Two mutations of the acquisition bound survived the suite:
- Reverting _try_adopt_writer_lease to a bare _run_json left all eight
tests green. Two new tests drive the real _ensure_writer_lease adoption
branch — fast-adoption capable, a heartbeat-prefixed predecessor silent
past both the row and guard edges — and assert the CAS runs under a
bounded deadline and that a persistently blocked CAS stops after
DEFAULT_LEASE_ACQUIRE_ATTEMPTS with a chained RuntimeError.
- Widening `except LedgerOperationTimeout` also left the suite green. A
new test asserts a non-timeout failure (the RuntimeError run_json raises
for a wrapped OperationalError, and an IdleInTransactionSessionTimeout —
sqlstate 25P03, an InternalError rather than an OperationalError)
propagates unchanged on the first attempt, with no retry and no
diagnostic printed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Match the orphan-lock comments and docs to what the guards deliver
Several comments from the bounded-acquisition change claimed more than
the code provides. The PostgresSessionGuards docstring said no session
the coordinator opens can become an unreapable lock holder; tcp_keepalives_*
are silently ignored on Unix-socket connections and on platforms that do
not implement them, and session settings reach only this coordinator's own
sessions — a foreign holder is bounded solely by the acquisition deadline.
The attempt-budget comments said the ~25s budget "exceeds" the 15s
idle_in_transaction_session_timeout so the orphan is reaped mid-budget;
that timer runs on the blocking backend's clock from when its transaction
went idle, not from when the successor starts retrying, so the budget is
sized to outlast a typical reap rather than guaranteed to contain one.
Also documents the deployment caveats in the ops guide: native connections
now always set the libpq `options` parameter, so a pooler that rejects
startup options fails visibly at connect (the default compose topology
connects directly and is unaffected); the psql backend carries the guards
in PGOPTIONS, which a wrapper command that drops its environment silently
discards; and Unix-socket DSNs ignore the keepalive GUCs, leaving the
idle-in-transaction timeout as the guard that still applies.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Reconcile the concurrency harness with the bounded acquisition path
The harness suites encode #123 as an unfixed defect; this branch fixes it,
so fourteen of their assertions had to be answered rather than muted.
The orphan-lock bound is now proven. OrphanedLeaseLockFailoverBoundTests
was authored against the fixed behaviour with an expectedFailure marker and
a docstring demanding its removal by the fixing commit; it passes, so the
marker is gone and the assertion is untouched. Its two siblings encoded
"the bug is present": the successor now parks under the configured
acquisition deadline instead of forever, so the wait is asserted as a
positive bound tied to the shipped tunable, and the "does not end" test now
pins how it ends -- the exact retry budget, the elapsed time that budget
implies, and a RuntimeError chaining the underlying LedgerOperationTimeout.
The orphan and the successor's park on it are kept as evidence throughout.
FakeSqlBackend needed a fidelity fix first, and it is the substantive one.
Both shipped LedgerSqlPort implementations translate a cancellation caused
by the caller's own deadline into LedgerOperationTimeout, and only when
that deadline was armed; the model did this for pool-slot exhaustion but
not for the tuple-lock wait, because before this branch no deadline ever
reached a pooled lease statement. Without the translation the model raised
a bare LockTimeout straight through the retry loop, so the suites appeared
to exercise a bounded retry that in fact ran once.
Arming the acquisition deadline also makes the acquire transaction-scoped,
which is a real widening: a coordinator that vanishes mid-acquire can now
orphan the lease lock on a path that previously could not produce one. The
model-fidelity test that used the acquire as its example of an autocommit
statement drives an unscoped renewal instead, the new shape is pinned by a
test of its own, and docs/prism-ledger-ops.md records the trade -- the new
orphan is bounded by idle_in_transaction_session_timeout, which is why that
guard is not optional. This harness models no reaper, so that test pins
only what the model supports and says where the guard itself is pinned.
Remaining changes are exact-trace updates: the acquire and adoption CAS now
commit explicitly, so each offers a precommit stop. Traces were replaced
with the observed ones rather than loosened to prefix or subset matches,
since exact order is what they exist to prove. The lost-adoption-CAS driver
needed one extra step to reach the same interleaving, the successor's
observation no longer committing itself.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Alex <alex@prismpool.io>
* Apply stability-review fixups from PRs 134-139 (PRISM core + infra/vardiff) (#141)
* Land PRISM core fixups from the PRs-134-139 stability review
F1: _acquire_operation_gate's slicing path now reads the injected
self._monotonic clock at all three deadline sites, so a virtual test
clock can drive admission-wait slicing deterministically; covered by a
new virtual-clock test.
F2: load_coordinator_config rejects PRISM_WATCHDOG_TIMEOUT_SECONDS
below 1.0 at startup, keeping the derived landing-budget ceiling
(half the tolerance) meaningfully below the watchdog tolerance.
F3: the sliced-admission liveness test drops its per-gap 0.3s wall
clock bound, which false-reds on loaded CI hosts; the determinism
intent (stamp count, total elapsed, raised timeout) is unchanged and
cadence is pinned by the new virtual-clock test instead.
F4: the concurrency harness baton timeout is overridable via
PRISM_HARNESS_BATON_TIMEOUT_SECONDS (default unchanged at 20.0); it
remains a false-red failure detector and never influences ordering.
F5: startup warns on stderr when the idle-in-transaction timeout
exceeds the writer-lease acquire budget (attempts x per-attempt lock
deadline), where an orphaned backend holding the lease row lock can
no longer be reaped mid-budget and startup crash-loops instead of
self-healing.
F6: the superseded-envelope restore authority proof now runs after
the payout balance lock and the publication order guard are acquired,
so a writer deposed during the unbounded lock wait no longer restores
on stale authority; the withheld-authority diagnostic is deferred
until the locks are released. A new test deposes the writer inside
the publication guard and fails against the old check-then-act shape.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Replace the racy socket-only prism-postgres healthcheck with a TCP-gated probe plus SELECT 1 confirm
The postgres image's initdb temporary server accepts unix-socket
connections and then shuts down, so a socket-only pg_isready can release
dependents against the throwaway server (the race PR #134 fixed in the
test scripts). Gate on the real TCP listener first, then confirm with a
real query.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Wire the #137 lease/session-guard env knobs into the ctv broadcaster and backfill scripts
Both operator-run scripts construct PsqlShareLedger directly and got the
class defaults for lease_acquire_lock_timeout_seconds,
lease_acquire_attempts, and the four Postgres session-guard knobs, so
operator tuning of PRISM_LEDGER_LEASE_ACQUIRE_* / PRISM_POSTGRES_* never
reached them. Mirror the coordinator: read the same six env vars with the
coordinator_config helpers and defaults and pass them through. The
backfill script exposes them as CLI flags with env defaults, matching its
existing --lease-ttl-seconds pattern, with construction extracted into
ledger_from_args so the path is testable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Document that vardiff retention is process-local and can strand a stuck-high resume
The SessionDifficultyStore is in-memory: a coordinator restart, including
the watchdog's terminal exit_process(1), empties it, so the post-restart
reconnect wave is not smoothed by retention. And with the idle sweep off
(the deployed default) a rig that converged high and reconnects within
TTL on weaker hardware can resume too high with no share signal to bring
it down, bounded by the resume TTL. Honest-disclosure comments and
.env.example notes only; no behavior change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Paginate startup candidate enumeration and sweep stranded prepared blocks (#142)
* Paginate startup pending-candidate enumeration
Startup replay must observe every durable pending candidate before child
job builds unblock, and it proved that by re-querying with a doubling
window until a batch came back short. Past MAX_BLOCK_REPLAY_ENUMERATION_ROWS
the window stops doubling and the pass fails closed, so a backlog larger
than the cap can never produce a complete enumeration: every pass reports
"still truncated", enumeration stays owed, and _await_pending_parent_payout_preview
raises TemplateRefreshBlocked for every job build until the outbox drains
under the cap on its own. Testnet4 hit this on 2026-08-18 with ~5,000
pending rows and refused job builds for ~25 minutes.
Replace the widening window with a keyset walk. pending_block_candidate_rows
gains an after_cursor keyword on both ledger backends and returns an opaque
per-row cursor built from the two ordering columns: the microsecond-precision
UTC created_at text plus block_hash in Postgres, (created_monotonic,
block_hash) in memory. The block-hash tiebreak is what makes the order
total -- creation stamps collide, and a cursor on a colliding stamp alone
would replay or skip its peers. In Postgres the predicate and the ORDER BY
both stay on (created_at, block_hash), so each page is one bounded range
scan of qbit_block_candidate_outbox_pending_idx no matter how deep into the
backlog it starts.
replay_pending walks pages of MAX_BLOCK_REPLAY_ENUMERATION_ROWS while
enumeration is owed and ends only on a page proven short. Row adoption
moves to _adopt_durable_block_candidate_rows so both enumeration shapes
restore rows through the same decode, dedupe, and poison path. Ledgers
without cursor support keep today's exact windowed semantics, fail-closed
truncation included: signature introspection detects them without spending
a ledger worker, and a keyword probe that raises TypeError falls back the
same way. A page that cannot advance its cursor also fails closed rather
than declare an unproven enumeration complete.
Each page carries its ordinal and cursor in the bounded-call dedupe key.
A timed-out outbox call stays registered for the next paced retry to reuse,
and without that key page N+1 would be answered by page N's in-flight call,
silently dropping a page from an enumeration that then declared itself
complete.
Steady-state polling is unchanged: one small fetch, the same early-outs,
no pagination.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Sweep stranded prepared pool blocks in the reorg reconciler
A prepared pool-block row is resolved by the live submit/replay path that
holds its outbox entry: block_finalization rejects it when the block turns
out to be orphaned. When that outbox entry is gone -- quarantined, or
completed by a process that died before confirming -- nothing re-examines
the ledger row again, because reorg_watch_blocks selects only
chain_state IN ('confirmed', 'inactive'). Testnet4 row
0000000000000190d487ff4d5bdcc920d7b5c7a978c2f82d29c2b1f2bb6d08e9 (height
30822, found 2026-07-08) sat prepared/immature for six weeks that way: the
canonical block at 30822 is a different hash, so the pool block was
orphaned, and its payout entries, carry forward, and CTV fanout artifacts
stayed pinned immature with it.
Add stranded_prepared_blocks to both ledger backends -- prepared, immature,
and buried at least min_depth below the active tip, ordered by height and
bounded by limit. The Postgres predicate leads with maturity_state and
block_height so it rides qbit_pool_blocks_maturity_idx.
The reconciler sweeps that read in the same guarded not-untrusted section
as the watch loop, feature-detected like every other step, and stamps
progress per row because each one costs a getblockhash plus possibly a
fenced ledger mutation. A row whose height no longer resolves to its own
hash is provably orphaned and goes through the existing fenced
reject_prepared_block, which cascades the payout entries, carry forward,
and fanout artifacts and returns 0 if the row already moved. A row that
is still canonical is never mutated -- the confirm path owns audit
publication sequencing -- and is reported loudly for an operator instead.
STRANDED_PREPARED_REJECT_MIN_DEPTH is 100: deep enough that no legitimate
in-flight finalization, ancestor replay included, can still be racing, and
shallow enough to heal within roughly two hours of block time. Rejections
count into the pass totals like inactive_blocks_total (error path included,
so a partial pass stays accounted), surface as summary
stranded_prepared_rejected/stranded_prepared_canonical, and drive the same
republication and cross-tip memo eviction as any other row mutation.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* Distinguish an idempotent confirm replay from a fresh block confirmation (#146)
qbit_confirm_pool_block collapsed two different outcomes into a return of 1:
the fresh flip, where the UPDATE matched a 'prepared' row and allocated an
audit publication ordinal through nextval, and the idempotent arm, where the
UPDATE matched nothing but a confirmed non-reversed row already exists at that
(hash, height) and no ordinal is allocated. A caller could not tell them apart,
and a genuinely new confirmation must publish for payout correctness, so every
replay of a candidate whose block is still the best tip republished identical
payout state -- bumping the payout generation, wiping the job-bundle cache and
aborting in-flight refreshes until a successor block arrived.
The audit publication ordinal already draws this line durably: only the flip
allocates one. Report that fact instead of discarding it.
Return contract, was 1 / 0 / -1, now 1 / 2 / 0 / -1:
1 fresh flip out of 'prepared'; allocates the publication ordinal
2 idempotent replay of an already-confirmed row; allocates none, and
returns the ordinal its original flip assigned (new)
0 no row, or a live row this confirmation does not match
-1 terminally disposed (reorg quarantine, rejection, reversal)
Threaded through every consumer:
* the SQL function itself
* the memory ledger's parity implementation, which now splits on whether
the row was already 'confirmed'
* the psql wrapper, which reads the ordinal on 1 and 2 (a covered replay
still needs it to address its audit publication) and stays a single
statement for the non-confirming dispositions
* the coordinator confirm branch, where 2 completes the candidate exactly
like a fresh flip instead of falling into the confirmation-failure arm
* the shared test ledger fake, so unit coverage models the split
* the A1 gate disposition matrix and the Postgres ledger suite
An at-tip replay now completes as a covered replay: it reserves no payout
source, bumps no generation, wipes no job-bundle cache and schedules no
refresh churn. The fence and pending-source checks that decide publication are
unchanged and unweakened -- a source left unpublished by a prior attempt, or a
leaked delivery fence, still publishes exactly as before.
Publication-ordering interaction with the audit ordinal work: the confirmation
dict rides in the published evidence, which publish_success compares in full on
an exact replay. A restart replay reporting 2 where the flip published 1 would
have been rejected as a payload conflict, so the disposition now joins the
share counters as an observational annotation reused from the durable
publication. The ordinal it rides with is block identity and is still compared
unchanged, so a replay returning a different ordinal still conflicts. The
ordinal is unaffected either way: the idempotent arm burns no sequence, so the
durable publication floor cannot move on a replay.
Refs #61
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Extend the deterministic concurrency harness to block candidate landing (#148)
* Extend the deterministic concurrency harness to block candidate landing
Issue #128 ranks block candidate landing first by incident history, and
the harness landed by #139 covered only the writer-lease lifecycle. This
teaches it the landing state machine, which is the one owner where two
threads of the same process reach the same durable rows.
The PostgreSQL model gains the rows landing turns on: qbit_pool_blocks,
qbit_block_candidate_outbox, the qbit_confirm_pool_block publication
ordinal allocator and the durable floor read. The allocator is modelled
on the shipped PL/pgSQL rather than on an approximation of it: the
ordinal comes from a sequence, so an exact replay of an already-confirmed
row burns nothing, a terminally disposed row reports the superseded
disposition, and an aborted confirmation leaves a gap the MAX-based floor
tolerates. That distinction is the whole of the two-tail defect -- a
later ordinal can publish first and leave an earlier one behind the
floor. Landing writes are staged per transaction for the same reason
lease writes already were: a deadline-scoped statement sends COMMIT as a
separate message, and the interesting window is inside it.
Statement classification is anchored on fragments lifted verbatim from
the production SQL, as the lease signatures already are, so a change to
that SQL breaks classification loudly instead of quietly changing what
the fake models. Where a modelled statement is a large CTE, the model
reproduces the effects a landing reads back and says so…
… (#538) GitHub fires schedule triggers, and offers workflow_dispatch, only for workflows on the default branch. This adds prism-load-nightly.yml to main, byte-identical to the copy merged to 3.x.x in #536. Its jobs check out the ref input, which defaults to 3.x.x. The presets, scripts, gate and composite actions they run therefore come from 3.x.x, and nothing else changes on main. Refs #521 Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…540) (#567) Brings prism-load-nightly.yml on main up to date with its copy on 3.x.x. The two files are now byte-identical: - #559: the plan job pins one commit for the whole matrix, a single build job shares the release binaries, and the preset names change; - #540: scenario 5's dense-cadence soak joins the live-nightly list, with a comment-only change here. Refs #521 Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…from 3.x.x (#565, #561) (#583) * main: sync the PRISM load-harness nightly workflow from 3.x.x (#565) Schedules run from the default branch's copy of this workflow, so #565's weekly 2.x.x migration-lifecycle schedule (Sunday 03:43 UTC) and its schedule-specific job conditions only fire once main carries them. The file is copied byte for byte from 3.x.x; jobs still build and test the `ref` input (3.x.x by default). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * main: copy the PRISM load-harness runner probe workflow from 3.x.x (#561) GitHub dispatches a workflow only when its file is on the default branch, so #561's dispatch-only runner probe can't run until main carries it, and #541's runs are blocked on that. The file is copied byte for byte from 3.x.x at e961113; every job that uses a repository file checks out the `ref` input (3.x.x by default) first. Refs #589 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Anatolie <anatolie@swaplabs.xyz>
(#590, #596, #588, #597, #552, #551, #553, #554) (#601) * main: sync the PRISM load-harness nightly workflow from 3.x.x (#590, #596) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * main: sync the PRISM load-harness nightly workflow from 3.x.x (#590, #596, #588, #597) Adds #588's Saturday 5.5 h soak schedule and its live-nightly step that runs every listed test binary, so qbit-prism-load::soak_smoke runs beside the live_regtest variants, and #597's L6 shipped-images lane on the Sunday schedule. Byte-identical to 3.x.x at 1932cb3. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * main: re-sync the PRISM load-harness nightly workflow from 3.x.x (#552, #551, #553, #554, #582) The file is 3.x.x's copy byte for byte again, as of 3.x.x c95febb; the last change to it there is bd1ddeb (#663). Since 1932cb3 (#597), five 3.x.x changes landed in it: - #552 (#613): the `bridging` job runs the short plan on the fake and the real node on one runner, nightly, on the run-load label and on a dispatch with the new `bridging` input. - #551 (#606): every preset job builds a trend row; on a schedule the `evidence` job (the one job with `contents: write`) appends the rows to the orphan `ci-evidence` branch, applies the report-only regression rule, promotes the soak's and flagged runs' bundles to the rolling `ci-evidence/YYYY-MM` pre-release, and keeps one `prism-load-regression` issue. PR and dispatch runs get the read-only `evidence-preview` instead. - #553 (#618): the live nightly and weekly lists gain the chain-event scenarios under 2,000 sessions; the workflow's comments say so. - #554 (#635): the run job installs fuse2fs and fuse3 for a preset whose faults fill the WAL volume (faults-long-real-node, Saturday). - #582 (#663): comment only. Without this, main's scheduled copy (#595) still has no qbitd install step, so every `--node qbitd` preset fails with "--node qbitd needs --qbitd-bin or QBITD_BIN" (scheduled runs 37188928798 and 37287515779). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Robert Clarke <robert@swaplabs.xyz>
* Freeze money-path vectors from 2.x.x and record the D2 payout differences (#302)
* Freeze money-path vectors from 2.x.x and record D2 payout differences (#269)
Export 48 money-path cases from 2.x.x (504846cc) into
crates/qbit-prism/fixtures/vectors/. Topics 1-7 (window clipping,
fractional oldest share, carry-only recipients, pool fee, dust recompute,
largest-remainder ties, CTV chunks) come from the 2.x.x qbit-prism entry
points. The bootstrap transition and below-target credit come from the
2.x.x Python rule code, with each rule's payout computed by the same
engine. The exporter source and its regeneration command sit under
fixtures/vectors/exporter/, outside any cargo target.
tests/money_path_vectors.rs replays every case through the 3.x.x engine
and reports all mismatches at once. Topics 1-7 match 2.x.x exactly. The
five differing cases carry expected_2xx, expected_3xx and a d2_entry
whose anchor must exist in docs/prism-rust-migration.md. That doc now
records D2a bootstrap pooling, D2b below-target credit, and D2c prior
balances during bootstrap, the last pending the umbrella owner's approval.
checked_in_hand_worked_fixtures_match_engine now also asserts the
power-law-accrual fixture's floor, on-chain count and accrued recipients.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Pin the money-path vectors and freeze the zero-fee boundary (#269)
The vector test now pins the sha256 of every vector file next to
SOURCE_COMMIT, so a vector changes only through a re-export from 2.x.x
in a reviewed commit that updates the pin. It also requires the set of
d2_entry anchors used by the cases to equal the entries documented in
the "Payout differences from 2.x.x (decision D2)" section, which must be
exactly D2a, D2b and D2c. Deleting a D2 case, or rewriting one into a
plain matching case, now fails.
pool_fee.json gains three cases where a 1-bps fee on a 9999-sat coinbase
floors to zero. With a 1000-sat fixed floor, 2.x.x records the pool-fee
account at 0 sats and emits no fee output, under both canonical and
pool-fee-first ordering; with the day-one floor the coinbase is below
14720 and the policy fails. 3.x.x reproduces all three.
The exporter's implemented_at citations now point at the lines that
implement each rule (coordinator.rs:747 and 759,773; job_bundle.py:3400;
accepted_preview_telemetry.py:144). The D2c reason labels
build_collection_bundle as 2.x.x code, and D2b states the confirmation
timing assumption behind its next-block payout. A real 1160-recipient
TRUC chunk is left as a documented gap: it would add about 1.6 MB.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Note the 2.x.x share-weight settings 3.x.x does not carry (#269)
2.x.x read PRISM_STRATUM_SHARE_WEIGHT and PRISM_STRATUM_SHARE_WEIGHTS_JSON.
A per-worker entry in the JSON object credited max(1, override) instead of
the share target's difficulty; the default weight only filled the job's
delivered share_weight field. 3.x.x supports neither. With the 2.x.x
defaults crediting is unchanged, so this is not a D2 payout difference,
but a deployment that set either variable must remove it before cutover.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* Add a native SQL oracle for the payout-window read (#306)
* Add a native SQL oracle for the payout-window read (#269)
Item 3 of #269 asks for an independent proof that `Ledger::snapshot`
returns exactly the eligible payout window. The 2.x.x oracle was a Python
module written against the old per-row read; 3.x.x pages the window
backwards through a different query, so this is a new oracle written
natively in Rust.
Each test gets a throwaway schema, drives the read through the public
`Ledger` API only, and recomputes the expected window in one unpaged
statement: an eligibility filter, a running-sum window function standing
in for the subtract-until-zero loop, and a single `json_agg`. The oracle
shares no SQL text, constants or private items with `ledger.rs`, so it
survives the pending split of that module. It then compares membership,
order and every field of every share, plus the captured cutoff.
The anchor barrier is expressed with exact integer millisecond
arithmetic rather than the read's `to_timestamp(double precision/1000)`,
so the two sides are genuinely independent. Scenarios cover the 4096-row
page boundary from either side and three pages deep, the row that
straddles the remaining weight, rows sitting exactly on the anchor and
one millisecond past it on each timestamp, rejected rows interleaved
with accepted ones, difficulties at the top of the `u128` range, anchors
ending in 999/000/001 near 2^41 and 2^43 milliseconds, and empty windows.
Each read is given a ceiling so a cursor that stops advancing fails with
a named error instead of hanging the run.
The database URL comes from `PRISM_TEST_DATABASE_URL`. One guard decides
the outcome: run when it is set, fail when it is missing while
`PRISM_TEST_REQUIRE_INTEGRATION=1` or `GITHUB_JOB=prism-native-postgres`
demands the integration suite, and otherwise print a skip line. Keying on
`CI` would break `rust-tests`, which runs the whole workspace with no
database.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Derive oracle anchors from the clock and assert the append path at the anchor (#269)
Two follow-ups to the payout-window oracle.
The suite pinned fixed anchors in 2030 and 2039. Because a pinned anchor
has to sit above the wall clock, those were dated fuses: from January 2030
every test would have failed in CI for no reason in the code. Anchors at an
ordinary magnitude are now derived from the database clock at run time --
a day of margin, rounded up to a round millisecond, plus the scenario's own
`999`, `000` or `001` suffix -- so the millisecond coverage is unchanged and
the date is gone. Fixed anchors are kept only for the far-future magnitude
cases, and the 2^41 anchors move to 2^42.
The second change turns a comment into an assertion. `Ledger::append`
stamps `accepted_at` with `GREATEST(ledger_clock_ms, floor(now))` and
requires `job_issued_at_ms <= accepted_at_ms`, so pinning the ledger clock
lets the public API alone place a real share exactly on an anchor and
another a millisecond past it. Every anchor in the barrier scenarios now
does that, and asserts the first is credited and the second is not.
Doing so surfaced why those two changes belong together. Past roughly 2^43
epoch-milliseconds a `double precision` no longer resolves a microsecond,
so the read's `to_timestamp(double precision/1000)` barrier lands up to a
microsecond either side of the true millisecond. A fixture storing the exact
millisecond and a share written through the writer's own conversion then
fall on opposite sides, and no single barrier makes both agree. The
exact-time oracle therefore runs only where the conversion is exact -- the
derived anchors and 2^42 -- with a runtime guard that refuses an anchor
which is not, so the fixture cannot silently pick one.
Behaviour past 2^43 gets its own test, in the integer milliseconds
`AcceptedShare` carries and with no oracle and no direct inserts: three
shares per anchor, all through `Ledger::append`. Membership follows the
contract at every anchor. The read-back does not: `share_from_row` floors,
so a millisecond whose conversion lands low reads back one millisecond
early -- 480000 of every 2000001 milliseconds measured near 2^43, none near
2^42 or at any realistic date. Rather than skip that check, the test pins it
from both sides: a millisecond converting at or above itself must round-trip
unchanged, one converting below must be early by exactly one millisecond and
no more. Nothing known-wrong is asserted as right, and a regression in
either direction fails.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Bound the far-future read-back drift instead of pinning it (#269)
`writer_rows_follow_integer_milliseconds_past_2_43` required the read-back
drift to be exactly one millisecond wherever the stored timestamp converts
below its own millisecond. That asserted a known defect as expected
behaviour: the day `share_from_row` rounds instead of flooring, the drift
becomes zero and the test would have failed the fix.
The branch now bounds the drift rather than requiring it -- at most one
millisecond early, never late -- and pushes to the printed
`read_back_early` list only when a millisecond actually drifted. The defect
stays visible in the per-scenario output, a silent change of behaviour is
still obvious, and a fix passes instead of failing.
The other side of the classification is unchanged: a millisecond the
conversion represents at or above itself must still round-trip unchanged.
Membership is still asserted exactly at every anchor, and both timestamps
must still drift alike.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Tolerate the read-back convention for off-grid fixture rows and tidy the oracle (#269)
The oracle held the read's millisecond fields to its own `floor(extract(epoch
…)*1000)`. Every timestamp a writer can produce lands on a whole millisecond,
so that was exact for real rows -- but scenarios c and e deliberately insert
rows 400 microseconds either side of the anchor, to prove the barrier compares
instants rather than milliseconds, and `AcceptedShare` cannot say which side of
a millisecond such a row fell on. Flooring one is no more correct than rounding
it, yet the oracle demanded the floor, so the separately-reported
`share_from_row` rounding fix would have failed `c:anchor-barrier` and every
`e:` sub-case on `accepted_at_ms` and `job_issued_at_ms`.
The oracle now reports, per share and per timestamp, whether the stored value
lies on a whole millisecond. On the grid the read-back must still be exact,
which covers every row the writer can write. Off the grid the read may return
the floor or the floor plus one and nothing else, so a read that lost a whole
millisecond or gained two still fails. Each timestamp is judged on its own
flag, so a read that treated the two fields differently is still caught, and
membership, order and every other field stay exact.
Every test now drops its throwaway schema on the failing path too, through a
helper that runs the body and then always closes the ledger and drops the
schema. The body's error is what a failing test reports; a cleanup failure is
attached to it as context rather than replacing it, so a leaked schema can
never hide the assertion that actually failed. That covers the per-sub-case
schemas in `check_anchor_boundary` and in the 2^43 loop, which previously
leaked one schema per sub-case whenever a run went red.
Three comments were also wrong. The module header claimed the oracle shares no
SQL text with `ledger.rs`, but `oracle_cutoff` repeats the read's cutoff query
and `compare_conversion` embeds the read's own `to_timestamp`; both are
deliberate and the header now says which, and why each is acceptable. The
guard said the repository sets both "required" signals, when it sets only
`PRISM_TEST_DATABASE_URL` -- GitHub sets `GITHUB_JOB`, and nothing sets
`PRISM_TEST_REQUIRE_INTEGRATION`, the opt-in switch proposed by #286. And the
2^43 measurement spans 2000001 milliseconds, about 33 minutes, not two seconds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* Retry CI apt installs through Ubuntu mirror blips (#311) (#312)
Three CI jobs install Ubuntu packages with a bare `apt-get update` and
`apt-get install`. On 2026-09-11 the Ubuntu archive refused connections
and served hash-sum mismatches to the runners. Each of those jobs failed
with exit 100 before any test ran.
Add `.github/scripts/apt-install.sh` and use it in all three jobs. apt
retries each download (Acquire::Retries=5, with 30 s timeouts). If the
update-and-install still fails, the helper clears the package lists,
waits 30 s, then 60 s, then 90 s, and tries again, for up to four
attempts. Every attempt but the last treats a partially failed
`apt-get update` as a failure, instead of installing from stale indexes.
The last keeps apt's default leniency, so the step is never stricter
than before. Each failed attempt is annotated as a warning.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Record the approved D2 payout decision in the migration guide (#310)
Decision D2 in #260 was approved on 2026-09-11: keep the 3.x.x behaviour
for D2a (bootstrap pooling), D2b (below-target block credit) and D2c
(prior balances during bootstrap). Mark D2c as approved, and point the
configuration note at the operator release notes.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* 3.x.x: native metrics registry with the cutover-minimum series (#308)
* feat(metrics): add native cutover registry and producer hooks
* fix(metrics): enforce registry ownership and live observation contracts
* docs(metrics): align operator guidance and collector validation
* test(metrics): pass registry to merged admission fixture
* test(metrics): reset whole socket before failed-write assertions
* fix(metrics): validate listeners before coordinator startup
* fix(metrics): bound progress guard to health publication
* fix(metrics): render registry before first publication
* Survive Ubuntu mirror outages in CI package installs (#311) (#314)
#312 made the three CI apt steps retry, which rides out short blips. An
outage like the one on 2026-09-11, about 35 minutes of refused
connections and hash-sum mismatches from Canonical's archive, still
fails them.
- Pin fallback mirrors in .github/apt/ubuntu-mirrors.txt: Azure's cloud
mirror, kernel.org and Princeton, with Canonical's archive last. After
the first failed attempt, apt-install.sh replaces the runner's Ubuntu
sources with that list. Priority tags make apt try the mirrors in
order, and retries skip apt's growing delay so a refused mirror doesn't
hold up every download. Healthy runs keep the runner's own sources.
- Cache the downloaded packages. The new composite action
.github/actions/apt-install restores and saves them with actions/cache,
keyed on the package set and the month. The helper downloads into its
own directory, keeps exactly the packages a successful install used,
seeds the next install from them, and if every online attempt fails,
installs them directly with dpkg. It never reinstalls or downgrades
anything the runner already has.
- Stop retrying after a deadline (480 s by default), so the cache still
gets its turn well inside the jobs' timeouts.
The three CI apt steps now use the composite action.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* 3.x.x: ratchet the payout window against the PostgreSQL JSONB ceiling (#298)
* 3.x.x: ratchet the payout window against the PostgreSQL JSONB ceiling (#264)
At the production window the native server stores the whole share window
inside single JSONB documents. PostgreSQL refuses any container whose
elements exceed 268,435,455 bytes. CI never saw this, because the largest
native fixture had 800 shares.
Add a deterministic production-shaped window fixture, an in-process fake
node, and a gate that drives refresh, enqueue, claim, landing and import
against PostgreSQL 16. After each phase the gate measures every JSONB
column, and it fits bytes per share at two reduced sizes to project to
400,000 shares. Four known violations are checked in as a ratchet. The
gate fails on a new crossing and on a listed write that stops crossing,
and it never skips when CI is set without a database. It runs in the
prism-native-postgres job; the database-free rust-tests job skips it by
name. The full-size run and the baseline sweep are #[ignore]d and
documented. test/prism-native-tests.sh gains a cargo-args mode, and the
measurement and capacity docs record the baselines.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* 3.x.x: attribute refused JSONB writes and check the sweep's fit stability (#264)
Two accepted review findings on the JSONB ceiling gate, both in test code and
the measurement document.
A refused write was attributed to a hard-coded column. PostgreSQL's ceiling
error names neither the table nor the column, and the server adds no context
naming the write, so each of the four `ceiling_rejection` sites was recording an
assumption: a phase that ever wrote a second oversized JSONB value would have
had the refusal blamed on the expected column, and the new offender would never
have been seen. Every refusal now goes through `attribute_refusal`, which takes
the refused phase, the refused share count and what that phase's accepted
measurements project for each of its columns there. Exactly one column must
project past `ATTRIBUTION_FLOOR`, 90% of the hard limit - seven times the 1.4%
the measured fits disagree by - and it must be the write the call site expects;
no candidate, two candidates, or the wrong one fails the gate with both names.
The full-size run and the baseline sweep now run the reduced pair first for
those projections; the reduced run attributes a refusal at `n2` from the `n1`
measurement scaled to it, and a refusal at `n1` fails outright, since nothing
smaller was accepted to attribute it from. The ratchet row says how it was
attributed and still never shows a byte count.
The baseline sweep never checked fit stability. Its comment claimed the CI-pair
projection had to agree with the large-pair one, but it only computed the large
pair, and with the default sizes refresh is refused there and has no projection
at all. The sweep now also runs the reduced pair for the CI projection, skipping
a size its list already has, and compares the two per write: the large pair is
the two largest sweep sizes at which that write was accepted, since the writes
are refused at different sizes and no single pair fits them all. A disagreement
over `FIT_STABILITY_TOLERANCE` fails the sweep, naming the write, both
projections and both pairs; a write accepted at fewer than two sweep sizes fails
rather than going unchecked; a CI projection below 1 MiB is printed as not
compared, because relative noise on a write that small says nothing about the
window's growth. The table has the columns of the document's own.
Unit tests cover attribution (unique, ambiguous, unexplained, another phase's
column, and a candidate that is not the expected key) and fit stability
(agreement, a 10% divergence, pair selection skipping refused sizes, fewer than
two accepted sizes, and the 1 MiB floor), all without a database.
Verified on PostgreSQL 16: the reduced gate ends with the ratchet holding at
exactly the four known violations, and the sweep at 10000,20000,40000 prints the
comparison table with every compared write inside 0.22%.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* 3.x.x: keep the n1-scaled projection for a write refused at n2 (#264)
A write PostgreSQL refuses at n2 gets a Rejected verdict in the reduced
fit, so projections_at gave it no projection. With PRISM_JSONB_GATE_N2
equal to the target, the full-size run then met the same refusal with
nothing to attribute it from, and failed as unexplained although the
write is a known violation. The sweep had the same gap at larger sizes.
reduced_projections keeps the projection that attributed the n2 refusal,
the n1 measurement scaled to the size in question, for every write the
reduced pair refused. The full-size run and the sweep now use it. A unit
test covers the refused write, the fitted write it leaves alone, and the
attribution at the target.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* 3.x.x: attribute refusals before any table names their column (#264)
The full-size run, the reduced run's n2 half and the baseline sweep printed
their measurement tables before attribute_refusals ran, and
write_table_lines named each refusal's table and column as the call site
recorded it. A moved or new oversized write would have been shown under
the old column before the attribution check disputed it.
Every path now attributes first and still prints its table whatever the
outcome. write_table_lines shows an unattributed refusal by phase only, as
(refused) (not attributed). attribute_refusals no longer empties the map
before every refusal has attributed, so a failure keeps the refusals in
the report. The sweep attributes through a new attribute_sweep before
printing any size. A unit test covers both rows.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* 3.x.x: fail on an unreadable database URL instead of skipping the gate (#264)
database_url read PRISM_TEST_DATABASE_URL and CI with .ok(), which turns
VarError::NotUnicode into "unset". Outside CI, a deliberately configured
but malformed URL therefore skipped the ratchet silently, and a
non-Unicode CI value counted as unset too.
Both reads now go through database_url_from, where an unreadable variable
is an error, as every other gate setting already is. The existing rule is
unchanged: with CI set and no URL, the gate fails. A unit test covers a
set URL, unset and blank values, CI without a URL, and non-UTF-8 values
of both variables on Unix.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* 3.x.x: Set one 3.0.0 release identity across VERSION and every crate (#315)
* 3.x.x: set one 3.0.0 release identity across VERSION and every crate
After #283 the tree carried three version strings on the 2.0.x line:
VERSION and the two library crates read 2.0.2 by inheritance from the
2.x.x merge, and the rewritten server still read 2.0.0 from #244. None
of them named a 3.x.x release.
Declare the version once, as [workspace.package].version = "3.0.0" in
the root Cargo.toml, and have every crate inherit it with
version.workspace = true. VERSION reads 3.0.0 to match. Cargo.lock only
moves the three workspace-member entries; no third-party dependency
changes, and `cargo metadata --locked` and `cargo check --workspace
--locked` pass.
The toolchain pin and builder-image digest stay with #286; the 3.0.0
release notes stay with #291.
Part of #284.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YRVX7BZXPMWRNnbY9QLm1H
* 3.x.x: gate the release identity with a version-skew check
scripts/check_version_skew.py asserts that VERSION equals the version
Cargo resolves for every workspace crate. It reads crate versions from
`cargo metadata --no-deps` rather than the manifests, so a crate that
later drops version.workspace = true for its own literal is still
caught. It names each mismatching crate with both versions and exits
non-zero on any mismatch, on a missing, empty, or malformed VERSION, and
when cargo itself cannot describe the workspace.
CI runs it as the first step of the rust-tests job, which already has a
Rust toolchain and is in the required checks aggregator's needs list, so
skew fails the merge gate without a new job. `make check-version-skew`
runs the same command locally.
tests/test_check_version_skew.py drives the script against temporary
workspaces: matching versions pass; VERSION behind the workspace names
every crate; one crate with a literal version is named alone; empty,
missing, and malformed VERSION files fail; an unusable cargo is reported
rather than passing. It also asserts the checked-in tree passes, that
the CI step lives in a job the gate needs, and that the make target
exists.
The 3.x.x development-line note records the single source and points
the real release notes at #291.
Part of #284.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YRVX7BZXPMWRNnbY9QLm1H
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* 3.x.x: assert the server-side D2 payout rules (#316)
* Cover the D2b below-target block credit against PostgreSQL
Decision D2 on #260 keeps 3.x.x's payout behaviour, and entry
d2b-below-target-credit of the migration guide is the one that changes a
miner's money: a proof that meets the network target but misses its
assigned share target is credited at network difficulty, and only once the
block it solved is observed on the active chain. 2.x.x credited the
assigned share difficulty the moment the node accepted the block. Until
now the only in-tree coverage of that shape ran against an in-memory
stratum backend, which can see the frame being forwarded but neither the
credited amount nor when it lands.
Add coordinator::d2_below_target_tests, which drives the real Coordinator
against a throwaway PostgreSQL schema and a scripted qbit JSON-RPC node,
one test per case of the frozen below_target_credit vectors. Each test
seeds the vector's ledger window, builds the found-block work, mines a
proof of the shape the case names, and starts a real submission, then
asserts:
* the credit is published only inside the confirmation transaction. While
the candidate is pending there is no credit row, exactly one deferred
share, and no acknowledgement to the miner;
* the credited difficulty is the proven network work and not the 2.x.x
assigned difficulty, read from both sides of the vector rather than
written here;
* a lost submitblock reply ends in exactly one credit, published by
reconciliation, and repeating that observation does not credit twice;
* a reorg after publication leaves the credit intact while the block goes
inactive;
* a block the node refuses is abandoned, credits nothing, and fails the
submission with reason ledger-confirmation-failed, with the internal
reason the miner never sees captured from the log;
* a share-passing proof is still credited its assigned difficulty at once,
with nothing deferred;
* the block after the solved one pays exactly what the vector projects,
compared leaf by leaf, which is where a wrong credit would become a
wrong payout.
The assigned share target is reached through the stratum difficulty floor,
because reassign clamps a target easier than the network target back to
the network target and the floor is the only way to get a share target
harder than the block target. The two difficulty units differ, so the
floor is converted from the vector's scaled unit and stepped up until it
is at least as hard as the vector's target; no f64 divides to that exact
256-bit value.
The tests carry the same integration guard as the window read oracle:
they run when PRISM_TEST_DATABASE_URL is set, fail by name when
PRISM_TEST_REQUIRE_INTEGRATION=1 or GITHUB_JOB=prism-native-postgres
demand the suite, and otherwise print a skip line, so a database outage in
that job cannot pass silently. Each test takes its own schema, serialises
against the cluster-wide advisory locks the other coordinator tests share,
bounds every wait, and drops its schema and aborts its tasks on the error
paths too.
Also point the stratum protocol test at this module, so the boundary
between the two is written down where a reader meets it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Pin the D2a bootstrap-pooling selection with coordinator tests
Decision D2 on #260 keeps 3.x.x's payout behaviour during pool bootstrap,
and nothing yet fails if that behaviour drifts back towards 2.x.x. This
adds an in-crate test module that drives Coordinator::refresh_once against
a fake qbit JSON-RPC node and asserts on the work it actually publishes.
What is asserted, and why it is asserted this way:
* D2a, bootstrap pooling. With two accepted shares from two miners -- below
2.x.x's readiness gate of three, above 3.x.x's rule of "any share at all"
-- refresh_once must select the ledger window, and the payout must be the
proportional split the bootstrap_transition vectors record for 3.x.x, and
must not be the 2.x.x whole-coinbase-to-the-finder outcome. Restoring any
readiness gate of three or more fails this test.
* The one-share boundary the vectors do not cover: a single share from a
miner who is not the solver takes the whole coinbase. A restored gate of
two or more fails this test.
* An empty ledger and no balances still pays the solver, through the one
synthetic bootstrap-share the bootstrap path builds.
Every expectation is read out of the vector file by case name, checked
against the D2 anchor the case carries, so a renamed, re-anchored or
weakened case fails here rather than passing quietly. The assertions read
the Prepared work refresh_once published and the bundle build_job hands a
solver; build_bundle is never called directly, because that would compute
the answer the selection under test was supposed to make.
The harness template's network difficulty is 1_000_000 rather than the
vectors' 100, so payouts that turn on the synthetic share are compared in
units of each side's own network difficulty, with exact divisibility
asserted rather than rounded away. Seeded-share payouts are compared
exactly: the 8x window counts every seeded share at either scale.
The integration guard is the one from the payout-window oracle: it runs
when PRISM_TEST_DATABASE_URL is set, fails and names the variable under
PRISM_TEST_REQUIRE_INTEGRATION=1 or GITHUB_JOB=prism-native-postgres, and
otherwise prints a skip line. Keying on CI would be wrong, because the
no-database job sets it too.
The D2c carry-only case is not covered yet: an empty payout window
alongside a non-zero carry-forward balance is not reachable from 3.x.x's
own write paths, and how to reach it is still being decided.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Pin D2c: a bootstrap block still pays carried-forward balances
Decision D2 on #260 keeps carried-forward balances in the payout of a
bootstrap block, where 2.x.x dropped them from its collection bundle and
left the account unpaid. Nothing failed if that regressed.
This adds the carry-only case from the bootstrap_transition vectors. With
an empty share window and one carry-only account whose balance is above
the day-one floor, refresh_once must publish no window bundle, the bundle
build_job hands the solver must carry the ledger's prior balances, and its
payout policy manifest must match the recorded 3.x.x outcome exactly -- the
solver paid the coinbase less the carried account's settlement, and the
carried account paid out of this block. It must also differ from the 2.x.x
outcome, which pays the solver everything. Passing an empty prior-balance
list on the bootstrap path fails this test and no other.
Reaching that state takes an explanation, because 3.x.x cannot produce it.
An empty window beside a non-zero carry is the migrated, or archived, state
of a pool that inherited balances without the share history behind them,
and every route to it from a running 3.x.x is closed: a block that accrues
must carry real ledger shares, because the landing path re-reads the
bundle's share range out of the ledger and requires a match; a
bootstrap-shaped block cannot accrue at all, because its single entitlement
always takes the whole miner reward and so always clears the floor that was
just checked against it; and shares can never be removed afterwards,
because the share ledger refuses every update, delete and truncate at the
database level, and one accepted row is enough to keep the window non-empty
forever.
So the test records the carrying parent block's rows directly, and does it
without inventing a single figure. The parent's payout is built by the
engine and verified through the public audit verifier, the test asserts the
engine accrued exactly the balance the vector expects before anything is
written, and then only that manifest's own accounts are recorded -- through
the same statements, the same account-type filter and the same order the
landing path uses, the block row prepared first and flipped to confirmed
separately so the summary triggers count its carry once. Nothing writes a
summary table by hand. A schema change that moves the real landing path
leaves this copy failing loudly rather than quietly seeding some other
state. The fake node's tip at that height is that parent block, so
reconciliation sees it on the active chain, and the block under test is its
child.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Share the D2 test harness and apply the review findings
Both decision-D2 coordinator test modules carried their own copy of the
same harness. Move it into coordinator::d2_test_support: the integration
guard, a throwaway schema that is removed on success, on failure and when
a fixture fails to open, the coordinator configuration both run, the
payout projection with its leaf-level diff, and the shared constants.
Also apply what review of the two modules found:
* the bootstrap tests' fake node served a fixed template time, so every
one of them would have started failing as stale in January 2027; it now
serves the wall clock, as the below-target node already did;
* each test now reports its own failure ahead of any cleanup failure,
instead of letting a cleanup error replace it;
* the below-target fixture now drops its schema when setup fails;
* assertions that could no longer fail once the exact 3.x.x match had
passed now check that the vector itself still records a difference
from 2.x.x, which is what they were meant to guard;
* the carry-seeding comment now says exactly which landing statements it
repeats verbatim and which parts of landing it leaves out.
No production code changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* 3.x.x: Make gated tests fail loudly, prove CI execution, add Rust hygiene jobs (#322)
* Fix every clippy warning without changing behaviour
`cargo clippy --locked --workspace --all-targets -- -D warnings` now
passes on the 1.98.1 toolchain. Each change keeps the code's behaviour:
- `(n + 1) / 2` becomes `n.div_ceil(2)` in both Merkle root loops.
- The two `Default` impls that picked one variant become `#[derive(Default)]`
with `#[default]` on that variant.
- The eligible-share sort keys on `Reverse(share_seq)` instead of a
comparator that spelled the same order.
- `&[x.clone()]` becomes `slice::from_ref(&x)` in the CTV manifest test.
- `seq % 4 == 0` becomes `seq.is_multiple_of(4)` in the window fixture.
- `chunks_exact(_mut)` with a constant size becomes `as_chunks(_mut)`, which
ignores the same remainder.
- The adversarial settlement case that looped over four blocks and broke
after the first now states that it checks block 0 only, which is what
it always did.
- `tools.rs` keeps its `live_instance_tests` module after the last item.
- The two owning audit-bundle entry points allow `too_many_arguments`
with the reason that they mirror the borrowed forms they forward to.
`cargo fmt --all --check` already passed on the base commit, so there is
no formatting-only change to land.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DzhiDBXX9hveszStKkyFt1
* Route every environment-gated test through one shared gate
Every test whose execution depends on PRISM_TEST_DATABASE_URL,
PRISM_TEST_PG_BIN_DIR or QBITD_BIN used to open with its own copy of a
skip: read the variable, print a line libtest hides, return Ok. The new
qbit-prism-test-gate crate holds the one decision table that
window_read_oracle.rs already followed, generalised per input: present,
run; missing while PRISM_TEST_REQUIRE_INTEGRATION=1 or
GITHUB_JOB=prism-native-postgres, fail naming the variable and the test;
missing otherwise, print one line with a fixed prefix past libtest's
capture and return. A variable that is not valid Unicode is an error,
never absent, and a switch value other than 1 or 0 is an error too. The
table is a pure function over injected values with a unit test per row,
and a child-process test drives the runtime path.
Every site is converted: the ledger, 2.x upgrade, rollups, API, readiness,
authorization, migration-rollback, failover, replica, live regtest,
candidate-lease, public-service and live-instance tests, and the window
oracle, which drops its private guard. The two explicitly selected
#[ignore] tests use the gate's required entry point, which never skips.
The JSONB ceiling gate moves off its CI keying (GitHub sets CI=true in
rust-tests, which has no database), so rust-tests no longer skips it by
name; its unreadable-variable test now exercises the shared table. The
live fixture keeps the qbitd path it was opened with for node restarts.
The crate is a dev-dependency of qbit-prism-server and qbit-prism, so a
gated test under either crate's tests, or in a #[cfg(test)] module, uses
the same gate. scripts/check_gate_env_reads.py, run in the lint job,
fails when any Rust file outside the gate crate names one of the gate
variables in a string literal, so no new test can read them directly.
Measured gated tests: 74 in the tree, 72 of which the native job runs
(the two JSONB #[ignore] measurement runs are gated but never selected
in CI).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DzhiDBXX9hveszStKkyFt1
* Prove in CI that every gated test executed
With PRISM_TEST_GATE_MANIFEST set, the gate appends one line per decision,
executed, skipped or failed, followed by <package>::<binary>::<test path>,
in a single O_APPEND write so parallel test threads and separate test
binaries share the file. test/prism-gated-tests.txt is the checked-in
list of the 72 gated tests the native job must execute, including the
two #[ignore] runs it selects explicitly; its length is the minimum.
scripts/check_gate_manifest.py fails when any expected test did not
execute, any skipped or failed line appears, fewer distinct tests
executed than the list holds, a test executed that the list does not
name, or the log shows the gate's skip prefix or the wording of the skip
messages the gate replaced. It prints the manifest to the log and to the
step summary.
The prism-native-postgres job keeps its id (the gate keys on it), sets
PRISM_TEST_REQUIRE_INTEGRATION=1 and the manifest path, runs the whole
workspace with --all-targets and --nocapture through tee, then the two
explicit --ignored runs, then the proof, and uploads the manifest as an
artifact. A gated test added under crates/qbit-prism is therefore covered
by a required job.
test/prism-native-tests.sh mirrors the job: its default mode runs the
workspace, both explicit runs and the proof, and exports the switch once
the PostgreSQL server tools and qbitd are present; the live, replica and
cargo-args modes always export it, since they refuse to start without
their inputs or name their own tests.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DzhiDBXX9hveszStKkyFt1
* Add fmt, clippy, cargo-deny and hadolint as required checks
Four new jobs join the checks aggregator: cargo fmt --all --check,
cargo clippy --locked --workspace --all-targets -- -D warnings,
cargo deny --locked check, and hadolint over every Dockerfile.
deny.toml checks advisories (nothing ignored), licenses (an allow-list of
what the tree uses, plus CDLA-Permissive-2.0 for the CA bundle that
webpki-roots ships), bans (duplicate versions warn; wildcard requirements
are refused except the workspace's own path dependencies, which needs
qbit-prism and qbit-prism-server to be publish = false, as they are in
practice) and sources (crates.io only).
cargo-deny 0.20.2 and hadolint 2.15.1 are installed by
.github/scripts/install-pinned-tool.sh from their GitHub releases after a
sha256 check. hadolint reads .hadolint.yaml, which ignores nothing
globally: lab/prism/Dockerfile pins its two apt packages, and the other
images, whose behaviour is not this workstream's to change, carry inline
ignores with reasons (unpinned apt versions, the named qbit-src build
context that DL3022 mistakes for a stage, the deliberate separate cmake
layers, and the checked pipes in the Bitcoin Core download).
.github/actionlint.yaml names the Blacksmith runner label so actionlint
runs clean locally.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DzhiDBXX9hveszStKkyFt1
* Pin the Rust toolchain to 1.98.1 in one place
rust-toolchain.toml pins the exact stable version with rustfmt and clippy
under the minimal profile. [workspace.package].rust-version carries the
same version and every crate inherits it. The lab/prism/Dockerfile
builder is rust:1.98.1-bookworm at the multi-arch index digest Docker
Hub serves for that tag (resolved through the registry API; both the
1.98.1-bookworm and 1.98-bookworm tags point at it today).
scripts/check_toolchain_pin.py fails when the three disagree, when a
crate stops inheriting rust-version, when the builder line lacks a
digest, or, with --installed, when the active rustc or cargo is not the
pin. The lint job runs the static check; the new
.github/actions/rust-toolchain composite action replaces the floating
dtolnay/rust-toolchain@stable in every Rust-using job, installing the
toolchain rustup reads from the file and asserting the installed
version. docker-builds builds the builder stage on its own first and
asserts that its cargo and rustc report the pin.
VERSION and scripts/check_version_skew.py from #315 are unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DzhiDBXX9hveszStKkyFt1
* Keep one make target per native test suite
test-prism-postgres-ledger, test-prism-postgres-scale,
test-prism-postgres-native-ledger and test-prism-postgres-seed-guard all
ran test/prism-native-tests.sh; they become test-prism-postgres.
test-prism-stratum-regtest-live, test-prism-stratum-postgres-regtest-live
and test-prism-combined-regtest all ran the live mode that
test-prism-regtest already names; they are removed.
test-prism-public-read-replica keeps its name and calls the script's
replica mode directly. The eight wrapper scripts under test/ that only
exec'd prism-native-tests.sh are deleted; test-prism-postgres-throughput
and its script are untouched (#271 owns the throughput floor).
README, the coordinator-refactor validation notes and the overload alert
spec now name the surviving targets; the historical release notes under
doc/ keep the names they shipped with.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DzhiDBXX9hveszStKkyFt1
* Document the integration test gate and the deleted-test map
docs/prism-integration-test-gate.md is the gate contract: the inputs, the
decision table, the switch, the skip prefix, test identity, the manifest,
the expected list and the proof, local runs, and how to add a gated test.
docs/prism-deleted-test-map.md rebuilds the readiness review's map from
git: every test file under tests/ and test/ on origin/2.x.x (504846c)
absent on 3.x.x, 142 files grouped by area, each with its native
replacement, or the open issue that closes the gap, or a needs-triage
note where neither exists. Both are linked from docs/README.md, and the
wrapper descriptions in PRISM.md, the server README and the ledger
operations guide now describe required mode and the manifest.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DzhiDBXX9hveszStKkyFt1
* Route the D2 payout tests through the shared gate
#316 landed nine PostgreSQL-gated coordinator tests with a hand-written
copy of the old guard in d2_test_support.rs, which read
PRISM_TEST_DATABASE_URL, PRISM_TEST_REQUIRE_INTEGRATION and GITHUB_JOB
directly and so failed scripts/check_gate_env_reads.py. The guard now
takes the URL through qbit_prism_test_gate like every other #[cfg(test)]
site, the nine callers drop the test name the gate derives itself, and
the tests join test/prism-gated-tests.txt, which the native job's
manifest is checked against: 81 expected gated tests, 83 in the tree.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DzhiDBXX9hveszStKkyFt1
* Require every explicit #[ignore] run and write manifest lines in one write
The two JSONB measurement runs, jsonb_ceiling_ratchet_at_full_size and
jsonb_ceiling_baseline_sweep, are #[ignore] tests someone asks for by
name, yet they reached the gate through the non-required entry point, so
a manual --ignored run with no database printed a skip line and reported
ok. They now use the gate's required entry point, as the
observability_database and stratum_admission_postgres runs already did,
and the contract says that every explicit #[ignore] run is required.
They stay out of test/prism-gated-tests.txt because CI never selects
them.
The gate's record_to used write_all, which can split a line across two
writes after a short write and let another thread's line land between
them. It now hands the whole line to one write and treats a short write
as an error, so each manifest line stays a single O_APPEND write; the
skip line on stderr goes out the same way.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DzhiDBXX9hveszStKkyFt1
* Leave the runtime image's apt packages unpinned
The PRISM runtime stage floats on debian:bookworm-slim, and Debian drops
superseded package versions from its archive, so the exact ca-certificates
and curl pins would have broken the build at the next point or security
release, with no Docker here to build-test a bump. The stage carries the
same inline DL3008 ignore and reason as the other images; pinning its base
and packages belongs to #288. The builder stage keeps its digest pin.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DzhiDBXX9hveszStKkyFt1
* Re-check the deleted-test map against the D2 payout tests
The map is now built against 8dccc74, the merge of #316, and pins both
ends of its file list (504846c and 8dccc74), since origin/2.x.x has moved
on and this branch removes wrapper scripts of its own. The nine D2 tests
were read against every row they could touch: ten partial rows now cite
them (below-target credit, lost-reply reconciliation, reorg survival,
bootstrap selection, the retired readiness gate), and the settlement
balance comparison row states why D2c does not cover it. No status
changes; the counts stay 5 full, 67 partial, 8 open gaps, 7 needs
triage, 55 retired.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DzhiDBXX9hveszStKkyFt1
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* 3.x.x: Assert stale-grace anchoring, retained-job and hot-path submit semantics (#313)
* fix: preserve PRISM issued-work and stale-grace semantics
Separate observed and published tip authority, retain original issued
worker context through replacement and reconnects, and preserve bounded
delivery-based grace without renewing expiry after awaited work.
Exercise real Coordinator decisions and socket interleavings, including
constant node-RPC load across 1, 32 and 128 clients.
Refs #280
* docs: reference PRISM miner parity settings
Co-Authored-By: OpenAI Codex <noreply@openai.com>
* fix(prism): bind share admission to one published payout view
* fix(prism): verify changed block notifications with fresh chain state
* docs(prism): record B8 review regressions and rebased qualification
* test(prism): adapt merged admission fixture to stale grace contract
* fix(prism): classify retained capacity by current published tip
* fix(prism): retain prepared dependencies for issued work
* test(prism): verify issued job dependency persistence
* fix(prism): keep sessions alive while retained work is usable
* docs(prism): record timer coverage and final qualification procedure
* docs: point D2 credit explanation to miner submit module
* test: align candidate lease tip fixture with coordinator state
* fix: integrate miner parity with native metrics registry
* fix: integrate miner parity with metrics registry API
* ci: run issued job dependency acceptance explicitly
* test: keep miner parity fixture publication reachable
* test: complete JSONB gate fixture config
* test: restore D2 coordinator coverage
* fix: bound configurable tip freshness durations
* test: retain tokio time controls under pinned toolchain
* chore: satisfy pinned toolchain checks
* test: route issued job dependency through integration gate
---------
Co-authored-by: OpenAI Codex <noreply@openai.com>
* docs: design record for WindowRef and Ledger::read_window (#297)
* docs: design record for WindowRef and Ledger::read_window (#264)
Add docs/prism-coordinator-refactor/window-ref.md, the design record that
#265 (migration 007) and #273 (migration 008) implement against, and link
it from the directory README.
The record specifies:
- WindowRef { anchor_ms, prior_balances_digest, shares: Option<ShareRange> }
- the six window_* columns and their three-state CHECK
- Ledger::read_window: one REPEATABLE READ READ ONLY transaction,
mandatory ascending keyset paging, a streaming digest, typed errors and
never a partial window
- per-caller revision fences that keep today's observe_candidate rule for
claims
- the non-window bundle inputs a rebuild stores instead of reading config
- the upgrade procedure, retention floor and expected cost at 400k shares
- the borrowed qbit-prism parts API merged in #296
It went through two adversarial reviews.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: hex-serialize WindowRef digests and clamp the resume timeout (#264)
Two fixes to the WindowRef design record after review:
- A derived Serialize writes a [u8; 32] as a JSON array of 32 integers, not
the lowercase 64-hex string the payload and column contract requires. Both
digest fields now carry #[serde(with = "hex32")], a small adapter in
ledger/window.rs that writes lowercase hex and reads exactly 64 lowercase
hex characters. The server's hex dependency has no serde feature.
- The proposed resume timeout, initial_job_timeout_seconds minus 5 s, is
zero or negative for any setting at or below 5 s, which validation allows.
It is now outer - min(5 s, outer / 2), computed from the Duration the
caller already builds: 25 s at the 30 s default, always positive, strictly
shorter than the outer timeout, and free of Duration underflow.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: bootstrap resumes use stored policy inputs (#264)
An empty-window resume rebuilt each miner's bootstrap bundle through
Coordinator::build_bundle. That function reads config.ctv_enabled,
payout_policy, ctv_direct_floor and ctv_config, so after a config or
fingerprint reset a resumed job could get a different signed bundle under
the same job ID. That is the drift the stored-inputs invariant forbids.
The record now has the per-miner bootstrap rebuild call the borrowing
builders directly. It takes its own build_slots permit after the
single-flight entry's permit is released. Its inputs are the synthetic
share build_bundle fabricates today, the template, anchor, payout_policy
and ctv stored in StoredPrepared, and the balances and revision read_window
returned. #273's must-not row now forbids build_bundle for any resume
rebuild.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: readable imported audits and a race-free upgrade order (#264)
Two fixes to the WindowRef design record after review.
Imported legacy audits: #265 stores canonical_audit_bytes and no inline
body. Only audit_canonical_bytes reads those bytes today. Ledger::audit_bundle
returns just the JSON column, so backfill-ctv would stop on imported rows,
and the bundle endpoint's fallback reads only body_uri. The record now
requires the same PR to make both readers decode the stored canonical
bytes first, digest-checked and under spawn_blocking. Leaving an imported
audit readable only through body_uri is now forbidden.
Upgrade order: a zero pending count taken while the pre-007 frontends still
run proves nothing, because a block can be found before they stop. The
count is now taken after every frontend is stopped. A non-zero count means
restarting the pre-007 frontends to drain and repeating, and 007's refusal
remains the backstop.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: one owned window per Prepared and the established lock order (#264)
Two fixes to the WindowRef design record after review.
Window ownership: Prepared and JobContext would have held an owned
AuditBundle beside the Snapshot. into_bundle takes the share Vec by value,
and two Arcs don't share storage, so every resume would keep a second
window: about 0.75 GB per 400k rebuild instead of 0.5 GB. Both structs now
hold Arc<Snapshot>, the only owned window, next to Arc<AuditBundleBody>.
Canonical bytes and verification come from the parts functions, and an
owned bundle is assembled only at claim.
Lock order: the D6 prune now takes SETTLEMENT_LOCK and then ORDER_LOCK,
the order every existing dual-lock site uses, so it can't form an
advisory-lock cycle with a refresh snapshot or a candidate finalization.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: version the audit builder behind durable references (#264)
Stored inputs reproduce a bundle only under the builder that first built
it. If a release changes builder output, a pending candidate that survived
the upgrade would rebuild into a different bundle than its coinbase commits
to, and fail on every retry or land changed audit data.
qbit-prism gains AUDIT_BUILDER_VERSION, which both the 007 candidate and
008's StoredPrepared store. A rebuild checks it before read_window: a
mismatched candidate is retried with an alert and never rebuilt with the
current builder, and a mismatched job is a cache miss. A release that
bumps the constant ships a migration step that refuses while pending
outbox rows carry another version; they drain through the 007 procedure.
Keeping frozen old builders is recorded as the rejected alternative.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: pin the signing keys behind durable references (#264)
The bundle embeds both public keys and their signatures, so a rebuild
reproduces the original bytes only under the original keys. Rotating a
key takes an operator resetting the cluster fingerprint, and a candidate
still pending across that reset would be re-signed into bytes its
coinbase does not commit to.
Both documents now store signer_keys, the two public keys the building
frontend signed with. A rebuild checks them next to the builder version,
before read_window, and never re-signs a mismatch with the local keys.
#265 extends configure to refuse a new fingerprint while a pending outbox
row stores other keys, so the operator drains with the old keys first.
Keeping retired seeds online to reproduce old signatures is recorded as
the rejected alternative.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: reuse the refresh-time WindowRef at submit (#264)
Submit built the candidate's reference from the prepared Snapshot, which
re-hashes and re-serializes the whole 233 to 260 MB window whenever a
block is found. That puts 0.7 to 1.4 s of digest CPU on the
latency-critical share path, on top of the once-per-refresh cost the
cost table budgets. refresh_once already computes the identical
reference, so Prepared now keeps it as window: WindowRef. Submit clones
that small value into the candidate, a resume takes it from
StoredPrepared, and re-digesting at submit is a must-not for #265.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: sort rebuilt prior balances canonically (#264)
prior_balances_digest sorts internally, but AuditBundle.prior_balances
serializes in vector order. read_prior_balances has no outer ORDER BY,
and SQL doesn't guarantee the function's internal order through a table
expression. Balances returned in a different order at claim or resume
would pass the digest check and still change the canonical bundle.
read_prior_balances now sorts with the digest's comparator (order_key,
recipient_id, p2mr_program_hex, byte order). Refresh and every rebuild
share that function, so the builder gets the same vector. A SQL ORDER BY
alone would follow the database collation, not the digest's byte order.
A #265 unit test requires permutations to give identical bytes.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: finish recovered claims from landed digest columns (#264)
A claim that finds its block's audit already landed finished through
materialize_audit_row. That function reads the whole range unpaged,
serializes and clones it on the async runtime, and builds an owned
AuditBundle: two share arrays at 400k on the submit loop, which the
cooperative 60 s deadline cannot interrupt.
The recovery branch now checks the landed row's columns alone, as
today's landing does for an existing audit. share_snapshot_sha256 must
equal window_snapshot_sha256, found_block_bits must match the header,
and the claim then finishes. materialize_audit_row on the submit loop is
a #265 must-not. If that path ever needs the body, #267's paged,
parts-based reader becomes a prerequisite.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: re-pin the WindowRef record to 3.x.x after #283 (#264)
#283 landed through #300: the 2.x.x tip merge and the split of ledger.rs
into candidates, jobs, window and connect submodules. The record's code
citations were pinned to 1398bbc, and many pointed into the old ledger.rs.
Every citation now resolves at 3.x.x 41a2afd: 76 of 252 changed, and all
247 cited line numbers were checked against that tree.
"Where the code lives" now describes what #300 landed. ledger/window.rs
exists under workstream B and already holds share_from_row and
read_prior_balances, and SELECT_SHARE stays in the module root. The only
question left for Anatolie is whether read_range moves out of A-owned
ledger/audit.rs.
Two review fixes in the landed-audit recovery path. A retried claim
authenticates the landed audit against the block it found: the landed
coinbase must match block_hex, and the root of the landed commitment
leaves must equal the coinbase's witness reserved value. For an empty
window, the snapshot comparison uses the digest of [bootstrap_share],
because window_snapshot_sha256 is NULL there.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: keep pool connections free during window rebuilds (#264)
build_slots bounds concurrent rebuilds at build_workers, but that doesn't
protect the database pool. PRISM_DATABASE_MAX_CONNECTIONS may be as low as
4, build_workers may exceed it, and share appends and candidate-lease
heartbeats share the pool with a 15 s acquire timeout. Rebuilds of
distinct jobs could hold every connection for their whole multi-page
reads, and a heartbeat that fails to get a connection drops recoverable
claim work.
read_window now also takes a permit from a window_reads semaphore sized
clamp(database_max_connections - 2, 1, build_workers). The caller takes it
after build_slots and releases it before the rebuild, so two connections
always stay free. Calling read_window without that permit is a must-not
for #265 and #273. A dedicated pool was rejected because it would exceed
the configured connection budget.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: continue to submission after recognizing a landed audit (#264)
The recovery path finished a claim as soon as it authenticated a landed
audit. But an inactive block lands before the lease renewal and
submitblock, so an earlier claim can land and lose its lease before it
submits. Finishing then would abandon a valid found block that was never
submitted.
A recognized landed audit now means landing is done. The claim skips
read_window, the rebuild and land_candidate, and continues through
today's observe, renew and submitblock flow, finishing only where today's
code does: an active block, or an inactive one whose revision or parent
moved. Finishing a claim only because its audit landed is a #265
must-not.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: reserve in-flight windows and scope the claim deadline (#264)
Two review fixes.
The retention floor covered written references only. Ledger::snapshot
releases its locks when its first transaction commits, and the scan, the
bundle and save_job follow with no row referencing the window yet. A prune
in that gap, after a fall in difficulty, could delete rows of the window
being built. D6 now adds a reservation. The snapshot's locked first
transaction inserts it, the refresh fills in the window start after the
scan, and save_job deletes it (publishing nothing if it expired or the
rows are gone). The prune skips while any reservation is unresolved and
otherwise keeps every reserved window. An age-based horizon was rejected
because a window's reach in time depends on hashrate.
The claim's 60 s timeout covered the whole call, including submitblock,
whose RPC timeout may be up to 86,400 s. It now covers read_window plus
the rebuild only, and the later steps keep their existing deadlines.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: answer djh58's four B contract points (#264)
djh58's review on #273 (comment 5624891010) raised four points against
the resume contract. The record now:
- lets B's published-identity and lease check decide resume eligibility
instead of revision equality, and rebuilds eligible work with its
as-issued balances from an immutable qbit_prism_balance_snapshots row
keyed by prior_balances_digest;
- keeps reconstruction failures (Incomplete, digest mismatch, decode,
database, expiry) distinct from missing work, under one end-to-end
deadline, the caller's existing initial_job_timeout_seconds, with no
inner timeout;
- names both share digests, the native snapshot_sha256 and #274's
canonical_digest_hex, amends #274's reuse instruction, and adds a cost
model for a sliding window;
- moves the template out of the prepared payload into an immutable
qbit_prism_templates row, keeping #273's 1 MB criterion whole-value, and
moves #265's block_hex into a bytea column.
It also records the permit-ownership and coalesced-waiter rules from the
same review.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: add the balance source to read_window and authenticate block bytes (#264)
The resume contract needs as-issued balances, but read_window's declared
API had no way to ask for them, and the read algorithm always used the
current set. Implemented literally, it would have returned
PriorBalancesChanged for the published work it was meant to rebuild.
read_window now takes a BalanceSource. Current reads the current view as
before. AsIssued reads the qbit_prism_balance_snapshots row keyed by the
reference's digest, sorts it with the digest's comparator and verifies it.
A missing row is the new BalanceSnapshotMissing, and a mismatching one is
Decode. Claims pass Current, and resumes of published work pass AsIssued.
Moving block_hex to a bytea column took it out of candidate_sha256. The
candidate JSON now keeps block_sha256. Claim decode checks it and that the
block header hashes to block_hash before any witness derivation, landing
or submission.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: prune externalized templates and balance snapshots with their jobs (#264)
Moving each full template into an immutable qbit_prism_templates row left
no cleanup for those rows. Templates change on every new transaction set
and at every reanchor, so each is a new, possibly multi-megabyte row, and
prune_expired_jobs deletes only job rows. The balance snapshots had the
same gap.
prune_expired_jobs becomes one transaction under SETTLEMENT_LOCK, the lock
save_job holds while it inserts the template, the balance snapshot and the
job row. It deletes the expired batch, then the template and balance rows
that batch referenced and no remaining job row references, in two
statements, because a data-modifying CTE's NOT EXISTS would still see the
deleted jobs. The job row gains an indexed template_sha256. Balance
snapshots follow job rows only, since claims read the current balances.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: keep the issued revision on resume and ratchet D6's horizon (#264)
Two review fixes.
A resumed Snapshot now keeps StoredPrepared.payout_revision, the revision
the job was issued with. read_window's current revision is only B's
eligibility input and the claim's probe hint. The job wire and any
candidate take their revision …
…knowledgement bound from dropping a disposition poll's connection (#781) * main: update tokio to 1.53.3 for RUSTSEC-2026-0386 The Dependency audit job (`cargo deny --locked check`) fails on RUSTSEC-2026-0386: tokio 1.53.1's safe `AsyncFd` constructors violate I/O safety. Move tokio to its first fixed patch release, 1.53.3, in the root and fuzz lockfiles, as #777 did for the TLS advisories. Nothing in the workspace uses `AsyncFd`. Only tokio's version and checksum change. `cargo update -p tokio --precise 1.53.3` also moved the windows-sys edges of four unrelated crates (errno, quinn-udp, rustix, tempfile) from 0.61.2 to 0.52.0, so the edit keeps those as they were; both lockfiles pass `cargo metadata --locked`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * main: keep the block-proof acknowledgement bound from dropping a disposition poll's connection b574_ack_cap's a_captured_proof_is_answered_at_the_cap_and_credited_after_its_landing failed on #780 with `proxy connection task failed: Connection reset by peer (os error 104)`, which the ledger execution proxy reported when the fixture closed. `deferred_share_disposition` wrapped each poll's checkout and SQL in `timeout_at(bound, ...)`. When the bound fell inside the checkout, during the pool's `before_acquire` ping of a connection idle for 100 ms or a new connection's setup, SQLx dropped the raw connection with the reply unread, and the kernel answered with a TCP reset. The test proxy reports that as a failed connection; PostgreSQL logs it as a client reset, and the pool loses a healthy connection. Each poll now runs in its own task, as the duplicate probe and enqueue do (#574), and only the wait for it is bounded. Answers, phases and `block_proof_ack_capped_total` are unchanged. A poll still running at the bound, or when the submission is cancelled, finishes on its own and returns its connection, so `database_pool_acquire_seconds` records its checkout when it ends instead of as a cancelled failure. Two acquire tests and docs/prism-pool-acquire-metrics.md now say so. A bound already past on entry, as for a #657 capture that committed in the share grace, starts no poll: the old code's poll there could only be cut off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* docs: design record for WindowRef and Ledger::read_window (#297)
* docs: design record for WindowRef and Ledger::read_window (#264)
Add docs/prism-coordinator-refactor/window-ref.md, the design record that
#265 (migration 007) and #273 (migration 008) implement against, and link
it from the directory README.
The record specifies:
- WindowRef { anchor_ms, prior_balances_digest, shares: Option<ShareRange> }
- the six window_* columns and their three-state CHECK
- Ledger::read_window: one REPEATABLE READ READ ONLY transaction,
mandatory ascending keyset paging, a streaming digest, typed errors and
never a partial window
- per-caller revision fences that keep today's observe_candidate rule for
claims
- the non-window bundle inputs a rebuild stores instead of reading config
- the upgrade procedure, retention floor and expected cost at 400k shares
- the borrowed qbit-prism parts API merged in #296
It went through two adversarial reviews.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: hex-serialize WindowRef digests and clamp the resume timeout (#264)
Two fixes to the WindowRef design record after review:
- A derived Serialize writes a [u8; 32] as a JSON array of 32 integers, not
the lowercase 64-hex string the payload and column contract requires. Both
digest fields now carry #[serde(with = "hex32")], a small adapter in
ledger/window.rs that writes lowercase hex and reads exactly 64 lowercase
hex characters. The server's hex dependency has no serde feature.
- The proposed resume timeout, initial_job_timeout_seconds minus 5 s, is
zero or negative for any setting at or below 5 s, which validation allows.
It is now outer - min(5 s, outer / 2), computed from the Duration the
caller already builds: 25 s at the 30 s default, always positive, strictly
shorter than the outer timeout, and free of Duration underflow.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: bootstrap resumes use stored policy inputs (#264)
An empty-window resume rebuilt each miner's bootstrap bundle through
Coordinator::build_bundle. That function reads config.ctv_enabled,
payout_policy, ctv_direct_floor and ctv_config, so after a config or
fingerprint reset a resumed job could get a different signed bundle under
the same job ID. That is the drift the stored-inputs invariant forbids.
The record now has the per-miner bootstrap rebuild call the borrowing
builders directly. It takes its own build_slots permit after the
single-flight entry's permit is released. Its inputs are the synthetic
share build_bundle fabricates today, the template, anchor, payout_policy
and ctv stored in StoredPrepared, and the balances and revision read_window
returned. #273's must-not row now forbids build_bundle for any resume
rebuild.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: readable imported audits and a race-free upgrade order (#264)
Two fixes to the WindowRef design record after review.
Imported legacy audits: #265 stores canonical_audit_bytes and no inline
body. Only audit_canonical_bytes reads those bytes today. Ledger::audit_bundle
returns just the JSON column, so backfill-ctv would stop on imported rows,
and the bundle endpoint's fallback reads only body_uri. The record now
requires the same PR to make both readers decode the stored canonical
bytes first, digest-checked and under spawn_blocking. Leaving an imported
audit readable only through body_uri is now forbidden.
Upgrade order: a zero pending count taken while the pre-007 frontends still
run proves nothing, because a block can be found before they stop. The
count is now taken after every frontend is stopped. A non-zero count means
restarting the pre-007 frontends to drain and repeating, and 007's refusal
remains the backstop.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: one owned window per Prepared and the established lock order (#264)
Two fixes to the WindowRef design record after review.
Window ownership: Prepared and JobContext would have held an owned
AuditBundle beside the Snapshot. into_bundle takes the share Vec by value,
and two Arcs don't share storage, so every resume would keep a second
window: about 0.75 GB per 400k rebuild instead of 0.5 GB. Both structs now
hold Arc<Snapshot>, the only owned window, next to Arc<AuditBundleBody>.
Canonical bytes and verification come from the parts functions, and an
owned bundle is assembled only at claim.
Lock order: the D6 prune now takes SETTLEMENT_LOCK and then ORDER_LOCK,
the order every existing dual-lock site uses, so it can't form an
advisory-lock cycle with a refresh snapshot or a candidate finalization.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: version the audit builder behind durable references (#264)
Stored inputs reproduce a bundle only under the builder that first built
it. If a release changes builder output, a pending candidate that survived
the upgrade would rebuild into a different bundle than its coinbase commits
to, and fail on every retry or land changed audit data.
qbit-prism gains AUDIT_BUILDER_VERSION, which both the 007 candidate and
008's StoredPrepared store. A rebuild checks it before read_window: a
mismatched candidate is retried with an alert and never rebuilt with the
current builder, and a mismatched job is a cache miss. A release that
bumps the constant ships a migration step that refuses while pending
outbox rows carry another version; they drain through the 007 procedure.
Keeping frozen old builders is recorded as the rejected alternative.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: pin the signing keys behind durable references (#264)
The bundle embeds both public keys and their signatures, so a rebuild
reproduces the original bytes only under the original keys. Rotating a
key takes an operator resetting the cluster fingerprint, and a candidate
still pending across that reset would be re-signed into bytes its
coinbase does not commit to.
Both documents now store signer_keys, the two public keys the building
frontend signed with. A rebuild checks them next to the builder version,
before read_window, and never re-signs a mismatch with the local keys.
#265 extends configure to refuse a new fingerprint while a pending outbox
row stores other keys, so the operator drains with the old keys first.
Keeping retired seeds online to reproduce old signatures is recorded as
the rejected alternative.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: reuse the refresh-time WindowRef at submit (#264)
Submit built the candidate's reference from the prepared Snapshot, which
re-hashes and re-serializes the whole 233 to 260 MB window whenever a
block is found. That puts 0.7 to 1.4 s of digest CPU on the
latency-critical share path, on top of the once-per-refresh cost the
cost table budgets. refresh_once already computes the identical
reference, so Prepared now keeps it as window: WindowRef. Submit clones
that small value into the candidate, a resume takes it from
StoredPrepared, and re-digesting at submit is a must-not for #265.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: sort rebuilt prior balances canonically (#264)
prior_balances_digest sorts internally, but AuditBundle.prior_balances
serializes in vector order. read_prior_balances has no outer ORDER BY,
and SQL doesn't guarantee the function's internal order through a table
expression. Balances returned in a different order at claim or resume
would pass the digest check and still change the canonical bundle.
read_prior_balances now sorts with the digest's comparator (order_key,
recipient_id, p2mr_program_hex, byte order). Refresh and every rebuild
share that function, so the builder gets the same vector. A SQL ORDER BY
alone would follow the database collation, not the digest's byte order.
A #265 unit test requires permutations to give identical bytes.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: finish recovered claims from landed digest columns (#264)
A claim that finds its block's audit already landed finished through
materialize_audit_row. That function reads the whole range unpaged,
serializes and clones it on the async runtime, and builds an owned
AuditBundle: two share arrays at 400k on the submit loop, which the
cooperative 60 s deadline cannot interrupt.
The recovery branch now checks the landed row's columns alone, as
today's landing does for an existing audit. share_snapshot_sha256 must
equal window_snapshot_sha256, found_block_bits must match the header,
and the claim then finishes. materialize_audit_row on the submit loop is
a #265 must-not. If that path ever needs the body, #267's paged,
parts-based reader becomes a prerequisite.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: re-pin the WindowRef record to 3.x.x after #283 (#264)
#283 landed through #300: the 2.x.x tip merge and the split of ledger.rs
into candidates, jobs, window and connect submodules. The record's code
citations were pinned to 1398bbc, and many pointed into the old ledger.rs.
Every citation now resolves at 3.x.x 41a2afd: 76 of 252 changed, and all
247 cited line numbers were checked against that tree.
"Where the code lives" now describes what #300 landed. ledger/window.rs
exists under workstream B and already holds share_from_row and
read_prior_balances, and SELECT_SHARE stays in the module root. The only
question left for Anatolie is whether read_range moves out of A-owned
ledger/audit.rs.
Two review fixes in the landed-audit recovery path. A retried claim
authenticates the landed audit against the block it found: the landed
coinbase must match block_hex, and the root of the landed commitment
leaves must equal the coinbase's witness reserved value. For an empty
window, the snapshot comparison uses the digest of [bootstrap_share],
because window_snapshot_sha256 is NULL there.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: keep pool connections free during window rebuilds (#264)
build_slots bounds concurrent rebuilds at build_workers, but that doesn't
protect the database pool. PRISM_DATABASE_MAX_CONNECTIONS may be as low as
4, build_workers may exceed it, and share appends and candidate-lease
heartbeats share the pool with a 15 s acquire timeout. Rebuilds of
distinct jobs could hold every connection for their whole multi-page
reads, and a heartbeat that fails to get a connection drops recoverable
claim work.
read_window now also takes a permit from a window_reads semaphore sized
clamp(database_max_connections - 2, 1, build_workers). The caller takes it
after build_slots and releases it before the rebuild, so two connections
always stay free. Calling read_window without that permit is a must-not
for #265 and #273. A dedicated pool was rejected because it would exceed
the configured connection budget.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: continue to submission after recognizing a landed audit (#264)
The recovery path finished a claim as soon as it authenticated a landed
audit. But an inactive block lands before the lease renewal and
submitblock, so an earlier claim can land and lose its lease before it
submits. Finishing then would abandon a valid found block that was never
submitted.
A recognized landed audit now means landing is done. The claim skips
read_window, the rebuild and land_candidate, and continues through
today's observe, renew and submitblock flow, finishing only where today's
code does: an active block, or an inactive one whose revision or parent
moved. Finishing a claim only because its audit landed is a #265
must-not.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: reserve in-flight windows and scope the claim deadline (#264)
Two review fixes.
The retention floor covered written references only. Ledger::snapshot
releases its locks when its first transaction commits, and the scan, the
bundle and save_job follow with no row referencing the window yet. A prune
in that gap, after a fall in difficulty, could delete rows of the window
being built. D6 now adds a reservation. The snapshot's locked first
transaction inserts it, the refresh fills in the window start after the
scan, and save_job deletes it (publishing nothing if it expired or the
rows are gone). The prune skips while any reservation is unresolved and
otherwise keeps every reserved window. An age-based horizon was rejected
because a window's reach in time depends on hashrate.
The claim's 60 s timeout covered the whole call, including submitblock,
whose RPC timeout may be up to 86,400 s. It now covers read_window plus
the rebuild only, and the later steps keep their existing deadlines.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: answer djh58's four B contract points (#264)
djh58's review on #273 (comment 5624891010) raised four points against
the resume contract. The record now:
- lets B's published-identity and lease check decide resume eligibility
instead of revision equality, and rebuilds eligible work with its
as-issued balances from an immutable qbit_prism_balance_snapshots row
keyed by prior_balances_digest;
- keeps reconstruction failures (Incomplete, digest mismatch, decode,
database, expiry) distinct from missing work, under one end-to-end
deadline, the caller's existing initial_job_timeout_seconds, with no
inner timeout;
- names both share digests, the native snapshot_sha256 and #274's
canonical_digest_hex, amends #274's reuse instruction, and adds a cost
model for a sliding window;
- moves the template out of the prepared payload into an immutable
qbit_prism_templates row, keeping #273's 1 MB criterion whole-value, and
moves #265's block_hex into a bytea column.
It also records the permit-ownership and coalesced-waiter rules from the
same review.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: add the balance source to read_window and authenticate block bytes (#264)
The resume contract needs as-issued balances, but read_window's declared
API had no way to ask for them, and the read algorithm always used the
current set. Implemented literally, it would have returned
PriorBalancesChanged for the published work it was meant to rebuild.
read_window now takes a BalanceSource. Current reads the current view as
before. AsIssued reads the qbit_prism_balance_snapshots row keyed by the
reference's digest, sorts it with the digest's comparator and verifies it.
A missing row is the new BalanceSnapshotMissing, and a mismatching one is
Decode. Claims pass Current, and resumes of published work pass AsIssued.
Moving block_hex to a bytea column took it out of candidate_sha256. The
candidate JSON now keeps block_sha256. Claim decode checks it and that the
block header hashes to block_hash before any witness derivation, landing
or submission.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: prune externalized templates and balance snapshots with their jobs (#264)
Moving each full template into an immutable qbit_prism_templates row left
no cleanup for those rows. Templates change on every new transaction set
and at every reanchor, so each is a new, possibly multi-megabyte row, and
prune_expired_jobs deletes only job rows. The balance snapshots had the
same gap.
prune_expired_jobs becomes one transaction under SETTLEMENT_LOCK, the lock
save_job holds while it inserts the template, the balance snapshot and the
job row. It deletes the expired batch, then the template and balance rows
that batch referenced and no remaining job row references, in two
statements, because a data-modifying CTE's NOT EXISTS would still see the
deleted jobs. The job row gains an indexed template_sha256. Balance
snapshots follow job rows only, since claims read the current balances.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: keep the issued revision on resume and ratchet D6's horizon (#264)
Two review fixes.
A resumed Snapshot now keeps StoredPrepared.payout_revision, the revision
the job was issued with. read_window's current revision is only B's
eligibility input and the claim's probe hint. The job wire and any
candidate take their revision from the snapshot, so a block found on R0
work carries R0, and observe_candidate takes the superseded path instead
of retrying PriorBalancesChanged forever.
A D6 horizon computed from today's difficulty fails when difficulty rises
after a prune: Ledger::snapshot stops when rows run out, so it would
publish an underweight window. D6 now needs a ratcheted horizon kept at a
durable maximum difficulty in qbit_prism_cluster. It must also record the
share_seq it deleted below, so snapshot fails when its scan reaches that
floor before the requested weight.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: drop rebuilt windows from resumed jobs once the coinbase is built (#264)
build_workers bounds how many rebuilds run at once, not how many rebuilt
windows stay resident. Each resumed JobContext kept its Prepared, with
Arc<Snapshot> and Arc<AuditBundleBody>, and a session keeps up to 64 jobs
per connection. Sequential resumes of different jobs could therefore pin
about 0.5 GB each, and the stated k x 0.5 GB peak didn't hold.
Once the rebuild has produced the job's coinbase, a resumed Prepared keeps
only what submit reads: the WindowRef, the snapshot scalars, the template,
the fee and the body's non-window fields. Submit reads nothing else, and
the whole-bundle clone into the candidate goes with #265. Snapshot.shares
and reward_manifest.shares are dropped before the job reaches the session,
so windows exist only inside rebuilds.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: stop old signers before a reset and fence writes on the fingerprint (#264)
configure's one-time pending-row check could pass while an old-key
frontend was still running. That frontend could then find a block and
enqueue a candidate with the old signer_keys, which no new-key frontend
can rebuild, and enqueue is not fenced by the fingerprint today.
A rotation now follows the 007 order: drain, stop every old-key frontend,
then check that nothing is pending, reset and start the new frontends.
#265 and #273 also re-read config_fingerprint FOR SHARE inside every
enqueue and save_job transaction and refuse writes under a foreign
fingerprint. FOR SHARE conflicts with configure's FOR UPDATE, so a write
and a reset cannot interleave, and a procedure violation fails loudly
instead of leaving rows nobody can rebuild.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: keep every stored input in slim resumed jobs (#264)
The slim resumed Prepared kept the WindowRef, the snapshot scalars, the
template, the fee and the body's non-window fields, but not the stored
payout_policy and nested ctv. A candidate found on that job must persist
those exact inputs, and neither the fee nor the coinbase manifest can
recover them, so submit would have had to read local configuration.
The slim form now keeps the decoded StoredPrepared itself: payout_policy,
the whole ctv, audit_builder_version, signer_keys, coinbase_suffix,
template_sha256, the WindowRef and the snapshot scalars, all O(1). For an
empty window it also keeps the miner's own bootstrap_share. Only the
window is dropped. Dropping a stored input before enqueue is a #273
must-not.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: let the published-work lease reach submission (#264)
Keeping R0 on a resumed snapshot doesn't help if its found block never
reaches observe_candidate. Today submit rejects work whose frontend or job
revision differs from the ledger's, apart from parent-stale work inside
the stale grace. That grace work is credited as a share, but a candidate
is enqueued only when the work isn't stale, so its block is dropped.
B8's lease now decides those submit checks the way it decides resume
eligibility. Work the lease covers is not stale because the revision
changed, and a block found on it is enqueued as a candidate carrying R0,
with its as-issued window and inputs. observe_candidate then decides at
claim whether it lands. Work outside the lease keeps today's handling.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: submit leased candidates first and clear terminal block bytes (#264)
Two review fixes.
A candidate enqueued from leased R0 work still met the unchanged claim
fence, which would finish it as superseded before submitblock and discard
the block. The candidate now records a leased flag. A leased candidate
skips the pre-submit supersession checks and the pre-submit landing,
renews its lease, calls submitblock, and only then meets
observe_candidate. Whether an active leased block lands is B8's rule with
#289.
Moving block_hex into the block_bytes bytea column meant the terminal
UPDATE no longer removed the block. It now NULLs block_bytes too, so the
outbox doesn't keep every finished block forever.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: correct claim-flow line citations to 3.x.x 41a2afd (#264)
The landed-audit continuation and leased-candidate paragraphs cited
coordinator.rs claim-flow lines read from a later 3.x.x, where #308 had
shifted them by five. They now point at the record's pin, 41a2afd:
the post-rebuild observe at :1039, the active branch at :1040-1048, the
inactive supersession at :1050-1063, the landing at :1067-1069, the
post-land check at :1070-1080, the lease renewal and submitblock at
:1084-1092, and the post-submit observe at :1095.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: submit leased candidates before any rebuild (#264)
A leased candidate skipped the pre-submit fences, but the #265 claim
rebuild still called read_window with BalanceSource::Current before
them. For an R0 candidate that returns PriorBalancesChanged and retries
forever, so the block never reached submitblock.
A leased candidate now submits before any rebuild, because submitblock
needs only the stored block. If the block is active, its audit rebuild
uses BalanceSource::AsIssued. A non-terminal leased outbox row therefore
keeps its balance snapshot. The prune checks for such rows and takes
SETTLEMENT_LOCK then ORDER_LOCK, so it cannot interleave with a leased
enqueue.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: release rebuilt windows off the runtime; hold the lease to unchanged balances (#264)
Two review fixes.
Freeing a completed rebuild's last Arc<Snapshot> and Arc<AuditBundleBody>
drops about 0.5 GB on whichever Tokio worker releases it, which can stall
lease heartbeats and share processing. The rebuild's spawn_blocking closure
now builds the coinbase and the slim resumed form and drops the window
there. Every other release of a rebuilt window goes through
spawn_blocking(drop).
An active leased block rebuilt with R0's balances could not land, because
landing requires the current balances. Until B8 and #289 define how such a
block lands, the lease covers only revision changes that leave
prior_balances_digest unchanged, and a balance change ends it at once. An
active leased block then lands through today's path, and the open
question asks djh58 and #289 for the full rule.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: land first-time claims from parts; sweep orphaned balance snapshots (#264)
Two review fixes.
A first-time landing assembled the rebuilt window into an AuditBundle, then
cloned it, serialized it whole for the stored body and the canonical length,
and re-read the range unpaged, all on a Tokio worker. At 400k shares that is
several half-gigabyte representations on the runtime. CandidateClaim now
holds the parts. Landing verifies, serializes and digests them in one
spawn_blocking closure before its transaction opens. It keeps the range
fence, but as a paged re-read. #267 is not a prerequisite.
The balance-snapshot prune only considered digests from the expired batch,
so a snapshot kept alive by a leased candidate could never be reclaimed once
its job had gone. Snapshots now get their own statement over every row with
no remaining reference.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: keep the bootstrap share in local jobs; correct local window residency (#264)
Three review fixes.
An empty-window local job held its only full AcceptedShare in
JobContext.bundle. Once JobContext holds a body, that share would be gone
and submit could not fill Candidate.bootstrap_share. JobContext now keeps
the synthetic share beside the body.
The record claimed the local frontend keeps one full window. Retired local
jobs keep their generation's Prepared for Stratum's retention period, as they
do today. The record now says so. Bounding that is logged as a #273
acceptance note.
The landing bind still copies the stored body into SQLx's argument buffer on
the runtime. The record now names that copy. Measuring it is logged as a
#265 acceptance note.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: keep null-response leased candidates pending; carry empty-window balances (#264)
Two review fixes.
The record finished an inactive leased candidate as superseded after
submission. Today's wrapper treats a null submitblock result on an inactive
block as unresolved, because it can still be a known side-chain block, and
retries with the candidate intact. Leased candidates now keep that outcome,
so a block that becomes active later still has its audit inputs.
For an empty window, resume waiters build their own bootstrap bundles from
the as-issued balances. The single-flight entry now carries those balances
until every waiter has built, instead of only the rebuilding caller holding
them.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: land a submitted leased candidate before any terminal outcome (#264)
After a null submitblock result on an inactive leased block, the retry
resubmitted before landing. A node can then answer "duplicate", which the
string branch treats as terminal, clearing the unlanded audit inputs.
Today's flow avoids that by landing an inactive block before its RPC, so a
late acceptance stays reconcilable.
A leased candidate still submits first, so delivery never waits on a
rebuild. It then rebuilds with AsIssued and lands at the observed
revision before any terminal outcome, whether or not the block is active,
and only then takes today's three outcomes.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: re-establish candidate references inside the enqueue transaction (#264)
Taking locks in order stops a prune and an enqueue from interleaving, but
it doesn't decide which runs first. A submission that passed its expiry
check could lose to a prune that deleted the job row and its balance
snapshot, and then enqueue a leased candidate pointing at missing data.
Enqueue now re-inserts a leased candidate's balance snapshot from the
as-issued balances the job carries (the slim resumed form keeps them),
and probes the window's first share row, both inside its ORDER_LOCK
transaction. Shares can't be written back, so an expired job row now holds
the retention floor until the prune deletes it, and the prune waits a grace
past expiry. A failed probe still publishes the candidate, with an alert.
Part of #261.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* 3.x.x: make Prism session sequence wrap-safe (#319)
* fix(prism): reserve session extranonces safely across sequence wrap
* fix(prism): fence stopped session owners by process incarnation
* fix(prism): drain session guards before shutdown cleanup
* fix(prism): scope migration readiness to mining startup
* test(prism): build complete legacy schema for read-only connect
* fix(tests): route integration databases through test gate
* test(prism): register all gated session coverage
* fix(tests): adapt fixtures to shared session backend
* 3.x.x: Generate native metric inventory and migrate deployed alerts (#320)
* docs(prism): generate metric inventory and migrate deployed alerts
* fix(prism): preserve the network-wide connected client alert floor
* fix(prism): evaluate standby lag from observed durable WAL progress
* fix(prism): delete provisioned native rules when gates disable them
* fix(prism): alert on invalid synchronous standby configuration
* docs(prism): record local review evidence and prepare PR draft
* fix(prism): reject stale alert and metric metadata
* fix(prism): alert when public scrape fails
* fix(prism): preserve native failure and paging alerts
* fix(prism): preserve native alert tuning overrides
* Revert "fix(prism): preserve native alert tuning overrides"
This reverts commit 31b26812ef133a6cfb7e76deefb11e50a2bf306e.
* fix(prism): preserve configurable alert tuning
* fix(prism): render tuning dwells as JSON
* fix(prism): validate rendered alert dwell overrides
* 3.x.x: finish native health contract evidence (#329)
* docs(prism): pin intentional legacy health differences
* docs(prism): correct legacy max blocks semantics
* style(prism): format health snapshot assertions
* v3.0.0 feat: add prepared-window reference foundation (#327)
* fix: preserve PRISM issued-work and stale-grace semantics
Separate observed and published tip authority, retain original issued
worker context through replacement and reconnects, and preserve bounded
delivery-based grace without renewing expiry after awaited work.
Exercise real Coordinator decisions and socket interleavings, including
constant node-RPC load across 1, 32 and 128 clients.
Refs #280
* docs: reference PRISM miner parity settings
Co-Authored-By: OpenAI Codex <noreply@openai.com>
* fix(prism): bind share admission to one published payout view
* fix(prism): verify changed block notifications with fresh chain state
* docs(prism): record B8 review regressions and rebased qualification
* test(prism): adapt merged admission fixture to stale grace contract
* fix(prism): classify retained capacity by current published tip
* fix(prism): retain prepared dependencies for issued work
* test(prism): verify issued job dependency persistence
* fix(prism): keep sessions alive while retained work is usable
* docs(prism): record timer coverage and final qualification procedure
* docs: point D2 credit explanation to miner submit module
* test: align candidate lease tip fixture with coordinator state
* fix: integrate miner parity with native metrics registry
* fix: integrate miner parity with metrics registry API
* ci: run issued job dependency acceptance explicitly
* test: keep miner parity fixture publication reachable
* test: complete JSONB gate fixture config
* docs: specify prepared window reference integration plan
* test: restore D2 coordinator coverage
* feat(prism): add window reference schema and authenticated reader
Add the independently approved B273 schema and reader foundation on PR313, with typed reconstruction errors, immutable blob tables, per-version migrations, and PostgreSQL qualification. Runtime prepared/resume, candidate, atomic repair and GC integration remain gated on A265 and shared owner agreement.
* feat(prism): add coherent payout state for lease eligibility
* Keep cancelled window cleanup off runtime workers and qualify current base
* Qualify landed 008 and 009 migration orders on current base
* Preserve legacy balance order and reject fatal window reads
* Prove legacy balance order preservation in PostgreSQL
* Run all gated suites before manifest proof
* Revert "Run all gated suites before manifest proof"
This reverts commit b11cbbb183927db9b51daf6bf0176a09c4afc0b8.
---------
Co-authored-by: OpenAI Codex <noreply@openai.com>
* 3.x.x: store imported legacy audits as canonical bytes and decode them on read (#325)
* 3.x.x: store imported legacy audits as canonical bytes and decode them on read
The legacy-audit import wrote the parsed bundle back into the
audit_bundle JSONB column. That value holds two copies of the payout
window (shares and reward_manifest.shares), so at 400k shares it
crosses PostgreSQL's JSONB container limit. The import now stores
only canonical_audit_bytes and the non-share metadata. It keeps its
idempotency and optimistic guards, verifies off the runtime, and drops
the parsed window before the write transaction opens. Rows that
already hold an inline body keep it unchanged, because the
body-present CHECK needs one of audit_bundle and body_uri.
found_block_bits stays as stored: no audit body carries the compact
bits.
A shared decode_canonical_audit_body helper checks the stored bytes
against audit_bundle_sha256 and parses them with
parse_audit_bundle_value. It returns the logical body the import used
to store inline, and all of that work runs under spawn_blocking.
Ledger::audit_bundle serves native snapshot rows first, then imported
canonical bytes, then inline bodies, so backfill-ctv works on imported
rows. The bundle endpoint serves imported bytes before any body_uri
fallback, on the same public pool. A digest mismatch or undecodable
bytes refuses the row, even when the legacy file still exists. The
settlement-artifacts audit link now also accepts stored canonical
bytes, so it does not decode a whole body just to show a link.
The JSONB ceiling gate drops its import-phase known violation, and the
measurement and capacity-readiness docs now count three
window-carrying writes.
Part of #265.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* 3.x.x: load only the served audit body and bound imported-audit decodes
The bundle queries now project no inline audit_bundle for a row whose
canonical bytes supersede it, the same rule Ledger::audit_bundle uses,
and report whether canonical bytes exist so bundle() skips the bytes
query for rows without them. The flag is removed before responding.
Imported-audit decodes run after their read connection is released, so
ApiState now carries a semaphore sized from the same read concurrency as
the public read pool. bundle() takes an owned permit inside the
request's existing deadline and moves it into the blocking decode, so a
dropped request cannot free it before the decode ends. Ledger::audit_bundle
takes no permit and releases its fetched row before decoding.
The decode refuses body-ref and v2 envelope schemas, since canonical
bytes are always a flat bundle. The storage-sizing doc describes the
canonical-bytes import, and the measurement doc gains a post-#265 note.
Tests pin the pre-change row shape (inline body, canonical bytes and
body_uri, no snapshot), bound the decode across a dropped request, and
refuse a digest-valid body-ref envelope.
Part of #265.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* 3.x.x: add deterministic HA readiness probe harness (#331)
* test: add deterministic HA readiness probe harness
* test: qualify readiness probe timing
* test: preserve probe state in jsonl harness
* test: harden readiness probe input boundaries
* test: reject overlapping readiness probes
* docs: use generic HA readiness contract wording
* test: classify malformed probe durations as failures
* 3.x.x: reconcile share-commit deadlines instead of dropping the append (#333)
* ledger: let append callers veto COMMIT at the last moment
append_at_revision_gated takes a non-optional pre-commit hook. It runs at
most once, after every statement including persist_candidate has
succeeded, while ORDER_LOCK is held and immediately before COMMIT. A false
return rolls the transaction back explicitly and fails with the typed
CommitGateClosed error, so COMMIT is never sent. append and
append_at_revision keep their signatures and pass no hook.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* metrics: count late share confirmations and unknown outcomes
A share whose commit is confirmed after the acknowledgement deadline is
still accepted, and a commit whose outcome cannot be established is no
longer reported as a clean rejection. Both need their own signal.
Add the late_confirmed_shares_total family, recorded through
record_late_confirmation and zero-registered like the other counters, and
the ledger-outcome-unknown reject reason. Without the new reason, an
unknown outcome would fold into internal-error.
PrismShareAppendFailures paged on any ledger-confirmation-failed, so it
now matches the unknown reason too and keeps its low-volume total-write-
failure coverage. The deployment patch is regenerated from the reviewed
snapshot, and a scenario covers the new reason alone.
Document the narrowed meaning of ledger-confirmation-failed, the new
reason, the statement-timeout guard, and the block-only acknowledgement
bound.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* coordinator: reconcile share-commit deadlines instead of dropping the append (#324)
submit wrapped the whole persistence future in a timeout and dropped it
when the deadline fired, answering ledger-confirmation-failed. If COMMIT
had already been flushed, sqlx drained the reply and reused the
connection, so the share was committed and credited without an
acknowledgement. A load harness reproduced that: 164 such rejections, two
of them committed rows. The block-only path committed its candidate
first, so its timeout always led to a later credit with no
acknowledgement.
Run the append in a task behind a one-shot commit gate. At the deadline
the acknowledgement path either closes the gate, which proves COMMIT was
never sent and aborts the doomed append, or finds the commit already in
flight and awaits its reply for share_commit_grace. Only a server error
of severity ERROR is a definite rejection; every other error, and a
commit that outlasts the grace, is answered ledger-outcome-unknown and
logged with its eventual outcome. A confirmation whose commit took at
least the effective statement_timeout is also unknown, because a
cancelled synchronous-replication wait commits only locally.
An append carrying a found-block candidate is never refused, since that
would discard the block; it waits for block_only_ack_timeout instead.
Block-only proofs always finish their enqueue and then follow the
candidate's disposition within the same bound, and poll errors after the
enqueue never become a rejection.
The new bounds are Config fields rather than environment variables, and
statement_timeout is read from the session the ledger pool actually uses.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* coordinator: bound the block-only enqueue wait and time COMMIT in the task
Review feedback on PR #333.
enqueue_candidate_once was awaited directly, outside the bound that covers
every other step of a block-only submission, so a degraded database could
hold the acknowledgement open well past block_only_ack_timeout. Wait for it
in a task until the bound instead. The enqueue still runs to completion,
because cutting it off could strand a COMMIT that was already sent, but the
miner is answered ledger-outcome-unknown on time and the enqueue's eventual
outcome is logged.
The synchronous-replication guard compared statement_timeout against a
COMMIT duration measured when the coordinator classified the finished task.
Scheduler delay after a successful COMMIT therefore counted as database
time and could turn a durable accepted share into an unknown one. Measure
the duration inside the append task, where COMMIT actually finishes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* metrics: count a late confirmation from when the append finished
Review feedback on PR #333.
late_confirmed_shares_total was decided by reading the clock while the
acknowledgement was being processed, which can be arbitrarily later than the
append itself. A coordinator task scheduled late therefore counted an
on-time confirmation as a late one. Decide it from the instant the append
finished, which the task already reports, and record it where that instant
is known.
The help text also claimed every counted share was confirmed within the
grace period. That was never true of a share-pass submission carrying a
found-block candidate: those are deliberately never refused, so they can be
confirmed later still, up to the block-only bound, and they are counted.
Say what the counter measures instead, and spell the found-block case out
in the metrics documentation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* perf(metrics): remove allocations from closed-label event recording (#344)
* 3.x.x: add bounded PostgreSQL WAL qualification helper (#330)
* test(prism): add bounded PostgreSQL WAL measurement helper
* fix(prism): harden WAL qualification failure semantics
* fix(prism): harden WAL helper validation and outcomes
* fix(prism): sanitize WAL helper connection inputs
* test(prism): cover WAL helper outcomes and scope
* fix(prism): preserve WAL operation outcome uncertainty
* fix(prism): bound WAL measurement subprocesses
* fix(prism): complete WAL helper diagnostics and test coverage
* test(prism): repair WAL helper test layout
* test(prism): stabilize WAL diagnostic mocks
* 3.x.x: Measure the share-append throughput floor against PostgreSQL (#323)
* 3.x.x: measure the share-append throughput floor against PostgreSQL
test/test-prism-postgres-throughput.sh measured `cargo run -- benchmark`, a
CPU-only run of the audit builder with no database at all, and the 2.x.x
report it replaced measured one bulk INSERT ... SELECT ... FROM
generate_series. Neither says anything about the path a share actually takes.
Every native share append is one transaction that takes the cluster-wide
ORDER_LOCK advisory lock, checks writability, bumps the single-row ledger
clock and writes two rows, so share appends are serialized cluster-wide, and
nobody had measured that with more than one writer.
tests/throughput_floor.rs seeds a payout window with the #264 fixture
generator and then appends through Ledger::append at 1, 2 and 4 concurrent
appenders, each appender holding its own Ledger with its own pool and its own
instance_id, as separate production frontends do. A connection outside every
appender pool samples pg_locks for backends queued behind ORDER_LOCK while a
level runs, and the report says how many waiters it saw and for how long --
labelled throughout as sampled estimates, with the episode count a lower
bound, because a wait shorter than the 10 ms interval can be missed entirely.
A sampler that fails reports null and the error, never zero.
The report is qbit.prism.postgres-throughput.v2, a deliberately new schema
name: its numbers must never be read against v1's bulk-insert rate. It is
written before the floor is asserted, so a run that fails the floor still
leaves its evidence behind. Every QBIT_PRISM_* input is validated at entry,
before a connection is opened: nan, inf, a negative or an empty floor are
configuration errors rather than extreme floors, and a window size that does
not divide the fixture's window weight would not produce the window it claims
to.
CI_MIN_SHARES_PER_SEC is provisional -- 25% of the slowest level rate measured
in a debug build on the writer's host. Debug is the right base because CI runs
this test in debug, and the wide margin is honest about the fact that nothing
has calibrated it on the 2-vCPU runner yet.
Part of #271
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* 3.x.x: point the throughput script and target at the share-append floor
test-prism-postgres-throughput ran `cargo run -- benchmark`, which measures
the audit builder's CPU cost and never opens a database, so the target named
after PostgreSQL throughput was not measuring PostgreSQL at all. It now runs
the full-size share-append floor: the ignored throughput_floor test, in
release, against PRISM_TEST_DATABASE_URL when one is set and otherwise through
prism-native-tests.sh, which initdb's a disposable cluster on a random
loopback port and removes it on exit.
The builder benchmark has not gone anywhere and is still worth running; the
script says so, and gives the exact command, because it answers a different
question rather than a worse version of this one.
`set -e` is suspended around the test so the report path is printed on a red
run too -- the test writes the report before it asserts the floor, so a run
that fails the floor is exactly the run whose report someone wants to read --
and the test's exit status is re-raised immediately afterwards. Every
QBIT_PRISM_* variable reaches the test process unchanged; the script reads
none of them except to say where the report landed.
Part of #271
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* 3.x.x: resolve pg_stat_statements by schema, not by search path
The pg_stat_statements probe was `to_regclass('pg_stat_statements')`, and the
read and the reset named the view unqualified. Every connection in this run
carries `options=-csearch_path=<run schema>`, which replaces the search path
rather than prepending to it, so `public` is not on it. On a server with the
extension installed in `public` the probe therefore returned NULL and the
report said "unavailable" about a view that was sitting right there -- an
EP-OBSERVABILITY failure of exactly the kind the rest of this file is careful
about, because it reads as "this server cannot tell us" when the truth was
"this query looked in the wrong place".
The extension's schema is now resolved from pg_extension and the server's own
quote_ident, and both the reset and the read qualify with it. Verified against
PostgreSQL 16.15 with `shared_preload_libraries=pg_stat_statements` and the
extension created in public: the block now reports 2000 calls per level, and
its total_exec_time corroborates the sampler closely -- 4254 ms against a
sampled 3.97-4.31 waiter-seconds at two appenders, and 17387 ms against a
sampled 16.9-17.3 at four. Two independent methods agreeing to within a few
percent is worth more than either alone, which is the whole reason the block
is there.
Part of #271
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* 3.x.x: attribute ORDER_LOCK waits to the run that caused them
The adversarial review found that the per-run schema isolates the tables but
not the lock. ORDER_LOCK is a PostgreSQL advisory lock and an advisory lock is
scoped to the database, so any other backend appending shares into the same
database serialises against this run's appenders, and the sampler -- filtered
only by database -- counted those foreign waits as this level's. Measured on a
quiet server against the same server with one other appending process, the
rates fell 40-57% and nothing in the report said why. A run like that either
fails a calibrated floor for a reason that is not the append path, or, once the
margin narrows, passes on a number describing two workloads.
Every connection this run opens now carries a per-run application_name,
`throughput-<uuid>`, pushed through the URL -- which is the only thing
Ledger::connect is handed, so the seeding ledger, every appender ledger and the
sampler all announce themselves without any production code knowing this test
exists. sqlx 0.8 parses the parameter out of the URL and sends it in the
startup packet, but the run does not take that on trust: it reads
current_setting('application_name') back on its first connection and refuses to
measure anything if the server disagrees, because a silently ignored parameter
would make every foreign backend look like ours and every contaminated run look
clean.
The sampler then reports this run's waiters as before, and foreign_waiters_max
and foreign_backends_seen beside them. A foreign backend counts whether it was
waiting on the lock or holding it: either way it is serialising against these
appends. A backend whose pg_stat_activity row this role may not read counts as
foreign, which is the safe direction. A level during which any foreign backend
was seen is marked contaminated, in the level, at the top of the report and in
the floor's failure message; it is still measured and still gated, because the
rate is what the appends really achieved, but the report now says plainly that
the number describes a shared database and must not be carried forward.
contaminated is null, not false, when the sampler could not look. "Nobody
checked" is not "nothing was there", and the same reasoning fixes the smaller
version of that bug next door: a level that ended before the sampler completed
a poll used to emit mean_waiters 0.0, max_waiters 0 and
estimated_waiter_seconds 0.0, which reads as "nobody ever waited". Those are
null now, samples stays 0, and the error says the level ended first.
The sampler's shutdown and its interval are fixed together. It tested the stop
flag only at the top of its loop and then blocked in sleep(interval), so every
level burned up to a whole interval of dead time after its appends had
finished -- 65 seconds of wall clock for 0.43 seconds of measurement at a
20-second interval. The flag is now paired with a Notify signalled alongside
it, so the sleep is interruptible and a level ends when the appends end;
notify_one stores the permit, so the signal cannot be lost in the window
between a poll and a sleep. The interval itself is bounded to 1..=1000 ms at
entry, before any connection, with a message saying that an interval coarser
than a second cannot resolve a lock wait that lasts milliseconds.
Two other gaps the review left open. A run that dies connecting, seeding or
appending now writes a report anyway -- identity, configuration, the full error
chain, passed_minimum false and an empty results array -- so the newest file on
disk belongs to this run rather than to some earlier one; a failure to write it
is attached to the original error as context and never replaces it. And the
default report path is per test, target/prism-postgres-throughput-<test>.json,
with a "test" field in the JSON to match, because the two floor tests run in
parallel threads under --include-ignored and the survivor of a shared path was
whichever finished last, unlabelled. QBIT_PRISM_THROUGHPUT_REPORT is still the
explicit override.
Cancellation is documented rather than fixed: SIGINT kills the process before
the schema drop runs, and handling it properly needs a signal handler, which
means a dependency this test may not add. The module docs now say so and give
the query that finds and drops the leftovers, next to the standing requirement
that this run wants a database nothing else is appending to.
Part of #271
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* 3.x.x: make the throughput script point at a report that exists
Two things the script got wrong, both of which send a reader looking for
evidence that is not there.
It printed the report path unconditionally, after the status had been
captured. On the fallback path with PRISM_TEST_DATABASE_URL unset and no local
server binaries, the run dies before the test starts and writes nothing, yet
the script still announced a file. It now prints the path only when the file is
actually on disk, and otherwise says that no report was written and where it
would have been. The exit status is unchanged either way.
It also pointed at the old shared default path. The test writes
target/prism-postgres-throughput-<test name>.json now, one per test, so the
script derives its path from the test it runs rather than from a constant that
has moved.
The header gains the two warnings the test itself carries. ORDER_LOCK is
database-wide, not schema-wide, so this must run against a database nothing
else is appending to; a run that shares one is marked contaminated and its
number describes two workloads. And a cancelled run leaves its
prism_throughput_<uuid> schema behind, so the header gives the query that finds
the leftovers and the statement that drops them -- unnecessary on a disposable
cluster, which is thrown away whole, and that is the reason to prefer one.
Part of #271
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* 3.x.x: print the floor's per-level rates where CI can see them
The floor test prints its rates with println!, and libtest throws away the
captured output of a passing test. CI keeps no report artifact either, so a
green prism-native-postgres run recorded nothing about how far above the floor
it was, and the provisional CI_MIN_SHARES_PER_SEC could not be calibrated from
CI at all.
After the report is written, the test now emits one summary line through
std::io::stderr() directly, which the test harness does not capture: the test
name, build profile, window, shares per level, the rate at each appender
count, the floor, the verdict and the contamination flag. Every CI log then
carries the margin the floor is set from. A failed write is ignored, because
the report file already holds everything the line summarises.
Part of #271
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* 3.x.x: move the throughput floor onto the shared integration gate
#322 replaced every test's private copy of the database guard with one
shared gate, qbit-prism-test-gate, and made the prism-native-postgres job
prove that each gated test executed against test/prism-gated-tests.txt.
The floor still carried its own copy of the old guard, which the new
check_gate_env_reads lint rejects, and it was not on the list, so the proof
step would have failed the moment the test ran.
The floor now asks the gate for its database URL, so it follows the same
decision table as every other gated test and records its decision in the
execution manifest. share_append_throughput_floor joins the list; the
full-size test is not selected in CI and stays off it.
Part of #271
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* 3.x.x: set the CI throughput floor from the CI runner's own numbers
CI_MIN_SHARES_PER_SEC was a provisional 71 shares/s, a quarter of what an
8-vCPU devbox managed, because nothing had measured the runner that enforces
it. Four runs of prism-native-postgres on blacksmith-2vcpu-ubuntu-2404, read
from the summary line the test now writes, put the slowest level of each run
at 1,290, 847, 1,242 and 1,200 shares/s (debug build, the job's PostgreSQL 16
service, a 20k-share window and 2,000 appends per level).
The floor is now 250, about 30% of the slowest level seen. Runs on that
runner differ by about a third, so the margin keeps a noisy run green, while a
3-4x collapse of the append path still fails the build. The report's source
label no longer calls the constant provisional.
Part of #271
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* 3.x.x: run the floor the disposable-cluster path was skipping
Four findings from the review of this PR, all in the paths that are hardest to exercise by hand.
The wrapper script passed cargo_args beginning with 'test' to prism-native-tests.sh, which runs 'cargo test "$@"' itself, so the second word became a test-name filter. With --exact alongside it the disposable-cluster path selected nothing, printed nothing and exited 0: the script that exists to measure the floor measured nothing whenever PRISM_TEST_DATABASE_URL was unset. The array now carries only the arguments, and the direct call adds 'test' itself.
The full-size test is selected by name, so a missing database must fail it rather than skip it; it now takes the gate's required entry point, and the CI-sized test keeps the skipping one. Resetting pg_stat_statements cleared the whole server's counters, including other databases; each level now reports the difference of two reads instead, and says so where the numbers are. The level clock started after the barrier opened, so an appender the runtime resumed first could work outside the measured window and inflate the rate; it starts before.
Part of #271
* 3.x.x: scope the lock statistics to this database and stop announcing stale reports
Two findings from review, both about a number that looks right and is not.
`pg_stat_statements` is server-wide. The level's advisory-lock figures
were the difference of two reads filtered only by query text, so an
advisory lock taken in any other database on the same server landed in
the level's count. A lock in another database cannot contend with this
run -- an advisory lock is scoped to its database -- so the level was
correctly marked uncontaminated while its lock figures described someone
else's work. Filter on `dbid`, and say so in the report's caveat.
Measured on a PostgreSQL 16 with `pg_stat_statements` loaded, over a
level during which another database took 3,000 advisory locks: the old
query attributed 3,000 calls to the level, the filtered one attributes 0.
With the fix and a concurrent foreign load of about 24,000 locks, each
level reports exactly 300 calls for 300 appended shares -- one lock per
share, which is what the append path takes.
The wrapper announced any file at the report path as this run's evidence,
so a run that died during compilation or cluster setup pointed the reader
at a previous run's report while returning a failure. The earlier file is
now moved aside before the run, which makes the report's presence
afterwards proof that this run wrote it, and it is put back if this run
wrote nothing. Deleting it instead would have destroyed the previous
run's data on a run that produced none of its own.
Timestamps were tried first and rejected: a warm build and a small window
can finish inside one filesystem timestamp tick, and a fresh report then
compares as no newer than the moment the run started, which reports a
good run as stale.
Verified end to end through the wrapper against a real PostgreSQL 16, and
for all three announcement cases: a failed run with an earlier report
present, a successful run that writes one, and no report at all.
Part of #271
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* 3.x.x: list the new reject reason in the metric exposition fixture (#348)
`3.x.x` has been red since #344 landed. Two pull requests collided
semantically while each was green on its own:
- #333 (e13051cf) added `ledger-outcome-unknown` to `RejectReason`, so
`/metrics` now emits one more `qbit_prism_rejections_total` series;
- #344 (037b04e4) added `event_exposition_matches_the_pre_allocation_change_byte_for_byte`,
whose fixture was captured at 2914a629, before that reason existed.
Neither branch could see the problem, because it only appears in the
merged tree. The whole difference is one line, so the fixture gains it.
The comment above the assertion is refreshed at the same time: it claimed
provenance the fixture no longer had, and it now says what the pin
implies -- adding a value to a closed label set has to be recorded here.
Verified on this branch: the four metrics_allocation tests pass, and the
failure reproduces on a6fe3fd9 without this change.
Part of #338
* v3.0.0 fix: record cancelled collector pool acquisitions (#345)
* fix: record cancelled collector pool acquisitions
* docs: align collector cancellation timing contract
* docs: qualify cancelled collector waits in alert contracts
* test(metrics): include approved unknown ledger outcome in fixture
PR #333 intentionally added the ledger-outcome-unknown rejection label and a producer contract test. Add exactly that sample to the older exposition fixture from #344, retaining the byte-for-byte equality assertion and leaving runtime behavior unchanged.
(cherry picked from commit 4e5b4c41e980a3ebd956f39d5ed4d4676083d04e)
* build: update rustls to 0.23.45 for RUSTSEC-2026-0285 (#354)
rustls 0.23.44 accepted TLS 1.3 handshake messages sent at the wrong
encryption level when they followed a key-changing message in the same
record, instead of terminating the connection as RFC 8446 section 5.1
requires. The transcript stays authenticated, so a network-position
attacker cannot alter or complete a handshake; the effect is that a peer
can send handshake messages in plaintext that should have been encrypted
without rustls rejecting the connection. It is the same bug as Go's
CVE-2025-61730.
The advisory names 0.23.45 as the fix. This is a lockfile-only change:
`cargo update -p rustls`, no manifest or source change.
The advisory database picked this up today, so the dependency audit now
fails on 3.x.x and on every open pull request.
* 3.x.x: Normalize stored audit bodies so they stop growing with the window (#353)
* 3.x.x: add PrismRewardManifestHeader and a checked reward-manifest rebuild (#267)
A native audit row is about to stop storing `reward_manifest.shares`. The
counted window is a pure fold over `(shares, found_block)`, so the row only
needs the manifest header; `restore_reward_manifest` reruns the fold and
returns its window only after proving the rebuilt header is identical to the
stored one, failing as the existing `reward_manifest` audit mismatch
otherwise. There is no unchecked variant.
`PrismRewardManifestHeader` denies unknown fields, so a legacy manifest that
still carries `shares` can never be read as a header; and the full manifest
still requires `shares`, so an older reader meeting a normalized row fails
explicitly instead of serving an empty window.
Tests prove the rebuilt bundle is byte-identical to the original (both
schemas, credit_policy included), that one changed field in a rebuilt share
moves the canonical sha256, that a tampered header is refused by name, and
that neither shape can be mistaken for the other.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXvnfUC8CvPxskAXLgd1xx
* 3.x.x: store native audit bodies without reward_manifest.shares and verify the range before the settlement lock (#267)
The landed native audit body kept `reward_manifest.shares`, one record per
counted share, so it grew with the payout window: about 235 MB in one JSONB
value at the production window, 88% of PostgreSQL's container limit. The
landing write now removes that copy alongside the top-level `shares`; the
stored body is O(distinct miners and payout accounts). Nothing else is added
or removed, `audit_body_byte_len` stays the full canonical artifact length,
and the GIN-indexed leaf arrays stay in the body.
Reads branch on the stored shape. A body whose `reward_manifest` has no
`shares` key is decoded as `PrismRewardManifestHeader` and its window rebuilt
through `restore_reward_manifest`, which refuses a header the fold does not
reproduce; a body that still has the key (landed before this change) is
served exactly as before. Both arms then run the unchanged canonical digest
check, so a rebuilt body is proven byte-identical, not merely equivalent.
Imported rows never enter this path. The snapshot digest, the rebuild and the
canonical hash run on one blocking thread, and the public API hands that job
the same permit imported decodes hold, so a dropped request cannot free the
limit before the rebuild ends.
The durable-range proof no longer runs inside the settlement transaction.
`verify_durable_range` runs before the lock, page by page (4096 rows, the
window reader's page), decoding and comparing each page on a blocking thread
against the same positions of the candidate window: same predicate, same
ordering, same full equality, same bootstrap exemption, same error text, and
the first mismatched page fails the landing. An already-landed block skips it
and takes the existing-row digest path as before. Under the lock the snapshot
now only counts the same anchored predicate and fails the landing if the count
differs from the window.
The JSONB ceiling gate's landing entry is deleted: that write no longer
crosses the threshold. Gated tests prove the stored shape and identical served
bytes, that a tampered stored header is refused by name, that pre-change
native rows still serve identical bytes, and that imported rows serve from
their canonical bytes even with a poisoned inline body. An ignored harness
measures the stored body, the settlement-lock hold and the landing peak RSS
at a configurable window size.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXvnfUC8CvPxskAXLgd1xx
* 3.x.x: document the measured native audit body before and after normalization (#267)
State the measured stored bytes per block at 20,000 and 100,000 shares before
and after `reward_manifest.shares` left the stored body,…
Merge the released main (b918755, #783) into 3.x.x, as the "Reconcile the development branch" step of docs/releasing.md asks. 3.x.x is still at the v3.1.0 tag (59194d0), so there is no later development to preserve, and the conflicts the squash promotion left resolve to main's files: the tree is main's, byte for byte. Against 3.x.x this brings: - #781's disposition-poll change: miner_submit.rs, its acquire tests, docs/prism-pool-acquire-metrics.md and scripts/prism_pool_acquires.json; - docs/releasing.md, and the README banner that names the version; - the released 3.0.0 notes in place of the pre-release copy, with the four links #782 pointed at the v3.0.0 tag back on the local file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs/releasing.md said neither 3.x.x nor 3.0.x had merged a released main. Say what this step merges into each: the released main into 3.x.x, and main at b9579c6, before 3.1.0's promotion, into 3.0.x, so that the 3.0 patch branch doesn't take the 3.1 dual writer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![qbit-reviewer[bot]](https://avatars.githubusercontent.com/in/5446813?s=60&v=4){kind=small}
This was referenced Oct 27, 2026
kiwidream
added a commit
that referenced
this pull request
Oct 28, 2026
VERSION, the workspace version and both lockfiles' workspace members move to 3.1.1, the same lines #782 bumped. doc/release-notes-3.1.1.md covers every change on 3.x.x since v3.1.0 (#786 to #808); the operator-facing changes: migration 032, the payout-revision listener's connection, the alert rule UIDs and gating; the upgrade from 3.1.0 (migrate while 3.1.0 serves, then replace the binaries one at a time), from 3.0.0 and earlier; the rollback to 3.1.0; and the known issues. The README banner, the development-line note and docs/releasing.md's current-release line name 3.1.1 and the v3.1.1 tag. The migration guide no longer says a 2.x audit-bundle.v2 body answers 500 until 3.x can verify it: since #809 the public route verifies it with the parser and canonical-digest check import-audits uses, and serves it. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merges the released
main(b9187551, #783) into3.x.x: the "Reconcile the development branch" step ofdocs/releasing.md.3.x.xis still at thev3.1.0tag (59194d03), so there is no later development to preserve.mainand3.x.xshare no history sincev1.1.0, so the merge conflicts in every file the squash promotion changed; each resolves tomain's file, and the merge commit's tree ismain's, byte for byte. Squash-merge it, as the repository requires. It adds no tag and changes no release.What
3.x.xgainsgit diff origin/3.x.x HEAD: 11 files, the ones #783 kept on top of thev3.1.0tree:miner_submit.rsandminer_submit_acquire_tests.rs, with the matchingdocs/prism-pool-acquire-metrics.mdandscripts/prism_pool_acquires.json. Each poll under the block-proof acknowledgement bound runs in its own task, so the bound no longer drops a connection partway through a checkout.docs/releasing.md, and the README banner that names the version ("Version 3.1.0: the Rust PRISM server …") rather than the branch, as the guide's reconcile step asks. It drops "3.x.xis the production line" and "not from the branch head"; the development-line note keeps that warning.blob/v3.0.0/point at the local file again, in the 3.1.0 notes, the development-line note,docs/prism-rust-migration.mdanddocs/prism-ledger-ops.md.One change beyond
maindocs/releasing.md's reconcile paragraph said neither3.x.xnor3.0.xhad merged a releasedmain. It now says what this step merges into each: the releasedmaininto3.x.x, andmainatb9579c67into3.0.x(#787). That commit is 3.0.0's promotion (#778) plus #781, before 3.1.0's, so the 3.0 patch branch doesn't take the 3.1 dual writer. Both PRs carry the same guide.Checked locally
main's (b9187551, the tree of v3.1.0 chore: consolidate release for main #783's head, where CI passed). The head differs frommainonly in that paragraph.check_version_skew.py(VERSION 3.1.0 matches every workspace crate),check_prism_settings.py,check_toolchain_pin.pyandcheck_prism_pool_acquires.pypass.cargo metadata --lockedpasses for the root and fuzz workspaces, andgit diff --checkis clean./code-review mediumof both sync branches. Its findings on main: update tokio for RUSTSEC-2026-0386, and keep the block-proof acknowledgement bound from dropping a disposition poll's connection #781's code are aboutmain's merged code, which this PR carries unchanged, so none is fixed here. It also noted the README banner point above, and that a squash merge records no merge parent, so the next sync or promotion conflicts on the same files again, as this one did.After merge
main's copy ofdocs/releasing.mdstill calls this step outstanding, until the next promotion from3.x.xbrings the new paragraph.🤖 Generated with Claude Code