Repository navigation
test, ci: foundations for hybrid v2 transport (Python ML-KEM, qbit releases, v1.0.0 interop, PQ CI, canary tools) - #224
Merged
Conversation
7 tasks done
Member
Author
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3dd57fcf73
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Member
Author
|
@codex review |
djh58
force-pushed
the
djh58/pq184-2-foundations
branch
from
October 5, 2026 19:54
3dd57fc to
5690a4f
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5690a4fd7a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Member
Author
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4e9eb66eb2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Member
Author
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 22fbdc7020
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
4 tasks done
Member
Author
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a423bbff2a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Add test_framework/crypto/mlkem.py, a pure-Python FIPS 203 ML-KEM-1024 written from the standard and independent of the C library: KeyGen from a 64-byte d || z seed, Encaps from a 32-byte m, Decaps with implicit rejection, and the section 7.2 (length and modulus) and 7.3 (length and hash) input checks. Hashing uses hashlib's SHA3 and SHAKE only. TestMLKEM1024, registered in feature_framework_unit_tests.py, checks: - NIST ACVP-Server ML-KEM-1024 keyGen and encapsulation vectors (exact ek, dk, ct and shared secret); - ACVP decapsulation vectors, including modified ciphertexts whose expected secret is the implicit-rejection value J(z || ct); - ACVP encapsulationKeyCheck vectors: valid keys pass the section 7.2 check and encapsulate, the others are rejected; - C2SP CCTV modulus vectors (q and 4095 at the first and last coefficient of every polynomial, plus a sample of the full run), each rejected, and accepted once the bad coefficient is set to 3328; - round trips, implicit rejection on corrupted ciphertexts, and input size and hash checks. mlkem1024_test_vectors.txt holds the subset with its sources and pinned commits in a header comment. The new tests take about 0.07 s; the whole feature_framework_unit_tests.py run about 4 s. Part of #184 (part 2, #186, work item B2). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Let functional tests download and run qbit release binaries the way they run Bitcoin Core releases, starting with v1.0.0. get_previous_releases.py resolves each tag through resolve_download(). qbit tags come from explicit descriptors: the GitHub release URL, the archive per host (the "-unsigned" tarballs on macOS, whose -unsigned.zip holds only the GUI app, and the "-unsigned" zip on win64), the archive root and each archive's SHA256 copied from the release's SHA256SUMS. Unsigned arm64 macOS binaries are self-signed as old Core ones are. A host without a qbit archive fails with a message instead of a guessed download. Bitcoin Core URLs, archives, hashes and messages are unchanged, and qbit tags join the default download list, so the previous-releases and Windows jobs fetch v1.0.0. The test framework maps client versions through explicit ReleaseInfo metadata (test_node.py): 10000 is qbit v1.0.0, based on Core v30.2 (core_compat_version 300200), with qbitd/qbit-cli binary names. A version from 140000 (Core v0.14) up is the Core release it names, as before; any lower version not in QBIT_RELEASES is an unknown qbit release and raises, instead of being read as a Core version. version_is_at_least() compares core_compat_version, so v1.0.0 nodes get -v2transport and the logging options, and qbit_version_is_at_least() gates qbit-only options: -p2mronly=0 now reaches qbit releases that have it, and supports_v2_pq marks which nodes may get -v2pqtransport. tool_qbit_release_resolution.py (qbit_release_resolution) checks v1.0.0 on every published host triple including win64 against a verbatim copy of SHA256SUMS, the Core mappings, the rejection of unknown qbit versions, and the options a v1.0.0, Core and current node get. It needs no downloads, so it runs in every functional job, Linux and Windows. Part of #184 (part 2, #186, work item B1). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add p2p_v2_previous_release.py, which pins how the qbit v1.0.0 release binary treats non-empty BIP324 version-packet contents, where the hybrid post-quantum handshake (#184) will carry its ML-KEM-1024 key and ciphertext. It drives a BIP324 peer on a plain socket, so it controls every byte and pause, against v1.0.0 in both roles: - test_version_contents: with 0, 1,572 and 65,000 bytes of contents, and with record-shaped contents (CompactSize(len) || header || payload): one 0xF0 record exactly as the hybrid offer and accept carry it (fd 21 06 f0 || 1,568 bytes), a 0xF0 record followed by an unknown 0xF1 record, and a single 65,000-byte record, each sent after three decoy packets, both immediately and after a 10 s pause under the default 60 s -peertimeout, v1.0.0 completes the handshake, exchanges version/verack and ping/pong, sends empty contents itself, and reports the plain BIP324 session id. As initiator it sends its version packet and VERSION message before receiving ours. - test_limits: v1.0.0 keeps a connection whose packet length is exactly MAX_CONTENTS_LEN (4,000,013), drops one at 4,000,014 bytes at once ("packet too large"), and drops a handshake stalled after the keys at -peertimeout ("V2 handshake timeout"), within a bounded time. The test skips without previous releases and runs in the previous-releases job, which downloads v1.0.0 since the previous commit. It takes about 24 s. Part of #184 (part 2, #186, work item B3). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add .github/workflows/ci-pq.yml, one reusable workflow for the post-quantum transport jobs, and make ci-nightly-heavy.yml callable and job-selectable, so main's callers (C4), on a schedule and on demand, can run either against a recorded commit. GitHub dispatches only workflows whose file is on the default branch, so on-demand PQ runs go through C4's caller on main (workflow_dispatch with source_ref and jobs, calling ci-pq.yml@1.x.x), available after this part merges; ci-pq.yml keeps its own workflow_dispatch trigger for when it is on main. ci-nightly-heavy.yml is already on main, so it can also be dispatched on a branch directly. ci-pq.yml runs on workflow_call and workflow_dispatch with two inputs: - source_ref: resolved once by the resolve-source job to a full commit (empty: the triggering commit; a branch or tag, looked up on the remote with annotated tags peeled; or a full hash). A missing, ambiguous or malformed ref fails the run. Every job checks out that commit, verifies HEAD against it, and the summaries record it. - jobs: a comma-separated allowlist; empty runs all jobs, and an unknown, repeated or empty name fails the run. The resolve-source job emits the matrix, since a job-level if cannot see matrix values. Its first job, aarch64-unit, runs on GitHub's ubuntu-24.04-arm runner with the public-runner environment (docker.io/library images, CI_ENFORCE_INTERNAL_REGISTRY=0, public DNS, no tailnet search domains), builds only test_bitcoin and runs bip324_tests and net_tests (ci/test/00_setup_env_native_aarch64_pq.sh). Its summary shows the source commit, labels this the pre-library phase, and lists both suites with their results. ci/test/03_test_script.sh: - CI_BUILD_TARGET (default all) selects the target of the build and of its verbose retry; GOAL's install/deploy targets apply to full builds; - CTEST_REGEX selects tests by name and is rejected together with CTEST_INCLUDE_RANGE; - ctest runs with --no-tests=error, lists the selected tests first and writes JUnit results; when CTEST_EXPECTED_SUITES is set, ci/checks/ctest_evidence.py fails the job unless every expected suite was selected, executed and passed (ctest itself passes a skipped suite, for example one compiled out). ci-nightly-heavy.yml gains workflow_call inputs mirroring workflow_dispatch (choices become strings, which run-scanners.py validates), an optional LIBBITCOINPQC_READ_TOKEN secret, and a jobs input over arm32-unit-1, arm32-unit-2, previous-releases, fuzz and scanner. The matrix entries are unchanged and now come from resolve-source. A called run gets its caller's github context, so github.event_name, github.ref and github.sha are main's and cannot choose what a called run validates. Only workflow_call takes a source_ref input (default 1.x.x, must not be empty), and a non-empty source_ref marks a run as called: - a called run resolves source_ref once, with the same function as ci-pq.yml (which the contract test keeps identical), never reads github.ref or github.sha, and records source_mode maintained when source_ref names 1.x.x and requested otherwise; it honors run_test_matrix, run_scanners, the scanner settings and jobs whatever its caller's event; - a direct scheduled run still resolves 1.x.x and follows the CI_NIGHTLY_HEAVY_RUN_* variables and scheduled defaults, and a direct dispatch keeps its selected ref and event commit; - a first step fails the run when source_ref disagrees with GitHub's job.workflow_ref, for example a caller that passes an empty source_ref, and skips that check, saying so in its summary, where job.workflow_ref is unavailable; - a called run's concurrency group names its caller's event and source_ref, so callers for different sources, and an on-demand call beside the scheduled one, never cancel each other; - the summaries record the source_ref input and source_mode. ci/checks/test_ci_pq_contract.py executes the workflows' inline steps and the 03_test_script.sh blocks against git fixtures and stub cmake/ctest: job selection, rejection of unknown jobs, an immutable source commit, suites that really ran, and called and direct nightly runs with their concurrency groups. The shared expression evaluator follows GitHub's loose equality and supports format() and toJSON(). test_scheduled_validation_contract.py reads the nightly matrix from its new catalog. Both now run in required CI: the Required Merge Gate's classify-changes job (.github/workflows/required-merge-gate.yml) gains one step that runs them, beside the classifier contract. Part of #184 (part 2, #186, work item C1). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Scripted evidence for the mainnet canary of the hybrid post-quantum v2 transport (#184, R5), ready before the 14-day R0 baseline starts on v1.0.0 nodes. Everything lives in contrib/pq-canary/. pq-canary-sample.sh runs every 5 minutes from cron or a systemd timer on each canary node and the control node. Through qbit-cli it appends one row to samples.csv: time, host label, uptime, version; enabled, instance_id, backends, since, every per-direction handshake counter, load_shedding.active and each ring's last_sequence and dropped from getpqtransportinfo; connections_in and connections_pq; and, on the pool node (--pinned), the pinned archive node's getpeerinfo connection_type, transport_pq and transport_pq_status. sample_ok is 1 only when uptime, getnetworkinfo and, on the pool node, getpeerinfo answered. A failed call or a missing field is NA, never 0, and a real 0 stays 0, so a v1.0.0 node writes NA for the new fields with sample_ok=1. The raw recent_failures and fallback_set go to failures.jsonl, which keeps its addresses between the team's canary nodes. --pinned and --host are validated. pq_canary.py holds the format shared with the evaluator. pq-canary-eval.py prints each figure with its sample count, coverage and pass, fail or unknown: the pinned link manual and hybrid in at least 99% of samples excluding restarts (with the disconnectnode remedy when an automatic connection holds the address); no malformed_record, first_packet_failed or fallback entry between known-good endpoints, on every monitored canary node (one node's failures file cannot pass it); closed_after_switch entries between them, listed for triage; and the archive node's connections_in within +-20% of the control's, relative to their baseline ratio, per UTC day, excluding 48 hours after a restart. A jump in boot time is a restart gap. A ring that wrapped stays judged while the entries read continue from the last sequence processed; only a wrap that evicted unread entries, a changed instance_id or an inconsistent ring makes that ring's interval unknown. A figure, node or day below --min-coverage (default 90% of the expected 5-minute samples) is unknown, not judged, and every figure reports its coverage. Missing evidence is unknown, never clean. The report names known-good peers by label and never prints a peer address, so it can be posted. Unknown, malformed or missing arguments are rejected with a message and exit status 2. Tests: contrib/pq-canary/test/test_pq_canary.py (stdlib only, fixtures in the #184 getpqtransportinfo schema) covers, among others, a monitored outbound failure surviving 300 unrelated inbound failures, unrelated noise, a restart gap, an RPC outage, a wrapped ring with and without lost entries, per-node evidence, coverage thresholds, address redaction and argument validation. The Required Merge Gate's PQ contract step (.github/workflows/required-merge-gate.yml) now runs it as well, and test/functional/tool_pq_canary.py runs the real sampler against the node under test and, when previous releases are present, a v1.0.0 node. The README covers install, the cron lines and how to read the output. Part of #184 (part 2, #186, work item C5). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The evaluator read a missing or malformed fallback_set as an empty list, so a sample without fallback evidence could pass Figure 2. Record it as an unknown interval for that host instead, as for malformed rings, and iterate fallback_set only when it is a list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A missing or malformed instance_id was compared like a real process identity, so a restart between samples whose new ring did not go backwards went unnoticed and lost failures could pass Figure 2. Mark both rings unknown from the previous sample through the next record with a valid 64-hex-digit instance_id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A ring entry whose endpoint was missing or malformed got no known-good label and was skipped like a valid unrelated peer, so a fallback, malformed_record or first_packet_failed entry without endpoint telemetry could still let the failure figure pass. Check that an endpoint has the getpqtransportinfo Endpoint shape: a kind and network that agree, an address of that network without a port, and a port from 0 to 65535. An entry that counts towards figure 2 or 3 and has neither a known-good nor a well-formed endpoint now makes that figure unknown. A malformed endpoint that still names a known-good peer remains that peer's failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Giving the same samples.csv for --archive and --control, or two files with the same host label, made figure 4 compare a node with itself: a perfect ratio that could pass at full coverage. The same applied to --pool and --archive, which also set the nodes monitored for figure 2. Require --pool, --archive and --control to be three different nodes. A file given for two roles, two roles sharing a host label or an instance_id, and a file whose rows carry no host label or more than one are bad input: the evaluator exits with status 2 and names the roles. The node test gave one node's samples as both pool and archive; give it the pool samples alone and check the rejection instead. The unit tests' simulated nodes now get distinct instance_ids, as real nodes do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On powerpc64le, a host set_host() recognizes, running get_previous_releases.py without tags now always included v1.0.0, which has no powerpc64le archive, so the default download stopped with "No qbit v1.0.0 archive". Without tags, download every Core tag as before but only the qbit releases that have an archive for the host, and say which were skipped. A tag given explicitly still fails when it has no archive for the host. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Audit every failures.jsonl field the evaluator reads, so that a missing, malformed or contradictory value makes its stretch unknown rather than clean, and never crashes the evaluator into a "fail" exit status: - recent_failures that is not an object, and ring counters or sequences that are booleans, negative or zero, are a malformed ring (a non-object recent_failures raised AttributeError); - an entry outcome outside the getpqtransportinfo vocabulary may be a monitored or a triage one, so both figures are unknown (it was skipped as clean; an object outcome raised TypeError); - an entry time that is missing, malformed, before its process started or more than CLOCK_SLACK after the sample that read it is unknown (it was read as 1970 and skipped, or raised TypeError); - a fallback-set entry that is not an object or lacks a well-formed endpoint is unknown, once per distinct entry (it was skipped); - since that is missing, malformed, after the sample or changed within one instance_id makes the record's process identity unusable, like a missing instance_id (it fell back to the sample time); - a failures.jsonl line needs a time in Unix seconds up to year 9999 and a host label, else it is bad input (a boolean time was accepted, and a huge one crashed the report); - a known-good failure whose reason, cause, connection type, peer id, streak or times are malformed is still a failure, and those fields print as "malformed": a malformed field could put an address in the report, and malformed times or unhashable values raised. Labels are now matched in full, so a host label with a trailing newline is rejected. The unit tests' simulated nodes now report only failures recorded by the time of each sample, and boot before their first sample. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Audit every samples.csv field the evaluator reads, so that a missing, malformed or contradictory value is unknown rather than a pass, a miss or a crash: - time, uptime and connections_in must be plain non-negative decimal integers. int() also took "-30", " 30", "3_0" and non-ASCII digits, so a negative uptime looked like a fresh restart and excluded samples from figures 1 and 4; a time such as "soon" raised ValueError. A row without a usable time is now a missing sample. --canary-start and --end must lie between 1970 and 9999, and so must a sample time, which the report prints. - A row with more or fewer fields than the header, such as a line cut short by a crash and continued by the next sample, has values in the wrong columns, so it is not a sample. - A connected pinned peer needs a connection type, transport_pq 0 or 1 and a transport_pq_status from the getpeerinfo vocabulary, with transport_pq 1 exactly when the status is hybrid. A malformed or contradictory value is unknown: it counted as a miss, and transport_pq=1 with status "pending" counted as hybrid. - Figure 4 pairs only samples with sample_ok=1. - An uptime whose boot time contradicts the one already seen for the same instance_id is unknown: it was read as a restart and excluded samples. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The sampler wrote any integer, string or boolean it found: true became a connections_in or counter of 1, a negative uptime or since was kept, and an instance_id of any shape was recorded. Write each column only when the RPC value has its contract type (counts are non-negative integers, flags booleans, backends and statuses strings, instance_id 64 lowercase hex digits); anything else is NA, which the evaluator reads as unknown. Reject --host NA, which would read back as a missing host label. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nknown Two of the new unknown rules reached further than the evidence: - An uptime was checked against the first boot time seen for its instance_id. A restart between the sampler's uptime and getpqtransportinfo calls pairs the old uptime with the new instance_id once, which made every later uptime of that process unknown. Check each uptime against the median boot time of its instance_id instead, so one racy or malformed value is the only unknown. - An entry with an implausible time made figure 2 or 3 unknown even when the sample that first read it was taken long before the window. Such an entry was recorded before that sample, so it cannot be in the window. Also rename the samples() helper to sample_rows(), which evaluate_failures shadowed with its sample count. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tool_qbit_release_resolution's default-tags case set HOST to make get_previous_releases.py act as a powerpc64le host. set_host() ignores HOST on Windows and answers win64, so on the Windows CI job the case downloaded the v1.0.0 win64 archive and failed. Stand in for set_host() so the case checks main()'s use of the host on any platform. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
djh58
force-pushed
the
djh58/pq184-2-foundations
branch
from
October 6, 2026 18:08
6317035 to
46cad5d
Compare
p2p_v2_previous_release.py only checked that previous releases were enabled. A releases directory that holds other releases, such as Bitcoin Core's, but not qbit v1.0.0 enables them, so the test went on and setup_nodes() aborted on the missing qbitd and qbit-cli. Skip unless both v1.0.0 binaries are present, as tool_pq_canary.py leaves out its v1.0.0 sample. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Figure 1 counted every samples.csv row in the window, so rows repeated for one sampling slot (overlapping files, a manual run beside cron) added to its total: 95 of 100 expected samples is unknown, but the same rows recorded six times read as 99.1% hybrid and passed. Expected samples are now --interval apart from --canary-start, a sample stands for the nearest one, and every figure counts an expected sample once. Several samples for one count as the least favorable of them, so an extra sample can never improve the evidence: for the pinned link a miss, then unknown, then excluded, then hybrid; for inbound connections unknown, then excluded, then a measurement, where measurements that disagree are unknown (figure 4 kept whichever row came last). Figure 2 counted a record nearest to an expected sample just outside the window as covering one inside it; it now uses the same expected samples. A row repeated exactly is one sample, so repeated rows cannot outvote the median boot time that decides whether an uptime is a restart: one racy uptime copied a hundred times made two misses look like a restart and excluded them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
Author
|
@codex review |
1 of 9 tasks
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a8aea727cf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
get_previous_releases.py registers the arm-linux-gnueabihf and powerpc64-linux-gnu v1.0.0 archives, but set_host() had no pattern for either host, so on 32-bit ARM or big-endian POWER it exited "Not sure which binary to download" before resolve_download() saw them. Map both hosts. Each pattern for a qbit host now accepts the depends triplet given in HOST, which the script's help documents, as well as config.guess output. This also fixes the upstream-derived aarch64, riscv64 and arm64 macOS patterns, which only matched config.guess output: HOST=aarch64-linux-gnu, riscv64-linux-gnu or arm64-apple-darwin exited the same way. The ARM patterns accept only little-endian ARMv7 or ARMv8 hosts with hard float, because the v1.0.0 ARM binaries are ARMv7 VFPv3 armhf builds; an ARMv6 host still gets "Not sure which binary". powerpc64le (Core archives only) and x86_64-w64-mingw32 (win64 comes from platform.system(), not HOST) are left as they were. tool_qbit_release_resolution now runs set_host() with a mocked platform, HOST and config.guess output for every published host, and checks that each answer resolves to that host's v1.0.0 archive. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tool_pq_canary enabled its v1.0.0 node whenever releases/v1.0.0/bin existed. get_previous_releases.py creates the release directory before extracting into it and keeps it after a failed extraction, so a stale or partial download could leave bin/ without qbitd or qbit-cli, and add_nodes() then aborted the whole test. Check both resolved executables, as p2p_v2_previous_release does, and otherwise skip only the optional v1.0.0 sample. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
Author
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b53d6e1366
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Figure 4 passes a day whose archive/control connections_in ratio is within 20% of the baseline ratio, inclusive. The evaluator computed the relative ratio in floats with two divisions, so a day exactly on the bound could round past it: 24/10 against a baseline of 3 came out as 0.7999999999999999, and 108/10 against 9 as 1.2000000000000002, and both failed the figure and the release gate. Compute the ratios as exact fractions, and read --baseline-ratio exactly too, since the float nearest 1.2 is below it. float() still screens the argument first, so nan, infinity and exponents outside its range are rejected as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
Author
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1c058363a4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The failure figures look endpoints up in the known-good list by string. The node reports an IPv6 address compressed, so a list entry spelled out, such as [2001:0db8:0:0:0:0:0:5]:8333, never matched 2001:db8::5: its first_packet_failed entries were counted as another endpoint's, and Figure 2 passed at full coverage despite a failure between known-good peers. Give every IP address one spelling on both sides, the list and the node's endpoints, address-only entries included: compressed, lower case, and an IPv4-mapped IPv6 address as IPv4, which is how the node reports it. Two spellings of one entry are now "listed twice". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
Author
|
@codex review |
|
Codex Review: Didn't find any major issues. 👍 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
23 tasks
djh58
added a commit
that referenced
this pull request
Oct 26, 2026
* ci: run pull request and push validation for feature/pq-transport
feature/pq-transport collects the #184 hybrid post-quantum transport work
before it goes to 1.x.x. Add it to the branch lists of Full Validation,
Core Checks and the Required Merge Gate, so pull requests into it and
merges to it get the same validation as 1.x.x. Remove the entry when the
branch merges into 1.x.x.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ci: build pull request RPC docs from the event commit (#228)
The pull request build of the development RPC docs checked out
refs/pull/<n>/merge and then required it to be the event's merge commit.
That ref moves whenever the base branch or the pull request's stack
changes, so a run that overlapped such a change checked out a different
merge commit and failed "Validate version source checkout".
Check out the event's merge commit by hash instead, as the release
entries already do, and drop the now unused --development-checkout-ref
plumbing.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* i2p: request post-quantum leasesets (6,4) and retry once with 4,0 on explicit rejection (#223)
* net: support bounded interruptible SAM session creation
Proxy::Connect() waits up to -timeout for a connection and does not
watch any interrupt, so a caller that has to finish several dependent
steps by one deadline, and stop promptly on shutdown, cannot use it.
Add a Proxy::Connect(deadline, interrupt) overload for that. It waits
for the connection in slices of at most MAX_CONNECT_POLL_INTERVAL
(100 ms), rechecking the interrupt and the deadline between them, and
creates no socket if either is already over. It does not use
nConnectTimeout; callers cap the deadline as they need. Proxy::Connect()
and ConnectDirectly() behave as before.
The new netbase tests use a socket whose connect stays in progress to
check that the overload connects, gives up at the deadline rather than
at -timeout, notices an interrupt within one slice, never waits longer
than a slice, and that the existing overload still waits once for
-timeout. TCP and Unix socket proxies are both covered.
Part of #185.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* i2p: prefer hybrid leasesets and retry explicit router rejection once
Backport Bitcoin Core #35696: ask the SAM router for the post-quantum
hybrid leaseset encryption type, i2cp.leaseSetEncType=6,4
(MLKEM768-X25519, then ECIES-X25519), instead of 4,0, for persistent
and transient sessions. Java I2P 2.10.0 or newer and i2pd 2.58.0 or
newer built with OpenSSL 3.5 or newer use it; other i2pd versions
ignore it.
Java I2P older than 2.10.0 fails the session instead. qbit-only, as
upstream has no retry: on an explicit rejection of 6,4, a well-formed
SESSION STATUS whose RESULT is present, not OK and not one of
DUPLICATED_ID, DUPLICATED_DEST, INVALID_ID or INVALID_KEY, close that
control socket and retry once on a fresh connection with a fresh HELLO
and session id, keeping the persistent destination, with 4,0. A SAM
reply with RESULT=TIMEOUT is such a rejection and is retried. No reply
(a timeout waiting for one, EOF, a socket error), a malformed reply, a
missing RESULT or a lost success reply is never retried. The original
rejection is logged at debug level with RESULT and a sanitized MESSAGE
capped at 200 characters. m_session_id and m_control_sock are set only
on success.
Only a rejection whose MESSAGE names the encryption type is evidence
that the router lacks 6,4. Java I2P 2.9.0 replies "... Failed to build
tunnels - ...: Unsupported crypto type: 6". The MESSAGE, parsed like
the logged one but before it is sanitized or capped, names the type if
it contains "crypto type", "encryption type", "enc type" or "enctype",
ASCII case-insensitively.
A process-lifetime, mutex-protected record per router (its TCP
endpoint or Unix socket path), shared by all sessions, counts the
sessions in a row in which the router rejected 6,4 naming the
encryption type and accepted 4,0. After two, later sessions through it
start with 4,0 and never retry, until qbit restarts. A session created
with 6,4 starts the count anew, so one transient rejection does not
turn 6,4 off, and a failed retry counts for nothing. The record keeps
at most 64 routers and forgets the least recently updated one. At the
first such session through a router, this is logged once at warning
level, with RESULT shown only if it is one of the SAM protocol's
RESULT values, otherwise "other":
I2P: the router rejected leaseSetEncType=6,4 (RESULT=I2P_ERROR) and
accepted 4,0, so this session has no post-quantum leaseset. Java I2P
older than 2.10.0 does this; upgrade the router to get it.
An info line says when later sessions through the router start using
4,0 until qbit restarts.
A rejection that does not name the encryption type, such as a passing
failure to build tunnels, RESULT=TIMEOUT or CANT_REACH_PEER, is
retried the same way but is no evidence either way: it neither counts
nor starts the count anew, and does not warn. One info line says that
the session uses 4,0 because of that rejection, with RESULT shown as
in the warning and the sanitized MESSAGE, and that the next session
asks for 6,4 again.
Session creation now has one deadline, SAM_CREATE_TIMEOUT (180 s), over
connecting, HELLO, the optional DEST GENERATE and both SESSION CREATE
attempts, through new Hello() and SendRequestAndGetReply() overloads
that take it. Each connect uses Proxy::Connect(deadline, interrupt),
capped by -timeout as before, so shutdown interrupts it promptly. Other
SAM requests behave as before. Session constructors take an optional
creation timeout for tests.
An unexpected reply to SESSION CREATE is now reported with the
sanitized request, "SESSION CREATE ...", instead of the raw one, which
carried the private key.
The new i2p tests use a recording mock SAM router to cover creation
with 6,4, the retry, the two-sessions-in-a-row rule and its reset,
rejections that name the encryption type (with the reply of Java I2P
2.9.0) and ones that do not, which are retried but neutral, the phrase
matching, the bounded record, sessions through a remembered router
including concurrent ones, rejections that are not retried, a failing
retry, transport failures and lost replies, interruption during
reconnect, HELLO and CREATE, an exhausted deadline across steps, during
DEST GENERATE and while connecting, the -timeout cap on connecting, and
that the persistent private key is never logged. Deadline checks use
the timeouts the mock router saw rather than wall-clock margins. Tests
and the i2p fuzz target forget the router record between runs. The
existing tests are unchanged.
Part of #185.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* doc: document I2P hybrid router requirements and compatibility retry
State in i2p.md which routers use the post-quantum leaseset type that
qbit now asks for, that Java I2P older than 2.10.0 rejects it naming
the encryption type, and how qbit retries once with 4,0 and warns at
the first such session through a router. After two such sessions in a
row, qbit uses 4,0 with that router until qbit restarts; a session that
gets the post-quantum type starts the count anew. Other rejections, a
SAM RESULT=TIMEOUT reply among them, are retried but neither count nor
warn. Tell users to restart qbit after upgrading the router. Note that
the retry is qbit-only, that no reply from the router is not retried,
and the single 3-minute limit on creating a session.
Add a release-note fragment, to be renamed to the pull request number.
Part of #185.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* bench: bound the Cadence benchmark chain without -asymptote (#229)
StarvationDistance() cast nanobench's complexityN() to size_t. It is -1
unless -asymptote sets it, so a plain bench_qbit run (including ctest's
bench_sanity_check) built a header chain of about 2^64 blocks and grew
until the host ran out of memory: 27 GiB before it was killed.
Use complexityN() only when -asymptote sets it to at least 1, and
default to 10,000 legacy blocks since the last AuxPoW block otherwise.
Both Cadence benchmarks now run in about 17 MB.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* multi_index: fix compilation failure with boost >= 1.91 (#237)
This effectively reverts a3cb309e7c31853f272bffaa65fb6ab0a7cc4083 from PR #30194.
That PR reduced the multi_index type signatures as recommended upstream, but
this is no longer supported as of boost 1.91 because it is no longer necessary.
1.91 drops support for the pre-c++11 work-arounds that bloated the type
signatures to begin with.
The upstream `BOOST_MULTI_INDEX_ENABLE_MPL_SUPPORT` define is meant to provide
compatibility with removed features, but it does not work for this case. Using
`indexed_by` directly when defining the `multi_index` (as opposed to inheriting
from it) works with all versions, and avoids the use of the back-compat define.
This is a slight regression when building against boost < 1.91 because the
bloated type signatures are reintroduced in that case, but it's not significant
enough to go to the trouble of introducing version detection and ifdefs.
Github-Pull: #35175
Rebased-From: 0bc9d354dfd8074d1c36a891a69b6585a8775c65
(cherry picked from commit 9632450ff65c1fee29c2564a6ad64dfaad4d663c)
Backport of bitcoin/bitcoin#35175.
Co-authored-by: Cory Fields <cory-nospam-@coryfields.com>
* ci: wait for a newer run instead of failing on an earlier one's result (#231)
* ci: wait for a newer run instead of failing on an earlier one's result
The Required Merge Gate reads the newest check run of each required name
on the commit. When CI runs again on the same commit (a re-run, a
reopened pull request, a label), the gate starts within seconds, before
the new workflows have created their own gate checks. It then found the
previous run's result and, if that run had been cancelled or failed,
failed at once, although the new run was still on its way. PR #229's
gate failed this way three seconds after a reopen.
Treat a check that concluded other than success before this gate run
started as pending: it belongs to an earlier run, and a newer one may
not have reported yet. The gate keeps polling and fails, with the reason
named, only if no newer result arrives before its deadline. A failure
after the gate run started still fails at once, and an earlier success
still counts, as a label starts no new Core Checks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ci: date the merge gate from its workflow run, not its step
The gate step starts only after the jobs it needs, so a check from the
current run can fail before the step's own start time and be taken for
an earlier run's result, making the gate wait out its deadline instead
of failing at once. Read this run attempt's run_started_at from the API
instead, and only when checks are polled, so profiles that poll nothing
still make no API calls. If it can't be read, fall back to the step's
start with a warning.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ci: never take a cancelled check as the merge gate's verdict
A newer push, reopen or label cancels the running Full Validation
(cancel-in-progress), and that cancellation can complete after this
gate run started, so the run-start cutoff alone took it for a current
failure. A cancelled check means superseded or never run, so treat it
as pending and wait for the run that replaces it; the gate still fails
at its deadline if none arrives.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ci: let the merge gate read its run attempt's start time
Getting a workflow run attempt needs Actions: read, and the workflow
grants only contents and checks, so the start-time lookup always fell
back to the step's own start. Grant actions: read, and pin it in the
poller contract test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ci: start Core Checks on every event that starts the merge gate
The gate now waits for a newer Core Checks Gate instead of trusting an
earlier failure, but Core Checks ran only on opened, synchronize and
reopened, so a label, unlabel or ready-for-review event left the gate
waiting out its deadline for a run that could not start. Give Core
Checks the same pull_request activity types as the gate and Full
Validation, and pin that in the poller contract test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ci: tell the merge gate's checks apart by the event that started them
The gate waited on a check that failed before its own run started,
taking it for an earlier run's result. A time cutoff cannot tell the
two apart: Core Checks, Full Validation and the gate are separate
workflows, so this event's check can fail before the gate's run starts,
an earlier event's can fail or be cancelled after it, and re-running
the gate alone moves its start but starts nothing else.
Date each failed check by its event instead. Every workflow run an
event starts is created at that event, and a re-run keeps the run, its
check suite and its created_at. The gate maps a failed check's check
suite to its workflow run (GET /actions/runs?check_suite_id=) and
compares that run's created_at with its own run's (GET
/actions/runs/{run_id}), allowing 60 seconds between one event's runs.
- An earlier event's failure is waited on until a newer run reports.
- A failure from this event or a later one fails at once, including
after a gate-only re-run.
- A success counts whichever event produced it, and a cancellation is
never a verdict; neither needs a lookup.
- If either run can't be read after the usual retries, the gate fails
closed under its own error title.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ci: let the newest event's check decide, not the newest-started one
The gate judged only the newest check run of each required name by
started_at. A manual re-run of an earlier event's workflow starts later
than this event's check, so it hid this event's result: a failed old
re-run was taken as an earlier event's failure and waited on, poll after
poll, until the deadline, even with this event's success already on the
commit.
Read every check run of the name instead. When they come from more than
one workflow run and not all of them succeeded, map each check suite to
its workflow run (cached per suite) and let the newest workflow run's
check decide, by created_at, then run_number for runs created in the
same second, then started_at for its attempts.
- An earlier event's re-run never outranks this or a later event's
result, whether it failed, passed, was cancelled or is still running.
- Attempts within one workflow run still rank by started_at alone, and
a check whose runs all succeeded passes, both without a lookup.
- If the runs to rank can't be read, or a check run names no check
suite, the gate fails closed under its "could not correlate" title.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ci: tell close events apart by a newer run of the check's workflow
The gate took any failure whose workflow run was created within 60 s of
its own as this event's. Two events that close together (a label, then
an unlabel or a reopen) made the second gate fail at once on the first
event's result. That is the usual case, not an edge one: the newer event
cancels the earlier event's Core Checks and Full Validation, and their
if: always() gate jobs then conclude failure (7 of 8 cancelled Core
Checks runs sampled here did), after the newer event has started.
GitHub's REST API names no event on a workflow run, check suite or
check run, so there is no marker to compare. It does give each
workflow's runs a run_number that grows with every new run and is kept
by re-runs. Before failing at once, the gate now lists the check's
workflow's runs on the commit (GET /actions/workflows/{id}/runs
?head_sha=): if one has a higher run_number, an event since has started
the run that replaces this one, and the failure is waited on. That
tells events apart at any spacing, even within the same second.
Tighten the allowance from 60 s to 10 s. Across 325 pull request events
on this repository (June to October 2026), one event's runs were
created within 4 s of each other, and the gate's up to 3 s after
another's, so exact equality would misdate a current failure; the
closest two events on one commit were 28 s apart.
- A current failure with no newer run of its workflow still fails at
once, including after a gate-only re-run.
- A current failure whose workflow already has a newer run, from a later
event, is waited on, and that run's result decides.
- A newer run, once listed, is cached; a listing that can't be read, or
lacks the check's own run, fails closed.
- Left open: two events within 10 s where the earlier one's check fails
and the later one's run is not yet listed when the gate judges it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* test, ci: foundations for hybrid v2 transport (Python ML-KEM, qbit releases, v1.0.0 interop, PQ CI, canary tools) (#224)
* test: add an independent ML-KEM-1024 Python reference
Add test_framework/crypto/mlkem.py, a pure-Python FIPS 203 ML-KEM-1024
written from the standard and independent of the C library: KeyGen from
a 64-byte d || z seed, Encaps from a 32-byte m, Decaps with implicit
rejection, and the section 7.2 (length and modulus) and 7.3 (length and
hash) input checks. Hashing uses hashlib's SHA3 and SHAKE only.
TestMLKEM1024, registered in feature_framework_unit_tests.py, checks:
- NIST ACVP-Server ML-KEM-1024 keyGen and encapsulation vectors (exact
ek, dk, ct and shared secret);
- ACVP decapsulation vectors, including modified ciphertexts whose
expected secret is the implicit-rejection value J(z || ct);
- ACVP encapsulationKeyCheck vectors: valid keys pass the section 7.2
check and encapsulate, the others are rejected;
- C2SP CCTV modulus vectors (q and 4095 at the first and last
coefficient of every polynomial, plus a sample of the full run), each
rejected, and accepted once the bad coefficient is set to 3328;
- round trips, implicit rejection on corrupted ciphertexts, and input
size and hash checks.
mlkem1024_test_vectors.txt holds the subset with its sources and pinned
commits in a header comment. The new tests take about 0.07 s; the whole
feature_framework_unit_tests.py run about 4 s.
Part of #184 (part 2, #186, work item B2).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: identify qbit releases independently of Core versions
Let functional tests download and run qbit release binaries the way
they run Bitcoin Core releases, starting with v1.0.0.
get_previous_releases.py resolves each tag through resolve_download().
qbit tags come from explicit descriptors: the GitHub release URL, the
archive per host (the "-unsigned" tarballs on macOS, whose -unsigned.zip
holds only the GUI app, and the "-unsigned" zip on win64), the archive
root and each archive's SHA256 copied from the release's SHA256SUMS.
Unsigned arm64 macOS binaries are self-signed as old Core ones are. A
host without a qbit archive fails with a message instead of a guessed
download. Bitcoin Core URLs, archives, hashes and messages are unchanged,
and qbit tags join the default download list, so the previous-releases
and Windows jobs fetch v1.0.0.
The test framework maps client versions through explicit ReleaseInfo
metadata (test_node.py): 10000 is qbit v1.0.0, based on Core v30.2
(core_compat_version 300200), with qbitd/qbit-cli binary names. A
version from 140000 (Core v0.14) up is the Core release it names, as
before; any lower version not in QBIT_RELEASES is an unknown qbit
release and raises, instead of being read as a Core version.
version_is_at_least() compares core_compat_version, so v1.0.0 nodes get
-v2transport and the logging options, and qbit_version_is_at_least()
gates qbit-only options: -p2mronly=0 now reaches qbit releases that have
it, and supports_v2_pq marks which nodes may get -v2pqtransport.
tool_qbit_release_resolution.py (qbit_release_resolution) checks v1.0.0
on every published host triple including win64 against a verbatim copy
of SHA256SUMS, the Core mappings, the rejection of unknown qbit
versions, and the options a v1.0.0, Core and current node get. It needs
no downloads, so it runs in every functional job, Linux and Windows.
Part of #184 (part 2, #186, work item B1).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: baseline v1.0.0 v2 transport interoperability
Add p2p_v2_previous_release.py, which pins how the qbit v1.0.0 release
binary treats non-empty BIP324 version-packet contents, where the hybrid
post-quantum handshake (#184) will carry its ML-KEM-1024 key and
ciphertext. It drives a BIP324 peer on a plain socket, so it controls
every byte and pause, against v1.0.0 in both roles:
- test_version_contents: with 0, 1,572 and 65,000 bytes of contents,
and with record-shaped contents (CompactSize(len) || header ||
payload): one 0xF0 record exactly as the hybrid offer and accept carry
it (fd 21 06 f0 || 1,568 bytes), a 0xF0 record followed by an unknown
0xF1 record, and a single 65,000-byte record, each sent after three
decoy packets, both immediately and after a 10 s pause under the
default 60 s -peertimeout, v1.0.0
completes the handshake, exchanges version/verack and ping/pong, sends
empty contents itself, and reports the plain BIP324 session id. As
initiator it sends its version packet and VERSION message before
receiving ours.
- test_limits: v1.0.0 keeps a connection whose packet length is exactly
MAX_CONTENTS_LEN (4,000,013), drops one at 4,000,014 bytes at once
("packet too large"), and drops a handshake stalled after the keys at
-peertimeout ("V2 handshake timeout"), within a bounded time.
The test skips without previous releases and runs in the
previous-releases job, which downloads v1.0.0 since the previous commit.
It takes about 24 s.
Part of #184 (part 2, #186, work item B3).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ci: add reusable maintained-branch validation entry points
Add .github/workflows/ci-pq.yml, one reusable workflow for the
post-quantum transport jobs, and make ci-nightly-heavy.yml callable and
job-selectable, so main's callers (C4), on a schedule and on demand,
can run either against a recorded commit. GitHub dispatches only
workflows whose file is on the default branch, so on-demand PQ runs go
through C4's caller on main (workflow_dispatch with source_ref and jobs,
calling ci-pq.yml@1.x.x), available after this part merges; ci-pq.yml
keeps its own workflow_dispatch trigger for when it is on main.
ci-nightly-heavy.yml is already on main, so it can also be dispatched on
a branch directly.
ci-pq.yml runs on workflow_call and workflow_dispatch with two inputs:
- source_ref: resolved once by the resolve-source job to a full commit
(empty: the triggering commit; a branch or tag, looked up on the remote
with annotated tags peeled; or a full hash). A missing, ambiguous or
malformed ref fails the run. Every job checks out that commit, verifies
HEAD against it, and the summaries record it.
- jobs: a comma-separated allowlist; empty runs all jobs, and an unknown,
repeated or empty name fails the run. The resolve-source job emits the
matrix, since a job-level if cannot see matrix values.
Its first job, aarch64-unit, runs on GitHub's ubuntu-24.04-arm runner
with the public-runner environment (docker.io/library images,
CI_ENFORCE_INTERNAL_REGISTRY=0, public DNS, no tailnet search domains),
builds only test_bitcoin and runs bip324_tests and net_tests
(ci/test/00_setup_env_native_aarch64_pq.sh). Its summary shows the
source commit, labels this the pre-library phase, and lists both suites
with their results.
ci/test/03_test_script.sh:
- CI_BUILD_TARGET (default all) selects the target of the build and of
its verbose retry; GOAL's install/deploy targets apply to full builds;
- CTEST_REGEX selects tests by name and is rejected together with
CTEST_INCLUDE_RANGE;
- ctest runs with --no-tests=error, lists the selected tests first and
writes JUnit results; when CTEST_EXPECTED_SUITES is set,
ci/checks/ctest_evidence.py fails the job unless every expected suite
was selected, executed and passed (ctest itself passes a skipped
suite, for example one compiled out).
ci-nightly-heavy.yml gains workflow_call inputs mirroring
workflow_dispatch (choices become strings, which run-scanners.py
validates), an optional LIBBITCOINPQC_READ_TOKEN secret, and a jobs input
over arm32-unit-1, arm32-unit-2, previous-releases, fuzz and scanner. The
matrix entries are unchanged and now come from resolve-source.
A called run gets its caller's github context, so github.event_name,
github.ref and github.sha are main's and cannot choose what a called run
validates. Only workflow_call takes a source_ref input (default 1.x.x,
must not be empty), and a non-empty source_ref marks a run as called:
- a called run resolves source_ref once, with the same function as
ci-pq.yml (which the contract test keeps identical), never reads
github.ref or github.sha, and records source_mode maintained when
source_ref names 1.x.x and requested otherwise; it honors
run_test_matrix, run_scanners, the scanner settings and jobs whatever
its caller's event;
- a direct scheduled run still resolves 1.x.x and follows the
CI_NIGHTLY_HEAVY_RUN_* variables and scheduled defaults, and a direct
dispatch keeps its selected ref and event commit;
- a first step fails the run when source_ref disagrees with GitHub's
job.workflow_ref, for example a caller that passes an empty source_ref,
and skips that check, saying so in its summary, where job.workflow_ref
is unavailable;
- a called run's concurrency group names its caller's event and
source_ref, so callers for different sources, and an on-demand call
beside the scheduled one, never cancel each other;
- the summaries record the source_ref input and source_mode.
ci/checks/test_ci_pq_contract.py executes the workflows' inline steps and
the 03_test_script.sh blocks against git fixtures and stub cmake/ctest:
job selection, rejection of unknown jobs, an immutable source commit,
suites that really ran, and called and direct nightly runs with their
concurrency groups. The shared expression evaluator follows GitHub's
loose equality and supports format() and toJSON().
test_scheduled_validation_contract.py reads the nightly matrix from its
new catalog. Both now run in required CI: the Required Merge Gate's
classify-changes job (.github/workflows/required-merge-gate.yml) gains
one step that runs them, beside the classifier contract.
Part of #184 (part 2, #186, work item C1).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* tools: add PQ canary sampling and evidence evaluation
Scripted evidence for the mainnet canary of the hybrid post-quantum v2
transport (#184, R5), ready before the 14-day R0 baseline starts on
v1.0.0 nodes. Everything lives in contrib/pq-canary/.
pq-canary-sample.sh runs every 5 minutes from cron or a systemd timer on
each canary node and the control node. Through qbit-cli it appends one
row to samples.csv: time, host label, uptime, version; enabled,
instance_id, backends, since, every per-direction handshake counter,
load_shedding.active and each ring's last_sequence and dropped from
getpqtransportinfo; connections_in and connections_pq; and, on the pool
node (--pinned), the pinned archive node's getpeerinfo connection_type,
transport_pq and transport_pq_status. sample_ok is 1 only when uptime,
getnetworkinfo and, on the pool node, getpeerinfo answered. A failed
call or a missing field is NA, never 0, and a real 0 stays 0, so a
v1.0.0 node writes NA for the new fields with sample_ok=1. The raw
recent_failures and fallback_set go to failures.jsonl, which keeps its
addresses between the team's canary nodes. --pinned and --host are
validated. pq_canary.py holds the format shared with the evaluator.
pq-canary-eval.py prints each figure with its sample count, coverage and
pass, fail or unknown: the pinned link manual and hybrid in at least 99%
of samples excluding restarts (with the disconnectnode remedy when an
automatic connection holds the address); no malformed_record,
first_packet_failed or fallback entry between known-good endpoints, on
every monitored canary node (one node's failures file cannot pass it);
closed_after_switch entries between them, listed for triage; and the
archive node's connections_in within +-20% of the control's, relative to
their baseline ratio, per UTC day, excluding 48 hours after a restart. A
jump in boot time is a restart gap. A ring that wrapped stays judged
while the entries read continue from the last sequence processed; only a
wrap that evicted unread entries, a changed instance_id or an
inconsistent ring makes that ring's interval unknown. A figure, node or
day below --min-coverage (default 90% of the expected 5-minute samples)
is unknown, not judged, and every figure reports its coverage. Missing
evidence is unknown, never clean. The report names known-good peers by
label and never prints a peer address, so it can be posted. Unknown,
malformed or missing arguments are rejected with a message and exit
status 2.
Tests: contrib/pq-canary/test/test_pq_canary.py (stdlib only, fixtures
in the #184 getpqtransportinfo schema) covers, among others, a monitored
outbound failure surviving 300 unrelated inbound failures, unrelated
noise, a restart gap, an RPC outage, a wrapped ring with and without
lost entries, per-node evidence, coverage thresholds, address redaction
and argument validation. The Required Merge Gate's PQ contract step
(.github/workflows/required-merge-gate.yml) now runs it as well, and
test/functional/tool_pq_canary.py runs the real sampler against the node
under test and, when previous releases are present, a v1.0.0 node. The
README covers install, the cron lines and how to read the output.
Part of #184 (part 2, #186, work item C5).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* contrib: treat missing PQ canary fallback-set evidence as unknown
The evaluator read a missing or malformed fallback_set as an empty list,
so a sample without fallback evidence could pass Figure 2. Record it as
an unknown interval for that host instead, as for malformed rings, and
iterate fallback_set only when it is a list.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* contrib: treat PQ canary records without an instance_id as unknown
A missing or malformed instance_id was compared like a real process
identity, so a restart between samples whose new ring did not go
backwards went unnoticed and lost failures could pass Figure 2. Mark
both rings unknown from the previous sample through the next record
with a valid 64-hex-digit instance_id.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* contrib: treat PQ canary entries with malformed endpoints as unknown
A ring entry whose endpoint was missing or malformed got no known-good
label and was skipped like a valid unrelated peer, so a fallback,
malformed_record or first_packet_failed entry without endpoint telemetry
could still let the failure figure pass.
Check that an endpoint has the getpqtransportinfo Endpoint shape: a kind
and network that agree, an address of that network without a port, and a
port from 0 to 65535. An entry that counts towards figure 2 or 3 and has
neither a known-good nor a well-formed endpoint now makes that figure
unknown. A malformed endpoint that still names a known-good peer remains
that peer's failure.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* contrib: reject PQ canary samples that compare a node with itself
Giving the same samples.csv for --archive and --control, or two files with
the same host label, made figure 4 compare a node with itself: a perfect
ratio that could pass at full coverage. The same applied to --pool and
--archive, which also set the nodes monitored for figure 2.
Require --pool, --archive and --control to be three different nodes. A
file given for two roles, two roles sharing a host label or an
instance_id, and a file whose rows carry no host label or more than one
are bad input: the evaluator exits with status 2 and names the roles.
The node test gave one node's samples as both pool and archive; give it
the pool samples alone and check the rejection instead. The unit tests'
simulated nodes now get distinct instance_ids, as real nodes do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: leave qbit releases without a host archive out of the default list
On powerpc64le, a host set_host() recognizes, running
get_previous_releases.py without tags now always included v1.0.0, which
has no powerpc64le archive, so the default download stopped with "No qbit
v1.0.0 archive".
Without tags, download every Core tag as before but only the qbit
releases that have an archive for the host, and say which were skipped.
A tag given explicitly still fails when it has no archive for the host.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* contrib: treat malformed PQ canary failure evidence as unknown
Audit every failures.jsonl field the evaluator reads, so that a missing,
malformed or contradictory value makes its stretch unknown rather than
clean, and never crashes the evaluator into a "fail" exit status:
- recent_failures that is not an object, and ring counters or sequences
that are booleans, negative or zero, are a malformed ring (a non-object
recent_failures raised AttributeError);
- an entry outcome outside the getpqtransportinfo vocabulary may be a
monitored or a triage one, so both figures are unknown (it was skipped
as clean; an object outcome raised TypeError);
- an entry time that is missing, malformed, before its process started or
more than CLOCK_SLACK after the sample that read it is unknown (it was
read as 1970 and skipped, or raised TypeError);
- a fallback-set entry that is not an object or lacks a well-formed
endpoint is unknown, once per distinct entry (it was skipped);
- since that is missing, malformed, after the sample or changed within one
instance_id makes the record's process identity unusable, like a missing
instance_id (it fell back to the sample time);
- a failures.jsonl line needs a time in Unix seconds up to year 9999 and a
host label, else it is bad input (a boolean time was accepted, and a
huge one crashed the report);
- a known-good failure whose reason, cause, connection type, peer id,
streak or times are malformed is still a failure, and those fields print
as "malformed": a malformed field could put an address in the report,
and malformed times or unhashable values raised.
Labels are now matched in full, so a host label with a trailing newline
is rejected. The unit tests' simulated nodes now report only failures
recorded by the time of each sample, and boot before their first sample.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* contrib: treat malformed PQ canary sample values as unknown
Audit every samples.csv field the evaluator reads, so that a missing,
malformed or contradictory value is unknown rather than a pass, a miss or
a crash:
- time, uptime and connections_in must be plain non-negative decimal
integers. int() also took "-30", " 30", "3_0" and non-ASCII digits, so a
negative uptime looked like a fresh restart and excluded samples from
figures 1 and 4; a time such as "soon" raised ValueError. A row without a
usable time is now a missing sample. --canary-start and --end must lie
between 1970 and 9999, and so must a sample time, which the report
prints.
- A row with more or fewer fields than the header, such as a line cut
short by a crash and continued by the next sample, has values in the
wrong columns, so it is not a sample.
- A connected pinned peer needs a connection type, transport_pq 0 or 1 and
a transport_pq_status from the getpeerinfo vocabulary, with transport_pq
1 exactly when the status is hybrid. A malformed or contradictory value
is unknown: it counted as a miss, and transport_pq=1 with status
"pending" counted as hybrid.
- Figure 4 pairs only samples with sample_ok=1.
- An uptime whose boot time contradicts the one already seen for the same
instance_id is unknown: it was read as a restart and excluded samples.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* contrib: write only well-typed values to PQ canary samples
The sampler wrote any integer, string or boolean it found: true became a
connections_in or counter of 1, a negative uptime or since was kept, and
an instance_id of any shape was recorded. Write each column only when the
RPC value has its contract type (counts are non-negative integers, flags
booleans, backends and statuses strings, instance_id 64 lowercase hex
digits); anything else is NA, which the evaluator reads as unknown.
Reject --host NA, which would read back as a missing host label.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* contrib: keep one stray PQ canary value from making whole stretches unknown
Two of the new unknown rules reached further than the evidence:
- An uptime was checked against the first boot time seen for its
instance_id. A restart between the sampler's uptime and
getpqtransportinfo calls pairs the old uptime with the new instance_id
once, which made every later uptime of that process unknown. Check each
uptime against the median boot time of its instance_id instead, so one
racy or malformed value is the only unknown.
- An entry with an implausible time made figure 2 or 3 unknown even when
the sample that first read it was taken long before the window. Such an
entry was recorded before that sample, so it cannot be in the window.
Also rename the samples() helper to sample_rows(), which evaluate_failures
shadowed with its sample count.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: check the release default list without the real host
tool_qbit_release_resolution's default-tags case set HOST to make
get_previous_releases.py act as a powerpc64le host. set_host() ignores
HOST on Windows and answers win64, so on the Windows CI job the case
downloaded the v1.0.0 win64 archive and failed. Stand in for set_host()
so the case checks main()'s use of the host on any platform.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: skip the v1.0.0 transport baseline unless v1.0.0 is downloaded
p2p_v2_previous_release.py only checked that previous releases were
enabled. A releases directory that holds other releases, such as Bitcoin
Core's, but not qbit v1.0.0 enables them, so the test went on and
setup_nodes() aborted on the missing qbitd and qbit-cli. Skip unless
both v1.0.0 binaries are present, as tool_pq_canary.py leaves out its
v1.0.0 sample.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* contrib: count each expected PQ canary sample once
Figure 1 counted every samples.csv row in the window, so rows repeated
for one sampling slot (overlapping files, a manual run beside cron)
added to its total: 95 of 100 expected samples is unknown, but the same
rows recorded six times read as 99.1% hybrid and passed.
Expected samples are now --interval apart from --canary-start, a sample
stands for the nearest one, and every figure counts an expected sample
once. Several samples for one count as the least favorable of them, so
an extra sample can never improve the evidence: for the pinned link a
miss, then unknown, then excluded, then hybrid; for inbound connections
unknown, then excluded, then a measurement, where measurements that
disagree are unknown (figure 4 kept whichever row came last). Figure 2
counted a record nearest to an expected sample just outside the window
as covering one inside it; it now uses the same expected samples.
A row repeated exactly is one sample, so repeated rows cannot outvote
the median boot time that decides whether an uptime is a restart: one
racy uptime copied a hundred times made two misses look like a restart
and excluded them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: find every host qbit publishes v1.0.0 archives for
get_previous_releases.py registers the arm-linux-gnueabihf and
powerpc64-linux-gnu v1.0.0 archives, but set_host() had no pattern for
either host, so on 32-bit ARM or big-endian POWER it exited "Not sure
which binary to download" before resolve_download() saw them.
Map both hosts. Each pattern for a qbit host now accepts the depends
triplet given in HOST, which the script's help documents, as well as
config.guess output. This also fixes the upstream-derived aarch64,
riscv64 and arm64 macOS patterns, which only matched config.guess
output: HOST=aarch64-linux-gnu, riscv64-linux-gnu or arm64-apple-darwin
exited the same way. The ARM patterns accept only little-endian ARMv7
or ARMv8 hosts with hard float, because the v1.0.0 ARM binaries are
ARMv7 VFPv3 armhf builds; an ARMv6 host still gets "Not sure which
binary".
powerpc64le (Core archives only) and x86_64-w64-mingw32 (win64 comes
from platform.system(), not HOST) are left as they were.
tool_qbit_release_resolution now runs set_host() with a mocked
platform, HOST and config.guess output for every published host, and
checks that each answer resolves to that host's v1.0.0 archive.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: skip the v1.0.0 canary sample unless both executables exist
tool_pq_canary enabled its v1.0.0 node whenever releases/v1.0.0/bin
existed. get_previous_releases.py creates the release directory before
extracting into it and keeps it after a failed extraction, so a stale
or partial download could leave bin/ without qbitd or qbit-cli, and
add_nodes() then aborted the whole test. Check both resolved
executables, as p2p_v2_previous_release does, and otherwise skip only
the optional v1.0.0 sample.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* contrib: pass a PQ canary day exactly 20% off its baseline ratio
Figure 4 passes a day whose archive/control connections_in ratio is
within 20% of the baseline ratio, inclusive. The evaluator computed the
relative ratio in floats with two divisions, so a day exactly on the
bound could round past it: 24/10 against a baseline of 3 came out as
0.7999999999999999, and 108/10 against 9 as 1.2000000000000002, and
both failed the figure and the release gate.
Compute the ratios as exact fractions, and read --baseline-ratio
exactly too, since the float nearest 1.2 is below it. float() still
screens the argument first, so nan, infinity and exponents outside its
range are rejected as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* contrib: match a known-good IP address however it is written
The failure figures look endpoints up in the known-good list by string.
The node reports an IPv6 address compressed, so a list entry spelled
out, such as [2001:0db8:0:0:0:0:0:5]:8333, never matched 2001:db8::5:
its first_packet_failed entries were counted as another endpoint's, and
Figure 2 passed at full coverage despite a failure between known-good
peers.
Give every IP address one spelling on both sides, the list and the
node's endpoints, address-only entries included: compressed, lower
case, and an IPv4-mapped IPv6 address as IPv4, which is how the node
reports it. Two spellings of one entry are now "listed twice".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* crypto: vendor mlkem-native and build ML-KEM-1024 with runtime backend selection (#225)
* crypto: vendor pinned mlkem-native without source changes
Vendor mlkem-native v2.0.0 (d1b2fe782888bdb761a50336012923180be7f502):
the whole upstream mlkem/ directory and LICENSE, byte for byte, in
src/mlkem-native. Nothing compiles it yet.
contrib/devtools/mlkem-native.pin records the commit, tag, the tree id
of mlkem/ and the LICENSE blob id. contrib/devtools/update-mlkem-native.sh
reproduces the tree from upstream and rewrites the pin from the fetched
object ids. test/lint/mlkem-native-check.sh, run by the subtree lint,
proves offline that HEAD still holds exactly the pinned tree. Lints that
scan src/ ignore the vendored tree, and doc/subtrees/mlkem-native.md is
the runbook.
Part of #184 (part 3, #190, D1).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* crypto: share x86 feature primitives and detect usable AVX2
Move the XGETBV instruction out of sha256.cpp into compat/cpuid.h as
XGetBV(), next to GetCPUID(). SHA256AutoDetect keeps its own selection
logic unchanged; it calls the shared primitive only.
Add compat/cpu_features.{h,cpp} with a standalone answer to "can this
CPU and OS run AVX2 code": the supported leaves are queried first, XCR0
is read only once leaf 1 reports OSXSAVE and AVX, and AVX2 is usable only
with XMM and YMM state enabled and the leaf 7 AVX2 bit set. Unlike the
SHA256 logic it does not depend on SSE4 or SHA-NI.
HasMlkemX86Native() adds the other extensions that mlkem-native's x86_64
assembly executes, found by auditing the assembled object: SSSE3,
SSE4.1, POPCNT and BMI2. A virtual machine can show AVX2 while hiding
those, and checking AVX2 alone would then end in SIGILL. Both checks
detect once and cache the result, because the ML-KEM capability hook
asks about a hundred times per operation. Non-x86 targets and compilers
without <cpuid.h> report false.
Part of #184 (part 3, #190, D2).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* crypto: require SSSE3 for the SSE4 and SHA-NI SHA256 paths
SHA256AutoDetect selects the SSE4 code whenever CPUID leaf 1 reports
SSE4.1, and only under that check also the 4-way SSE4.1 code and the
SHA-NI code. All three execute SSSE3 instructions. An audit of the built
objects finds pshufb and palignr in the SSE4 and SHA-NI code and pshufb
in the 4-way code, besides SSE4.1's pinsrd, pextrd and pblendw; no SSE4.2
instruction occurs. A virtual machine can report SSE4.1 while masking
SSSE3, and the self-test then dies with SIGILL at startup.
Require SSSE3 as well. Under qemu-x86_64, -cpu Haswell,-ssse3 and the
SHA-NI model EPYC,-ssse3 die with SIGILL before this change and select
the standard implementation after it. Unmasked, Haswell still selects
sse4, sse41 and avx2, and EPYC and Denverton x86_shani.
This is an upstream Bitcoin Core bug, so the change is kept minimal
enough to go upstream.
Part of #184 (part 3, #190).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* crypto: add ML-KEM-1024 wrapper and runtime backend dispatch
Build the vendored mlkem-native as ML-KEM-1024 into bitcoin_node only,
behind a small C++ wrapper. Nothing in qbitd calls it yet.
- src/crypto/mlkem_config.h configures the library (MLK_CONFIG_FILE):
parameter set 1024, the qbit_mlkem1024 prefix, deterministic APIs only,
zeroization through memory_cleanse, and a capability hook.
- Native backends are chosen by qbit-owned backend headers instead of
upstream's compile-time defaults, which enable x86_64 only for AVX2
compilers. No source is compiled with an ISA flag: every native entry
point asks the hook, which reads only cached CPU detection and one
atomic portable override. On x86_64 that detection requires every
extension the assembly executes (AVX2, SSSE3, SSE4.1, POPCNT, BMI2),
not only the AVX2 that upstream's capability name says.
- Upstream's AArch64 single-lane Keccak entry point skips the hook, so
the override could not reach it. qbit mirrors that header's dispatch
glue with the check added; every other entry point checks itself.
- mlkem::KeyGen, CheckPublicKey, Encaps and Decaps take fixed-size spans,
map upstream's invalid-key results separately from internal failures,
and clear their outputs on every failure. GetBackendNames() reports the
implementation that runs. ForcePortableForTesting and
InjectResultForTesting are test-only guards.
- cmake/mlkem-native.cmake adds WITH_MLKEM_NATIVE=AUTO|ON|OFF and the
configure summary line. The library gets core_interface's hardening
flags but not qbit's C++ warnings, then APPEND_CPPFLAGS and
APPEND_CFLAGS, so the MSan jobs' -U_FORTIFY_SOURCE applies to it as to
the other C libraries. MSVC builds without assembly, MSan
builds portable C, and AArch64 ELF targets build portable C because
the upstream assembly has no BTI landing pads or GNU property note and
would strip BTI/PAC marking from the binary (doc/subtrees has the
evidence and the revisit trigger). ON fails where native is impossible.
mlkem_tests covers NIST ACVP keygen, encapsulation, decapsulation and key
checks, all 1040 C2SP CCTV modulus vectors and the 3328/3329 boundary,
implicit rejection against an independent SHAKE256, injected failures,
moves and wiping, and backend selection, each on the native backend and
with portable C forced. ci/checks/test_mlkem_build_policy.py checks the
WITH_MLKEM_NATIVE matrix, builds the library with each compiler found
(core-checks requires the AArch64 cross compiler), and counts assembly
calls to show that the portable override reaches every x86_64 routine.
Part of #184 (part 3, #190, D3).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fuzz: exercise ML-KEM validation and backend equivalence
Add two fuzz targets over the mlkem:: wrapper:
- mlkem: honest key generation, encapsulation and decapsulation agree;
an arbitrary encapsulation key is accepted exactly when an independent
modulus check passes, and a rejected one leaves no output; any
ciphertext of the right length decapsulates to a secret that differs
from the honest one (implicit rejection is never proof of validity);
an arbitrary decapsulation key is rejected exactly when an independent
SHA3-256 hash check fails, and the secret is then cleared.
- mlkem_backend_diff: the detected backend and forced portable C give
equal statuses and outputs for the same inputs, and the override is
restored afterwards.
Both honor MLKEM_FORCE_PORTABLE=1. They join the required qbit corpus
with nine generated seeds each (CCTV modulus key and 3328/3329/4095
coefficients, NIST ACVP decapsulation vectors, a ciphertext bit flip).
CI replays them a second time with MLKEM_FORCE_PORTABLE=1 and includes
them in the mutation phase; the PQ CI contract test checks that wiring.
UBSan's -fsanitize=integer checks are suppressed for the vendored tree,
as for the other external libraries.
Part of #184 (part 3, #190, D5).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* bench: measure ML-KEM operations by active backend
Benchmark ML-KEM-1024 key generation, encapsulation and decapsulation
twice each: on the detected backend and with portable C forced. Like the
per-implementation SHA256 benchmarks, each is named after the backend
that ran, so on an AVX2 host bench_qbit -filter='MLKEM.*' lists both,
and the portable guard restores auto-detection afterwards.
Part of #184 (part 3, #190, D5).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ci: add s390x and macOS arm64 PQ unit jobs
Add two jobs to the PQ workflow, selectable by name like aarch64-unit:
- s390x-unit cross-compiles only test_bitcoin for big-endian s390x on
the x64 runner, with Ubuntu's g++-s390x-linux-gnu and the depends tree,
and runs the suites through ctest with qemu-s390x as the crosscompiling
emulator, so only the tests run emulated. It stays out of the merge
gate. Its timeout is 180 minutes until runs are measured.
- macos-arm64-unit runs natively on a GitHub-hosted macos-14 runner
without Docker, asserts that uname -m is arm64, and shares the unit
job's source checks and evidence report through YAML anchors. It is
the job that runs native AArch64 ML-KEM, and configures with
WITH_MLKEM_NATIVE=ON so that losing native code fails it.
All three jobs now expect mlkem_tests besides bip324_tests and net_tests
(CTEST_REGEX ^(mlkem_tests|bip324_tests|net_tests)$), and report the
suites through ctest_evidence.py. The aarch64 job's description says
that ML-KEM runs portable C on AArch64 Linux. The PQ CI contract test
covers selecting each job, the s390x cross and emulator settings, the
macOS architecture check and report, and that every expression uses a
context GitHub makes available there (no runner in job env, no matrix
in a job without one), which GitHub enforces when loading the file.
Part of #184 (part 3, #190, C2 and the CI part of D3).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ci: gate ML-KEM and transport changes on the aarch64 PQ job
The merge-profile classifier gains a pq_aarch64_required output, true when
a change touches the ML-KEM library (src/mlkem-native, src/crypto/mlkem*,
cmake/mlkem-native.cmake), the v2 transport (src/bip324*, src/net.h,
src/net.cpp), the CPU feature checks, the PQ workflow or its env files,
or the PQ test suites and vectors. A path it cannot read also requires
the job.
Full Validation calls ci-pq.yml with jobs: aarch64-unit on the change
itself (a pull request's merge commit) when the output is true. The gate
accepts a skipped job only when the classifier says it is not required;
a required job that was skipped, failed or cancelled, or a missing or
unreadable classifier output, fails it. The changed-file lists that feed
the classifier, in Full Validation and in the Required Merge Gate, use
--no-renames, so moving a file out of a gated path still names it.
ci/checks/test_pq_merge_gate_contract.py runs the real classifier, job
condition, ci-pq selection step and gate script: a pull request touching
src/bip324.cpp requires and runs the job, while docs-only and other
source pull requests skip it and pass, and a rename out of a gated path
still requires it. The Required Merge Gate runs it with the other PQ CI
contracts.
Part of #184 (part 3, #190, C3).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ci: run the PQ macOS arm64 job on macos-15
GitHub is retiring the macos-14 runner image: brownouts start on
2026-10-05 and the image is removed on 2026-11-02. Move the
macos-arm64-unit job to macos-15, which is also Apple Silicon.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* build: probe the ML-KEM target with the appended flags
APPEND_CPPFLAGS and APPEND_CFLAGS reach mlkem_native's C and assembly
compiles, but the architecture probes ran without them. A build that
appends a target-changing flag such as -m32 was detected as x86_64,
enabled the native backend, and then failed in mlkem_config.h. Run the
probes with the appended flags, and probe afresh on every configure,
since those flags may change between configures.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ci: require the macOS arm64 and x86_64 PQ jobs on PQ changes
No pull request ever built or ran ML-KEM's native code on macOS: the
macos-arm64-unit job could only be dispatched, and nothing ran the x86_64
native code on an Intel Mac at all. Apple Silicon and Intel macOS are
release targets with native code enabled.
ci-pq.yml gains a macos-x86_64-unit job on GitHub's Intel image
(macos-15-intel), sharing every step with macos-arm64-unit. Both build
with WITH_MLKEM_NATIVE=ON, check the runner's architecture, and after the
suites pass require the native backend by name from
mlkem_tests/backend_selection, so a runner whose CPU lacks an extension
cannot pass on portable C.
Full Validation's PQ job, renamed pq-unit, now calls ci-pq.yml with
aarch64-unit, macos-arm64-unit and macos-x86_64-unit, so all three are
required exactly when the classifier says the change needs them. The
classifier output is renamed pq_unit_required to match. On-demand runs
accept the new job name; an empty jobs input runs it too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: check the portable override on AArch64 and per entry point
The build-policy test counted assembly calls only for the x86_64 backend,
so nothing in CI showed that the portable override reaches every AArch64
assembly routine. With upstream's x1 Keccak glue in place of qbit's
mirror, 221 assembly calls ran with portable C forced, and every test
passed: outputs agree, and the backend name comes from the hook.
The counting harness now also runs on AArch64: a static binary built with
aarch64-linux-gnu-g++, from a copy of mlkem_config.h without the AArch64
ELF refusal, and run under qemu-aarch64. A negative control builds
upstream's x1 glue and requires the leaked calls to show.
Each entry point also runs alone on a new thread while portable C is
forced, and must make no assembly call: one that skipped BeginOperation()
would run on the new thread's stale copy of the override. The earlier
round trip could not show this, since KeyGen set the copy for the whole
thread.
The build smoke job installs g++-aarch64-linux-gnu and qemu-user and
lists them as required, so the AArch64 case cannot skip.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: cover each ML-KEM entry point's override and its per-thread copy
Two wrapper mutations passed every test. An entry point without
BeginOperation() went unnoticed because ForEachBackend calls
GetBackendNames() on the same thread first, which takes the override for
it. A shared (not thread_local) copy of the override went unnoticed
because only one thread ran operations.
every_entry_point_takes_the_override runs each entry point alone on a new
thread with portable C forced, then asks the capability hooks what that
thread's copy says. override_copy_is_per_thread lets one thread take the
override set while another holds it unset, and checks that the first
thread's copy did not change. override_changed_by_another_thread now runs
operations on two threads at once, so a shared copy also gives wrong
outputs on x86_64 and a data race under ThreadSanitizer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fuzz: check that the ML-KEM portable replay runs portable C
CI replays the mlkem and mlkem_backend_diff targets a second time with
MLKEM_FORCE_PORTABLE=1, but nothing checked that the binary honored it:
any other value, such as "true", silently selected native code, and the
second replay would have repeated the first.
Initialize now refuses any value but unset, empty, 0 or 1, prints the
active backend once, and aborts when MLKEM_FORCE_PORTABLE=1 but the
active backend is not portable/portable. A host without native code also
shows in the log, where mlkem_backend_diff compares portable C with
itself.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: cover SHA256's SSSE3 requirement with synthetic CPUs
SHA256AutoDetect requires SSSE3 besides SSE4.1 before it selects its SSE4,
SSE4.1 or SHA-NI code, all of which execute SSSE3 instructions. Nothing
tested that: dropping the SSSE3 term passed every test, because real
SSE4.1 hardware always has SSSE3.
An overload of SHA256AutoDetect takes the CPUID and XGETBV functions, as
cpu_features::DetectX86Features does, and the existing one passes the real
ones, so the queries and the selection are unchanged. A cpu_features_tests
case runs it on synthetic CPUs derived from this one: unmasked it selects
what this CPU gets, and with SSSE3 or SSE4.1 masked the standard code.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* doc: say exactly where ML-KEM native code runs
The wrapper's header said native code runs on "x86_64 with AVX2,
AArch64", and the WITH_MLKEM_NATIVE help text spoke of "the x86_64 and
AArch64 ML-KEM assembly". AArch64 runs native code only on macOS (AArch64
ELF builds stay portable to keep their BTI and PAC marking), and x86_64
also needs SSSE3, SSE4.1, POPCNT and BMI2, as
cpu_features::HasMlkemX86Native checks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* contrib: keep the clone unshallow when vendoring mlkem-native
update-mlkem-native.sh fetched the upstream release with --depth=1 into
the qbit repository itself. That writes .git/shallow, so a full clone
then reported itself as shallow to git and every tool that asks.
Fetch into a temporary bare repository instead, and read the commit, the
object ids and the archive from there. Staging and the byte-for-byte
check still run in the qbit repository. Re-vendoring v2.0.0 by tag and by
commit leaves no diff, and the clone stays complete.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* doc: record the local x86_64 GNU property evidence for mlkem-native
#190 asks for ELF property evidence that linking the x86_64 assembly keeps
the CET marking. Record what a local Ubuntu build shows: the assembly and
C objects and the test_qbit and fuzz binaries that link the native
backend all carry IBT and SHSTK, the binaries need only baseline x86-64,
and the assembly loses its note only when built with
-fcf-protection=none. The Guix release evidence is still outstanding and
is required before the post-quantum feature branch merges into 1.x.x.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: name the ML-KEM entry point instead of capturing a binding
every_entry_point_takes_the_override captured a structured binding in
the thread's lambda, which compilers accept only from C++20 (P1091R3,
P1381R1). Bind the label and the entry point to ordinary variables, so
the test does not depend on that support.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ci: build the macOS PQ jobs' Boost and libevent through depends
The first macOS arm64 run failed to compile txmempool.h: with
NO_DEPENDS=1 the job used Homebrew's Boost, now 1.92, and Boost 1.91
stopped accepting a struct derived from multi_index::indexed_by as an
index list. Bitcoin Core fixed the source in bitcoin/bitcoin#35175; that
backport is handled on its own.
Build Boost and libevent with depends, natively for the runner, so the
jobs compile against the versions qbit releases with. Homebrew now
installs GNU Make 4 for depends instead of the two libraries. A step
sets HOST and reads depends' own build id (packages, recipes and
compilers), which keys an actions/cache of depends/built. It is saved
once the toolchain file exists, even if the suites fail. After the
build, a step requires that depends built that id and that CMake took
the toolchain, libevent and Boost at boost.mk's version from it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* net: hybrid post-quantum v2 handshake with key confirmation (#184 part 4) (#230)
* bip324: derive hybrid keys from an ordered transport transcript
Add the hybrid ML-KEM-1024 key recipe of #184 to BIP324Cipher. Nothing
calls it from the transport yet; E3 does.
- Initialize(..., retain_for_hybrid) keeps the raw BIP324 ECDH secret,
starts a running SHA256 transcript with ellswift_I || ellswift_R, and
snapshots the network magic and cipher direction. The default (false)
keeps today's secret lifetime and the existing vectors.
- AddVersionContents hashes CompactSize(len) || contents, responder
first, through a small stream adapter without copying the contents.
- SwitchToHybrid derives HKDF-SHA256 with salt
"qbit_v2_hybrid_mlkem1024" || magic over ss_ECDH || ss_MLKEM || TH,
re-emplaces all four packet ciphers with counters at 0 and replaces
the session id; garbage terminators stay. Misuse (not initialized, not
retained, wrong contents count, a second switch) returns false and
changes nothing.
- The retained secret, IKM, HKDF state and derived keys are wiped on
switch, in DiscardHybridSecret and in the destructor. Copying is
deleted.
The same recipe goes into test_framework/crypto/bip324_cipher.py, and
contrib/devtools/generate-pq-transport-vectors.py uses it, the Python
ML-KEM-1024 and the BIP324 ECDH helpers (imported by path, never the C++
code) to write src/test/data/pq_transport_vectors.json: inputs, every
intermediate, both confirmations and first packets, and negative cases.
Its --check mode runs in lint through test/lint/lint-pq-transport-vectors.py.
bip324_tests gains hybrid_vectors (both roles and self-decrypt mode,
with every intermediate recomputed from qbit's own primitives),
hybrid_transcript_binding, hybrid_api_order and hybrid_rekey_boundaries.
The bip324_cipher_roundtrip fuzz target gains an optional switch,
transcript and secret mutations and discards. The existing BIP324
vectors are unchanged.
Part of #184 (#199, E1).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* doc: specify hybrid v2 transport (doc/design/pq-transport.md)
Add the specification of the hybrid ML-KEM-1024 v2 transport from #184's
design, written for interoperating implementations and outside review:
the record grammar and header registry (0x00 reserved, 0xF0 for the
ML-KEM-1024 offer and accept, qbit's values from 0xF0 upward), byte
layouts, parse rules (unparseable contents are ignored, unlike the
Transport Feature Negotiation draft), the key recipe byte by byte, the
responder and initiator message rules and why progress still holds, the
switch boundary, key confirmation MUSTs, failure handling, security
notes, an interop walk-through of the vectors file, the convergence and
retirement policy, and a contributor section.
The spec quotes vector 0 in a marked block, and…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Part 2 of #184 (#186): the foundations later parts build on. There is no change to node behavior. This is the bottom of the #184 PR stack on
1.x.x. C4, a small scheduler PR onmain, follows after this merges, and completes #186.B2. Independent Python ML-KEM-1024.
test_framework/crypto/mlkem.pyis a pure-Python FIPS 203 implementation with self-tests, written independently of the C library. Its public API is four functions (mlkem1024_keygen,mlkem1024_encaps,mlkem1024_decaps,mlkem1024_check_ek), which later parts' vector generator and Python hybrid peer use. The self-tests cover pinned NIST ACVP keyGen, encapDecap and key-check vectors, C2SP CCTV modulus vectors and implicit rejection.B1. qbit releases in the previous-release tooling. It uses explicit release metadata (
ReleaseInfo). It resolves qbit's GitHub release assets, including the-unsignedmacOS and win64 names, and takes hashes from the release'sSHA256SUMS.version_is_at_leastcomparescore_compat_version(qbit 1.0.x → 300200), and a newqbit_version_is_at_leasthandles qbit-only options. Unknown or too-low qbit versions are rejected. Every Bitcoin Core mapping still resolves as before.B3. v1.0.0 interoperability baseline.
p2p_v2_previous_release.pychecks v1.0.0 in both roles with these version-packet contents:0xF0record;This pins today's behavior before any node change.
C1. Reusable PQ CI workflow.
.github/workflows/ci-pq.yml: asource_refinput resolved once to a SHA, ajobsallowlist and matrix, andaarch64-uniton GitHub's arm64 runner with a public-runner environment.03_test_script.sh:CI_BUILD_TARGETandCTEST_REGEX, with--no-tests=errorand JUnit evidence that every expected suite ran and passed.ci-nightly-heavy.yml:workflow_callwith asource_refinput (default1.x.x) and ajobsinput. A called run validatessource_ref, resolved once, never the caller's ref or commit; it honors the caller's inputs, and fails closed ifsource_refis empty (cross-checked withjob.workflow_ref). Direct scheduled and dispatched runs behave as before. Concurrency groups include the source, so callers for different sources don't cancel each other.ci/checks/, run in the Required Merge Gate's classify step.GitHub only dispatches workflows that exist on the default branch, so on-demand PQ runs will go through C4's caller on
main.C5. Canary tools.
contrib/pq-canary/adds a 5-minute sampler and an evaluator for the mainnet canary in the release gate.NA, never 0, and missing evidence isunknown, never clean.unknown, never divided by.tool_pq_canary.pyruns the real sampler against the node under test, and against v1.0.0 when previous releases are present.Testing
feature_framework_unit_tests.py(ML-KEM self-tests),tool_qbit_release_resolution.py,p2p_v2_previous_release.pyagainst the real v1.0.0 binaries,tool_pq_canary.pyand the canary suite (34 tests);wallet_listtransactions.pythat passed 3 of 3 times on rerun.ci/checkscontract tests, of which 14 of 14 mutations were caught.ci-pq.ymlruns follow C4.Two independent adversarial reviews found no correctness defect in B1–B3; their fixes are folded in.
SHA256SUMS.Target Branch
1.x.x, where the p2p: add hybrid post-quantum key exchange (ML-KEM-1024) to BIP324 v2 transport #184 work lands.Risk / Review Notes
Notes: these are CI and test-framework changes only.
required-merge-gate.ymlgains one fast, stdlib-only step that runs the new contract tests and the canary evaluator tests. Nosrc/file changes.Docs / Process Impact
contrib/pq-canary/README.mdand the CI contract notes.Refs #186 · Part of #184
🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.