Skip to content

build(tooling): pin maintained-client downloads to HTTP/1.1 - #1476

Open
doublewhy wants to merge 3 commits into
devfrom
1475-curl-http1-pin
Open

doublewhy wants to merge 3 commits into
devfrom
1475-curl-http1-pin

Conversation

@doublewhy

@doublewhy doublewhy commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Summary

curl_transfer_argv (tools/maintained_client_acquisition.py) builds the one argv that qualification and every real acquisition share. It set no HTTP version, so curl negotiates HTTP/2 with the download hosts. #1475 reproduces this on dev 3512210 with macOS /usr/bin/curl 8.7.1:

  • A size-limit breach exits 56 over HTTP/2. run_curl_transfer then reports curl-transfer-failed instead of curl-size-limit-enforced.
  • A 503 gets one request instead of three (the budget allows two retries).

With --http1.1, the same transfers exit 63 and, for the 503, 22 after three requests. Qualification never saw the difference: its http.server fixture offers no ALPN protocol, so curl always used HTTP/1.1 there. The #1344 fixture-provenance audit recorded the same exit codes (finding NS-1 in the record #1461 proposes).

This PR adds --http1.1 to the shared argv, inside the design #1246 introduced. No outcome classification, budget, retry or redirect setting changes. The real-curl fixture now offers h2 before http/1.1 through ALPN, as most download hosts do (the Cambridge Isabelle mirror, www.cl.cam.ac.uk, offers only http/1.1). http.server speaks only HTTP/1.x, so a client that accepted h2 fails the real-curl tests. That makes those tests, and the large-object ones in test_issue_1220_isabelle_acquisition.py that reuse the fixture, fail without the pin.

The participant-opacity proof evidence binds each prover-acquisition source by digest, so the second commit records the new digest of tools/maintained_client_acquisition.py there.

Requirement UIDs

  • None. Repository tooling fix with no requirement scope.

Related Issues

Closes #1475

ADR Impact

  • None. ADR-106 decision 4 keeps HTTP behavior in maintained curl ("Repository adapters own selection, local integrity checks and installation, never HTTP behavior"). --http1.1 is native curl configuration, like the --proto, --max-redirs and --retry flags already in the argv. No repository HTTP code is added.

Changes

  • tools/maintained_client_acquisition.py: curl_transfer_argv adds --http1.1 after the --proto and --proto-redir restrictions, with a comment giving the reason.
  • implementations/python/tests/test_issue_1217_bootstrap_profiles.py:
    • test_curl_qualification_uses_fixed_hardened_argv asserts that --http1.1 is the argv's only --http* flag.
    • _https_fixture offers h2 and http/1.1 through ALPN (ssl.SSLContext.set_alpn_protocols), so the real-curl tests see a client that negotiates HTTP/2. Its comment says most download hosts offer h2 first (third commit, 578c6fec): every host probed below negotiated h2 except the Cambridge Isabelle mirror.
  • specs/formal/participant-semantics/participant-opacity-proof-evidence.json (second commit, 7d2f81d6): the toolchain.tool_sources digest of tools/maintained_client_acquisition.py. The proof checker binds that file by digest (PROOF_TOOL_SOURCE_PATHS in tools/check_participant_opacity_proof.py). On the first head, 76cbe186, the old digest failed three CI jobs. canonical / proof and six test_issue_963_participant_opacity_proof.py tests in canonical / integration reported proof tool source digest does not match its repository source (CI run 38046775269). The Bootstrap dev-container smoke expects the proof session to stop at its bubblewrap check, and it did not find that line (Bootstrap qualification run 38046775103). Locally the checker validates this digest before its platform and sandbox checks. Only that digest changes. refactor(tooling): simplify connected developer and release workflows #1317 re-pinned other tool sources in this manifest the same way.

Test Plan

  • Unit tests pass
  • Integration tests pass if applicable
  • Full completion suite required in CI before merge
  • No coverage regression

All runs were on macOS, where /usr/bin/curl --version reports curl 8.7.1 (x86_64-apple-darwin26.0) libcurl/8.7.1 (SecureTransport) LibreSSL/3.3.6 zlib/1.2.12 nghttp2/1.69.0. Commands ran from implementations/python.

Real-curl tests on 578c6fec, with the selection the Bootstrap qualification workflow uses: uv run --frozen --all-extras python -m pytest tests/test_issue_1217_bootstrap_profiles.py tests/test_issue_1220_isabelle_acquisition.py -q -p no:cacheprovider -m integration -k real_curl passed (8 passed).

The same run fails without the pin. With tools/maintained_client_acquisition.py restored to dev's version and the new tests kept, 7 failed and 1 passed:

  • Every transfer returned curl-transfer-failed, for example {'reason_code': 'curl-transfer-failed'} != {'reason_code': 'curl-size-limit-enforced'}.
  • The one pass is the /disconnect case, which expects that reason.
  • In a separate default-marker run, test_curl_qualification_uses_fixed_hardened_argv failed with assert [] == ['--http1.1'].

A scratch http.server TLS fixture checked the fixture change before it was written:

  • With no ALPN offer, as on dev, dev's argv and the pinned argv got the same results.
  • With the h2 offer, the server saw h2 from dev's argv and every transfer failed. From the pinned argv it saw http/1.1, and the transfers passed.

Download hosts, for the fixture's ALPN offer: for the first source_urls entry of each of the 12 hosts in implementations/tooling/artifacts.lock.json, /usr/bin/curl -sS -o /dev/null -I -w '%{http_version}', which offers h2, negotiated HTTP/2 with 11 hosts and HTTP/1.1 with www.cl.cam.ac.uk, the Isabelle archive's second URL. Against that host, openssl s_client -alpn h2,http/1.1 selects http/1.1, and an h2-only offer negotiates no ALPN protocol.

The #1475 reproduction against real hosts, rerun on this branch, uses the argv from curl_transfer_argv plus --verbose:

  • The ATLAS asset with the limit one byte short exits 63 over http/1.1 after 2 requests (redirect and asset).
  • At the exact limit it writes 625,790 bytes.
  • https://httpbin.org/status/503 and https://postman-echo.com/status/503 exit 22 after 3 requests.

Through the production entry points, run_curl_transfer returns curl-size-limit-enforced for the oversize transfer (dev: curl-transfer-failed), and run_curl_qualification returns passed for it. The exact-limit transfer returns curl-transfer-qualified.

Module runs on 578c6fec:

  • Default markers, test_issue_1217_bootstrap_profiles.py, test_issue_1137_maintained_client_acquisition.py, test_issue_1221_vocabulary_maintained_client_acquisition.py, test_tooling_artifact_policy.py and test_issue_1313_workflow_policy.py: 238 passed.
  • Default markers, test_issue_1220_isabelle_acquisition.py: 93 passed, 11 failed. The same 11 fail on this host with dev's code (macOS temporary-directory PermissionErrors, and Isabelle checks that support only Linux x86_64; tracked in Lock qualification and Isabelle acquisition tests fail on macOS #1536). CI runs them on Linux.
  • -m integration, all six modules: 18 passed. -m fuzz selects no test in the 1217 module.

Proof manifest checks on 578c6fec:

  • With the new digest, uv run --project implementations/python --frozen python tools/check_participant_opacity_proof.py gets past manifest validation and stops at the pinned Isabelle proof tool supports Linux x86_64 only on this Mac. With dev's manifest it stops at proof tool source digest does not match its repository source.
  • tests/test_issue_963_participant_opacity_proof.py -m integration: 35 passed, including the six that failed in CI.
  • The same module with default markers: 11 passed and 2 failed. Both failures are Linux-only checks that fail the same way on this host with dev's code.
  • The research evidence checkers still pass: tools/check_specification_coverage.py exits 0, and tools/check_formal_semantic_validation.py reports Formal semantic-validation evidence bundle passed integrity and replay checks.
  • nox -s contracts -- --base-rev origin/dev passed 14 stages.

On 578c6fec, nox -s verify-fast-feedback -- --base-rev origin/dev passed: every stage PASS or SKIP, its ruff format and check stages passed on the tool file and the test, and the changed test module ran (45 passed). nox -s lint passed. The pinned tooling ruff format and ruff check, run from implementations/python, pass on the test. make policy passed 15 stages. Requirement governance was skipped: the branch has no requirement UID, and no scope file has ever existed for #1475.

The changed source line is an argv element that every transfer test builds, so coverage does not change. The Bootstrap qualification workflow reruns the real-curl tests on Ubuntu x86_64, Ubuntu arm64 and macOS 15.

On 578c6fec, CI run 38056743267 passed and all 14 required checks pass (32 checks pass, deploy skipped), including canonical / proof and canonical / integration. Bootstrap qualification run 38056743136 passed: the real-curl selection passed on Ubuntu x86_64, Ubuntu arm64 and macOS 15 (8 passed each), and the dev-container smoke passed. SonarCloud's quality gate is OK with no new issues.

Ground Control Checks

  • Repository policy command passes
  • The real-curl tests fail with dev's argv against the ALPN fixture and pass with the pin

make policy and the test runs are recorded in the Test Plan.

Traceability

Checklist

  • Code follows the project coding standards
  • FM: not applicable, no semantic change (one curl argv element in repository tooling, and its tests); no executable or runtime adoption is claimed.
  • No package code, published schema or research capture changes. One evidence-bound file changes: the proof manifest's digest for the edited tool source, which the CI proof lane re-verifies by replay.
  • PR title is a Conventional Commit; build does not cut a release.

curl_transfer_argv set no HTTP version, so curl negotiated HTTP/2 with
the download hosts. Over HTTP/2, macOS /usr/bin/curl 8.7.1 exits 56
for a size-limit breach, which run_curl_transfer does not classify as
the enforced limit (63), and it does not retry a 503. Qualification
only exercised HTTP/1.1, because its http.server fixture offers no
ALPN protocol.

Pass --http1.1 in the shared argv. The fixture now offers h2 before
http/1.1 through ALPN, as the download hosts do, so a client that
accepted h2 fails the real-curl tests.
The participant-opacity proof manifest binds each source that acquires
the prover by digest, tools/maintained_client_acquisition.py included.
Record the digest of the file with the --http1.1 argv. The proof lane,
the participant-opacity proof tests and the dev-container smoke failed
on the old digest.
The Cambridge Isabelle mirror (www.cl.cam.ac.uk) negotiates only
http/1.1; the other download hosts in the tooling artifact lock offer h2.
@doublewhy
doublewhy marked this pull request as ready for review October 10, 2026 14:09
doublewhy added a commit that referenced this pull request Oct 12, 2026
…ter entry

Each open entry in the divergence register now names its issue: #1525
to #1557 for the production changes, and #1521 for the smaller
corrections. The entries that open pull requests now address move to
"Addressed by pull requests" with their issues: GH-7 (#1491, #1490),
NS-1 (#1476, #1475), NS-2 (#1480, #1479), MF-2 (#1483, #1482), CL-2
(#1474, #1473), ST-2 (#1472, #1559) and NS-7 and NS-8 (#1488, #1560
and #1561). ST-1 names #1558, GH-3 records that #1415 merged, and
"What remains" names #1522 for the unverified and unexamined
boundaries.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant