perf: parallelize native Python consumer checks #336

Summary

Speed up native la-stack Python consumer checks using the bounded pytest worker approach already used by research-repo-tools. Requested by the maintainer during review of #335; target v0.5.3. Preserve the complete test inventory, scientific assertions and required CI job/evidence identities.

Current state

The last successful main Windows job ran its 306 Python tests serially in 968.40 seconds. This is historical baseline evidence, not a prediction for the expanded current suite.

The current research-repo-tools pin already includes native Windows hardware queries without PowerShell/WMI subprocesses. Upstream wheel/sdist parallelization has no corresponding local package phase because la-stack's Python environment is not a distribution. Its pytest worker strategy is applicable here.

Implementation and acceptance

• Pin pytest-xdist in the existing development group and lock without changing the accepted research-repo-tools commit or unrelated dependency versions.
• Keep the existing test-python entry point and CI jobs. Run the full inventory with a bounded worker count and work-stealing scheduling; do not restart crashed workers or narrow test selection.
• Provide a documented serial override and record slow test durations.
• Inspect temporary paths, process environments, shared fixtures and installed-tool interactions for worker isolation. Preserve all existing test contracts; fix any exposed races without skipping or weakening tests.
• Test actual recipe forwarding for default/custom/serial worker selections and propagation of failures.
• Complete local Python/configuration/documentation validation and the existing native Linux/macOS/Windows checks before merge. Record actual timings and limitations; local concurrency is not proof of Windows speedup.
• Close through feat: resume and reuse verified release benchmark acquisitions #335's closing reference only after required review/CI and merge.

Related: #326 (workstream readiness), #333 (release adoption and final evidence). No benchmark sampling, release publication, new CI matrix or upstream dependency upgrade is required.

Local validation — 2026-11-07

Commit d809c06 in #335 passes just check, just python-ci, documentation validation, uv lock --check, Windows-targeted typing for changed Python source/tests, and git diff --check. All 376 Python tests passed in 758.45 seconds (12m 38s) with four workers. The prior local serial run passed 367 tests in 1999.26 seconds; the new run includes nine additional review/runner regressions, so this is a local observed comparison rather than a controlled Windows timing claim.

All original scientific/evidence tests remain. The dependency diff adds only pytest-xdist 3.8.0 and execnet 2.1.2. All checks on the previous PR head finished before commit d809c06 was pushed. Native Linux/macOS/Windows CI and release packaging passed; the Windows Python suite passed all 367 tests serially in 2863.35 seconds (47m 43s). Updated native validation remains incomplete: Windows passed, while Linux and macOS failed the Cargo-update integration test. #335 now includes Closes #336.

Native failure investigation — 2026-11-08 UTC

At d809c06, Windows CI passed, including all 376 Python tests in 2093.59 seconds (34m 53s), compared with 367 tests in 2863.35 seconds (47m 43s) on the prior serial head. These are observed CI runs with different test counts, not a controlled benchmark.

Linux and macOS each passed 375 tests and failed test_native_updates_advance_eligible_dependencies_and_preserve_coupled_requirements[update-cargo-dependencies]. The captured subprocess diagnostic was omitted by the exception report, so the underlying cause is not established.

Diagnostic commit 3676349, now pushed with maintainer authorization, retains the zero-exit requirement and prints captured stdout/stderr in its assertion. With that diagnostic change, all 376 local tests passed again in 759.69 seconds; just check, Windows-targeted typing, and git diff --check passed. Focused concurrent runs, Python 3.15.2, CI environment flags, a cold registry cache, and recycled registry-address probes did not reproduce the failure. No retry/skip/assertion weakening has been added. Native confirmation of a root-cause fix remains outstanding. Diagnostic CI passed on all three platforms and release packaging. On this diagnostic-only head, Linux passed 376 Python tests in 1547.68 seconds, macOS in 1472.54 seconds, and Windows in 2104.63 seconds. These passing runs alone do not establish a fix. The maintainer subsequently authorized pushing the fix after all local validation passed.

Reproduced connection race and local fix — 2026-11-08 UTC

The fixture serves HTTP/1.0 responses. After a bodyless 304 Not Modified, cargo-edit 0.13.14 can pool the connection before the server closes it and attempt the next index lookup on that retiring connection. Holding the server close until the client either closes or sends another request reproduces Error: io: Peer disconnected and recipe exit 1 immediately after the first 304. This matches the visible request/exit pattern in the original native failures; their hidden subprocess stderr cannot be recovered retrospectively.

Commit 9376250, pushed after the maintainer-authorized local validation, explicitly sends Connection: close for registry responses. Both actual update recipes retain all success/dependency assertions and now run with normal closure and with the retiring-connection ordering forced using socket readiness (no sleep). All four cases pass. Negative control: removing only the header makes the new regression fail in 1.50 seconds with Peer disconnected; restoring it passes. Windows-targeted typing passes. Final local just check and just python-ci passed: all 378 tests passed in 965.70 seconds. Windows-targeted typing and git diff --check also passed. Native CI for the fix and the current-head review are pending; standard auto-merge still requires their gates.

The independently reported upstream defect is ureq #1224, matching the ureq 3.5.0 / ureq-proto 0.6.5 versions in cargo-edit 0.13.14. #337 tracks removal once a published cargo-edit contains the fix, with the race regression retained.

Observed full Windows CI job duration decreased from 61m 30s on the serial head to 49m 38s on the diagnostic parallel head, approximately 19% shorter. Its Python portion decreased from 2863.35 seconds (367 tests) to 2104.63 seconds (376 tests), approximately 27% shorter. These remain comparisons of observed runs, not controlled benchmarks.