Investigate CI test speedups with baseline evidence and cost/fidelity gates before adoption

Objective

Reduce the elapsed time of Netsuke's CI test gate without increasing Ubicloud expenditure or weakening test fidelity. Establish a current baseline, run controlled experiments, and publish the CI evidence before adopting any optimization.

This is an evidence-led investigation, not authorization to implement every proposed change. A candidate may be rejected or deferred. Do not create additional paid benchmark jobs, enlarge runners, or enable an optimization by default merely because its isolated microbenchmark looks promising.

Verified starting point, not a performance baseline

Source inspection on 10 October 2026 used main at 24293a1b3616872100d379e277540bb1df89203c. Revalidate these details against the implementation base before beginning.

The Linux merge gate uses ubicloud-standard-4-ubuntu-2404 for eligible runs, with four Cargo jobs and four Nextest workers. Fork PRs use a separate GitHub-hosted fallback. The PR lane combines test execution and coverage, passes all-features, all-targets and doctests, and pins generate-coverage to 7d751eded578e48dacf43ab173d5820e6437c474.
The cache action hashes the complete ci.yml and cache-action file into pins_hash. Both installed-tool and Whitaker cache keys depend on it. Only trusted trunk pushes publish these archives. The action deliberately does not archive target/; compiler output belongs to sccache.
The public-API UI harness has eight direct-rustc fixture tests which each call NetsukeRlib::build(). Each call invokes Cargo and discovers the rlib and dependency search paths through Cargo's JSON messages. The verbose-timing fixture also checks that compilation fails without --extern netsuke. The separate cli_configuration_fixture_compiles test exercises an actual Cargo consumer project.
Nextest configuration already serializes build-capable child-Cargo tests in nested-cargo-builds, with anchored filters. Its comments record a historical 452.4-second PR Nextest run and a further 274 seconds for profile merging/report generation on run 34914144521.

Those September figures are historical observations recorded in the repository, not measurements of today's gate and not an expected saving. Do not reuse the earlier unvalidated 8m07 figure as a baseline. Repeated Cargo calls do not imply eight cold builds: identify freshness checks, real recompilation and lock waits separately.

Related work: #732 is closed/completed and addressed earlier nested-build inefficiencies. Preserve its feature-alignment improvements rather than reopening the same fix. #693 covers a distinct split-build-dir/Windows fidelity question. This investigation does not authorize reducing that test's scope or replacing real consumers with weaker stand-ins.

Non-negotiable constraints

Keep the existing Ubicloud runner shape, job topology, build-worker limits and test-worker limits. No extra sharding, matrix expansion, scheduled benchmark workflow, repeated manual dispatches, or reruns solely to grow the sample. Do not move work to another paid lane and call that a saving.
Use existing logs first. Collect additional evidence through normal, already-required CI executions. Before enabling experimental work, record its overhead and how it fits the existing budget. If convincing evidence cannot be collected within that envelope, leave the candidate unadopted and document the limitation.
Preserve the complete required test inventory and its selection: unit, integration, behavioural, public-API compilation, real-Ninja, packaging, workflow-contract and other existing gates, plus the separate doctest pass. Preserve feature/target/workspace scope, assertions, negative controls, property-test budgets, isolation, and failure propagation. Keep mutation/Kani and other independent workflows unchanged.
Keep the existing coverage instrumentation, full detailed report, exclusions, denominator and ratchet semantics. No summary-only substitution, silently ignored malformed profiles, discarded subprocess coverage, new exclusions, or removal of uncovered files to improve either time or percentage.
Preserve toolchain pins, warnings-as-errors, coverage-specific compiler/linker settings, trusted single-writer cache policy, and existing timeout ordering. No blanket retries, skipped slow tests, lowered test counts, or timeout increases as a performance fix.
Do not pursue a restore-and-delete optimization for target/: that archive does not exist in this cache design. Do not add one as part of this work.

Phase 1: establish the baseline and evidence format

Reproducible provenance

For every observation, retain the run URL, job URL/ID, run attempt, event type, source and tested merge SHA, experiment variant, relevant diff, runner label/image/architecture and CPU/memory capacity. Record the Rust/Cargo/LLVM/Nextest/cargo-llvm-cov versions, shared-action SHAs, feature/target selections, effective compiler flags, profile and concurrency limits.

Record the effective cache keys and exact/fallback/miss results separately for Cargo downloads, installed tools, Whitaker and sccache. Capture sccache hit/miss and compilation statistics rather than treating all restored runs as equally warm.

Keep PR coverage, trunk coverage, trunk cache-writer jobs and GitHub-hosted fork runs in separate cohorts. Trace the pinned shared coverage action and its scripts to confirm actual commands and phase boundaries. Record missing evidence explicitly instead of reconstructing timings from assumptions.

Measure the whole paid job and the components

Use the existing resource sampler and cache observations where possible. Add only bounded, tested telemetry needed to distinguish:

Component	Required evidence
Queue and provisioning	Record separately from execution; do not count reduced queue time as reduced paid compute without billing evidence
Cache restore and tool installation	Per-step elapsed time, hit/miss state, keys, archive sizes where available, and whether installers actually did work
Compilation	Instrumented build time, real compiler invocations/freshness, compiler-cache behaviour, and nested Cargo preparation time
Test execution	Nextest execution time, complete selected/executed test identity sets, per-test durations, failures/skips/ignored tests/retries and slow-test tail
Nested-build scheduling	Number and duration of Cargo invocations, build-capable group occupancy, lock-wait evidence where available, and the exclusive tail on the critical path
Coverage processing	Raw-profile count/bytes, merge time, export time, report parsing/ratchet time and profile diagnostics
Doctests	Separate duration, selection, counts and outcome
Whole job	Start-to-finish execution including setup, telemetry, uploads, cleanup and post-job cache work; peak memory, CPU use and available I/O indicators

Do not add together overlapping per-test durations and describe the result as wall-clock savings. Include shared preparation outside Nextest's own timer in the primary end-to-end measurement.

Prefer native timings or lightweight instrumentation of the pinned action. Splitting coverage into execution and reporting merely to obtain timings must not execute the tests twice, clean profiles prematurely, omit objects, or change failure semantics. Apply equivalent instrumentation to control and candidate.

Important confound: editing ci.yml to add telemetry currently changes the broad installed-tool keys. Distinguish that instrumentation-induced miss and any subsequent cache warm-up from both baseline performance and the optimization's effect. Do not purge production caches or provoke cold builds on Ubicloud for benchmarking.

Sampling and analysis plan

Before evaluating candidates, commit an analysis plan with a primary endpoint, minimum useful improvement, non-regression checks and sample/exclusion rules. Use whole-job duration and total chargeable consumption as primary outcomes; phase timings explain the mechanism.

Aim initially for at least five comparable control observations and five candidate observations per evaluated condition, accumulated from ordinary runs. This is a collection floor, not automatic proof. Continue observing naturally when noise or sample size prevents a reliable conclusion; do not buy extra runs to meet it.

Keep application source, dependency lockfile, toolchain, test inventory and relevant runner/cache conditions equivalent, apart from the candidate and identical telemetry. Where normal development prevents matching, label the comparison observational and explain the confound. A neighbouring PR with different tests or a different toolchain is not a controlled baseline.

Publish individual observations, medians and spread, absolute and percentage differences, and uncertainty appropriate to the sample. Do not present a p95 from a handful of runs as a reliable tail estimate. Separate naturally warm and cold/miss observations; never compare a cold control with a warm candidate as proof of improvement. Retain failed, cancelled, timed-out and retried attempts in the cost/reliability accounting even when they cannot enter successful-run timing comparisons.

Phase 2: test the candidates independently

Experiment A: installation-input cache fingerprints

Hypothesis: unrelated workflow edits unnecessarily invalidate expensive installed-tool caches. Narrowing the fingerprints may reduce setup time on CI-changing PRs without changing any executed checks.

Inventory every actual installation input, including values sourced indirectly through Makefiles/configuration, tool versions, action/installer revisions, checksums, Python environments, relevant toolchain/ABI settings and runner compatibility.
Derive explicit compatibility fingerprints, separately for general tools and Whitaker where appropriate. Preserve version checks and the existing trusted writer/restore boundaries. Do not merely remove ci.yml from the hash and hope the remaining inputs suffice.
Add mutation/contract tests proving that a relevant pin, installer, ABI, runner-image or suite input invalidates the correct cache; an unrelated comment/test-step edit does not; and restore/save render identical keys. Retain separate Cargo-download keying.
Measure cache restore plus installation time and whole-job time, not cache-hit percentage alone. Evaluate the source-only warm path and naturally occurring CI-edit cases separately.
Account for one-off key-schema migration/cold-start cost and how an authorized trunk run populates the candidate generation. Do not let a permanently cold PR-only key masquerade as a fair trial, weaken the single-writer policy, or perform redundant paid population runs.

This is a setup-time candidate, not a claim that the test executable itself will become faster.

Experiment B: prepare public-API compilation artifacts once per run

Hypothesis: the eight direct-rustc fixtures can retain independent compilation/assertions while sharing one Cargo build/artifact-discovery preparation.

Prototype a run-scoped manifest containing the exact rlib and dependency directories resolved from Cargo JSON. Prepare it once under the same toolchain, manifest, feature selection, target/build layout and effective instrumentation environment as the existing helper.
Preserve eight separately identifiable fixture tests and their individual rustc invocations, the verbose-timing negative control, temporary-output isolation, complete dependency search paths and response-file handling. Preserve the real Cargo-project boundary in cli_configuration_fixture_compiles.
Nextest's process-per-test model means a process-local OnceLock is not sufficient. Evaluate a scoped setup script or equivalent run-scoped preparation mechanism. Nextest setup scripts currently require experimental opt-in; using them needs an explicit compatibility/maintenance decision rather than an incidental dependency.
Scope data to the current invocation. Never reuse another run's manifest or guess the newest rlib. Detect stale/mismatched/missing/malformed artifacts, propagate preparation failures, and avoid overwritten uplifted rlibs/version skew when other nested builds run. Keep a correct fallback for individual local cargo test invocations and existing supported platforms.
Keep current serialization for the first experiment. Any subsequent change to group membership is a separate, measured variant justified by the remaining resource usage and race analysis; preserve anchored filter contracts.
Add focused tests for one preparation serving all relevant tests, zero unnecessary preparation when none are selected, error propagation, invalid manifests, configuration mismatch, split layouts and long response-file paths.
Measure the number of preparations, Cargo freshness/rebuild work, total preparation-plus-fixture duration, serialized tail, whole-job time and peak memory. Reject an approach that merely moves the same cost outside Nextest's reported duration or adds another build configuration.

Experiment C: reduce coverage merge/export overhead, only if the baseline confirms it

Hypothesis: profile-file proliferation or reporting work may explain a substantial post-test tail. The historical 274 seconds identifies a phase to investigate, not a proven fix.

First measure the actual LLVM_PROFILE_FILE pattern, raw-profile inventory, binary/object inventory and separate merge/export timings. Establish whether reporting is still material before changing it.
If raw-profile proliferation is the cause, test a small, predeclared, sequential set of bounded LLVM merge-pool configurations on the existing runner. LLVM documents %Nm pools, but validate support and behaviour in the repository's pinned runtime/tooling. Inspect the entire filename pattern: a per-process component can defeat cross-process pooling.
Preserve run isolation and binary identity; validate collision avoidance, profile collection paths, subprocess contributions, locking and crash/error handling. Do not assume %b is effective without the necessary build-ID support, or silently change the coverage linker settings to obtain it.
Measure any work shifted into test-process exit and any lock contention, as well as reduced report time. A faster merge with equally slower tests is not a win.
Retain full LCOV records and identical report scope. Compare normalized source/file/function/line/branch records where currently emitted, including uncovered entries. Use deterministic controls to verify exact counter accumulation across repeated/concurrent processes. Investigate application-suite counter differences against documented natural nondeterminism; equal headline percentages alone do not establish fidelity.
Do not assume llvm-profdata -j 4 introduces parallelism: LLVM already supports automatic thread selection. Record the actual invocation before proposing a thread override. Keep aggregate work within the same resource limits.

Do not sweep a large parameter matrix. Rule out incompatible or unpromising configurations with cheap local checks first, then evaluate only justified candidates through normal CI.

Phase 3: evidence review and adoption decision

Keep each candidate independently reversible and opt-in during its evaluation. Instrumentation may land separately, but default behaviour must remain unchanged until the relevant evidence review accepts the candidate. Experimental activation must select one implementation for an existing run, not execute both full suites in the same paid job.

For each candidate, publish this decision record in the issue and link it from the adoption PR:

Field	Required content
Hypothesis and variant	Exact change, base/candidate SHAs and predeclared success criterion
Cohort	Run/job links, attempts, cache states, versions, source/test comparability and sample size
Effect	Whole-job and target-phase baseline/candidate timings, absolute/relative change and uncertainty
Cost	Total chargeable consumption across affected jobs, including failed/cancelled attempts, setup/reporting and telemetry; migration cost separately
Fidelity	Test-identity diff, individual outcomes and doctest evidence, negative controls, detailed coverage comparison and relevant contract checks
Resources/reliability	Peak-memory and CPU/I/O observations, timeout/flake/retry changes and confounders
Decision	Adopt, reject or inconclusive, with rationale and an exact rollback path

Use billing data when available. Otherwise label runner-seconds on the unchanged shape as a cost proxy, not verified invoice savings, and document the limitation. Include any new storage/transfer footprint. A statistically inconclusive increase is not evidence that cost remained unchanged.

Adoption requires a repeatable, useful improvement beyond observed noise in the intended cohort, no end-to-end regression elsewhere, no increased Ubicloud consumption, and no loss of test or coverage fidelity. A cold-cache setup win must not hide a warm-path regression. Reject candidates whose complexity, uncertainty or overhead outweighs the demonstrated benefit.

If several candidates pass independently, check their combination through normal CI before enabling the combination by default. Validate affected shared harnesses on the existing Windows/macOS lanes without adding jobs. Any necessary shared-actions change must retain backwards-compatible defaults for other consumers and use a reviewed immutable pin.

After adoption, keep lightweight regression telemetry for a documented window of ordinary runs, publish the observed follow-through here, and revert on fidelity loss, resource/cost regression or disappearance of the claimed benefit. Do not close the issue on implementation alone.

Acceptance checklist

No CI runs were dispatched and no implementation or workflow changes were made when filing this issue.

@CodeRabbit

@CodeRabbit plan

Coding Plan

Summary

Revalidate first. Fix the endpoints, cohorts, statistics and cost envelope in a living ExecPlan before any candidate data exists. Use existing logs before adding anything.
Add one bounded, contract-tested telemetry PR: Nextest JUnit, a pre-cleanup coverage inventory, a timestamped sampler, matched-key reporting and nested-Cargo lines. Account for its single cache-key rotation explicitly.
Evaluate each candidate in an opt-in, reversible form inside ordinary runs: A as zero-cost shadow v2 keys with completeness and mutation contracts, B as validated lazy preparation keyed by NEXTEST_RUN_ID, and C only if the baseline confirms a material merge and export tail.
Adopt only through published decision records, combination checks, ADRs and a follow-through window. Keep inconclusive candidates disabled.

Design Choices

Design Choice 1: Where to keep the analysis plan, observations and decision records

Options Considered:

Post the evidence only in the issue.
Keep a living ExecPlan in docs/execplans/, a normalized observation dataset in docs/ci-performance/, and mirror each decision record in the issue.
Use a new ADR for every intermediate result.

Chosen Option: 2

Rationale: This follows the ExecPlan convention, keeps provenance under review, and still publishes each decision in the issue. Use ADRs only for adopted architectural changes.

Design Choice 2: How to separate coverage phases inside the single `cargo llvm-cov nextest` command

Options Considered:

Split coverage into separate test and report invocations.
Fork the shared action to add timers.
Use GitHub log line timestamps at native markers: the Nextest summary line, the cargo-llvm-cov verbose Running lines if they can be enabled, the doctest start and the step end. Combine these with job and step timestamps from the API.

Chosen Option: 3

Rationale: This adds no execution, keeps failure semantics, and applies the same way to control and candidate. If verbose Running lines cannot be enabled without a shared-action change, report merge and export as one post-test phase and record that limitation.

Design Choice 3: How to evaluate candidate A without a cold PR-only key or extra trunk population runs

Options Considered:

Switch to v2 keys on PRs only.
Have trunk save v1 and v2 generations in parallel.
Render a shadow v2 key in every run without restoring or saving it. Compute counterfactual hits against the v2 keys from trunk writer runs. Take hit and miss setup costs from the observed v1 cohorts.

Chosen Option: 3

Rationale: This adds no paid work or storage and keeps the single-writer policy. It gives hit-rate evidence. Confirm the effect after adoption through the follow-through window.

Design Choice 4: Source of the candidate A fingerprint inputs

Options Considered:

Compose a fingerprint string in the with: blocks of ci.yml. This duplicates it across the restore and save calls.
Parse a projection of ci.yml at key-render time. This needs YAML tooling before any installation.
Commit per-cache fingerprint manifests that both modes read. Make a contract test derive the expected pin values from ci.yml, Makefile and the installer files. Make it fail on any drift or omission.

Chosen Option: 3

Rationale: Restore and save then read one file by construction. The contract test also proves the manifest is complete. This extends the existing python_toolchain_sync_test.py sync pattern.

Design Choice 5: Run-scoped preparation mechanism for candidate B

Options Considered:

Use a Nextest setup script. This needs experimental = ["setup-scripts"] and a cross-platform helper. It also runs serially before tests, which blocks all four workers. It must also work on both 0.9.133 and the coverage action's 0.9.145.
Use a process-local OnceLock. This is ineffective under process-per-test.
Use lazy preparation inside the first grouped consumer. Key it by NEXTEST_RUN_ID, guard it with a file lock in a run-scoped directory, and validate it with a provenance manifest. Fall back to today's per-test build when NEXTEST_RUN_ID or the opt-in flag is absent.

Chosen Option: 3

Rationale: This needs no experimental Nextest feature, does no preparation when no consumer is selected, keeps preparation inside the one-slot group, and works on all platforms. Record the setup-script compatibility decision as "rejected for the first variant" in the decision record.

Design Choice 6: Opt-in activation during evaluation

Options Considered:

Edit ci.yml per trial.
Use a repository variable. This affects all PRs at once and confounds time with treatment.
Use per-PR labels such as ci-experiment:public-api-prep and ci-experiment:profile-pool. Read them only in the same-repository PR coverage step. Set one environment flag for each label.

Chosen Option: 3

Rationale: Labels select one implementation for an existing ordinary run. They do not trigger runs under the current pull_request event types, and they never reach trunk or coverage-main.yml.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Revalidation, analysis plan and evidence tooling

Fix the reference state and the rules for evidence before any change to a workflow. Keep all work in this phase to documentation and offline tooling. Add no CI cost.

Preserve the planning context: AGENTS.md requires living ExecPlans and ADRs. The ticket requires publication in the issue. The pinned coverage action has no separate merge or export step. cargo-llvm-cov 0.9.0 has no timers.

Task 1: Task 1.1: Publish the revalidation record

Record the verified current state against the ticket's reference SHA 24293a1b….

Create the living ExecPlan docs/execplans/ci-test-speedups-investigation.md. Use the status and section format in docs/documentation-style-guide.md.
Record the runner topology, cohorts (same-repository PR, fork PR, trunk build-test, trunk coverage-main, trunk cache writer), worker limits, timeout hierarchy and action pins from the exploration.
Record the discrepancies found:
- Coverage enforces Nextest 0.9.145, while the tool manifest says 0.9.143 and the workflow says 0.9.133.
- The profile pattern contains a per-process %p.
- The Linux report has no cache-matched-key.
- test_support/dylint.toml is not hashed.
- The mold, Makefile and pyproject.toml inputs are absent from the keys.
- The docs/developers-guide.md lines on coverage without Nextest and on an old action pin are stale.
Record that #732 is complete and that #693 closed via #736 as a Windows split-build-dir question. Mark both as out of scope for change.
Label every September figure as historical context. Do not use any of them as a baseline.

Task 2: Task 1.2: Commit the analysis plan and cost envelope

Predeclare endpoints and rules before any candidate data exists.

In the ExecPlan, set two primary endpoints:
- Whole-job elapsed time (job started_at to completed_at, including post-job steps) for the same-repository PR build-test job.
- Summed runner-seconds across the affected jobs for each run attempt. Label this as a cost proxy, because no Ubicloud billing data is available.
Record queue and provisioning time separately. Never count it as a compute saving.
Predeclare the minimum useful improvement as the larger of a fixed absolute floor (propose 60 seconds) and the control interquartile range. Fix the numbers before candidate evaluation.
Predeclare these non-regression checks:
- Identical executed test-identity set and outcomes.
- Identical doctest count and outcome.
- A normalized LCOV comparison.
- Peak memory no higher than the control range.
- No new timeouts or retries.
- Coverage percentage within the ratchet band.
Set the sample rules. Use at least five control and five candidate observations per condition, from ordinary runs only. Identify each observation by run ID plus attempt. Use these cache cohorts: exact hit, prefix restore, miss, and instrumentation-induced rotation. Exclude cancelled and superseded attempts from timing comparisons. Keep them in the cost and reliability accounting.
Set the statistics: individual values, median, range, and a distribution-free shift estimate with a permutation interval. Do not report p95 from small samples.
Record the cost envelope for telemetry. Measure its step durations and artefact bytes. Confirm that it adds nothing inside the 1800-second Cargo watchdog.

Task 3: Task 1.3: Build the offline evidence collector

Turn GitHub API exports, logs and artefacts into normalized observations. Keep missing values explicit.

Create scripts/ci_test_evidence.py. Follow the pattern of scripts/release_operation_observability.py: fixed labels, None for missing or backwards timestamps, and no secrets or raw errors.
Ingest saved JSON from gh api …/runs/{id}/attempts/{n}/jobs and the job logs. Also ingest the telemetry artefacts from Phase 2 when they are present.
Emit one record per job attempt with these fields:
- Provenance: run and job URL, attempt, event, head and merge SHA, variant label, runner label, image and architecture.
- Versions taken from the logs.
- Per-step durations.
- Cache keys and outcomes.
- sccache statistics.
- Phase timestamps from native log markers.
- Doctest counts.
- Test outcome counts.
Add tests in tests/workflow_contracts/ci_test_evidence_test.py for parsing, missing-evidence handling, attempt separation and cohort classification.
Store normalized observations in docs/ci-performance/ as JSONL with a short README. Do not commit raw logs.

Task 4: Task 1.4: Harvest the pre-telemetry baseline from existing logs

Use the evidence that already exists before any workflow edit.

Collect recent ordinary runs of ci.yml and coverage-main.yml on the current base, within log retention. Classify them by cohort.
From the step summaries, extract tools and Whitaker hit or miss status, the sccache text statistics and the resource aggregates. Also measure how often PR runs miss the tools cache because ci.yml or action.yml changed. Use this frequency to define how large candidate A's opportunity can be.
From the log timestamps, extract the Nextest summary time, the post-test tail before the doctests, and the doctest duration.
Record each unavailable field as missing. This includes the matched Cargo fallback key, the profile inventory, per-test durations and process RSS.

🤖 Prompt for AI agents

Implement documentation and offline evidence tooling to establish the reference
state and baseline rules. Do not edit workflows or trigger CI runs in this
phase.

- Create `docs/execplans/ci-test-speedups-investigation.md` using
`docs/documentation-style-guide.md`. Record the reference SHA `24293a1b…`,
topology, cohorts, limits, timeout hierarchy, action pins, discrepancies and
out-of-scope issues. Treat September figures as historical context only.
- Predeclare whole-job elapsed time and summed runner-seconds per attempt as
primary endpoints. Label runner-seconds as a cost proxy. Separate queue and
provisioning time.
- Fix the useful-improvement threshold before evaluation: the larger of the
proposed 60-second floor and control interquartile range. Record fidelity,
memory, timeout, retry and ratchet checks.
- Require at least five control and five candidate observations per condition
from ordinary runs. Separate cache cohorts and run attempts. Exclude cancelled
and superseded attempts only from timing comparisons, not cost or reliability
accounting.
- Report individual values, median, range and a distribution-free shift estimate
with a permutation interval. Do not report small-sample p95. Define the
telemetry cost envelope, including durations and artefact bytes, with no
additions inside the Cargo watchdog.
- Implement `scripts/ci_test_evidence.py` following
`scripts/release_operation_observability.py`. Use fixed labels and `None` for
missing or backwards timestamps. Exclude secrets and raw errors.
- Parse saved attempt-job API JSON, logs and optional telemetry artefacts. Emit
provenance, versions, durations, cache evidence, sccache statistics, native
phase timestamps, doctest counts and test outcomes per job attempt. Classify the
specified cohorts.
- Test parsing, missing evidence, attempt separation and cohort classification
in `tests/workflow_contracts/ci_test_evidence_test.py`. Store JSONL and a README
in `docs/ci-performance/`; do not commit raw logs.
- Harvest recent retained ordinary-run logs before workflow edits. Extract
existing cache, sccache, resource and phase evidence. Measure tools-cache misses
caused by workflow or action edits. Keep unavailable matched keys, inventories,
per-test durations and RSS explicitly missing.

Phase 2: Bounded baseline telemetry

Land one telemetry PR so that the instrumentation-induced key rotation happens once. Apply identical telemetry to control and candidate runs. Do not change default execution.

Task 1: Task 2.1: Emit structured Nextest results

Capture executed test identities, per-test durations, outcomes and retries natively.

In .config/nextest.toml, enable JUnit output for profile ci with no stored success output. Do not change timeouts, groups or selection.
Confirm locally where JUnit lands under cargo-llvm-cov's target directory. Also confirm whether testcase timestamps are emitted, because group-occupancy reconstruction needs them.
Update tests/workflow_contracts/whole_run_profile_test.py and timeout_ordering_test.py so that they accept the JUnit stanza and still assert unchanged budgets.

Task 2: Task 2.2: Capture coverage inventories and upload a telemetry bundle

Record profile, object and version evidence before cleanup deletes it.

Add an always() step to .github/workflows/ci.yml (PR coverage) and .github/workflows/coverage-main.yml. Place it after the coverage step and before the rm … target/llvm-cov-target target/llvm-cov cleanup.
Make that step call a new tested script in .github/scripts/. Make the script write JSON with the .profraw count, total bytes and size distribution, the .profdata size, the instrumented object count, the rustc -vV output (including the LLVM version) and the copied JUnit file.
Upload a telemetry bundle with the pinned upload-artifact action. Follow the pattern in .github/workflows/release.yml: always(), missing-file tolerance, and a fixed short retention. Include the inventory JSON, JUnit, the sccache JSON and text, and the timestamped resource log.
Check the v0.9.0 source for an environment-based verbose switch that emits Running lines for merge and export. If one exists, set it identically in both coverage steps. If none exists, record the combined post-test phase as a limitation.

Task 3: Task 2.3: Extend resource and cache observations

Close the attribution gaps in the existing telemetry.

Update .github/actions/memory-sampler/action.yml so that each sample line has an epoch timestamp. Keep the aggregates unchanged.
Move the sampler stop to just before the telemetry upload, so that the sample window includes cache saves and CodeScene work. Record the remaining post-job hooks as outside the window.
Update the observation step in .github/actions/linux-gate-cache/action.yml so that it reports cache-matched-key for the Cargo-download restore. Separate exact hit, prefix restore and miss states.
Record in the ExecPlan that this PR rotates the tools and Whitaker keys. Label the runs after it as instrumentation-induced misses until trunk has repopulated the cache.

Task 4: Task 2.4: Add nested Cargo telemetry to the existing harness

Make freshness, real rebuilds and lock waits visible for the control path.

Update NetsukeRlib::build in tests/command_env_ui_tests.rs and the analogous preparation in tests/support/test_support_rlib.rs to print one fixed-format line per Cargo invocation. Include the test name, elapsed time, the counts of fresh and non-fresh compiler-artefact messages, and whether stderr contains Cargo's "Blocking waiting for file lock" text.
Count freshness in a small helper next to tests/support/cargo_artifacts.rs. Do not change the existing selection API.
Rely on the existing immediate success output of the group to put these lines in the logs. Do not change assertions, the negative control or isolation.

Task 5: Task 2.5: Contract-test the telemetry and accumulate the baseline

Keep the telemetry bounded. Ensure that it cannot change gate semantics.

Add contracts in tests/workflow_contracts/. Require the inventory and upload steps to be always() and to sit before cleanup. Require upload to tolerate missing files. Ensure that telemetry does not alter the coverage inputs, RUSTFLAGS, the watchdog or the job ceiling. Require the sampler to log timestamps. Require the Linux observation to report the matched key.
Run make test-workflow-contracts and the affected Rust tests locally.
Accumulate ordinary-run observations through the collector until each cohort meets the floor. Then publish the baseline table, the phase attribution, the test inventory and the cache and resource evidence in the ExecPlan and the issue.

🤖 Prompt for AI agents

Implement one bounded telemetry PR to establish a comparable baseline for
control and candidate runs. Preserve default execution and all gate semantics.

- Enable JUnit for Nextest profile `ci` without stored success output. Verify
the cargo-llvm-cov target location and whether testcase timestamps support
group-occupancy reconstruction. Preserve timeouts, groups, selection and
budgets.
- Add tested inventory tooling in `.github/scripts/`. Call it with `always()`
after coverage and before target cleanup in PR coverage and `coverage-main.yml`.
Record `.profraw` count, bytes and size distribution, `.profdata` size,
instrumented object count, `rustc -vV` including LLVM, and copied JUnit.
- Upload inventory JSON, JUnit, sccache JSON and text, and the resource log with
the pinned `upload-artifact` action. Follow `release.yml` for `always()`,
missing-file tolerance and fixed short retention.
- Check cargo-llvm-cov 0.9.0 for an environment-based verbose switch. If
available, enable it identically in both coverage steps. Otherwise document
merge and export as a combined post-test phase.
- Add epoch timestamps to memory samples without changing aggregates. Stop
sampling just before upload to include cache saves and CodeScene. Document
post-job hooks outside the sample window.
- Report Cargo-download `cache-matched-key` in the Linux observation step.
Separate exact hits, prefix restores and misses. Document the tools and Whitaker
key rotation and mark misses as instrumentation-induced until trunk repopulates
them.
- Add fixed-format per-Cargo lines to `NetsukeRlib::build` and
`tests/support/test_support_rlib.rs`. Include test name, elapsed time, fresh and
non-fresh compiler-artefact counts, and the Cargo lock-wait marker. Add a small
freshness helper next to `cargo_artifacts.rs`; preserve the selection API,
assertions, negative control and isolation.
- Update profile and timeout contracts. Add contracts for step ordering,
`always()`, missing-file tolerance, timestamps and matched-key reporting.
Preserve coverage inputs, `RUSTFLAGS`, watchdog and job ceiling.
- Run `make test-workflow-contracts` and affected Rust tests locally. Collect
ordinary runs until cohort floors are met. Publish the baseline and attribution
evidence in the ExecPlan and issue.

Phase 3: Experiment A – installation-input cache fingerprints

Derive explicit compatibility fingerprints and evaluate them as shadow keys. Keep live keys unchanged until adoption.

Preserve the fingerprint-input constraint: installer uses: SHAs cannot be referenced as expressions. Make the restore call and save call render identical keys.

Task 1: Task 3.1: Inventory inputs and commit fingerprint manifests

Make every real installation input explicit.

Add tools and whitaker fingerprint manifests under .github/actions/linux-gate-cache/.
List these inputs in the general-tools manifest:
- The lane and the toolchain.
- The values of NEXTEST_VERSION, Ruff, ty, Python baseline, mdtablefix, yamllint, and actionlint (version, SHA256 and installer commit).
- The SHAs of setup-rust, install-action, setup-uv, the mdtablefix installer and the Nixie installer.
- The Makefile tool pins: interrogate, pylint, the typos builder, the df12 Git refs, the CV-005 commit, and the pytest and pytest-cov pins.
- Hashes of tools/mold/VERSION, tools/mold/SHA256SUMS, scripts/install-build-tools.sh and scripts/build-tools-common.sh.
- A schema version.
List these inputs in the Whitaker manifest: the lane, the toolchain, the installer action SHA, installer-version, and hashes of the root dylint.toml and test_support/dylint.toml.
Record the known residual gaps without hiding them: the unversioned df12 pylint, the unverified Whitaker suite revision and the upstream installer behaviour. Keep the installers' own version and checksum checks unchanged.

Task 2: Task 3.2: Extract a tested key renderer with a shadow v2 key

Prove restore and save symmetry by construction. Keep live behaviour unchanged.

Move the inline "Render cache keys" logic of .github/actions/linux-gate-cache/action.yml into a script next to the action. Keep the v1 keys byte-identical and keep Cargo-download keying separate.
Render the shadow keys netsuke-tools-v2-… and netsuke-whitaker-v2-… from the manifests. Write them to the step summary and the telemetry bundle only. Do not restore or save them.
Keep the writer condition github.event_name == 'push' && github.ref == 'refs/heads/main' and the per-save hit gating unchanged.
If this change is ready, land it together with the Phase 2 telemetry PR so that only one key rotation happens.

Task 3: Task 3.3: Add mutation and completeness contracts

Show what invalidates each cache and what does not.

Add tests/workflow_contracts/linux_cache_fingerprint_test.py with these tests:
- Restore mode and save mode render identical keys.
- Mutating each manifest input (pin, installer SHA, toolchain, lane or runner image, mold file, either dylint.toml) changes the correct v2 key and only that key.
- A comment or unrelated test-step edit in a ci.yml fixture leaves the v2 keys unchanged.
- Every installer step pin in ci.yml and every Makefile tool pin matches a manifest value, so adding a new installer without a fingerprint fails.
Keep cache_ownership_test.py and cache_write_policy_test.py passing without loosening them.

Task 4: Task 3.4: Evaluate counterfactual hits and setup cost

Measure whole-job effects, not hit rate alone.

Extend the collector. Count a PR run's v2 key as a counterfactual hit if an earlier successful trunk writer run rendered the same v2 key within the cache retention window.
Estimate the saving as the observed difference in tools and Whitaker restore-plus-install duration and whole-job time between the v1 miss cohort and the v1 hit cohort on PRs. Report source-only PRs and CI-editing PRs separately.
Estimate the one-off migration cost: one cold trunk installation, plus the PR misses before the next ordinary trunk push.
Write the decision record. If adoption is chosen, switch the live key to v2 in the adoption PR, add an ADR, and give an exact rollback: revert to the v1 key rendering.

🤖 Prompt for AI agents

Implement candidate A as explicit installation-input fingerprints and shadow v2
keys. Evaluate counterfactual hits without changing live cache behaviour or
adding paid work or storage.

- Add tools and Whitaker manifests under `.github/actions/linux-gate-cache/`.
Include the lane, toolchain, specified tool versions and installer SHAs,
Makefile pins, mold and build-tool file hashes, and schema version. Include
Whitaker installer version and both `dylint.toml` hashes.
- Document the unversioned df12 `pylint`, unverified Whitaker suite revision and
upstream installer behaviour. Preserve installer version and checksum checks.
- Extract key rendering into a script next to the action. Make restore and save
read the same manifests. Keep v1 keys byte-identical and Cargo-download keying
separate.
- Render `netsuke-tools-v2-…` and `netsuke-whitaker-v2-…` only to the step
summary and telemetry bundle. Do not restore or save shadow keys. Preserve the
trunk-only writer condition and per-save hit gating.
- If ready, combine the renderer change with the Phase 2 PR to produce only one
key rotation.
- Add `linux_cache_fingerprint_test.py`. Test restore/save equality,
correct-key-only invalidation for every specified input mutation, stability
under unrelated workflow edits, and manifest completeness against workflow
installer pins and Makefile tool pins. Preserve cache ownership and
writer-policy contracts without weakening them.
- Extend the collector to identify counterfactual hits from earlier successful
trunk writer keys within retention. Estimate restore-plus-install and whole-job
savings from observed v1 hit and miss cohorts. Separate source-only and
CI-editing PRs.
- Include the cold trunk installation and intervening PR misses in migration
cost. Publish the decision record. If accepted, switch live keys only in the
adoption PR, add an ADR and specify rollback to v1 rendering.

Phase 4: Experiment B – run-scoped public-API artefact preparation

Share one Cargo preparation across the eight direct-rustc fixtures. Keep every test, invocation and control.

Task 1: Task 4.1: Implement opt-in run-scoped preparation with a provenance manifest

Prepare once per Nextest run. Validate before every consumer uses the preparation.

Create a support module in tests/support/ that NetsukeRlib::build calls. Activate it only when NETSUKE_PUBLIC_API_PREP=run-scoped and NEXTEST_RUN_ID are both present. Otherwise run the current per-test build unchanged, including for local cargo test.
Under an exclusive file lock in a directory keyed by NEXTEST_RUN_ID, make the first consumer run the existing Cargo command with GATE_FEATURE_ARGUMENTS. Parse the JSON with the existing parser.
Then write a manifest with these fields:
- Schema version and run ID.
- Manifest path and feature arguments.
- An environment fingerprint: toolchain, RUSTFLAGS, CARGO_ENCODED_RUSTFLAGS set or unset and its value, wrappers, target root and build root.
- The selected artefact path, size and SHA-256.
- Every reported dependency artefact with size and modification time.
- The ordered dependency directories.
Make later consumers validate the run ID, the schema, the environment fingerprint, the selected-artefact digest and the dependency stat fingerprints. On any mismatch, malformed manifest or missing file, make the consumer fail with a diagnostic. Do not rebuild silently.
If preparation fails, write no manifest. Make each consumer fail independently, as it does today.
Leave the eight test identities, the rustc response-file invocation, the fresh output directories, the verbose-timing negative control, cli_configuration_fixture_compiles and the nested-cargo-builds membership unchanged.

Task 2: Task 4.2: Add focused tests

Cover the required behaviour of the preparation mechanism.

Add tests next to the new module with an injectable builder. Check these behaviours:
- One preparation serves several consumers.
- No preparation runs when no consumer runs.
- A builder failure propagates.
- A malformed manifest, a wrong run ID and a schema mismatch are rejected.
- A configuration mismatch, a changed artefact digest and a missing dependency file are detected.
- A split layout works, using tests/ui/split_build_dir_cargo_messages.jsonl.
- The fallback runs when the opt-in flag is absent.
Add an oversized response-file argument test in tests/support/rustc_response_file.rs that goes beyond the Windows command-line limit.
Assign every new test that runs Cargo to nested-cargo-builds. Keep the anchored selector contracts in nextest_child_cargo_group_test.py passing.

Task 3: Task 4.3: Wire label-gated activation and evaluate

Measure the real end-to-end effect.

In the same-repository PR coverage step of .github/workflows/ci.yml, set NETSUKE_PUBLIC_API_PREP only when the label ci-experiment:public-api-prep is present. Add a contract that forbids the variable in fork, trunk and coverage-main.yml paths.
Measure the number of preparations, the freshness and rebuild counts from the Task 2.4 lines, the summed preparation-plus-fixture duration, the reconstructed exclusive tail of the group, whole-job time and peak memory.
If the cost only moves elsewhere or the whole-job time does not improve, reject the variant.
Record the setup-script decision (Assumption 5).
Treat any later change to group membership as a separate, measured variant.
Before adoption, validate the change on the existing Windows and macOS lanes in the adoption PR.

🤖 Prompt for AI agents

Implement candidate B as opt-in lazy preparation shared within one Nextest run.
Preserve all eight fixtures and validate artefact provenance before every
consumer.

- Add a `tests/support/` module called by `NetsukeRlib::build`. Activate it only
when `NETSUKE_PUBLIC_API_PREP=run-scoped` and `NEXTEST_RUN_ID` are present.
Otherwise preserve the per-test path, including local `cargo test`.
- Use an exclusive file lock in a directory keyed by run ID. Make the first
consumer execute the existing Cargo command with `GATE_FEATURE_ARGUMENTS` and
parse JSON through the existing parser. Do no preparation when no consumer runs.
- Write the specified provenance manifest: schema, run ID, manifest path,
feature arguments, complete environment fingerprint, selected artefact
path/size/SHA-256, dependency artefact sizes/modification times, and ordered
dependency directories.
- Make later consumers validate schema, run ID, environment, selected digest and
dependency stats. Fail diagnostically on mismatch, malformed data or missing
files. Do not silently rebuild. Write no manifest after failed preparation so
consumers fail independently.
- Preserve test identities, response-file invocation, fresh output directories,
verbose-timing negative control, `cli_configuration_fixture_compiles` and
existing group membership.
- Add injectable-builder tests for sharing, no consumers, failures, malformed or
mismatched provenance, changed digests, missing dependencies, split layout and
fallback. Use `tests/ui/split_build_dir_cargo_messages.jsonl`. Add a
response-file test beyond the Windows command-line limit.
- Assign new Cargo-running tests to `nested-cargo-builds` and preserve anchored
selector contracts.
- Gate the environment variable on `ci-experiment:public-api-prep` only in
same-repository PR coverage. Contract-test its absence from fork, trunk and
`coverage-main.yml` paths.
- Measure preparation count, freshness, rebuilds, preparation-plus-fixture
duration, reconstructed exclusive group tail, whole-job time and peak memory.
Reject shifted costs or no whole-job gain. Record setup scripts as rejected for
the first variant.
- Treat group changes as separate measured variants. Validate Windows and macOS
in the adoption PR before adoption.

Phase 5: Experiment C – coverage profile merge overhead (conditional)

Proceed only if the baseline shows a material post-test tail. Otherwise defer with evidence.

Task 1: Task 5.1: Gate the experiment on baseline evidence and local checks

Confirm the mechanism before any CI trial.

Use the Phase 2 data to decide whether the post-test tail (merge and export) is material against the predeclared threshold. If it is not, record "deferred" in the decision record and stop this phase.
If it is material, confirm locally with the pinned nightly LLVM that %Nm pooling works for process-per-test runs. The default pattern has %p before %Nm, so the default creates per-process files.
Write a deterministic local control. Run a small instrumented program in concurrent repeated processes with the default pattern and with a pooled pattern. Compare counters exactly with llvm-profdata show.
Check collision avoidance across binaries through %m, profile collection paths, subprocess contributions and crash behaviour. Do not rely on %b. Do not change linker settings.

Task 2: Task 5.2: Add an LCOV fidelity comparator

Compare full detailed reports, not headline percentages.

Create a tested script in scripts/ that normalizes LCOV SF, FN, FNDA, DA and BRDA records, including uncovered entries. Make it report set differences and count differences.
Add tests in tests/workflow_contracts/.
Use the comparator on control and candidate reports. Explain any differences against documented natural nondeterminism.

Task 3: Task 5.3: Run the predeclared pool configuration through ordinary CI

Test one small, sequential set of configurations.

In the same-repository PR coverage step, set LLVM_PROFILE_FILE_NAME to a pooled pattern ending in .profraw only when the label ci-experiment:profile-pool is present. Start with a pool size equal to the existing four test threads. Evaluate a second predeclared size only if lock contention explains an inconclusive result.
Confirm in the telemetry that the variable reaches cargo-llvm-cov through the composite action.
Measure the profile count and bytes, the Nextest duration (work moved into process exit and lock contention), the post-test tail and whole-job time.
Do not add a thread override to llvm-profdata. It already selects threads automatically.
Keep the ratchet, the exclusions and the report scope unchanged. Add a contract that forbids the variable outside the gated PR path.

🤖 Prompt for AI agents

Implement candidate C only if the baseline post-test merge and export tail meets
the predeclared materiality threshold. Otherwise publish a deferred decision and
stop this phase.

- Validate `%Nm` pooling locally with the pinned nightly LLVM for
process-per-test runs. Account for the default `%p` before `%Nm`, which produces
per-process files.
- Build a deterministic local control using concurrent repeated processes of a
small instrumented program. Compare default and pooled counters exactly with
`llvm-profdata show`.
- Verify `%m` collision avoidance across binaries, collection paths, subprocess
contributions and crash behaviour. Do not rely on `%b` or change linker
settings.
- Add a tested LCOV normalizer/comparator in `scripts/` with tests in
`tests/workflow_contracts/`. Normalize `SF`, `FN`, `FNDA`, `DA` and `BRDA`,
including uncovered entries. Report set and count differences. Explain
control/candidate differences against documented natural nondeterminism.
- Gate a pooled `LLVM_PROFILE_FILE_NAME` ending in `.profraw` on
`ci-experiment:profile-pool` only in same-repository PR coverage. Start with a
pool of four. Test a second predeclared size only when lock contention explains
an inconclusive result.
- Verify propagation through the composite action in telemetry. Measure profile
counts and bytes, Nextest duration including exit work and contention, post-test
tail and whole-job time.
- Preserve instrumentation, exclusions, ratchet, report scope, linker flags and
automatic `llvm-profdata` thread selection. Add a contract forbidding the
variable outside the gated PR path.

Phase 6: Evidence review, adoption and follow-through

Decide on each candidate separately. Adopt only with evidence. Keep a working rollback.

Task 1: Task 6.1: Publish per-candidate decision records

Fill in the ticket's decision-record fields for candidates A, B and C.

In the ExecPlan and the issue, record for each candidate: the hypothesis and variant SHAs, the cohort links and attempts, the effect with uncertainty, the cost proxy (including failed attempts, telemetry and migration), the fidelity diffs (test identities, doctests, negative controls, LCOV), resources and reliability, and the decision with its rollback.
Label every observational comparison and its confounds.

Task 2: Task 6.2: Adopt accepted candidates and check combinations

Enable only the candidates that pass, through ordinary CI.

For each accepted candidate, open an adoption PR that makes it the default, adds an ADR under docs/, updates docs/developers-guide.md, and removes the experiment label path.
If more than one candidate passes, run the combination through normal CI before enabling the combination by default.
If a shared-actions change is needed, keep backwards-compatible defaults and use a reviewed immutable pin.

Task 3: Task 6.3: Run post-adoption follow-through

Confirm that the benefit lasts.

Keep the Phase 2 telemetry for a documented window of ordinary runs. Publish the follow-through observations in the issue.
Revert on fidelity loss, a resource or cost regression, or loss of the claimed benefit.
Keep inconclusive candidates disabled. Close the issue only after the follow-through is published.

🤖 Prompt for AI agents

Publish separate evidence-based decisions for A, B and C, then adopt only
accepted candidates. Verify combinations and sustained benefits with ordinary CI
and retain explicit rollback paths.

- Record each hypothesis, variant SHA, cohort link and attempt, effect with
uncertainty, cost proxy including failed attempts/telemetry/migration, fidelity
diffs, resources, reliability, decision and rollback in the ExecPlan and issue.
Label observational comparisons and confounds.
- For each accepted candidate, open an adoption PR to make it the default, add
an ADR under `docs/`, update `docs/developers-guide.md` and remove its
experiment label path.
- If multiple candidates pass, test the combination through normal CI before
making the combination the default.
- If shared-actions changes are required, preserve backwards-compatible defaults
and use a reviewed immutable pin.
- Retain Phase 2 telemetry for a documented ordinary-run window. Publish
follow-through observations in the issue.
- Revert on fidelity loss, resource or cost regression, or loss of the claimed
benefit. Keep inconclusive candidates disabled. Close the issue only after
publishing follow-through.

Research

Ticket Summary

Run an evidence-led investigation into faster Netsuke CI test gates. Revalidate the current state and publish a provenance-rich baseline. Add only bounded telemetry. Evaluate candidates A (cache fingerprints), B (run-scoped public-API artefact preparation) and C (coverage profile merge) independently and opt-in. Adopt a candidate only after a published decision record shows a real gain, no extra Ubicloud consumption and no loss of test or coverage fidelity.

Codebase Summary

The Linux merge gate in .github/workflows/ci.yml runs on ubicloud-standard-4-ubuntu-2404 for same-repository runs and on ubuntu-latest for fork PRs. It uses CARGO_BUILD_JOBS=4, NEXTEST_TEST_THREADS=4 and Nextest profile ci.

PR coverage calls leynos/shared-actions/.github/actions/generate-coverage@7d751ede…. That action runs one cargo llvm-cov nextest --workspace command, which builds, tests, merges and exports. It then runs an uninstrumented cargo test --doc. Each Cargo process has a 1800-second watchdog. At this SHA, the action installs and enforces cargo-nextest 0.9.145 and cargo-llvm-cov 0.9.0. Coverage therefore does not use the workflow's 0.9.133 pin.

cargo-llvm-cov 0.9.0 sets LLVM_PROFILE_FILE to <workspace>-%p-%<N>m.profraw under Nextest unless LLVM_PROFILE_FILE_NAME is set. It merges with llvm-profdata merge -sparse -f <list> and exports with llvm-cov export -format=lcov. It prints no phase timings.

Trunk coverage runs in .github/workflows/coverage-main.yml. That workflow owns ratchet and CodeScene publication.

.github/actions/linux-gate-cache/action.yml renders Cargo-download, tools and Whitaker keys. pins_hash hashes the whole of ci.yml and action.yml. Only a trunk push saves an archive. target/ is never archived. The Linux observation step does not report cache-matched-key.

.github/actions/memory-sampler/action.yml writes 15-second samples without timestamps and does not upload them.

In tests/command_env_ui_tests.rs, eight direct-rustc fixtures call NetsukeRlib::build with --all-features and parse Cargo JSON through tests/support/cargo_artifacts.rs. They pass arguments through tests/support/rustc_response_file.rs. These fixtures run serially in the one-slot nested-cargo-builds group in .config/nextest.toml.

Python workflow contracts in tests/workflow_contracts/ enforce cache ownership, writer policy, timeout ordering, profile and selector rules. scripts/release_operation_observability.py is the tested pattern for "missing means no value, not zero" telemetry.

File-Level Change Summary and Follow-Up Agent Prompts

File-level changes

docs/execplans/ci-test-speedups-investigation.md (new): revalidation, analysis plan, cost envelope, progress and decision records.
docs/ci-performance/ (new): normalized observation JSONL and README.
scripts/ci_test_evidence.py (new), with tests/workflow_contracts/ci_test_evidence_test.py: offline collector.
.config/nextest.toml: JUnit output for profile ci. Do not change budgets or groups.
.github/workflows/ci.yml and .github/workflows/coverage-main.yml: always() inventory step before cleanup, telemetry upload, sampler stop moved later, and label-gated variables for B and C (PR coverage only).
.github/scripts/ (new script): profile, object and version inventory.
.github/actions/memory-sampler/action.yml: timestamped samples.
.github/actions/linux-gate-cache/action.yml, a key-render script and fingerprint manifests (new): matched-key reporting, extracted renderer, shadow v2 keys.
tests/workflow_contracts/: telemetry contracts, linux_cache_fingerprint_test.py, experiment-gating contracts, and updates to the profile and timeout contracts.
tests/command_env_ui_tests.rs, tests/support/test_support_rlib.rs, tests/support/cargo_artifacts.rs, a new tests/support/ run-scoped preparation module, and tests/support/rustc_response_file.rs: telemetry, opt-in preparation and focused tests.
scripts/ (new): LCOV comparator. Use it only if Phase 5 proceeds.
docs/adr-0NN-*.md and docs/developers-guide.md: change only at adoption.

Prompt: evidence collector (Phase 1)

Create scripts/ci_test_evidence.py. Model it on scripts/release_operation_observability.py: fixed labels, None for missing or backwards timestamps, no secrets.
Input: saved `gh api` JSON for run attempts and jobs, job logs, and optional telemetry artefacts.
Output: one JSONL record per job attempt with these fields:
- run and job URL, attempt, event, head and merge SHA
- variant label, runner label, image and architecture
- tool versions found in the logs
- per-step durations
- cache keys with outcome: exact, prefix or miss
- sccache statistics
- Nextest summary, post-test tail and doctest timestamps
- test and doctest counts
Classify each record by cohort: same-repository PR, fork PR, trunk build-test, trunk coverage-main, trunk cache writer.
Add tests in tests/workflow_contracts/ci_test_evidence_test.py.
Do not trigger CI runs.

Prompt: baseline telemetry PR (Phase 2)

Add bounded telemetry without changing gate semantics.
1. Enable JUnit for Nextest profile `ci` in .config/nextest.toml.
2. In ci.yml PR coverage and in coverage-main.yml, add an always() inventory step before the `rm … target/llvm-cov-target target/llvm-cov` cleanup. Record:
   - .profraw count, bytes and size distribution
   - .profdata size and object count
   - `rustc -vV`
   - a copy of the JUnit file
3. Upload a telemetry bundle with the pinned upload-artifact action. Use always() and missing-file tolerance.
4. Timestamp the memory-sampler lines. Stop the sampler just before the upload.
5. Report cache-matched-key for the Linux Cargo-download restore.
6. Make NetsukeRlib::build print one line per Cargo call: elapsed time, fresh and non-fresh counts, lock-wait marker.
Keep RUSTFLAGS, the coverage inputs, the timeouts, the selectors and the assertions unchanged.
Add workflow contracts. Run `make test-workflow-contracts`.

Prompt: Experiment A

Add tools and Whitaker fingerprint manifests under .github/actions/linux-gate-cache/.
Cover:
- installer action SHAs and tool versions
- Makefile tool pins
- the mold VERSION and SHA256SUMS files and the build-tool scripts
- the toolchain and the lane
- both dylint.toml files
- a schema version
Move key rendering into a script. Keep the v1 keys byte-identical. Render the v2 keys as shadow keys only: no restore, no save.
Add linux_cache_fingerprint_test.py with these checks:
- restore and save produce identical keys
- every input mutation invalidates the correct key
- an unrelated edit leaves the v2 keys unchanged
- the manifest is complete against ci.yml and Makefile
Do not change the writer policy or the Cargo-download key.

Prompt: Experiment B

Add opt-in run-scoped preparation for NetsukeRlib::build.
Activate it only if NETSUKE_PUBLIC_API_PREP=run-scoped and NEXTEST_RUN_ID are both set. Otherwise keep today's per-test build.
1. Under a file lock in a directory keyed by NEXTEST_RUN_ID, make the first consumer build with GATE_FEATURE_ARGUMENTS and parse with the existing parser.
2. Write a manifest:
   - schema and run ID
   - environment fingerprint
   - selected artefact SHA-256
   - stat fingerprints of the dependency artefacts
   - ordered dependency directories
3. Make later consumers validate the manifest. On any mismatch, make them fail loudly.
Do not change test names, rustc invocations, the negative control, cli_configuration_fixture_compiles or the nested-cargo-builds membership.
Add focused tests and an oversized response-file test.

Prompt: Experiment C

First confirm that the baseline post-test tail is material. If it is not, stop and record "deferred".
Otherwise:
1. Verify %Nm pooling locally with the pinned nightly LLVM.
2. Run a deterministic concurrent-process counter control.
3. Build a tested LCOV normalizer and comparator.
4. Gate LLVM_PROFILE_FILE_NAME (pooled, ending in .profraw) behind the label `ci-experiment:profile-pool`, in the same-repository PR coverage step only.
Do not change instrumentation, exclusions, the ratchet, the linker flags or the llvm-profdata thread settings.

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Implement documentation and offline evidence tooling to establish the reference
state and baseline rules. Do not edit workflows or trigger CI runs in this
phase.

- Create `docs/execplans/ci-test-speedups-investigation.md` using
`docs/documentation-style-guide.md`. Record the reference SHA `24293a1b…`,
topology, cohorts, limits, timeout hierarchy, action pins, discrepancies and
out-of-scope issues. Treat September figures as historical context only.
- Predeclare whole-job elapsed time and summed runner-seconds per attempt as
primary endpoints. Label runner-seconds as a cost proxy. Separate queue and
provisioning time.
- Fix the useful-improvement threshold before evaluation: the larger of the
proposed 60-second floor and control interquartile range. Record fidelity,
memory, timeout, retry and ratchet checks.
- Require at least five control and five candidate observations per condition
from ordinary runs. Separate cache cohorts and run attempts. Exclude cancelled
and superseded attempts only from timing comparisons, not cost or reliability
accounting.
- Report individual values, median, range and a distribution-free shift estimate
with a permutation interval. Do not report small-sample p95. Define the
telemetry cost envelope, including durations and artefact bytes, with no
additions inside the Cargo watchdog.
- Implement `scripts/ci_test_evidence.py` following
`scripts/release_operation_observability.py`. Use fixed labels and `None` for
missing or backwards timestamps. Exclude secrets and raw errors.
- Parse saved attempt-job API JSON, logs and optional telemetry artefacts. Emit
provenance, versions, durations, cache evidence, sccache statistics, native
phase timestamps, doctest counts and test outcomes per job attempt. Classify the
specified cohorts.
- Test parsing, missing evidence, attempt separation and cohort classification
in `tests/workflow_contracts/ci_test_evidence_test.py`. Store JSONL and a README
in `docs/ci-performance/`; do not commit raw logs.
- Harvest recent retained ordinary-run logs before workflow edits. Extract
existing cache, sccache, resource and phase evidence. Measure tools-cache misses
caused by workflow or action edits. Keep unavailable matched keys, inventories,
per-test durations and RSS explicitly missing.
===============================================================================

Task: 2

Implement one bounded telemetry PR to establish a comparable baseline for
control and candidate runs. Preserve default execution and all gate semantics.

- Enable JUnit for Nextest profile `ci` without stored success output. Verify
the cargo-llvm-cov target location and whether testcase timestamps support
group-occupancy reconstruction. Preserve timeouts, groups, selection and
budgets.
- Add tested inventory tooling in `.github/scripts/`. Call it with `always()`
after coverage and before target cleanup in PR coverage and `coverage-main.yml`.
Record `.profraw` count, bytes and size distribution, `.profdata` size,
instrumented object count, `rustc -vV` including LLVM, and copied JUnit.
- Upload inventory JSON, JUnit, sccache JSON and text, and the resource log with
the pinned `upload-artifact` action. Follow `release.yml` for `always()`,
missing-file tolerance and fixed short retention.
- Check cargo-llvm-cov 0.9.0 for an environment-based verbose switch. If
available, enable it identically in both coverage steps. Otherwise document
merge and export as a combined post-test phase.
- Add epoch timestamps to memory samples without changing aggregates. Stop
sampling just before upload to include cache saves and CodeScene. Document
post-job hooks outside the sample window.
- Report Cargo-download `cache-matched-key` in the Linux observation step.
Separate exact hits, prefix restores and misses. Document the tools and Whitaker
key rotation and mark misses as instrumentation-induced until trunk repopulates
them.
- Add fixed-format per-Cargo lines to `NetsukeRlib::build` and
`tests/support/test_support_rlib.rs`. Include test name, elapsed time, fresh and
non-fresh compiler-artefact counts, and the Cargo lock-wait marker. Add a small
freshness helper next to `cargo_artifacts.rs`; preserve the selection API,
assertions, negative control and isolation.
- Update profile and timeout contracts. Add contracts for step ordering,
`always()`, missing-file tolerance, timestamps and matched-key reporting.
Preserve coverage inputs, `RUSTFLAGS`, watchdog and job ceiling.
- Run `make test-workflow-contracts` and affected Rust tests locally. Collect
ordinary runs until cohort floors are met. Publish the baseline and attribution
evidence in the ExecPlan and issue.
===============================================================================

Task: 3

Implement candidate A as explicit installation-input fingerprints and shadow v2
keys. Evaluate counterfactual hits without changing live cache behaviour or
adding paid work or storage.

- Add tools and Whitaker manifests under `.github/actions/linux-gate-cache/`.
Include the lane, toolchain, specified tool versions and installer SHAs,
Makefile pins, mold and build-tool file hashes, and schema version. Include
Whitaker installer version and both `dylint.toml` hashes.
- Document the unversioned df12 `pylint`, unverified Whitaker suite revision and
upstream installer behaviour. Preserve installer version and checksum checks.
- Extract key rendering into a script next to the action. Make restore and save
read the same manifests. Keep v1 keys byte-identical and Cargo-download keying
separate.
- Render `netsuke-tools-v2-…` and `netsuke-whitaker-v2-…` only to the step
summary and telemetry bundle. Do not restore or save shadow keys. Preserve the
trunk-only writer condition and per-save hit gating.
- If ready, combine the renderer change with the Phase 2 PR to produce only one
key rotation.
- Add `linux_cache_fingerprint_test.py`. Test restore/save equality,
correct-key-only invalidation for every specified input mutation, stability
under unrelated workflow edits, and manifest completeness against workflow
installer pins and Makefile tool pins. Preserve cache ownership and
writer-policy contracts without weakening them.
- Extend the collector to identify counterfactual hits from earlier successful
trunk writer keys within retention. Estimate restore-plus-install and whole-job
savings from observed v1 hit and miss cohorts. Separate source-only and
CI-editing PRs.
- Include the cold trunk installation and intervening PR misses in migration
cost. Publish the decision record. If accepted, switch live keys only in the
adoption PR, add an ADR and specify rollback to v1 rendering.
===============================================================================

Task: 4

Implement candidate B as opt-in lazy preparation shared within one Nextest run.
Preserve all eight fixtures and validate artefact provenance before every
consumer.

- Add a `tests/support/` module called by `NetsukeRlib::build`. Activate it only
when `NETSUKE_PUBLIC_API_PREP=run-scoped` and `NEXTEST_RUN_ID` are present.
Otherwise preserve the per-test path, including local `cargo test`.
- Use an exclusive file lock in a directory keyed by run ID. Make the first
consumer execute the existing Cargo command with `GATE_FEATURE_ARGUMENTS` and
parse JSON through the existing parser. Do no preparation when no consumer runs.
- Write the specified provenance manifest: schema, run ID, manifest path,
feature arguments, complete environment fingerprint, selected artefact
path/size/SHA-256, dependency artefact sizes/modification times, and ordered
dependency directories.
- Make later consumers validate schema, run ID, environment, selected digest and
dependency stats. Fail diagnostically on mismatch, malformed data or missing
files. Do not silently rebuild. Write no manifest after failed preparation so
consumers fail independently.
- Preserve test identities, response-file invocation, fresh output directories,
verbose-timing negative control, `cli_configuration_fixture_compiles` and
existing group membership.
- Add injectable-builder tests for sharing, no consumers, failures, malformed or
mismatched provenance, changed digests, missing dependencies, split layout and
fallback. Use `tests/ui/split_build_dir_cargo_messages.jsonl`. Add a
response-file test beyond the Windows command-line limit.
- Assign new Cargo-running tests to `nested-cargo-builds` and preserve anchored
selector contracts.
- Gate the environment variable on `ci-experiment:public-api-prep` only in
same-repository PR coverage. Contract-test its absence from fork, trunk and
`coverage-main.yml` paths.
- Measure preparation count, freshness, rebuilds, preparation-plus-fixture
duration, reconstructed exclusive group tail, whole-job time and peak memory.
Reject shifted costs or no whole-job gain. Record setup scripts as rejected for
the first variant.
- Treat group changes as separate measured variants. Validate Windows and macOS
in the adoption PR before adoption.
===============================================================================

Task: 5

Implement candidate C only if the baseline post-test merge and export tail meets
the predeclared materiality threshold. Otherwise publish a deferred decision and
stop this phase.

- Validate `%Nm` pooling locally with the pinned nightly LLVM for
process-per-test runs. Account for the default `%p` before `%Nm`, which produces
per-process files.
- Build a deterministic local control using concurrent repeated processes of a
small instrumented program. Compare default and pooled counters exactly with
`llvm-profdata show`.
- Verify `%m` collision avoidance across binaries, collection paths, subprocess
contributions and crash behaviour. Do not rely on `%b` or change linker
settings.
- Add a tested LCOV normalizer/comparator in `scripts/` with tests in
`tests/workflow_contracts/`. Normalize `SF`, `FN`, `FNDA`, `DA` and `BRDA`,
including uncovered entries. Report set and count differences. Explain
control/candidate differences against documented natural nondeterminism.
- Gate a pooled `LLVM_PROFILE_FILE_NAME` ending in `.profraw` on
`ci-experiment:profile-pool` only in same-repository PR coverage. Start with a
pool of four. Test a second predeclared size only when lock contention explains
an inconclusive result.
- Verify propagation through the composite action in telemetry. Measure profile
counts and bytes, Nextest duration including exit work and contention, post-test
tail and whole-job time.
- Preserve instrumentation, exclusions, ratchet, report scope, linker flags and
automatic `llvm-profdata` thread selection. Add a contract forbidding the
variable outside the gated PR path.
===============================================================================

Task: 6

Publish separate evidence-based decisions for A, B and C, then adopt only
accepted candidates. Verify combinations and sustained benefits with ordinary CI
and retain explicit rollback paths.

- Record each hypothesis, variant SHA, cohort link and attempt, effect with
uncertainty, cost proxy including failed attempts/telemetry/migration, fidelity
diffs, resources, reliability, decision and rollback in the ExecPlan and issue.
Label observational comparisons and confounds.
- For each accepted candidate, open an adoption PR to make it the default, add
an ADR under `docs/`, update `docs/developers-guide.md` and remove its
experiment label path.
- If multiple candidates pass, test the combination through normal CI before
making the combination the default.
- If shared-actions changes are required, preserve backwards-compatible defaults
and use a reviewed immutable pin.
- Retain Phase 2 telemetry for a documented ordinary-run window. Publish
follow-through observations in the issue.
- Revert on fidelity loss, resource or cost regression, or loss of the claimed
benefit. Keep inconclusive candidates disabled. Close the issue only after
publishing follow-through.

💡 Iterate on the plan with: @coderabbitai <feedback>

Example Feedback
- @coderabbitai You can skip phase 3. Add a simple unit test case for phase 2.
- @coderabbitai For design choice 1 go ahead with option 3 and replan.

Start Coding →

Offline evidence checkpoint for #894; no optimization is adopted.

The investigation plan fixes the endpoints, cohorts, useful-improvement threshold, cost envelope and fidelity gates. The evidence report links individual observations and explains missing fields.

The original harvest contains 21 ordinary attempts and 81 job records. A dated supplement retains all five jobs from ordinary attempt 38750711833, attempt 1, the latest ordinary PR attempt returned by the 15 October query without an outcome filter. These are 22 observational attempts and 86 records, not a controlled baseline. No CI execution or rerun was requested for collection.

The supplemental Linux job took 1,369 seconds end to end, with 495 seconds in the combined coverage step and 344.029 seconds of native Nextest execution. It reported 3,933 passes and six skips; three native doctest batches total 129 passes and 32 ignored tests. Its source and CI stages differ: the executed identity sets have 25 additions and four removals compared with the earlier retained job. Keep it separate from the earlier summaries; it is not a fifth matched control.

All three archive caches hit exactly in that job, while sccache recorded 2,934 hits and 559 misses. Peak system-used memory was 2,591 MiB; process RSS, separate merge/export timings and raw-profile inventory remain missing. Archive warmth does not establish compiler-cache equivalence.

The supplement's complete attempt accounts for 4,230 runner-seconds: 1,369 on the four-vCPU Ubicloud shape, 787 on the two-vCPU shape and 2,074 on GitHub-hosted jobs. The original harvest separately retains failed and cancelled attempts. These are consumption proxies for existing ordinary runs, not invoice savings or new benchmark expenditure. The two JSONL files add approximately 10.3 MiB and 0.75 MiB before Git compression; storage/transfer billing is unavailable. Queue/provisioning is not counted as a saving.

Candidate	Current decision	Evidence required before evaluation/adoption
A: installation fingerprints	Deferred; live v1 keys unchanged	Complete installer/suite compatibility inputs, shadow-key contracts and naturally observed writer availability. Two unrelated Kani workflow edits produced installed-tool misses, but counterfactual v2 hits and savings are unproved.
B: run-scoped API preparation	Deferred; no prototype enabled	Comparable freshness/rebuild/lock-wait and preparation-plus-fixture evidence, then whole-job and fidelity comparisons. Repeated Cargo calls are not eight demonstrated cold builds.
C: profile pools	Deferred; profile pattern unchanged	A measured material merge/export tail, exact deterministic counters and detailed LCOV comparison. The historical report tail is not a current measurement.

Each candidate has no evaluated variant SHA, candidate cohort or estimated effect yet. Existing runner shape, topology, worker limits, toolchain/action pins, Rust harness, selection, assertions, doctests, detailed coverage, ratchet, cache ownership and independent workflows remain unchanged by this checkpoint. There is no optimization to roll back. Future variants remain opt-in and independently reversible.

Adoption still requires at least five comparable ordinary controls and five candidate observations per condition, useful improvement beyond the larger of 60 seconds and control IQR, no cost increase and preserved fidelity. Source/test differences prevent causal comparisons here. Do not close the issue on this checkpoint.

Validation/publication status: the reviewed documentation and saved evidence are published in draft PR #918. Formatting, Markdown, spelling and Mermaid checks pass; the latest documentation increment has a complete CodeRabbit review with no findings. Earlier documentation review reported one unnamed unreviewed file, retained as a review-coverage limitation.

Six offline-collector Python files remain staged locally and are not part of that published head. Their linting and typing pass; 1,311 workflow contracts pass with three skips, followed by 27 passing script checks. Separate doctests pass with 129 passes and 32 ignored cases. Full Rust validation remains blocked by the unchanged packaging timeout; the single diagnostic awaits its predeclared shared-queue condition. Collector code commit/review and addition to the draft remain pending the required gate. No complete Rust-gate pass is claimed.

Documentation checkpoint update: draft PR #918, head 55a66ca91d1989a6caa5fc8fe9a0e5e5cff8e130, now has all required checks green. This includes ordinary CI run 38762276230, attempt 1, Windows gates and the Release Dry Run across Linux, macOS and Windows. No rerun or dispatch was requested.

Native checkout identifies tested merge 0ced8474e03ba6ec22d45abf8d81145e90d41a35, with newer main 486b876c7244c81b4fa9787a990448beebafea26 and the PR head as parents. Linux job 116516379338 passed 3,928 Nextest tests with six skips and the separate doctests (129 passes, 32 ignored). The original packaging test passed in 161.125 seconds under CI's existing configuration.

The tested merge differs from the local implementation tree across 78 paths, including Rust tests and Nextest configuration. Newer main also raises the packaging-specific allowance from 300 to 600 seconds; this branch retains 300 seconds. It therefore cannot replace the local failed Rust gate or establish comparable performance: the local suite selected 3,912 tests. Its packaging timeout and the other local failures remain recorded. H4's quiet-queue condition produced no qualifying pair in 424 observations before a temporary hold; no diagnostic test ran. Monitoring has resumed with the same launch condition and original timeout.

The six added Python collector files remain staged locally and are not part of this published head. Their Python gates pass, but code commit/review remain pending the required Rust validation. Candidates A, B and C remain deferred and disabled. This update is integration/validation evidence, not a controlled timing baseline, an adoption decision or a cost-saving claim. The issue remains open.

Record validation and recovery evidence (#894)

Clarify the diagnostic date range (#894)

Baseline attribution checkpoint: direct-rustc fixture durations from retained ordinary logs.

All eight direct-rustc fixtures have recorded passing outcomes in six trunk-coverage jobs and seven PR-coverage jobs. Each recorded fixture body took less than 0.6 seconds. The tables show within-job fixture medians and ranges, with cohorts kept separate. These are descriptive native timings: source, runner and cache conditions vary, and no candidate was executed. No CI run or rerun was requested for this analysis.

Trunk coverage

Run / job (attempt 1)	Source / tested SHA	Whole job (s)	Recorded fixtures	Fixture median (s)	Fixture range (s)
38103238271 / 114363328295	`3029d496` / `3029d496`	330	8	0.3240	0.300–0.345
38123563613 / 114423115185	`a75420e6` / `a75420e6`	337	8	0.2740	0.254–0.349
38125672660 / 114429422524	`b73f8d62` / `b73f8d62`	399	8	0.3640	0.280–0.562
38132248056 / 114448797246	`7539e32e` / `7539e32e`	416	8	0.3505	0.263–0.405
38147234655 / 114493045355	`557966c8` / `557966c8`	372	8	0.2690	0.239–0.328
38165545414 / 114547235263	`9e1bf27e` / `9e1bf27e`	388	8	0.3045	0.267–0.316

PR coverage

Run / job (attempt 1)	Source / tested SHA	Whole job (s)	Recorded fixtures	Fixture median (s)	Fixture range (s)
38174992874 / 114575472300	`dc07efdd` / `210e963a`	1134	8	0.3960	0.332–0.413
38177653022 / 114583406647	`7faf1f1d` / `96901a78`	954	8	0.2805	0.258–0.314
38178639487 / 114586289926	`70b8b0b7` / `02256c19`	1438	8	0.2885	0.246–0.327
38179177460 / 114587902925	`c89535d0` / `8f1fdd7d`	1213	8	0.3320	0.286–0.432
38194233518 / 114633283143	`3317c8ce` / `78a62891`	1341	8	0.3365	0.313–0.396
38276341866 / 114902625996	`ec0afc1c` / `1d4bbde6`	1062	8	0.3805	0.294–0.430
38750711833 / 116476064720	`2b01a3e3` / `820bfdaa`	1369	8	0.3415	0.249–0.502

Full source/tested SHAs, event, job provenance, versions, cache evidence, individual test identities/outcomes/durations and missing fields remain in the immutable original JSONL and dated supplement. Jobs without these fixture outcomes remain in cost/reliability accounting; this component table is not an exclusion from those datasets.

No overlapping durations were added into a wall-clock saving. Cargo freshness/rebuild counts, lock waits, shared preparation and the group's exclusive critical-path tail remain missing. The observed short fixture bodies do not establish those quantities. Candidate B remains deferred; there is no measured candidate effect, cost-saving claim or fidelity/adoption decision. The matched-control/candidate sample floors and whole-job/cost/fidelity gates remain unmet.

Record CI evidence and adoption criteria (#894)

Punctuate the coverage experiment condition (#894)

Clarify the CI evidence validation state (#894)

Retain current CI observations and parser evidence (#894)

Record validation and recovery evidence (#894)