Skip to content

ci: add coverage (tarpaulin) + complexity (crap) gates - #139

Closed
bkataru wants to merge 1 commit into
mainfrom
ci/coverage-crap-gates
Closed

bkataru wants to merge 1 commit into
mainfrom
ci/coverage-crap-gates

Conversation

@bkataru

@bkataru bkataru commented Oct 10, 2026 •

Copy link
Copy Markdown
Member

What this adds

One CI job, coverage-crap, with TWO gates fed by ONE instrumented run:

  • cargo-tarpaulin (pinned 0.37.5) runs the whole suite under instrumentation. The coverage run IS the test run — tarpaulin exits non-zero on any test failure — gated with --fail-under 59.
  • cargo-crap (pinned 0.6.1) scores complexity from the LCOV that same run writes (cargo crap --lcov lcov.info --workspace --exclude '**/main.rs' --threshold 510 --fail-above). cargo-crap never runs tests itself: it parses source with syn and computes CRAP(m) = CC²·(1−cov)³ + CC. With no coverage feed every function scores cov=0 and CRAP collapses to CC³+CC, so the ceiling would measure nothing — that is why the two gates live in one job.

Both tools install from per-arch release tarballs at exact asserted versions (cargo-tarpaulin --version | grep -qx "tarpaulin 0.37.5", same for crap), so two runs cannot disagree because one host had a newer tool.

Measured first-run numbers (2026-10-10, tarpaulin 0.37.5)

  • Line coverage: 62.47% (10249/16405 lines), whole suite green.
  • Worst maintained-surface CRAP: 425.5 = handle_session_browser_key (crates/pawan-cli/src/tui/events/default.rs:36, CC 22, 5.9% coverage). Next: LspTool::execute 377.8 (crates/pawan-core/src/tools/lsp_tool.rs:276, CC 30), OllamaBackend::streaming 240.0.
  • CLI entry points excluded from the ceiling: run (crates/pawan-cli/src/main.rs:918, CC 42, CRAP 1806) and the other main.rs functions (870/506/462/380) — argument parsing and process wiring, 0% covered by design, like skywalkers' spike exclusion and git-sync's main.rs exclusion. 1053 functions are scored.

Thresholds + rationale

Gate Threshold Rationale
Coverage floor 59 Measured 62.47% − ~3.5pp margin. The margin covers the llvm-vs-ptrace engine delta (fleet-measured to be inside ~3pp) and the denominator artifact below.
CRAP ceiling 510 Worst maintained 425.5 + ~10%; deliberately 510 rather than 470 so the first fleet run cannot red on an engine artifact — that function's score becomes 506 if the llvm engine measures its single covered line as uncovered. Tighten to ~470 once a green llvm run re-measures.

Ratchet rule

  • Coverage floor +2pp per merged PR (raise the floor whenever the suite's coverage improves past floor+2).
  • CRAP ceiling lowered on schedule, and immediately on any fail-regression: once no function exceeds 30, drop the ceiling to the measured max +10% and delete this paragraph's escape hatch.

Runner choice

ubuntu-latest, matching every existing job in this workflow. pawan is a public repo, so GitHub-hosted runners work (private dirmacs repos get zero-step admission refusal there). The self-hosted arm64 fleet labels are not needed.

Engine choice (measured, not preferred)

--engine Ptrace was the first choice and is blocked on the fleet: tarpaulin's ptrace engine fails in the sysbox-confined runner containers with ASLR disable failed: EPERM: Operation not permitted (measured on dirmacs/skywalkers run 38064108686). auto would fall back silently, so the job pins --engine Llvm with the llvm-tools-preview component. The thresholds above were measured under ptrace on the build host and carry the stated margins for that delta.

DB needs

None. No Postgres/Redis; live tests are env-gated (PAWAN_NVIDIA_LIVE_TUI, PAWAN_RMUX_LIVE, #[ignore] live cases) and run nowhere in CI. The job does install the same test deps as the test job (protobuf-compiler, ripgrep, fd-find + the fd symlink) because the suite shells out to rg/fd and the workspace builds prost types.

Caveats

  • grind/ is a separate 16-crate workspace (own Cargo.lock, run by mise grind-test), not a root-workspace member. tarpaulin lists 9 of its files in the LCOV at 0% coverage as a scanning artifact (it is in the 16405 denominator, which makes the coverage floor conservative); cargo-crap --workspace does not score it. No grind/ gate is proposed here.
  • crates/pawan-core/src/agent/backend/lancor.rs is behind the optional lancor feature and is not built by the default test run, so it appears in neither the LCOV nor the CRAP report — consistent in CI, which also runs default features only.
  • Doctests: the test job runs cargo test --workspace --doc separately; tarpaulin runs them as part of its cargo test invocation.
  • The summary step parses crap.json with shell only (no python3 — the runner images carry none) and can never fail the job.

Required-check intent

pawan is public, so this gate is intended to become a required check on main once its first CI run is green — the wiring (branch protection) is the repo owner's action and is deliberately not taken by this PR.


First CI run results (verified green)

Run 38067951458 — all five jobs pass, including Coverage + CRAP:

  • Coverage: 59.73% (9107/15248 lines) under the llvm engine on ubuntu-latest/stable, against the floor of 59. The ptrace-vs-llvm delta measured here is ~2.7pp, inside the floor's ~3.5pp margin. Test suite under instrumentation: green (the plain test job: 1836 tests passed, 12 skipped).
  • CRAP: green — no scored function above the ceiling of 510 with **/main.rs excluded (worst measured locally under ptrace: 425.5).

One job, one instrumented run, two gates fed by it: cargo-tarpaulin
runs the whole suite (the coverage run IS the test run -- tarpaulin
exits non-zero on any test failure) and writes lcov.info, which
cargo-crap then scores (CRAP = CC^2*(1-cov)^3 + CC; without the
coverage feed every function scores cov=0 and the ceiling measures
nothing).

Both tools are pinned release tarballs (tarpaulin 0.37.5, cargo-crap
0.6.1) with asserted exact versions. The ptrace engine is blocked on
the fleet's sysbox-confined runners (EPERM), so the job uses
--engine Llvm with llvm-tools-preview.

Measured on the first run (ptrace, build host): 62.47% line coverage
(10249/16405), worst maintained-surface CRAP 425.5
(handle_session_browser_key, crates/pawan-cli/src/tui/events/
default.rs:36, CC 22, 5.9% cov). Floors: coverage 59, CRAP 510.
@bkataru

bkataru commented Oct 11, 2026

Copy link
Copy Markdown
Member Author

Superseded by #141 — please keep that one for review; this one can close.

To be fair to the work here: this was a working gate, not a failed attempt — the job already produced the byte-exact Coverage + CRAP context with all checks green, the same pinned tools (tarpaulin 0.37.5, cargo-crap 0.6.1), the same one-instrumentated-run design, the same ceiling (510) and exclusions. The two substantive differences, both reasons this is superseded rather than merged:

  1. The measurement base is stale. This measured 62.47% on a 16405-line denominator on 2026-10-10; main's denominator has since changed (15248 lines), and ci: coverage + CRAP gate (unblocks protected merges) #141 re-measured both bases with the identical command: origin/main 60.13% (floor 58, sized with the fleet delta — this PR's own runner run measured 59.75% ~0.40pp under the build host) and the appa branch head 60.76%. A floor measured against a different tree state is exactly the shape that reds spuriously later.
  2. Failure directions proven. ci: coverage + CRAP gate (unblocks protected merges) #141 proved both gates can fail before committing the thresholds (--fail-under 99 → exit 1; --threshold 400 → exit 1 vs worst 431.04), added the belt-and-braces naming (workflow name: and job name: both Coverage + CRAP, so the context is right however GitHub derives it), and the actual job log shows the real measured number with the enforced floor — not just a green badge.

Also practical: both PRs need the owner's approval under pawan's protection, and two competing gate PRs would split that review. The ratchet rule from here (raise the floor in whole points as coverage improves) carries over to #141.

@bkataru bkataru closed this Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant