Skip to content

ci: add coverage (tarpaulin) + CRAP gates - #4

Merged
bkataru merged 1 commit into
mainfrom
ci/coverage-crap-gates
Oct 14, 2026
Merged

bkataru merged 1 commit into
mainfrom
ci/coverage-crap-gates

Conversation

@bkataru

@bkataru bkataru commented Oct 10, 2026 •

Copy link
Copy Markdown
Member

Gates added

One new job coverage + CRAP in .github/workflows/ci.yml: a single tarpaulin run under the llvm engine (one instrumented run IS the test run — tarpaulin exits non-zero on any test failure) writes lcov.info, which cargo-crap then scores. The two gates share the run because CRAP without a coverage feed scores every function cov=0 and measures nothing.

Public repo, so the job runs on ubuntu-latest following this repo's existing convention (pages.yml). This check is intended to become required once green (branch protection is wired separately after the run exists).

Is this crate even gateable natively?

dwasm is a build tool for wasm frontends, not a wasm-only crate: Cargo.toml has no crate-type = ["cdylib"], no wasm-bindgen dependency, and the binary is a native clap CLI that shells out to cargo build --target wasm32-unknown-unknown, wasm-bindgen, and wasm-opt. Verified before shipping: the suite compiles and runs natively (6/6 tests pass) under both engines. No rust-toolchain.toml, so the toolchain step installs stable explicitly.

Measured numbers (first run, on the build host)

metric value
tests 6 passed, 0 failed
line coverage 5.97% (8/134 lines, tarpaulin 0.37.5, ptrace engine)
line coverage (CI engine) 5.67% (8/141 lines, llvm engine)
worst maintained-surface CRAP 72.0 = patch_index_html (src/main.rs:202, CC 8)
other clean_old_artifacts 42 (CC 6), main 42 (CC 6), find_crate_dir/find_wasm_opt 20 (CC 4)

Coverage is low by measurement, not by failure: the suite exercises pure helpers (content hashing, reference extraction, stale-artifact cleanup) and never invokes the external toolchains in main()'s 5-step pipeline. The floor is therefore a regression tripwire at measured level, not a coverage target.

Thresholds

  • Coverage floor 3% — measured 5.97% (ptrace) / 5.67% (llvm) minus ~3pp margin for fleet variance between engines and runners.
  • CRAP ceiling 80 — worst measured function 72 plus ~10% headroom. --path src scores the whole crate surface; the CLI entry main scores 42, below the worst function, so no artificial exclusion is needed and the ceiling measures the real surface.

Tooling pins

tarpaulin 0.37.5 and cargo-crap 0.6.1, installed from per-arch release tarballs with the exact versions asserted in the same step ($GITHUB_PATH only affects later steps). Cargo.lock is committed and enforced via --locked plus a guard step.

Engine note

--engine Ptrace is blocked on the fleet's sysbox-confined runner containers (ASLR disable failed: EPERM: Operation not permitted, measured on dirmacs/skywalkers run 38064108686); auto would fall back silently. The job uses the documented Llvm engine with llvm-tools-preview. Both engines were measured on the host and agree within the floor's margin.

Summary step

Shell-only (no python3 on runner images — a crashing tracker must never repaint a green gate red) and structurally unable to fail the build (|| true).

Ratchet

  • Coverage floor rises +2pp per merged PR that adds pipeline-level tests; target ≥50% once the external-toolchain pipeline has a testable seam.
  • CRAP ceiling is lowered on schedule (target: remove headroom as functions are refactored) and immediately when a merged change lowers the true maximum; the ceiling stays at measured-max + ~10% until then. The honest refactoring target is patch_index_html (CC 8, CRAP 72): extract its hashed-reference rewriting and integrity-stripping loops into tested helpers — that both lowers the CRAP ceiling (room opens under the current 80) and raises coverage.

Caveats

  • The suite is small (6 tests) and covers helpers only; do not read the 3% floor as an endorsement of current coverage — it exists so the number cannot silently regress.
  • No DB, no Postgres container, no secrets, no external services; no cron needed for this gate. Cargo.lock is committed and enforced.

- one job, one tarpaulin run under the llvm engine feeding both gates
- pinned tools: tarpaulin 0.37.5, cargo-crap 0.6.1 (version-asserted)
- floor 3% (measured 5.97%% ptrace / 5.67%% llvm: 8/134 lines) as a
  measured-level regression tripwire
- CRAP ceiling 80 (worst maintained-surface function patch_index_html
  at 72, CC 8)
- public repo: ubuntu-latest; shell-only summary that cannot fail the build
@bkataru
bkataru merged commit a22b366 into main Oct 14, 2026
1 check passed
@bkataru
bkataru deleted the ci/coverage-crap-gates branch October 14, 2026 06:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant