Roadmap: Strengthened Testing & Nightly Mutation Scans to 1.0 (Phase 1–3) #688

Suggested labels: testing, meta, roadmap
This is a tracking/meta issue — each item will be a separate PR. Checkboxes track progress.

Related

#676 Mutation-Gated Testing, #668 (313 mutants canceled at 30m), #678 (short gates share one runner), #679 (original PR sharding — superseded by #739), #739 (nightly workspace mutants, PR shards dropped), #685 (policy docs), #687 (ast-grep exact-variant)

Docs: docs/road-to-1.0.md (1.0 promises: datadir freeze, chain, IBD ~1s, 2 GiB RSS, P2P eclipse resistance), docs/quality.md Q-41 Rank 1 = 71 run / 196 skip, SCHEMA.md v24, docs/crash-recovery.md, TESTING.md, AGENTS.md

CI: .github/workflows/ci.yml required jobs fmt, deny, clippy, ast-grep, test, windows, macos, coverage, nixos-module-eval (9 jobs). Mutants no longer run or block on PRs. mutants.yml = nightly ~07:00 UTC, 90-min window, -j 1, --test-workspace=true, new diff lines first then rotating backlog, MISSED artifact + cursor — advisory only. fuzz.yml nightly 600s daily + Sunday 3600s extended. miri.yml nightly primitives-only Q-53.

Motivation

Owner direction (confirmed #739):

• Focus on high-level workspace journey tests — catch behavioral changes at the system level
• Mutants run nightly only — full workspace suite per mutant, ~30 per night, report misses, do NOT block merge
• Prevention first — ast-grep catches weak assertion patterns before they merge
• No PR gating — feedback is post-merge/advisory; fixes happen over time

Testing strength is the biggest gap to rbitcoin v1.0 release — goal: automated suite so strong only minimal real-world testing needed per release. Aligns with road-to-1.0.md freeze-last gate.

Current gaps addressed:

• Weak assertions is_err()/is_ok() and matches!(_) pass coverage but let survivors through
• Non-deterministic tests → nightly results vary → hard to trust or act on
• No structured baseline to track improvement as tests strengthen
• Crash/recovery paths untested → datadir freeze guarantee unvalidated

Research: "Investigating the application of differential fuzzing to support the identification and suppression of equivalent mutants" (Reis Garcia & Souza) — 1090 mutants across Bitcoin Core/OpenSSL/LND/Arrow: time-bounded differential fuzzing achieves 98–100% kill rate; existing fuzz seed corpora already contain inputs that unit tests miss. Limitation §7.3: compares return values only — side effects (output buffers, pass-by-ref, global state) need complementary coverage via fault-injection harness.

Phase 1 — Contract, Lints, Feedback Loop

Goal: Prevention first — catch weak patterns before merge; make nightly results actionable

• 1.1 Policy docs (docs: Mutation-Gated Testing — demonstrable killer per #676 #685) — TESTING.md, AGENTS.md, .agents/skills/ship-pr/SKILL.md

  • Exact error-variant assertion standard, waiver format // MUTANT-WAIVER expires YYYY-MM-DD reason
  • Zero production Rust changes
  • Solves: shared definition of what constitutes a complete test

• 1.2 ast-grep exact-variant rules (lint: weak asserts outside test files, and a waiver date token #687) — weak-assert.yml, waiver-expiry.yml

  • Flags bare is_err()/is_ok(), loose matches!(_) without specific value
  • Requires expiry on all waivers; scope: src/ including #[cfg(test)] blocks
  • Ignores: benches, core vector fixtures; time-source ban deferred (separate cleanup PR per code-shape.md)
  • Primary defense now — PRs not gated, so preventing weak patterns is critical
  • Solves: weak assertions pass coverage but miss behavioral changes

• 1.3 Mutants baseline-diff feedback (advisory — next 1–2d)
  Builds on ci: nightly workspace mutants, drop the PR shards #739 (nightly workspace scan). Upload artifacts from nightly run; maintain known-survivor baseline in docs/mutants/. Post PR comment: compares changed area vs baseline.

  • Advisory only — never blocks merge
  • First run initializes empty baseline; script merges shard/nightly results
  • Goal: transparency — contributors see what nightly will check; track improvement over time
  • Solves: survivors visible but not overwhelming; progress measurable
  • Risk 🟢🟡

• 1.4 Seed corpus mining (after 1.3)
  Extract fuzz inputs that kill missed mutants → convert to permanent assert_eq! tests

  • Solves: free test improvements using existing infrastructure
  • Aligns with: strengthen high-level workspace tests
  • Risk 🟢

Phase 2 — Determinism Infrastructure

Goal: Nightly results must be reproducible — flake = noise = ignored

• 2.1 Unified Clock API (2–3d, 🟡🔴)
  Consolidate NodeClock + thread-local → single wall_time() + mockable override; remove scattered Instant::now workarounds; setmocktime preserved.

  • Critical now: non-deterministic timing → nightly passes/fails randomly → trust erodes
  • ast-grep blocks NEW violations; legacy cleanup follows
  • Solves: time flake → consistent nightly results
  • Why 1.0: reliable IBD measurement + deterministic test runs
  • Depends: 1.2

• 2.2 Wall vs Monotonic Split (1–2d, 🟡🔴)
  Distinct monotonic() for deadlines — NTP-drift immune, both mockable

  • Solves: wall-clock changes breaking timeout logic → false negatives in nightly
  • Depends: 2.1

• 2.3 Exposed Decision State (2–3d, 🟡)
  Return Decision { peer, window, reason }; assert intermediate values, not just final tip height

  • Aligns with: high-level journey tests need meaningful intermediate assertions
  • Depends: 1.1

• 2.4 Deterministic RNG Injection (1–2d, 🟡)
  Single seeding point; ban rand::thread_rng outside it; stable seed in test mode

  • Solves: journey flake → identical code produces different results nightly
  • Depends: 2.3

• 2.5 Crash/Recovery Harness (1–2w, 🔴)
  Per docs/crash-recovery.md: 3 outcomes → recover-to-prefix / rebuild-from-lower-layer / typed Corrupt

  • Harness: inject → kill → reopen → verify invariant; covers 0/N, 1/N, N-1/N torn writes
  • Solves: Q-57/Q-58 fail-closed + store freeze-last gate — independent of mutants scheduling
  • Enables: remaining large store PRs
  • Depends: 1.1

Phase 3 — Evidence-Driven Improvement

Goal: Address survivors surfaced by nightly — systematically, not by panic

• 3.1 Differential Fuzzing Harness — Trigger: baseline shows persistent survivors
  Duplicate target → same input to original + mutated → compare outputs

  • Budget: 500K–1M executions; kill → promote input to permanent test
  • Limitation §7.3: return values only → torn-write coverage via 2.5
  • Solves: equivalent mutant bottleneck → resolve nightly findings efficiently
  • Depends: 2.3, 2.4

• 3.2 Deterministic In-Mem P2P Transport — Trigger: journey flake rate > 0
  Abstract tokio::net → control delivery, delay, reordering

  • Depends: 2.2, 2.4

• 3.3 Loom Concurrency — Trigger: audit identifies reducible lock ordering
  cfg(loom) model; skip if atomics-only; gated on 2.5 fault-seam patterns

  • Depends: 2.5

• 3.4 Shuttle Async Scheduling — Trigger: confirmed unreproducible bug

  • Depends: 3.2

• 3.5 Electrum Golden Transcripts — Trigger: measurable survivors in index paths
  Record/replay vs Fulcrum/Electrs; enforce in matrix

  • Depends: 2.3

• 3.6 Mutant Triage & Resolution Workflow — Trigger: baseline stable → process established
  Nightly findings → categorized (fixed / waived with expiry / accepted equivalent) → tracked in baseline

  • Goal: unresolved rate drops → candidate for optional future enforcement only if consensus supports it
  • Replaces earlier "selective promotion" — build trust first, do not force
  • Depends: 1.3

Verification / Alignment

1. ✅ No PR gating proposed — aligns with ci: nightly workspace mutants, drop the PR shards #739
2. ✅ ast-grep = primary prevention — fills gap left by removed checks
3. ✅ Determinism prioritized — nightly results must be trusted
4. ✅ 2.1 unchanged — more critical now, not less
5. ✅ Crash/recovery independent — store 1.0 path unaffected
6. ✅ Seed mining + differential fuzzing = strengthen tests — matches stated focus
7. ✅ Schedule verified: mutants.yml nightly advisory, not PR-gated per ci: nightly workspace mutants, drop the PR shards #739
8. ✅ No production code changes in Phase 1 — remains true

Non-Goals

• Not restoring PR merge gates — direction confirmed
• Not blocking on nightly results — advisory only
• Not formal verification of whole codebase — targeted islands only
• Not solving 2 GiB RSS measurement, IBD ~1s re-baseline, Tor, BIP331, Apple notarization — separate 1.0 gates in docs/quality.md

Next Steps

1. Merge docs: Mutation-Gated Testing — demonstrable killer per #676 #685, lint: weak asserts outside test files, and a waiver date token #687
2. Submit 2.1 Unified Clock API — critical for consistent, trusted nightly results
3. Adjust 1.3 baseline-diff: advisory comment only, no merge-fail
4. Seed corpus mining → strengthen high-level tests per stated direction

Change log:
Updated 2026-09-24 — aligned with #739 (nightly scans, no PR gating)
Updated 2026-09-22 — first published, initial plan