Skip to content

fix: pin the gate harness across an incompatible requirement - #196

Merged
Verdenroz merged 4 commits into
masterfrom
fix/harness-pin
Sep 12, 2026
Merged

Verdenroz merged 4 commits into
masterfrom
fix/harness-pin

Conversation

@Verdenroz

@Verdenroz Verdenroz commented Sep 12, 2026

Copy link
Copy Markdown
Owner

What changed

  • Rewrote the merge-base worktree's soothfast version requirements when cargo update --precise is refused, then retried the pin. Only the requirement string moves, so features, optional and default-features beside it survive.
  • Changed sync_harness_versions to return a HarnessSync outcome instead of always Ok(()).
  • Marked a run whose reference could not be pinned in the gate banner, in the verdict line, and in .soothfast/triage/harness-mismatch.txt. The verdict line is what the PR comment's tail carries.
  • Added --allow-harness-change, which waives comparison regressions on a harness-mismatched run. Checked claims hold HEAD against its own declared numbers and are not waived.
  • Stamped the mismatch into the stored reference document, so a reference served from the run cache carries the mark of the run that measured it.
  • Added a "When the harness moves under a PR" section to docs/gating.md.

Fixes #194

Why

A --against-ref gate builds the merge-base's bench binary from the merge-base's lockfile. When a pull request bumps soothfast across a caret boundary, cargo update -p soothfast@0.2.0 --precise 0.3.1 is refused by the merge-base's own soothfast = "0.2.0" requirement. The pinning code printed a WARN and carried on, so the reference embedded the old harness while HEAD embedded the new one, and the harness delta was reported as the project's regression. That looked like about +5% callgrind instructions on a 30K-instruction bench, under a red verdict, with the only trace scrolled off the top of the log.

The worktree is a throwaway checkout, so its requirement is not worth respecting when it blocks the pin. Where the pin still cannot land, the run says so rather than deciding quietly. A reference measured with a different instrument is not evidence in either direction, so a quiet pass would be as wrong as the red verdict.

Merge ordering

The docs/gating.md section here inserts immediately before ## Ratchets, which is the same anchor another open branch uses. Both additions stand alone, so whichever lands second keeps both sections. Nothing else in this branch overlaps.

How was this tested

  • cargo test -p cargo-soothfast: 215 passed, 1 failed. The failure is invoke::tests::a_plain_directory_in_the_repo_is_not_a_worktree_root, which asserts the workspace root is not a linked worktree. It fails from any linked-worktree checkout and passes from a plain one, on this branch and on master.
  • cargo fmt --all -- --check: clean.
  • cargo clippy --workspace --all-targets -- -D warnings: clean.
  • New unit tests cover the caret-incompatible case on harness_mismatches, cargo's refusal text told apart from a network failure, all three requirement shapes, the path-dependency and non-dependency-table skips, trailing-comment preservation, the waiver conditions, both verdict texts, the run-cache round trip, and the triage note.
  • cargo test --workspace and make gate BASE=master were not run. The change touches no measured function in the three bench crates.
  • The incompatible-requirement path was not reproduced against a live registry. The proof here is the constructed lockfile and manifest fixtures in the unit tests.

Checklist

  • make check passes (fmt, clippy -D warnings, cargo test --workspace)
  • make gate BASE=master passes, or any intentional cost change is explained above
  • Tests added/updated for new behavior (not required for docs/CI-only changes)
  • Public API items have /// doc comments
  • No new dependency, or its justification is included above (see Dependency Policy)
  • Docs (README.md, docs/, soothfast:bind/soothfast:claim markers) updated if behavior changed

- a caret-incompatible soothfast bump made the reference bench embed
  the old harness, and the delta was reported as the project's
  regression
- the merge-base worktree is disposable, so its own requirement is not
  worth respecting when it blocks the pin
@Verdenroz
Verdenroz deployed to soothfast-bot September 12, 2026 00:13 — with GitHub Actions Active
@Verdenroz
Verdenroz deployed to soothfast-bot September 12, 2026 00:13 — with GitHub Actions Active
@Verdenroz
Verdenroz deployed to soothfast-bot September 12, 2026 00:13 — with GitHub Actions Active
@Verdenroz
Verdenroz deployed to soothfast-bot September 12, 2026 00:13 — with GitHub Actions Active
@Verdenroz
Verdenroz deployed to soothfast-bot September 12, 2026 00:13 — with GitHub Actions Active
@Verdenroz
Verdenroz deployed to soothfast-bot September 12, 2026 00:13 — with GitHub Actions Active
@Verdenroz
Verdenroz deployed to soothfast-bot September 12, 2026 00:13 — with GitHub Actions Active
- a reference the pin could not reach measures a different protocol, so
  its deltas are not the change's in either direction
- the mark reaches the header, the verdict line the PR comment tails,
  and the triage artifact
- a run served from the run cache carries the mark of the run that
  measured it
@soothfast-bot

soothfast-bot Bot commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

soothfast gate

gate: measuring merge-base of origin/master in worktree (interleaved rounds)
gate: bench binaries identical (code and data match) — no measurable change possible
gate: gating backend = walltime
gate: build=08f03a noise_floor=0.11% thresholds: instructions +5% ir +5% walltime +10.0% alloc/size +5% polls/wakes +5%
ok    soothfast_sdk::bench_emit_typescript walltime_median_ns 2104630.0 -> 2104630.0 (+0.0%)
ok    soothfast_sdk::bench_emit_typescript allocs 42930 -> 42930 (allowed <= 45076)
ok    soothfast_sdk::bench_emit_typescript alloc_bytes 2938940 -> 2938940 (allowed <= 3085887)
ok    soothfast_sdk::bench_lower walltime_median_ns 1354211.0 -> 1354211.0 (+0.0%)
ok    soothfast_sdk::bench_lower allocs 23907 -> 23907 (allowed <= 25102)
ok    soothfast_sdk::bench_lower alloc_bytes 2064788 -> 2064788 (allowed <= 2168027)
ok    soothfast_sdk::bench_emit_typescript assert complexity: claimed O(n); growth drift x0.99 over sizes [16, 64, 256] (limit x2.5)
ok    soothfast_sdk::bench_lower assert complexity: claimed O(n); growth drift x1.31 over sizes [16, 64, 256] (limit x2.5)
gate: passed (2 item(s))

@Verdenroz
Verdenroz deployed to soothfast-bot September 12, 2026 00:14 — with GitHub Actions Active
@Verdenroz
Verdenroz deployed to soothfast-bot September 12, 2026 00:14 — with GitHub Actions Active
@Verdenroz
Verdenroz deployed to soothfast-bot September 12, 2026 00:14 — with GitHub Actions Active
@Verdenroz
Verdenroz deployed to soothfast-bot September 12, 2026 00:14 — with GitHub Actions Active
@Verdenroz
Verdenroz deployed to soothfast-bot September 12, 2026 00:14 — with GitHub Actions Active
@Verdenroz
Verdenroz deployed to soothfast-bot September 12, 2026 00:14 — with GitHub Actions Active
@Verdenroz
Verdenroz deployed to soothfast-bot September 12, 2026 00:14 — with GitHub Actions Active
@Verdenroz
Verdenroz deployed to soothfast-bot September 12, 2026 00:22 — with GitHub Actions Active
@Verdenroz
Verdenroz deployed to soothfast-bot September 12, 2026 00:22 — with GitHub Actions Active
@Verdenroz
Verdenroz deployed to soothfast-bot September 12, 2026 00:22 — with GitHub Actions Active
@Verdenroz
Verdenroz deployed to soothfast-bot September 12, 2026 00:22 — with GitHub Actions Active
@Verdenroz
Verdenroz deployed to soothfast-bot September 12, 2026 00:22 — with GitHub Actions Active
@Verdenroz
Verdenroz deployed to soothfast-bot September 12, 2026 00:22 — with GitHub Actions Active
@Verdenroz
Verdenroz deployed to soothfast-bot September 12, 2026 00:22 — with GitHub Actions Active
@Verdenroz
Verdenroz merged commit ffefa67 into master Sep 12, 2026
31 of 33 checks passed
@Verdenroz
Verdenroz deleted the fix/harness-pin branch September 12, 2026 00:25
@Verdenroz Verdenroz mentioned this pull request Sep 12, 2026
6 tasks
Verdenroz added a commit that referenced this pull request Sep 12, 2026
## What changed

- Froze the `## Unreleased` heading in `CHANGELOG.md` to `## 0.3.2 - 2026-09-11` and wrote its Overview and Upgrade notes.
- Bumped `[workspace.package] version` and the eight `[workspace.dependencies]` requirements from `0.3.1` to `0.3.2`.
- Regenerated `Cargo.lock`. Only version lines moved; no dependency was added, removed, or changed.
- Bumped the version strings in `README.md` and `soothfast.toml`.

## Why

0.3.1 gave the action its `features` and regeneration inputs, so CI could be
told once what a repository needs. A developer running the CLI by hand still
repeated the same flags on every invocation, and two defects surfaced once
adopters started bumping across versions. This release carries all three
fixes: `soothfast.toml` holds the feature and package defaults, the gate stops
letting a reference it could not pin decide a verdict quietly, and the
changelog drops the bot's own regeneration commits instead of listing the
paperwork as changes.

Everything here is additive and opt-in. A repository on `soothfast = "0.3"`
picks it up without touching its manifest.

## How was this tested

- `cargo fmt --all -- --check`: clean.
- `cargo clippy --workspace --all-targets -- -D warnings`: clean.
- `cargo test --workspace --no-fail-fast`: 817 passed, 4 ignored, 1 failed. The failure is `invoke::tests::a_plain_directory_in_the_repo_is_not_a_worktree_root`, which asserts the workspace root is not a linked git worktree. It fails from any linked-worktree checkout and passes from a plain one, which is what CI uses. Confirmed by running the same built test binary from both.
- `cargo check --workspace` after the bump: clean, all ten crates resolve at `0.3.2`.
- Read the frozen section against `git log v0.3.1..master`: it lists #195, #196 and #199, and none of the four soothfast-bot regeneration commits. Those exclusions are the filter from #199 running against real history.
- `make gate BASE=master` was not run. This commit changes no measured code.

## Checklist

- [ ] `make check` passes (fmt, clippy `-D warnings`, `cargo test --workspace`)
- [ ] `make gate BASE=master` passes, or any intentional cost change is explained above
- [ ] Tests added/updated for new behavior (not required for docs/CI-only changes)
- [ ] Public API items have `///` doc comments
- [x] No new dependency, or its justification is included above (see [Dependency Policy](../CONTRIBUTING.md#dependency-policy))
- [x] Docs (`README.md`, `docs/`, `soothfast:bind`/`soothfast:claim` markers) updated if behavior changed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

gate: an unpinnable reference harness still decides the verdict

1 participant