fix(benches): correct release guard decision gating - #13
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Correct the 0.3.0b4 release guard's basic statistical decision error without changing codec behavior: Python collects raw timings and R owns inference. For a non-inferiority endpoint at the predeclared +3% regression margin, declare regression and block only when the simultaneous Bonferroni interval's lower bound exceeds +3%; declare non-inferiority only when its upper bound is below +3%; otherwise emit an explicit INCONCLUSIVE decision that remains published evidence but does not block release. Preserve improvement-family gating, fail closed on invalid evidence, update the public release/GitHub Actions path and executable tests, do not rerun the long benchmark while implementing or fish for a different result, keep version 0.3.0b4, and ship the correction safely.
What Changed
INCONCLUSIVE, preserving them as evidence without blocking release; established regressions and unmet improvement endpoints still fail the gate.Risk Assessment
✅ Low: The change is well-bounded and consistently implements the required non-inferiority, improvement-gating, evidence-validation, and release-path behavior.
Testing
No prior baseline test output was supplied; targeted pytest checks, same-evidence base/target R reproductions, exact-boundary and tamper probes, manifest-bound publication, release-path inspection, and version/codec smoke checks all succeeded without running the long benchmark or full suite.
Evidence: Published INCONCLUSIVE release evidence
Evidence: Manifest-conformant raw timing evidence
/var/folders/pt/19wwr80d5bs38zmww5rm2vsc0000gn/T/no-mistakes-evidence/01KZZMYEE88J6HQ6QGS8TV71HT/release-guard-inconclusive-result.json) - Evidence: Base incorrect FAIL result (local file:/var/folders/pt/19wwr80d5bs38zmww5rm2vsc0000gn/T/no-mistakes-evidence/01KZZMYEE88J6HQ6QGS8TV71HT/release-guard-inconclusive-base-result.json) - Evidence: Exact +3% boundary result (local file:/var/folders/pt/19wwr80d5bs38zmww5rm2vsc0000gn/T/no-mistakes-evidence/01KZZMYEE88J6HQ6QGS8TV71HT/release-guard-exact-margin-result.json)Evidence: End-to-end release decision transcript
Evidence: Public release-path proof
Evidence: Version and codec smoke evidence
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
🔧 **Review** - 1 issue found → auto-fixed ✅
benches/_panel.py:262- The acceptance criterion requires “fail closed on invalid evidence,” but this hunk acceptsINCONCLUSIVEsolely by enum membership. A result whose endpoint isregressed(or whosefailure_endpointsis nonempty) can claimINCONCLUSIVE, pass this validator andrelease_guard, and avoid blocking publication. At this shared boundary, reject disagreement between the R-emitted endpoint roles/statuses,failure_endpoints,inconclusive_endpoints, andgate_decisionwithout recomputing inference in Python.🔧 Fix: Validate release gate evidence consistency
✅ Re-checked - no issues remain.
✅ **Test** - passed
✅ No issues found.
git diff --name-status 33daa52a0aa5de3b5d442e2aad470b2a150951b0..bd936f346a3fb5b9119c5bb710903b02798f734e.venv/bin/python -m pytest -q tests/test_performance_evidence.py::test_r_owns_directional_decisions_order_term_and_insufficient_precision tests/test_performance_evidence.py::test_r_does_not_treat_an_inconclusive_noninferiority_interval_as_a_regression tests/test_performance_evidence.py::test_ab_cli_blocks_only_an_r_fail_decision tests/test_performance_evidence.py::test_one_simultaneous_family_spans_mixed_confirmatory_roles tests/test_performance_evidence.py::test_r_result_digest_and_engine_fail_closed tests/test_performance_evidence.py::test_r_result_rejects_disagreement_between_endpoint_and_gate_decisions tests/test_performance_evidence.py::test_missing_r_fails_closed_without_python_inference tests/test_release_report.py::test_performance_evidence_check_requires_and_validates_all_four_outputs tests/test_release_report.py::test_release_guard_preserves_each_required_untyped_shape tests/test_release_report.py::test_release_guard_rejects_an_exploratory_r_decisionExecuted base and targetanalyze_ab.Ragainst identical synthetic raw timings to reproduce the regression and compare decisionsExecuted the target R analyzer on a complete 36-pair, 22-endpoint manifest-conformant release-guard fixtureExecutedscripts/release-report.py::release_guard(...)in required-evidence mode and persisted its accepted INCONCLUSIVE outputExecuted the R analyzer on an interval exactly equal to +3%; it returned INCONCLUSIVERelabeled the INCONCLUSIVE result as PASS and confirmedrelease_guard(...)rejected the inconsistent evidencemake -n release-performance RELEASE_BENCH_VENV=.venvSemantically loaded.github/workflows/wheels.ymlwith PyYAML and verified the evidence command, uploaded guard files, and publication dependencyExecuted a typedmsgspec_toon.encode/decoderoundtrip and queried Python/Cargo package versionsgit status --short✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.