Skip to content

feat: enforce signed V3 release evidence gates - #312

Draft
EvanProgramming wants to merge 51 commits into
Evan/v3-matched-evaluationfrom
Evan/v3-release-evidence
Draft

EvanProgramming wants to merge 51 commits into
Evan/v3-matched-evaluationfrom
Evan/v3-release-evidence

Conversation

@EvanProgramming

@EvanProgramming EvanProgramming commented Oct 21, 2026 •

Copy link
Copy Markdown
Owner

Tracks #268; stacked on PR #311. Closes #268 only after the required release evidence validates.

Change

The release validator preserves passed, failed, inconclusive, and waived states. A limited-evidence preview requires all non-waived gates to pass, and each eligible inconclusive gate requires a GPG-signed waiver bound to the exact candidate and preview scope. Failed controls, missing safety evidence, stale or tampered evidence, and an ordinary full release with inconclusive gates remain blocked. The validator checks the exact safety-log bytes against the signed platform record before interpreting them.

The eligible evaluation gates are R259 (zero native protocol recovery), R262 (dynamic replanning quality), R263 (matched task success), and R264 (matched request and token efficiency). A positive R259 recovery count is a failure and cannot be waived. A waiver leaves the original evidence inconclusive; it is not a pass. The rationale is to allow an explicitly scoped preview after independently verified safety controls pass, without weakening full-release requirements. The alternative of treating missing results as passes would conceal the actual evidence gap.

The preview version is 3.0.0.dev1. The local publisher revalidates the signed manifest and scoped evidence, builds from the signed candidate, signs an artifact attestation, verifies a signed tag and uploaded draft assets, and publishes only a GitHub prerelease. Automatic V3 tag publication is excluded from the ordinary release workflow. Preview updates fail closed if the source version cannot be verified or would downgrade the installation. These changes add no Python dependency.

Candidate and evidence

Candidate c942491405fa4130a75891dd45f95f789cfa523f has a locally and GitHub-verified commit signature from the pinned release-owner key. Six-platform CI and 30-minute soak run 41082489932 passed all six platform jobs, their 30-minute soaks, and the R266 collector. The R266 artifact was inspected for candidate binding, artifact hashes and required safety tests and signed with the release-owner key. The preceding candidate run failed in all six platform jobs and skipped the R266 collector. An earlier successful six-platform run is bound to 668a3e43f1b17365ab24d4f0fee3bf8af6186e4c and cannot establish results for this candidate.

No complete candidate-bound 17-gate evidence set, signed waivers, or accepted manifest exists. The original R259 and R262–R264 source reports were not found; further searching was stopped at the user’s request. Their results cannot be inferred from summaries. Missing safety evidence blocks the preview. Proposed scope: Linux and macOS preview with KYROZEN_SANDBOX_REQUIRED=1; Windows distribution, production use, general availability, and claims that full V3 gates passed are excluded.

The matched benchmark remains incomplete: the 3.5 run stopped at 133/180 rows with incomplete usage; the correctly configured 3.1 run stopped after one request with unknown usage. A report labeled 3.1 that actually used 3.5 is excluded as 3.1 evidence. Live benchmark work and budget-ledger reconciliation are paused as requested; unknown usage remains reserved. No new model calls were made.

Validation and risks

  • 61 focused release-gate, signed-evidence, publication, and update tests passed on the candidate source; tests include safety-log replacement, stale and tampered waivers, failed safety evidence, and downgrade prevention.
  • make check, make lint, make docs-check, Go tests and Go build passed locally on the candidate source.
  • The local full Python run and installed-wheel smoke failed when the disk filled; they are not passes. The candidate wheel and source distribution built, and local agent, subagent, and interactive TUI acceptance passed. A previous identical-tree candidate passed CI Docker persistence, wheel installation, installer syntax, TUI, and Linux/macOS installed-update jobs, but those results do not bind this SHA. The preceding CI run failed: Linux/macOS Python jobs failed three updater tests whose fixtures assumed a now-blocked unverified fallback. The fixtures now use a pinned version-checked source, and all three tests pass locally. Windows Python 3.12 and 3.13 supported-feature jobs failed the cancellation-safety test test_cancel_during_provider_call_prevents_late_write after a worker timed out; this does not establish that a late write occurred, but the safety check did not pass. The cancellation test now emits worker stacks on timeout without weakening its assertions. The new exact-SHA CI and all six soaks passed, but the previous Windows timeout did not recur and its underlying cause remains unconfirmed. The R266 record is signed; the other required candidate-bound gate records and waivers remain incomplete. Publication is blocked.
  • No preview manifest, tag, or public release has been created. This PR remains unmerged.

Stacked native planning follow-up

PR #317 is stacked on this PR. It addresses a reproduced R252 gap in which a complex native turn could execute tools without first establishing a Working Plan. Its signed candidate and fresh CI run supersede this PR head for proposed preview evidence. The preview remains blocked until that run, gate records, and scoped waivers validate.

PR #318 is stacked on #317 and replaces a fragile deadline test assertion; the preceding run also showed an unresolved Windows subagent acceptance failure. The proposed preview candidate is PR #318 head, not this PR head.

@coderabbitai

coderabbitai Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@ghfind-review ghfind-review Bot added review: high ghfind author score; see https://ghfind.com documentation Improvements or additions to documentation enhancement New feature or request release-blocker Required acceptance gate that must pass before the release labels Oct 21, 2026
@EvanProgramming
EvanProgramming force-pushed the Evan/v3-release-evidence branch 15 times, most recently from 7276872 to 992b027 Compare October 22, 2026 19:17
@EvanProgramming
EvanProgramming force-pushed the Evan/v3-release-evidence branch 3 times, most recently from 22543f7 to 068b8f0 Compare October 22, 2026 22:03
@EvanProgramming
EvanProgramming changed the base branch from Evan/v3-matched-evaluation to main October 28, 2026 12:49
@EvanProgramming
EvanProgramming changed the base branch from main to Evan/v3-matched-evaluation October 28, 2026 18:30
@EvanProgramming
EvanProgramming changed the base branch from Evan/v3-matched-evaluation to main October 29, 2026 02:38
@EvanProgramming
EvanProgramming changed the base branch from main to Evan/v3-matched-evaluation October 29, 2026 10:34

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation enhancement New feature or request release-blocker Required acceptance gate that must pass before the release review: high ghfind author score; see https://ghfind.com

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[V3] Enforce V3 release gates and publish traceable acceptance evidence

1 participant