feat: raise the default irreversible hold threshold to 0.9 - #138
Merged
Merged
Conversation
The action guard held every call the judge scored at or above 0.7. On the recorded action-guard corpus the judge is wrong on 15% of calls below confidence 0.8 and under 1% above it, and the 0.7 to 0.9 band held no call the user regretted or rejected. A 0.9 cutoff chosen on one half of the corpus and checked on the other removed about 52 false alarms without losing a true catch, so `action.irreversible.confirm` is 0.9 and that band warns instead. `warn` stays 0.5; pattern holds and user or project overrides are unchanged. Tests move the old 0.85 and 0.87 judge scores into the new hold band and pin a judge-only 0.8 at warn and the default at 0.9. Docs and CHANGELOG carry the confidence table, the held-out split, the limits, and the method source.
The calibration heading carried the wrong day. The selection seed stays 20260930. No behaviour or numbers change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The action guard held every call the judge scored at or above 0.7 for
irreversible. On the recorded action-guard corpus, those judge-only holds were false alarms: the judge's error rate is 15–42% below confidence 0.8 and under 1% above it.Change
action.irreversible.confirm(the hold threshold) is 0.9 instead of 0.7. Calls the judge scores 0.5 to 0.9 warn instead of holding.warnstays 0.5. Pattern holds (destructive commands, deny rules) and user or project overrides are unchanged."action": { "irreversible": { "confirm": 0.7 } }.Evidence
A 0.9 cutoff chosen on a random half of the corpus and checked on the other half removed about 52 false alarms and lost no true catch. The offline calibration report: holds 48 → 25 (all 23 judge-only holds become warnings), regretted calls held 0 → 0. Limits: only 26 regretted calls, so this is a direction rather than a fitted threshold; the remaining misses sit at high confidence, where no threshold reaches them.
Verification
npm run check: 1207 pass / 0 fail before and after; typecheck and build clean.