Skip to content

feat: raise the default irreversible hold threshold to 0.9 - #138

Merged
DevMortimer merged 3 commits into
mainfrom
feat/irreversible-hold-threshold
Sep 29, 2026
Merged

DevMortimer merged 3 commits into
mainfrom
feat/irreversible-hold-threshold

Conversation

@DevMortimer

Copy link
Copy Markdown
Owner

The action guard held every call the judge scored at or above 0.7 for irreversible. On the recorded action-guard corpus, those judge-only holds were false alarms: the judge's error rate is 15–42% below confidence 0.8 and under 1% above it.

Change

  • Default action.irreversible.confirm (the hold threshold) is 0.9 instead of 0.7. Calls the judge scores 0.5 to 0.9 warn instead of holding.
  • warn stays 0.5. Pattern holds (destructive commands, deny rules) and user or project overrides are unchanged.
  • Restore the old behaviour with "action": { "irreversible": { "confirm": 0.7 } }.
  • Docs: the confidence-band table, the held-out split, the limits of the evidence, and the method source (Li, Miao, Krishnan, Padman, "JEV-as-a-Judge: Accept When Confident, Escalate When Unsure", arXiv:2609.26550).
  • Version bump to 0.75.0.

Evidence

confidence items error false alarms misses
0.50–0.60 118 42.4% 49 1
0.60–0.80 493 15.2% 72 3
0.80–0.95 4,937 0.8% 28 13
0.95–1.00 9,798 0.1% 0 8

A 0.9 cutoff chosen on a random half of the corpus and checked on the other half removed about 52 false alarms and lost no true catch. The offline calibration report: holds 48 → 25 (all 23 judge-only holds become warnings), regretted calls held 0 → 0. Limits: only 26 regretted calls, so this is a direction rather than a fitted threshold; the remaining misses sit at high confidence, where no threshold reaches them.

Verification

  • npm run check: 1207 pass / 0 fail before and after; typecheck and build clean.

The action guard held every call the judge scored at or above 0.7. On the
recorded action-guard corpus the judge is wrong on 15% of calls below
confidence 0.8 and under 1% above it, and the 0.7 to 0.9 band held no call
the user regretted or rejected. A 0.9 cutoff chosen on one half of the
corpus and checked on the other removed about 52 false alarms without
losing a true catch, so `action.irreversible.confirm` is 0.9 and that band
warns instead. `warn` stays 0.5; pattern holds and user or project
overrides are unchanged.

Tests move the old 0.85 and 0.87 judge scores into the new hold band and
pin a judge-only 0.8 at warn and the default at 0.9. Docs and CHANGELOG
carry the confidence table, the held-out split, the limits, and the
method source.
The calibration heading carried the wrong day. The selection seed stays
20260930. No behaviour or numbers change.
@DevMortimer
DevMortimer merged commit 4d11cfd into main Sep 29, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant