Repository navigation
Add jev_audit: verify extracted values against their source - #48
Merged
Merged
Conversation
Jev reads text only, so multimodal intake lands as a cascade: the host vision/ASR model produces a dense transcript and extracted values, jev_screen checks the transcript, and the missing piece was a verifier for the values themselves (jkudish#45). jev_audit runs the SDE cascade cookbook's per-value failure-mode battery (hallucinated / off_target / incomplete / format, each framed true = something is wrong) in one request, with the cookbook's dedicated omission question for empty values. p_wrong is the max over a value's checks — never a mean, so one fired flag cannot be diluted by clean siblings — and any value at wrong_at (default 0.7) escalates the whole audit. Empty values get only the omission check; malformed answers mark the record invalid_response and escalate; a truncated source demotes pass to review. The README documents the full multimodal intake flow, and the agent skill routes it.
Owner
|
Thank you for this — the proposal and implementation are both excellent, and the failure-mode battery is a strong fit as the twelfth tool. Merged with a few amendments during review:
Shipping in the next release. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements #45 — the twelfth tool, proposed there with the research notes.
What
jev_auditaudits extracted values against the text they claim to come from, before the values are trusted. One request per call carries the SDE cascade cookbook's per-value failure-mode battery:hallucinated/off_target/incomplete/formatNouls, each framed so true = something is wrong;p_wrongis the max over its checks — never a mean, so one fired flag cannot be diluted by clean siblings — and any value atwrong_at(default 0.7, the cookbook'sFIRE_T) escalates the whole audit.Fail-closed throughout: a malformed answer marks the record
invalid_responseand escalates; a truncated source demotespasstoreview. The state carries{ purpose, source, records }with backticked path references in the questions, and every question gets the anti-injection framing.The README's Multimodal intake section documents the full cascade this enables — Jev is text-only by contract, so images/scans/audio go: extract (host vision/ASR model: dense transcript + values) →
jev_screenthe transcript →jev_auditthe values against it → judge with the existing tools. The cross-check is sound because the transcript and the values are independent passes over the same source.Bounds: 32 records per call, 500 chars per request line, 2,000 per value, source at the shared 50,000 cap. Skill routing added (
SKILL.mdtable + policy bullet,reference/tools.mdblock); tool-count assertions updated everywhere.Testing
npm run typecheck,npm run build,npm test: 244 tests. New coverage: the one-request battery and scoped wire shape, clean pass, fabrication escalation without diluting clean siblings, the empty-value omission path, fail-closed on malformed answers, truncated-source demotion, andwrong_atas a parameter.Notes
jev_extract.