Repository navigation
Feat/phase c surfaces - #12
Merged
Merged
Conversation
…ool output
"vedei-hook --fallback /path/to/vedei" ran the binary with no arguments. vedei
printed its usage, exited 0, and the client wrote that usage screen back to the
agent as the tool's result:
$ echo '{"tool_response":"cpf ..."}' | vedei-hook --fallback ./bin/vedei
vedei finds Brazilian sensitive data and leaked credentials, and ...
Every other fallback source — VEDEI_BIN, a vedei beside the client, one on
PATH — was run as "vedei hook --no-daemon". The flag's own help called its
value "a command", but the only thing that works there is a vedei binary, so it
is now run the same way as the others.
The test for this path passed the whole time. It checked only that the CPF was
absent from the output, and a help screen has no CPF in it. It now requires the
output to be the event, with the value redacted, and fails with the old
behaviour put back.
Found while writing the --fail-closed tests, which use the same path.
Fail-open (ADR-005) passes unscanned text through when detection fails,
because breaking a session is worse than missing one redaction. That is right
for most users and wrong where an unscanned CPF reaching a model is a
reportable incident — the regulated environments the project is aimed at.
--fail-closed, on vedei hook, vedei stream and vedei-hook, changes only the
failure path. When detection cannot run — the engine fails, the input is over
the size cap, or the thin client has no daemon and no vedei to fall back to —
the tool output is replaced by a notice that says nothing about the content:
[vedei: output withheld — detection could not run, and --fail-closed is set]
vedei stream writes only that notice and exits 1. vedei-hook passes the flag on
when it falls back to the full binary, so a fallback that cannot scan withholds
too. A scan that runs still redacts, a clean result still passes, and the
default is unchanged.
Malformed JSON and unrecognized event shapes stay fail-open even with the flag:
there is no tool output in them to withhold. Documented in the hook README.
Tests force a degraded scan with a 5-byte input cap. The fail-closed hook test
fails with the branch disabled.
The thesis behind vedei rests on one number nobody has measured: how often a developer's agent transcripts hold a credential or a Brazilian document. The way to measure it is to run the audit on volunteers' machines — which only works if a volunteer can hand over the result without handing over the data. "vedei transcript scan --stats" prints counts and nothing else: transcripts scanned, how many hold a finding, a credential, Brazilian personal data, and per type how many transcripts and how many distinct values. --json gives the same as an object a script can aggregate across machines. The Stats type has no field that could carry a value, a path or a fingerprint, so nothing can leak through it by mistake. Fingerprints are excluded on purpose, not only values: a CPF's fingerprint is a hash of eleven digits, few enough to enumerate, so publishing it would publish the CPF. A test marshals a summary and fails if any value, redacted value, fingerprint or path appears in it. Distinct values are counted per machine — one CPF in ten transcripts is one value — so the study measures exposure, not repetition.
transcript scan and scrub read Claude Code and Codex. Gemini CLI and Cursor
keep the same kind of plaintext record and are not read, and nothing said so —
a user of either would run the audit, see "nothing found", and conclude their
transcripts were clean.
Both READMEs and the command's help now name them, where each keeps its
sessions, and why they are not supported yet:
Gemini CLI ~/.gemini/tmp/<project hash>/chats/, JSON files. The format is
documented upstream but untested here against a real session;
reading it blind would report coverage it does not have.
Cursor state.vscdb, a SQLite database. Reading it needs a SQLite driver,
and scrub cannot safely rewrite a database.
Not implemented on purpose: support written against a format seen only in
documentation is support that has not been verified.
vedei builds for Windows and Windows 10 1803+ has Unix-domain sockets, so the daemon would start there — and quietly lose the property that makes it safe. os.Chmod on Windows only toggles the read-only attribute; the 0600 socket and 0700 directory restrict nobody, because Windows controls access with ACLs. The socket would be as private as the profile directory's default ACL happens to make it, which vedei does not check. It is also untested on Windows. SECURITY.md says so in the daemon section. hooks/README.md tells Windows users to run "vedei hook --no-daemon" for now — 29 ms per call instead of 8, and nothing crosses a socket — and lays out the plan: an explicit security descriptor granting the current user's SID only, via golang.org/x/sys/windows (already a transitive dependency), a test that reads it back, and a Windows CI runner. A named pipe with the same descriptor is the fallback design.
transcript/stats_test.go uses the corpus's made-up credential as a finding's raw value, and its fingerprint had been dropped when corpus/cases/ moved to a path rule — it only lived in the corpus then. The self-scan caught it in the test file and in the commit that added it, so the fingerprint comes back with the reason updated.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Adds
--fail-closedfor regulated environments andtranscript scan --stats, a shareable counts-only summary. It also documents two gaps that were silent (Cursor and Gemini CLI transcripts aren't read; on Windows the daemon's socket privacy doesn't hold) and fixes a bug wherevedei-hook --fallbacksent vedei's help text to the agent in place of the tool output.Why
These close the surface gaps the project's problem thesis listed under technical validation.
Fail-open is the wrong trade for part of the audience. ADR-005 passes unscanned text through when detection fails, because a broken session is worse than one missed redaction. That holds for most users. It doesn't hold where an unscanned CPF reaching a model is a reportable incident, and those regulated environments are exactly who vedei is for.
The thesis needs a number nobody has measured: how often a developer's agent transcripts hold a credential or a Brazilian document. Measuring it means running the audit on volunteers' machines, and that only works if a volunteer can share the result without sharing the data.
Two coverage gaps said nothing. A Cursor or Gemini CLI user who ran the audit saw "nothing found" and could fairly conclude their transcripts were clean. Separately, the daemon builds and starts on Windows while quietly losing the property that makes it safe.
Closes #
Checklist
make testpasses, including-racemake lintreports zero issues;make sastcleanWithhold,runHook,Summarizeand the fallback path each arrive with testsIf this adds or changes a detector
Not applicable. No detector or validator changes; corpus unchanged at 119 cases, 1.000 / 1.000.
If this touches anything a detected value passes through
Two changes sit on that path, and both are built to emit less, not more.
Finding.Rawout of the process.Statshas no field that could carry a value, a path or a fingerprint, and a test marshals a summary and fails if any value, redacted value, fingerprint or path shows up in it--fail-closedreplaces the whole output with a notice that says nothing about the contentValidityInvalidstill never leave the engineTrade-offs and limitations
--fail-closedwithholds a whole result, not a value. When detection can't run, the agent loses that tool output and has to retry. That cost is the point of the flag, and it's opt-in; the default is unchanged.Two cases stay fail-open even with the flag: malformed JSON, and an event shape vedei doesn't recognize. Neither has a tool output it knows how to withhold. Documented in the hook README.
--statsleaves out fingerprints, not just values. A CPF's fingerprint is a hash of eleven digits, few enough to enumerate, so publishing it would publish the CPF. The cost is that two machines can't tell whether they saw the same value. The study measures exposure per machine, not overlap between machines.Cursor and Gemini CLI are documented as unsupported, not supported. Gemini keeps JSON files whose format I've only read about; nothing here was tested against a real session, and support written blind would claim coverage it doesn't have. Cursor keeps a SQLite database, which needs a driver and can't be scrubbed safely.
Windows is documented, not fixed. On Windows,
os.Chmodonly toggles the read-only attribute, so the0600/0700modes restrict nobody. The recommendation until it's fixed isvedei hook --no-daemon, at 29 ms per call instead of 8. The fix (an explicit security descriptor, a test that reads it back, a Windows CI runner) is laid out but not built.The
--fallbackfix changes the flag's meaning. It was described as "a command"; it's now a vedei binary, run like every other fallback source. Anything else passed there was already broken.How to verify
The fallback bug, before and after:
The existing test for this path passed the whole time because it only checked that the CPF was absent, and a help screen has no CPF in it. It now requires the output to be the event with the value redacted, and it fails if the old behaviour is put back.
--fail-closed, with detection forced to fail through a 5-byte input cap:Without the flag, both pass the input through as before. With the fail-closed branch disabled,
TestRunHook_FailClosedWithholdsfails.--stats, on three synthetic transcripts:The same CPF in two transcripts counts as one distinct value.
Self-scan, tree and history:
By opening this PR you agree it is licensed under the MIT License and that you follow the Code of Conduct.
🤖 Generated with Claude Code