Skip to content

Feat/phase c surfaces - #12

Merged
mkmuniz merged 6 commits into
mainfrom
feat/phase-c-surfaces
Oct 7, 2026
Merged

mkmuniz merged 6 commits into
mainfrom
feat/phase-c-surfaces

Conversation

@mkmuniz

@mkmuniz mkmuniz commented Oct 7, 2026 •

Copy link
Copy Markdown
Owner

What this changes

Adds --fail-closed for regulated environments and transcript scan --stats, a shareable counts-only summary. It also documents two gaps that were silent (Cursor and Gemini CLI transcripts aren't read; on Windows the daemon's socket privacy doesn't hold) and fixes a bug where vedei-hook --fallback sent vedei's help text to the agent in place of the tool output.

Why

These close the surface gaps the project's problem thesis listed under technical validation.

Fail-open is the wrong trade for part of the audience. ADR-005 passes unscanned text through when detection fails, because a broken session is worse than one missed redaction. That holds for most users. It doesn't hold where an unscanned CPF reaching a model is a reportable incident, and those regulated environments are exactly who vedei is for.

The thesis needs a number nobody has measured: how often a developer's agent transcripts hold a credential or a Brazilian document. Measuring it means running the audit on volunteers' machines, and that only works if a volunteer can share the result without sharing the data.

Two coverage gaps said nothing. A Cursor or Gemini CLI user who ran the audit saw "nothing found" and could fairly conclude their transcripts were clean. Separately, the daemon builds and starts on Windows while quietly losing the property that makes it safe.

Closes #

Checklist

  • make test passes, including -race
  • make lint reports zero issues; make sast clean
  • New code does not lower coverage — Withhold, runHook, Summarize and the fallback path each arrive with tests

If this adds or changes a detector

Not applicable. No detector or validator changes; corpus unchanged at 119 cases, 1.000 / 1.000.

If this touches anything a detected value passes through

Two changes sit on that path, and both are built to emit less, not more.

  • No path sends Finding.Raw out of the process. Stats has no field that could carry a value, a path or a fingerprint, and a test marshals a summary and fails if any value, redacted value, fingerprint or path shows up in it
  • Redaction still hides the identifying part of the value. --fail-closed replaces the whole output with a notice that says nothing about the content
  • Findings with ValidityInvalid still never leave the engine

Trade-offs and limitations

--fail-closed withholds a whole result, not a value. When detection can't run, the agent loses that tool output and has to retry. That cost is the point of the flag, and it's opt-in; the default is unchanged.

Two cases stay fail-open even with the flag: malformed JSON, and an event shape vedei doesn't recognize. Neither has a tool output it knows how to withhold. Documented in the hook README.

--stats leaves out fingerprints, not just values. A CPF's fingerprint is a hash of eleven digits, few enough to enumerate, so publishing it would publish the CPF. The cost is that two machines can't tell whether they saw the same value. The study measures exposure per machine, not overlap between machines.

Cursor and Gemini CLI are documented as unsupported, not supported. Gemini keeps JSON files whose format I've only read about; nothing here was tested against a real session, and support written blind would claim coverage it doesn't have. Cursor keeps a SQLite database, which needs a driver and can't be scrubbed safely.

Windows is documented, not fixed. On Windows, os.Chmod only toggles the read-only attribute, so the 0600/0700 modes restrict nobody. The recommendation until it's fixed is vedei hook --no-daemon, at 29 ms per call instead of 8. The fix (an explicit security descriptor, a test that reads it back, a Windows CI runner) is laid out but not built.

The --fallback fix changes the flag's meaning. It was described as "a command"; it's now a vedei binary, run like every other fallback source. Anything else passed there was already broken.

How to verify

$ make test && make lint && make sast
0 issues.

The fallback bug, before and after:

$ echo '{"tool_response":"cpf 529.982.247-25"}' | vedei-hook --socket /tmp/absent.sock --fallback ./bin/vedei
# before:
vedei finds Brazilian sensitive data and leaked credentials, and ...
# after:
{"tool_response":"cpf ***.***.***-25 [vedei: cpf redacted]"}

The existing test for this path passed the whole time because it only checked that the CPF was absent, and a help screen has no CPF in it. It now requires the output to be the event with the value redacted, and it fails if the old behaviour is put back.

--fail-closed, with detection forced to fail through a 5-byte input cap:

$ printf 'cpf 529.982.247-25' | vedei stream --no-daemon --max-input 5 --fail-closed
[vedei: output withheld — detection could not run, and --fail-closed is set]
$ echo $?
1

$ echo '{"tool_response":"cpf 529.982.247-25"}' | vedei-hook --socket /tmp/absent.sock --fallback - --fail-closed
{"tool_response":"[vedei: output withheld — detection could not run, and --fail-closed is set]"}

Without the flag, both pass the input through as before. With the fail-closed branch disabled, TestRunHook_FailClosedWithholds fails.

--stats, on three synthetic transcripts:

$ vedei transcript scan --path /tmp/t --stats
3 transcript(s) scanned
2 contain at least one finding
0 contain a credential
2 contain Brazilian personal data

TYPE  TRANSCRIPTS  DISTINCT VALUES
cnpj  1            1
cpf   2            1

Counts only: no values, paths or fingerprints. Safe to share.

$ vedei transcript scan --path /tmp/t --stats --json | grep -cE '529|982|11\.222|\.jsonl|/tmp/'
0

The same CPF in two transcripts counts as one distinct value.

Self-scan, tree and history:

$ vedei scan .
.: nothing found in 141 file(s)
$ vedei git .
history: nothing found in 357 file(s)
$ GOOS=windows GOARCH=amd64 go build ./cmd/...

By opening this PR you agree it is licensed under the MIT License and that you follow the Code of Conduct.

🤖 Generated with Claude Code

…ool output

"vedei-hook --fallback /path/to/vedei" ran the binary with no arguments. vedei
printed its usage, exited 0, and the client wrote that usage screen back to the
agent as the tool's result:

    $ echo '{"tool_response":"cpf ..."}' | vedei-hook --fallback ./bin/vedei
    vedei finds Brazilian sensitive data and leaked credentials, and ...

Every other fallback source — VEDEI_BIN, a vedei beside the client, one on
PATH — was run as "vedei hook --no-daemon". The flag's own help called its
value "a command", but the only thing that works there is a vedei binary, so it
is now run the same way as the others.

The test for this path passed the whole time. It checked only that the CPF was
absent from the output, and a help screen has no CPF in it. It now requires the
output to be the event, with the value redacted, and fails with the old
behaviour put back.

Found while writing the --fail-closed tests, which use the same path.
Fail-open (ADR-005) passes unscanned text through when detection fails,
because breaking a session is worse than missing one redaction. That is right
for most users and wrong where an unscanned CPF reaching a model is a
reportable incident — the regulated environments the project is aimed at.

--fail-closed, on vedei hook, vedei stream and vedei-hook, changes only the
failure path. When detection cannot run — the engine fails, the input is over
the size cap, or the thin client has no daemon and no vedei to fall back to —
the tool output is replaced by a notice that says nothing about the content:

    [vedei: output withheld — detection could not run, and --fail-closed is set]

vedei stream writes only that notice and exits 1. vedei-hook passes the flag on
when it falls back to the full binary, so a fallback that cannot scan withholds
too. A scan that runs still redacts, a clean result still passes, and the
default is unchanged.

Malformed JSON and unrecognized event shapes stay fail-open even with the flag:
there is no tool output in them to withhold. Documented in the hook README.

Tests force a degraded scan with a 5-byte input cap. The fail-closed hook test
fails with the branch disabled.
The thesis behind vedei rests on one number nobody has measured: how often a
developer's agent transcripts hold a credential or a Brazilian document. The way
to measure it is to run the audit on volunteers' machines — which only works if
a volunteer can hand over the result without handing over the data.

"vedei transcript scan --stats" prints counts and nothing else: transcripts
scanned, how many hold a finding, a credential, Brazilian personal data, and per
type how many transcripts and how many distinct values. --json gives the same as
an object a script can aggregate across machines.

The Stats type has no field that could carry a value, a path or a fingerprint,
so nothing can leak through it by mistake. Fingerprints are excluded on purpose,
not only values: a CPF's fingerprint is a hash of eleven digits, few enough to
enumerate, so publishing it would publish the CPF. A test marshals a summary
and fails if any value, redacted value, fingerprint or path appears in it.

Distinct values are counted per machine — one CPF in ten transcripts is one
value — so the study measures exposure, not repetition.
transcript scan and scrub read Claude Code and Codex. Gemini CLI and Cursor
keep the same kind of plaintext record and are not read, and nothing said so —
a user of either would run the audit, see "nothing found", and conclude their
transcripts were clean.

Both READMEs and the command's help now name them, where each keeps its
sessions, and why they are not supported yet:

  Gemini CLI  ~/.gemini/tmp/<project hash>/chats/, JSON files. The format is
              documented upstream but untested here against a real session;
              reading it blind would report coverage it does not have.
  Cursor      state.vscdb, a SQLite database. Reading it needs a SQLite driver,
              and scrub cannot safely rewrite a database.

Not implemented on purpose: support written against a format seen only in
documentation is support that has not been verified.
vedei builds for Windows and Windows 10 1803+ has Unix-domain sockets, so the
daemon would start there — and quietly lose the property that makes it safe.
os.Chmod on Windows only toggles the read-only attribute; the 0600 socket and
0700 directory restrict nobody, because Windows controls access with ACLs. The
socket would be as private as the profile directory's default ACL happens to
make it, which vedei does not check. It is also untested on Windows.

SECURITY.md says so in the daemon section. hooks/README.md tells Windows users
to run "vedei hook --no-daemon" for now — 29 ms per call instead of 8, and
nothing crosses a socket — and lays out the plan: an explicit security
descriptor granting the current user's SID only, via golang.org/x/sys/windows
(already a transitive dependency), a test that reads it back, and a Windows CI
runner. A named pipe with the same descriptor is the fallback design.
transcript/stats_test.go uses the corpus's made-up credential as a finding's
raw value, and its fingerprint had been dropped when corpus/cases/ moved to a
path rule — it only lived in the corpus then. The self-scan caught it in the
test file and in the commit that added it, so the fingerprint comes back with
the reason updated.
@mkmuniz mkmuniz added bug Something isn't working enhancement New feature or request labels Oct 7, 2026
@mkmuniz mkmuniz self-assigned this Oct 7, 2026
@mkmuniz
mkmuniz merged commit 05c87d9 into main Oct 7, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant