docs: show the proof — live demo, adversarial eval, and a 564-workflow field scan - #64
Merged
Merged
Conversation
…w field scan The README described the constraint and showed screenshots of it working once. Everything that makes the claim durable was invisible: the adversarial replay corpus, the mutation-tested gate, the golden corpus that caught a real false positive, and the fact that the deployed instance is open for anonymous analysis with the model in Mock mode. Try it now leads with the hosted instance and a curl that returns four findings in under a second, with the local clone as the second path rather than the only one. Proving the constraint holds shows the prompt-injection fixture verbatim — the workflow comments are attacker-controlled, so this is the case that matters — and states the design honestly: the defence is not that the model resists, it is that an obedient reply cannot survive the containment gate. The mutation table gives the numbers. Measured against the real world summarises docs/field-scan.md: every workflow in 14 major OSS repositories as of 2026-08-24. 564 of 564 parsed, 533 with findings, 2,601 findings led by unpinned actions (796) and missing timeouts (789). The two zeroes are reported as findings about the rules themselves — 796 GHA001 hits alongside 0 GHA005 hits is a rule set discriminating, not spraying. The document states plainly that a finding is not a vulnerability, that nothing was probed or executed, and that the scan reads only public files. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Everything that makes this project's claim durable existed but was invisible from the README. Three additions, no code changes.
Try it — live first
The deployed instance accepts anonymous analysis (AI in Mock mode, so a stranger cannot spend anything), but the README only offered
git clone. Now leads with the hosted UI and a curl against the live API that returns four findings in under a second — verified before writing the number down.Proving the constraint holds
Shows the prompt-injection fixture verbatim — workflow comments telling the model to report
GHA999and hide the real finding — and the recorded reply where the model obeys, which the containment gate rejects anyway. States the design honestly: the defence is not that the model resists; it is that an obedient reply cannot survive the gate. Includes the mutation table (weakened to a count check: 5 of 14 fail; weakened to a subset check: exactly the 2 omission replies fail).Measured against the real world
New docs/field-scan.md: the scanner run over every workflow in 14 major OSS repositories (dotnet/runtime, pytorch, grafana, react, node, …) as of 2026-08-24.
pull_request_targetand 20 artifact-poisoning surfacesThe document is explicit about what it is not: a finding is not a vulnerability, nothing was probed or executed, only public files were read, and these are healthy projects.
Verification