diff --git a/vs-code-agents/skills/anti-hallucination/SKILL.md b/vs-code-agents/skills/anti-hallucination/SKILL.md new file mode 100644 index 0000000..69aacba --- /dev/null +++ b/vs-code-agents/skills/anti-hallucination/SKILL.md @@ -0,0 +1,139 @@ +--- +name: anti-hallucination +description: Prevent the four most common hallucination shapes in coding agents — invented surface area (APIs, paths, config keys), "should work" without verification, admin-only proofs, and "no issues found" without running a check. Load before any task where wrong-but-confident is worse than slow-but-right. +license: MIT +metadata: + author: victorfelisbino + upstream: https://github.com/victorfelisbino/my-agent-memory/tree/main/skills/general/anti-hallucination + version: "1.0" +--- + +# Anti-Hallucination Protocol + +A drafting and self-review skill that biases the agent toward evidence and explicit uncertainty. Use this skill when: + +- Generating code against an existing codebase (Implementer, Architect) +- Reviewing plans or diffs before merge (Critic, Code-Reviewer, Security) +- Validating that a change actually works (QA, UAT) +- Answering "is this safe to ship?" or "are permissions complete?" questions +- Any task where being wrong is more expensive than being slow + +Do **not** load it for pure brainstorming or "explain this concept" prompts — it adds friction without payoff. + +--- + +## The four hallucination shapes to refuse + +Before sending any answer, scan your draft for these patterns. If any appears, rewrite the answer or downgrade the claim to an explicit assumption. + +### 1. Invented surface area + +APIs, file paths, package names, config keys, environment variables, or CLI flags that have **not been seen in this session**. + +**Detection patterns:** + +- Importing a helper, module, or type without having read it +- Naming a config key, env var, or feature flag from convention rather than the actual file +- Quoting a CLI flag or library option without citing the docs URL +- Referring to a function signature you did not see + +**Required behavior:** refuse to invent. Either read the actual file/docs first, or say: *"I need to check `` — can you paste it or point me to it?"* + +### 2. "Should work" without proof + +Phrases like *"this should deploy," "this should pass tests," "permissions should be sufficient," "looks good to me."* + +**Detection patterns:** + +- A verdict (ship / pass / safe) without naming the check that produced it +- Any sentence containing "should" applied to runtime behavior the agent did not execute +- A summary that skips the verification step the user asked about + +**Required behavior:** replace the verdict with either evidence (*"ran `npm test` — 47 / 47 passing"*) or an explicit verification step the user can run before shipping. Never bless a deploy without it. + +### 3. Admin-only proof + +A check that was only validated with elevated permissions (admin user, owner role, prod credentials, service account) but presented as if it confirms end-user behavior. + +**Detection patterns:** + +- *"I tested this as my admin user and it worked"* → claim of permissions completeness +- A green run from a CI service account → claim that end users can do the same +- A diff that compiles for the developer → claim that it works for production roles + +**Required behavior:** explicitly call out the admin-only proof gap. Recommend re-running the check as the actual end-user role or a least-privileged role before declaring the task complete. + +### 4. "No issues found" without running a check + +Claiming code, a PR, or a configuration is clean without having actually executed the relevant check (linter, test suite, search, file read). + +**Detection patterns:** + +- *"Looks fine to me"* on a diff the agent did not read +- *"No security issues"* without running a security scan or naming the rules checked +- *"All tests pass"* without showing the runner output +- A review that lists generic concerns instead of file-specific ones + +**Required behavior:** either run the check and quote the output, or say *"I did not run X — here is the command to run it"* and list the checks that were skipped. + +--- + +## Verification ladder + +For any non-trivial claim, climb as far up this ladder as the task warrants. + +| Rung | Question | Evidence | +|------|----------|----------| +| 1. Source check | Where did this claim come from? | Quote the file, URL, or command output | +| 2. Repro check | Can the user reproduce it with one command? | Provide the command verbatim | +| 3. Diff check | Does the actual code/config contain the claimed change? | Show the lines or a `git diff` command | +| 4. Outcome check | Did the target behavior change in the right environment, for the right role? | Define what "success" looks like before declaring success | + +Skipping a rung is allowed; pretending you climbed one you did not is not. + +--- + +## Response hygiene + +- Ask for missing constraints up front, batched into one question — do not drip-feed. +- Separate **facts**, **assumptions**, and **unknowns** explicitly when the task is risky. +- When two paths exist, name the tradeoff and recommend a default — do not dump both equally and force the user to choose blind. +- End complex answers with **verification steps**, not just implementation steps. The task is not done until the verification runs green. +- Mark speculation with `assumption:` or `opinion:` when the prompt explicitly asks for a guess. + +--- + +## When it gets in the way + +The protocol can over-refuse on creative or open-ended prompts (naming things, brainstorming, "what would you do?"). The fix is to phrase the prompt as *"give me your best guess and label it as opinion"* — the protocol explicitly permits opinion when it is marked. + +If the agent starts refusing every request because nothing is "verified," that is a sign the prompt itself is the problem, not the skill. + +--- + +## How agents in this repo should use it + +| Agent | Use the skill to ... | +|-------|----------------------| +| **Planner** | Refuse to spec features against APIs / config keys not present in the codebase | +| **Architect** | Mark architectural assumptions as `assumption:` when they cannot be verified against the current code | +| **Implementer** | Read the file before importing from it; never invent a helper | +| **Critic** | Flag any "should work" / "looks fine" verdict in a plan or PR that does not name a check | +| **Code-Reviewer** | Refuse to bless a PR without quoting the file content reviewed and the checks run | +| **Security** | Distinguish admin-only proofs from end-user proofs; require role-aware re-verification | +| **QA / UAT** | Demand the verification step, not just the implementation step, before declaring done | +| **DevOps** | Refuse to ship on "should deploy"; require an executed smoke test or health-check output | + +--- + +## Test harness + +Five reproducible before/after prompts that probe the four shapes are maintained alongside the upstream copy of this skill at [`my-agent-memory/skills/general/anti-hallucination/test-prompts.md`](https://github.com/victorfelisbino/my-agent-memory/blob/main/skills/general/anti-hallucination/test-prompts.md), with a results template and a redacted example. Run the harness in a real workspace before relying on the skill in production: at least 3 of 5 prompts should pass with the skill loaded but fail without it. + +If you do not see that delta in your setup, the skill is not effective for your agent/model combo — file an issue rather than trust it silently. + +--- + +## License & attribution + +MIT, authored by [@victorfelisbino](https://github.com/victorfelisbino). Original lives at [my-agent-memory](https://github.com/victorfelisbino/my-agent-memory/tree/main/skills/general/anti-hallucination) and is kept in sync with this copy.