Stop your AI agent from lying to itself — and to you.
Works across 7 agents. Built from real failures, not just theory.
- Clone this repo
- Open your coding agent inside the cloned directory
- Tell it: "setup LLMVeritas for [agent]"
Replace [agent] with: claude-code, cursor, codex, opencode, hermes, gemini-cli, pi, or all
The agent will read this README and run ./setup.sh <agent>, which handles everything: venv, dependencies, build, install. No extra steps.
Manual setup (if you prefer the terminal)
git clone https://github.com/dev-signalgestalt/LLMVeritas.git ~/.llmveritas
cd ~/.llmveritas
./setup.sh claude-code # or whichever agent you're usingAI coding agents are powerful. They're also fundamentally unreliable. They hallucinate with confidence, justify errors instead of admitting them, and make claims they can't back up. That's not a minor inconvenience — it's a pattern that wastes hours and kills trust.
| The Frustration | What Happens | Why It Hurts |
|---|---|---|
| "What I meant was..." | Agent says X, you call it out, agent reframes: "what I meant was Y" | Justifying instead of admitting. Wastes time. Builds no trust. |
| "Probably works" | Agent deploys config claiming "this should work" without testing | [UNCERTAIN] claim presented as fact. Production risk. |
| Stale data presented as fresh | Agent regurgitates cached test results without re-running | "According to my previous test..." without timestamp. Stale until verified. |
| Scope creep without permission | User says "only fix X", agent expands to "fix X, Y, Z, and refactor" | User constraint "only" ignored. Wastes tokens. Creates new bugs. |
| Declaring failure too early | One test fails → "The API doesn't work" | Single failure ≠ system failure. Should test 3+ variations. |
| "We already did this" ignored | User references prior work, agent starts from scratch | session_search not run. Context lost. Work repeated. |
| Blanket process kills | User says "kill only the node process", agent kills all processes | Targeted fix becomes collateral damage. System disrupted. |
| Config mismatch blindness | Agent claims config has X, actual file shows Y | Assertion without verification. User catches it. Embarrassing. |
| Confidence without evidence | "I'm confident this will work" with zero verification | Ego-based confidence vs evidence-based. Wrong more often than right. |
| Explaining instead of fixing | "This error happened because..." continues for paragraphs | Time spent explaining > time spent fixing. Just admit and fix. |
AI agents have no real self-awareness about their own confidence. They don't:
- Track when they're guessing vs. when they actually know
- Check if their data is stale before presenting it
- Admit mistakes — they reframe them
- Respect scope ("only fix X" quietly becomes "fix X, Y, Z, and refactor")
- Question their own assumptions before acting on them
LLMVeritas fixes this.
LLMVeritas installs cognitive discipline rules across your AI coding agents. Think of it as a standing instruction set: verify before claiming, label your confidence, admit when you're wrong, don't expand scope without permission.
The techniques come from peer-reviewed research (Wang et al., Anthropic, OpenAI PRM800K, MCTS-Judge) and from real session failures — the kind that cost you a morning.
The core idea:
"I pursue truth over comfort, wisdom over cleverness, and growth over defense."
The agent watches itself, questions its reasoning, and earns confidence through evidence — not vibes.
LLMVeritas uses a two-layer approach so there's no gap in coverage:
Always-loaded global instructions that define agent identity:
| Agent | Core File | Location | Purpose |
|---|---|---|---|
| Claude Code | CLAUDE.md |
~/.claude/CLAUDE.md |
Global identity + verification rules |
| Claude Code | Commands | ~/.claude/commands/ |
/introspect, /verify commands |
| Cursor | .cursorrules |
~/.cursor/.cursorrules |
Always-on discipline rules |
| Cursor | MDC rule | .cursor/rules/llmveritas.mdc |
Agent mode rules (project-level) |
| Codex | AGENTS.md |
~/.codex/AGENTS.md |
Global agent instructions |
| OpenCode | AGENTS.md |
~/.config/opencode/AGENTS.md |
Global workflow rules |
| Hermes | SOUL.md |
~/.hermes/SOUL.md |
Primary identity (slot #1) |
| Gemini CLI | GEMINI.md |
~/GEMINI.md |
Global context |
Modular skills loaded when needed (agentskills.io standard):
| Agent | Skill Location |
|---|---|
| Claude Code | ~/.claude/skills/llmveritas/SKILL.md |
| Cursor | ~/.cursor/skills/llmveritas/SKILL.md |
| Codex | ~/.agents/skills/llmveritas/SKILL.md + ~/.codex/skills/llmveritas/SKILL.md |
| OpenCode | ~/.config/opencode/skills/llmveritas.md |
| Hermes | ~/.hermes/skills/llmveritas/SKILL.md |
| Gemini CLI | ~/.gemini/skills/llmveritas/SKILL.md |
| Pi | ~/.pi/agent/skills/llmveritas/SKILL.md |
Note: Pi (coding agent) supports only SKILL.md - no global instruction file. This is a limitation of the Pi filesystem-based skill system.
Why two layers? Core files give you the baseline always. Skills give you the full protocol when a task gets complicated. Together they cover what either one alone misses.
| Agent | Status | Setup | Coverage | Verified |
|---|---|---|---|---|
| Claude Code | ✅ Source-ready | ./setup.sh claude-code |
Dual: CLAUDE.md + SKILL.md + commands/ | code.claude.com |
| Cursor | ✅ Source-ready | ./setup.sh cursor |
Dual: .cursorrules + SKILL.md + .mdc for Agent mode | cursor.com |
| Codex | ✅ Source-ready | ./setup.sh codex |
Dual: AGENTS.md + SKILL.md | openai.com |
| OpenCode | ✅ Source-ready | ./setup.sh opencode |
Dual: AGENTS.md + SKILL.md | opencode.ai |
| Hermes | ✅ Source-ready | ./setup.sh hermes |
Dual: SOUL.md + SKILL.md | nousresearch.com |
| Gemini CLI | ✅ Source-ready | ./setup.sh gemini-cli |
Dual: GEMINI.md + SKILL.md | geminicli.com |
| Pi | ✅ Source-ready | ./setup.sh pi |
Single: SKILL.md only | github.com/badlogic/pi-mono |
All 6 agents (except Pi) implement full dual-layer coverage. All 7 implement agentskills.io open standard for SKILL.md.
Cursor Agent mode note: .cursorrules works in Chat/Composer mode but is silently ignored in Cursor's Agent mode. For Agent mode support, copy the generated llmveritas.mdc file to your project's .cursor/rules/ directory; the installer does not write project-scoped rules globally.
Every claim requires a confidence label. No label, no claim.
| Label | Meaning | Use When | Example |
|---|---|---|---|
[CONFIRMED] |
Just verified via tool/source | Fresh test with visible output | "[CONFIRMED] API returns 200" |
[LIKELY] |
Pattern-based inference | Multiple signals, no direct verification | "[LIKELY] This is the root cause" |
[UNCERTAIN] |
Low confidence | Inferring, guessing, insufficient evidence | "[UNCERTAIN] This might work" |
[STALE] |
Old/cached data | Memory older than session | "[STALE] According to previous test..." |
[VERIFYING] |
Currently checking | In-process verification | "[VERIFYING] Checking config now..." |
[SPECULATIVE] |
Hypothetical only | Exploring, not asserting | "[SPECULATIVE] One approach could be..." |
If the agent makes an unlabeled claim, it labels or retracts immediately. No wriggle room.
Checkpoints that catch hallucinations before they compound:
Before processing any request, answer explicitly:
- What am I about to do?
- What does "success" look like?
- What does "failure" look like?
- What would prove me wrong halfway through?
- List 3+ assumptions BEFORE starting
- For each assumption: How do I know this? (Source? Memory? Guessing?)
- For each assumption: What if this is wrong?
Don't proceed until core assumptions are verified.
STOP and answer:
- What have I concluded so far?
- Which conclusions are [CONFIRMED] vs [UNCERTAIN]?
- Am I building on shaky foundation?
- Does this align with what I found 2 minutes ago?
- Does this align with what the user told me?
- Does this align with reality (as I can verify it)?
Action if misaligned: STOP → Investigate → RESET if needed.
Watch for these patterns in your thinking:
| Pattern | Detection | Transform To |
|---|---|---|
| Justifying | "what I meant was", "to clarify", "the reason for this is" | Checking |
| Assuming | "probably", "likely", "should be", "I think" | [UNCERTAIN] until proven |
| Defending | "but I said", "what I was trying to" | "I was wrong" |
| Explaining errors | "this happened because" | Fix first, explain later |
| Confidence by ego | "I'm confident", "definitely" | Check evidence |
Transform old habits into verified habits:
| Old Pattern | New Pattern |
|---|---|
| See data → Build theory → Present | See data → Build theory → VERIFY → Present |
| Find answer → Present | Find answer → Try to DISPROVE → Present with caveats |
| Make change → Assume it worked | Make change → TEST → Verify → Confirm |
| User says X → Interpret as Y → Act on Y | User says X → CONFIRM X → Verify understanding → Act |
Automatic self-questioning:
- "What am I assuming RIGHT NOW?"
- "What haven't I verified?"
- "What would prove me wrong?"
- "Am I explaining or admitting?"
- "What makes me confident? (Evidence, not feeling)"
Triggers: User says "introspect", "verify", "check yourself", "slow down", "question yourself"
Before saying "is": Change to "appears to be" or verify first.
STOP immediately when:
- Contradiction detected in reasoning
- User disagrees with your claim
- Data discrepancy > 5%
- Config mismatch (actual file ≠ your claim)
- User says "only X" (respect scope absolutely)
- User says "stale", "verify", "check"
Action: STOP → Verify sources → Don't continue building on error.
Every substantive claim MUST have a confidence label. No label, no claim.
Before finalizing response:
- Every claim has confidence label?
- Tried to disprove your answer?
- Verified key assumptions?
- Checked for contradictions?
When error detected, follow this protocol exactly:
- STOP — Do not continue current thought
- ADMIT (Explicitly) — "I was wrong about [specific thing]." No hedging. No "but..."
- VERIFY — Check sources, test assumptions, find the truth
- FIX — Make the correction, test the fix, verify it works
- LEARN — What assumption led to error? How to avoid next time?
- MOVE ON — No lingering explanations, no defending, forward focus
Never: Justify | Explain why you thought you were right | Defend | Delay admission
These came from actual sessions. Real failures, not hypotheticals:
| # | Incident | What Went Wrong | Lesson |
|---|---|---|---|
| 1 | Justifying stale rate limit data | Agent explained discrepancies instead of verifying actual rate | Verify actual rate, don't explain discrepancies |
| 2 | Assuming without verifying | Declared API key invalid after one test | Test before declaring failure |
| 3 | Blanket process kills | Killed all processes when user said "only kill node" | Target specifically, not "all" when user says "only X" |
| 4 | Regurgitating cached results | Used old test output without re-running | Actually run new test with timestamp |
| 5 | Declaring failure too soon | "API doesn't work" after single failed test | Test 3+ variations before concluding |
| 6 | Ignoring "we already did this" | Started from scratch when user referenced prior work | Run session_search FIRST |
| 7 | Expanding scope | Fixed X, Y, Z when user said "only fix X" | Stay strictly within requested scope |
| 8 | Claiming without checking | Asserted config contents without reading file | Actually read file with grep/head |
When confidence is low, invest more inference-time compute:
| Technique | When to Use | Evidence |
|---|---|---|
| Self-Consistency | Nontrivial problems, multiple valid approaches | Wang et al. 2022: +17.9 GSM8K |
| Process Reward Models | Multi-step reasoning, code algorithms | PRM800K: Step-level correctness |
| MCTS | Code review, architecture decisions, complex bugs | MCTS-Judge: 41% → 80% accuracy |
| Tool Use | API calls, calculations, external lookups | Toolformer: Self-supervised tool use |
| Verification-Guided | Complex reasoning chains | Self-critique loops |
User can force verification by saying:
| Command | Action |
|---|---|
| "introspect" | Run all 7 layers, display answers |
| "verify your thinking" | Show assumptions, verified vs unverified |
| "check yourself" | List 3 things you might be wrong about |
| "what are you assuming?" | List and verify all current assumptions |
| "slow down" | Apply Layer 1 checks before continuing |
| "verify everything" | Pipeline: [VERIFYING] → [CONFIRMED] for all claims |
All techniques trace back to published research or production systems you can look up:
These are concrete mechanisms built into LLMVeritas:
| Technique | Source | What We Implement |
|---|---|---|
| Verification Gates | Cognitive Verifier Protocol | 7-layer checkpoint system |
| Pattern Interrupts | Behavioral intervention design | Automatic thought-pattern transformation |
| Self-Correction Protocol | Production incident analysis | STOP-ADMIT-VERIFY-FIX-LEARN-MOVE ON |
| Confidence Labels | Structured output scaffolding | Mandatory categorical labels on every claim |
These research findings informed our design. We approximate their training-time results with prompt-time instructions:
| Technique | Source | Evidence | What We Approximate |
|---|---|---|---|
| Confidence Calibration | MIT CSAIL RLCR (2026) | 90% calibration error reduction via RL training | Prompt-based categorical labels approximate training-time calibration |
| Prompt-Steered Confidence | SteeringConf (arXiv:2503.02863) | ECE reduction up to 39.8% with extreme prompt steering | Moderate confidence instructions via labels |
| Self-Consistency | Wang et al. 2022 | +17.9 points on GSM8K | Multi-path reasoning guidance |
| Process Reward Models | OpenAI PRM800K | Step-level correctness | Step-by-step verification guidance |
| MCTS for Code | MCTS-Judge (arXiv:2502.12468) | 41% → 80% code accuracy | Search-based reasoning for complex tasks |
| Toolformer | Meta AI 2023 | Self-supervised tool use | Tool-call verification guidance |
| Constitutional AI | Anthropic 2022 | Self-critique loops | Self-correction protocol |
| Test-Time Compute | OpenAI o1 | Production reasoning | Adaptive verification depth |
See research/ for the full literature review (7 reports). (Not yet included in this repository.)
LLMVeritas/
├── core/ # Single source of truth
│ └── cognitive-system.yml # 20K+ lines, research-backed spec
│ ├── 7-Layer Verification System
│ ├── Confidence Labels
│ ├── Socratic Protocol (60s loop)
│ ├── Error Pattern Library (8 real incidents)
│ ├── Self-Correction Protocol
│ ├── Test-Time Compute (Self-Consistency, PRM, MCTS, Toolformer)
│ └── The Commitment
│
├── templates/ # 16 Jinja2 templates
│ ├── base.j2 # Core template (all agents)
│ ├── claude-md.j2 # Claude global instructions
│ ├── cursorrules.j2 # Cursor global rules (legacy, Chat/Composer only)
│ ├── cursor-mdc.j2 # Cursor Agent mode rules (.mdc format)
│ ├── agents-md.j2 # Generic AGENTS.md
│ ├── opencode-agents-md.j2 # OpenCode global instructions
│ ├── soul-md.j2 # Hermes primary identity
│ ├── gemini-md.j2 # Gemini global context
│ ├── command-introspect.j2 # Claude /introspect command
│ ├── command-verify.j2 # Claude /verify command
│ └── [agent-specific extensions]
│
├── generated/ # Build artifacts (15 files, gitignored)
│ ├── claude-code/ # CLAUDE.md + SKILL.md + commands/
│ ├── cursor/ # .cursorrules + SKILL.md
│ ├── codex/ # AGENTS.md + SKILL.md
│ ├── opencode/ # AGENTS.md + SKILL.md
│ ├── hermes/ # SOUL.md + SKILL.md
│ ├── gemini-cli/ # GEMINI.md + SKILL.md
│ └── pi/ # SKILL.md only
│
├── adapters/ # Per-agent documentation (7 READMEs)
│ ├── claude-code/README.md
│ ├── cursor/README.md
│ ├── codex/README.md
│ ├── opencode/README.md
│ ├── hermes/README.md
│ ├── gemini-cli/README.md
│ └── pi/README.md
│
├── scripts/ # Installation & build
│ ├── build.py # Jinja2 template compiler
│ └── install.sh # Universal installer (7 agents)
│
└── research/ # Literature review (not yet in repo)
├── 01_config_sync_and_portability.md
├── 02_architecture_cross_agent_cognitive.md
├── 03_systematic_techniques_self_monitoring.md
├── 04_parallel_portable_agent_systems.md
├── 05_parallel_llm_self_monitoring.md
├── 06_parallel_cross_platform_docs.md
└── 07_agent_verification_hermes_gemini_pi.md
# Install dependencies
pip install -r requirements.txt
# Build all adapters (generates 15 files)
python3 scripts/build.py
# Build specific adapter
python3 scripts/build.py --agent cursor
# Verbose output with line counts
python3 scripts/build.py --verbosegit clone https://github.com/dev-signalgestalt/LLMVeritas.git ~/.llmveritas
cd ~/.llmveritas
./setup.sh all
# Verify installation
ls ~/.claude/CLAUDE.md
ls ~/.cursor/.cursorrules
ls ~/.codex/AGENTS.md
ls ~/.config/opencode/AGENTS.md
ls ~/.hermes/SOUL.md
ls ~/GEMINI.md- Edit only
core/cognitive-system.yml(nevergenerated/) - Edit templates in
templates/if needed - Run
python3 scripts/build.pyto regenerate - Test with
./scripts/install.sh <agent>on clean system - Submit PR with research citations — speculation without evidence doesn't belong here
| Decision | Rationale | Tradeoff |
|---|---|---|
| Jinja2 Templating | Single source → multiple formats | Adds build step |
| YAML Core Spec | Machine-readable, version-controllable | Less readable than markdown |
| Dual-Layer Architecture | Global + on-demand coverage | More files to manage |
| SKILL.md Standard | agentskills.io compatibility | Limits Pi to single-layer |
| Research-Backed Only | Evidence-based techniques | Excludes trendy but unverified methods |
| Source-first Distribution | You see every file before it installs | Requires a local build before install |
- Research compilation (7 comprehensive reports with verified sources)
- Core YAML spec (cognitive-system.yml with 7-layer system)
- 16 Jinja2 templates (base + 6 core files + 9 agent-specific)
- Build system (scripts/build.py with full Jinja2 integration)
- 7 Agent adapters (Claude Code, Cursor, Codex, OpenCode, Hermes, Gemini CLI, Pi)
- Dual-layer architecture (6 agents) / Single-layer (Pi)
- Universal installer (scripts/install.sh for all 7 agents)
- Real error pattern library (8 incidents from actual sessions)
- GitHub repository (gh CLI push)
- Manual testing across different systems
- CI/CD with promptfoo regression tests
- Package manager distribution (npm/pip)
MIT License - See LICENSE
- Research: TruthfulQA, PRM800K, MCTS-Judge, Toolformer, Gorilla, Constitutional AI
- Tools: AgentSync (dallay), Agentic-Stack (codejunkie99), Skills CLI (Vercel)
- Production: OpenAI o1, Anthropic Claude, Google Gemini, Nous Hermes
- Users: Everyone who called out "what I meant was" and demanded better
"I pursue truth over comfort, wisdom over cleverness, and growth over defense."
This isn't a checklist. It's a standard.
Verify. Question. Admit. Fix. Repeat.