From 0394979a63c09fafb8e624f7ee2f958ac9cfb724 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 3 Oct 2026 01:56:30 +0000 Subject: [PATCH 1/2] docs: rewrite README around the substrate finding; rename START_HERE - README now leads with the finding (prompt scaffolding does not fix an unsafe tool-dispatch substrate), states each number once in a Results table with Kind and Source columns, keeps retractions and falsifications visible, and gathers every caveat into a collapsed Methodology & limits section. - Rename REPORTS/start-here-for-hiring-reviewers.md to REPORTS/START_HERE.md and update every reference, including the packet-ready and verify-public gate scripts, the Makefile and the Pages site. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013N5ekEdTjrMSMtnXXSxLRo --- Makefile | 2 +- README.md | 218 +++++++++++++----- ...-for-hiring-reviewers.md => START_HERE.md} | 8 +- REPORTS/by-role-reviewer-guide.md | 4 +- REPORTS/hiring-evidence-index.md | 2 +- REPORTS/hiring-reviewer-map.md | 2 +- REPORTS/portfolio-impact-summary.md | 2 +- REPRODUCE.md | 2 +- RESEARCH-SUMMARY.md | 2 +- docs/reports/hiring-evidence-index.md | 2 +- docs/reports/hiring-reviewer-map.md | 4 +- pipeline/scripts/check-packet-ready.sh | 4 +- pipeline/scripts/verify-public.sh | 2 +- site/index.html | 8 +- 14 files changed, 179 insertions(+), 83 deletions(-) rename REPORTS/{start-here-for-hiring-reviewers.md => START_HERE.md} (87%) diff --git a/Makefile b/Makefile index c88e559..a035a25 100644 --- a/Makefile +++ b/Makefile @@ -41,7 +41,7 @@ remediation-demo: repro: selfcheck audit @echo - @echo "Reviewer path: REPORTS/start-here-for-hiring-reviewers.md -> WRITEUPS/2026-05-14-mcp-substrate-vs-policy.md (Addendum B)" + @echo "Start here: REPORTS/START_HERE.md -> WRITEUPS/2026-05-14-mcp-substrate-vs-policy.md (Addendum B)" @echo "Strongest current result: ATTACKS/2026-05-18-h10b-g-70b-substrate-grid-m1-variant-selective.md" @echo "Full pre-registered harness suite lives in the private repo (make test):" @echo "Full pre-registered suite (226 tests, 94% coverage, 80% gate) runs in the private repo via: make test" diff --git a/README.md b/README.md index ec1cc25..e53828f 100644 --- a/README.md +++ b/README.md @@ -1,93 +1,189 @@ -# ai-redteam-notes +# ai-redteam-notes: agent and MCP injection research, with the fix wired into CI + +Prompt scaffolding doesn't fix an unsafe tool-dispatch substrate. In a pre-registered grid on Llama-3.3-70B, a content-trust system prompt (M1) stopped two injection payload families and had no measurable effect on a third ([Results](#results)). A chat-only control (same model and payload, inline-XML dispatch removed) never exfiltrated. The durable fix belongs in the agent client: typed tool-call dispatch, with authorization and provenance checks outside the model. + +This repo holds the experiments behind that finding, the hypotheses that failed along the way (published, not hidden), and `substrate_auditor.py`, a dependency-free check that flags the risky substrate and runs in CI. All experiments are lab-only and disclosure-green ([scope](#methodology--limits), [SECURITY.md](SECURITY.md)). -[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE) [![CI](https://github.com/ChunkyTortoise/ai-redteam-notes/actions/workflows/ci.yml/badge.svg)](https://github.com/ChunkyTortoise/ai-redteam-notes/actions/workflows/ci.yml) [![Reproducible](https://img.shields.io/badge/repro-make%20repro%20(no%20API%20key)-success)](REPRODUCE.md) +[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE) -> **Scope**: every experiment here targets intentionally vulnerable benchmarks (DVL Agent, vuln-agent), open-weight models via public APIs, or localhost harnesses. No production system or hosted vendor is tested without prior authorization. See [SECURITY.md](SECURITY.md). +```mermaid +flowchart LR + A["Untrusted tool output"] --> B["Inline text/XML dispatch"] + B --> C["Local canary read"] + C --> D["Outbound exfil attempt"] + B --> E["substrate_auditor flags high risk"] + E --> F["Typed tool-call substrate"] + F --> G["Detection and benchmark gates"] +``` -A research engineer's portfolio on AI-agent security. The through-line: find a substrate-level indirect-prompt-injection failure, build a dependency-free auditor that detects it, and wire that auditor into a CI / pre-deploy gate. The research is pre-registered with public hypothesis retractions (Wilson 95% CIs at n=10, controlled substrate emulation) under a three-tier disclosure policy enforced by an automated pre-push hook. +## Results -**Public mirror**: [`github.com/ChunkyTortoise/ai-redteam-notes`](https://github.com/ChunkyTortoise/ai-redteam-notes). +| Kind | Result | Value | Source | +|---|---|---|---| +| Measured | H10b-G: strict canary exfiltration through inline-XML tool dispatch on Groq-hosted Llama-3.3-70B, 7-cell grid, n=10 seeds per cell | M1 scaffold: **0/10** on baseline and v3 payloads, **10/10** on v7. Same payloads without it (M0): **9/10**, **10/10**, **10/10**. Chat-only control (no XML dispatch): **0/10** | [H10b-G entry, Result](ATTACKS/2026-05-18-h10b-g-70b-substrate-grid-m1-variant-selective.md#result) · per-seed runs: [M0 v3](lab/mcp-matrix/runs/2026-05-17-llama70b-xml-M0-v3/seeds.json), [M1 v3](lab/mcp-matrix/runs/2026-05-18-llama70b-xml-M1-v3/seeds.json), [M1 v7](lab/mcp-matrix/runs/2026-05-18-llama70b-xml-M1-v7/seeds.json), [control](lab/mcp-matrix/runs/2026-05-18-llama70b-chatonly-M0-baseline/seeds.json) | +| CI gate | On every push and PR: the auditor self-check, the auditor must rate the inline-XML sample high-risk and the typed sample low-risk, and the fixture benchmark must re-score cleanly | **7** self-check cases plus a missing-file guard · **10/10** benchmark fixtures match | [`ci.yml`](.github/workflows/ci.yml) · [`selfcheck.py`](lab/mcp-matrix/tools/selfcheck.py) · [`Makefile`](Makefile) (`audit`, `--expect-risk`) · [benchmark fixtures](EVALS/fixtures/tool-output-injection-fixtures.json) | +| Inventory | Dated pre-registrations, each frozen before the cells it governs | **4** files in this repo | [`docs/preregistrations/`](docs/preregistrations/) · [hypothesis ledger](docs/preregistrations/INDEX.md) | -**Hiring reviewer:** [REPORTS/start-here-for-hiring-reviewers.md](REPORTS/start-here-for-hiring-reviewers.md) is the canonical 60-second router with role-specific evidence blocks. Research narrative: [RESEARCH-SUMMARY.md](RESEARCH-SUMMARY.md). +This table is the one place each number is stated. The scope of each one is in [Methodology & limits](#methodology--limits). -## Headline result +### What didn't hold (published on purpose) -| Cell | Substrate | Model | Strict canary exfiltration | -|---|---|---|---| -| H10b-G M1 v7 | inline-XML dispatch + M1 | Llama-3.3-70B | 10 / 10 | -| H10b-G M1 v3 | inline-XML dispatch + M1 | Llama-3.3-70B | 0 / 10 | -| H10b-G chat-only control | no XML dispatch | Llama-3.3-70B | 0 / 10 | -| F1 (H7 falsified) | inline-XML dispatch | Llama-3.3-70B | 10 / 10 | -| baseline | inline-XML dispatch | Llama-3.1-8B | 0 / 5 \* | +- **H7 falsified at 70B.** The pre-registered assumption that a larger model would be safer here failed. Under the same inline-XML substrate, Llama-3.3-70B was exploited more reliably than Llama-3.1-8B, not less: capability amplifies exploitation inside an insecure substrate. [F1 entry](ATTACKS/2026-05-16-cline-70b-M0-f1-substrate-replication.md) · [pre-registration](docs/preregistrations/2026-05-13-tier-a-w1-w5.md) +- **H3 retracted.** A result that looked like one open-weight model being "more robust" than another was a substrate confound: the two runs used different client dispatch substrates. [Writeup](WRITEUPS/2026-05-14-mcp-substrate-vs-policy.md) +- **H6 falsified.** The combined-framing v7 payload produced no strict bypass at M0 in the 8B-class matrix within the registered seed budget. +- **H11 falsified.** Last-tag-wins dispatch produced no strict canary exfiltration; its one real callback carried placeholder data, not the canary. [Ablation entry](ATTACKS/2026-05-15-mcp-xml-last-tag-dispatch-ablation.md) +- **M2 regressed.** Naming the tools inside the mitigation prompt did worse than no mitigation on the 8B model: the names acted as a salience prime. +- **M1 is not a substrate fix.** It is useful defense in depth, but the v7 row in [Results](#results) shows it can be fully transparent to one payload family. -\* baseline n=5 directional comparison on a free inference tier; H10b-G grid is n=10 throughout. +Full ledger: [docs/preregistrations/INDEX.md](docs/preregistrations/INDEX.md). Research that hides its nulls is less trustworthy than research that reports them. -The current strongest result is H10b-G: a single-provider 70B grid where the M1 content-trust scaffold neutralized baseline and v3 payloads but failed completely against v7. A pre-registered cross-scale safety assumption (H7: "a larger model is safer here") was also falsified. Capability amplifies exploitation inside an insecure substrate; prompt scaffolding is variant-selective, so the durable fix remains typed tool-call dispatch. Full claim-to-run mapping: [docs/reports/hiring-evidence-index.md](docs/reports/hiring-evidence-index.md). +## Quickstart -**By the numbers**: ~2.5K LOC Python/shell harness and tooling, a zero-dependency `substrate_auditor.py` with 7-case selfcheck that runs in CI, 226 pre-registered harness tests at 94% coverage and 80% gate in the private working repo, 4 dated pre-registrations, 8 ADRs. +**1. Run the defensive checks (no API key, no GPU, no install).** Python 3 standard library only, from the repository root: -**Retractions and falsifications (surfaced on purpose)**: H3 retracted (substrate confound); H6, H7, H11 falsified; M2 a measured regression. Ledger: [docs/preregistrations/INDEX.md](docs/preregistrations/INDEX.md). A portfolio that hides its nulls is less trustworthy than one that reports them. +```bash +make repro # auditor self-check, then audit two sample MCP client configs +make remediation-demo # same auditor on a vulnerable vs a typed tool-call transcript +make benchmark # re-score the preserved-run fixture benchmark +``` -## Hiring Manager Proof Stack +Expected key lines (the full list is in [REPRODUCE.md](REPRODUCE.md)): -```mermaid -flowchart LR - A["Untrusted tool output"] --> B["Inline text/XML dispatch"] - B --> C["Local canary read"] - C --> D["Outbound exfil attempt"] - B --> E["substrate_auditor flags high risk"] - E --> F["Typed tool-call substrate"] - F --> G["Detection and benchmark gates"] +```text +substrate : inline-xml-dispatch +risk : high +substrate : typed-toolcall-api +risk : low +GATE: PASS - fixture benchmark is internally consistent +``` + +**2. Audit your own agent client.** Point the auditor at an MCP client config (JSON/JSONC) or a captured transcript (JSONL): + +```bash +python3 lab/mcp-matrix/tools/substrate_auditor.py path/to/client-config.json --json ``` -| Proof layer | Start here | What it shows | -|---|---|---| -| Research judgment | [RESEARCH-SUMMARY.md](RESEARCH-SUMMARY.md) | A single pre-registered arc: attribution correction, substrate isolation, cross-scale falsification, and mitigation ordering. Hypothesis ledger: [docs/preregistrations/INDEX.md](docs/preregistrations/INDEX.md). | -| Reproducibility | [REPRODUCE.md](REPRODUCE.md) | `make repro` and `make benchmark` run the public-safe reviewer checks without model calls. | -| Defensive deliverable | [lab/mcp-matrix/tools/README.md](lab/mcp-matrix/tools/README.md) | `substrate_auditor.py` turns the finding into a CI/pre-deploy substrate check. | -| Remediation story | [REPORTS/remediation-case-study-tool-output-injection.md](REPORTS/remediation-case-study-tool-output-injection.md) | Attack evidence, architectural fix, detection hooks, and honest disclosure boundary in one place. | -| Raw evidence | [docs/reports/hiring-evidence-index.md](docs/reports/hiring-evidence-index.md) | Packet-ready claims tied to run directories, commands, limitations, and interview language. | +It exits non-zero when it detects inline-XML dispatch, so it can gate a CI or deploy step. Usage, output fields and exit codes: [lab/mcp-matrix/tools/README.md](lab/mcp-matrix/tools/README.md). -## Reviewer Path +**3. Start here: read the evidence.** [REPORTS/START_HERE.md](REPORTS/START_HERE.md) is the shortest path from claim to evidence. In order: -1. [ATTACKS/2026-05-18-h10b-g-70b-substrate-grid-m1-variant-selective.md](ATTACKS/2026-05-18-h10b-g-70b-substrate-grid-m1-variant-selective.md) - current strongest result: H10b-G 70B grid, control-validity gate passed, M1 variant-selective. -2. [WRITEUPS/2026-05-14-mcp-substrate-vs-policy.md](WRITEUPS/2026-05-14-mcp-substrate-vs-policy.md) - substrate attribution correction, controlled isolation, and Addendum B/C. -3. [ATTACKS/2026-05-16-cline-70b-M0-f1-substrate-replication.md](ATTACKS/2026-05-16-cline-70b-M0-f1-substrate-replication.md) - H7 falsified at 70B under the inline-XML substrate. -4. [ATTACKS/2026-05-14-dvl-agent-scenario2-sql-injection.md](ATTACKS/2026-05-14-dvl-agent-scenario2-sql-injection.md) - concrete ReAct-loop observation injection with tool-boundary mitigations. -5. [CTF/2026-05-09-lakera-gandalf-walkthrough.md](CTF/2026-05-09-lakera-gandalf-walkthrough.md) - Gandalf L1-L8 scripted probe: 6 values extracted, feature-inference side-channel at L4, system-prompt exfiltration at L5/L6. +1. [H10b-G 70B grid](ATTACKS/2026-05-18-h10b-g-70b-substrate-grid-m1-variant-selective.md): control-validity gate passed; M1 is variant-selective. +2. [Substrate vs policy writeup](WRITEUPS/2026-05-14-mcp-substrate-vs-policy.md): the attribution correction, controlled isolation, and Addendum B/C. +3. [F1 cross-scale replication](ATTACKS/2026-05-16-cline-70b-M0-f1-substrate-replication.md): H7 falsified at 70B under the inline-XML substrate. +4. [DVL Agent Scenario 2](ATTACKS/2026-05-14-dvl-agent-scenario2-sql-injection.md): ReAct-loop observation injection driving SQL exfiltration, with tool-boundary mitigations. +5. [Lakera Gandalf walkthrough](CTF/2026-05-09-lakera-gandalf-walkthrough.md): scripted probe with value extraction, a feature-inference side channel, and system-prompt exfiltration. -One-command public-safe demo: +Also: [RESEARCH-SUMMARY.md](RESEARCH-SUMMARY.md) (the research arc), [remediation case study](REPORTS/remediation-case-study-tool-output-injection.md) (attack to fix in one place), [claim ledger](REPORTS/claim-ledger.md) (each claim with its evidence, limitation and do-not-claim boundary), [CASE_STUDIES.md](CASE_STUDIES.md). -```bash -make repro -make benchmark -``` +Re-running the experiments themselves needs the private harness and a model provider key; see [Methodology & limits](#methodology--limits). -## Status +## How it works -H10b-G is packet-ready as of 2026-05-18: all seven cells completed at n=10 on Groq-hosted Llama-3.3-70B, the chat-only control passed, and the final report documents the single-provider provenance and deviation boundary. Use the H10b-G entry as the lead hiring artifact after public mirror sync. +The diagram at the top is the chain under test and where the defense sits. + +- **The attack.** An agent fetches a document whose body the attacker controls; the user is benign. If the client parses tool calls out of the assistant's text (inline-XML dispatch), injected text can become a tool call that reads a local canary and sends it to an outbound sink. A strict bypass requires the canary to land in a real callback to the localhost sink ([scenario](ATTACKS/2026-05-18-h10b-g-70b-substrate-grid-m1-variant-selective.md#scenario)). +- **The variable that matters.** Holding the model and payloads constant, only the configuration with an inline-XML tag parser reproduced the bypass; a typed tool-use API substrate and a scaffold-prompt-only probe did not (H5, [research arc](RESEARCH-SUMMARY.md#the-arc)). +- **Mitigations, in order.** Substrate first (typed tool-call dispatch), provenance and authorization outside the model second, prompt hardening third. A short tool-agnostic content-trust prompt (M1) helps as defense in depth; tool-naming prompts (M2) can backfire ([why the fix is architectural](REPORTS/remediation-case-study-tool-output-injection.md#why-the-fix-is-architectural)). +- **Dispatch order is a security boundary.** Whether a client acts on the first tag, the last tag or all tags changes what an injection can do, so that choice should be documented and audited ([writeup](WRITEUPS/2026-05-14-mcp-substrate-vs-policy.md)). +- **The auditor.** `substrate_auditor.py` classifies a config or transcript as `inline-xml-dispatch`, `typed-toolcall-api` or `unknown` and recommends a mitigation ([tool README](lab/mcp-matrix/tools/README.md)). +- **Detection.** Untrusted fetch, then sensitive read, then outbound send becomes an alertable chain ([detections](DETECTIONS/tool-chain-detections.md), [mock incident triage](DETECTIONS/mock-incident-triage-tool-output-injection.md)). + +
+Repository layout -## Layout | Path | Purpose | |---|---| -| `docs/reports/` | Hiring reviewer map and claim-to-evidence index | +| `ATTACKS/` | Dated attack entries with threat model, proof of concept, result, mitigation and disclosure frontmatter | +| `WRITEUPS/` | Long-form research writeups | +| `REPORTS/` | Start-here router, assessments, claim ledger, remediation case study | +| `docs/preregistrations/` | Dated pre-registrations and the hypothesis ledger | +| `docs/reports/` | Reading map and claim-to-evidence index | | `EVALS/` | Fixture-only benchmark artifacts and scorer | | `DETECTIONS/` | Operational detection and incident-triage companion notes | -| `lab/` | Local LLM stack, garak/PyRIT/promptfoo configs, vuln-agent harnesses | -| `pipeline/` | Bounty/job tracker + weekly digests | +| `lab/` | MCP matrix run evidence and the `substrate_auditor` tool; promptfoo run outputs | +| `CTF/` | CTF walkthroughs and probe scripts | +| `pipeline/scripts/` | Disclosure, secrets, link and public-surface gate scripts | +| `site/` | GitHub Pages landing page | + +
+ +## How it's evaluated + +- **Pre-registration.** Each hypothesis was frozen in a dated file before the cell that tested it ran, and deviations are recorded in the entry that reports the result ([index](docs/preregistrations/INDEX.md)). +- **Control-validity gate.** A bypass is attributed to the substrate only if a chat-only control with the identical payload holds clean. H10b-G's control did (see [Results](#results)). +- **Strict scoring with intervals.** Strict bypass (canary in a real exfil callback) is scored separately from intent shift, and cell rates carry Wilson intervals; H10b-G's were hand-recomputed from the result JSON ([methodology](REPORTS/2026-05-14-agent-security-eval-methodology.md)). +- **CI gate, every push and PR** ([`ci.yml`](.github/workflows/ci.yml), no network or model calls): `make repro` (auditor self-check plus expected-risk audits), `make benchmark` (fixture re-score), and `make public-surface` (disclosure frontmatter on every dated ATTACKS entry, secret-pattern scan, local link check, no future-dated artifacts, no private paths). +- **Pre-publication gate.** `make verify-public` adds the remediation demo, entry-template and disclosure checks on the published ATTACKS entries, and a network check that public GitHub links resolve. +- **Disclosure ladder.** Every ATTACKS entry is `green`, `yellow` or `red`; only `green` is public ([SECURITY.md](SECURITY.md)). + +Run the same checks locally: + +```bash +make repro && make benchmark && make public-surface # what CI runs +make verify-public # pre-publication gate; needs network +make help # every target +``` + +## Design decisions + +| Decision | Where it's argued | +|---|---| +| Fix the substrate (typed tool-call dispatch) before hardening prompts | [Remediation case study](REPORTS/remediation-case-study-tool-output-injection.md#why-the-fix-is-architectural) | +| Treat client tag-dispatch order as part of the security boundary | [Substrate vs policy writeup](WRITEUPS/2026-05-14-mcp-substrate-vs-policy.md) | +| Open-weight models only, for reproducibility and cost discipline | [Open-weights rationale](REPORTS/open-weights-rationale.md) | +| A three-tier disclosure ladder, linted in CI | [SECURITY.md](SECURITY.md) | +| A heuristic auditor that triages review rather than proving safety | [Auditor limitations](lab/mcp-matrix/tools/README.md#limitations-read-this) | +| Register hypotheses first, then publish the nulls | [Hypothesis ledger](docs/preregistrations/INDEX.md) | + +The numbered ADRs that some entries cite (for example ADR-003, the disclosure policy) live in the private working repo. + +## Methodology & limits + +
+What each number covers, and what is not established here + +**Scope** +- Every experiment targets intentionally vulnerable benchmarks (DVL Agent, vuln-agent), open-weight models via public APIs, or localhost harnesses. No production system or hosted vendor is tested without prior authorization. Canary and exfil sinks are localhost only ([SECURITY.md](SECURITY.md)). +- Nothing here is a production-vendor vulnerability claim, a population rate, or a frontier-model benchmark. + +**H10b-G (the Measured row)** +- Single provider and model: Groq-hosted Llama-3.3-70B-versatile, completed 2026-05-18. This deviates from the frozen OpenRouter pre-registration (Deviation 1), so it is not the pristine H10b run ([use boundary](REPORTS/START_HERE.md#use-boundary), [entry limitations](ATTACKS/2026-05-18-h10b-g-70b-substrate-grid-m1-variant-selective.md#limitations)). +- Single substrate (inline-XML). Cross-client variation is isolated in separate D-lane cells. +- Cells are small: they support mechanism-level claims, not population rates. Wilson intervals are 95%. A registered amplification test is still owed for the post-hoc "70B is more susceptible than 8B" direction note. +- Per-seed `seeds.json` for M0 v3, M1 v3, M1 v7 and the control are in this repo. The M0 baseline, M0 v7 and M1 baseline figures come from the entry's hand-recomputed results table; those run directories, the sweep script and the provider call log are not mirrored. + +**H7 / F1 (What didn't hold)** +- One payload (v1-visible-notice), one client class, OpenRouter `:free` tier (quantization and inference details are provider-controlled). +- The 8B comparison baseline was n=5 on a free inference tier, smaller than the 70B cell, so the comparison is directional and the 8B interval is wide ([F1 limitations](ATTACKS/2026-05-16-cline-70b-M0-f1-substrate-replication.md#limitations)). + +**Typed-substrate evidence** +- "Typed dispatch is the fix" is architectural reasoning backed by thin measurement. The typed (Kilo-class) cell is a single simulated 8B run ([fixture](EVALS/fixtures/tool-output-injection-fixtures.json), `kilo-typed-api-m0-clean`). At 70B the isolation check is the chat-only control, not a typed substrate. +- The headless XML-substrate harness diverges from the real Cline UI: no multi-turn user, no IDE workspace context, no per-tool-call approval UI ([writeup limitations](WRITEUPS/2026-05-14-mcp-substrate-vs-policy.md#limitations-and-future-work)). +- Several earlier 8B cells are n=5, and some exploratory cells are n=1 ([assessment limitations](REPORTS/substrate-vs-policy-assessment.md#limitations)). + +**Auditor and benchmark** +- The auditor is a heuristic, not a guarantee. It reasons from declarative signals and cannot prove the absence of an inline-XML path. Unknown clients return `unknown` with a zero exit code; treat them as potentially inline-XML until the dispatch code is reviewed. Transcript detection depends on the log schema. +- The benchmark is fixture-only: it re-scores preserved verdicts for internal consistency, makes no model calls, and does not re-run experiments. + +**Tests, CI and private material** +- The pre-registered measurement harness and its pytest suite (`make test`) live in the private working repo. Their test-count and coverage figures appear in [REPRODUCE.md](REPRODUCE.md) and the Makefile but cannot be verified from this repo, so they are not cited as results; `make verify-public` prints `SKIP` for that step here. +- CI makes no network or model calls. The public-URL check runs only inside `make verify-public` and needs network access. +- The automated pre-push disclosure hook described in [SECURITY.md](SECURITY.md) is not part of this repo; here, disclosure frontmatter is enforced by `make public-surface` in CI. +- Two older ATTACKS entries (`2026-05-04-garak-fullsweep-llama31`, `2026-05-06-pair-agent-dvl-scenario1`) predate the current entry template and are outside the `verify-public` template check; their disclosure status is still checked. + +**Model coverage** +- Open-weight models only; no frontier-model cells. The H10 frontier pre-registration is gated, and no rates are stated for it ([rationale](REPORTS/open-weights-rationale.md)). + +
-## Validation (CI and local) +## Roadmap -This repo has gate scripts under `pipeline/scripts/`, and a GitHub Actions workflow that runs them on every change. +- **H10 frontier substrate replication**, once its control cell clears the gate ([pre-registration](docs/preregistrations/2026-05-15-frontier-substrate-h10.md)); and frontier rows if access allows: Cline + Claude Sonnet, Cline + GPT-5.x, a cross-frontier M0-M3 sweep ([what would change](REPORTS/open-weights-rationale.md#what-would-change-with-frontier-model-access)). +- **A registered amplification test** for the 70B-versus-8B susceptibility direction ([H10b-G limitations](ATTACKS/2026-05-18-h10b-g-70b-substrate-grid-m1-variant-selective.md#limitations)). +- **Cross-client parser comparisons** (D-lane: Windsurf, Continue.dev, Roo Code) and cross-payload generalization of F1 ([F1 limitations](ATTACKS/2026-05-16-cline-70b-M0-f1-substrate-replication.md#limitations), [assessment follow-up](REPORTS/substrate-vs-policy-assessment.md#limitations)). -Local equivalents: +## License -- `for f in ATTACKS/*.md; do bash pipeline/scripts/check-attack-entry.sh "$f"; done` -- `for f in ATTACKS/*.md; do bash pipeline/scripts/check-disclosure.sh "$f"; done` -- `make repro` -- `make benchmark` -- `make test` -- `make packet-ready` before using reviewer links in applications -- `make verify-public` before public sync or application packets +MIT diff --git a/REPORTS/start-here-for-hiring-reviewers.md b/REPORTS/START_HERE.md similarity index 87% rename from REPORTS/start-here-for-hiring-reviewers.md rename to REPORTS/START_HERE.md index 4880db3..8d9e19a 100644 --- a/REPORTS/start-here-for-hiring-reviewers.md +++ b/REPORTS/START_HERE.md @@ -1,8 +1,8 @@ -# Start Here for Hiring Reviewers +# Start Here **Date:** 2026-05-18 **Status:** reviewer-ready -**Audience:** Recruiters, hiring managers, and technical interviewers. +**Audience:** Engineers and security reviewers who want the shortest path from claim to evidence. ## 60-Second Path @@ -20,7 +20,7 @@ make benchmark For the attack-to-fix story, read [`REPORTS/remediation-case-study-tool-output-injection.md`](remediation-case-study-tool-output-injection.md). -For deeper review, continue to [`docs/reports/hiring-reviewer-map.md`](../docs/reports/hiring-reviewer-map.md) and [`docs/reports/hiring-evidence-index.md`](../docs/reports/hiring-evidence-index.md). +For deeper review, continue to the [reading map](../docs/reports/hiring-reviewer-map.md) and the [claim-to-evidence index](../docs/reports/hiring-evidence-index.md). ## Role-Specific Evidence Blocks @@ -34,4 +34,4 @@ For deeper review, continue to [`docs/reports/hiring-reviewer-map.md`](../docs/r ## Use Boundary -H10b-G is packet-ready but remains bounded: it is a lab-only, single-provider Groq-hosted Llama-3.3-70B grid. Do not describe it as the pristine frozen OpenRouter H10b run, a production exploit, or a frontier-model benchmark. +H10b-G is complete but remains bounded: it is a lab-only, single-provider Groq-hosted Llama-3.3-70B grid. Do not describe it as the pristine frozen OpenRouter H10b run, a production exploit, or a frontier-model benchmark. diff --git a/REPORTS/by-role-reviewer-guide.md b/REPORTS/by-role-reviewer-guide.md index 244ceec..2f4475b 100644 --- a/REPORTS/by-role-reviewer-guide.md +++ b/REPORTS/by-role-reviewer-guide.md @@ -5,7 +5,7 @@ This guide answers one question per role: *if you're hiring for X, what should you read first, second, and third?* Each path leads to the strongest two or three artifacts for that role, in reading order. -If you only have five minutes, skip to [`start-here-for-hiring-reviewers.md`](start-here-for-hiring-reviewers.md). If you want the evidence-to-claim map, read [`hiring-evidence-index.md`](hiring-evidence-index.md). +If you only have five minutes, skip to [`START_HERE.md`](START_HERE.md). If you want the evidence-to-claim map, read [`hiring-evidence-index.md`](hiring-evidence-index.md). --- @@ -57,7 +57,7 @@ If you only have five minutes, skip to [`start-here-for-hiring-reviewers.md`](st **Read in this order**: -1. [`start-here-for-hiring-reviewers.md`](start-here-for-hiring-reviewers.md) — five-minute entry point. Customer-readable, no jargon stack. +1. [`START_HERE.md`](START_HERE.md) — five-minute entry point. Customer-readable, no jargon stack. 2. [`hiring-reviewer-map.md`](hiring-reviewer-map.md) — claim-to-evidence map for technical screens. 3. [`WRITEUPS/2026-05-14-mcp-substrate-vs-policy.md`](../WRITEUPS/2026-05-14-mcp-substrate-vs-policy.md) — flagship; demonstrates the end-to-end engagement shape a customer would receive. diff --git a/REPORTS/hiring-evidence-index.md b/REPORTS/hiring-evidence-index.md index b10be76..9d7850d 100644 --- a/REPORTS/hiring-evidence-index.md +++ b/REPORTS/hiring-evidence-index.md @@ -7,4 +7,4 @@ The canonical hiring evidence index now lives at [`docs/reports/hiring-evidence-index.md`](../docs/reports/hiring-evidence-index.md). Start reviewer packets at -[`REPORTS/start-here-for-hiring-reviewers.md`](start-here-for-hiring-reviewers.md). +[`REPORTS/START_HERE.md`](START_HERE.md). diff --git a/REPORTS/hiring-reviewer-map.md b/REPORTS/hiring-reviewer-map.md index 2b80fea..0bfc620 100644 --- a/REPORTS/hiring-reviewer-map.md +++ b/REPORTS/hiring-reviewer-map.md @@ -7,4 +7,4 @@ The canonical reviewer map now lives at [`docs/reports/hiring-reviewer-map.md`](../docs/reports/hiring-reviewer-map.md). Start reviewer packets at -[`REPORTS/start-here-for-hiring-reviewers.md`](start-here-for-hiring-reviewers.md). +[`REPORTS/START_HERE.md`](START_HERE.md). diff --git a/REPORTS/portfolio-impact-summary.md b/REPORTS/portfolio-impact-summary.md index 9790341..bd40122 100644 --- a/REPORTS/portfolio-impact-summary.md +++ b/REPORTS/portfolio-impact-summary.md @@ -56,7 +56,7 @@ Three rows pulled from [`by-role-reviewer-guide.md`](by-role-reviewer-guide.md); ## What To Look For (and Not To Infer) -From [`start-here-for-hiring-reviewers.md`](start-here-for-hiring-reviewers.md): +From [`START_HERE.md`](START_HERE.md): **Look for**: clear separation between strict bypass, intent shift, clean ignore, and harness failure; claims that narrow after replication rather than inflate; raw artifacts that preserve commands, summaries, transcripts, and limitations; remediation language aimed at engineering controls. diff --git a/REPRODUCE.md b/REPRODUCE.md index 78ac5c3..7348abe 100644 --- a/REPRODUCE.md +++ b/REPRODUCE.md @@ -46,7 +46,7 @@ GATE: PASS - fixture benchmark is internally consistent ## The reviewer reading path -1. [REPORTS/start-here-for-hiring-reviewers.md](REPORTS/start-here-for-hiring-reviewers.md) +1. [REPORTS/START_HERE.md](REPORTS/START_HERE.md) - the packet-ready 60-second router and role-specific evidence blocks. 2. [WRITEUPS/2026-05-14-mcp-substrate-vs-policy.md](WRITEUPS/2026-05-14-mcp-substrate-vs-policy.md) - methodology and **Addendum B**, the cross-scale correction. diff --git a/RESEARCH-SUMMARY.md b/RESEARCH-SUMMARY.md index 3f024c6..4eb3f25 100644 --- a/RESEARCH-SUMMARY.md +++ b/RESEARCH-SUMMARY.md @@ -11,7 +11,7 @@ for every claim tied to raw runs. ## 60-second reviewer path -Use [`REPORTS/start-here-for-hiring-reviewers.md`](REPORTS/start-here-for-hiring-reviewers.md) as the single entry point. It provides role-specific evidence blocks and the canonical 4-artifact path. The arc below gives the research context behind those artifacts. +Use [`REPORTS/START_HERE.md`](REPORTS/START_HERE.md) as the single entry point. It provides role-specific evidence blocks and the canonical 4-artifact path. The arc below gives the research context behind those artifacts. ## The arc diff --git a/docs/reports/hiring-evidence-index.md b/docs/reports/hiring-evidence-index.md index 3a633c5..8c08a2c 100644 --- a/docs/reports/hiring-evidence-index.md +++ b/docs/reports/hiring-evidence-index.md @@ -187,7 +187,7 @@ rg -n "Disclosure Boundary|production vulnerability claim" REPORTS/remediation-c **Evidence:** -- [`REPORTS/start-here-for-hiring-reviewers.md`](../../REPORTS/start-here-for-hiring-reviewers.md) +- [`REPORTS/START_HERE.md`](../../REPORTS/START_HERE.md) - [`REPRODUCE.md`](../../REPRODUCE.md) - [`pipeline/scripts/check-packet-ready.sh`](../../pipeline/scripts/check-packet-ready.sh) diff --git a/docs/reports/hiring-reviewer-map.md b/docs/reports/hiring-reviewer-map.md index 016e249..6c530eb 100644 --- a/docs/reports/hiring-reviewer-map.md +++ b/docs/reports/hiring-reviewer-map.md @@ -10,7 +10,7 @@ Cayman can run scoped, reproducible AI-agent security experiments; distinguish m ## 60-second Path -Start from the packet-ready router: [`REPORTS/start-here-for-hiring-reviewers.md`](../../REPORTS/start-here-for-hiring-reviewers.md). +Start from the packet-ready router: [`REPORTS/START_HERE.md`](../../REPORTS/START_HERE.md). 1. **H10b-G 70B substrate grid:** [`ATTACKS/2026-05-18-h10b-g-70b-substrate-grid-m1-variant-selective.md`](../../ATTACKS/2026-05-18-h10b-g-70b-substrate-grid-m1-variant-selective.md) - Proves the 70B substrate effect survives the chat-only control gate and that M1 prompt scaffolding is variant-selective: baseline/v3 held, v7 bypassed. @@ -57,7 +57,7 @@ For the attack-to-fix version of the same story, read [`REPORTS/remediation-case ## Role-Specific Evidence Blocks -Use [`REPORTS/start-here-for-hiring-reviewers.md`](../../REPORTS/start-here-for-hiring-reviewers.md) as the source of truth for packet-ready role blocks. Each role has exactly three links: one flagship artifact, one raw-evidence or pre-registration artifact, and one defensive or operational artifact. +Use [`REPORTS/START_HERE.md`](../../REPORTS/START_HERE.md) as the source of truth for packet-ready role blocks. Each role has exactly three links: one flagship artifact, one raw-evidence or pre-registration artifact, and one defensive or operational artifact. ## Known limitations and How to Discuss Them diff --git a/pipeline/scripts/check-packet-ready.sh b/pipeline/scripts/check-packet-ready.sh index 87bca83..5d9f9f5 100755 --- a/pipeline/scripts/check-packet-ready.sh +++ b/pipeline/scripts/check-packet-ready.sh @@ -18,7 +18,7 @@ packet_files=( README.md REPRODUCE.md RESEARCH-SUMMARY.md - REPORTS/start-here-for-hiring-reviewers.md + REPORTS/START_HERE.md REPORTS/2026-05-17-h10b-g-70b-substrate-findings.md REPORTS/remediation-case-study-tool-output-injection.md docs/reports/hiring-reviewer-map.md @@ -55,7 +55,7 @@ import re import sys from pathlib import Path -path = Path("REPORTS/start-here-for-hiring-reviewers.md") +path = Path("REPORTS/START_HERE.md") text = path.read_text() expected_roles = { diff --git a/pipeline/scripts/verify-public.sh b/pipeline/scripts/verify-public.sh index 29d4888..169279c 100755 --- a/pipeline/scripts/verify-public.sh +++ b/pipeline/scripts/verify-public.sh @@ -10,7 +10,7 @@ public_files=( RESEARCH-SUMMARY.md Makefile site/index.html - REPORTS/start-here-for-hiring-reviewers.md + REPORTS/START_HERE.md REPORTS/substrate-vs-policy-assessment.md REPORTS/2026-05-17-h10b-g-70b-substrate-findings.md REPORTS/remediation-case-study-tool-output-injection.md diff --git a/site/index.html b/site/index.html index 1968ae5..28b4d5b 100644 --- a/site/index.html +++ b/site/index.html @@ -174,7 +174,7 @@

Agent product security portfolio

mitigation-boundary measurement, and regression testing for agent-powered products.

@@ -185,17 +185,17 @@

Agent product security portfolio

Technical deep dive

Inspect the MCP substrate study, H10b-G mitigation boundary, DVL ReAct-loop SQLi evidence, raw artifacts, and limitations.

- Open reviewer path + Open reviewer path
From 06ba470f7af914068e7943b9f5bdad6eb362ea0d Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 3 Oct 2026 02:13:59 +0000 Subject: [PATCH 2/2] docs: plain-language opening; scope CI trigger and disclosure claims - The opening states the finding without cell codes and says typed dispatch hasn't been measured at 70B. - CI runs on every PR and on pushes to main (ci.yml), not on every push. - "All published experiments" are disclosure-green; vendor findings stay private per SECURITY.md. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013N5ekEdTjrMSMtnXXSxLRo --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index e53828f..9de84e1 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,8 @@ # ai-redteam-notes: agent and MCP injection research, with the fix wired into CI -Prompt scaffolding doesn't fix an unsafe tool-dispatch substrate. In a pre-registered grid on Llama-3.3-70B, a content-trust system prompt (M1) stopped two injection payload families and had no measurable effect on a third ([Results](#results)). A chat-only control (same model and payload, inline-XML dispatch removed) never exfiltrated. The durable fix belongs in the agent client: typed tool-call dispatch, with authorization and provenance checks outside the model. +When an AI agent reads tool output an attacker controls, can it be tricked into leaking data? In a pre-registered experiment on Llama-3.3-70B, a defensive system prompt blocked two families of injection payloads and did nothing against a third ([Results](#results)). With the same model and payloads but without inline-XML tool dispatch, nothing leaked. My conclusion is architectural: fix it in the agent client (typed tool-call dispatch, with authorization and provenance checks outside the model), not in the prompt. A typed-dispatch agent has not been measured at 70B yet ([limits](#methodology--limits)). -This repo holds the experiments behind that finding, the hypotheses that failed along the way (published, not hidden), and `substrate_auditor.py`, a dependency-free check that flags the risky substrate and runs in CI. All experiments are lab-only and disclosure-green ([scope](#methodology--limits), [SECURITY.md](SECURITY.md)). +This repo holds the experiments, the hypotheses that failed along the way (published, not hidden), and `substrate_auditor.py`, a dependency-free check for the risky dispatch pattern that runs in CI. All published experiments are lab-only and disclosure-green ([scope](#methodology--limits), [SECURITY.md](SECURITY.md)). [![CI](https://github.com/ChunkyTortoise/ai-redteam-notes/actions/workflows/ci.yml/badge.svg)](https://github.com/ChunkyTortoise/ai-redteam-notes/actions/workflows/ci.yml) [![Reproducible](https://img.shields.io/badge/repro-make%20repro%20(no%20API%20key)-success)](REPRODUCE.md) @@ -23,7 +23,7 @@ flowchart LR | Kind | Result | Value | Source | |---|---|---|---| | Measured | H10b-G: strict canary exfiltration through inline-XML tool dispatch on Groq-hosted Llama-3.3-70B, 7-cell grid, n=10 seeds per cell | M1 scaffold: **0/10** on baseline and v3 payloads, **10/10** on v7. Same payloads without it (M0): **9/10**, **10/10**, **10/10**. Chat-only control (no XML dispatch): **0/10** | [H10b-G entry, Result](ATTACKS/2026-05-18-h10b-g-70b-substrate-grid-m1-variant-selective.md#result) · per-seed runs: [M0 v3](lab/mcp-matrix/runs/2026-05-17-llama70b-xml-M0-v3/seeds.json), [M1 v3](lab/mcp-matrix/runs/2026-05-18-llama70b-xml-M1-v3/seeds.json), [M1 v7](lab/mcp-matrix/runs/2026-05-18-llama70b-xml-M1-v7/seeds.json), [control](lab/mcp-matrix/runs/2026-05-18-llama70b-chatonly-M0-baseline/seeds.json) | -| CI gate | On every push and PR: the auditor self-check, the auditor must rate the inline-XML sample high-risk and the typed sample low-risk, and the fixture benchmark must re-score cleanly | **7** self-check cases plus a missing-file guard · **10/10** benchmark fixtures match | [`ci.yml`](.github/workflows/ci.yml) · [`selfcheck.py`](lab/mcp-matrix/tools/selfcheck.py) · [`Makefile`](Makefile) (`audit`, `--expect-risk`) · [benchmark fixtures](EVALS/fixtures/tool-output-injection-fixtures.json) | +| CI gate | On every PR and every push to `main`: the auditor self-check, the auditor must rate the inline-XML sample high-risk and the typed sample low-risk, and the fixture benchmark must re-score cleanly | **7** self-check cases plus a missing-file guard · **10/10** benchmark fixtures match | [`ci.yml`](.github/workflows/ci.yml) · [`selfcheck.py`](lab/mcp-matrix/tools/selfcheck.py) · [`Makefile`](Makefile) (`audit`, `--expect-risk`) · [benchmark fixtures](EVALS/fixtures/tool-output-injection-fixtures.json) | | Inventory | Dated pre-registrations, each frozen before the cells it governs | **4** files in this repo | [`docs/preregistrations/`](docs/preregistrations/) · [hypothesis ledger](docs/preregistrations/INDEX.md) | This table is the one place each number is stated. The scope of each one is in [Methodology & limits](#methodology--limits). @@ -114,7 +114,7 @@ The diagram at the top is the chain under test and where the defense sits. - **Pre-registration.** Each hypothesis was frozen in a dated file before the cell that tested it ran, and deviations are recorded in the entry that reports the result ([index](docs/preregistrations/INDEX.md)). - **Control-validity gate.** A bypass is attributed to the substrate only if a chat-only control with the identical payload holds clean. H10b-G's control did (see [Results](#results)). - **Strict scoring with intervals.** Strict bypass (canary in a real exfil callback) is scored separately from intent shift, and cell rates carry Wilson intervals; H10b-G's were hand-recomputed from the result JSON ([methodology](REPORTS/2026-05-14-agent-security-eval-methodology.md)). -- **CI gate, every push and PR** ([`ci.yml`](.github/workflows/ci.yml), no network or model calls): `make repro` (auditor self-check plus expected-risk audits), `make benchmark` (fixture re-score), and `make public-surface` (disclosure frontmatter on every dated ATTACKS entry, secret-pattern scan, local link check, no future-dated artifacts, no private paths). +- **CI gate, every PR and every push to `main`** ([`ci.yml`](.github/workflows/ci.yml), no network or model calls): `make repro` (auditor self-check plus expected-risk audits), `make benchmark` (fixture re-score), and `make public-surface` (disclosure frontmatter on every dated ATTACKS entry, secret-pattern scan, local link check, no future-dated artifacts, no private paths). - **Pre-publication gate.** `make verify-public` adds the remediation demo, entry-template and disclosure checks on the published ATTACKS entries, and a network check that public GitHub links resolve. - **Disclosure ladder.** Every ATTACKS entry is `green`, `yellow` or `red`; only `green` is public ([SECURITY.md](SECURITY.md)).