From 2f7a4b66c81b2787d745e2b8f68379cb90c53fdf Mon Sep 17 00:00:00 2001 From: Ken Okamoto Date: Thu, 13 Aug 2026 17:40:31 +0900 Subject: [PATCH] Add CITADEL-style title confirmation and an AMPL ship gate. The skill is company QA: confirm claimed titles after T4, check retractions on T1, and refuse to ship fabricated identifiers as real. Grey literature stays T2. Engine T4 is never overridden. --- skills/medical-fact-check/SKILL.md | 26 ++++++++-- .../references/adversarial-review.md | 3 +- .../references/citadel-confirmation.md | 40 ++++++++++++++++ .../references/ship-gate.md | 47 +++++++++++++++++++ .../references/verification-workflow.md | 18 ++++++- .../templates/report-template.md | 9 +++- 6 files changed, 136 insertions(+), 7 deletions(-) create mode 100644 skills/medical-fact-check/references/citadel-confirmation.md create mode 100644 skills/medical-fact-check/references/ship-gate.md diff --git a/skills/medical-fact-check/SKILL.md b/skills/medical-fact-check/SKILL.md index 6297311..af41fb0 100644 --- a/skills/medical-fact-check/SKILL.md +++ b/skills/medical-fact-check/SKILL.md @@ -1,21 +1,21 @@ --- name: medical-fact-check -description: "Comprehensive medical fact-checking and critical appraisal skill. Evaluates any medical content — research papers, articles, social media posts, newsletters, YouTube/podcast transcripts, conference slides, clinical guidelines, pharma marketing, patient leaflets, health app content, and more — across 15 criteria for accuracy, evidence quality, and appropriateness. Generates a structured Markdown report with an A–F score and actionable improvement suggestions. Triggers: 'fact-check', 'evidence check', 'evaluate this article', 'check this post', 'ファクトチェック', 'エビデンスチェック', 'この記事を評価して', 'この投稿の問題点'." +description: "Pre-publish QA for medical writing (AMPL company gate, open protocol). Deterministic citation engine plus CITADEL-style claimed-title confirmation, 15-criteria appraisal, adversarial KILL, and a ship gate. Evaluates papers, posts, slides, guidelines, marketing, patient materials, AI-generated copy. Report with A-F, integrity stamp, ship: no/caveats/human-ack. Triggers: fact-check, evidence check, evaluate this article, check this post, ship gate, QA, ファクトチェック, エビデンスチェック, この記事を評価して, この投稿の問題点." --- # Medical Fact-Check Skill -Comprehensive critical appraisal and fact-checking for medical information, producing a structured report. +AMPL's quality assurance for medical writing, published as an open protocol. Critical appraisal plus a ship gate: the piece does not go out with a fabricated identifier presented as real. > **Scope.** This is a pre-publication aid for writers, editors, and researchers — **not clinical decision support.** It evaluates how medical *content* is written and sourced; it does not diagnose, treat, or replace professional medical judgment. Do not recommend treatments. -> **Must-read before scoring.** Use the `Read` tool on `references/verification-workflow.md` (operating model, loops, hard rules) and `references/adversarial-review.md` (red-team, KILL/MAJOR/MINOR/PASS) before you emit a letter grade. Also load `references/checklist.md`, `references/evidence-levels.md`, `templates/claim-ledger.md`, and `templates/report-template.md`. Depending on install, the skill directory is `~/.claude/skills/medical-fact-check/` (manual copy) or `${CLAUDE_PLUGIN_ROOT}/skills/medical-fact-check/` (Evidentia plugin). +> **Must-read before scoring.** Use the `Read` tool on `references/verification-workflow.md` (operating model, loops, hard rules), `references/adversarial-review.md` (red-team, KILL/MAJOR/MINOR/PASS), `references/citadel-confirmation.md` (claimed-title confirmation after T4), and `references/ship-gate.md` (company / publisher QA) before you emit a letter grade or a ship stamp. Also load `references/checklist.md`, `references/evidence-levels.md`, `templates/claim-ledger.md`, and `templates/report-template.md`. Depending on install, the skill directory is `~/.claude/skills/medical-fact-check/` (manual copy) or `${CLAUDE_PLUGIN_ROOT}/skills/medical-fact-check/` (Evidentia plugin). > **Deterministic citation engine.** When checking citations (Step 4), prefer the local `evidentia` engine over verifying each identifier by hand. Use the `evidentia` binary first; fall back to `npx -y evidentia` only if a local install is unavailable. Evidentia resolves DOI/PMID/arXiv/NCT identifiers against CrossRef, PubMed, OpenAlex, arXiv, and ClinicalTrials.gov, and emits the 4-tier classification plus `lookupVerified` and `resolverOutcomes` lookup traces. Use a cache path when possible so repeat checks are stable and fast. See **Step 4** for how to call it. ## Overview -This skill evaluates medical information across **15 criteria** — evidence quality, citation accuracy, statistical interpretation, ethical considerations, and more — then generates a structured Markdown report with an overall **A–F score** and actionable improvement suggestions. +This skill is a **pre-publish QA gate**. It evaluates medical information across **15 criteria**, confirms fabrications the CITADEL way (claimed title absent from independent databases), then generates a structured Markdown report with an overall **A–F score**, an integrity stamp, and a ship decision (no / caveats / human-ack). ### Supported Media Types @@ -182,6 +182,21 @@ Fallback if the local binary is unavailable: Use its output as the ground truth for citation *existence*. Inspect `lookupVerified` and `resolverOutcomes` when explaining why a citation was classified. Write the engine tier into the claim ledger. +#### 4a2. CITADEL-style title confirmation (every T4) + +Read `references/citadel-confirmation.md`. For every engine **Tier 4**, search the **claimed title from the draft** in PubMed, Crossref/OpenAlex, and one Scholar-like web search *before* you write KILL. + +- 0 hits in all → fabrication confirmed. Keep T4. KILL if presented as real. +- Title exists under another identifier → keep T4 for the cited id. Do **not** upgrade to T1. Still KILL if the draft presents that id as that paper. Note "claimed work may exist under another id." +- Never override T4 because authors sound real or the journal/year match. + +This is not a new engine tier. The engine remains ground truth for the identifier. + +#### 4a3. Retraction check (every T1) + +For every **Tier 1**, check PubMed "Retracted Publication" / Crossref `update-to`. If retracted: ledger note `retracted`. T1 stays T1 (the paper existed). It cannot ship as current evidence unless the draft says so. + + - **Tier 1 (Verified)** — the paper, preprint, or trial exists and metadata matches. Proceed to the context check in 4b. - **Tier 2 (Content review needed)** — the source may be real, but the engine cannot deterministically verify it in registries, or semantic use still needs review. - **Tier 3 (Bibliographic mismatch)** — a real record exists, but the DOI/PMID/arXiv/NCT identifier or metadata is wrong. Record the discrepancy. @@ -295,6 +310,7 @@ Save the completed report as a Markdown file using `Write`: - The file path - A concise summary of findings (3–5 sentences) - The overall score, adversarial verdict, and risk level + - The **ship stamp** (no / caveats / human-ack) from `references/ship-gate.md` — default on for AMPL / org QA - Top 3 most important issues to address ### Step 9: Correction loop (up to 3) @@ -403,6 +419,8 @@ Patient materials prioritize accessibility and safety: - `references/verification-workflow.md` — operating model, two layers, named loops, hard rules (must-read before scoring) - `references/adversarial-review.md` — five lenses, attack checklist, KILL/MAJOR/MINOR/PASS (must-read before scoring) +- `references/citadel-confirmation.md` — claimed-title confirmation after T4; T1 retraction check +- `references/ship-gate.md` — AMPL / publisher pre-publish QA (cannot-ship rules, integrity stamp) - `references/checklist.md` — detailed 15-item evaluation checklist - `references/evidence-levels.md` — evidence hierarchy & quality assessment tools - `templates/claim-ledger.md` — claim table filled before the engine call diff --git a/skills/medical-fact-check/references/adversarial-review.md b/skills/medical-fact-check/references/adversarial-review.md index c45876b..5b5b812 100644 --- a/skills/medical-fact-check/references/adversarial-review.md +++ b/skills/medical-fact-check/references/adversarial-review.md @@ -17,6 +17,7 @@ Trust the engine. Look for swapped DOIs, invented PMIDs, "the author exists but - Engine Tier 4 presented as a real source → this lens fails hard (KILL). - Engine Tier 3 (bibliographic mismatch) presented as clean → at least MAJOR unless already disclosed in the draft. - Do not "rescue" a Tier 4 because a similarly titled paper exists. +- **CITADEL dominant pattern (KILL):** a plausible title attached to a real PMID/DOI that resolves to an unrelated paper in the **same journal and year**. Confirm the claimed title is absent (or lives under another id) per `references/citadel-confirmation.md`. Keep the engine T4 either way. ### 2. Claim support @@ -55,7 +56,7 @@ KILL or MAJOR: do **not** emit a clean A-score as if publishable. Tell the user The agent must actually answer each line: **yes / no** plus a one-line evidence note. "n/a" is allowed only where a line cannot apply (e.g. no citations). Blank is a fail. 1. Did I run the engine (`evidentia check --format json` or MCP `verify_citations`)? If not, did I mark citations `unresolved` instead of guessing a tier? -2. Did I refuse to override any engine Tier 4 to "probably real"? +2. Did I refuse to override any engine Tier 4 to "probably real," and did I confirm the claimed title is absent from independent databases or swapped onto another id? 3. Did I treat ISBN / guideline / title-only cites as Tier 2 (Content review needed), never Hallucination? 4. Did I read an abstract for each Tier 1 citation used to support a numeric or causal claim? (One extra lookup, then stop.) 5. Does every cited numeric claim match the paper's actual primary finding, population, and effect direction? diff --git a/skills/medical-fact-check/references/citadel-confirmation.md b/skills/medical-fact-check/references/citadel-confirmation.md new file mode 100644 index 0000000..1f7418e --- /dev/null +++ b/skills/medical-fact-check/references/citadel-confirmation.md @@ -0,0 +1,40 @@ +# CITADEL-style title confirmation + +Skill-side check after the engine, before a **KILL**. Stolen from CITADEL (Topaz et al., *Lancet* 2026; [maxtopaz.com/citadel](https://www.maxtopaz.com/citadel)). Existence is still the engine. This step only **confirms** a Tier 4. It must not invent a tier and must not override one. + +CITADEL's fabrication rule: the *claimed title* returns **0 results** in PubMed, Crossref, OpenAlex, **and** Google Scholar. The dominant pattern in their audit is a plausible title attached to a **real PMID/DOI that resolves to an unrelated paper in the same journal and year**. + +The core of detection is not AI judgment. It is a binary check against independent databases. Do not copy CITADEL's LLM title-comparison for existence, and do not copy their exclusion of the 23% of references that lacked a PMID. Grey literature stays **Tier 2**. + +## After every engine T4 + +Take the **claimed title from the draft**, not the registry title the identifier resolved to. + +1. Search that claimed title in PubMed (WebSearch / PubMed). +2. Search it in Crossref or OpenAlex (engine traces count if they already did a title search; otherwise WebSearch). +3. One Scholar-like web search of the quoted title. + +Record hits in the report's integrity stamp. + +| Hits | What it means | What you do | +|------|----------------|-------------| +| **0 in all** | Fabrication confirmed (CITADEL definition). | Keep T4. **KILL** if the draft presents it as a real source. | +| **Title exists under a different identifier** | Swapped PMID/DOI. Engine T4 for the *cited id* is still right. | Keep T4. Do **not** upgrade to T1. Still **KILL** if the draft presents that id as that paper. Note: "claimed work may exist under another id." | +| **Authors "sound real" / same journal-year** | The CITADEL trap. | Never override T4. | + +Do not rescue a T4 because a similarly titled paper exists. Do not search until you find something that "could be" the paper. + +## After every engine T1 + +One extra integrity check CITADEL asked publishers to attach as metadata, done here at write time: + +- PubMed publication type **Retracted Publication**, or +- Crossref `update-to` / retraction notice. + +If yes: ledger note `retracted`. The paper existed (T1 stays T1). It cannot ship as **current** evidence unless the draft says so. + +## What this is not + +- Not a new engine tier. +- Not permission to override T4. +- Not clinical decision support. diff --git a/skills/medical-fact-check/references/ship-gate.md b/skills/medical-fact-check/references/ship-gate.md new file mode 100644 index 0000000..c93da5e --- /dev/null +++ b/skills/medical-fact-check/references/ship-gate.md @@ -0,0 +1,47 @@ +# Ship gate (company / publisher QA) + +CITADEL's authors wrote that the barrier is **institutional, not technological**: verify references **before** the next human sees the piece. This file is that barrier for a medical organization. + +First customer: **AMPL** (Ken Okamoto). The gate is generic. Any medical org using this skill should run it before shipping. + +Not clinical decision support. The gate decides whether *content* may ship. It does not diagnose, treat, or tell a patient what to take. The human still owns the publish click. + +## When it applies + +Default **on** for AMPL medical copy, and for any org that asked for QA: + +- learn / pedia pages +- note / X posts +- client decks and paper-writer output +- READMEs or skill pages that make clinical or epidemiologic claims + +A one-off public fact-check still produces the report. The stamp says whether the piece **may ship**. + +## Cannot ship + +Human still owns the click. The skill's job is to refuse a clean pass. + +- Any **T4** presented as a real source +- Adversarial **KILL** +- **HIGH** public-health risk with an unfixed KILL or MAJOR +- Engine **`unresolved`** on a numeric or causal claim (retry already done), unless a human explicitly acks +- **T2** (unindexed or semantic misuse) presented as if it were T1 + +## Can ship with caveats + +- Verdict **PASS** or **MINOR** +- T1 / T3 disclosed (T3 = bibliographic mismatch, already in the draft or the report) +- Retractions labeled +- T2 only if labeled "not registry-verified" + +## Integrity stamp (required in the report) + +- Engine command and version +- Databases: CrossRef, PubMed, OpenAlex, arXiv, ClinicalTrials.gov +- Timestamp +- CITADEL-style title confirmation done (yes/no) for **each T4** +- Retractions found +- Ship: **no** / **caveats** / **human-ack** +- AMPL QA: yes / no + +If the stamp is missing, the gate did not run. diff --git a/skills/medical-fact-check/references/verification-workflow.md b/skills/medical-fact-check/references/verification-workflow.md index 5d3cb74..a5aaecb 100644 --- a/skills/medical-fact-check/references/verification-workflow.md +++ b/skills/medical-fact-check/references/verification-workflow.md @@ -74,7 +74,9 @@ Four named loops. Each has a stop condition. "FAIL" means the gate did not clear **Then:** if still down, mark affected citations `unresolved`. Continue the rest of the appraisal. Do **not** guess Hallucination without a failed identifier lookup. -**Do not:** override a returned Tier 4 because the title "sounds real." Do not invent Tier 1 from WebSearch when the engine already said 3 or 4. Manual WebSearch is the fallback only when the engine is unavailable, and even then a 404/empty registry hit is Hallucination — a book, guideline, or title-only cite is still Tier 2. +**Then (skill, not a new tier):** for each T4, run claimed-title confirmation (`references/citadel-confirmation.md`). For each T1, check retraction. + +**Do not:** override a returned Tier 4 because the title "sounds real." Do not invent Tier 1 from WebSearch when the engine already said 3 or 4. Manual WebSearch is the fallback only when the engine is unavailable, and even then a 404/empty registry hit is Hallucination — a book, guideline, or title-only cite is still Tier 2. ### 2. Semantic loop @@ -119,6 +121,17 @@ Step 9 is a protocol, not an optional courtesy. **FAIL:** a new Tier 4, a new KILL/MAJOR, or an unfixed previous KILL/MAJOR. Do not raise the letter grade while those remain. + +## CITADEL overlap + +Stolen from CITADEL (Topaz et al., *Lancet* 2026) and refused where it would make us worse. + +**Steal.** Fabrication is confirmed when the *claimed title* is absent from independent databases (PubMed, Crossref, OpenAlex, plus a Scholar-like web search). The dominant pattern is a plausible title on a real PMID/DOI of an unrelated paper in the same journal and year. The core of detection is a binary registry check, not an LLM vibe. The institutional move is a **ship gate** before the next human sees the piece (`references/ship-gate.md`). + +**Refuse.** Do not drop grey literature because it lacks a PMID (CITADEL excluded 23%). Those stay **Tier 2**. Do not let an LLM decide existence. Do not override engine T4. Do not use their "12-fold" figure in our materials (~10-fold is the 1 in 2,828 to 1 in 277 ratio). + +Title confirmation and retraction checks are **skill-side**. They do not create a new engine tier. See `references/citadel-confirmation.md`. + ## Hard rules 1. Engine output is ground truth for existence. The LLM must not override Tier 4 to "probably real." @@ -127,6 +140,9 @@ Step 9 is a protocol, not an optional courtesy. 4. Semantic mismatch of a real paper is Tier 2, not Tier 4. 5. Not CDS. Do not recommend treatments, doses, or "what the patient should do." 6. The AI recommends; the human publishes. +7. After every T4, confirm the claimed title (CITADEL-style) before KILL. Do not override T4. +8. After every T1, check retraction / Crossref update-to. Unlabeled retraction cannot ship as current evidence. +9. AMPL / org shipping applies `references/ship-gate.md`. ## What FAIL does *not* mean diff --git a/skills/medical-fact-check/templates/report-template.md b/skills/medical-fact-check/templates/report-template.md index 5692a74..6090f80 100644 --- a/skills/medical-fact-check/templates/report-template.md +++ b/skills/medical-fact-check/templates/report-template.md @@ -278,6 +278,13 @@ Scoring gates: a **KILL** forces overall score ≤ D (F if ethics, harm, or fabr --- + +## 4b. Integrity stamp / ship gate + +See `references/ship-gate.md`. **AMPL QA**: [yes/no]. **Ship**: [no / caveats / human-ack]. **T4 title confirmation**: [done / n/a]. **Retractions**: [none / list]. **Databases**: CrossRef, PubMed, OpenAlex, arXiv, ClinicalTrials.gov. **Timestamp**: [ISO-8601]. + +--- + ## 5. Adversarial review **Pass number**: [1 / 2 / 3 of 3] @@ -303,7 +310,7 @@ Scoring gates: a **KILL** forces overall score ≤ D (F if ethics, harm, or fabr | # | Question | Yes / No / n/a | Evidence | |---|----------|----------------|----------| | 1 | Did I run the engine? If not, marked unresolved rather than guessed a tier? | | | -| 2 | Refused to override any engine Tier 4 to "probably real"? | | | +| 2 | Refused to override T4, and confirmed claimed title absent or swapped? | | | | 3 | ISBN / guideline / title-only treated as Content review needed, never Hallucination? | | | | 4 | Read an abstract for each T1 citation used to support a numeric or causal claim? | | | | 5 | Every cited numeric claim matches the paper's primary finding, population, and effect direction? | | |