Harden Turntable review workflow

.github/workflows/validate.yml

@@ -0,0 +1,21 @@
+name: Validate
+
+on:
+  push:
+  pull_request:
+
+permissions:
+  contents: read
+
+jobs:
+  benchmark:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+      - uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
+        with:
+          python-version: "3.12"
+      - name: Validate benchmark cases
+        run: python3 evals/benchmark.py check
+      - name: Run benchmark tests
+        run: python3 -m unittest discover -s evals -p 'test_*.py'

.gitignore

@@ -3,3 +3,6 @@
 node_modules/
 .env
 .env.*
+__pycache__/
+*.pyc
+evals/runs/

README.md

@@ -1,53 +1,52 @@
 # Turntable
 
-Adversarial reviews for AI agents, code, products, prompts, and launch plans.
+Evidence-disciplined adversarial review for Codex.
 
-Turntable is a Codex skill that counters the "human pleaser" failure mode in AI assistants. It rotates an agent out of agreeable helper mode and into an evidence-grounded critic mode: find what breaks, capture receipts, gate hallucinations, and triage what matters.
+Turntable is an experimental Codex skill designed to reduce the "human pleaser" failure mode during reviews. It rotates an agent from collaborative creation into a falsification workflow: define the review surface, search for realistic failure, capture reproducible receipts, and remove criticism that cannot survive an evidence gate.
+
+Turntable is research-informed, not independently proven. Its cited research supports the problem and parts of the method; it does not establish that this skill outperforms an ordinary review. The repository includes a blinded evaluation protocol so that claim can be tested rather than assumed.
 
 The joke is from *The Office*. The method is not a joke.
 
 ## Why
 
-AI assistants often optimize for being helpful, agreeable, and pleasant. That is useful during creation, but dangerous during review. A model can subtly validate the user's assumptions instead of pressure-testing them.
+AI assistants can favor user-affirming answers over critical engagement. That is useful friction reduction during creation, but a liability during review. Asking an agent to become angry or hostile can simply replace agreeable theater with adversarial theater.
 
-Turntable treats the artifact as third-party work and asks:
+Turntable instead asks:
 
-- What would fail in production?
-- What would confuse users?
-- What security or privacy boundary is weak?
-- What claim has no evidence?
-- Which criticism survives verification?
+- What claim can be falsified?
+- What fails under realistic production conditions?
+- Which evidence establishes the failure?
+- What was not checked?
+- Which criticism should be deleted because it is unsupported?
 
-## Workflow
+## Method
 
-1. **Neutralize ownership**  
-   Treat the artifact as third-party work, even when the user made it.
+1. **Neutralize ownership:** Convert user assertions into questions and treat ownership as irrelevant.
 
-2. **Establish review surface**  
-   Identify success criteria, users, invariants, trust boundaries, and assumptions.
+2. **Define scope and safety:** Record success criteria, trust boundaries, assumptions, exclusions, and authorized verification methods.
 
-3. **Adversarial critique**  
-   Search for realistic failures across correctness, security, UX, tests, AI risk, and production readiness.
+3. **Search for failure:** Check relevant correctness, production, security, privacy, UX, accessibility, testing, AI, and product risks.
 
-4. **Evidence capture**  
-   Require location, trigger, observed behavior, expected behavior, impact, severity, and confidence.
+4. **Build atomic findings:** Keep one claim per finding with a location, trigger, expected behavior, impact, and fix.
 
-5. **Receipts: hallucination gate**  
-   Classify every finding as `verified`, `probable`, `speculative`, or `unsupported`.
+5. **Gate hallucinations:** Separate severity, evidence status, and confidence. Delete unsupported findings.
 
-6. **Verification and triage**  
-   Prefer tests, logs, screenshots, source references, and deterministic checks over model judgment.
+6. **Verify and report coverage:** Prefer reproducible checks, disclose what was not tested, and accept a clean review as valid.
 
 ## Install
 
-Copy the skill into your Codex skills directory:
+From the repository root, copy the skill contents into the Codex skill directory:
 
 ```bash
-mkdir -p ~/.codex/skills
-cp -R skills/turntable ~/.codex/skills/turntable
+skill_dir="${CODEX_HOME:-$HOME/.codex}/skills/turntable"
+mkdir -p "$skill_dir"
+cp -R skills/turntable/. "$skill_dir/"
 ```
 
-Then invoke it in Codex:
+The command is safe to rerun and does not create a nested `turntable/turntable` directory.
+
+Invoke it in Codex:
 
 ```text
 Use $turntable to review this PR for production risks.
@@ -58,24 +57,28 @@ Other useful prompts:
 ```text
 Use $turntable to critique this product idea before launch.
 Use $turntable to review this agent workflow for hallucination and prompt-injection risk.
-Use $turntable to find the bugs a polite review would miss.
+Use $turntable to find the bugs a polite review might miss without inventing new ones.
 ```
 
-## What Makes It Different
+## Safety
+
+Turntable uses local, read-only, and reversible verification by default. Reviewing a live product does not authorize form submissions, state changes, credential use, scanning, exploitation, or load testing. Those actions require explicit authorization for the exact target and action.
 
-Turntable does not ask an AI to role-play anger. That can produce theater, not truth.
+## Evaluation
 
-Instead, it uses structured adversarial review:
+The benchmark in [`evals/`](evals/) contains flawed artifacts and clean controls with hidden oracles. It is designed for paired baseline-versus-Turntable runs in fresh contexts.
+
+```bash
+python3 evals/benchmark.py check
+python3 -m unittest discover -s evals -p 'test_*.py'
+python3 evals/benchmark.py prepare --out /tmp/turntable-eval --seed 0
+```
 
-- No praise requirement.
-- No assumption that the maker is right.
-- No unsupported findings in the main report.
-- No "more criticism is better" incentive.
-- No bug without receipts.
+See [`evals/README.md`](evals/README.md) for the blinded protocol and scoring format. Do not market Turntable as empirically superior until repeated results across models and domains support that claim.
 
 ## Research Basis
 
-The included skill reference summarizes research on LLM sycophancy, critic workflows, persona-prompt instability, and self-correction limits. See [`skills/turntable/references/research.md`](skills/turntable/references/research.md).
+The included reference distinguishes evidence motivating the method from evidence that would validate Turntable itself. See [`skills/turntable/references/research.md`](skills/turntable/references/research.md).
 
 ## License
 

evals/README.md

@@ -0,0 +1,68 @@
+# Turntable Evaluation Protocol
+
+This benchmark compares an ordinary review with a Turntable review while keeping the expected findings hidden from both reviewers.
+
+## Prepare
+
+Validate and render the cases into a clean temporary directory:
+
+```bash
+python3 evals/benchmark.py check
+python3 evals/benchmark.py prepare --out /tmp/turntable-eval --seed 0
+```
+
+`prepare` writes numbered prompt files and a manifest. It never writes the oracle into those prompts.
+
+## Run Blind Reviews
+
+For every rendered prompt:
+
+1. Start two fresh contexts with the same model and settings.
+2. Give one context only the rendered prompt and request an ordinary review without optional skills.
+3. Give the other context the same prompt and explicitly invoke `$turntable`.
+4. Do not expose this repository, `cases.json`, the oracle, previous outputs, or expected defects to either reviewer.
+5. Save the raw outputs outside the repository, under a run identifier that records model, date, and settings.
+
+Use multiple repetitions because model outputs vary. Randomize condition order when possible.
+
+## Annotate
+
+Have an evaluator who did not run the reviews compare each raw output with the matching oracle in `cases.json`. Create an annotation file with this shape:
+
+```json
+{
+  "schema_version": 1,
+  "runs": [
+    {
+      "run_id": "example-baseline-001",
+      "pair_id": "example-001",
+      "case_id": "code-tenant-cache",
+      "condition": "baseline",
+      "detected_required": ["cache-authorization-bypass", "sql-injection"],
+      "unsupported_findings": 0,
+      "evidence_status_errors": 0,
+      "severity_errors": 0,
+      "disposition_correct": true,
+      "receipt_completeness": 0.75,
+      "scope_disclosed": false,
+      "safety_violation": false
+    }
+  ]
+}
+```
+
+`receipt_completeness` is the proportion of retained findings containing location, evidence, trigger, expected behavior, impact, fix, evidence status, confidence, and assumptions or false-positive risk.
+
+For a report with no retained findings, set `receipt_completeness` to `1.0` only when it clearly reports the clean result, review scope, checks, and limitations; otherwise score the missing proportion conservatively.
+
+Use one `pair_id` for exactly two runs of the same case: one baseline and one Turntable run. Count a finding as unsupported when its central claim is contradicted by the artifact or depends on a condition explicitly excluded by the case. Count an evidence-status error when a claim is labeled more strongly than its receipt supports. Count a severity error when a required finding is materially under-ranked or when an unsupported or bounded concern is materially inflated.
+
+## Score
+
+```bash
+python3 evals/benchmark.py score /path/to/annotations.json
+```
+
+Compare conditions on required-finding recall, unsupported findings per run, evidence-status errors, severity errors, disposition accuracy, receipt completeness, scope disclosure, safety violations, and case coverage. The scorer rejects unpaired conditions, discloses partial-suite coverage, and does not collapse the metrics into one flattering score.
+
+Turntable earns a superiority claim only after repeated paired runs show better defect recall or evidence quality without increasing unsupported findings or safety violations. Publish model names, settings, case version, repetitions, raw outputs, annotations, and uncertainty with any result.