Finalize benchmark release surface and comparison artifacts

CHANGELOG.md

@@ -1,5 +1,13 @@
 # Changelog
 
+## 0.1.0-beta.2 — 2026-04-08
+
+Second public beta.
+
+- adds a consolidated repo-level roadmap in `ROADMAP.md`
+- links the roadmap from the main `README.md`
+- bumps the published package and server version to `0.1.0-beta.2`
+
 ## 0.1.0-beta.1 — 2026-03-31
 
 First public beta.
@@ -16,4 +24,3 @@ First public beta.
 - Ungrounded entity detection
 - Claim classification for tool routing
 - Iterative enforcement with explicit caller-provided context
-

README.md

@@ -1,4 +1,4 @@
-# critical-thinking-mcp
+# ct-mcp
 
 > **BETA** — Under active development. Interfaces may change between versions.
 
@@ -9,21 +9,30 @@ No LLM calls in enforcement logic. No configuration. No API keys. Runs locally.
 ## Install
 
 ```bash
-npm install -g critical-thinking-mcp
+npm install -g ct-mcp
 ```
 
 Add to Claude Desktop, Cursor, or any MCP client:
 
 ```json
 {
   "mcpServers": {
-    "critical-thinking": {
-      "command": "critical-thinking-mcp"
+    "ct-mcp": {
+      "command": "ct-mcp"
     }
   }
 }
 ```
 
+## Roadmap
+
+The repo-wide roadmap is consolidated in [`ROADMAP.md`](ROADMAP.md).
+
+That document separates:
+
+- `ct-mcp` package milestones and beta-exit criteria
+- `Invisible Tea Party` benchmark release milestones and calibration goals
+
 ## What Changes
 
 A billing system claims 90% confidence. Three assumptions support it — but two can't state what would prove them wrong.
@@ -84,6 +93,45 @@ Coverage includes confidence inflation, concurrency patterns (race conditions, s
 
 Full benchmark results: [benchmark/reports/BENCHMARK_REPORT.md](benchmark/reports/BENCHMARK_REPORT.md)
 
+## Benchmark Suites
+
+This repo now has two distinct benchmark tracks under [`benchmark/`](benchmark/):
+
+- [`benchmark/invisible-tea-party/`](benchmark/invisible-tea-party/) — `The Invisible Tea Party: A Benchmark for Coherence vs Truth`
+- [`benchmark/duckexperiments/`](benchmark/duckexperiments/) — critique-improvement workflow using CT-MCP as deterministic critique support
+
+For Tea Party specifically:
+
+- benchmark overview: [`benchmark/invisible-tea-party/README.md`](benchmark/invisible-tea-party/README.md)
+- benchmark release line: [`benchmark/invisible-tea-party/RELEASES.md`](benchmark/invisible-tea-party/RELEASES.md)
+- benchmark foundation: [`benchmark/invisible-tea-party/FOUNDATION.md`](benchmark/invisible-tea-party/FOUNDATION.md)
+- pass contracts: [`benchmark/invisible-tea-party/PASS_SCHEMA.md`](benchmark/invisible-tea-party/PASS_SCHEMA.md)
+- verifier architecture: [`benchmark/invisible-tea-party/PASS4_ARCHITECTURE.md`](benchmark/invisible-tea-party/PASS4_ARCHITECTURE.md)
+- current agent handoff: [`benchmark/invisible-tea-party/AGENT_HANDOFF.md`](benchmark/invisible-tea-party/AGENT_HANDOFF.md)
+- results layout and reproduction notes: [`benchmark/invisible-tea-party/results/README.md`](benchmark/invisible-tea-party/results/README.md)
+
+Current published Tea Party surfaces:
+
+- preserved official baseline (`v1.0`): [`benchmark/invisible-tea-party/results/live-gemini-official-2026-04-06/aggregate_report.md`](benchmark/invisible-tea-party/results/live-gemini-official-2026-04-06/aggregate_report.md)
+- current comparison pack (`v1.1`): [`benchmark/invisible-tea-party/results/live-expanded-comparison-2026-04-07/aggregate_report.md`](benchmark/invisible-tea-party/results/live-expanded-comparison-2026-04-07/aggregate_report.md)
+- dedicated Gemini 3.1 preview comparison: [`benchmark/invisible-tea-party/results/live-gemini-3-1-preview-2026-04-07/aggregate_report.md`](benchmark/invisible-tea-party/results/live-gemini-3-1-preview-2026-04-07/aggregate_report.md)
+- benchmark overview and interpretation: [`benchmark/invisible-tea-party/README.md`](benchmark/invisible-tea-party/README.md)
+
+What these additions are for:
+
+- Tea Party measures whether models accept coherent nonsense, repair reasoning under critique, and stay anchored to logical and ontological constraints.
+- Duck Experiments measures whether structured critique actually improves answers in a repeatable review workflow.
+- Together they separate two different questions:
+  - can the model detect persuasive invalidity at all?
+  - does deterministic critique support materially improve the result?
+
+What we are trying to get from the new benchmark work:
+
+- a preserved official baseline plus versioned comparison packs for coherence-vs-truth failures
+- replayable artifacts that combine prompts, raw pass outputs, and final scores in one place
+- benchmark outputs that are usable for publication, scorecards, and downstream engineering work
+- a clean handoff surface for future agents to improve matcher coverage and rerun calibration
+
 ## Iterative Enforcement (No Hidden Memory)
 
 CT-MCP retains nothing between calls. For multi-step workflows, callers pass explicit `context`:
@@ -98,7 +146,7 @@ No hidden state — all context is in the request.
 
 ## Experimental Workflow And Formulas
 
-The public comparison workflow in [`duckexperiments/`](duckexperiments/) uses CT-MCP as critique support, not as the final judge of truth.
+The public comparison workflow in [`benchmark/duckexperiments/`](benchmark/duckexperiments/) uses CT-MCP as critique support, not as the final judge of truth.
 
 Process:
 

ROADMAP.md

@@ -0,0 +1,104 @@
+# Roadmap
+
+This repo now has two parallel release lines:
+
+- `ct-mcp` package semver for the MCP server and tool surface
+- `Invisible Tea Party` benchmark releases for benchmark artifacts and published result bundles
+
+They are related, but they do not move in lockstep.
+
+## Current State
+
+### `ct-mcp` package
+
+- Current package version: `0.1.0-beta.2`
+- Current shipped surface: 9 deterministic, stateless MCP tools
+- Current validation baseline: 56 benchmark scenarios (`42` defect, `14` clean control)
+- Current publication caveat: baseline and prompted benchmark conditions are still self-assessed
+
+### `Invisible Tea Party` benchmark
+
+- `v1.0` official baseline preserved
+- `v1.1` comparison pack published
+- Core single-agent benchmark path is live:
+  - Pass 1 -> Pass 2 -> Pass 3 loop
+  - Pass 4A deterministic verifier
+  - Pass 4B arbiter verifier
+  - Pass 4C reconciler
+  - calibration reporting, scorecard rendering, failure gallery, and official ingest flow
+
+## Near-Term Priorities
+
+### Track A: `ct-mcp` package
+
+These are the clearest next improvements for the MCP server itself:
+
+- independent human scoring of baseline and prompted benchmark conditions
+- cross-tool routing via the existing claim classifier
+- chained arithmetic verification for multi-step formulas
+- escalate `ordering_assumption` from warning to blocking when no protections are listed
+
+The most important gate is independent human scoring. That is the explicit blocker for stronger non-beta benchmark claims.
+
+### Track B: `Invisible Tea Party`
+
+The benchmark core is already built, so the next work is hardening and release discipline:
+
+- keep the official baseline stable unless a new certified official path is intentionally minted
+- publish new comparison packs without rewriting prior releases
+- improve rerun guidance and raw calibration hygiene for outside users
+- extend the benchmark from failure isolation toward stronger end-to-end agent evaluation
+- keep separating official benchmark claims from research-only comparison runs
+
+## Milestones
+
+### `ct-mcp` `v1.0`
+
+The package is ready to move beyond beta when all of the following are true:
+
+- independent human evaluation has been completed for benchmark scoring
+- benchmark claims no longer rely on self-assessed baseline and prompted conditions
+- the current tool surface and routing behavior are stable enough for a non-beta contract
+
+Until then, `0.1.x-beta` should be treated as a real public beta with explicit evidence limits.
+
+### `Invisible Tea Party` `v1.5 Calibrated`
+
+The next major benchmark milestone is not another architecture build-out. It is calibration strength.
+
+`v1.5 Calibrated` is the earliest release that should support stronger publication-grade claims about:
+
+- score stability
+- threshold quality
+- calibration behavior across runs
+- broader confidence in benchmark interpretation
+
+## Longer-Term Scope
+
+These items are intentionally not part of the immediate roadmap:
+
+- multi-agent contagion in the default Tea Party benchmark
+- folding Tea Party releases into npm package semver
+- pretending the current benchmark is a general fact-verification system
+
+For Tea Party specifically:
+
+- multi-agent contagion is reserved for `v2` or explicit extension-mode evaluation
+- any early contagion work should remain outside the core public benchmark contract
+
+## Working Rule
+
+If a change affects the MCP server surface, tool semantics, or package claims, treat it as `ct-mcp` roadmap work.
+
+If a change affects benchmark schemas, pass orchestration, calibration bundles, release packs, or leaderboard policy, treat it as `Invisible Tea Party` roadmap work.
+
+## Source Docs
+
+This roadmap consolidates the currently scattered plan from:
+
+- `README.md`
+- `DEVELOPMENT.md`
+- `benchmark/invisible-tea-party/README.md`
+- `benchmark/invisible-tea-party/RELEASES.md`
+- `benchmark/invisible-tea-party/NEXT_STEPS_PLAN.md`
+- `docs/designs/invisible-tea-party-next-evolution.md`

benchmark/BENCHMARK_RUNNER_PROMPT.md

@@ -35,7 +35,7 @@ The CT-MCP MCP server should be configured in your MCP settings:
   "mcpServers": {
     "ct-mcp": {
       "command": "node",
-      "args": ["./node_modules/.bin/critical-thinking-mcp"]
+      "args": ["./node_modules/.bin/ct-mcp"]
     }
   }
 }