From b92ccddec7fd80e467d160c22b9d7aa55f2ed6fd Mon Sep 17 00:00:00 2001 From: tn-pisama Date: Tue, 28 Jul 2026 11:55:45 -0700 Subject: [PATCH 1/2] docs: correct false claims in the README Three claims were verified false against the published 0.5.5 artifact and against this project's own benchmark evidence. 1. Four documented detectors do not exist. `delegation`, `grounding`, `retrieval_quality` and `compaction_quality` all raise UnknownDetectorError; the installed registry has exactly 32 names and none of them is present. Replaced with four that do exist and were undocumented: propagation, citation, routing, mcp_protocol (each verified to run). 2. "plus framework-specific detectors for n8n, LangGraph, Dify, and OpenClaw" is not true. registry.count == 32 and get_for_platform() returns subsets of that same set (n8n 31, langgraph 32, dify 32, openclaw 32). There is no detector unique to any framework; the only difference is per-platform gating, e.g. coordination is restricted to multi-agent platforms. 3. The TRAIL table presented 59.9% against 11.9% (GPT-5.4) and 6.8% (Gemini 3.1 Pro) with no caveat. Our own linked artifact contradicts it. evidence.json says verbatim: "The run is in-distribution because 144 of 148 traces appeared in calibration material", "The run does not establish held-out generalization", and "The archive does not contain the prediction-level data needed to reproduce joint accuracy". trail_llm_baselines.json carries confusion counts but "result": null for all four models, so 11.9% and 6.8% are not sourced from the artifact the README cites as evidence. The table now reports only the figure the archive supports, states the in-distribution caveat inline, and names macro-F1 0.7535 / micro-F1 0.7463 as the numbers that ARE reproducible via benchmarks/verify_report.py. The LLM-judge comparison is removed rather than restated, because no comparable number exists in the evidence. The evidence.json claim boundaries were already careful and honest. The README simply did not carry them, which turns rigorous internal work into an unsupportable public claim. Co-Authored-By: Claude --- README.md | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 8c81d7b..48d5d95 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE) [![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/) -Pisama ships 32 core heuristic detectors plus framework-specific detectors for n8n, LangGraph, Dify, and OpenClaw. They run locally with zero LLM cost on the heuristic tier. An archived run on the [TRAIL benchmark](https://arxiv.org/abs/2505.08638) reports **59.9% joint accuracy** (span and category), compared with 11.9% for the best general-purpose LLM judge tested. The public confusion counts reproduce the reported macro-F1 and micro-F1. See the [benchmark evidence and its reproducibility boundary](https://github.com/Pisama-AI/pisama-detectors/tree/main/benchmarks). +Pisama ships 32 core heuristic detectors. They apply across frameworks including n8n, LangGraph, Dify and OpenClaw, with per-platform gating (for example `coordination` runs only on multi-agent platforms). They run locally with zero LLM cost on the heuristic tier. An archived, in-distribution run on the [TRAIL benchmark](https://arxiv.org/abs/2505.08638) reports 59.9% joint accuracy (span and category). It is not a held-out result: 144 of the 148 traces appeared in calibration material, and the published archive does not contain the prediction-level data needed to recompute joint accuracy. The public confusion counts do reproduce the reported macro-F1 (0.7535) and micro-F1 (0.7463). See the [benchmark evidence and its reproducibility boundary](https://github.com/Pisama-AI/pisama-detectors/tree/main/benchmarks). ## Install @@ -51,7 +51,7 @@ Works in Cursor, Claude Desktop, and Windsurf. No API key is needed: ## Detectors -32 core detectors plus framework-specific detectors for n8n, LangGraph, Dify, and OpenClaw. A representative selection: +32 core detectors, gated per platform (n8n, LangGraph, Dify, OpenClaw and others). A representative selection: | Detector | What It Catches | |----------|----------------| @@ -71,20 +71,26 @@ Works in Cursor, Claude Desktop, and Windsurf. No API key is needed: | `withholding` | Suppressed findings, hidden errors | | `convergence` | Metric plateau, regression, thrashing | | `overflow` | Context window exhaustion | -| `delegation` | Delegation quality and task handoff failures | -| `grounding` | Claims not supported by source documents | -| `retrieval_quality` | Poor retrieval relevance or coverage | -| `compaction_quality` | Information loss during context compaction | +| `propagation` | Silent error propagation across steps | +| `citation` | Fabricated citations and source misattribution | +| `routing` | Inputs misrouted to the wrong specialist agent | +| `mcp_protocol` | MCP tool-communication failures | ## Benchmark Results -**TRAIL** (trace-level failure detection, 148 traces): +**TRAIL** (trace-level failure detection, 148 traces). Archived April 2026 run, **in-distribution and not held out**: 144 of 148 traces appeared in calibration material. | Method | Joint Accuracy | |--------|---------------| -| GPT-5.4 | 11.9% | -| Gemini 3.1 Pro | 6.8% | -| **Pisama archived run** | **59.9%** | +| Pisama archived run | 59.9% | + +Reproducible from the archive: macro-F1 0.7535, micro-F1 0.7463 (`benchmarks/verify_report.py`). +Joint accuracy is **not** recomputable from the public archive, which lacks prediction-level +data. LLM-judge baselines are omitted here because +[`trail_llm_baselines.json`](https://github.com/Pisama-AI/pisama-detectors/blob/main/benchmarks/trail_llm_baselines.json) +carries confusion counts but `"result": null` for every model, so no comparable joint-accuracy +figure exists in the artifact. See the +[reproducibility boundary](https://github.com/Pisama-AI/pisama-detectors/tree/main/benchmarks). **Who&When** (ICML 2025, multi-agent attribution, 58 hand-crafted cases): From b9d518113623ded2987e19a36eaa5976d51a9282 Mon Sep 17 00:00:00 2001 From: tn-pisama Date: Tue, 28 Jul 2026 12:36:16 -0700 Subject: [PATCH 2/2] fix(deps): upper-bound the mcp extra so the documented MCP server starts `pip install "pisama[mcp]"` declared `mcp>=1.0.0` with no upper bound, so a fresh install today resolves mcp 2.0.0 and the documented quickstart is dead on arrival: $ pisama mcp-server File ".../pisama/mcp/server.py", line 138, in create_local_server @server.list_tools() AttributeError: 'Server' object has no attribute 'list_tools' pisama/mcp/server.py uses the mcp 1.x decorator API, which 2.0.0 removed. Verified against the published 0.5.5: clean venv + `pip install "pisama[mcp]"` pulls mcp 2.0.0 and crashes as above; the same venv with `"mcp<2"` pulls 1.29.0 and serves pisama_analyze, pisama_detect, pisama_status, pisama_explain. Installing from this patched source resolves 1.29.0 and create_local_server() returns a Server. Pinning rather than porting: the port to the 2.x API is real work and this unbreaks the install path today. The comment says to lift the bound once the server is ported. Co-Authored-By: Claude --- pyproject.toml | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/pyproject.toml b/pyproject.toml index 2e0e169..96ac61e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -40,7 +40,12 @@ dependencies = [ ] [project.optional-dependencies] -mcp = ["mcp>=1.0.0"] +# Upper-bounded: pisama/mcp/server.py uses the mcp 1.x decorator API +# (@server.list_tools()), which mcp 2.0.0 removed. Without the bound, a fresh +# `pip install "pisama[mcp]"` resolves mcp 2.0.0 and `pisama mcp-server` dies at +# server.py:138 with AttributeError: 'Server' object has no attribute +# 'list_tools'. Lift this only once the server is ported to the 2.x API. +mcp = ["mcp>=1.0.0,<2"] auto = ["pisama-auto>=0.1.0", "opentelemetry-api>=1.20.0", "opentelemetry-sdk>=1.20.0"] langgraph = ["langgraph>=0.2.0", "langchain-core>=0.3.0"] dev = [