docs: correct false detector list, framework claim, and TRAIL benchmark table - #10
Merged
Conversation
Three claims were verified false against the published 0.5.5 artifact and against this project's own benchmark evidence. 1. Four documented detectors do not exist. `delegation`, `grounding`, `retrieval_quality` and `compaction_quality` all raise UnknownDetectorError; the installed registry has exactly 32 names and none of them is present. Replaced with four that do exist and were undocumented: propagation, citation, routing, mcp_protocol (each verified to run). 2. "plus framework-specific detectors for n8n, LangGraph, Dify, and OpenClaw" is not true. registry.count == 32 and get_for_platform() returns subsets of that same set (n8n 31, langgraph 32, dify 32, openclaw 32). There is no detector unique to any framework; the only difference is per-platform gating, e.g. coordination is restricted to multi-agent platforms. 3. The TRAIL table presented 59.9% against 11.9% (GPT-5.4) and 6.8% (Gemini 3.1 Pro) with no caveat. Our own linked artifact contradicts it. evidence.json says verbatim: "The run is in-distribution because 144 of 148 traces appeared in calibration material", "The run does not establish held-out generalization", and "The archive does not contain the prediction-level data needed to reproduce joint accuracy". trail_llm_baselines.json carries confusion counts but "result": null for all four models, so 11.9% and 6.8% are not sourced from the artifact the README cites as evidence. The table now reports only the figure the archive supports, states the in-distribution caveat inline, and names macro-F1 0.7535 / micro-F1 0.7463 as the numbers that ARE reproducible via benchmarks/verify_report.py. The LLM-judge comparison is removed rather than restated, because no comparable number exists in the evidence. The evidence.json claim boundaries were already careful and honest. The README simply did not carry them, which turns rigorous internal work into an unsupportable public claim. Co-Authored-By: Claude <noreply@anthropic.com>
`pip install "pisama[mcp]"` declared `mcp>=1.0.0` with no upper bound, so a fresh
install today resolves mcp 2.0.0 and the documented quickstart is dead on
arrival:
$ pisama mcp-server
File ".../pisama/mcp/server.py", line 138, in create_local_server
@server.list_tools()
AttributeError: 'Server' object has no attribute 'list_tools'
pisama/mcp/server.py uses the mcp 1.x decorator API, which 2.0.0 removed.
Verified against the published 0.5.5: clean venv + `pip install "pisama[mcp]"`
pulls mcp 2.0.0 and crashes as above; the same venv with `"mcp<2"` pulls 1.29.0
and serves pisama_analyze, pisama_detect, pisama_status, pisama_explain.
Installing from this patched source resolves 1.29.0 and create_local_server()
returns a Server.
Pinning rather than porting: the port to the 2.x API is real work and this
unbreaks the install path today. The comment says to lift the bound once the
server is ported.
Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Verified against the published 0.5.5 artifact and this project's own benchmark evidence.
1. Four documented detectors do not exist.
delegation,grounding,retrieval_quality,compaction_qualityall raiseUnknownDetectorError. Replaced with four that exist and were undocumented.2. "framework-specific detectors" is false.
registry.count == 32;get_for_platform()returns subsets of the same 32. No detector is unique to n8n/LangGraph/Dify/OpenClaw.3. The TRAIL table was unsupportable. Our own
evidence.jsonsays the run is in-distribution (144/148 traces in calibration material), does not establish held-out generalization, and lacks the data to reproduce joint accuracy.trail_llm_baselines.jsonhas"result": nullfor all four models, so 11.9% and 6.8% are not in the cited artifact.Now reports only what the archive supports, with the caveat inline, and names macro-F1 0.7535 / micro-F1 0.7463 as reproducible via
verify_report.py.🤖 Generated with Claude Code