From fcc9c0c889e0df8780d9521c38b66c9eb1a81d53 Mon Sep 17 00:00:00 2001 From: tn-pisama Date: Wed, 29 Jul 2026 08:30:32 -0700 Subject: [PATCH] docs: disclose the TRAIL precision boundary in the installed package The README publishes macro-F1 0.7535 and micro-F1 0.7463 as reproducible from the archive and already discloses the in-distribution caveat, but omits the precision caveat its sibling pisama-detectors README carries. `pisama` is the package people actually install, so the caveat needs to be here. Verified from benchmarks/trail.json in Pisama-AI/pisama-detectors: fp = 0 in 14 of 14 categories, and prediction_count == mapped_annotations == 813. The archive scored only annotated errors, so no false positive was recordable. Precision is 1.000 by construction rather than by measurement, F1 reduces to exactly 2R / (1 + R), and the informative figure is micro-recall 0.5953. The F1 values are not withdrawn. They are arithmetically reproducible from the public confusion counts; they simply carry no precision information. Also backfills the 0.5.6 CHANGELOG entry, which the 0.5.6 release commit (f17a80f) omitted. Co-Authored-By: Claude --- CHANGELOG.md | 24 ++++++++++++++++++++++++ README.md | 12 +++++++++++- pyproject.toml | 2 +- src/pisama/__init__.py | 2 +- 4 files changed, 37 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e1415a5..5a62794 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,30 @@ All notable changes to the `pisama` meta-package are documented here. The packag ## [Unreleased] +## [0.5.7] - 2026-07-29 + +### Changed + +- Disclose the precision boundary of the archived TRAIL run beside the F1 + numbers in the README. The archive scored only annotated errors, so no false + positive was recordable (`fp = 0` in 14 of 14 categories, `prediction_count` + equals `mapped_annotations` at 813). Precision is 1.000 by construction rather + than by measurement, F1 reduces to `2R / (1 + R)`, and the informative figure + is micro-recall 0.5953. The macro-F1 0.7535 and micro-F1 0.7463 values remain + arithmetically reproducible; they simply carry no precision information. This + caveat already shipped in the sibling `pisama-detectors` README and in + `benchmarks/evidence.json`, and is now carried by the package people install. + +## [0.5.6] - 2026-07-28 + +### Fixed + +- Pin `mcp>=1.0.0,<2` so `pip install "pisama[mcp]"` stops resolving mcp 2.0.0, + which removed the 1.x decorator API that `pisama mcp-server` uses. +- Correct the published README: drop four detectors that do not exist + (`delegation`, `grounding`, `retrieval_quality`, `compaction_quality`), drop a + false framework-specific-detectors claim, and correct the TRAIL table. + ## [0.5.5] - 2026-07-25 ### Changed diff --git a/README.md b/README.md index 48d5d95..5f2809f 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE) [![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/) -Pisama ships 32 core heuristic detectors. They apply across frameworks including n8n, LangGraph, Dify and OpenClaw, with per-platform gating (for example `coordination` runs only on multi-agent platforms). They run locally with zero LLM cost on the heuristic tier. An archived, in-distribution run on the [TRAIL benchmark](https://arxiv.org/abs/2505.08638) reports 59.9% joint accuracy (span and category). It is not a held-out result: 144 of the 148 traces appeared in calibration material, and the published archive does not contain the prediction-level data needed to recompute joint accuracy. The public confusion counts do reproduce the reported macro-F1 (0.7535) and micro-F1 (0.7463). See the [benchmark evidence and its reproducibility boundary](https://github.com/Pisama-AI/pisama-detectors/tree/main/benchmarks). +Pisama ships 32 core heuristic detectors. They apply across frameworks including n8n, LangGraph, Dify and OpenClaw, with per-platform gating (for example `coordination` runs only on multi-agent platforms). They run locally with zero LLM cost on the heuristic tier. An archived, in-distribution run on the [TRAIL benchmark](https://arxiv.org/abs/2505.08638) reports 59.9% joint accuracy (span and category). It is not a held-out result: 144 of the 148 traces appeared in calibration material, and the published archive does not contain the prediction-level data needed to recompute joint accuracy. The public confusion counts do reproduce the reported macro-F1 (0.7535) and micro-F1 (0.7463), but those F1 values carry no precision information: the archive recorded no false positives, so they reduce to recall (micro-recall 0.5953). See the [benchmark evidence and its reproducibility boundary](https://github.com/Pisama-AI/pisama-detectors/tree/main/benchmarks). ## Install @@ -85,6 +85,16 @@ Works in Cursor, Claude Desktop, and Windsurf. No API key is needed: | Pisama archived run | 59.9% | Reproducible from the archive: macro-F1 0.7535, micro-F1 0.7463 (`benchmarks/verify_report.py`). + +**Read those F1 figures as a recall measurement.** The archive scored only annotated errors, +so no false positive was recordable: `fp = 0` in 14 of 14 categories, and `prediction_count` +equals `mapped_annotations` (813). Precision is therefore 1.000 by construction rather than by +measurement, and F1 collapses to `2R / (1 + R)`, carrying no information beyond recall. The +informative figure is micro-recall 0.5953: the heuristic tier alone missed roughly 40% of the +labelled failures. The same boundary is recorded in +[`benchmarks/evidence.json`](https://github.com/Pisama-AI/pisama-detectors/blob/main/benchmarks/evidence.json) +and in the [`pisama-detectors` README](https://github.com/Pisama-AI/pisama-detectors#archived-trail-platform-benchmark). + Joint accuracy is **not** recomputable from the public archive, which lacks prediction-level data. LLM-judge baselines are omitted here because [`trail_llm_baselines.json`](https://github.com/Pisama-AI/pisama-detectors/blob/main/benchmarks/trail_llm_baselines.json) diff --git a/pyproject.toml b/pyproject.toml index dea2652..96d2618 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "pisama" -version = "0.5.6" +version = "0.5.7" description = "Multi-agent failure detection for production AI systems" readme = "README.md" license = "MIT" diff --git a/src/pisama/__init__.py b/src/pisama/__init__.py index 1c7a563..995a527 100644 --- a/src/pisama/__init__.py +++ b/src/pisama/__init__.py @@ -14,7 +14,7 @@ __version__ = _pkg_version("pisama") except PackageNotFoundError: # running from a source checkout without an install - __version__ = "0.5.6" + __version__ = "0.5.7" from pisama._analyze import AnalyzeResult, Issue, analyze, async_analyze from pisama._http import PisamaAuthError