Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,30 @@ All notable changes to the `pisama` meta-package are documented here. The packag

## [Unreleased]

## [0.5.7] - 2026-07-29

### Changed

- Disclose the precision boundary of the archived TRAIL run beside the F1
numbers in the README. The archive scored only annotated errors, so no false
positive was recordable (`fp = 0` in 14 of 14 categories, `prediction_count`
equals `mapped_annotations` at 813). Precision is 1.000 by construction rather
than by measurement, F1 reduces to `2R / (1 + R)`, and the informative figure
is micro-recall 0.5953. The macro-F1 0.7535 and micro-F1 0.7463 values remain
arithmetically reproducible; they simply carry no precision information. This
caveat already shipped in the sibling `pisama-detectors` README and in
`benchmarks/evidence.json`, and is now carried by the package people install.

## [0.5.6] - 2026-07-28

### Fixed

- Pin `mcp>=1.0.0,<2` so `pip install "pisama[mcp]"` stops resolving mcp 2.0.0,
which removed the 1.x decorator API that `pisama mcp-server` uses.
- Correct the published README: drop four detectors that do not exist
(`delegation`, `grounding`, `retrieval_quality`, `compaction_quality`), drop a
false framework-specific-detectors claim, and correct the TRAIL table.

## [0.5.5] - 2026-07-25

### Changed
Expand Down
12 changes: 11 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)

Pisama ships 32 core heuristic detectors. They apply across frameworks including n8n, LangGraph, Dify and OpenClaw, with per-platform gating (for example `coordination` runs only on multi-agent platforms). They run locally with zero LLM cost on the heuristic tier. An archived, in-distribution run on the [TRAIL benchmark](https://arxiv.org/abs/2505.08638) reports 59.9% joint accuracy (span and category). It is not a held-out result: 144 of the 148 traces appeared in calibration material, and the published archive does not contain the prediction-level data needed to recompute joint accuracy. The public confusion counts do reproduce the reported macro-F1 (0.7535) and micro-F1 (0.7463). See the [benchmark evidence and its reproducibility boundary](https://github.com/Pisama-AI/pisama-detectors/tree/main/benchmarks).
Pisama ships 32 core heuristic detectors. They apply across frameworks including n8n, LangGraph, Dify and OpenClaw, with per-platform gating (for example `coordination` runs only on multi-agent platforms). They run locally with zero LLM cost on the heuristic tier. An archived, in-distribution run on the [TRAIL benchmark](https://arxiv.org/abs/2505.08638) reports 59.9% joint accuracy (span and category). It is not a held-out result: 144 of the 148 traces appeared in calibration material, and the published archive does not contain the prediction-level data needed to recompute joint accuracy. The public confusion counts do reproduce the reported macro-F1 (0.7535) and micro-F1 (0.7463), but those F1 values carry no precision information: the archive recorded no false positives, so they reduce to recall (micro-recall 0.5953). See the [benchmark evidence and its reproducibility boundary](https://github.com/Pisama-AI/pisama-detectors/tree/main/benchmarks).

## Install

Expand Down Expand Up @@ -85,6 +85,16 @@ Works in Cursor, Claude Desktop, and Windsurf. No API key is needed:
| Pisama archived run | 59.9% |

Reproducible from the archive: macro-F1 0.7535, micro-F1 0.7463 (`benchmarks/verify_report.py`).

**Read those F1 figures as a recall measurement.** The archive scored only annotated errors,
so no false positive was recordable: `fp = 0` in 14 of 14 categories, and `prediction_count`
equals `mapped_annotations` (813). Precision is therefore 1.000 by construction rather than by
measurement, and F1 collapses to `2R / (1 + R)`, carrying no information beyond recall. The
informative figure is micro-recall 0.5953: the heuristic tier alone missed roughly 40% of the
labelled failures. The same boundary is recorded in
[`benchmarks/evidence.json`](https://github.com/Pisama-AI/pisama-detectors/blob/main/benchmarks/evidence.json)
and in the [`pisama-detectors` README](https://github.com/Pisama-AI/pisama-detectors#archived-trail-platform-benchmark).

Joint accuracy is **not** recomputable from the public archive, which lacks prediction-level
data. LLM-judge baselines are omitted here because
[`trail_llm_baselines.json`](https://github.com/Pisama-AI/pisama-detectors/blob/main/benchmarks/trail_llm_baselines.json)
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "hatchling.build"

[project]
name = "pisama"
version = "0.5.6"
version = "0.5.7"
description = "Multi-agent failure detection for production AI systems"
readme = "README.md"
license = "MIT"
Expand Down
2 changes: 1 addition & 1 deletion src/pisama/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@

__version__ = _pkg_version("pisama")
except PackageNotFoundError: # running from a source checkout without an install
__version__ = "0.5.6"
__version__ = "0.5.7"

from pisama._analyze import AnalyzeResult, Issue, analyze, async_analyze
from pisama._http import PisamaAuthError
Expand Down
Loading