Skip to content

docs: disclose the TRAIL precision boundary in the installed package (0.5.7) - #12

Merged
tn-pisama merged 1 commit into
mainfrom
docs/trail-precision-caveat
Jul 29, 2026
Merged

docs: disclose the TRAIL precision boundary in the installed package (0.5.7)#12
tn-pisama merged 1 commit into
mainfrom
docs/trail-precision-caveat

Conversation

@tn-pisama

Copy link
Copy Markdown
Contributor

Why

The README publishes TRAIL macro-F1 0.7535 and micro-F1 0.7463 as "reproducible from the
archive" and already discloses the in-distribution caveat (144 of 148 traces appeared in
calibration material). It omits the precision caveat that the sibling
pisama-detectors README
already carries. pisama is the package people actually install, and PyPI renders this
README, so the caveat needs to live here too.

The fact, verified from the artifact

Parsed from benchmarks/trail.json in Pisama-AI/pisama-detectors:

Check Value
Categories with fp = 0 14 of 14
prediction_count 813
mapped_annotations 813
micro-recall (tp 481 / support 808) 0.5953
2R / (1 + R) 0.7463 (equals the reported micro-F1 exactly)
macro-F1 recomputed from the 14 per-category F1 values 0.7535 (equals the reported value)

The archive scored only annotated errors, so no false positive was recordable. Precision is
1.000 by construction rather than by measurement, and F1 reduces to exactly 2R / (1 + R),
carrying no information beyond recall. benchmarks/evidence.json records the same boundary:
"The archive does not contain a negative candidate set that independently establishes
precision."

What this PR does NOT claim

The F1 figures are not withdrawn or corrected. They are arithmetically reproducible from the
public confusion counts, exactly as the README says. The only addition is that they carry no
precision information, and that the informative figure is micro-recall 0.5953: the heuristic
tier alone missed roughly 40% of the labelled failures.

Changes

  • README.md: precision caveat added directly under the F1 line in Benchmark Results, plus
    a one-clause version in the intro paragraph (the part PyPI shows first).
  • pyproject.toml, src/pisama/__init__.py: 0.5.6 to 0.5.7. A README fix only reaches users
    through a release, since PyPI renders the README from the published artifact.
  • CHANGELOG.md: 0.5.7 entry, plus a backfilled 0.5.6 entry. The 0.5.6 release commit
    (f17a80f) bumped the version without a changelog entry, so the file jumped 0.5.5 to 0.5.7.
    The backfilled text is taken from that commit's own message and diff.

Verification run locally (Python 3.13)

  • ruff check src tests passed
  • mypy src/pisama passed, no issues in 36 source files
  • pytest -q --cov=pisama --cov-fail-under=60 passed, 59 passed, coverage 61.73%
  • python -m build plus twine check dist/* passed on both artifacts
  • Confirmed the sdist PKG-INFO (what PyPI renders) carries the new caveat text and
    Version: 0.5.7; pisama --version reports 0.5.7

No source behavior changed. Docs, version, changelog only.

🤖 Generated with Claude Code

The README publishes macro-F1 0.7535 and micro-F1 0.7463 as reproducible from
the archive and already discloses the in-distribution caveat, but omits the
precision caveat its sibling pisama-detectors README carries. `pisama` is the
package people actually install, so the caveat needs to be here.

Verified from benchmarks/trail.json in Pisama-AI/pisama-detectors: fp = 0 in
14 of 14 categories, and prediction_count == mapped_annotations == 813. The
archive scored only annotated errors, so no false positive was recordable.
Precision is 1.000 by construction rather than by measurement, F1 reduces to
exactly 2R / (1 + R), and the informative figure is micro-recall 0.5953.

The F1 values are not withdrawn. They are arithmetically reproducible from the
public confusion counts; they simply carry no precision information.

Also backfills the 0.5.6 CHANGELOG entry, which the 0.5.6 release commit
(f17a80f) omitted.

Co-Authored-By: Claude <noreply@anthropic.com>
@tn-pisama
tn-pisama merged commit cce62e1 into main Jul 29, 2026
9 checks passed
@tn-pisama
tn-pisama deleted the docs/trail-precision-caveat branch July 29, 2026 15:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant