docs: disclose the TRAIL precision boundary in the installed package (0.5.7) - #12
Merged
Conversation
The README publishes macro-F1 0.7535 and micro-F1 0.7463 as reproducible from the archive and already discloses the in-distribution caveat, but omits the precision caveat its sibling pisama-detectors README carries. `pisama` is the package people actually install, so the caveat needs to be here. Verified from benchmarks/trail.json in Pisama-AI/pisama-detectors: fp = 0 in 14 of 14 categories, and prediction_count == mapped_annotations == 813. The archive scored only annotated errors, so no false positive was recordable. Precision is 1.000 by construction rather than by measurement, F1 reduces to exactly 2R / (1 + R), and the informative figure is micro-recall 0.5953. The F1 values are not withdrawn. They are arithmetically reproducible from the public confusion counts; they simply carry no precision information. Also backfills the 0.5.6 CHANGELOG entry, which the 0.5.6 release commit (f17a80f) omitted. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The README publishes TRAIL macro-F1 0.7535 and micro-F1 0.7463 as "reproducible from the
archive" and already discloses the in-distribution caveat (144 of 148 traces appeared in
calibration material). It omits the precision caveat that the sibling
pisama-detectorsREADMEalready carries.
pisamais the package people actually install, and PyPI renders thisREADME, so the caveat needs to live here too.
The fact, verified from the artifact
Parsed from
benchmarks/trail.jsoninPisama-AI/pisama-detectors:fp = 0prediction_countmapped_annotations2R / (1 + R)The archive scored only annotated errors, so no false positive was recordable. Precision is
1.000 by construction rather than by measurement, and F1 reduces to exactly
2R / (1 + R),carrying no information beyond recall.
benchmarks/evidence.jsonrecords the same boundary:"The archive does not contain a negative candidate set that independently establishes
precision."
What this PR does NOT claim
The F1 figures are not withdrawn or corrected. They are arithmetically reproducible from the
public confusion counts, exactly as the README says. The only addition is that they carry no
precision information, and that the informative figure is micro-recall 0.5953: the heuristic
tier alone missed roughly 40% of the labelled failures.
Changes
README.md: precision caveat added directly under the F1 line in Benchmark Results, plusa one-clause version in the intro paragraph (the part PyPI shows first).
pyproject.toml,src/pisama/__init__.py: 0.5.6 to 0.5.7. A README fix only reaches usersthrough a release, since PyPI renders the README from the published artifact.
CHANGELOG.md: 0.5.7 entry, plus a backfilled 0.5.6 entry. The 0.5.6 release commit(f17a80f) bumped the version without a changelog entry, so the file jumped 0.5.5 to 0.5.7.
The backfilled text is taken from that commit's own message and diff.
Verification run locally (Python 3.13)
ruff check src testspassedmypy src/pisamapassed, no issues in 36 source filespytest -q --cov=pisama --cov-fail-under=60passed, 59 passed, coverage 61.73%python -m buildplustwine check dist/*passed on both artifactsPKG-INFO(what PyPI renders) carries the new caveat text andVersion: 0.5.7;pisama --versionreports 0.5.7No source behavior changed. Docs, version, changelog only.
🤖 Generated with Claude Code