Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 28 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,32 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

## [0.2.1] - 2026-09-02

Two ways a run could report `looks safe to adopt` over something it never measured are
closed, and the first error a stranger hits no longer hands them the command they just ran.

**A green exit code is still not proof of full coverage.** A scenario with no recorded
baseline is disclosed everywhere a reviewer looks, and costs the run its affirmative
clearance, but it does not by itself fail the build (ADR-0033). A published Report's headline
also does not yet qualify itself when scenarios were skipped. Read the coverage disclosure,
not the exit code.

### Fixed
- **User-facing error text is text, not markup.** `[M]` A user whose install lacked the SDK
extra was told to run `pip install 'modelpin'` — the command they had just run — because
rich parsed `[providers]` as a style tag and ate it. It is the first error such a user
hits. The same defect had a quieter half: a message containing anything bracket-shaped was
silently rewritten, with no exception, naming something that did not exist. Errors are now
escaped once at the root rather than at each call site, where the class had been closed
five separate times.
- **`modelpin scan` reported file paths that do not exist**, and `modelpin init` misdescribed
the directory you had just named. `[M]` A folder called `src [experimental]` rendered as
`src ` — a rich table parses its cells as markup, and those cells are paths discovered in
your repo. Any project using bracketed directory names, such as Next.js route folders, saw
every affected row corrupted.


### Added
- **The semantic judge runs cross-vendor.** It accepted OpenAI judge models only, which left a
hole with teeth: cross-vendor is a core promise, and for a suite that declares no `tools` the
Expand Down Expand Up @@ -870,7 +896,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
opinion-framed Markdown + JSON report.
- BYO-key throughout, with key-shaped-secret scrubbing on all output.

[Unreleased]: https://github.com/samarthputhraya/modelpin/compare/v0.2.0...HEAD
[Unreleased]: https://github.com/samarthputhraya/modelpin/compare/v0.2.1...HEAD
[0.2.1]: https://github.com/samarthputhraya/modelpin/compare/v0.2.0...v0.2.1
[0.2.0]: https://github.com/samarthputhraya/modelpin/compare/v0.1.2...v0.2.0
[0.1.2]: https://github.com/samarthputhraya/modelpin/compare/v0.1.1...v0.1.2
[0.1.1]: https://github.com/samarthputhraya/modelpin/compare/v0.1.0...v0.1.1
Expand Down
15 changes: 9 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Install (Python 3.12+):

```bash
pip install "modelpin[providers]" # or: pipx install "modelpin[providers]"
modelpin version # -> modelpin 0.2.0
modelpin version # -> modelpin 0.2.1
```

> **Windows PowerShell:** run `modelpin …`, not `mp …`. PowerShell ships a built-in `mp` alias
Expand Down Expand Up @@ -73,10 +73,13 @@ dated copy under `.modelpin/runs/` that the next run will not overwrite, for cit
| `invoice_parse` | `changed_minor` | `"Total: $5"` → `"Total: 5"` breaks the scenario's assertion, but nothing refused and no tool moved |

`modelpin check` exits **1** only on a real `regression` — that's the CI gate, and it is why
the demo exits 1. It also exits **3** when a scenario it compared could not be measured, or when
nothing was compared at all — a different claim from "it broke". A scenario with no recorded
baseline is named in the report and costs the run its clearance, but does not fail the build. Then edit `traces.json`, re-run, and watch the verdict move:
the answer is computed from the traces, not baked in.
the demo exits 1. It also exits **3** when a scenario it compared could not be measured, when the
provider rejected one, or when nothing could be compared at all — a different claim from "it
broke". A scenario with no recorded baseline is named in the report and costs the run its
clearance, but does not by itself fail the build.

Then edit `traces.json`, re-run, and watch the verdict move: the answer is computed from the
traces, not baked in.

None of this is bundled inside the installed package — the wheel is code only, and the demo is
generated on your machine. That is deliberate: shipping them would mean the quickstart depends
Expand Down Expand Up @@ -518,7 +521,7 @@ is what keeps the false-positive promise honest and the tool small enough to tru
## Status

**Phase 0 (core engine MVP) — detection demonstrated but NOT characterised; the false-positive half is NOT met**
(see [`docs/fp-measurement.md`](https://github.com/samarthputhraya/modelpin/blob/main/docs/fp-measurement.md)); `v0.2.0` live on PyPI. Live-validated cross-vendor
(see [`docs/fp-measurement.md`](https://github.com/samarthputhraya/modelpin/blob/main/docs/fp-measurement.md)); `v0.2.1` live on PyPI. Live-validated cross-vendor
(OpenAI ↔ Google ↔ Groq/Llama); **false-positive rate not established** (the "0 in 8 held-out
trials" claim is withdrawn — those 8 could not have fired, so the honest score is 0/0); multi-turn replay; a real
GitHub Action; the public-report engine (`mp report`) + the open suite (in this repo, not
Expand Down
2 changes: 1 addition & 1 deletion modelpin/__init__.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
"""Modelpin — Dependabot for AI models. Know before the model breaks you."""

__version__ = "0.2.0"
__version__ = "0.2.1"
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[project]
name = "modelpin"
version = "0.2.0"
version = "0.2.1"
description = "Dependabot for AI models — catch model-migration regressions before they hit production."
readme = "README.md"
requires-python = ">=3.12"
Expand Down