Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/actions/jevcheck/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ inputs:
jevcheck-version:
description: PyPI version pin installed as `jevcheck==<version>`.
required: false
default: "0.2.0"
default: "0.2.1"
working-directory:
description: >
Directory to run the CLI from. Contract, replay, and `out` paths are
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.11", "3.12"]
python-version: ["3.11", "3.12", "3.13", "3.14"]
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
Expand Down
62 changes: 62 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# Changelog

All notable changes to this project are documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.2.1] - 2026-09-30

Library source since [0.2.0] is the version string only. CLI commands, exit codes, and eval rules are unchanged.

### Added

- Composite GitHub Action [`.github/actions/jevcheck`](https://github.com/sathariels/jevcheck/blob/main/.github/actions/jevcheck/action.yml). It installs `jevcheck` from PyPI and runs `eval`, `compare`, or `record`.
- Example workflow [`.github/workflows/jevcheck-example.yml`](https://github.com/sathariels/jevcheck/blob/main/.github/workflows/jevcheck-example.yml) for replay `eval` and `compare` with no API key.
- Optional [pr-triage workflow](https://github.com/sathariels/jevcheck/blob/main/.github/workflows/pr-triage.yml). It calls `sathariels/jevtriage@v0.1.0` with model `jev-1.13.0` only when `TYPESAFE_API_KEY` is set.
- First-run tutorial ([`docs/tutorial.md`](https://github.com/sathariels/jevcheck/blob/main/docs/tutorial.md)) and an offline [`examples/`](https://github.com/sathariels/jevcheck/blob/main/examples/README.md) pack for compatible (exit 0) and breaking (exit 1) replays with no API key.
- `SECURITY.md`, Dependabot, `requirements.lock`, and the Plugin Security Scan workflow. That change does not alter public CLI or Action behavior.
- This changelog and [`RELEASING.md`](https://github.com/sathariels/jevcheck/blob/main/RELEASING.md).

### Changed

- Package version is `0.2.1` in `pyproject.toml`, `jevcheck.__version__`, and the Action input `jevcheck-version` (default `0.2.1`).
- Documented Action refs pin `@v0.2.1`. That is the first tag that contains `.github/actions/jevcheck`. `v0.2.0` does not.
- License metadata is the SPDX expression `MIT`, with `license-files = ["LICENSE"]`. The `License :: OSI Approved :: MIT License` classifier is removed. Builds require `setuptools>=77`.
- Classifiers and the CI test matrix include Python 3.13 and 3.14. `requires-python` stays `>=3.11`.
- `project.urls` lists Homepage, Source, Documentation, Issues, and Changelog.
- README install leads with `pip install jevcheck`. README links to docs, examples, and workflows are absolute GitHub URLs.

## [0.2.0] - 2026-09-20

### Added

- `jevcheck record` writes a baseline replay JSON in the same shape as `eval --answers`. The contract schema stays `"0.1"`.
- `jevcheck compare` checks a candidate against that snapshot (`--from`) or fetches both models live (`--from-model` and `--to`).
- Record and compare summaries print `alias → resolved` when an opted-in alias resolves to a different model id.
- [ADR 009](https://github.com/sathariels/jevcheck/blob/v0.2.0/docs/adr-009-two-model-compare.md) locks this path. `eval`, identity rules, exit codes, and Gate stay as in 0.1.0.

### Fixed

- `record`, and `compare` when it loads a replay, reject an answer whose kind does not match the question (choice, noul, or score) instead of writing a snapshot `compare` later refuses.

## [0.1.0] - 2026-09-20

### Added

- Fixture-versus-candidate `jevcheck eval` for a v0.1 JSON or JSONL contract (baseline model, cases, and expected answers).
- Choice, noul, and score checks, including confidence floors and confidence drops.
- Explicit model pinning. `jev-latest` and `jev-preview` are rejected unless `--allow-unpinned`.
- CLI (`jevcheck` and `python -m jevcheck`), Jev client, and optional Gate helper.
- Contract loader, replay fixtures, the support-triage example, and API docs (`docs/contract.md`, `docs/jev-api.md`).
- GitHub Actions CI that runs a CLI eval smoke test on replay fixtures.

### Fixed

- Fail-closed model pinning and response identity, rejection of ineffective expects and invalid probability maps, inclusive float tolerances, and SDK HTTP errors as a distinct ops exit with secret redaction.
- A baseline-confidence-only expect needs a floor or tolerance after defaults. An opted-in floating alias accepts a concrete resolved response model. Replay and adapter paths require nonempty normalized score probabilities and a legend.
- The sdist ships tests and fixtures.

[0.2.1]: https://github.com/sathariels/jevcheck/compare/v0.2.0...v0.2.1
[0.2.0]: https://github.com/sathariels/jevcheck/compare/v0.1.0...v0.2.0
[0.1.0]: https://github.com/sathariels/jevcheck/releases/tag/v0.1.0
2 changes: 2 additions & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ graft docs
graft examples
include LICENSE
include README.md
include CHANGELOG.md
include RELEASING.md
include SECURITY.md
include pyproject.toml
include requirements.lock
Expand Down
35 changes: 18 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@

Probabilities and model versions move. A raw `0.94` is not a release decision. jevcheck records a **production contract** (baseline model + fixtures + expected answers) and evals a candidate against that fixture.

**New here?** Walk through the [first-run tutorial](docs/tutorial.md) (about 10 minutes, offline replay). Copy the [examples pack](examples/README.md) into your repo.
**New here?** Walk through the [first-run tutorial](https://github.com/sathariels/jevcheck/blob/main/docs/tutorial.md) (about 10 minutes, offline replay). Copy the [examples pack](https://github.com/sathariels/jevcheck/blob/main/examples/README.md) into your repo.

**v0.1** `eval` is fixture-versus-candidate. **v0.2** adds two-model execution: `record` a baseline model's answers, then `compare` a candidate against that snapshot (or fetch both models live). See [`docs/adr-009-two-model-compare.md`](docs/adr-009-two-model-compare.md).
**v0.1** `eval` is fixture-versus-candidate. **v0.2** adds two-model execution: `record` a baseline model's answers, then `compare` a candidate against that snapshot (or fetch both models live). See [`docs/adr-009-two-model-compare.md`](https://github.com/sathariels/jevcheck/blob/main/docs/adr-009-two-model-compare.md).

The repo’s `jev-1.13` / `jev-1.14` strings are **unverified example pin labels** used by fixtures. A documented TypeSafe version pin (2026-09-19 model list) is `jev-1.13.0`. Floating aliases `jev-latest` and `jev-preview` are rejected unless you pass `--allow-unpinned`. With that opt-in, a response whose `model` is the concrete resolved ID (for example `jev-1.13.0`) is accepted; the eval report prints that resolved model. Concrete pins still require exact identity.

Expand All @@ -22,15 +22,14 @@ Reports: **unchanged** / **confidence regressions** / **answer flips**, with exa
## Install

```bash
pip install -e ".[dev]"
export TYPESAFE_API_KEY=... # live Jev only; never commit this
pip install jevcheck
```

Auth is `TYPESAFE_API_KEY` only. Unit tests mock the network.
Auth is `TYPESAFE_API_KEY` only. Set it for live Jev calls and never commit the key. Unit tests mock the network.

## Pin → eval → upgrade

1. Write a contract (see [`docs/contract.md`](docs/contract.md)) against the model you ship.
1. Write a contract (see [`docs/contract.md`](https://github.com/sathariels/jevcheck/blob/main/docs/contract.md)) against the model you ship.
2. Call Jev with that **explicit** model. `jev-latest` and `jev-preview` are rejected unless you opt in. Concrete pins require `response.model` to match exactly. An opted-in alias may resolve to a nonempty concrete (non-alias) response model, which is reported.
3. Before upgrading, eval the candidate (example fixture label — not a verified live ID):

Expand Down Expand Up @@ -93,7 +92,7 @@ jevcheck compare fixtures/support-triage.json \

`--from` and `--from-model` are mutually exclusive. Identity rules are the v0.1 pins: concrete names must match `response.model` exactly; `jev-latest` / `jev-preview` still need `--allow-unpinned`. Exit codes stay 0 compatible / 1 breaking / 2 usage-or-identity / 3 ops.

Verified System One fields: [`docs/jev-api.md`](docs/jev-api.md).
Verified System One fields: [`docs/jev-api.md`](https://github.com/sathariels/jevcheck/blob/main/docs/jev-api.md).

## Example

Expand Down Expand Up @@ -121,12 +120,12 @@ A `Gate` helper exists for auto / ask-human / reject thresholds. It is optional

## GitHub Action

Reuse jevcheck from other repos as a [composite action](.github/actions/jevcheck/action.yml). The action sets up Python, installs `jevcheck` from PyPI, and runs `eval` (default), `compare`, or `record`. A nonzero CLI exit fails the job (exit 1 is a breaking contract).
Reuse jevcheck from other repos as a [composite action](https://github.com/sathariels/jevcheck/blob/main/.github/actions/jevcheck/action.yml). The action sets up Python, installs `jevcheck` from PyPI, and runs `eval` (default), `compare`, or `record`. A nonzero CLI exit fails the job (exit 1 is a breaking contract).

Composite actions live in a subdirectory, so the `uses:` ref must point at a commit or tag that contains `.github/actions/jevcheck`. **Use `@main` until a dedicated action tag exists:**
Composite actions live in a subdirectory, so the `uses:` ref must point at a commit or tag that contains `.github/actions/jevcheck`. Pin `@v0.2.1`, the first tag that includes this action:

```yaml
- uses: sathariels/jevcheck/.github/actions/jevcheck@main
- uses: sathariels/jevcheck/.github/actions/jevcheck@v0.2.1
with:
contract: contracts/support.json
command: compare
Expand All @@ -135,9 +134,9 @@ Composite actions live in a subdirectory, so the `uses:` ref must point at a com
answers: fixtures/candidate-replay.json
```

`v0.2.0` is the PyPI package tag and **does not include this action**. Do not retag that release. `uses: ...@v0.2.0` will fail (or stay on a tree without the action) until a new tag that contains `.github/actions/jevcheck` is cut — for example `action-v1` after this lands on `main`.
`v0.2.0` does not contain `.github/actions/jevcheck`. Pin `@v0.2.1` (or a later tag) for the Action.

This repo’s [example workflow](.github/workflows/jevcheck-example.yml) is the green CI proof: `eval` (and `compare`) on `fixtures/support-triage.json` + `fixtures/replay-unchanged.json`. No `TYPESAFE_API_KEY`. A breaking replay (`fixtures/replay-breaking.json`) exits 1; do not mark a job that uses it as a required check.
This repo’s [example workflow](https://github.com/sathariels/jevcheck/blob/main/.github/workflows/jevcheck-example.yml) is the green CI proof: `eval` (and `compare`) on `fixtures/support-triage.json` + `fixtures/replay-unchanged.json`. No `TYPESAFE_API_KEY`. A breaking replay (`fixtures/replay-breaking.json`) exits 1; do not mark a job that uses it as a required check.

### Inputs

Expand All @@ -154,7 +153,7 @@ This repo’s [example workflow](.github/workflows/jevcheck-example.yml) is the
| `out` | — | `record --out` (required when `command` is `record`) |
| `allow-unpinned` | `false` | `--allow-unpinned` |
| `python-version` | `3.12` | `actions/setup-python` |
| `jevcheck-version` | `0.2.0` | `pip install jevcheck==…` |
| `jevcheck-version` | `0.2.1` | `pip install jevcheck==…` |
| `working-directory` | — | `cd` before the CLI; paths are relative to it |

`candidate-model` and `to` are aliases for the same candidate id. If both are set they must match. `from` and `from-model` stay mutually exclusive, same as the CLI.
Expand All @@ -164,7 +163,7 @@ This repo’s [example workflow](.github/workflows/jevcheck-example.yml) is the
Omit `answers` (and use `from-model` instead of `from`) only when the job has a TypeSafe key:

```yaml
- uses: sathariels/jevcheck/.github/actions/jevcheck@main
- uses: sathariels/jevcheck/.github/actions/jevcheck@v0.2.1
env:
TYPESAFE_API_KEY: ${{ secrets.TYPESAFE_API_KEY }}
with:
Expand Down Expand Up @@ -192,12 +191,14 @@ python -m jevcheck compare fixtures/support-triage.json \

## Related

[jevtriage](https://github.com/sathariels/jevtriage) is an optional PR triage gate (`ready` / `needs_review` / `risky`). This repo’s [pr-triage workflow](.github/workflows/pr-triage.yml) runs it only when the `TYPESAFE_API_KEY` repository secret is set, so public forks and default CI stay green without the secret.
[jevtriage](https://github.com/sathariels/jevtriage) is an optional PR triage gate (`ready` / `needs_review` / `risky`). This repo’s [pr-triage workflow](https://github.com/sathariels/jevcheck/blob/main/.github/workflows/pr-triage.yml) runs it only when the `TYPESAFE_API_KEY` repository secret is set, so public forks and default CI stay green without the secret.

## For agents / audits

v0.2 two-model lock: [`docs/adr-009-two-model-compare.md`](docs/adr-009-two-model-compare.md).
v0.2 two-model lock: [`docs/adr-009-two-model-compare.md`](https://github.com/sathariels/jevcheck/blob/main/docs/adr-009-two-model-compare.md).

See [`docs/release-readiness-audit-v0.1.md`](https://github.com/sathariels/jevcheck/blob/main/docs/release-readiness-audit-v0.1.md). Recheck: [`docs/release-readiness-audit-v0.1-recheck.md`](https://github.com/sathariels/jevcheck/blob/main/docs/release-readiness-audit-v0.1-recheck.md).

See [`docs/release-readiness-audit-v0.1.md`](docs/release-readiness-audit-v0.1.md). Recheck: [`docs/release-readiness-audit-v0.1-recheck.md`](docs/release-readiness-audit-v0.1-recheck.md).
Release history: [`CHANGELOG.md`](https://github.com/sathariels/jevcheck/blob/main/CHANGELOG.md). Tag and publish steps: [`RELEASING.md`](https://github.com/sathariels/jevcheck/blob/main/RELEASING.md).

MIT.
91 changes: 91 additions & 0 deletions RELEASING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
# Releasing

Tag, GitHub Release, and PyPI upload happen after the release pull request is merged. The release pull request does not push a tag, open a GitHub Release, or upload to PyPI.

There is no publish workflow. `.github/workflows/` contains `ci.yml`, `jevcheck-example.yml`, `pr-triage.yml`, and `plugin-scan.yml` only. PyPI upload is a manual `twine upload` from the tag.

`v0.2.1` is the first tag that contains `.github/actions/jevcheck`. Documented `uses:` refs pin `@v0.2.1`. The composite action installs `jevcheck==0.2.1` from PyPI, so that install succeeds only after the upload below.

## Post-merge steps for 0.2.1

1. Merge the release pull request into `main`.
2. Tag that merge commit and push the tag:

```bash
git switch main
git pull origin main
# HEAD must be the merge commit of the 0.2.1 release PR.
git tag -a v0.2.1 -m "v0.2.1 — GitHub Action, tutorial, and packaging metadata"
git push origin v0.2.1
```

3. Create the GitHub Release `v0.2.1` from that tag. Title:

```text
v0.2.1 — GitHub Action, tutorial, and packaging metadata
```

Notes: use the [v0.2.1 release notes](#github-release-notes-for-v021) below. Paste them as the release body. They use `pip install jevcheck==0.2.1` with no "after PyPI publish" heading.

4. From a clean checkout of the tag, build and upload:

```bash
git switch --detach v0.2.1
python3 -m venv .venv-release
source .venv-release/bin/activate
python -m pip install --upgrade build twine
python -m build
python -m twine check dist/*
python -m twine upload dist/*
```

`twine check` should pass with no deprecation warnings. `twine upload` uses the owner's PyPI credentials for the `jevcheck` project.

5. After the upload, `.github/workflows/jevcheck-example.yml` on `main` can install `jevcheck==0.2.1`. Until then, that workflow's PyPI install fails. `ci.yml` does not install from PyPI.

If the tag date is not 2026-09-30, update the `[0.2.1]` heading in `CHANGELOG.md` on a follow-up commit. The compare link at the bottom of the changelog resolves once `v0.2.1` exists.

## GitHub release notes for v0.2.1

Paste the following as the release body:

~~~~markdown
## Highlights

- Composite GitHub Action `.github/actions/jevcheck`. Pin `sathariels/jevcheck/.github/actions/jevcheck@v0.2.1`. `v0.2.0` does not contain the Action.
- First-run tutorial (`docs/tutorial.md`) and an offline `examples/` pack. Replay eval and compare need no API key.
- Optional jevtriage pull-request workflow, skipped when `TYPESAFE_API_KEY` is unset.
- SPDX `license = "MIT"`, Python 3.13 and 3.14 in classifiers and CI, changelog, and Homepage / Source / Issues / Changelog URLs.
- CLI commands, exit codes, and eval rules match 0.2.0.

## Install

```
pip install jevcheck==0.2.1
```

## Action

```yaml
- uses: sathariels/jevcheck/.github/actions/jevcheck@v0.2.1
with:
contract: contracts/support.json
command: eval
candidate-model: jev-1.14
answers: fixtures/candidate-replay.json
```
~~~~

## Existing v0.2.0 GitHub Release

The published [v0.2.0 release](https://github.com/sathariels/jevcheck/releases/tag/v0.2.0) still says "Install (after PyPI publish)". PyPI has had `jevcheck==0.2.0` since 2026-09-19. A release pull request cannot edit that release. If the owner edits it by hand, replace that section with:

~~~~markdown
## Install

```
pip install jevcheck==0.2.0
```
~~~~

Leave the rest of the v0.2.0 highlights as published. Leave the `v0.2.0` tag where it is.
6 changes: 3 additions & 3 deletions docs/tutorial.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ pip install -e ".[dev]"
Or install the published package and copy the [examples pack](../examples/README.md) into your own repo:

```bash
pip install "jevcheck==0.2.0"
pip install "jevcheck==0.2.1"
```

`jevcheck` and `python -m jevcheck` run the same CLI. Use the module form when the `jevcheck` script is not on your `PATH`.
Expand Down Expand Up @@ -152,7 +152,7 @@ Other repos can call the [composite action](../.github/actions/jevcheck/action.y
Copy [`examples/jevcheck.yml`](../examples/jevcheck.yml) to `.github/workflows/jevcheck.yml`. It runs fixture-only `eval` and `compare` and does not set `TYPESAFE_API_KEY`. The breaking replay stays local so the workflow can remain a green check.

```yaml
- uses: sathariels/jevcheck/.github/actions/jevcheck@main
- uses: sathariels/jevcheck/.github/actions/jevcheck@v0.2.1
with:
contract: examples/support.json
command: compare
Expand All @@ -161,7 +161,7 @@ Copy [`examples/jevcheck.yml`](../examples/jevcheck.yml) to `.github/workflows/j
answers: examples/replay-unchanged.json
```

Use `@main` until a tag that contains `.github/actions/jevcheck` exists. The PyPI tag `v0.2.0` does not include the action. This repo's in-tree proof is [`.github/workflows/jevcheck-example.yml`](../.github/workflows/jevcheck-example.yml), which points `uses:` at `./.github/actions/jevcheck` and the fuller `fixtures/` set.
Pin `@v0.2.1`, the first tag that contains `.github/actions/jevcheck`. `v0.2.0` does not include the action. This repo's in-tree proof is [`.github/workflows/jevcheck-example.yml`](../.github/workflows/jevcheck-example.yml), which points `uses:` at `./.github/actions/jevcheck` and the fuller `fixtures/` set.

## 7. Optional PR gate (jevtriage)

Expand Down
4 changes: 2 additions & 2 deletions examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,6 @@ Exit 1.

Copy `jevcheck.yml` to `.github/workflows/jevcheck.yml`. Leave this directory at the repo root so the workflow paths (`examples/support.json`, `examples/replay-unchanged.json`, `examples/replay-baseline.json`) resolve. If you move the JSON files, change those inputs.

The workflow calls `sathariels/jevcheck/.github/actions/jevcheck@main`, installs `jevcheck==0.2.0` from PyPI, and runs `eval` plus `compare` on the unchanged replay. It does not set `TYPESAFE_API_KEY`. The breaking replay is omitted because exit 1 fails the job.
The workflow calls `sathariels/jevcheck/.github/actions/jevcheck@v0.2.1`, installs `jevcheck==0.2.1` from PyPI, and runs `eval` plus `compare` on the unchanged replay. It does not set `TYPESAFE_API_KEY`. The breaking replay is omitted because exit 1 fails the job.

Use `@main` until a tag that contains `.github/actions/jevcheck` exists. `v0.2.0` is the PyPI package tag and does not include the action.
Pin `@v0.2.1`, the first tag that contains `.github/actions/jevcheck`. `v0.2.0` does not include the action.
Loading
Loading