Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/actions/jevcheck/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ inputs:
jevcheck-version:
description: PyPI version pin installed as `jevcheck==<version>`.
required: false
default: "0.2.1"
default: "0.2.2"
working-directory:
description: >
Directory to run the CLI from. Contract, replay, and `out` paths are
Expand All @@ -89,7 +89,7 @@ runs:
using: composite
steps:
- name: Set up Python
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: ${{ inputs.python-version }}

Expand Down
10 changes: 5 additions & 5 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,24 +15,24 @@ jobs:
matrix:
python-version: ["3.11", "3.12", "3.13", "3.14"]
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: ${{ matrix.python-version }}
- name: Install
run: pip install -e ".[dev]"
- name: Test
run: pytest
run: pytest --cov=jevcheck --cov-report=term-missing

packaging:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- name: Build wheel and sdist
Expand Down
9 changes: 7 additions & 2 deletions .github/workflows/jevcheck-example.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,15 @@ jobs:
name: eval unchanged replay
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Gate candidate against contract (replay, no API key)
uses: ./.github/actions/jevcheck
with:
# Action default is 0.2.2, which is not on PyPI until the post-merge
# upload. Install the published package so this check stays green.
jevcheck-version: "0.2.1"
contract: fixtures/support-triage.json
command: eval
candidate-model: jev-1.14
Expand All @@ -32,12 +35,14 @@ jobs:
name: compare unchanged replay
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Compare candidate replay to recorded baseline
uses: ./.github/actions/jevcheck
with:
# Same published pin as the eval job. Drop this input after 0.2.2 is on PyPI.
jevcheck-version: "0.2.1"
contract: fixtures/support-triage.json
command: compare
from: fixtures/replay-baseline.json
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/plugin-scan.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@ jobs:
scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: hashgraph-online/ai-plugin-scanner-action@caba2e96aa8ad2feb6cf6fca52442b52e22e779f # v1.2.635
- uses: hashgraph-online/ai-plugin-scanner-action@6f088dedc8f1f035eddd1597de8eacd8262a3862 # v1.2.777
with:
plugin_dir: "."
min_score: 80
Expand Down
19 changes: 19 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,24 @@ All notable changes to this project are documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.2.2] - 2026-10-13

### Added

- Eval tests for tolerance edges, several expectations on one case, wrong-type and missing answers, identity failures, and alias resolution.
- `pytest-cov` on the `dev` extra. CI prints a coverage report and does not fail on a coverage threshold.

### Changed

- `requirements.lock` pins `typesafe-sdk==0.7.4`. The `pyproject.toml` floor stays `typesafe-sdk>=0.7.0`. Releases 0.7.1 through 0.7.4 are bug fixes, an optional `http2` extra, stricter request validation, and non-finite number serialization. They do not change a successful System One call or jevcheck's response adaptation.
- GitHub Actions stay pinned by commit SHA: `actions/checkout` 7.0.1, `actions/setup-python` 7.0.0, and `hashgraph-online/ai-plugin-scanner-action` 1.2.777 (from 4.2.2, 5.6.0, and 1.2.635). Checkout v5+ and setup-python v6+ run on Node 24 and need Actions runner v2.327.1 or newer. Checkout v7 refuses fork-PR checkouts for `pull_request_target` and `workflow_run`; these workflows do not use those events. `persist-credentials` and `python-version` are unchanged. setup-python v7 removes the unused `pip-install` input. The scanner bump is a patch; `plugin_dir`, `min_score`, and `fail_on_severity` are unchanged.
- Package version is `0.2.2` in `pyproject.toml`, `jevcheck.__version__`, and the Action input `jevcheck-version` (default `0.2.2`). Documented Action refs pin `@v0.2.2`. `@v0.2.1` remains the first tag that contains the Action.
- The v0.1 recheck's four P1/P2 findings are marked resolved in be58041 (v0.2.0).

### Fixed

- `evaluate()` identity-checks the stripped candidate returned by `require_pinned`. A concrete name with surrounding whitespace no longer passes the pin check and then fails against the same model id.

## [0.2.1] - 2026-09-30

Library source since [0.2.0] is the version string only. CLI commands, exit codes, and eval rules are unchanged.
Expand Down Expand Up @@ -57,6 +75,7 @@ Library source since [0.2.0] is the version string only. CLI commands, exit code
- A baseline-confidence-only expect needs a floor or tolerance after defaults. An opted-in floating alias accepts a concrete resolved response model. Replay and adapter paths require nonempty normalized score probabilities and a legend.
- The sdist ships tests and fixtures.

[0.2.2]: https://github.com/sathariels/jevcheck/compare/v0.2.1...v0.2.2
[0.2.1]: https://github.com/sathariels/jevcheck/compare/v0.2.0...v0.2.1
[0.2.0]: https://github.com/sathariels/jevcheck/compare/v0.1.0...v0.2.0
[0.1.0]: https://github.com/sathariels/jevcheck/releases/tag/v0.1.0
12 changes: 6 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,10 +122,10 @@ A `Gate` helper exists for auto / ask-human / reject thresholds. It is optional

Reuse jevcheck from other repos as a [composite action](https://github.com/sathariels/jevcheck/blob/main/.github/actions/jevcheck/action.yml). The action sets up Python, installs `jevcheck` from PyPI, and runs `eval` (default), `compare`, or `record`. A nonzero CLI exit fails the job (exit 1 is a breaking contract).

Composite actions live in a subdirectory, so the `uses:` ref must point at a commit or tag that contains `.github/actions/jevcheck`. Pin `@v0.2.1`, the first tag that includes this action:
Composite actions live in a subdirectory, so the `uses:` ref must point at a commit or tag that contains `.github/actions/jevcheck`. Pin `@v0.2.2`. `@v0.2.1` is the first tag that includes this action:

```yaml
- uses: sathariels/jevcheck/.github/actions/jevcheck@v0.2.1
- uses: sathariels/jevcheck/.github/actions/jevcheck@v0.2.2
with:
contract: contracts/support.json
command: compare
Expand All @@ -134,7 +134,7 @@ Composite actions live in a subdirectory, so the `uses:` ref must point at a com
answers: fixtures/candidate-replay.json
```

`v0.2.0` does not contain `.github/actions/jevcheck`. Pin `@v0.2.1` (or a later tag) for the Action.
`v0.2.0` does not contain `.github/actions/jevcheck`. Pin `@v0.2.2` (or a later tag). `@v0.2.1` is the first tag that includes the Action.

This repo’s [example workflow](https://github.com/sathariels/jevcheck/blob/main/.github/workflows/jevcheck-example.yml) is the green CI proof: `eval` (and `compare`) on `fixtures/support-triage.json` + `fixtures/replay-unchanged.json`. No `TYPESAFE_API_KEY`. A breaking replay (`fixtures/replay-breaking.json`) exits 1; do not mark a job that uses it as a required check.

Expand All @@ -153,7 +153,7 @@ This repo’s [example workflow](https://github.com/sathariels/jevcheck/blob/mai
| `out` | — | `record --out` (required when `command` is `record`) |
| `allow-unpinned` | `false` | `--allow-unpinned` |
| `python-version` | `3.12` | `actions/setup-python` |
| `jevcheck-version` | `0.2.1` | `pip install jevcheck==…` |
| `jevcheck-version` | `0.2.2` | `pip install jevcheck==…` |
| `working-directory` | — | `cd` before the CLI; paths are relative to it |

`candidate-model` and `to` are aliases for the same candidate id. If both are set they must match. `from` and `from-model` stay mutually exclusive, same as the CLI.
Expand All @@ -163,7 +163,7 @@ This repo’s [example workflow](https://github.com/sathariels/jevcheck/blob/mai
Omit `answers` (and use `from-model` instead of `from`) only when the job has a TypeSafe key:

```yaml
- uses: sathariels/jevcheck/.github/actions/jevcheck@v0.2.1
- uses: sathariels/jevcheck/.github/actions/jevcheck@v0.2.2
env:
TYPESAFE_API_KEY: ${{ secrets.TYPESAFE_API_KEY }}
with:
Expand Down Expand Up @@ -197,7 +197,7 @@ python -m jevcheck compare fixtures/support-triage.json \

v0.2 two-model lock: [`docs/adr-009-two-model-compare.md`](https://github.com/sathariels/jevcheck/blob/main/docs/adr-009-two-model-compare.md).

See [`docs/release-readiness-audit-v0.1.md`](https://github.com/sathariels/jevcheck/blob/main/docs/release-readiness-audit-v0.1.md). Recheck: [`docs/release-readiness-audit-v0.1-recheck.md`](https://github.com/sathariels/jevcheck/blob/main/docs/release-readiness-audit-v0.1-recheck.md).
The v0.1 audit ([`docs/release-readiness-audit-v0.1.md`](https://github.com/sathariels/jevcheck/blob/main/docs/release-readiness-audit-v0.1.md)) and recheck ([`docs/release-readiness-audit-v0.1-recheck.md`](https://github.com/sathariels/jevcheck/blob/main/docs/release-readiness-audit-v0.1-recheck.md)) are historical. Each P1/P2 finding in the recheck is resolved in be58041 (v0.2.0).

Release history: [`CHANGELOG.md`](https://github.com/sathariels/jevcheck/blob/main/CHANGELOG.md). Tag and publish steps: [`RELEASING.md`](https://github.com/sathariels/jevcheck/blob/main/RELEASING.md).

Expand Down
41 changes: 22 additions & 19 deletions RELEASING.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,33 +4,35 @@ Tag, GitHub Release, and PyPI upload happen after the release pull request is me

There is no publish workflow. `.github/workflows/` contains `ci.yml`, `jevcheck-example.yml`, `pr-triage.yml`, and `plugin-scan.yml` only. PyPI upload is a manual `twine upload` from the tag.

`v0.2.1` is the first tag that contains `.github/actions/jevcheck`. Documented `uses:` refs pin `@v0.2.1`. The composite action installs `jevcheck==0.2.1` from PyPI, so that install succeeds only after the upload below.
`v0.2.1` is on PyPI and is the first tag that contains `.github/actions/jevcheck`. Documented `uses:` refs for this release pin `@v0.2.2`. The composite action's default install is `jevcheck==0.2.2`, so that install succeeds only after the upload below.

## Post-merge steps for 0.2.1
`.github/workflows/jevcheck-example.yml` passes `jevcheck-version: "0.2.1"` so the in-repo proof installs the published package and stays green before the upload. `ci.yml` installs the checkout and does not use PyPI.

## Post-merge steps for 0.2.2

1. Merge the release pull request into `main`.
2. Tag that merge commit and push the tag:

```bash
git switch main
git pull origin main
# HEAD must be the merge commit of the 0.2.1 release PR.
git tag -a v0.2.1 -m "v0.2.1 — GitHub Action, tutorial, and packaging metadata"
git push origin v0.2.1
# HEAD must be the merge commit of the 0.2.2 release PR.
git tag -a v0.2.2 -m "v0.2.2 — dependency pins, eval tests, and audit status"
git push origin v0.2.2
```

3. Create the GitHub Release `v0.2.1` from that tag. Title:
3. Create the GitHub Release `v0.2.2` from that tag. Title:

```text
v0.2.1 — GitHub Action, tutorial, and packaging metadata
v0.2.2 — dependency pins, eval tests, and audit status
```

Notes: use the [v0.2.1 release notes](#github-release-notes-for-v021) below. Paste them as the release body. They use `pip install jevcheck==0.2.1` with no "after PyPI publish" heading.
Notes: use the [v0.2.2 release notes](#github-release-notes-for-v022) below. Paste them as the release body. They use `pip install jevcheck==0.2.2` with no "after PyPI publish" heading.

4. From a clean checkout of the tag, build and upload:

```bash
git switch --detach v0.2.1
git switch --detach v0.2.2
python3 -m venv .venv-release
source .venv-release/bin/activate
python -m pip install --upgrade build twine
Expand All @@ -41,33 +43,34 @@ There is no publish workflow. `.github/workflows/` contains `ci.yml`, `jevcheck-

`twine check` should pass with no deprecation warnings. `twine upload` uses the owner's PyPI credentials for the `jevcheck` project.

5. After the upload, `.github/workflows/jevcheck-example.yml` on `main` can install `jevcheck==0.2.1`. Until then, that workflow's PyPI install fails. `ci.yml` does not install from PyPI.
5. After the upload, remove `jevcheck-version: "0.2.1"` from `.github/workflows/jevcheck-example.yml` so that workflow uses the Action default `0.2.2`.

If the tag date is not 2026-09-30, update the `[0.2.1]` heading in `CHANGELOG.md` on a follow-up commit. The compare link at the bottom of the changelog resolves once `v0.2.1` exists.
If the tag date is not 2026-10-13, update the `[0.2.2]` heading in `CHANGELOG.md` on a follow-up commit. The compare link at the bottom of the changelog resolves once `v0.2.2` exists.

## GitHub release notes for v0.2.1
## GitHub release notes for v0.2.2

Paste the following as the release body:

~~~~markdown
## Highlights

- Composite GitHub Action `.github/actions/jevcheck`. Pin `sathariels/jevcheck/.github/actions/jevcheck@v0.2.1`. `v0.2.0` does not contain the Action.
- First-run tutorial (`docs/tutorial.md`) and an offline `examples/` pack. Replay eval and compare need no API key.
- Optional jevtriage pull-request workflow, skipped when `TYPESAFE_API_KEY` is unset.
- SPDX `license = "MIT"`, Python 3.13 and 3.14 in classifiers and CI, changelog, and Homepage / Source / Issues / Changelog URLs.
- CLI commands, exit codes, and eval rules match 0.2.0.
- `typesafe-sdk` lock is 0.7.4. The install floor stays `typesafe-sdk>=0.7.0`.
- GitHub Actions pins: `actions/checkout` 7.0.1, `actions/setup-python` 7.0.0, and `hashgraph-online/ai-plugin-scanner-action` 1.2.777, still by commit SHA.
- Eval coverage for tolerance edges, multiple expectations, and error paths. CI prints a coverage report.
- `evaluate()` accepts a concrete candidate pin that has surrounding whitespace.
- The v0.1 recheck P1/P2 findings are marked resolved in be58041 (v0.2.0).
- Pin `sathariels/jevcheck/.github/actions/jevcheck@v0.2.2`. `@v0.2.1` is the first tag that contains the Action.

## Install

```
pip install jevcheck==0.2.1
pip install jevcheck==0.2.2
```

## Action

```yaml
- uses: sathariels/jevcheck/.github/actions/jevcheck@v0.2.1
- uses: sathariels/jevcheck/.github/actions/jevcheck@v0.2.2
with:
contract: contracts/support.json
command: eval
Expand Down
Loading
Loading