ratchetlock

English | 한국어

A CLI that manages the LLM prompts you run over and over like code under regression control. Evaluation is left to promptfoo; ratchetlock adds one layer promptfoo doesn't have — a "ratchet."

To hand this tool to an AI agent, give it AGENTS.md as context — the command contract, the situation-to-action decision table, and the forbidden actions are all written up as an executable contract.

Why this exists

You've probably lived this. A prompt kept wrapping its output in markdown code fences, breaking your parser, so you added a constraint — "don't use code fences" — and moved on. Two months later you tweaked the prompt again while fixing something unrelated, and the code-fence problem quietly came back. You find out when the parser breaks again.

This rarely happens with code. When you fix a bug you leave a test behind, and if someone later resurrects that bug, CI turns red. Prompts don't have this safety net. Evaluation tools like promptfoo check "does this prompt pass its tests right now" — but they don't persist state, so they can't answer "did something that passed yesterday break today."

The one thing ratchetlock does is attach that memory.

• Save the currently-passing results, output snapshots included, as a baseline — freeze
• Register any failure you've caught once as a permanent regression test. No matter how much you rewrite the prompt afterward, this list only grows, never shrinks — add-fail
• Reject with exit 1 the moment anything drops below the baseline. Wire it straight into CI — check

The name comes from the ratchet — a gear mechanism that only turns one way. Every gain in prompt quality gets a pawl dropped behind it, so it can't slide back.

There's one more thing. It's not just prompt regressions this catches — it also catches the grader itself quietly going soft. Run a gate like this long enough, and sooner or later someone loosens the scoring criteria without meaning to. It's the measuring instrument's needle drifting, and nobody notices, because from that point on a green light guarantees nothing. ratchetlock re-scores frozen outputs with today's probe every time, so the instant a probe change flips the verdict on something that used to pass, it goes red. promptfoo doesn't have this, and in real use it's the single feature that has earned back its keep the most (the mechanics are in ARCHITECTURE.md).

What kind of prompts this is for

The target is a prompt that's saved to a file and run repeatedly with only the input swapped out — a daily summarization pipeline, a rewrite prompt baked into a service, that kind of thing. Three things need to hold for a ratchet to make sense.

• You run the same prompt repeatedly. A prompt you write once and throw away has no concept of "an old failure coming back."
• You have a growing set of representative inputs to test against.
• The output can be scored pass/fail — does the JSON parse, are forbidden phrases absent, that kind of check.

One-off prompts typed into a chat window aren't the target. Neither are prompts where a human reviews every single result — that eyeball is already the gate. Where a ratchet earns its keep is the automated path where results ship without a human looking.

Installation

For local development, clone + link is recommended.

git clone https://github.com/calintzy/ratchetlock.git
cd ratchetlock
npm install          # the prepare hook builds it too
npm link             # registers the ratchetlock command on PATH

After that, run ratchetlock <command> from anywhere. To remove it, npm unlink -g ratchetlock.

In CI (GitHub Actions ubuntu), installing the git dependency directly works fine in practice — the prepare hook finishes the build. Pinning the install to a tag is recommended.

npm install --no-save github:calintzy/ratchetlock#v0.4.0   # builds cleanly on ubuntu-latest (verified)

Without #v0.4.0, github:calintzy/ratchetlock alone installs main's HEAD fresh every time. The moment a tool update lands on this repo, every consuming repo's CI that didn't pin a tag quietly picks up the new version — with nothing on their side having changed. Deterministic check guaranteeing a green CI is the whole point of this tool, and an unpinned install path undermines exactly that guarantee. Pinning is the cheapest way to close that gap.

On some local macOS npm versions, this global git install skips devDependencies (typescript), which can break the prepare-hook build. On that setup, use the clone + npm link route above. Once this ships to the npm registry (v0.2.0 target), npm install -g ratchetlock will be the one line and this branch goes away.

The only runtime dependency is promptfoo. Everything else runs on Node's standard library.

After upgrading to v0.4.0, run ratchetlock freeze once. Starting in v0.4.0, ratchetlock also hashes the local modules a probe pulls in via require (see "Tracking a probe's dependency modules" below), and that baseline is recorded at freeze time. Snapshots frozen under v0.3.0 or earlier don't have this baseline, so until you freeze again, dep-drift judgment is silently skipped — not a false negative, just a check that hasn't started watching yet.

Full instructions for registering a new prompt as a contract are in docs/REGISTER.md; probe-writing pitfalls and reference implementations are in docs/PROBES.md.

Usage flow

Every command runs from the directory that holds promptfooconfig.yaml. State accumulates in one ratchet.json file next to it, so you can version it with your prompts in git.

cd my-prompts/           # where promptfooconfig.yaml lives
ratchetlock init         # once — creates the state file
ratchetlock freeze       # freeze the currently-passing results as the baseline

From there, every time you edit the prompt you run this loop.

vi prompt.txt                          # edit the prompt
ratchetlock check --live               # call the real model to see the effect
ratchetlock add-fail --from-last <case>  # register any newly-caught failure as a permanent guard
ratchetlock freeze                     # freeze the improved state as the new baseline

When you just want to run the registered probes against one fresh output — not a whole case replay — use lint.

ratchetlock lint --output today.json   # run registered probes on one new output, exit 1 on violation

If lint catches a new defect, add that input to tests.yaml and promote it with add-fail — this is how a hard case you hit in the wild gets folded into the contract.

lint is only valid when the input unit matches what the probes expect. lint reuses the probes registered in the contract as-is, so if a probe is reference-comparison style (it reads grading metadata like creation date or star count from vars to catch distortions), you must pass that same vars via --vars. Call it without vars and those probes are silently skipped — a violation will pass with exit 0, which means the violation was missed, not absent. Conversely, scanning an already-rendered document as a whole for surface patterns is out of scope for lint. That job belongs to an adapter on the application side that splits the document into item-level (output, vars) pairs and feeds them to lint (the pattern is in docs/PROBES.md).

In CI, ratchetlock check is the one line you need. If the baseline breaks, it exits 1 and fails the build.

Commands

The binary is ratchetlock, and there are six commands.

Command	What it does	Calls an LLM
init	Reads promptfooconfig, extracts prompts/probes/tests, and creates ratchet.json.	No
check	Checks whether the baseline (frozen snapshots + registered failure cases) still passes. Exits 1 on regression.	No (only with --live)
freeze	Freezes currently-passing cases, output snapshots included, raising the baseline.	Yes
add-fail	Registers a failure caught in the last evaluation as a permanent regression guard. Stays enforced across prompt edits.	No
lint --output <file> [--vars <JSON>] [--prompt <label>]	Applies registered probes to one new output to check for a new defect. Exits 1 on violation.	No
status	Shows the active prompt, freeze count, baseline size, and whether prompt/probe/dependency-module files have drifted.	No

Why check doesn't call the LLM

Evaluating a prompt is really two steps. Calling the model to get an output is expensive, and even with the same input the result differs a little every time. Scoring that output with a probe is just code execution — fast, and it gives the same result a hundred times over.

Because freeze saves the output of that first step (the raw model output), plain check skips step one and re-scores the saved output. It's not re-administering the exam to the student — it's re-grading a kept answer sheet against today's grading criteria. That's why it runs without an API key and gives the same verdict on every CI run.

What this catches: a probe changing and flipping the verdict on something that used to pass, a frozen snapshot being tampered with or corrupted, and violations in the registered failure cases. What it doesn't catch: the effect of an actual prompt edit — the saved output was produced by the old prompt. That's what check --live or freeze are for; they call the model fresh. The reasoning behind splitting these two paths is in ARCHITECTURE.md.

Tracking a probe's dependency modules (v0.4.0)

Back when only the probe file's own hash was tracked, there was a gap. Following PROBES.md's shared-module pattern, asserts.js typically delegates the actual grading logic to another file via require("./rules.cjs"), either inside the contract directory or outside it. But ratchet.json only ever hashed asserts.js itself — if rules.cjs changed and the grading criteria shifted entirely, the probe file itself hadn't changed a single byte, so deterministic check had no way to see it.

Starting in v0.4.0, freeze recursively follows the local modules a probe file requires by relative path and records their hashes alongside it. check compares that combined hash against the current state, and if anything changed it prints a separate [dep drift] warning (distinct from [probe drift], the probe's own hash mismatch — because what changed here is a module the probe depends on, not the probe itself). By default this only warns and keeps exit 0, but with --probe-locked a dep drift is promoted to a hard fail (exit 1), same as a probe drift.

Tracking is a static scan, so it has a defined safe range. It follows only relative-path string literals in require("./foo")-style calls, and flags paths that resolve outside the contract directory with a warning. Bare specifiers (require("lodash"), a monorepo package name) and dynamic require calls assembled from variables are never tracked to begin with — they're only tallied as [dep 미추적] (dep untracked) and excluded from judgment. If a frozen snapshot lacks this field (any snapshot made under v0.3.0 or earlier), dep judgment is skipped entirely — behavior stays 100% identical to before this feature existed.

Real case study: the cardnews prompt, v1 → v2

examples/cardnews isn't a made-up demo — it's what actually happened while fixing a prompt that was in real use. It rewrites tech news into plain-language Instagram cardnews copy, and the outputs in fixtures/ are actual claude (sonnet) outputs from both v1 and v2.

v1 (prompt.txt) passed only 0 of 5 test cases. Despite explicitly instructing "no code fences," 4 cases came back wrapped in them, and the 5th failed to parse as JSON at all. Rewriting the prompt as v2 (prompt_v2.txt) brought it up to 4/5.

The last case, OmniRoute, shows what this tool is actually worth. v2 fixed the code-fence problem, but a hedge phrase from the source text — the qualifier that marked a claim as the author's, not a verified fact — got dropped during the rewrite. A framing check already present in asserts.js caught it. This isn't an unsolved problem deferred to the roadmap; it's an existing probe catching a real defect that the v2 rewrite actually introduced. So this one case was deliberately left unfrozen, and the honest headline is "0/5 → 4/5, and we know about the fifth."

demo.sh reproduces this whole flow.

bash examples/cardnews/demo.sh

It runs to completion with no claude subscription or API key. The default is fixture-replay mode — it feeds back the real claude outputs saved in fixtures/ and re-scores them with the current probes, so no LLM gets called. It starts with init, confirms v1 really is 0/5, registers those failures with add-fail, switches to v2 and freezes the 4 passing cases, and confirms deterministic check passes everything with exit 0. Along the way it deliberately corrupts one frozen output to show check catching it. The full transcript is in TRANSCRIPT.md.

The framing check is not "semantic-distortion detection"

The framing check that caught OmniRoute above is a deterministic keyword check — a regex looking for the presence of an author-claim hedge. It's not an LLM rubric judging "did this sentence distort the meaning of the source." Semantic-distortion detection, which is on the roadmap, is a problem a level above this and is out of scope right now.

CI integration

Because check reports regressions through its exit code, it wires straight into CI.

ratchetlock check   # exit 1 on regression → build fails

--probe-locked is recommended in CI.

ratchetlock check --probe-locked   # also rejects with exit 1 on probe or probe-dependency drift

Plain check only warns and keeps exit 0 even when a probe (or a module it depends on) has changed since freeze time. To fully block the measuring instrument going quietly soft at the CI stage, you need --probe-locked.

Honest limitations

This is partly a portfolio project and partly a tool that has had its own optical illusions in real use, so here's a clear list of what this gate does not guarantee.

• A live pass is an approximation. Contract evaluation runs a single prompt in isolation, while in production that same prompt runs inside the full spec context, through different call paths, and sometimes with a different model. A check --live pass is no guarantee the same thing passes identically in production — this tool does not replace a production gate. Don't read "contract passes" as "it's safe."
• A green CI from deterministic check means "nobody touched the contract," not "the prompt still works well." check only re-scores output captured at freeze time, so it can't see the model quietly drifting or the real input distribution shifting. You only see that by running check --live again.
• freeze and check --live require a working local LLM CLI. The provider has to be actually runnable (e.g., the claude CLI needs to be on PATH), so remote or mobile environments can't freeze a new prompt.

Sample discipline — a ratchet only guards as much as its sample

The frozen cases are the entire contract. If you've only frozen 6 news items from one day, harder categories full of political content, deaths, or English proper nouns simply aren't in the contract at all, and the ratchet doesn't protect against them. The first rule is not mistaking the tool for guaranteeing anything outside its sample.

So operations should run like this. Every time you hit a hard input in production, add it to tests.yaml, and once it passes live, fold it in with freeze. New snapshots don't overwrite old ones — they accumulate as a union: the floor is the sum of everything frozen plus every registered failure case, so once a case is folded in it keeps being required to pass. The contract tightens as the sample grows.

Roadmap

• Determinizing LLM judgment — verify blind spots binary asserts can't catch (numeric/unit/ framing distortions) with an LLM grader, while reconciling that grader's non-determinism with deterministic check. The plan is to freeze the judgment itself like a fixture — save the LLM's verdict at freeze time, have check replay it, and only re-judge when the rubric hash changes. This lets an LLM grader in without giving up the gate's reproducibility.
• Absorbing promptfoo optimize — a loop that auto-generates improvement candidates and only adopts the ones that pass the check gate.
• Probe-lock enforced by default — promote a probe hash mismatch to a hard fail by default, closing off measurement drift at the source.
• Multiple targets — manage several prompt contracts in parallel from one repository.

How we got here

Built it, put it into two real automations, and fixed what broke across three rounds. The reasoning behind each fix and what we learned along the way is in docs/RETROSPECTIVE.md — a record of why the biggest defect showed up only in real usage and not the example, and why this tool's real enemy turned out to be the measuring instrument itself, not regressions.