A small, inspectable evaluation system for short-form product messages.
Turn vague prompt tweaks into a repeatable loop: define cases, generate candidates, grade constraints, inspect failures, and protect improvements with a holdout set.
Product messages are deceptively hard. A push notification can be fluent and still be wrong. It can invent urgency, omit a required date, exceed a character limit, or introduce a number that never appeared in the brief.
This repository treats those failures as testable product requirements. It includes:
- synthetic cases for common consumer product messages
- separate development and holdout sets
- three prompt strategies, from a naive prompt to a four-step chain
- deterministic graders for facts and hard constraints
- illustrative outputs that make the evaluation workflow runnable without an API key
- an optional Anthropic provider for live experiments
- scorecards, tests, and a CI workflow
The bundled outputs are hand-authored fixtures. They test the harness and demonstrate expected failure patterns. They are not a benchmark of any model and they do not claim business impact.
Prompt quality is often judged by reading a few outputs and deciding that one version feels better. That approach is difficult to repeat, easy to bias, and weak at catching regressions.
This project replaces that process with a small evaluation loop:
The core principle is simple: write down what success means before changing the prompt.
You only need Python 3.9 or newer for the offline workflow.
git clone https://github.com/abhishekl-offl/eval-driven-messaging.git
cd eval-driven-messaging
python3 -m unittest discover -s tests -vGenerate all three scorecards from the included holdout fixtures:
python3 -m eval_driven_messaging evaluate \
--cases cases/holdout.jsonl \
--outputs examples/outputs/v0.jsonl \
--label "V0: naive single prompt" \
--json-report results/v0.json \
--markdown-report results/v0-scorecard.md
python3 -m eval_driven_messaging evaluate \
--cases cases/holdout.jsonl \
--outputs examples/outputs/v1.jsonl \
--label "V1: constraint-aware prompt" \
--json-report results/v1.json \
--markdown-report results/v1-scorecard.md
python3 -m eval_driven_messaging evaluate \
--cases cases/holdout.jsonl \
--outputs examples/outputs/v2.jsonl \
--label "V2: normalize, draft, critique, revise" \
--json-report results/v2.json \
--markdown-report results/v2-scorecard.md
python3 -m eval_driven_messaging compare \
results/v0.json results/v1.json results/v2.json \
--markdown-report results/comparison.mdThen open the comparison and drill into any scorecard.
The included holdout run makes the prompt progression visible without requiring an API key. Each strategy was graded against the same 8 synthetic holdout cases and the same deterministic checks.
| Strategy | Prompt design | Hard-pass cases | Hard-pass rate | Average check score |
|---|---|---|---|---|
| V0 | vague single prompt | 0 of 8 | 0.00% | 76.78% |
| V1 | constraint-aware single prompt | 7 of 8 | 87.50% | 98.21% |
| V2 | normalize, draft, critique, revise | 8 of 8 | 100.00% | 100.00% |
These results come from hand-authored fixtures that demonstrate the harness. They are not live model benchmarks. A real comparison must generate fresh outputs, record model settings, and repeat runs before making a performance claim.
| Version | Change | Failure pattern in the included fixtures | Lesson |
|---|---|---|---|
| V0 | asks only for a catchy notification | 7 forbidden-phrase failures, 4 missing-phrase failures, 2 unsupported-number failures | fluent copy is not the same as controlled copy |
| V1 | states the factual and formatting rules in one prompt | 1 required-phrase failure | explicit constraints remove most avoidable errors |
| V2 | separates interpretation, writing, critique, and revision | no deterministic failures | chaining can help when one call has too many distinct jobs |
The hard-pass rate is intentionally strict. A message fails the case if even one required check fails. The average check score is diagnostic, but it should not be used to excuse a critical failure.
The detailed artifacts remain available for inspection:
Each JSONL row describes one messaging task and its hard constraints:
{
"id": "holdout-gate-change",
"category": "travel",
"brief": {
"event": "Flight CL308 now departs from Gate B12",
"action": "Open the boarding pass"
},
"constraints": {
"max_title_chars": 38,
"max_body_chars": 120,
"required_phrases": ["CL308", "Gate B12"],
"forbidden_phrases": ["final call", "hurry"],
"allowed_numbers": ["308", "12"],
"require_action": true
}
}This format keeps the task readable for a product manager and executable for the evaluator.
The repository includes three strategies:
| Strategy | Design | Purpose |
|---|---|---|
v0 |
one vague instruction | establishes a weak baseline |
v1 |
one constraint-aware instruction | tests whether clearer requirements are enough |
v2 |
normalize, draft, critique, revise | separates factual extraction from writing and verification |
The chain is intentionally explicit. It is useful when one prompt must interpret a brief, write copy, and audit itself. Each step has one job and can be inspected independently.
The deterministic graders check:
- title and body length
- required phrases
- forbidden phrases
- unsupported numbers
- unresolved template placeholders
- clear action language when required
These checks are cheap, fast, and stable. They are good regression gates because the same input always receives the same grade.
They do not judge tone, relevance, clarity, or persuasion. Those qualities need a carefully designed rubric, calibrated human review, or a model grader. The quality rubric shows how that second layer can be added without pretending that subjective quality is deterministic.
Use development cases while changing prompts. Use holdout cases only when you want a less biased comparison.
If every prompt decision is made after looking at every case, it becomes easy to tune the wording to the examples instead of improving the general system. The split is small here because this is a teaching repository, but the discipline is the same in a production evaluation pipeline.
- Load cases. The runner reads either the development set or the holdout set. Each case contains the factual brief and the rules that can be checked objectively.
- Choose a strategy. V0 and V1 load one prompt template. V2 loads four templates for normalization, drafting, critique, and revision.
- Generate candidates. The provider boundary sends the rendered prompt to a text model. The optional Anthropic provider is the only component that knows about a model SDK.
- Save structured outputs. Every candidate becomes one JSONL row with a case ID, title, and body. Saving this boundary makes a run inspectable and repeatable.
- Run hard graders. Pure Python functions check lengths, required and forbidden phrases, numbers, placeholders, and action language.
- Build reports. The same grade objects produce JSON for automation and Markdown for human review.
- Gate changes. Tests and CI can compare saved outputs without a model key, while a reviewer can separately assess tone and usefulness.
The system has four boundaries:
- Cases define the product requirement.
- Prompts define the candidate behavior.
- Providers isolate model-specific API calls.
- Graders and reports stay offline and deterministic.
This separation lets you change a prompt or model without rewriting the evaluator. It also lets CI validate saved outputs without exposing an API key or paying for live inference.
| Layer | Best for | Implementation in this repository |
|---|---|---|
| deterministic checks | facts with one defensible answer | executable Python graders and regression tests |
| quality judgment | clarity, tone, relevance, and persuasion | a documented rubric for calibrated human or model review |
Combining both layers into one unexplained number would hide uncertainty. The architecture keeps hard failures visible and treats subjective quality as a separate judgment that must be calibrated.
The evaluator does not depend on a specific model. A small provider interface accepts a prompt and returns text. This makes three workflows possible:
- run the full offline demo with saved fixtures and no credentials
- test prompt orchestration with a fake provider
- run a live Anthropic experiment without changing the graders
The boundary also makes a future second provider straightforward, but the repository does not add one speculatively.
Read SYSTEM-DESIGN.md for the detailed decisions and tradeoffs.
Live generation is optional. The offline tests and reports do not require a model account.
Install the optional dependency:
python3 -m pip install -e ".[live]"Set your API key using the method recommended by the Anthropic SDK, then select a model explicitly:
export ANTHROPIC_MODEL="YOUR_MODEL_ID"Generate one strategy against the holdout set:
python3 -m eval_driven_messaging generate \
--cases cases/holdout.jsonl \
--strategy v2 \
--output runs/v2-live.jsonlGrade that run with the same evaluate command used for the fixtures. Do not commit private briefs, API keys, user data, or production outputs. See PRIVACY.md.
- Give the case a unique ID.
- Use synthetic or approved data.
- State the brief as facts, not desired copy.
- Encode only objectively testable requirements as hard constraints.
- Add it to the development set first.
- Run the tests and all saved strategies.
Only use code when the rule has one defensible answer. Examples include a maximum length, an exact required phrase, valid JSON, or the absence of unsupported numbers.
Add a unit test that fails before adding the grader. Keep the failure detail specific enough to guide a prompt revision.
Start with rubrics/quality-judge.md. Calibrate the rubric against examples scored by people. Measure agreement. Keep the model grader separate from the hard checks so readers can see which scores are objective and which depend on judgment.
| Choice | Reason | Tradeoff |
|---|---|---|
| JSONL for cases and outputs | easy to diff, stream, inspect, and generate | less compact than a database |
| Python standard library for the core | fast setup and few supply-chain risks | fewer convenience libraries |
| saved fixture outputs | makes the full loop runnable without cost or credentials | fixtures are not live benchmarks |
| deterministic checks first | stable feedback for clear constraints | cannot judge nuanced writing quality |
| development and holdout split | reduces prompt tuning against the final comparison | small sample sizes remain noisy |
| explicit prompt chain | makes intermediate reasoning artifacts inspectable | four model calls cost more and take longer |
| provider boundary | keeps model access separate from evaluation | one extra interface to understand |
This repository is deliberately small. A production system would likely add:
- versioned datasets with review ownership
- privacy classification and retention rules
- model and prompt metadata for every run
- retries, rate-limit handling, and cost tracking
- calibrated quality graders for tone and relevance
- human review for high-risk categories
- statistical confidence intervals on larger test sets
- experiment links that connect offline metrics to product outcomes
Those features are useful only after the basic loop is trusted. They are not included here because the repository is meant to explain the core system clearly.
cases/ synthetic development and holdout tasks
prompts/ three prompt strategies
eval_driven_messaging/ generation, graders, reporting, and CLI
examples/outputs/ hand-authored fixtures for the offline demo
results/ generated scorecards
rubrics/ rubric for a future quality grader
tests/ unit and integration tests
assets/ repository visuals
This project adapts evaluation and prompt-chaining ideas from:
The implementation, synthetic cases, prompt progression, and explanation are original to this repository.