A reproducible, local-first CLI evaluation harness for comparing GPT prompt styles on identical tasks. PromptBench measures how different prompt-design approaches affect correctness, constraint compliance, evidence grounding, token usage, latency, and cost.
PromptBench answers questions like:
- Does a declarative contract outperform a procedural prompt?
- Does prompt simplification improve correctness or merely reduce cost?
- Do verbose prompts cause instruction conflicts?
- How stable is each prompt style across repeated trials?
- Does the best prompt style change across GPT models?
- What is the quality, latency, token, and cost trade-off?
promptbench/
├── configs/ # Model registry, pricing, experiment definitions
├── datasets/ # JSONL evaluation cases + fixtures
├── prompts/ # Shared components + 6 prompt style templates
├── src/
│ ├── cli.ts # Commander CLI entry point
│ ├── config/ # YAML config loader with Zod validation
│ ├── dataset/ # JSONL dataset loader
│ ├── prompts/ # Template renderer + semantic parity validator
│ ├── providers/ # OpenAI Responses API + fake provider
│ ├── runner/ # Experiment engine, retry, concurrency
│ ├── graders/ # Deterministic graders + blinded model judge
│ ├── statistics/ # Aggregate metrics, paired comparisons, bootstrap CI
│ ├── storage/ # SQLite persistence via sql.js (WASM)
│ ├── reporting/ # Markdown + standalone HTML report generation
│ ├── pricing/ # Cost estimation
│ └── types/ # TypeScript interfaces + Zod schemas
├── tests/ # Unit tests (vitest)
└── runs/ # Experiment output (one directory per run)
- Node.js 24+
- An OpenAI API key with access to GPT-family models
git clone <repo-url>
cd promptbench
npm install
cp .env.example .env
# Edit .env with your OPENAI_API_KEYEdit configs/models.yaml to match the models available in your OpenAI account:
models:
- id: gpt-5.6-sol
label: "GPT-5.6 Sol"
reasoning:
effort: medium
enabled: trueUpdate configs/pricing.yaml with current pricing from openai.com/pricing.
npm run cli validate --config configs/experiments/default.yamlnpm run cli run --config configs/experiments/default.yaml --dry-runnpm run cli run --config configs/experiments/default.yamlThis runs 2 models × 6 prompt styles × 20 cases × 5 trials = 1,200 API requests.
npm run cli resume <experiment-id>npm run cli grade <experiment-id>
npm run cli grade <experiment-id> --no-judge # skip model judge (cheaper)npm run cli analyze <experiment-id>npm run cli report <experiment-id>Reports are written to runs/<experiment-id>/report.md and report.html.
JSONL files where each line is a JSON object:
{
"caseId": "refund-standard-deny",
"category": "policy-reasoning",
"input": {
"request": "The customer requests a refund after 35 days.",
"evidence": ["Standard refunds are permitted within 30 days.", ...]
},
"expected": {
"decision": "deny",
"requiredFacts": ["The request occurred after the standard refund period."],
"forbiddenClaims": ["Refunds are never permitted after 30 days."]
},
"tags": ["grounding", "policy"]
}Six prompt styles are implemented as first-class variants. All express the same task, evidence policy, output schema, and constraints.
| Style | Description | Purpose |
|---|---|---|
minimal |
Goal + evidence + output schema only | Smallest viable baseline |
procedural |
Explicit step-by-step workflow | Test imperative prompting |
declarative-contract |
Role, goal, success criteria, constraints, stopping condition | Test contract-oriented prompting |
verbose-scaffolded |
Detailed role, repeated reminders, extensive guidance | Measure instruction dilution |
policy-based |
Invariants, decision policies, preferences, fallback rules | Test policy classification |
example-driven |
Compact instruction + 2-3 representative examples | Test few-shot guidance |
- JSON parse validity
- Schema validity
- Decision match
- Required fact coverage
- Forbidden claim detection
- Output completeness
- Refusal/abstention detection
- Explanation length
All scores normalized to [0, 1].
Blinded rubric-based judge using a separate model. The judge sees the case, evidence, criteria, and candidate response — but NOT the prompt style name, model identity, trial number, latency, or tokens.
npm test # unit tests (no API key needed)
npm run test:live # live smoke test (requires OPENAI_API_KEY)promptbench init Create .env from template
promptbench validate --config <path> Validate config, dataset, prompts
promptbench run --config <path> Run experiment
promptbench resume <id> Resume interrupted experiment
promptbench grade <id> Grade responses
promptbench analyze <id> Compute aggregate metrics
promptbench report <id> Generate markdown + HTML reports
promptbench export <id> Export results as JSONL
promptbench compare <id-a> <id-b> Compare two experiments
--models <a,b> Filter models
--styles <a,b> Filter styles
--cases <a,b> Filter case IDs
--trials <n> Override trial count
--concurrency <n> Override concurrency
--seed <n> Override seed
--dry-run Validate + estimate cost without API calls
--force Re-run completed conditions
--no-judge Skip model judge
--judge-model <id> Override judge model
--max-cost <n> Stop when estimated cost exceeded
Every run produces a manifest (runs/<id>/manifest.json) containing:
- Experiment configuration hash
- Dataset hash
- Prompt hashes per style
- Model IDs
- SDK version
- Node.js version
- Timestamps
- Pricing version
- Analysis seed
A past run remains interpretable even if project configuration changes.
- Multi-trial evaluation: 5 trials per condition to measure variance
- Paired comparisons: Same dataset cases across conditions, bootstrap confidence intervals
- Blinded judge: Model judge sees anonymized responses
- Separate quality and cost: Never mix quality and efficiency into one score
- Controlled variables: Identical task, evidence, schema, and model settings across styles
- Conclusions apply only to the tested tasks, models, and settings
- Cost estimates depend on pricing configuration accuracy
- Model judge introduces its own bias; deterministic graders are the ground truth
- Current dataset is synthetic (policy reasoning domain)
- No distributed execution support in MVP
- Create
prompts/styles/<name>.mdusing the template syntax - Add the style name to
configs/experiments/default.yaml - Run
promptbench validateto verify parity
Current: v0.1.0 — Core pipeline verified. 26 tests pass, live smoke test (20 API calls, 8 graders + blinded judge) green.
| Task | Status |
|---|---|
| Atomic writes for run manifests | pending |
promptbench compare command |
pending |
Fix README reference to src/pricing/ directory |
pending |
| Structured JSONL error logging | pending |
| Handling corrupted output artifacts | pending |
| Task | Status |
|---|---|
| Pairwise judge (anonymized A/B, position-bias check) | pending |
Human review export (promptbench export-review) |
pending |
Human review import (promptbench import-review) |
pending |
| Position-bias validation for judge | pending |
| Task | Status |
|---|---|
| Integration tests with fake provider | pending |
| Mermaid diagrams (execution flow, eval hierarchy) | pending |
| Full test coverage for all modules | pending |
| Provenance documentation | pending |
Full benchmark execution (1,200 requests, ~$40, 2–3 GPT-family models, 6 styles × 20 cases × 5+ trials). This phase is documented in detail at ROADMAP.md and will be executed once hardening and the grading matrix are complete. Key deliverables:
- Full benchmark run against 2–3 models at 10 trials
- Pairwise prompt-style comparison with bootstrap CIs
- Pareto frontier analysis (quality vs. cost vs. latency)
- 3-model comparison (does best style vary across models?)
- Published reference report
MIT