Statistically compare LLM prompts using Wald's Sequential Probability Ratio Test (SPRT), then build reproducible prompt-engineering benchmarks instead of relying on anecdotal prompt advice.
- What it is
- Position statement
- Features
- Quick start
- Understanding the result
- Run your own experiment
- Use via Telegram (optional)
- Configuration
- CLI
- How it fits together
- Benchmark corpus
- Further reading
- FAQ
Varex asks one question: is prompt A better than prompt B, or is the difference just noise?
You supply two prompt variants and a dataset. Each trial generates an answer with each prompt, scores a winner, and updates a Sequential Probability Ratio Test (SPRT). When the log-likelihood ratio hits a boundary for your α and β, the experiment stops. That is usually sooner than a fixed-size A/B test.
You can drive it from the command line (scripts, CI) or from Telegram through an OpenClaw skill.
Tip
If you have no cloud API keys, use the objective demo on local Ollama. See Quick start.
Most prompt engineering advice is anecdotal.
One guide says use XML. Another prefers Markdown. Some people swear by role prompts; others say roles barely move the needle. Many of these recommendations are presented as best practices without statistical evidence you could reproduce.
Varex exists to test those claims on a given model, dataset, and scoring method — not to declare universal prompt truths.
It compares two prompt variants with Wald's Sequential Probability Ratio Test (SPRT) and stops when the data support a conclusion at the error rates you set. Preference and one-off screenshots are not the decision procedure.
Longer term, the project aims to grow an open corpus of reproducible prompt-engineering experiments so others can rerun them, check the numbers, and add their own.
- Stops at
accept_H1oraccept_H0, or ends inconclusive onmax_trials, budget, repeated errors, or too many ties - Objective scoring (reference answers) or LLM-as-judge (optional rubric)
- JSON
ExperimentReportwith a short verdict sentence, provenance fields, and caveats the code fills in - Resume interrupted runs; re-judge stored answers with a new judge without regenerating them
- Ollama locally, or any OpenAI-compatible API (OpenAI, Groq, Gemini, Grok, vLLM, and similar)
- Optional token budget; warnings when
max_trialslooks underpowered or larger than the dataset
| Tool | Why |
|---|---|
| Python 3.11+ | Runtime |
| uv | Install deps and run commands |
| Ollama | Local model for the demos |
# Install uv (macOS / Linux / WSL)
curl -LsSf https://astral.sh/uv/install.sh | sh
# Windows (PowerShell)
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"git clone https://github.com/coeusyk/varex.git
cd varex
uv syncOn macOS and Windows, open the Ollama app; it starts the server.
On Linux / WSL:
ollama serve &Then:
ollama pull llama3.1:latest
curl http://localhost:11434 # expect: Ollama is runninguv run python src/cli.py --config configs/demo_qa_objective.jsonIn a TTY you get a live progress panel. Piped output uses NDJSON for progress. The last thing on stdout is the experiment report JSON.
Note
configs/demo_summarization_judge.json needs a cloud judge API key (see .env.example). Stick to the objective demo if you only have Ollama.
Short example (full schema in ARCHITECTURE.md):
{
"experiment_id": "exp-abc123",
"winner": "A",
"stopped_by": "accept_H1",
"trials_used": 42,
"p_hat": 0.714,
"verdict_sentence": "Prompt A is statistically better than B (p̂=0.714, LLR=3.102, α=0.05)."
}winner |
Meaning |
|---|---|
"A" |
Prompt A won (accept_H1) |
"no_difference" |
No evidence that A beats B (accept_H0) |
"inconclusive" |
Stopped without a statistical decision (cap, budget, errors, or ties) |
"B" |
Kept for a possible two-sided test later; this release never emits it |
Write a JSON config, then:
uv run python src/cli.py --config my_experiment.jsonMinimal objective config:
{
"task_type": "qa",
"evaluation_mode": "objective",
"prompt_a": "Answer in one sentence.",
"prompt_b": "Answer briefly and directly.",
"generator": {
"protocol": "ollama",
"provider": "ollama",
"model": "llama3.1:latest"
},
"dataset": {
"path": "datasets/qa_demo.jsonl"
},
"sprt": {
"p0": 0.50,
"p1": 0.65,
"alpha": 0.05,
"beta": 0.10,
"max_trials": 200
}
}| Mode | Use when | How a trial is scored |
|---|---|---|
objective |
You have reference answers | Compare outputs to reference_answer in the dataset |
judge |
No ground truth | A separate judge model picks A or B (optional rubric) |
Important
In judge mode the generator and judge must be different endpoints. Varex rejects same-endpoint judging.
| Parameter | Meaning | Common values |
|---|---|---|
p0 |
Win rate for A under “no real edge” | 0.50 |
p1 |
Win rate for A you care about detecting | 0.60–0.70 |
alpha |
Max false positive rate | 0.05 (stricter), 0.10 (looser) |
beta |
Max false negative rate | 0.10 |
max_trials |
Hard stop if still undecided | 200 in the shipped demos |
Higher α/β usually finishes in fewer trials. Lower ones need more evidence, but the test still stops early when the signal is clear.
If OpenClaw is installed, skills/varex/SKILL.md can walk through a comparison in chat.
- Create a Telegram bot with @BotFather and copy the token.
- Point OpenClaw at it and restart:
openclaw config set channels.telegram.botToken "YOUR_TOKEN"
openclaw gateway restart- Message the bot from an account your gateway is supposed to allow.
Warning
Trust boundary: Varex does not authenticate Telegram senders. There is no allowlist inside Varex; it runs the config it is given. Who may talk to the bot is your OpenClaw gateway’s job. Check that yourself before you expose a bot that can burn API budget. See ARCHITECTURE.md — Trust Boundary.
Everything is one JSON file. Main blocks:
| Block | Role |
|---|---|
prompt_a / prompt_b |
The two system prompts under test |
generator |
Model that writes answers (protocol: ollama or openai_compat) |
judge |
Only for evaluation_mode: "judge" |
dataset |
Path to .jsonl / .csv; sample is "random" or omitted |
sprt |
p0, p1, alpha, beta, max_trials |
budget |
Optional max_tokens_total across generate and judge calls |
Optional on any provider block: temperature, top_p, seed. Leave them out to keep provider defaults. Setting them fixes the sampling knobs for that run; it does not promise identical model output every time.
Note
"dataset.sample": "stratified" is not implemented and fails validation. Use "random" or omit the field.
Field-by-field tables and the report schema are in ARCHITECTURE.md.
If provider is a known name (openai, groq, gemini, xai, ollama, vllm, …), the registry fills base_url. Otherwise set provider: "custom" and give base_url yourself. Keys come from env vars named in api_key_env, never from the config file.
uv run python src/cli.py [OPTIONS]| Flag | Purpose |
|---|---|
--config PATH |
Start a new experiment |
--resume ID |
Resume from SQLite state |
--rerun ID + --new-judge SPEC |
Re-judge stored answers (judge-mode sources only) |
--export ID |
Export trials (--format csv|markdown, optional --output) |
uv run python src/cli.py --config configs/demo_qa_objective.json
uv run python src/cli.py --resume exp-abc123
uv run python src/cli.py --rerun exp-abc123 --new-judge groq/llama-3.3-70b-versatile
uv run python src/cli.py --export exp-abc123 --format markdown --output results.mdcli.py
├── config.py validate experiment JSON
├── data/sampler.py task sampling + resume
├── sprt/engine.py LLR update + boundaries
├── evaluation/ objective or judge scoring
├── models/ Ollama + OpenAI-compatible clients
├── storage/ SQLite trials + CSV/Markdown export
└── report/formatter.py ExperimentReport JSON
Design notes, independence assumptions, and the trust boundary live in ARCHITECTURE.md.
Varex is meant to be more than the SPRT runner.
The project is accumulating a growing collection of reproducible prompt-engineering experiments. That corpus is still early. Example categories include:
- Structured prompts vs simple prompts
- XML vs Markdown
- Role prompting vs no role
- Few-shot vs zero-shot
- Chain-of-thought vs concise prompting
- Long context vs concise context
Each archived experiment keeps the configuration, report, observations, and statistical evidence so results remain reproducible as models change. Layout and archive rules live under benchmarks/.
The point is to replace "trust me, this prompt is better" with something you can re-run.
| Document | Contents |
|---|---|
| RELEASE.md | Release notes, known limits, verification status |
| ARCHITECTURE.md | Full schema, SPRT design, trust boundary |
| TRACKER.md | Remediation status (living doc) |
| skills/varex/SKILL.md | OpenClaw skill behaviour |
No. Small Ollama models run comfortably on a CPU. Larger local models require more RAM or VRAM, while cloud providers (such as OpenAI-compatible APIs or Groq) only require a valid API key.
Yes. Set protocol to openai_compat and configure the appropriate provider and api_key_env values for the service you want to use.
An inconclusive result means the experiment ended without reaching a statistical conclusion. This can happen because of:
max_trials- Token budget exhaustion
- Repeated evaluation or provider errors
- Too many consecutive ties
Check the report's stopped_by field to see why the run terminated, and review the caveats field for any additional context.
Yes. SPRT is specifically designed for sequential analysis, so monitoring the log-likelihood ratio (LLR) while an experiment is running is statistically valid.
However, do not change the prompts, hypotheses, or evaluation configuration midway through a run and still treat it as the same experiment. Doing so invalidates the statistical assumptions behind the test.