Skip to content

About

Prompt regression testing for RAG pipelines. Like git diff, but for AI.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

LLM Regress

Prompt regression testing for RAG pipelines. Like git diff, but for AI.

PyPI version License: MIT Provider Agnostic


LLM Regress dashboard — side-by-side prompt comparison and score trend chart


Why LLM Regress

  • You changed a prompt. Did it get better? Find out in 2 minutes.
  • Works with any LLM — OpenAI, Anthropic, Google Gemini, and more.
  • One env var. Set your API key and you're done.

Quickstart

# 1. Install
pip install llmregress

# 2. Set your API key (pick any provider you already have)
export ANTHROPIC_API_KEY=your_key_here
# or: export OPENAI_API_KEY=your_key_here
# or: export GOOGLE_API_KEY=your_key_here

# 3. Copy an example test file
cp examples/rag_pipeline.yaml my_tests.yaml

# 4. Compare your prompts
llmregress compare my_tests.yaml

# 5. Open the web dashboard
llmregress serve
# → http://localhost:7331

I have a LangChain RAG app — how do I use this?

If your app looks like:

result = chain.invoke({"question": q, "context": c})

Translate it into a YAML test case:

model: anthropic/claude-3-5-haiku-20241022
judge_model: openai/gpt-4o-mini
test_cases:
  - id: my_test
    input: "What is the default chunk size?"
    context: "LangChain's default chunk_size is 1000 characters..."
    reference_answer: >
      LangChain's RecursiveCharacterTextSplitter defaults to a chunk_size of 1000 characters
      and a chunk_overlap of 200 characters.
    criteria:
      - "Answer is factually correct"
      - "Response is concise (under 50 words)"
    prompt_v1: |
      You are a helpful assistant. Context: {context}
      Question: {input}
    prompt_v2: |
      Answer only from context. Be concise.
      Context: {context}
      Question: {input}

Then run: llmregress compare my_tests.yaml

Providers & model strings

Change 1–2 lines in your YAML — no code changes. You can use any model from each provider family:

Provider Example model string Env var
Anthropic anthropic/claude-3-5-haiku-20241022 ANTHROPIC_API_KEY
Anthropic anthropic/claude-opus-4 ANTHROPIC_API_KEY
OpenAI openai/gpt-4o-mini OPENAI_API_KEY
OpenAI openai/gpt-4o OPENAI_API_KEY
Google Gemini gemini/gemini-2.0-flash GOOGLE_API_KEY
Google Gemini gemini/gemini-1.5-pro GOOGLE_API_KEY
Ollama (local) ollama/llama3 (none)

The model string format is always provider/model-name. Any model supported by LiteLLM works — just set the matching API key.

Reduce judge bias: use a different model family for judge_model than model. Example: Anthropic runner + OpenAI judge = cross-family, lowest self-preference bias.

CLI reference

Command Description
llmregress compare <yaml> Run + print colored diff to terminal
llmregress run <yaml> Run + store results (no terminal output)
llmregress history List past runs
llmregress serve Start web dashboard at localhost:7331
llmregress demo Try it without an API key

Web dashboard

llmregress serve

Opens at http://localhost:7331. Features:

  • Side-by-side output comparison per test case
  • Live streaming — results appear as they complete
  • Run history with score trend chart
  • Click any past run while a new test is running — views are independent

Environment variables

Variable Default Description
ANTHROPIC_API_KEY — Anthropic API key
OPENAI_API_KEY — OpenAI API key
GOOGLE_API_KEY — Google Gemini API key
LLMREGRESS_DB_PATH ~/.llmregress/history.db SQLite database path
LLMREGRESS_PORT 7331 Web server port
LLMREGRESS_HOST 127.0.0.1 Web server bind address
LLMREGRESS_YAML_DIR ~/.llmregress/tests Allowed directory for YAML test files
LLMREGRESS_JUDGE_VOTES 3 Calls per criterion: 1=fast, 3=reliable majority vote

Docker

docker-compose up
# Dashboard at http://localhost:7331

Mount your YAML test files into the container:

# docker-compose.yml — add a volume:
volumes:
  - ./my_tests:/workspace/tests

Contributing

  1. Fork the repo
  2. Create a branch: git checkout -b feat/my-feature
  3. Run tests: pytest tests/ -v
  4. Open a PR

About

Prompt regression testing for RAG pipelines. Like git diff, but for AI.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages