vcr-llm

Record and replay LLM API conversations for deterministic testing. Zero deps. 14 providers.

The Problem

Your agent tests hit the LLM API every run. They're:

• Flaky — model responses change between runs
• Slow — each test waits for API round-trips
• Expensive — every test run costs real tokens
• Broken offline — can't run tests without API access

VCR.py doesn't work for LLMs because:

• All requests POST to the same URL (wrong response matching)
• SSE streaming frame boundaries are lost
• No conversation turn sequencing
• No token usage tracking

The Fix

from vcr_llm import use_cassette

@use_cassette("tests/cassettes/my_agent.jsonl")
def test_my_agent():
    result = my_agent("What is 2+2?")
    assert result == "4"  # 0 API calls. Deterministic. Free.

Install

pip install vcr-llm

Quick Demo

python -m vcr_llm demo

Recording 5-turn conversation with openai...
Cassette saved: demo.jsonl (5 turns, 312 total tokens)

Replaying from cassette...
All 5 turns matched. 0 API calls. 0 tokens used.

Supported Providers

Provider	Auto-Detected	Wire Format	Streaming Protocol
OpenAI	✓	OpenAI	SSE
Anthropic	✓	Messages API	SSE (named events)
Google Gemini	✓	generateContent	SSE
Mistral	✓	OpenAI-compat	SSE
Cohere	✓	Custom (v2)	SSE (typed events)
DeepSeek	✓	OpenAI-compat	SSE
Ollama	✓	Native / OpenAI-compat	NDJSON / SSE
Together AI	✓	OpenAI-compat	SSE
Groq	✓	OpenAI-compat	SSE
Fireworks AI	✓	OpenAI-compat	SSE
Azure OpenAI	✓	OpenAI-compat	SSE
Amazon Bedrock	✓	Converse / OpenAI-compat	SSE
OpenRouter	✓	OpenAI-compat	SSE
LiteLLM	✓	OpenAI-compat	SSE

Plus: any custom or local provider via the manual record_turn() / next_response() API.

Usage

Decorator (primary)

@use_cassette("tests/cassettes/my_flow.jsonl")
def test_flow():
    ...

Context Manager

with Cassette("flow.jsonl", mode="record") as c:
    result = my_agent("Hello")

with Cassette("flow.jsonl", mode="playback") as c:
    result = my_agent("Hello")
    assert c.total_tokens == 150

Async Support

@use_cassette("tests/cassettes/async_flow.jsonl")
async def test_async_flow():
    result = await my_async_agent("Hello")

Pytest Plugin

@pytest.mark.vcr_llm("my_flow.jsonl")
def test_with_marker(vcr_cassette):
    result = my_agent("Hello")
    assert vcr_cassette.turns == 1

Manual Recording (non-HTTP providers)

with Cassette("custom.jsonl", mode="record") as c:
    c.record_turn(
        request={"messages": [{"role": "user", "content": "Hello"}]},
        response={"content": "Hi there"},
        provider="vllm",
        usage={"prompt_tokens": 10, "completion_tokens": 5, "total_tokens": 15},
    )

Configuration

Auto mode (default): records on first run, replays on subsequent runs.

@use_cassette("flow.jsonl")  # mode="auto" by default

Strict matching: validates request bodies match the recording.

@use_cassette("flow.jsonl", strict_matching=True)

Custom normalizers: strip fields before comparing.

def strip_session_id(body):
    body.pop("session_id", None)
    return body

@use_cassette("flow.jsonl", custom_normalizers=[strip_session_id])

Cassette Format

Human-readable JSONL. One turn per line. Git-diff friendly.

{"turn": 0, "provider": "openai", "request": {...}, "response": {...}, "usage": {...}}
{"turn": 1, "provider": "anthropic", "request": {...}, "response": {...}, "usage": {...}}

How It Works

• Records HTTP traffic at the transport layer by patching httpx / requests
• Auto-detects which provider is being called from the request URL
• Replays recorded responses sequentially (not by URL matching)

Known Limitations

• Requires httpx or requests for automatic interception
• Sequential matching assumes deterministic call order
• Model version changes make recordings stale — re-record when upgrading
• Streaming responses are replayed as non-streaming (content is identical)

Part of the Agent Toolkit

vcr-llm is part of a suite of zero-dependency Python tools for the AI agent era:

Tool	What it does
ghostlines	Find code your team merged but never understood
agent-circuit	Circuit breaker for agent tool calls
agent-guard	Block prompt injection and path traversal at the tool boundary
agent-bill	Track LLM costs with itemized receipts
crowdllm	Multi-model voting for better answers
singleflight-agents	Deduplicate parallel agent tool calls

License

MIT