diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 7feacb1..3706c7f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -39,6 +39,21 @@ jobs: file: ./coverage.xml fail_ci_if_error: false + minimum-sdks: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.10" + - name: Install tested SDK floors and legacy Gemini + run: >- + python -m pip install -e ".[all,dev,google-legacy]" + "openai==1.109.1" "anthropic==0.75.0" "google-genai==1.47.0" + "langchain-core==0.3.80" "langchain-openai==0.3.35" + - name: Run all tests + run: python -m pytest + lint: runs-on: ubuntu-latest steps: @@ -97,7 +112,7 @@ jobs: path: dist/ publish: - needs: [test, lint, build] + needs: [test, minimum-sdks, lint, build] runs-on: ubuntu-latest if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v') permissions: diff --git a/CHANGELOG.md b/CHANGELOG.md index af6e6f4..b2c5279 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,102 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [0.4.0] - Unreleased + +### Breaking changes / upgrading from 0.3.x + +- Recording is now implemented: the recording flags and markers no longer always + fail closed as unavailable. `--llm-record=once` can make real, billable API calls + when a cassette does not exist; `new_episodes` records unmatched calls and `all` + replaces recordings. Use `--llm-record=none` (the default) for offline replay; + missing or mismatched interactions still fail closed. Review the selected mode + before running tests with real credentials, and regenerate old experimental + cassettes with the new recorder. +- Replace direct `LLMRecorder(mode="auto"|"record"|"replay")` usage with + `record_mode="once"|"all"|"none"`, respectively. The old spellings warn but still + work. Legacy `mode="none"` means live pass-through; new `record_mode="none"` + means offline replay. Do not substitute one for the other without checking intent. +- Anthropic fixtures keep `mock_anthropic`, `add_response`, `simulate_error`, and + their existing arguments. Helpers that assumed a bare generator or a configurable + `MagicMock` from `messages.stream()` must use the SDK's stream context manager: + `with client.messages.stream(...) as stream:` (or `async with` for async clients), + then consume `stream.text_stream` or call `stream.get_final_message()` (await it + for async clients). `messages.create(stream=True)` yields official typed events; + access text with `event.delta.text` after checking its event/delta type. Configure + fixture output through `add_response(..., stream_chunks=..., tool_calls=...)` + rather than mutating private response/event-building helpers. The Anthropic SDK + minimum is now 0.75.0; public fixture helpers do not require renaming. +- `[google]` now uses `google-genai`; import `from google import genai` and construct + `genai.Client(...)`. Install `[google-legacy]` to retain the deprecated + `google.generativeai` adapter temporarily. Enable `--llm-stats` if you relied on + the previously automatic cost summary. + +### Added + +- Working OpenAI and Anthropic recording/replay using readable YAML cassettes, + with `--llm-record=none|once|new_episodes|all`, the `llm_recorder` fixture, + automatic recording/replay markers, and offline replay by default. +- SDK-parsed sync/async JSON and SSE playback, including fragmented tool calls, + OpenAI Responses structured parsing and embeddings, Anthropic stream helpers, + and HTTP error/retry sequences. Recording buffers complete streams; arrival + timing and partial transport failures are not replayed. +- Request matching includes provider, method, URL/query, JSON body, and selected + semantic headers. Identical requests consume interactions in order. Misses in + offline modes and unsupported endpoints fail before network access. +- Credential/header redaction for requests, response JSON and SSE, atomic cassette + writes, and collision-resistant per-test paths. Redaction is not general PII removal. +- Native `google-genai` typed generation, chat sessions, sync/async streaming, + function calls, structured output, and provider error classes. +- Typed OpenAI Responses streams and function-call outputs, SDK stream context + managers, and final structured output through `responses.stream`. +- `--llm-stats`, `--no-llm-stats`, and `llm_stats` configuration; statistics are now + off by default. +- A tested README quickstart, provider/stream/tool/error/recording examples, and + an explicit comparison with mock, transport, recording, and evaluation tools. +- An actual asciinema demo (`docs/demo.cast`) and a portable regeneration script. +- `py.typed` package marker and version consistency checks. + +### Changed + +- `[google]` and `[all]` now install `google-genai`. `[google-legacy]` retains the + old `google-generativeai` adapter with a deprecation warning. Existing fixture + names, response configuration arguments, and the legacy module accessor remain. +- New direct recorder usage selects `record_mode`. Legacy `mode=auto|record|replay` + maps to once/all/none with a deprecation warning. Legacy `mode='none'` retains + explicit live pass-through; `record_mode='none'` is offline replay. +- Provider dependency floors now match tested SDK versions: OpenAI 1.109.1, + Anthropic 0.75.0, google-genai 1.47.0, langchain-core 0.3.80, and + langchain-openai 0.3.35. Removed the Anthropic `<1.0` cap after testing 1.15.0. + HTTPX and HTTPX2 response types follow the installed provider SDK. + +### Fixed + +- Redaction preserves JSON/tool-schema fields named `key`, `token`, `auth`, and + similar names. Credential headers/query parameters and known credential values + remain redacted; `sk-ant-...` keys carry the Anthropic label. +- Tiny nonzero estimated costs display as `<$0.0001` instead of `$0.0000`. +- Simulated timeout/server errors now use the SDK's correct constructors. +- Async OpenAI Responses latency yields the event loop. +- Responses tool-arguments completion events include the function name required + by some SDK versions; the Google dependency floor preserves Python 3.9 installs. +- Chat tool streams no longer require explicit argument fragments and support + multiple tools. Anthropic streams preserve both text and tool-use blocks. +- Legacy Gemini streaming no longer passes duplicate `stream` arguments, and + positional prompts/model names are included in the request ledger. +- Global strict mode also applies to LangChain. +- Implicit token estimates no longer load/download tiktoken encoders. Already-loaded + encoders remain usable; otherwise counting falls back to a character estimate. +- Package `__version__` now agrees with project metadata at 0.4.0. + +### Validation and remaining limits + +- Full suite exercised on Python 3.10–3.13; minimum/current SDK contract tests, + actual local HTTP record/replay, executable README examples, and package checks. +- No paid/live-provider validation or PyPI publication. Gemini recording, LiteLLM, + new assertion helpers, and a live-test environment marker are not included. +- See [SDK compatibility](docs/sdk-compatibility.md) for versions, test evidence, + private-SDK-hook risk, buffered streaming, and unsupported interfaces. + ## [0.3.1] - 2026-08-30 ### Fixed @@ -116,6 +212,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - No external network calls in mock mode (completely isolated testing) [Unreleased]: https://github.com/godhiraj-code/pytest-mockllm/compare/v0.3.1...HEAD +[0.4.0]: https://github.com/godhiraj-code/pytest-mockllm/compare/v0.3.1...HEAD [0.3.1]: https://github.com/godhiraj-code/pytest-mockllm/compare/v0.3.0...v0.3.1 [0.3.0]: https://github.com/godhiraj-code/pytest-mockllm/releases/tag/v0.3.0 [0.2.3]: https://github.com/godhiraj-code/pytest-mockllm/releases/tag/v0.2.3 diff --git a/README.md b/README.md index 7394ddd..05f2422 100644 --- a/README.md +++ b/README.md @@ -1,374 +1,336 @@ -

pytest-mockllm

- -

- Fixture-scoped LLM mocking for pytest -

- -

- PyPI version - Python versions - CI - License - Downloads -

- -

- Quick Start • - Supported Integrations • - Examples • - Recording • - Configuration -

- ---- - -> [!IMPORTANT] -> **Interception is fixture-scoped.** Calls made through an active provider fixture such as -> `mock_openai` are intercepted locally. Installing the plugin alone does not block network -> access from tests that do not request a mock fixture. Keep real API keys out of unit-test -> environments and use normal CI egress controls as a second safety layer. - -## Why pytest-mockllm? - -Provider calls make unit tests slower, non-deterministic, dependent on secrets, and potentially -costly. `pytest-mockllm` supplies auto-discovered pytest fixtures that return configured responses -and record call details without contacting the provider on supported fixture paths. +Tests that call real LLM APIs are slow, cost money, and give different answers each run. -```python -def test_my_chatbot(mock_openai): - mock_openai.add_response("Hello! I'm here to help.") - - response = my_chatbot.chat("Hi there!") - - assert "help" in response.lower() - assert mock_openai.call_count == 1 -``` - -## Quick Start +# pytest-mockllm -### Installation +Configure the answer, run your Python code, and assert what it sent to the model. +Auto-discovered pytest fixtures for OpenAI, Anthropic, Google Gemini, and LangChain. +MIT licensed. No API key is needed for mocked tests. -Install the extra for the SDK used by your test: +## 30-second quickstart ```bash pip install "pytest-mockllm[openai]" ``` -Available extras are: - -```bash -pip install "pytest-mockllm[anthropic]" -pip install "pytest-mockllm[google]" -pip install "pytest-mockllm[langchain]" -``` - -The `langchain` extra includes `langchain-core` and `langchain-openai`, which are required by the -LangChain example below. Use `pytest-mockllm[all]` to install every supported integration. A base -`pip install pytest-mockllm` installs the pytest plugin and its core runtime dependencies, but not -the optional provider SDKs. - -The plugin is discovered automatically through its `pytest11` entry point; no `pytest_plugins` -setting or fixture import is required. - -### Your First Test +Save this as `test_chat.py`, then run `pytest`: ```python -def test_customer_support_bot(mock_openai): - mock_openai.add_response("I can help with your order. What's your order number?") +from openai import OpenAI - from openai import OpenAI - client = OpenAI(api_key="fixture-only") - response = client.chat.completions.create( - model="gpt-4o", - messages=[{"role": "user", "content": "I need help with my order"}], - ) - - assert "order number" in response.choices[0].message.content.lower() +def test_chat(mock_openai): + mock_openai.add_response("Hello, Ada!") + with OpenAI(api_key="fixture-only") as client: + reply = client.chat.completions.create( + model="gpt-4o-mini", + messages=[{"role": "user", "content": "My name is Ada"}], + ) + assert reply.choices[0].message.content == "Hello, Ada!" assert mock_openai.call_count == 1 - assert mock_openai.last_call["model"] == "gpt-4o" + assert mock_openai.last_call["model"] == "gpt-4o-mini" + assert "Ada" in mock_openai.last_call["messages"][0]["content"] ``` -The placeholder key only satisfies the SDK constructor. The active fixture intercepts the -supported call before transport. - -## Supported Integrations - -| Fixture | Install extra | Intercepted interfaces | -|---|---|---| -| `mock_openai` | `openai` | Chat Completions, Responses create/parse, embeddings; sync/async; chat streaming and tool-call chunks | -| `mock_anthropic` | `anthropic` | Messages; sync/async; text and tool-use streaming | -| `mock_gemini` | `google` | `GenerativeModel.generate_content`, async generation, streaming, and chat sessions | -| `mock_langchain` | `langchain` | Installed ChatOpenAI-compatible classes plus direct mock model `invoke`, `ainvoke`, `stream`, `astream`, and structured output | -| `mock_llm` | matching provider extra | OpenAI by default; select `openai`, `anthropic`, or `gemini` with `@pytest.mark.llm_mock` | +In your application tests, call your own function in place of the SDK call above. +The fixture intercepts supported SDK calls, including clients imported or created +before the fixture starts. No fixture imports or `pytest_plugins` setting are needed. +The placeholder key satisfies SDK construction; it is never sent by the mock. -OpenAI and Anthropic supported paths return official SDK response/event objects when their SDKs -are installed. Their active fixtures also block unhandled requests at the SDK base-request layer, -including requests from clients created before fixture activation. Gemini and LangChain replace -the supported high-level entry points listed above; they are not global network guards for every -function exposed by those libraries. +## Providers -The `google` extra currently targets the legacy `google-generativeai` package. That upstream SDK is -deprecated; support for its replacement, `google-genai`, has not been implemented yet. +Install only what you use: `[openai]`, `[anthropic]`, `[google]`, or `[langchain]`. +`[all]` installs the current integrations; the base package does not install provider SDKs. -All fixtures support queued deterministic responses, strict mode, call tracking, token/cost -estimates, and deterministic error simulation. Response fidelity is limited to the interfaces in -the table; this package does not claim complete coverage of every provider API. +| Fixture | Supported interfaces | +| --- | --- | +| `mock_openai` | Chat Completions, Responses create/parse, embeddings; sync/async; text and function-call streams | +| `mock_anthropic` | Messages create/stream; sync/async; text and tool-use streams | +| `mock_gemini` | `google.genai.Client` models generate_content/generate_content_stream, chat sessions; sync/async | +| `mock_langchain` | ChatOpenAI-compatible invoke/ainvoke, stream/astream, structured output; direct `mock_langchain.model` | +| `mock_llm` | OpenAI by default; select a provider with `@pytest.mark.llm_mock(provider="anthropic")` | -## Provider Examples +These are scoped test doubles, not complete emulators. Installing the plugin alone +blocks no traffic. Active OpenAI, Anthropic, and modern Gemini mocks reject unhandled +HTTP SDK requests. Realtime/WebSocket APIs, file uploads, and arbitrary network traffic +are outside this guarantee. LangChain and legacy Gemini intercept their documented +methods only. SDK constructor validation still applies. -### OpenAI +### OpenAI Responses API ```python -def test_openai(mock_openai): - mock_openai.add_response("The answer is 42") +from openai import OpenAI +from pydantic import BaseModel - from openai import OpenAI - client = OpenAI(api_key="fixture-only") - response = client.chat.completions.create( - model="gpt-4o", - messages=[{"role": "user", "content": "What is the meaning of life?"}], - ) +class Contact(BaseModel): + name: str - assert response.choices[0].message.content == "The answer is 42" + +def test_responses(mock_openai): + mock_openai.add_response('{"name":"Ada"}') + with OpenAI(api_key="fixture-only") as client: + result = client.responses.parse( + model="gpt-4o-mini", input="Extract Ada's name", text_format=Contact, + ) + assert result.output_parsed == Contact(name="Ada") ``` ### Anthropic ```python -def test_anthropic(mock_anthropic): - mock_anthropic.add_response("I'd be happy to help!") - - from anthropic import Anthropic +from anthropic import Anthropic - client = Anthropic(api_key="fixture-only") - response = client.messages.create( - model="claude-3-5-sonnet-20241022", - max_tokens=1024, - messages=[{"role": "user", "content": "Hello Claude!"}], - ) - assert "happy" in response.content[0].text +def test_claude(mock_anthropic): + mock_anthropic.add_response("Your order is on its way.") + with Anthropic(api_key="fixture-only") as client: + reply = client.messages.create( + model="claude-sonnet-4-5", max_tokens=100, + messages=[{"role": "user", "content": "Where is my order?"}], + ) + assert reply.content[0].text == "Your order is on its way." ``` ### Google Gemini -```python -def test_gemini(mock_gemini): - mock_gemini.add_response("Here's what I found...") +The `[google]` extra uses **google-genai**, not the retired google-generativeai SDK. +For a migration period, `[google-legacy]` also installs the old SDK; its existing +`GenerativeModel` adapter remains available with a deprecation warning. - import google.generativeai as genai +```python +from google import genai - model = genai.GenerativeModel("gemini-1.5-pro") - response = model.generate_content("Tell me about AI") - assert "found" in response.text +def test_gemini(mock_gemini): + mock_gemini.add_response("Paris") + with genai.Client(api_key="fixture-only") as client: + reply = client.models.generate_content( + model="gemini-2.5-flash", contents="Capital of France?", + ) + assert reply.text == "Paris" + assert mock_gemini.last_call["contents"] == "Capital of France?" ``` +`client.chats.create(...).send_message(...)`, `client.aio.models.generate_content(...)`, +and `await client.aio.models.generate_content_stream(...)` use the same response queue. +Generated responses expose SDK `candidates`, `function_calls`, and `usage_metadata`. + ### LangChain ```python -def test_langchain(mock_langchain): - mock_langchain.add_response("Paris is the capital of France.") - - from langchain_core.prompts import ChatPromptTemplate - from langchain_openai import ChatOpenAI +from langchain_core.prompts import ChatPromptTemplate +from langchain_openai import ChatOpenAI - llm = ChatOpenAI(model="gpt-4o", api_key="fixture-only") - prompt = ChatPromptTemplate.from_template("What is the capital of {country}?") - chain = prompt | llm - result = chain.invoke({"country": "France"}) - - assert "Paris" in result.content +def test_chain(mock_langchain): + mock_langchain.add_response("Paris") + prompt = ChatPromptTemplate.from_template("Capital of {country}?") + model = ChatOpenAI(model="gpt-4o-mini", api_key="fixture-only") + assert (prompt | model).invoke({"country": "France"}).content == "Paris" ``` -## Examples +This replaces the model call, so it does not exercise LangChain's provider transport, +callbacks, or automatic tool execution. Tool responses do not execute your Python tools. -### Multiple Responses +## Async and streaming ```python -def test_conversation(mock_openai): - mock_openai.add_responses( - "Hi! How can I help you today?", - "I can help with that order.", - "Your order has been updated.", - ) - - assert "help" in chatbot.send("Hello") - assert "order" in chatbot.send("I need to change my order") - assert "updated" in chatbot.send("Change quantity to 5") -``` +import pytest +from openai import AsyncOpenAI, OpenAI -### Streaming Responses -```python -def test_streaming(mock_openai): - mock_openai.add_response("This is a streaming response that comes in chunks") - - from openai import OpenAI - - client = OpenAI(api_key="fixture-only") - stream = client.chat.completions.create( - model="gpt-4o", - messages=[{"role": "user", "content": "Tell me a story"}], - stream=True, - ) - - full_response = "".join( - chunk.choices[0].delta.content or "" - for chunk in stream - ) - assert "streaming" in full_response -``` +@pytest.mark.asyncio +async def test_async(mock_openai): + mock_openai.add_response("Async answer") + async with AsyncOpenAI(api_key="fixture-only") as client: + reply = await client.chat.completions.create(model="gpt-4o-mini", messages=[]) + assert reply.choices[0].message.content == "Async answer" -### Function and Tool Calling -```python -def test_function_calling(mock_openai): - mock_openai.add_response( - "", - tool_calls=[{ - "id": "call_123", - "function": { - "name": "get_weather", - "arguments": {"location": "San Francisco", "unit": "celsius"}, - }, - }], - ) - - from openai import OpenAI - - client = OpenAI(api_key="fixture-only") - response = client.chat.completions.create( - model="gpt-4o", - messages=[{"role": "user", "content": "What's the weather?"}], - ) - - tool_call = response.choices[0].message.tool_calls[0] - assert tool_call.function.name == "get_weather" - assert mock_openai.last_call["model"] == "gpt-4o" +def test_stream(mock_openai): + mock_openai.add_response("Hello world", stream_chunks=["Hello ", "world"]) + with OpenAI(api_key="fixture-only") as client: + stream = client.chat.completions.create( + model="gpt-4o-mini", messages=[], stream=True, + ) + text = "".join(chunk.choices[0].delta.content or "" for chunk in stream) + assert text == "Hello world" ``` -### Token Usage and Cost Assertions +Anthropic `messages.create(stream=True)` yields official SDK events, including +`content_block_delta`. Use `tool_call_chunks` for fragmented JSON arguments in OpenAI +chat and Anthropic tool streams. Both provider mocks and record/replay support SDK stream context managers. +OpenAI Responses streams also support `get_final_response()` and structured parsing. -```python -def test_stays_within_budget(mock_openai): - from pytest_mockllm.core import TokenUsage, estimate_cost - - mock_openai.add_response( - "A detailed response...", - token_usage=TokenUsage(prompt_tokens=500, completion_tokens=1000), - ) - - my_function() - - assert mock_openai.total_tokens < 2000 - cost = estimate_cost( - "gpt-4o", - mock_openai.total_prompt_tokens, - mock_openai.total_completion_tokens, - ) - assert cost < 0.05 -``` - -### Error Simulation +## Tool calls ```python -def test_handles_rate_limit(mock_openai): - mock_openai.simulate_error("rate_limit", times=1) - mock_openai.add_response("recovered") +import json +from openai import OpenAI + + +def test_tool_call(mock_openai): + mock_openai.add_response("", tool_calls=[{ + "id": "call_weather", + "function": {"name": "weather", "arguments": {"city": "Paris"}}, + }]) + with OpenAI(api_key="fixture-only") as client: + reply = client.chat.completions.create(model="gpt-4o-mini", messages=[]) + tool = reply.choices[0].message.tool_calls[0] + assert tool.function.name == "weather" + assert json.loads(tool.function.arguments) == {"city": "Paris"} +``` - # The first supported provider call raises a provider-style rate-limit error. - # The next call returns "recovered". +The same `tool_calls` configuration produces Anthropic `tool_use` blocks and Gemini +function-call parts. Configure what the model returns, then test whether your application +validates the arguments and invokes the right tool. +## Errors, retries, and strict mode -def test_handles_jitter(mock_openai): - mock_openai.simulate_jitter(max_ms=500) +```python +import pytest +from openai import OpenAI, RateLimitError -def test_random_failures(mock_openai): - mock_openai.simulate_random_errors(probability=0.1) +def test_retry_logic(mock_openai): + mock_openai.simulate_error("rate_limit", times=1) + mock_openai.add_response("Recovered") + with OpenAI(api_key="fixture-only") as client: + with pytest.raises(RateLimitError): + client.chat.completions.create(model="gpt-4o-mini", messages=[]) + reply = client.chat.completions.create(model="gpt-4o-mini", messages=[]) + assert reply.choices[0].message.content == "Recovered" + assert [call["outcome"] for call in mock_openai.request_ledger] == ["rate_limit", "success"] ``` -### Strict Mode +Mock errors occur at the SDK method boundary: use them to test **your application's** +retry handling. To exercise the SDK's internal HTTP retry loop, use record/replay. +Supported errors include `rate_limit`, `auth`, `timeout`, `server`, and `invalid_request`; +Anthropic also supports `overloaded`. `after_calls=N` delays an error until N successes. +`times=N` bounds it. `call_count` counts successful configured responses; `calls` and +`request_ledger` include failed attempts. -```python -import pytest +Use `add_responses("first", "second")` to queue answers. After the queue is exhausted, +`set_default_response(...)` supplies a fallback; otherwise non-strict mode returns a +placeholder answer. Strict mode catches missing responses: +```bash +pytest --llm-strict +``` -def test_catches_unconfigured_calls(mock_openai): - mock_openai.set_strict_mode(True) +Or call `mock_openai.set_strict_mode()` in one test. Token usage is configurable with +`TokenUsage`; cost estimates use a static price table and are not billing measurements. +Token counting uses preloaded tiktoken encoders when available, otherwise a character +estimate. It never downloads tokenizers implicitly; preload an encoder explicitly if +you need exact tokenization, or supply `TokenUsage` for deterministic budget tests. - with pytest.raises(RuntimeError, match="No mock response configured"): - my_function_that_calls_llm() -``` +## Record real responses once, replay offline -## Recording and Replay +Use `llm_recorder` **instead of** a provider mock fixture. It captures OpenAI Chat +Completions, Responses (including structured parsing), embeddings, and Anthropic Messages. +The SDK parses the recorded HTTP JSON or SSE into its normal response objects. -Recording and replay are currently unavailable. Earlier releases exposed the interface without -safely intercepting provider calls. Current `auto`, `record`, and `replay` modes fail before test -code can reach a provider instead of allowing an apparent replay test to fall through to a live -API. Use the deterministic provider fixtures until recording returns with provider-level -behavioral tests. +```python +import os +from openai import OpenAI -The `--llm-record`, `--llm-cassette-dir`, `llm_record`, and `llm_replay` names remain reserved for -future compatibility; using them with `llm_recorder` does not create or replay cassettes today. -## Configuration +def test_recorded_reply(llm_recorder): + with OpenAI(api_key=os.getenv("OPENAI_API_KEY", "replay-only")) as client: + reply = client.responses.create(model="gpt-4o-mini", input="Say hello.") + assert reply.output_text +``` -### Strict Mode +Set your provider key in the environment for the first run: ```bash -pytest --llm-strict +pytest test_recorded.py --llm-record=once # real API calls if no cassette exists +pytest test_recorded.py # offline replay; no key needed ``` -Strict mode makes a provider fixture fail when a supported call has no configured response. It can -also be enabled in pytest configuration: +| Mode | Existing matching interaction | Missing/exhausted interaction | +| --- | --- | --- | +| `none` (default) | Replay | Fail without sending HTTP | +| `once` | Replay | Record only if the cassette file did not exist on entry | +| `new_episodes` | Replay | Record and append | +| `all` | Record again | Record; replace the old cassette | + +Bare `--llm-record` means `all` for compatibility. Live modes make billable API calls. +The flag affects tests using `llm_recorder` or the markers; it does not activate recording +across your entire suite. `@pytest.mark.llm_record` activates the fixture, with offline +defaults. `@pytest.mark.llm_record(mode="once")` selects a mode for one test. +`@pytest.mark.llm_replay` forces offline replay even when the CLI requests recording. + +Cassettes live under `tests/llm_cassettes/`, relative to pytest's root directory. Names +include a hash of the full test ID, including module, class, and parameters. Override +with `--llm-cassette-dir=...` or `llm_cassette_dir`. Review and commit cassettes for CI; +the repository's own local cassette directory is ignored by Git. + +Matching includes provider, HTTP method, URL/query, JSON request body (including model, +prompt, tools, sampling parameters, and stream flag), and selected API-version/project +headers. Credential and SDK telemetry headers do not affect matching. Identical requests +consume recorded responses in order; a cassette cannot be replayed infinitely in one test. +Changing the base URL or a prompt requires a matching recording. + +Credential headers and query parameters, known key/token patterns, and credentials used +by the SDK are redacted, including echoes in JSON or SSE. JSON fields named `key`, `token`, +or `auth` are preserved unless their values contain a recognized secret. This is **not general +PII removal**: prompts and model output remain readable. Review cassettes before committing. +Redacted fields replay as redacted values. + +Recording buffers the complete response, including streams, before handing it to the SDK; +replay preserves SSE events and tool fragments, not network timing. Incomplete HTTP reads +are not saved. Completed interactions are saved atomically even if a later test assertion +fails. OpenAI/Anthropic HTTPX and HTTPX2 clients are covered; custom clients that override +`send` are rejected (use a custom transport instead). WebSockets, non-JSON requests, +other SDK endpoints, Gemini recording, overlapping recorder contexts, and concurrent +writers to the **same cassette file** are not supported. Separate tests get separate files. + +For direct context-manager use, choose `LLMRecorder(path, record_mode="once")`. +Legacy `mode="auto"`, `"record"`, and `"replay"` map to once/all/none with a deprecation +warning. Legacy `mode="none"` still means explicit live pass-through; migrate to +`record_mode="none"` for offline replay. + +## How this compares + +| Tool | Choose it when | Where pytest-mockllm fits | +| --- | --- | --- | +| [unittest.mock](https://docs.python.org/3/library/unittest.mock.html) | You want general-purpose patching, specs, and call assertions with no extra dependency. | Queues, provider-shaped responses, streams, and errors require less setup here; `mock` gives more control. | +| [respx](https://lundberg.github.io/respx/) | You want precise HTTPX route matching and transport-level assertions. | Provider fixtures avoid constructing HTTP payloads. Use a transport tool when wire behavior is the subject of your test. | +| [vcrpy](https://vcrpy.readthedocs.io/en/latest/) | You need mature HTTP recording across multiple services and client libraries. | Our narrower recorder targets OpenAI/Anthropic JSON and SSE; vcrpy has a broader ecosystem and configuration surface. | +| [pytest-recording](https://github.com/kiwicom/pytest-recording) | You want pytest integration for VCR.py, cassette markers, and network controls. | This plugin combines handwritten LLM responses and scoped LLM cassettes. Keep pytest-recording for general HTTP integration tests. | +| [deepeval](https://github.com/confident-ai/deepeval) / [promptfoo](https://www.promptfoo.dev/docs/intro/) | You need to evaluate answer quality, prompt changes, or model behavior. | Those judge generated answers; this plugin tests the code around the model. Use both when you need both kinds of evidence. | + +A passing mocked test does not prove a prompt produces a good answer or that the live +service accepts your request. Keep a small live integration suite and separate evaluations. + +## Configuration ```toml [tool.pytest.ini_options] llm_strict = true +llm_stats = false +llm_cassette_dir = "tests/llm_cassettes" ``` -### Universal Fixture - -```python -import pytest - - -@pytest.mark.llm_mock(provider="anthropic") -def test_with_anthropic(mock_llm): - mock_llm.add_response("Configured through the universal fixture") - # Calls through the Anthropic Messages API are intercepted here. -``` +Statistics are **off by default**. Enable with `pytest --llm-stats` or `llm_stats = true`; +`--no-llm-stats` overrides both. There is no cost banner in a normal run. -## Development +## Demo and development -See [CONTRIBUTING.md](CONTRIBUTING.md) for contribution guidelines. +[Demo tests](examples/test_demo.py) run offline. Generate an asciinema v2 recording of +an actual run with `python scripts/record_demo.py`; play it with `asciinema play docs/demo.cast`. +The script needs only the `[openai]` extra and does not fabricate test output. ```bash -pip install -e ".[dev,all]" +pip install -e ".[all,dev]" build pytest ruff check src/ mypy src/ python -m build ``` -## Roadmap - -- [x] Async provider fixtures -- [x] Token and cost estimates -- [x] Deterministic latency and error simulation -- [ ] Safe provider-level recording and replay -- [ ] More providers -- [ ] Process-safe pytest-xdist aggregation - -## Author - -[Dhiraj Das](https://www.dhirajdas.dev) · [Follow @dkkkkd on X](https://x.com/dkkkkd) - -## License +See [CONTRIBUTING.md](CONTRIBUTING.md), [CHANGELOG.md](CHANGELOG.md), and the +[SDK compatibility notes](docs/sdk-compatibility.md) for tested versions and limits. -MIT License — see [LICENSE](LICENSE) for details. +MIT licensed — [LICENSE](LICENSE). Maintained by [Dhiraj Das](https://www.dhirajdas.dev). diff --git a/docs/demo.cast b/docs/demo.cast new file mode 100644 index 0000000..dcfed02 --- /dev/null +++ b/docs/demo.cast @@ -0,0 +1,9 @@ +{"version": 2, "width": 100, "height": 24, "timestamp": 1792416897, "title": "pytest-mockllm: streams, tools, retries"} +[1.656, "o", "\u001b[1m======================================= test session starts =======================================\u001b[0m\r\n"] +[2.531, "o", "\u001b[1mcollecting ... \u001b[0mcollected 3 items\r\n"] +[2.531, "o", "\r\n"] +[2.781, "o", "examples/test_demo.py::test_stream_without_an_api_key \u001b[32mPASSED\u001b[0m\u001b[32m [ 33%]\u001b[0m\r\n"] +[2.796, "o", "examples/test_demo.py::test_weather_tool_arguments \u001b[32mPASSED\u001b[0m\u001b[32m [ 66%]\u001b[0m\r\n"] +[2.796, "o", "examples/test_demo.py::test_recovers_after_rate_limit \u001b[32mPASSED\u001b[0m\u001b[32m [100%]\u001b[0m\r\n"] +[2.796, "o", "\r\n"] +[2.796, "o", "\u001b[32m======================================== \u001b[32m\u001b[1m3 passed\u001b[0m\u001b[32m in 1.14s\u001b[0m\u001b[32m ========================================\u001b[0m\r\n"] diff --git a/docs/sdk-compatibility.md b/docs/sdk-compatibility.md new file mode 100644 index 0000000..6cbd214 --- /dev/null +++ b/docs/sdk-compatibility.md @@ -0,0 +1,78 @@ +# SDK compatibility and validation + +Checked on 2026-10-19 using PyPI metadata and the installed SDK source. + +| Package | Current version tested | Minimum version tested | +| --- | --- | --- | +| [openai](https://pypi.org/project/openai/3.35.1/) | 3.35.1 | 1.109.1 | +| [anthropic](https://pypi.org/project/anthropic/1.15.0/) | 1.15.0 | 0.75.0 | +| [google-genai](https://pypi.org/project/google-genai/2.31.0/) | 2.31.0 | 1.47.0 | +| [langchain-core](https://pypi.org/project/langchain-core/1.6.12/) | 1.6.12 | 0.3.80 | +| [langchain-openai](https://pypi.org/project/langchain-openai/1.7.2/) | 1.7.2 | 0.3.35 | + +Provider dependency floors reflect the older SDK set actually exercised here. This +is not a claim that every version between the endpoints has been tested. OpenAI and +Anthropic extras require Pydantic 2 for typed response/stream validation. All provider +SDKs remain optional. The google-genai 1.47.0 floor preserves Python 3.9 support; +newer SDK releases can require newer Python versions. The deprecated google-generativeai adapter is tested separately +with 0.8.6 and issues a deprecation warning on model construction. + +## Reproducible checks + +```bash +pip install -e ".[all,dev]" +pytest +ruff check src/ +mypy src/pytest_mockllm --ignore-missing-imports --python-version 3.11 --follow-imports=silent +``` + +The full suite was run locally on Windows with Python 3.9, 3.10, 3.11, 3.12, and 3.13. +Python 3.9 uses the newest SDK versions that still support that interpreter. +The optional legacy Gemini module skips when google-generativeai is absent. The +older SDK set and legacy adapter were also tested on Python 3.12. The existing +cross-platform CI matrix covers Linux, macOS, and Windows; CI results, not a local +Windows run, establish those other platform results. + +The CI minimum-SDK job installs: + +```bash +pip install -e ".[all,dev,google-legacy]" \ + "openai==1.109.1" "anthropic==0.75.0" "google-genai==1.47.0" \ + "langchain-core==0.3.80" "langchain-openai==0.3.35" +pytest +``` + +## Evidence and limits + +- [Recording tests](../tests/test_recording.py) exercise actual SDK parsing with + controlled HTTP responses: sync/async JSON, SSE text/tool fragments, SDK retries, + error types, credential redaction, all four modes, matching, exhaustion, file + replacement, cleanup, and incomplete reads. Local HTTP-server tests record over + a real socket and replay after the server stops with socket connections blocked. +- [SDK regressions](../tests/acceptance/test_latest_sdk.py) exercise Responses API + text/usage/function calls, typed events, stream managers, parsed final output, + nonblocking async delay, provider errors, and an OpenAI-compatible base URL. +- [Gemini tests](../tests/test_gemini.py) exercise google-genai typed responses, + chat state, sync/async streams, tools, structured output, strict mode, and errors. +- [Plugin tests](../tests/test_plugin.py) cover CLI modes, automatic markers, + cassette names, mock/recorder conflicts, strict mode, and opt-in statistics. +- [README tests](../tests/test_readme.py) execute the offline Python examples. + [Demo tests](../examples/test_demo.py) are part of the normal suite. + +No paid/live provider calls were used for this validation. Passing these tests does +not establish that a model exists, your account can access it, or a live API accepts +every parameter combination. Model names in mocked examples are illustrative. + +Record/replay uses private SDK request-building hooks plus HTTPX/HTTPX2 send hooks. +The compatibility suites are intended to catch upstream changes to those boundaries. +It buffers full UTF-8 JSON/SSE bodies, so it does not reproduce chunk arrival timing +or partial HTTP disconnects. It rejects unsupported SDK endpoints, non-JSON requests, +and custom HTTP clients overriding send. It is not a network sandbox for WebSockets, +other providers, or unrelated HTTP calls. Concurrent writes to the same cassette +path are unsupported; per-test paths include the full pytest ID's hash. + +Handwritten mocks cover the documented interfaces, not entire SDKs. In particular, +Gemini automatic tool execution and LangChain transport/callback behavior are not +emulated. Token/cost totals are estimates based on configured responses and a static +price table. Assertion helpers, LiteLLM-specific integration, and an environment-gated +live-test marker remain future work. diff --git a/examples/test_demo.py b/examples/test_demo.py new file mode 100644 index 0000000..d3e7b71 --- /dev/null +++ b/examples/test_demo.py @@ -0,0 +1,38 @@ +"""Offline tests shown in the launch demo: pytest examples/test_demo.py -v.""" + +import json + +import pytest +from openai import OpenAI, RateLimitError + + +def test_stream_without_an_api_key(mock_openai): + mock_openai.add_response("Hello, Ada!", stream_chunks=["Hello, ", "Ada!"]) + with OpenAI(api_key="fixture-only") as client: + stream = client.chat.completions.create(model="gpt-4o-mini", messages=[], stream=True) + assert "".join(chunk.choices[0].delta.content or "" for chunk in stream) == "Hello, Ada!" + assert mock_openai.call_count == 1 + + +def test_weather_tool_arguments(mock_openai): + mock_openai.add_response( + "", + tool_calls=[ + {"function": {"name": "weather", "arguments": {"city": "Paris"}}}, + ], + ) + with OpenAI(api_key="fixture-only") as client: + reply = client.chat.completions.create(model="gpt-4o-mini", messages=[]) + tool = reply.choices[0].message.tool_calls[0] + assert tool.function.name == "weather" + assert json.loads(tool.function.arguments) == {"city": "Paris"} + + +def test_recovers_after_rate_limit(mock_openai): + mock_openai.simulate_error("rate_limit", times=1) + mock_openai.add_response("Recovered") + with OpenAI(api_key="fixture-only") as client: + with pytest.raises(RateLimitError): + client.chat.completions.create(model="gpt-4o-mini", messages=[]) + reply = client.chat.completions.create(model="gpt-4o-mini", messages=[]) + assert reply.choices[0].message.content == "Recovered" diff --git a/pyproject.toml b/pyproject.toml index c7cd957..0ff646c 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "pytest-mockllm" -version = "0.3.1" +version = "0.4.0" description = "Pytest plugin for mocking OpenAI, Anthropic, Gemini, and LangChain calls" readme = "README.md" license = "MIT" @@ -49,17 +49,19 @@ dependencies = [ ] [project.optional-dependencies] -openai = ["openai>=1.0.0", "tiktoken>=0.7.0"] -anthropic = ["anthropic>=0.18.0,<1.0"] -google = ["google-generativeai>=0.3.0"] -langchain = ["langchain-core>=0.1.0", "langchain-openai>=0.1.0"] +openai = ["openai>=1.109.1", "tiktoken>=0.7.0", "pydantic>=2.9.0"] +anthropic = ["anthropic>=0.75.0", "pydantic>=2.9.0"] +google = ["google-genai>=1.47.0"] +google-legacy = ["google-genai>=1.47.0", "google-generativeai>=0.3.0"] +langchain = ["langchain-core>=0.3.80", "langchain-openai>=0.3.35"] all = [ - "openai>=1.0.0", - "anthropic>=0.18.0,<1.0", - "google-generativeai>=0.3.0", - "langchain-core>=0.1.0", - "langchain-openai>=0.1.0", + "openai>=1.109.1", + "anthropic>=0.75.0", + "google-genai>=1.47.0", + "langchain-core>=0.3.80", + "langchain-openai>=0.3.35", "tiktoken>=0.7.0", + "pydantic>=2.9.0", ] dev = [ "pytest-cov>=4.0.0", @@ -84,6 +86,9 @@ mockllm = "pytest_mockllm.plugin" include = [ "/src", "/tests", + "/examples", + "/docs", + "/scripts/record_demo.py", "/README.md", "/CHANGELOG.md", "/LICENSE", @@ -93,7 +98,8 @@ include = [ packages = ["src/pytest_mockllm"] [tool.pytest.ini_options] -testpaths = ["tests"] +testpaths = ["tests", "examples"] +asyncio_default_fixture_loop_scope = "function" python_files = ["test_*.py"] python_functions = ["test_*"] addopts = "-v --tb=short" diff --git a/scripts/record_demo.py b/scripts/record_demo.py new file mode 100644 index 0000000..b51757b --- /dev/null +++ b/scripts/record_demo.py @@ -0,0 +1,75 @@ +"""Record actual pytest output as asciinema v2, without an asciinema dependency. + +Run from an environment with pytest-mockllm[openai] installed. The recording uses +the real elapsed times of pipe output (not a terminal capture); no output is invented. +""" + +from __future__ import annotations + +import argparse +import json +import os +import subprocess +import sys +import time +from pathlib import Path + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--output", type=Path, default=Path("docs/demo.cast")) + args = parser.parse_args() + root = Path(__file__).resolve().parents[1] + output = args.output.resolve() + output.parent.mkdir(parents=True, exist_ok=True) + env = {**os.environ, "PYTHONUNBUFFERED": "1", "PYTHONIOENCODING": "utf-8", "COLUMNS": "100"} + for key in ("OPENAI_API_KEY", "ANTHROPIC_API_KEY", "GOOGLE_API_KEY", "GEMINI_API_KEY"): + env.pop(key, None) + command = [ + sys.executable, + "-m", + "pytest", + "examples/test_demo.py", + "-v", + "--color=yes", + "--no-header", + "-o", + "addopts=", + ] + with output.open("w", encoding="utf-8", newline="\n") as recording: + recording.write( + json.dumps( + { + "version": 2, + "width": 100, + "height": 24, + "timestamp": int(time.time()), + "title": "pytest-mockllm: streams, tools, retries", + } + ) + + "\n" + ) + started = time.monotonic() + with subprocess.Popen( + command, + cwd=root, + env=env, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + text=True, + encoding="utf-8", + ) as process: + assert process.stdout is not None + for line in process.stdout: + sys.stdout.write(line) + recording.write( + json.dumps( + [round(time.monotonic() - started, 6), "o", line.replace("\n", "\r\n")] + ) + + "\n" + ) + return process.wait() + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/src/pytest_mockllm/__init__.py b/src/pytest_mockllm/__init__.py index 7df73e6..85b5556 100644 --- a/src/pytest_mockllm/__init__.py +++ b/src/pytest_mockllm/__init__.py @@ -14,7 +14,7 @@ def test_my_chatbot(mock_openai): ``` For more examples and documentation, visit: -https://github.com/pytest-mockllm/pytest-mockllm +https://github.com/godhiraj-code/pytest-mockllm """ from pytest_mockllm.core import MockLLM, MockResponse @@ -23,7 +23,7 @@ def test_my_chatbot(mock_openai): from pytest_mockllm.providers.openai import OpenAIMock from pytest_mockllm.recording import LLMRecorder -__version__ = "0.2.3" +__version__ = "0.4.0" __all__ = [ # Core "MockLLM", diff --git a/src/pytest_mockllm/_recording_http.py b/src/pytest_mockllm/_recording_http.py new file mode 100644 index 0000000..2a7ea65 --- /dev/null +++ b/src/pytest_mockllm/_recording_http.py @@ -0,0 +1,137 @@ +"""SDK-scoped HTTP interception. Provider SDKs still own parsing and SSE decoding.""" + +from __future__ import annotations + +import importlib +from contextvars import ContextVar +from functools import wraps +from typing import Any +from unittest.mock import patch + +from pytest_mockllm.recording import RecordingError + +_provider: ContextVar[str | None] = ContextVar("mockllm_provider", default=None) + + +def _unwrap(error: BaseException) -> None: + """SDKs may wrap transport failures; preserve actionable cassette errors.""" + cause: BaseException | None = error + while cause is not None: + if isinstance(cause, RecordingError): + raise cause from None + cause = cause.__cause__ + + +def install(recorder: Any, stack: Any) -> None: + """Patch only installed SDKs; unrelated HTTP traffic passes through unchanged.""" + + def validate_client(client: Any) -> None: + if not getattr(client._client.send, "_mockllm_intercepted", False): + raise RecordingError( + "Custom HTTP clients overriding send are unsupported; use a custom transport instead" + ) + + for name in ("openai", "anthropic"): + try: + sdk = importlib.import_module(f"{name}._base_client") + except ImportError: + continue + for asynchronous, cls in ((False, sdk.SyncAPIClient), (True, sdk.AsyncAPIClient)): + original = cls.request + + def scoped(original: Any, name: str, asynchronous: bool) -> Any: + @wraps(original) + def sync_call(*args: Any, **kwargs: Any) -> Any: + validate_client(args[0]) + recorder.remember_credentials(args[0]) + token = _provider.set(name) + try: + return original(*args, **kwargs) + except Exception as error: + _unwrap(error) + raise + finally: + _provider.reset(token) + + @wraps(original) + async def async_call(*args: Any, **kwargs: Any) -> Any: + validate_client(args[0]) + recorder.remember_credentials(args[0]) + token = _provider.set(name) + try: + return await original(*args, **kwargs) + except Exception as error: + _unwrap(error) + raise + finally: + _provider.reset(token) + + return async_call if asynchronous else sync_call + + stack.enter_context(patch.object(cls, "request", scoped(original, name, asynchronous))) + + # Validate before the SDK's transport exception handler/retry loop. + # This also catches unsupported endpoints before any network access. + build = cls._build_request + + def checked(build: Any) -> Any: + @wraps(build) + def build_request(*args: Any, **kwargs: Any) -> Any: + request = build(*args, **kwargs) + provider = _provider.get() + if provider is not None: + recorder.prepare(provider, request, consume=False) + return request + + return build_request + + stack.enter_context(patch.object(cls, "_build_request", checked(build))) + + for module_name in ("httpx", "httpx2"): + try: + http = importlib.import_module(module_name) + except ImportError: + continue + for asynchronous, cls in ((False, http.Client), (True, http.AsyncClient)): + original = cls.send + + def intercept(original: Any, http: Any, asynchronous: bool) -> Any: + @wraps(original) + def send(client: Any, request: Any, *args: Any, **kwargs: Any) -> Any: + provider = _provider.get() + if provider is None: + return original(client, request, *args, **kwargs) + prepared, replay = recorder.prepare(provider, request) + if replay is not None: + return recorder.http_response(http, request, replay) + response = original(client, request, *args, **kwargs) + try: + response.read() + recorder.capture(provider, prepared, response) + except BaseException: + response.close() + raise + return response + + @wraps(original) + async def async_send(client: Any, request: Any, *args: Any, **kwargs: Any) -> Any: + provider = _provider.get() + if provider is None: + return await original(client, request, *args, **kwargs) + prepared, replay = recorder.prepare(provider, request) + if replay is not None: + return recorder.http_response(http, request, replay) + response = await original(client, request, *args, **kwargs) + try: + await response.aread() + recorder.capture(provider, prepared, response) + except BaseException: + await response.aclose() + raise + return response + + replacement: Any = async_send if asynchronous else send + replacement._mockllm_intercepted = True + return replacement + + stack.enter_context(patch.object(cls, "send", intercept(original, http, asynchronous))) diff --git a/src/pytest_mockllm/_sdk.py b/src/pytest_mockllm/_sdk.py new file mode 100644 index 0000000..99a8559 --- /dev/null +++ b/src/pytest_mockllm/_sdk.py @@ -0,0 +1,87 @@ +"""Small SDK compatibility boundary for HTTPX and HTTPX2 response types.""" + +from __future__ import annotations + +import importlib +from types import SimpleNamespace +from typing import Any + + +def http_module(provider: str) -> Any: + sdk = importlib.import_module(f"{provider}._base_client") + return getattr(sdk, "httpx2", getattr(sdk, "httpx", None)) + + +def raise_sdk_error(provider: str, error: Any) -> None: + sdk = importlib.import_module(provider) + http = http_module(provider) + request = http.Request("POST", f"https://api.{provider}.com/v1/mock") + if error.error_type == "timeout": + raise sdk.APITimeoutError(request=request) + name, status = { + "rate_limit": ("RateLimitError", 429), + "auth": ("AuthenticationError", 401), + "server": ("InternalServerError", 500), + "overloaded": ("InternalServerError", 529), + "invalid_request": ("BadRequestError", 400), + }.get(error.error_type, ("InternalServerError", 500)) + body = {"error": {"message": error.message}} + response = http.Response(status, request=request, json=body) + raise getattr(sdk, name)(message=error.message, response=response, body=body) + + +def sdk_stream( + provider: str, events: Any, *, asynchronous: bool = False, cast_to: Any = None +) -> Any: + """Use the installed SDK's real SSE parser and stream/context-manager classes.""" + sdk = importlib.import_module(provider) + streaming = importlib.import_module(f"{provider}._streaming") + http = http_module(provider) + from pydantic import TypeAdapter + + def process_data(*, data: Any, cast_to: Any, response: Any) -> Any: + return TypeAdapter(cast_to).validate_python(data) + + client = SimpleNamespace( + _make_sse_decoder=streaming.SSEDecoder, + _process_response_data=process_data, + ) + + def encode(event: Any) -> bytes: + data = event.model_dump_json() + prefix = f"event: {event.type}\n" if provider == "anthropic" else "" + return (prefix + "data: " + data + "\n\n").encode() + + sync_base: Any = http.SyncByteStream + async_base: Any = http.AsyncByteStream + + class SyncBody(sync_base): + def __iter__(self) -> Any: + for event in events: + yield encode(event) + if provider == "openai": + yield b"data: [DONE]\n\n" + + def close(self) -> None: + if hasattr(events, "close"): + events.close() + + class AsyncBody(async_base): + async def __aiter__(self) -> Any: + async for event in events: + yield encode(event) + if provider == "openai": + yield b"data: [DONE]\n\n" + + async def aclose(self) -> None: + if hasattr(events, "aclose"): + await events.aclose() + + response = http.Response( + 200, + headers={"content-type": "text/event-stream"}, + request=http.Request("POST", "https://mock.invalid"), + stream=AsyncBody() if asynchronous else SyncBody(), + ) + cls = sdk.AsyncStream if asynchronous else sdk.Stream + return cls(cast_to=cast_to, response=response, client=client) diff --git a/src/pytest_mockllm/core.py b/src/pytest_mockllm/core.py index 9d157db..7d3be6d 100644 --- a/src/pytest_mockllm/core.py +++ b/src/pytest_mockllm/core.py @@ -98,8 +98,9 @@ def count_tokens(text: str, model: str = "gpt-4o") -> int: """ Count tokens in a string for a specific model. - Uses real tokenizers (tiktoken, anthropic) if installed, otherwise - falls back to a rough character-based estimate. + Uses an already-loaded tiktoken encoder when available. Otherwise it + returns a character estimate; implicit counting must never download data. + Applications wanting exact counts can explicitly preload a tiktoken encoder. """ if not text: return 0 @@ -107,13 +108,16 @@ def count_tokens(text: str, model: str = "gpt-4o") -> int: # OpenAI models if any(x in model.lower() for x in ["gpt-", "text-embedding-", "o1-"]): try: - import tiktoken + import tiktoken.model + import tiktoken.registry try: - encoding = tiktoken.encoding_for_model(model) + name = tiktoken.model.encoding_name_for_model(model) except KeyError: - encoding = tiktoken.get_encoding("cl100k_base") - return len(encoding.encode(text)) + name = "cl100k_base" + encoding = tiktoken.registry.ENCODINGS.get(name) + if encoding is not None: + return len(encoding.encode(text)) except ImportError: pass diff --git a/src/pytest_mockllm/fixtures.py b/src/pytest_mockllm/fixtures.py index cb28148..d49f7d4 100644 --- a/src/pytest_mockllm/fixtures.py +++ b/src/pytest_mockllm/fixtures.py @@ -13,6 +13,7 @@ import pytest +from pytest_mockllm.core import MockLLM from pytest_mockllm.providers.anthropic import AnthropicMock from pytest_mockllm.providers.gemini import GeminiMock from pytest_mockllm.providers.openai import OpenAIMock @@ -21,6 +22,8 @@ if TYPE_CHECKING: from _pytest.fixtures import FixtureRequest + from pytest_mockllm.integrations.langchain import LangChainMock + def _strict_enabled(request: FixtureRequest) -> bool: """Resolve strict mode from the CLI flag or pytest configuration.""" @@ -104,16 +107,16 @@ def mock_gemini(request: FixtureRequest) -> Generator[GeminiMock, None, None]: """ Mock Google Gemini API for testing. - Automatically patches the Google GenerativeAI client to return mock responses. + Automatically patches the Google GenAI client to return mock responses. No API key or network access required. Example: >>> def test_gemini_bot(mock_gemini): ... mock_gemini.add_response("Here's what I found...") ... - ... import google.generativeai as genai - ... model = genai.GenerativeModel("gemini-1.5-pro") - ... response = model.generate_content("Tell me about AI") + ... from google import genai + ... client = genai.Client(api_key="fixture-only") + ... response = client.models.generate_content(model="gemini-2.5-flash", contents="Tell me about AI") ... ... assert "found" in response.text.lower() """ @@ -123,7 +126,7 @@ def mock_gemini(request: FixtureRequest) -> Generator[GeminiMock, None, None]: @pytest.fixture -def mock_llm(request: FixtureRequest): # type: ignore[no-untyped-def] +def mock_llm(request: FixtureRequest) -> Generator[MockLLM, None, None]: """ Universal LLM mock - defaults to OpenAI. @@ -160,7 +163,7 @@ def mock_llm(request: FixtureRequest): # type: ignore[no-untyped-def] @pytest.fixture -def mock_langchain(): +def mock_langchain(request: FixtureRequest) -> Generator[LangChainMock, None, None]: """ Mock LangChain LLM and ChatModel for testing. @@ -184,39 +187,39 @@ def mock_langchain(): from pytest_mockllm.integrations.langchain import LangChainMock with LangChainMock() as mock: + mock.set_strict_mode(_strict_enabled(request)) yield mock @pytest.fixture def llm_recorder(request: FixtureRequest) -> Generator[LLMRecorder, None, None]: - """ - Reserved recording/replay fixture. - - Provider interception is not implemented for this fixture, so record, replay, and auto - modes raise before test code can reach a provider. Use a provider fixture such as - ``mock_openai`` for deterministic offline tests. - """ - # Get cassette directory from config or use default + """Replay offline by default; opt into live recording with --llm-record=once.""" + import hashlib + import re + + conflicting = set(request.fixturenames) & { + "mock_openai", + "mock_anthropic", + "mock_llm", + "mock_gemini", + "mock_langchain", + } + if conflicting: + raise pytest.UsageError("Use llm_recorder separately from provider mock fixtures") cassette_dir = request.config.getoption("--llm-cassette-dir") or request.config.getini( "llm_cassette_dir" ) - record_mode = request.config.getoption("--llm-record", default=False) - - # Use test name as cassette name - test_name = request.node.name - cassette_path = Path(cassette_dir) / f"{test_name}.yaml" - - # Check for llm_record or llm_replay markers + # Include file, class and parameter IDs; rootpath makes invocation location irrelevant. + nodeid = request.node.nodeid + name = re.sub(r"[^a-zA-Z0-9_-]", "_", request.node.name)[:80] + digest = hashlib.sha256(nodeid.encode()).hexdigest()[:16] + cassette_path = Path(request.config.rootpath) / cassette_dir / f"{name}-{digest}.yaml" record_marker = request.node.get_closest_marker("llm_record") replay_marker = request.node.get_closest_marker("llm_replay") - - mode = "auto" - if record_marker or record_mode: - mode = "record" - elif replay_marker: - mode = "replay" - - recorder = LLMRecorder(cassette_path=cassette_path, mode=mode) - - with recorder: + mode = request.config.getoption("--llm-record") + if record_marker and "mode" in record_marker.kwargs: + mode = record_marker.kwargs["mode"] + if replay_marker: + mode = "none" + with LLMRecorder(cassette_path=cassette_path, record_mode=mode) as recorder: yield recorder diff --git a/src/pytest_mockllm/plugin.py b/src/pytest_mockllm/plugin.py index 5874fbe..2b224f6 100644 --- a/src/pytest_mockllm/plugin.py +++ b/src/pytest_mockllm/plugin.py @@ -29,9 +29,12 @@ def pytest_addoption(parser: Parser) -> None: group.addoption( "--llm-record", - action="store_true", - default=False, - help="Reserved for recording support; currently fails closed", + action="store", + nargs="?", + const="all", + default="none", + choices=("none", "once", "new_episodes", "all"), + help="Cassette mode: none (offline default), once, new_episodes, all; bare flag means all", ) group.addoption( @@ -48,6 +51,20 @@ def pytest_addoption(parser: Parser) -> None: help="Fail tests if no mock response is configured (strict mode)", ) + group.addoption( + "--llm-stats", + action="store_true", + default=False, + help="Show estimated token usage and cost savings (off by default)", + ) + group.addoption( + "--no-llm-stats", + action="store_true", + default=False, + help="Hide token/cost statistics even when llm_stats is enabled in configuration", + ) + parser.addini("llm_stats", "Show estimated token/cost statistics", type="bool", default=False) + parser.addini( "llm_cassette_dir", "Directory to store LLM response cassettes", @@ -70,14 +87,21 @@ def pytest_configure(config: Config) -> None: ) config.addinivalue_line( "markers", - "llm_record: Reserved for recording support; currently fails closed", + "llm_record(mode=None): Activate recording/replay; CLI mode defaults to offline", ) config.addinivalue_line( "markers", - "llm_replay: Reserved for replay support; currently fails closed", + "llm_replay: Activate offline cassette replay (overrides recording CLI flags)", ) +def pytest_runtest_setup(item: Any) -> None: + """Markers activate the recorder without requiring a fixture parameter.""" + if item.get_closest_marker("llm_record") or item.get_closest_marker("llm_replay"): + if "llm_recorder" not in item.fixturenames: + item.fixturenames.insert(0, "llm_recorder") + + def pytest_sessionstart(session: Any) -> None: """Initialize global statistics.""" from pytest_mockllm.stats import GLOBAL_STATS @@ -89,7 +113,8 @@ def pytest_terminal_summary(terminalreporter: Any, exitstatus: int, config: Any) """Display LLM usage and cost savings summary.""" from pytest_mockllm.stats import GLOBAL_STATS - if GLOBAL_STATS.total_calls == 0: + enabled = config.getoption("--llm-stats") or config.getini("llm_stats") + if not enabled or config.getoption("--no-llm-stats") or GLOBAL_STATS.total_calls == 0: return terminalreporter.section("pytest-mockllm stats") @@ -103,8 +128,6 @@ def pytest_terminal_summary(terminalreporter: Any, exitstatus: int, config: Any) # Detailed rows (sorted by calls) sorted_models = sorted(GLOBAL_STATS.model_counts.items(), key=lambda x: x[1], reverse=True) for model, count in sorted_models: - # Note: We don't currently track per-model tokens in GLOBAL_STATS easily, - # but we can improve GLOBAL_STATS later. For now, show what we have. terminalreporter.write_line(f"{model:<30} {count:<8}") terminalreporter.write_line("-" * 75) @@ -116,7 +139,8 @@ def pytest_terminal_summary(terminalreporter: Any, exitstatus: int, config: Any) ) # Estimated cost avoided by configured mock responses - cost_str = f"${GLOBAL_STATS.total_cost_saved:,.4f}" + cost = GLOBAL_STATS.total_cost_saved + cost_str = "<$0.0001" if 0 < cost < 0.0001 else f"${cost:,.4f}" terminalreporter.write_line("") terminalreporter.write_sep("=", f"Total Estimated Cost Saved: {cost_str}", bold=True, blue=True) terminalreporter.write_line("") diff --git a/src/pytest_mockllm/providers/_gemini_legacy.py b/src/pytest_mockllm/providers/_gemini_legacy.py new file mode 100644 index 0000000..11cb23b --- /dev/null +++ b/src/pytest_mockllm/providers/_gemini_legacy.py @@ -0,0 +1,293 @@ +""" +Google Gemini API mock for pytest-mockllm. + +Provides comprehensive mocking for the Google GenerativeAI SDK including: +- Content generation (sync, async, streaming) +- Chat sessions +- Embeddings +""" + +from __future__ import annotations + +from collections.abc import AsyncIterator, Iterator +from contextvars import ContextVar +from typing import Any +from unittest.mock import MagicMock, patch + +from pytest_mockllm.core import MockError, MockLLM, MockResponse + +_LEGACY_ACTIVE: ContextVar[bool] = ContextVar("mockllm_legacy_gemini", default=False) + + +class LegacyGeminiMock(MockLLM): + """ + Mock for Google Generative AI (Gemini) SDK. + + Supports: + - model.generate_content() + - model.generate_content() with streaming + - Chat sessions + - Embeddings + + Example: + >>> def test_gemini_bot(mock_gemini): + ... mock_gemini.add_response("Here's what I found...") + ... + ... import google.generativeai as genai + ... model = genai.GenerativeModel("gemini-1.5-pro") + ... response = model.generate_content("Tell me about AI") + ... assert "found" in response.text.lower() + """ + + def __init__(self) -> None: + super().__init__() + self._default_model: str = "gemini-1.5-pro" + + def _legacy_next_response(self, **kwargs: Any) -> MockResponse: + token = _LEGACY_ACTIVE.set(True) + try: + return self._get_next_response(**kwargs) + finally: + _LEGACY_ACTIVE.reset(token) + + def _raise_provider_error(self, error: MockError) -> None: + """Raise a Gemini-style API error.""" + try: + from google.api_core import exceptions + + error_classes = { + "rate_limit": exceptions.ResourceExhausted, + "auth": exceptions.Unauthenticated, + "timeout": exceptions.DeadlineExceeded, + "server": exceptions.InternalServerError, + "invalid_request": exceptions.InvalidArgument, + } + + error_class = error_classes.get(error.error_type, exceptions.GoogleAPIError) + raise error_class(error.message) + except ImportError: + raise RuntimeError(f"Gemini API Error ({error.error_type}): {error.message}") from None + + def _create_generation(self, **kwargs: Any) -> Any: + """Create a mock content generation response.""" + self._record_call(type="generate_content", **kwargs) + model = kwargs.get("model", self._default_model) + response = self._legacy_next_response(model=model) + self._simulate_delay(self._get_delay_ms(response)) + return self._build_generation_response(response) + + def _build_generation_response(self, response: MockResponse) -> Any: + """Build a Gemini-style GenerateContentResponse.""" + try: + from google.ai.generativelanguage_v1beta import ( + Candidate, + Content, + Part, + ) + from google.ai.generativelanguage_v1beta import ( + GenerateContentResponse as ProtoResponse, + ) + from google.generativeai.types import GenerateContentResponse + + # Build proto-style response + proto = ProtoResponse( + candidates=[ + Candidate( + content=Content( + parts=[Part(text=response.content)], + role="model", + ), + finish_reason=1, # STOP + index=0, + ) + ], + usage_metadata={ + "prompt_token_count": response.token_usage.prompt_tokens + if response.token_usage + else 10, + "candidates_token_count": response.token_usage.completion_tokens + if response.token_usage + else 50, + "total_token_count": response.token_usage.total_tokens + if response.token_usage + else 60, + }, + ) + + return GenerateContentResponse.from_response(proto) + except ImportError: + return self._build_mock_generation(response) + + def _build_mock_generation(self, response: MockResponse) -> MagicMock: + """Build a MagicMock generation when google-generativeai not installed.""" + mock = MagicMock() + mock.text = response.content + + # Candidates + candidate = MagicMock() + candidate.content.parts = [MagicMock(text=response.content)] + candidate.content.role = "model" + candidate.finish_reason = "STOP" + mock.candidates = [candidate] + + # Usage metadata + mock.usage_metadata = MagicMock() + mock.usage_metadata.prompt_token_count = ( + response.token_usage.prompt_tokens if response.token_usage else 10 + ) + mock.usage_metadata.candidates_token_count = ( + response.token_usage.completion_tokens if response.token_usage else 50 + ) + mock.usage_metadata.total_token_count = ( + response.token_usage.total_tokens if response.token_usage else 60 + ) + + return mock + + async def _create_async_generation(self, **kwargs: Any) -> Any: + """Create a mock async content generation response.""" + self._record_call(type="generate_content", **kwargs) + model = kwargs.get("model", self._default_model) + response = self._legacy_next_response(model=model) + await self._simulate_delay_async(self._get_delay_ms(response)) + return self._build_generation_response(response) + + async def _create_async_streaming_generation(self, **kwargs: Any) -> AsyncIterator[Any]: + """Create an async streaming content generation response.""" + self._record_call(type="generate_content", stream=True, **kwargs) + model = kwargs.get("model", self._default_model) + response = self._legacy_next_response(model=model) + await self._simulate_delay_async(self._get_delay_ms(response)) + + # Split content into chunks + if response.stream_chunks: + chunks = response.stream_chunks + else: + words = response.content.split() + chunks = [word + " " for word in words[:-1]] + [words[-1]] if words else [""] + + for i, chunk_content in enumerate(chunks): + is_last = i == len(chunks) - 1 + yield self._build_stream_chunk(chunk_content, is_last, response) + + def _create_streaming_generation(self, **kwargs: Any) -> Iterator[Any]: + """Create a streaming content generation response.""" + self._record_call(type="generate_content", stream=True, **kwargs) + model = kwargs.get("model", self._default_model) + response = self._legacy_next_response(model=model) + self._simulate_delay(self._get_delay_ms(response)) + + # Split content into chunks + if response.stream_chunks: + chunks = response.stream_chunks + else: + words = response.content.split() + chunks = [word + " " for word in words[:-1]] + [words[-1]] if words else [""] + + for i, chunk_content in enumerate(chunks): + is_last = i == len(chunks) - 1 + yield self._build_stream_chunk(chunk_content, is_last, response) + + def _build_stream_chunk( + self, + content: str, + is_last: bool, + response: MockResponse, + ) -> Any: + """Build a streaming chunk.""" + mock = MagicMock() + mock.text = content + + candidate = MagicMock() + candidate.content.parts = [MagicMock(text=content)] + candidate.content.role = "model" + candidate.finish_reason = "STOP" if is_last else None + mock.candidates = [candidate] + + return mock + + def _create_chat_session(self) -> MagicMock: + """Create a mock chat session.""" + session = MagicMock() + + def send_message(content: str, **kwargs: Any) -> Any: + self._record_call(type="chat.send_message", content=content, **kwargs) + response = self._legacy_next_response() + return self._build_mock_generation(response) + + session.send_message = send_message + session.history = [] + + return session + + def __enter__(self) -> LegacyGeminiMock: + """Start mocking Gemini API calls.""" + mock_module = MagicMock() + + # Create mock GenerativeModel class + def mock_model_class(model_name: str = "gemini-1.5-pro", **kwargs: Any) -> MagicMock: + import warnings + + warnings.warn( + "google.generativeai support is deprecated; migrate to google.genai.Client", + DeprecationWarning, + stacklevel=2, + ) + model = MagicMock() + model.model_name = model_name + + def generate_content(*args: Any, **kw: Any) -> Any: + kw.setdefault("contents", args[0] if args else None) + kw.setdefault("model", model_name) + if kw.pop("stream", False): + return self._create_streaming_generation(**kw) + return self._create_generation(**kw) + + async def generate_content_async(*args: Any, **kw: Any) -> Any: + kw.setdefault("contents", args[0] if args else None) + kw.setdefault("model", model_name) + if kw.pop("stream", False): + return self._create_async_streaming_generation(**kw) + return await self._create_async_generation(**kw) + + model.generate_content = generate_content + model.generate_content_async = generate_content_async + model.start_chat = lambda **kw: self._create_chat_session() + + async def start_chat_async(**kw: Any) -> Any: + return self._create_chat_session() + + model.start_chat_async = start_chat_async + model.count_tokens = lambda content: MagicMock( + total_tokens=len(str(content).split()) * 1.3 + ) + + async def count_tokens_async(content: Any) -> Any: + return MagicMock(total_tokens=len(str(content).split()) * 1.3) + + model.count_tokens_async = count_tokens_async + + return model + + mock_module.GenerativeModel = mock_model_class + mock_module.configure = MagicMock() + + try: + import google.generativeai as genai + except ImportError: + # Do not inject an obsolete SDK into sys.modules when it is absent. + pass + else: + # Patch the real module object so imports performed before fixture activation + # observe the replacement too. + model_patcher = patch.object(genai, "GenerativeModel", mock_model_class) + self._patches.append(model_patcher) + model_patcher.start() + + self._mock_module = mock_module + return self + + @property + def module(self) -> MagicMock: + """Access the mock module directly for advanced usage.""" + return self._mock_module diff --git a/src/pytest_mockllm/providers/_responses.py b/src/pytest_mockllm/providers/_responses.py new file mode 100644 index 0000000..0be9dbb --- /dev/null +++ b/src/pytest_mockllm/providers/_responses.py @@ -0,0 +1,97 @@ +"""OpenAI Responses SSE events for text and function-call outputs.""" + +from __future__ import annotations + +from collections.abc import Iterator +from typing import Any + + +def response_events(completed: Any, configured: Any) -> Iterator[Any]: + from openai.types import responses as types + + sequence = 0 + + def event(cls: str, kind: str, **fields: Any) -> Any: + nonlocal sequence + result = getattr(types, cls)(type=kind, sequence_number=sequence, **fields) + sequence += 1 + return result + + yield event( + "ResponseCreatedEvent", + "response.created", + response=completed.model_copy( + update={"output": [], "status": "in_progress", "usage": None} + ), + ) + for index, item in enumerate(completed.output): + is_tool = item.type == "function_call" + updates: dict[str, Any] = {"status": "in_progress"} + if is_tool: + updates["arguments"] = "" + else: + updates["content"] = [] + initial = item.model_copy(update=updates) + yield event( + "ResponseOutputItemAddedEvent", + "response.output_item.added", + output_index=index, + item=initial, + ) + common = {"output_index": index, "item_id": item.id} + if is_tool: + fragments = configured.tool_call_chunks if index == 0 else None + for fragment in fragments or [item.arguments]: + yield event( + "ResponseFunctionCallArgumentsDeltaEvent", + "response.function_call_arguments.delta", + delta=fragment, + **common, + ) + yield event( + "ResponseFunctionCallArgumentsDoneEvent", + "response.function_call_arguments.done", + arguments=item.arguments, + name=item.name, + **common, + ) + else: + part = item.content[0] + yield event( + "ResponseContentPartAddedEvent", + "response.content_part.added", + content_index=0, + part=part.model_copy(update={"text": ""}), + **common, + ) + for fragment in configured.stream_chunks or [part.text]: + yield event( + "ResponseTextDeltaEvent", + "response.output_text.delta", + content_index=0, + delta=fragment, + logprobs=[], + **common, + ) + yield event( + "ResponseTextDoneEvent", + "response.output_text.done", + content_index=0, + text=part.text, + logprobs=[], + **common, + ) + yield event( + "ResponseContentPartDoneEvent", + "response.content_part.done", + content_index=0, + part=part, + **common, + ) + yield event( + "ResponseOutputItemDoneEvent", + "response.output_item.done", + output_index=index, + item=item, + ) + yield event("ResponseCompletedEvent", "response.completed", response=completed) diff --git a/src/pytest_mockllm/providers/anthropic.py b/src/pytest_mockllm/providers/anthropic.py index 1603ec1..71a6c06 100644 --- a/src/pytest_mockllm/providers/anthropic.py +++ b/src/pytest_mockllm/providers/anthropic.py @@ -9,6 +9,8 @@ from __future__ import annotations +import inspect +import json import uuid from collections.abc import AsyncIterator, Iterator from typing import Any, cast @@ -57,52 +59,13 @@ def simulate_stream_disconnect( return self def _raise_provider_error(self, error: MockError) -> None: - """Raise an Anthropic-style API error.""" - try: - from anthropic import ( - APIError, - APITimeoutError, - AuthenticationError, - BadRequestError, - InternalServerError, - RateLimitError, - ) - - error_classes = { - "rate_limit": RateLimitError, - "auth": AuthenticationError, - "timeout": APITimeoutError, - "server": APIError, - "overloaded": InternalServerError, - "invalid_request": BadRequestError, - } - - error_class = error_classes.get(error.error_type, APIError) - - import httpx - - status_code = { - "rate_limit": 429, - "auth": 401, - "timeout": 408, - "server": 500, - "overloaded": 529, - "invalid_request": 400, - }.get(error.error_type, 500) - mock_response = httpx.Response( - status_code, - request=httpx.Request("POST", "https://api.anthropic.com/v1/messages"), - json={"error": {"message": error.message}}, - ) + from pytest_mockllm._sdk import raise_sdk_error - raise error_class( - message=error.message, - response=mock_response, - body={"error": {"message": error.message}}, - ) + try: + raise_sdk_error("anthropic", error) except ImportError: raise RuntimeError( - f"Anthropic API Error ({error.error_type}): {error.message}" + f"anthropic API Error ({error.error_type}): {error.message}" ) from None def _create_message(self, **kwargs: Any) -> Any: @@ -223,11 +186,32 @@ def _stream_events(self, response: MockResponse, model: str) -> Iterator[Any]: message.usage = usage yield MessageStartEvent(type="message_start", message=message) - if response.tool_calls and response.tool_call_chunks: - tool_call = response.tool_calls[0] + block_index = 0 + emitted = 0 + if response.content or not response.tool_calls: yield ContentBlockStartEvent( type="content_block_start", - index=0, + index=block_index, + content_block=TextBlock(type="text", text=""), + ) + chunks = response.stream_chunks + if chunks is None: + words = response.content.split() + chunks = [word + " " for word in words[:-1]] + [words[-1]] if words else [""] + for chunk in chunks: + self._maybe_disconnect_stream(emitted) + emitted += 1 + yield ContentBlockDeltaEvent( + type="content_block_delta", + index=block_index, + delta=TextDelta(type="text_delta", text=chunk), + ) + yield ContentBlockStopEvent(type="content_block_stop", index=block_index) + block_index += 1 + for tool_index, tool_call in enumerate(response.tool_calls or []): + yield ContentBlockStartEvent( + type="content_block_start", + index=block_index, content_block=ToolUseBlock( type="tool_use", id=tool_call.get("id", f"toolu_{uuid.uuid4().hex[:8]}"), @@ -235,43 +219,25 @@ def _stream_events(self, response: MockResponse, model: str) -> Iterator[Any]: input={}, ), ) - chunks = response.tool_call_chunks - for index, chunk in enumerate(chunks): - self._maybe_disconnect_stream(index) + arguments = tool_call["function"].get("arguments", {}) + encoded = arguments if isinstance(arguments, str) else json.dumps(arguments) + fragments = response.tool_call_chunks if tool_index == 0 else None + for chunk in fragments or [encoded]: + self._maybe_disconnect_stream(emitted) + emitted += 1 yield ContentBlockDeltaEvent( type="content_block_delta", - index=0, + index=block_index, delta=InputJSONDelta(type="input_json_delta", partial_json=chunk), ) - stop_reason = "tool_use" - else: - yield ContentBlockStartEvent( - type="content_block_start", - index=0, - content_block=TextBlock(type="text", text=""), - ) - if response.stream_chunks: - chunks = response.stream_chunks - else: - words = response.content.split() - chunks = [word + " " for word in words[:-1]] + [words[-1]] if words else [""] - for index, chunk in enumerate(chunks): - self._maybe_disconnect_stream(index) - yield ContentBlockDeltaEvent( - type="content_block_delta", - index=0, - delta=TextDelta(type="text_delta", text=chunk), - ) - stop_reason = "end_turn" - - yield ContentBlockStopEvent(type="content_block_stop", index=0) + yield ContentBlockStopEvent(type="content_block_stop", index=block_index) + block_index += 1 + stop_reason = "tool_use" if response.tool_calls else "end_turn" yield MessageDeltaEvent( type="message_delta", delta=Delta(stop_reason=cast(Any, stop_reason), stop_sequence=None), usage=MessageDeltaUsage( - output_tokens=response.token_usage.completion_tokens - if response.token_usage - else 50 + output_tokens=response.token_usage.completion_tokens if response.token_usage else 50 ), ) yield MessageStopEvent(type="message_stop") @@ -282,7 +248,9 @@ def _maybe_disconnect_stream(self, emitted_chunks: int) -> None: return if self._calls: self._calls[-1]["outcome"] = "stream_disconnect" - import httpx + from pytest_mockllm._sdk import http_module + + httpx = http_module("anthropic") from anthropic import APIConnectionError raise APIConnectionError( @@ -296,6 +264,7 @@ async def _create_async_streaming_message(self, **kwargs: Any) -> AsyncIterator[ self._record_call(type="messages.create", **request) model = kwargs.get("model", self._default_model) response = self._get_next_response(model=model) + await self._simulate_delay_async(self._get_delay_ms(response)) for event in self._stream_events(response, model): yield event @@ -305,6 +274,7 @@ def _create_streaming_message(self, **kwargs: Any) -> Iterator[Any]: self._record_call(type="messages.create", **request) model = kwargs.get("model", self._default_model) response = self._get_next_response(model=model) + self._simulate_delay(self._get_delay_ms(response)) yield from self._stream_events(response, model) def __enter__(self) -> AnthropicMock: @@ -313,7 +283,15 @@ def __enter__(self) -> AnthropicMock: def create_message(*args: Any, **kwargs: Any) -> Any: if kwargs.get("stream", False): - return self._create_streaming_message(**kwargs) + from anthropic.types import RawMessageStreamEvent + + from pytest_mockllm._sdk import sdk_stream + + return sdk_stream( + "anthropic", + self._create_streaming_message(**kwargs), + cast_to=RawMessageStreamEvent, + ) return self._create_message(**kwargs) mock_client.messages.create = create_message @@ -324,13 +302,49 @@ def create_message(*args: Any, **kwargs: Any) -> Any: async def create_async_message(*args: Any, **kwargs: Any) -> Any: if kwargs.get("stream", False): - return self._create_async_streaming_message(**kwargs) + from anthropic.types import RawMessageStreamEvent + + from pytest_mockllm._sdk import sdk_stream + + return sdk_stream( + "anthropic", + self._create_async_streaming_message(**kwargs), + asynchronous=True, + cast_to=RawMessageStreamEvent, + ) return await self._create_async_message(**kwargs) async_mock_client.messages.create = create_async_message async_mock_client.beta = MagicMock() async_mock_client.beta.messages = async_mock_client.messages + def stream_message(*args: Any, **kwargs: Any) -> Any: + from anthropic import NOT_GIVEN + from anthropic.lib.streaming import MessageStreamManager + + output_format = kwargs.pop("output_format", NOT_GIVEN) + options = ( + {"output_format": output_format} + if "output_format" in inspect.signature(MessageStreamManager).parameters + else {} + ) + return MessageStreamManager(lambda: create_message(stream=True, **kwargs), **options) + + def async_stream_message(*args: Any, **kwargs: Any) -> Any: + from anthropic import NOT_GIVEN + from anthropic.lib.streaming import AsyncMessageStreamManager + + output_format = kwargs.pop("output_format", NOT_GIVEN) + options = ( + {"output_format": output_format} + if "output_format" in inspect.signature(AsyncMessageStreamManager).parameters + else {} + ) + return AsyncMessageStreamManager(create_async_message(stream=True, **kwargs), **options) + + mock_client.messages.stream = stream_message + async_mock_client.messages.stream = async_stream_message + def blocked_request(*args: Any, **kwargs: Any) -> Any: raise RuntimeError("Anthropic network blocked by active mock_anthropic fixture") @@ -340,6 +354,8 @@ async def blocked_async_request(*args: Any, **kwargs: Any) -> Any: targets = [ ("anthropic.resources.messages.Messages.create", create_message), ("anthropic.resources.messages.AsyncMessages.create", create_async_message), + ("anthropic.resources.messages.Messages.stream", stream_message), + ("anthropic.resources.messages.AsyncMessages.stream", async_stream_message), ("anthropic._base_client.SyncAPIClient.request", blocked_request), ("anthropic._base_client.AsyncAPIClient.request", blocked_async_request), ] diff --git a/src/pytest_mockllm/providers/gemini.py b/src/pytest_mockllm/providers/gemini.py index 793b86c..95e579f 100644 --- a/src/pytest_mockllm/providers/gemini.py +++ b/src/pytest_mockllm/providers/gemini.py @@ -1,275 +1,165 @@ -""" -Google Gemini API mock for pytest-mockllm. - -Provides comprehensive mocking for the Google GenerativeAI SDK including: -- Content generation (sync, async, streaming) -- Chat sessions -- Embeddings -""" +"""Fixture-scoped google-genai content generation, streaming and chat mocking.""" from __future__ import annotations +import json from collections.abc import AsyncIterator, Iterator +from dataclasses import replace from typing import Any -from unittest.mock import MagicMock, patch - -from pytest_mockllm.core import MockError, MockLLM, MockResponse +from unittest.mock import patch +from pytest_mockllm.core import MockError, MockResponse +from pytest_mockllm.providers._gemini_legacy import _LEGACY_ACTIVE, LegacyGeminiMock -class GeminiMock(MockLLM): - """ - Mock for Google Generative AI (Gemini) SDK. - Supports: - - model.generate_content() - - model.generate_content() with streaming - - Chat sessions - - Embeddings +class GeminiMock(LegacyGeminiMock): + """Mock google.genai.Client and its async models and chat interfaces. - Example: - >>> def test_gemini_bot(mock_gemini): - ... mock_gemini.add_response("Here's what I found...") - ... - ... import google.generativeai as genai - ... model = genai.GenerativeModel("gemini-1.5-pro") - ... response = model.generate_content("Tell me about AI") - ... assert "found" in response.text.lower() + The optional deprecated google.generativeai adapter is retained for one + migration cycle. New responses use real google.genai.types models. """ - def __init__(self) -> None: - super().__init__() - self._default_model: str = "gemini-1.5-pro" - def _raise_provider_error(self, error: MockError) -> None: - """Raise a Gemini-style API error.""" - try: - from google.api_core import exceptions - - error_classes = { - "rate_limit": exceptions.ResourceExhausted, - "auth": exceptions.Unauthenticated, - "timeout": exceptions.DeadlineExceeded, - "server": exceptions.InternalServerError, - "invalid_request": exceptions.InvalidArgument, - } - - error_class = error_classes.get(error.error_type, exceptions.GoogleAPIError) - raise error_class(error.message) - except ImportError: - raise RuntimeError(f"Gemini API Error ({error.error_type}): {error.message}") from None - - def _create_generation(self, **kwargs: Any) -> Any: - """Create a mock content generation response.""" - self._record_call(type="generate_content", **kwargs) - model = kwargs.get("model", self._default_model) - response = self._get_next_response(model=model) - self._simulate_delay(self._get_delay_ms(response)) - return self._build_generation_response(response) - - def _build_generation_response(self, response: MockResponse) -> Any: - """Build a Gemini-style GenerateContentResponse.""" - try: - from google.ai.generativelanguage_v1beta import ( - Candidate, - Content, - Part, - ) - from google.ai.generativelanguage_v1beta import ( - GenerateContentResponse as ProtoResponse, - ) - from google.generativeai.types import GenerateContentResponse - - # Build proto-style response - proto = ProtoResponse( - candidates=[ - Candidate( - content=Content( - parts=[Part(text=response.content)], - role="model", - ), - finish_reason=1, # STOP - index=0, - ) - ], - usage_metadata={ - "prompt_token_count": response.token_usage.prompt_tokens - if response.token_usage - else 10, - "candidates_token_count": response.token_usage.completion_tokens - if response.token_usage - else 50, - "total_token_count": response.token_usage.total_tokens - if response.token_usage - else 60, - }, + if _LEGACY_ACTIVE.get(): + return super()._raise_provider_error(error) + from google.genai import errors + + code, status = { + "rate_limit": (429, "RESOURCE_EXHAUSTED"), + "auth": (401, "UNAUTHENTICATED"), + "timeout": (504, "DEADLINE_EXCEEDED"), + "server": (500, "INTERNAL"), + "invalid_request": (400, "INVALID_ARGUMENT"), + }.get(error.error_type, (500, "INTERNAL")) + cls = errors.ClientError if code < 500 else errors.ServerError + raise cls(code, {"error": {"code": code, "status": status, "message": error.message}}) + + def _build_genai_response(self, response: MockResponse, model: str, config: Any = None) -> Any: + from google.genai import types + + parts = [types.Part(text=response.content)] if response.content else [] + for tool in response.tool_calls or []: + function = tool["function"] + arguments = function.get("arguments", {}) + if isinstance(arguments, str): + arguments = json.loads(arguments) + parts.append( + types.Part(function_call=types.FunctionCall(name=function["name"], args=arguments)) ) - - return GenerateContentResponse.from_response(proto) - except ImportError: - return self._build_mock_generation(response) - - def _build_mock_generation(self, response: MockResponse) -> MagicMock: - """Build a MagicMock generation when google-generativeai not installed.""" - mock = MagicMock() - mock.text = response.content - - # Candidates - candidate = MagicMock() - candidate.content.parts = [MagicMock(text=response.content)] - candidate.content.role = "model" - candidate.finish_reason = "STOP" - mock.candidates = [candidate] - - # Usage metadata - mock.usage_metadata = MagicMock() - mock.usage_metadata.prompt_token_count = ( - response.token_usage.prompt_tokens if response.token_usage else 10 + if not parts: + parts = [types.Part(text="")] + usage = response.token_usage + result = types.GenerateContentResponse( + candidates=[ + types.Candidate( + content=types.Content(role="model", parts=parts), + finish_reason=types.FinishReason.STOP, + index=0, + ) + ], + model_version=model, + usage_metadata=types.GenerateContentResponseUsageMetadata( + prompt_token_count=usage.prompt_tokens if usage else 0, + candidates_token_count=usage.completion_tokens if usage else 0, + total_token_count=usage.total_tokens if usage else 0, + ), ) - mock.usage_metadata.candidates_token_count = ( - response.token_usage.completion_tokens if response.token_usage else 50 + schema = ( + config.get("response_schema") + if isinstance(config, dict) + else getattr(config, "response_schema", None) ) - mock.usage_metadata.total_token_count = ( - response.token_usage.total_tokens if response.token_usage else 60 + if schema is not None: + result.parsed = ( + schema.model_validate_json(response.content) + if hasattr(schema, "model_validate_json") + else json.loads(response.content) + ) + return result + + def _genai_response( + self, model: str, contents: Any, config: Any, stream: bool = False + ) -> MockResponse: + self._record_call( + type="generate_content_stream" if stream else "generate_content", + model=model, + contents=contents, + config=config, + stream=stream, ) + return self._get_next_response(model=model) - return mock - - async def _create_async_generation(self, **kwargs: Any) -> Any: - """Create a mock async content generation response.""" - self._record_call(type="generate_content", **kwargs) - model = kwargs.get("model", self._default_model) - response = self._get_next_response(model=model) - await self._simulate_delay_async(self._get_delay_ms(response)) - return self._build_generation_response(response) - - async def _create_async_streaming_generation(self, **kwargs: Any) -> AsyncIterator[Any]: - """Create an async streaming content generation response.""" - self._record_call(type="generate_content", stream=True, **kwargs) - model = kwargs.get("model", self._default_model) - response = self._get_next_response(model=model) - await self._simulate_delay_async(self._get_delay_ms(response)) - - # Split content into chunks - if response.stream_chunks: - chunks = response.stream_chunks - else: + def _genai_chunks(self, response: MockResponse, model: str) -> Iterator[Any]: + chunks = response.stream_chunks + if chunks is None: words = response.content.split() chunks = [word + " " for word in words[:-1]] + [words[-1]] if words else [""] - - for i, chunk_content in enumerate(chunks): - is_last = i == len(chunks) - 1 - yield self._build_stream_chunk(chunk_content, is_last, response) - - def _create_streaming_generation(self, **kwargs: Any) -> Iterator[Any]: - """Create a streaming content generation response.""" - self._record_call(type="generate_content", stream=True, **kwargs) - model = kwargs.get("model", self._default_model) - response = self._get_next_response(model=model) - self._simulate_delay(self._get_delay_ms(response)) - - # Split content into chunks - if response.stream_chunks: - chunks = response.stream_chunks - else: - words = response.content.split() - chunks = [word + " " for word in words[:-1]] + [words[-1]] if words else [""] - - for i, chunk_content in enumerate(chunks): - is_last = i == len(chunks) - 1 - yield self._build_stream_chunk(chunk_content, is_last, response) - - def _build_stream_chunk( - self, - content: str, - is_last: bool, - response: MockResponse, - ) -> Any: - """Build a streaming chunk.""" - mock = MagicMock() - mock.text = content - - candidate = MagicMock() - candidate.content.parts = [MagicMock(text=content)] - candidate.content.role = "model" - candidate.finish_reason = "STOP" if is_last else None - mock.candidates = [candidate] - - return mock - - def _create_chat_session(self) -> MagicMock: - """Create a mock chat session.""" - session = MagicMock() - - def send_message(content: str, **kwargs: Any) -> Any: - self._record_call(type="chat.send_message", content=content, **kwargs) - response = self._get_next_response() - return self._build_mock_generation(response) - - session.send_message = send_message - session.history = [] - - return session - - def __enter__(self) -> GeminiMock: - """Start mocking Gemini API calls.""" - mock_module = MagicMock() - - # Create mock GenerativeModel class - def mock_model_class(model_name: str = "gemini-1.5-pro", **kwargs: Any) -> MagicMock: - model = MagicMock() - model.model_name = model_name - - def generate_content(*args: Any, **kw: Any) -> Any: - if kw.get("stream", False): - return self._create_streaming_generation(**kw) - return self._create_generation(**kw) - - async def generate_content_async(*args: Any, **kw: Any) -> Any: - if kw.get("stream", False): - return self._create_async_streaming_generation(**kw) - return await self._create_async_generation(**kw) - - model.generate_content = generate_content - model.generate_content_async = generate_content_async - model.start_chat = lambda **kw: self._create_chat_session() - - async def start_chat_async(**kw: Any) -> Any: - # Chat sessions in Gemini are mostly local state management - # but we'll provide an async variant if needed - return self._create_chat_session() - - model.start_chat_async = start_chat_async - model.count_tokens = lambda content: MagicMock( - total_tokens=len(str(content).split()) * 1.3 + for index, text in enumerate(chunks): + last = index == len(chunks) - 1 + result = self._build_genai_response( + replace(response, content=text, tool_calls=response.tool_calls if last else None), + model, ) + if not last: + result.candidates[0].finish_reason = None + result.usage_metadata = None + yield result - async def count_tokens_async(content: Any) -> Any: - return MagicMock(total_tokens=len(str(content).split()) * 1.3) - - model.count_tokens_async = count_tokens_async - - return model - - mock_module.GenerativeModel = mock_model_class - mock_module.configure = MagicMock() - + def __enter__(self) -> GeminiMock: try: - import google.generativeai as genai - except ImportError: - module_patcher = patch.dict("sys.modules", {"google.generativeai": mock_module}) - self._patches.append(module_patcher) - module_patcher.start() - else: - # Patch the real module object so imports performed before fixture activation - # observe the replacement too. - model_patcher = patch.object(genai, "GenerativeModel", mock_model_class) - self._patches.append(model_patcher) - model_patcher.start() - - self._mock_module = mock_module - return self - - @property - def module(self) -> MagicMock: - """Access the mock module directly for advanced usage.""" - return self._mock_module + super().__enter__() + + def generate(_instance: Any, *, model: str, contents: Any, config: Any = None) -> Any: + response = self._genai_response(model, contents, config) + self._simulate_delay(self._get_delay_ms(response)) + return self._build_genai_response(response, model, config) + + async def async_generate( + _instance: Any, *, model: str, contents: Any, config: Any = None + ) -> Any: + response = self._genai_response(model, contents, config) + await self._simulate_delay_async(self._get_delay_ms(response)) + return self._build_genai_response(response, model, config) + + def stream( + _instance: Any, *, model: str, contents: Any, config: Any = None + ) -> Iterator[Any]: + response = self._genai_response(model, contents, config, stream=True) + self._simulate_delay(self._get_delay_ms(response)) + yield from self._genai_chunks(response, model) + + async def async_stream( + _instance: Any, *, model: str, contents: Any, config: Any = None + ) -> AsyncIterator[Any]: + response = self._genai_response(model, contents, config, stream=True) + await self._simulate_delay_async(self._get_delay_ms(response)) + + async def chunks() -> AsyncIterator[Any]: + for chunk in self._genai_chunks(response, model): + yield chunk + + return chunks() + + def blocked(*args: Any, **kwargs: Any) -> Any: + raise RuntimeError("Gemini network blocked by active mock_gemini fixture") + + async def async_blocked(*args: Any, **kwargs: Any) -> Any: + raise RuntimeError("Gemini network blocked by active mock_gemini fixture") + + for target, replacement in [ + ("google.genai.models.Models.generate_content", generate), + ("google.genai.models.Models.generate_content_stream", stream), + ("google.genai.models.AsyncModels.generate_content", async_generate), + ("google.genai.models.AsyncModels.generate_content_stream", async_stream), + ("google.genai._api_client.BaseApiClient.request", blocked), + ("google.genai._api_client.BaseApiClient.request_streamed", blocked), + ("google.genai._api_client.BaseApiClient.async_request", async_blocked), + ("google.genai._api_client.BaseApiClient.async_request_streamed", async_blocked), + ]: + patcher = patch(target, new=replacement) + patcher.start() + self._patches.append(patcher) + return self + except BaseException: + self.__exit__(None, None, None) + raise diff --git a/src/pytest_mockllm/providers/openai.py b/src/pytest_mockllm/providers/openai.py index 8b59441..41d6ed8 100644 --- a/src/pytest_mockllm/providers/openai.py +++ b/src/pytest_mockllm/providers/openai.py @@ -54,49 +54,12 @@ def set_embedding_dimension(self, dim: int) -> OpenAIMock: return self def _raise_provider_error(self, error: MockError) -> None: - """Raise an OpenAI-style API error.""" - try: - from openai import ( - APIError, - APITimeoutError, - AuthenticationError, - BadRequestError, - RateLimitError, - ) - - error_classes = { - "rate_limit": RateLimitError, - "auth": AuthenticationError, - "timeout": APITimeoutError, - "server": APIError, - "invalid_request": BadRequestError, - } + from pytest_mockllm._sdk import raise_sdk_error - error_class = error_classes.get(error.error_type, APIError) - - import httpx - - status_code = { - "rate_limit": 429, - "auth": 401, - "timeout": 408, - "server": 500, - "invalid_request": 400, - }.get(error.error_type, 500) - mock_response = httpx.Response( - status_code, - request=httpx.Request("POST", "https://api.openai.com/v1/mock"), - json={"error": {"message": error.message}}, - ) - - raise error_class( - message=error.message, - response=mock_response, - body={"error": {"message": error.message}}, - ) + try: + raise_sdk_error("openai", error) except ImportError: - # If openai not installed, fall back to RuntimeError - raise RuntimeError(f"OpenAI API Error ({error.error_type}): {error.message}") from None + raise RuntimeError(f"openai API Error ({error.error_type}): {error.message}") from None def _create_chat_completion(self, **kwargs: Any) -> Any: """Create a mock chat completion response.""" @@ -118,16 +81,60 @@ def _create_response(self, *, text_format: type[Any] | None = None, **kwargs: An response = self._get_next_response(model=model) self._simulate_delay(self._get_delay_ms(response)) + return self._response_result(response, model, text_format, kwargs.get("stream", False)) + + async def _create_async_response( + self, *, text_format: type[Any] | None = None, **kwargs: Any + ) -> Any: + call_type = "responses.parse" if text_format is not None else "responses.create" + self._record_call(type=call_type, text_format=text_format, **kwargs) + model = kwargs.get("model", self._default_model) + response = self._get_next_response(model=model) + await self._simulate_delay_async(self._get_delay_ms(response)) + return self._response_result( + response, model, text_format, kwargs.get("stream", False), asynchronous=True + ) + + def _response_result( + self, + response: MockResponse, + model: str, + text_format: Any, + stream: bool, + asynchronous: bool = False, + ) -> Any: + result = self._build_response(response, model, text_format) + if not stream: + return result + from openai.types.responses import ResponseStreamEvent + + from pytest_mockllm._sdk import sdk_stream + from pytest_mockllm.providers._responses import response_events + + events = response_events(result, response) + if asynchronous: + + async def async_events() -> AsyncIterator[Any]: + for event in events: + yield event + + return sdk_stream( + "openai", async_events(), asynchronous=True, cast_to=ResponseStreamEvent + ) + return sdk_stream("openai", events, cast_to=ResponseStreamEvent) + + def _build_response(self, response: MockResponse, model: str, text_format: Any = None) -> Any: from openai.types.responses import ( ParsedResponse, ParsedResponseOutputMessage, ParsedResponseOutputText, Response, + ResponseFunctionToolCall, ResponseOutputMessage, ResponseOutputText, ) - common = { + common: dict[str, Any] = { "id": response.id, "created_at": time.time(), "model": model, @@ -136,6 +143,21 @@ def _create_response(self, *, text_format: type[Any] | None = None, **kwargs: An "tool_choice": "auto", "tools": [], "status": "completed", + "error": None, + "incomplete_details": None, + "instructions": None, + "metadata": {}, + "temperature": 1.0, + "top_p": 1.0, + "usage": { + "input_tokens": response.token_usage.prompt_tokens, + "output_tokens": response.token_usage.completion_tokens, + "total_tokens": response.token_usage.total_tokens, + "input_tokens_details": {"cached_tokens": 0, "cache_write_tokens": 0}, + "output_tokens_details": {"reasoning_tokens": 0}, + } + if response.token_usage + else None, } if text_format is not None: from pydantic import TypeAdapter @@ -153,9 +175,7 @@ def _create_response(self, *, text_format: type[Any] | None = None, **kwargs: An ) return ParsedResponse[Any](output=[parsed_output_message], **common) - output_text = ResponseOutputText( - annotations=[], type="output_text", text=response.content - ) + output_text = ResponseOutputText(annotations=[], type="output_text", text=response.content) output_message = ResponseOutputMessage( id=f"msg_{response.id}", content=[output_text], @@ -163,7 +183,20 @@ def _create_response(self, *, text_format: type[Any] | None = None, **kwargs: An status="completed", type="message", ) - return Response(output=[output_message], **common) + output: list[Any] = [output_message] if response.content or not response.tool_calls else [] + for tool in response.tool_calls or []: + arguments = tool["function"].get("arguments", {}) + output.append( + ResponseFunctionToolCall( + id=f"fc_{uuid.uuid4().hex[:8]}", + call_id=tool.get("id", f"call_{uuid.uuid4().hex[:8]}"), + name=tool["function"]["name"], + type="function_call", + status="completed", + arguments=arguments if isinstance(arguments, str) else json.dumps(arguments), + ) + ) + return Response(output=output, **common) def _build_completion_response(self, response: MockResponse, model: str) -> Any: """Build an OpenAI-style ChatCompletion response object.""" @@ -184,7 +217,9 @@ def _build_completion_response(self, response: MockResponse, model: str) -> Any: choice = Choice( index=0, message=message, - finish_reason=cast(Any, response.finish_reason), + finish_reason=cast( + Any, "tool_calls" if response.tool_calls else response.finish_reason + ), ) usage = None @@ -256,7 +291,9 @@ def _format_tool_calls(self, tool_calls: list[dict[str, Any]]) -> list[Any]: type="function", function=Function( name=tc["function"]["name"], - arguments=json.dumps(tc["function"].get("arguments", {})), + arguments=tc["function"].get("arguments", "{}") + if isinstance(tc["function"].get("arguments"), str) + else json.dumps(tc["function"].get("arguments", {})), ), ) for tc in tool_calls @@ -273,72 +310,49 @@ async def _create_async_chat_completion(self, **kwargs: Any) -> Any: return self._build_completion_response(response, model) async def _create_async_streaming_completion(self, **kwargs: Any) -> AsyncIterator[Any]: - """Create an async streaming chat completion response.""" - request = {**kwargs, "stream": True} - self._record_call(type="chat.completions.create", **request) + self._record_call(type="chat.completions.create", **{**kwargs, "stream": True}) model = kwargs.get("model", self._default_model) response = self._get_next_response(model=model) - - if response.tool_calls and response.tool_call_chunks: - for i, arguments in enumerate(response.tool_call_chunks): - yield self._build_tool_stream_chunk( - response=response, - model=model, - arguments=arguments, - is_first=i == 0, - is_last=i == len(response.tool_call_chunks) - 1, - ) - return - - if response.stream_chunks: - chunks = response.stream_chunks - else: - words = response.content.split() - chunks = [word + " " for word in words[:-1]] + [words[-1]] if words else [""] - - for i, chunk_content in enumerate(chunks): - yield self._build_stream_chunk( - chunk_content, - model=model, - is_first=(i == 0), - is_last=(i == len(chunks) - 1), - response_id=response.id, - ) + await self._simulate_delay_async(self._get_delay_ms(response)) + for chunk in self._completion_chunks(response, model): + yield chunk def _create_streaming_completion(self, **kwargs: Any) -> Iterator[Any]: - """Create a streaming chat completion response.""" - request = {**kwargs, "stream": True} - self._record_call(type="chat.completions.create", **request) + self._record_call(type="chat.completions.create", **{**kwargs, "stream": True}) model = kwargs.get("model", self._default_model) response = self._get_next_response(model=model) + self._simulate_delay(self._get_delay_ms(response)) + yield from self._completion_chunks(response, model) - if response.tool_calls and response.tool_call_chunks: - for i, arguments in enumerate(response.tool_call_chunks): + def _completion_chunks(self, response: MockResponse, model: str) -> Iterator[Any]: + if response.content or not response.tool_calls: + chunks = response.stream_chunks + if chunks is None: + words = response.content.split() + chunks = [word + " " for word in words[:-1]] + [words[-1]] if words else [""] + for i, content in enumerate(chunks): + yield self._build_stream_chunk( + content, + model, + i == 0, + i == len(chunks) - 1 and not response.tool_calls, + response.id, + ) + for tool_index, tool in enumerate(response.tool_calls or []): + arguments = tool["function"].get("arguments", {}) + encoded = arguments if isinstance(arguments, str) else json.dumps(arguments) + fragments = response.tool_call_chunks if tool_index == 0 else None + chunks = fragments or [encoded] + for i, fragment in enumerate(chunks): yield self._build_tool_stream_chunk( response=response, model=model, - arguments=arguments, + arguments=fragment, is_first=i == 0, - is_last=i == len(response.tool_call_chunks) - 1, + is_last=i == len(chunks) - 1 + and tool_index == len(response.tool_calls or []) - 1, + tool_index=tool_index, ) - return - - # Split content into chunks - if response.stream_chunks: - chunks = response.stream_chunks - else: - # Default: split by words - words = response.content.split() - chunks = [word + " " for word in words[:-1]] + [words[-1]] if words else [""] - - for i, chunk_content in enumerate(chunks): - yield self._build_stream_chunk( - chunk_content, - model=model, - is_first=(i == 0), - is_last=(i == len(chunks) - 1), - response_id=response.id, - ) def _build_tool_stream_chunk( self, @@ -348,6 +362,7 @@ def _build_tool_stream_chunk( arguments: str, is_first: bool, is_last: bool, + tool_index: int = 0, ) -> Any: """Build an official chunk containing one function-argument fragment.""" from openai.types.chat import ChatCompletionChunk @@ -360,10 +375,10 @@ def _build_tool_stream_chunk( if not response.tool_calls: raise ValueError("A tool call is required to build a tool stream chunk") - tool_call = response.tool_calls[0] + tool_call = response.tool_calls[tool_index] delta_tool_call = ChoiceDeltaToolCall( - index=0, - id=tool_call.get("id") if is_first else None, + index=tool_index, + id=tool_call.get("id", f"call_{response.id}_{tool_index}") if is_first else None, type="function" if is_first else None, function=ChoiceDeltaToolCallFunction( name=tool_call["function"]["name"] if is_first else None, @@ -476,7 +491,15 @@ def __enter__(self) -> OpenAIMock: def create_chat(*args: Any, **kwargs: Any) -> Any: if kwargs.get("stream", False): - return self._create_streaming_completion(**kwargs) + from openai.types.chat import ChatCompletionChunk + + from pytest_mockllm._sdk import sdk_stream + + return sdk_stream( + "openai", + self._create_streaming_completion(**kwargs), + cast_to=ChatCompletionChunk, + ) return self._create_chat_completion(**kwargs) def create_response(*args: Any, **kwargs: Any) -> Any: @@ -488,15 +511,24 @@ def parse_response(*args: Any, **kwargs: Any) -> Any: async def create_async_chat(*args: Any, **kwargs: Any) -> Any: if kwargs.get("stream", False): - return self._create_async_streaming_completion(**kwargs) + from openai.types.chat import ChatCompletionChunk + + from pytest_mockllm._sdk import sdk_stream + + return sdk_stream( + "openai", + self._create_async_streaming_completion(**kwargs), + asynchronous=True, + cast_to=ChatCompletionChunk, + ) return await self._create_async_chat_completion(**kwargs) async def create_async_response(*args: Any, **kwargs: Any) -> Any: - return self._create_response(**kwargs) + return await self._create_async_response(**kwargs) async def parse_async_response(*args: Any, **kwargs: Any) -> Any: text_format = kwargs.pop("text_format") - return self._create_response(text_format=text_format, **kwargs) + return await self._create_async_response(text_format=text_format, **kwargs) async def create_async_embedding(*args: Any, **kwargs: Any) -> Any: return self._create_embedding(**kwargs) diff --git a/src/pytest_mockllm/py.typed b/src/pytest_mockllm/py.typed new file mode 100644 index 0000000..e69de29 diff --git a/src/pytest_mockllm/recording.py b/src/pytest_mockllm/recording.py index b7164f1..fc77894 100644 --- a/src/pytest_mockllm/recording.py +++ b/src/pytest_mockllm/recording.py @@ -9,11 +9,18 @@ import hashlib import json +import os import re +import tempfile +import threading import time +import warnings +from contextlib import ExitStack +from contextvars import ContextVar from dataclasses import dataclass, field from pathlib import Path from typing import Any +from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit import yaml @@ -57,17 +64,20 @@ class Cassette: name: str interactions: list[RecordedInteraction] = field(default_factory=list) - version: str = "1.0" + version: str = "2.0" created: float = field(default_factory=time.time) def to_dict(self, redact: bool = False) -> dict[str, Any]: - data = { + data: dict[str, Any] = { "name": self.name, "version": self.version, "created": self.created, "interactions": [i.to_dict() for i in self.interactions], } if redact: + for interaction in data["interactions"]: + for side in ("request", "response"): + interaction[side] = PIIRedactor.redact_http(interaction[side]) return PIIRedactor.redact_dict(data) return data @@ -85,8 +95,19 @@ def save(self, path: Path) -> None: path.parent.mkdir(parents=True, exist_ok=True) # Always redact on save for security data = self.to_dict(redact=True) - with open(path, "w", encoding="utf-8") as f: - yaml.safe_dump(data, f, default_flow_style=False, sort_keys=False) + temporary = None + try: + with tempfile.NamedTemporaryFile( + mode="w", encoding="utf-8", dir=path.parent, suffix=".tmp", delete=False + ) as f: + temporary = Path(f.name) + yaml.safe_dump( + data, f, default_flow_style=False, sort_keys=False, allow_unicode=True + ) + os.replace(temporary, path) + finally: + if temporary is not None: + temporary.unlink(missing_ok=True) @classmethod def load(cls, path: Path) -> Cassette: @@ -101,40 +122,72 @@ class PIIRedactor: # Patterns for common API keys and sensitive tokens PATTERNS = [ - # OpenAI keys: sk-... - (re.compile(r"sk-[a-zA-Z0-9]{20,}", re.IGNORECASE), "[REDACTED_OPENAI_KEY]"), - # Anthropic keys: ant-api-... - (re.compile(r"ant-api-[a-zA-Z0-9\-_]{20,}", re.IGNORECASE), "[REDACTED_ANTHROPIC_KEY]"), + # Match Anthropic before the broader OpenAI prefix. + ( + re.compile(r"(?:sk-ant-|ant-api-)[a-zA-Z0-9_-]{20,}", re.IGNORECASE), + "[REDACTED_ANTHROPIC_KEY]", + ), + (re.compile(r"sk-(?!ant-)[a-zA-Z0-9_-]{20,}", re.IGNORECASE), "[REDACTED_OPENAI_KEY]"), # Azure OpenAI (re.compile(r"api-key: [a-zA-Z0-9]{32}", re.IGNORECASE), "api-key: [REDACTED]"), # Google/GCP keys: AIza... - (re.compile(r"AIza[0-9A-Za-z-_]{35}", re.IGNORECASE), "[REDACTED_GCP_KEY]"), + (re.compile(r"AIza[0-9A-Za-z_-]{35}", re.IGNORECASE), "[REDACTED_GCP_KEY]"), # Generic Bearer tokens - (re.compile(r"Bearer [a-zA-Z0-9\.\-_]{20,}", re.IGNORECASE), "Bearer [REDACTED]"), + (re.compile(r"Bearer [a-zA-Z0-9._~+/-]+=*", re.IGNORECASE), "Bearer [REDACTED]"), ] - SENSITIVE_KEYS = { + SENSITIVE_HEADERS = { "api_key", "api-key", "authorization", "x-api-key", - "token", - "access_token", - "secret", - "auth", + "cookie", + "set-cookie", + "proxy-authorization", + "x-goog-api-key", + } + SENSITIVE_QUERY_PARAMS = { + "key", "api_key", "api-key", "apikey", "access_token", "auth_token", + "token", "client_secret", "signature", "sig", } + # Retained for callers that inspect the old attribute; never applied to JSON keys. + SENSITIVE_KEYS = SENSITIVE_HEADERS | SENSITIVE_QUERY_PARAMS + + @classmethod + def redact_headers(cls, headers: dict[str, Any]) -> dict[str, Any]: + """Redact authentication headers without interpreting JSON field names.""" + return { + key: "[REDACTED]" if key.lower() in cls.SENSITIVE_HEADERS else cls.redact_dict(value) + for key, value in headers.items() + } + + @classmethod + def redact_url(cls, value: str) -> str: + """Remove URL credentials and sensitive query parameters.""" + url = urlsplit(value) + query = [ + (key, "[REDACTED]" if key.lower() in cls.SENSITIVE_QUERY_PARAMS else cls.redact_text(val)) + for key, val in parse_qsl(url.query, keep_blank_values=True) + ] + return urlunsplit( + (url.scheme, url.netloc.rsplit("@", 1)[-1], url.path, urlencode(sorted(query)), "") + ) + + @classmethod + def redact_http(cls, message: dict[str, Any]) -> dict[str, Any]: + """Redact an HTTP envelope; nested body objects retain their field names.""" + result = cls.redact_dict(message) + if isinstance(message.get("headers"), dict): + result["headers"] = cls.redact_headers(message["headers"]) + if isinstance(message.get("url"), str): + result["url"] = cls.redact_url(message["url"]) + return result @classmethod def redact_dict(cls, data: Any) -> Any: - """Deeply redact a dictionary or list of any sensitive information.""" + """Redact recognizable secret values, preserving arbitrary JSON schemas.""" if isinstance(data, dict): - redacted = {} - for k, v in data.items(): - if str(k).lower() in cls.SENSITIVE_KEYS: - redacted[k] = "[REDACTED]" - else: - redacted[k] = cls.redact_dict(v) - return redacted + return {k: cls.redact_dict(v) for k, v in data.items()} elif isinstance(data, list): return [cls.redact_dict(item) for item in data] elif isinstance(data, str): @@ -150,56 +203,262 @@ def redact_text(cls, text: str) -> str: return text +class RecordingError(RuntimeError): + """A cassette cannot safely satisfy the requested SDK call.""" + + +class CassetteMissError(RecordingError): + """No unused interaction matches, and the mode forbids recording.""" + + +_active: ContextVar[bool] = ContextVar("mockllm_recorder_active", default=False) +_recorder_lock = threading.Lock() + + class LLMRecorder: - """Fail-closed placeholder for recording and replay. + """Record/replay JSON and SSE HTTP responses from OpenAI and Anthropic. - ``auto``, ``record``, and ``replay`` raise on entry until safe provider interception is - implemented. ``none`` is the only pass-through mode and must be selected explicitly. + ``record_mode`` is ``none`` (offline, the default), ``once`` (record only + when the file does not exist), ``new_episodes`` (append misses), or ``all`` + (replace). Each recorded request can be replayed once per context entry. + Recording buffers complete responses, including SSE, before returning them. + SDK parsers and stream managers run normally during both record and replay. + + The legacy ``mode`` argument remains supported with a deprecation warning; + notably legacy ``mode='none'`` retains its explicit pass-through behavior. """ def __init__( self, cassette_path: str | Path, - mode: str = "auto", + mode: str | None = None, + *, + record_mode: str | None = None, ) -> None: + if mode is not None and record_mode is not None: + raise ValueError("Use record_mode or legacy mode, not both") + if mode is not None: + aliases = {"auto": "once", "record": "all", "replay": "none", "none": "passthrough"} + if mode not in aliases: + raise ValueError( + "Legacy mode must be auto, record, replay, or none; use record_mode" + ) + warnings.warn( + "LLMRecorder(mode=...) is deprecated; use record_mode='none', 'once', " + "'new_episodes', or 'all'. Legacy mode='none' still passes through live calls.", + DeprecationWarning, + stacklevel=2, + ) + record_mode = aliases[mode] + elif record_mode is not None and record_mode not in {"none", "once", "new_episodes", "all"}: + raise ValueError(f"Unknown record_mode: {record_mode!r}") self.cassette_path = Path(cassette_path) - self.mode = mode + self.mode = mode if mode is not None else (record_mode or "none") + self.record_mode = record_mode or "none" self._cassette: Cassette | None = None - self._interaction_index: int = 0 - self._patches: list[Any] = [] - self._recording: bool = False + self._interaction_index = 0 + self._used: set[int] = set() + self._recording = False + self._dirty = False + self._stack: ExitStack | None = None + self._lock = threading.RLock() + self._secrets: set[str] = set() @property def is_recording(self) -> bool: - """Check if currently in recording mode.""" return self._recording @property def cassette(self) -> Cassette | None: - """Access the current cassette.""" return self._cassette - def _should_record(self) -> bool: - """Determine if we should record or replay.""" - if self.mode == "record": - return True - elif self.mode == "replay": - if not self.cassette_path.exists(): - raise FileNotFoundError( - f"Cassette not found: {self.cassette_path}. " - "Run with --llm-record or @pytest.mark.llm_record to record first." + def __enter__(self) -> LLMRecorder: + if self._stack is not None or _active.get(): + raise RecordingError("Nested or overlapping LLMRecorder contexts are not supported") + if not _recorder_lock.acquire(blocking=False): + raise RecordingError("Nested or overlapping LLMRecorder contexts are not supported") + self._stack = ExitStack() + self._stack.callback(_recorder_lock.release) + token = _active.set(True) + self._stack.callback(_active.reset, token) + try: + if self.record_mode == "passthrough": + return self + exists = self.cassette_path.exists() + self._recording = self.record_mode in {"all", "new_episodes"} or ( + self.record_mode == "once" and not exists + ) + self._cassette = ( + Cassette.load(self.cassette_path) + if exists and self.record_mode != "all" + else Cassette(name=self.cassette_path.stem) + ) + if self._cassette.version != "2.0": + raise RecordingError( + "Unsupported cassette version; record a new cassette with --llm-record=all" ) - return False - elif self.mode == "auto": - return not self.cassette_path.exists() - else: # mode == "none" - return False + self._used.clear() + self._secrets.clear() + self._interaction_index = 0 + self._dirty = False + from pytest_mockllm._recording_http import install + + install(self, self._stack) + return self + except BaseException: + self._stack.close() + self._stack = None + raise + + def __exit__(self, exc_type: Any, exc_val: Any, exc_tb: Any) -> None: + if self._stack is not None: + self._stack.close() + self._stack = None + if self._dirty and self._cassette is not None: + # Scrub again so secrets learned from later requests cannot leak from + # earlier echoed responses. Atomic replacement preserves old files. + self._cassette = Cassette.from_dict(self._redact(self._cassette.to_dict())) + self._cassette.save(self.cassette_path) + + def _redact(self, value: Any) -> Any: + value = PIIRedactor.redact_dict(value) + if isinstance(value, dict): + return {k: self._redact(v) for k, v in value.items()} + if isinstance(value, list): + return [self._redact(v) for v in value] + if isinstance(value, str): + for secret in sorted(self._secrets, key=len, reverse=True): + value = value.replace(secret, "[REDACTED]") + return value + + def remember_credentials(self, client: Any) -> None: + for name in ("api_key", "auth_token"): + value = getattr(client, name, None) + if isinstance(value, str) and value: + self._secrets.add(value) + + def _request(self, provider: str, request: Any) -> dict[str, Any]: + url = urlsplit(str(request.url)) + endpoints = { + "openai": ("/chat/completions", "/responses", "/embeddings"), + "anthropic": ("/messages",), + } + if request.method != "POST" or not url.path.rstrip("/").endswith(endpoints[provider]): + raise RecordingError(f"Unsupported {provider} endpoint for recording/replay") + try: + body = json.loads(request.content) + except (ValueError, UnicodeError) as error: + raise RecordingError("Recording supports JSON request bodies only") from error + headers = dict(request.headers) + query = parse_qsl(url.query, keep_blank_values=True) + for values, sensitive in ( + (headers.items(), PIIRedactor.SENSITIVE_HEADERS), + (query, PIIRedactor.SENSITIVE_QUERY_PARAMS), + ): + for key, value in values: + if key.lower() in sensitive and value: + self._secrets.add(value) + if value.lower().startswith("bearer "): + self._secrets.add(value[7:]) + clean_url = PIIRedactor.redact_url(str(request.url)) + return self._redact( + { + "method": request.method, + "url": clean_url, + "headers": PIIRedactor.redact_headers(headers), + "body": body, + } + ) def _hash_request(self, request: dict[str, Any]) -> str: - """Create a hash of the request for matching.""" - # Normalize the request for hashing - normalized = json.dumps(request, sort_keys=True, default=str) - return hashlib.sha256(normalized.encode()).hexdigest()[:16] + semantic_headers = { + "content-type", + "anthropic-version", + "anthropic-beta", + "openai-beta", + "openai-organization", + "openai-project", + } + normalized = { + **request, + "headers": { + k: v for k, v in request.get("headers", {}).items() if k.lower() in semantic_headers + }, + } + return hashlib.sha256(json.dumps(normalized, sort_keys=True).encode()).hexdigest() + + def prepare( + self, provider: str, request: Any, *, consume: bool = True + ) -> tuple[dict[str, Any], dict[str, Any] | None]: + with self._lock: + prepared = self._request(provider, request) + assert self._cassette is not None + if self.record_mode != "all": + wanted = self._hash_request(prepared) + for index, interaction in enumerate(self._cassette.interactions): + if ( + index not in self._used + and interaction.provider == provider + and self._hash_request(interaction.request) == wanted + ): + if consume: + self._used.add(index) + return prepared, interaction.response + if not self._recording: + raise CassetteMissError( + f"No unused {provider} interaction matches in {self.cassette_path}. " + "Network access refused. Record explicitly with --llm-record=once " + "(new file), new_episodes (append), or all (replace)." + ) + return prepared, None + + def capture(self, provider: str, request: dict[str, Any], response: Any) -> None: + """Store a completed HTTP response; incomplete reads never reach this method.""" + content_type = response.headers.get("content-type", "") + text = response.content.decode("utf-8") + if "json" in content_type: + body = self._redact(json.loads(text)) + encoding = "json" + else: + # SSE data may contain JSON credentials unrelated to the actual key. + lines = [] + for line in text.splitlines(keepends=True): + if line.startswith("data:"): + try: + data = self._redact(json.loads(line[5:].strip())) + line = "data: " + json.dumps(data, ensure_ascii=False) + "\n" + except ValueError: + pass + lines.append(line) + body = self._redact("".join(lines)) + encoding = "text" + headers = { + k: v + for k, v in response.headers.items() + if k.lower() not in {"content-length", "content-encoding", "transfer-encoding"} + } + self.record_interaction( + request, + { + "status_code": response.status_code, + "headers": PIIRedactor.redact_headers(headers), + "encoding": encoding, + "body": body, + }, + provider, + str(request["body"].get("model", "")), + ) + + @staticmethod + def http_response(http: Any, request: Any, replay: dict[str, Any]) -> Any: + body = replay["body"] + content = json.dumps(body, ensure_ascii=False) if replay["encoding"] == "json" else body + return http.Response( + replay["status_code"], + headers=replay["headers"], + content=content.encode("utf-8"), + request=request, + ) def record_interaction( self, @@ -209,77 +468,31 @@ def record_interaction( model: str, latency_ms: float = 0, ) -> None: - """Record a new interaction.""" - if self._cassette is None: - self._cassette = Cassette(name=self.cassette_path.stem) - - interaction = RecordedInteraction( - request=request, - response=response, - provider=provider, - model=model, - latency_ms=latency_ms, - ) - self._cassette.interactions.append(interaction) + with self._lock: + if self._cassette is None: + self._cassette = Cassette(name=self.cassette_path.stem) + self._used.add(len(self._cassette.interactions)) + self._cassette.interactions.append( + RecordedInteraction( + self._redact(request), + self._redact(response), + provider, + model, + latency_ms=latency_ms, + ) + ) + self._dirty = True def get_next_replay(self) -> RecordedInteraction | None: - """Get the next recorded interaction for replay.""" if self._cassette is None or self._interaction_index >= len(self._cassette.interactions): return None - interaction = self._cassette.interactions[self._interaction_index] self._interaction_index += 1 return interaction - def __enter__(self) -> LLMRecorder: - """Start recording/replaying.""" - if self.mode != "none": - raise RuntimeError( - "LLM recording and replay are temporarily unavailable because provider " - "interception is not yet implemented safely. pytest-mockllm refuses to " - "fall through to a live API. Use a provider mock fixture such as " - "mock_openai, or construct LLMRecorder(..., mode='none') for explicit " - "pass-through behavior." - ) - - self._recording = self._should_record() - - if not self._recording and self.cassette_path.exists(): - # Load existing cassette for replay - self._cassette = Cassette.load(self.cassette_path) - self._interaction_index = 0 - elif self._recording: - # Start fresh cassette - self._cassette = Cassette(name=self.cassette_path.stem) - - if not self._recording: - # Set up replay patches - self._setup_replay_patches() - - return self - - def _setup_replay_patches(self) -> None: - """Set up patches to replay recorded responses.""" - # For now, the replay is handled by the individual provider mocks - # This can be extended to intercept real API calls - pass - - def __exit__(self, exc_type: Any, exc_val: Any, exc_tb: Any) -> None: - """Stop recording and save cassette if needed.""" - # Stop all patches - for p in reversed(self._patches): - p.stop() - self._patches.clear() - - # Save cassette if we were recording - if self._recording and self._cassette and self._cassette.interactions: - self._cassette.save(self.cassette_path) - def clear(self) -> None: - """Clear all recorded interactions.""" - self._cassette = None + self._cassette = Cassette(name=self.cassette_path.stem) self._interaction_index = 0 - - # Delete cassette file if it exists - if self.cassette_path.exists(): - self.cassette_path.unlink() + self._used.clear() + self._dirty = False + self.cassette_path.unlink(missing_ok=True) diff --git a/tests/acceptance/test_anthropic_sdk_contract.py b/tests/acceptance/test_anthropic_sdk_contract.py index a0da8ba..daf4838 100644 --- a/tests/acceptance/test_anthropic_sdk_contract.py +++ b/tests/acceptance/test_anthropic_sdk_contract.py @@ -4,7 +4,9 @@ import json -import httpx +from pytest_mockllm._sdk import http_module + +httpx = http_module("anthropic") import pytest from anthropic import ( Anthropic, diff --git a/tests/acceptance/test_latest_sdk.py b/tests/acceptance/test_latest_sdk.py new file mode 100644 index 0000000..0236443 --- /dev/null +++ b/tests/acceptance/test_latest_sdk.py @@ -0,0 +1,181 @@ +"""Regression contracts for the current SDK models and streaming interfaces.""" + +import asyncio +import importlib +import json + +import pytest +from openai import AsyncOpenAI, OpenAI +from openai.types.responses import Response +from pydantic import BaseModel + +from pytest_mockllm.core import TokenUsage + + +class Answer(BaseModel): + answer: str + + +def test_responses_text_usage_and_tools(mock_openai): + mock_openai.add_response("hello", token_usage=TokenUsage(3, 4)) + mock_openai.add_response( + "", + tool_calls=[ + {"id": "call_1", "function": {"name": "weather", "arguments": {"city": "Paris"}}} + ], + ) + with OpenAI(api_key="fake") as client: + response = client.responses.create(model="gpt-4o-mini", input="hi") + assert isinstance(response, Response) + assert response.output_text == "hello" + assert response.usage.total_tokens == 7 + tool = client.responses.create(model="gpt-4o-mini", input="weather?").output[0] + assert tool.type == "function_call" + assert tool.call_id == "call_1" + assert json.loads(tool.arguments) == {"city": "Paris"} + + +@pytest.mark.parametrize("asynchronous", [False, True]) +@pytest.mark.parametrize("tool", [False, True]) +@pytest.mark.asyncio +async def test_responses_stream_types_and_context_manager(mock_openai, asynchronous, tool): + if tool: + mock_openai.add_response( + "", + tool_calls=[ + {"id": "call_1", "function": {"name": "weather", "arguments": {"city": "Paris"}}} + ], + tool_call_chunks=['{"city":', '"Paris"}'], + ) + else: + mock_openai.add_response("hello", stream_chunks=["hel", "lo"]) + client = (AsyncOpenAI if asynchronous else OpenAI)(api_key="fake") + if asynchronous: + async with await client.responses.create( + model="gpt-4o-mini", input="hi", stream=True + ) as stream: + events = [event async for event in stream] + await client.close() + else: + with client.responses.create(model="gpt-4o-mini", input="hi", stream=True) as stream: + events = list(stream) + client.close() + assert events[0].type == "response.created" + assert events[-1].type == "response.completed" + assert [e.sequence_number for e in events] == list(range(len(events))) + assert isinstance(events[-1].response, Response) + if tool: + fragments = [e.delta for e in events if e.type == "response.function_call_arguments.delta"] + assert json.loads("".join(fragments)) == {"city": "Paris"} + done = next(e for e in events if e.type == "response.function_call_arguments.done") + assert done.name == "weather" + assert json.loads(done.arguments) == {"city": "Paris"} + else: + assert "".join(e.delta for e in events if e.type == "response.output_text.delta") == "hello" + assert events[-1].response.output_text == "hello" + + +@pytest.mark.parametrize("asynchronous", [False, True]) +@pytest.mark.asyncio +async def test_responses_stream_helper_final_parsed_output(mock_openai, asynchronous): + mock_openai.add_response('{"answer":"hello"}', stream_chunks=['{"answer":', '"hello"}']) + client = (AsyncOpenAI if asynchronous else OpenAI)(api_key="fake") + manager = client.responses.stream(model="gpt-4o-mini", input="hi", text_format=Answer) + if asynchronous: + async with manager as stream: + response = await stream.get_final_response() + await client.close() + else: + with manager as stream: + response = stream.get_final_response() + client.close() + assert response.output_parsed == Answer(answer="hello") + + +@pytest.mark.asyncio +async def test_responses_async_delay_yields_event_loop(mock_openai): + mock_openai.add_response('{"answer":"hello"}', latency_ms=30) + advanced = False + + async def other_task(): + nonlocal advanced + await asyncio.sleep(0) + advanced = True + + task = asyncio.create_task(other_task()) + async with AsyncOpenAI(api_key="fake") as client: + response = await client.responses.parse(model="gpt-4o-mini", input="hi", text_format=Answer) + assert advanced + assert response.output_parsed == Answer(answer="hello") + await task + + +@pytest.mark.parametrize("provider", ["openai", "anthropic"]) +@pytest.mark.parametrize( + "kind,expected", + [ + ("rate_limit", "RateLimitError"), + ("auth", "AuthenticationError"), + ("timeout", "APITimeoutError"), + ("server", "InternalServerError"), + ("invalid_request", "BadRequestError"), + ], +) +def test_official_error_constructors(request, provider, kind, expected): + mock = request.getfixturevalue("mock_" + provider) + sdk = importlib.import_module(provider) + mock.simulate_error(kind) + with getattr(sdk, "OpenAI" if provider == "openai" else "Anthropic")(api_key="fake") as client: + with pytest.raises(getattr(sdk, expected)) as error: + if provider == "openai": + client.chat.completions.create(model="test", messages=[]) + else: + client.messages.create(model="test", messages=[], max_tokens=1) + assert error.value.request.method == "POST" + + +@pytest.mark.parametrize("asynchronous", [False, True]) +@pytest.mark.asyncio +async def test_anthropic_stream_helper_text_and_tools(mock_anthropic, asynchronous): + import anthropic + + mock_anthropic.add_response( + "Looking up weather", + tool_calls=[ + {"id": "tool_1", "function": {"name": "weather", "arguments": {"city": "Paris"}}} + ], + ) + client = (anthropic.AsyncAnthropic if asynchronous else anthropic.Anthropic)(api_key="fake") + manager = client.messages.stream(model="test", max_tokens=100, messages=[]) + if asynchronous: + async with manager as stream: + result = await stream.get_final_message() + await client.close() + else: + with manager as stream: + result = stream.get_final_message() + client.close() + assert result.content[0].text == "Looking up weather" + assert result.content[1].type == "tool_use" + assert result.content[1].input == {"city": "Paris"} + assert result.stop_reason == "tool_use" + + +def test_chat_stream_multiple_tools_without_manual_fragments(mock_openai): + mock_openai.add_response( + "", + tool_calls=[ + {"function": {"name": name, "arguments": {"city": "Paris"}}} + for name in ["weather", "time"] + ], + ) + with OpenAI(api_key="fake", base_url="https://compatible.invalid/v1") as client: + with client.chat.completions.create( + model="another-provider-model", messages=[], stream=True + ) as stream: + chunks = list(stream) + tools = [c.choices[0].delta.tool_calls[0] for c in chunks] + assert [t.index for t in tools] == [0, 1] + assert [t.function.name for t in tools] == ["weather", "time"] + assert all(json.loads(t.function.arguments) == {"city": "Paris"} for t in tools) + assert chunks[-1].choices[0].finish_reason == "tool_calls" diff --git a/tests/test_gemini.py b/tests/test_gemini.py index b597b7c..9924d5e 100644 --- a/tests/test_gemini.py +++ b/tests/test_gemini.py @@ -1,14 +1,118 @@ -"""Gemini provider integration contracts.""" +"""google-genai contracts using real clients, messages, streams and error types.""" -import google.generativeai as genai +import pytest +from google import genai +from google.genai import errors, types +from pydantic import BaseModel +from pytest_mockllm.core import TokenUsage -def test_module_imported_before_fixture_is_intercepted(mock_gemini): - """A normal module-level SDK import must not bypass the active fixture.""" - mock_gemini.add_response("offline") - model = genai.GenerativeModel("gemini-1.5-pro") - response = model.generate_content("Tell me about AI") +def test_client_created_before_fixture_is_intercepted(request): + with genai.Client(api_key="fixture-only") as client: + mock = request.getfixturevalue("mock_gemini") + mock.add_response("hello", token_usage=TokenUsage(3, 5)) + response = client.models.generate_content(model="gemini-2.5-flash", contents="hi") + assert isinstance(response, types.GenerateContentResponse) + assert response.text == "hello" + assert response.model_version == "gemini-2.5-flash" + assert response.usage_metadata.total_token_count == 8 + assert mock.last_call["contents"] == "hi" + assert mock.last_call["model"] == "gemini-2.5-flash" + with pytest.raises(RuntimeError, match="Gemini network blocked"): + client.models.list() - assert response.text == "offline" - assert mock_gemini.call_count == 1 + +def test_sync_stream_and_chat_history(mock_gemini): + mock_gemini.add_response("hello", stream_chunks=["hel", "lo"]) + mock_gemini.add_responses("first", "second") + with genai.Client(api_key="fixture-only") as client: + stream = list( + client.models.generate_content_stream(model="gemini-2.5-flash", contents="hi") + ) + assert all(isinstance(c, types.GenerateContentResponse) for c in stream) + assert [c.text for c in stream] == ["hel", "lo"] + assert stream[0].candidates[0].finish_reason is None + assert stream[-1].candidates[0].finish_reason == "STOP" + chat = client.chats.create(model="gemini-2.5-flash") + assert chat.send_message("one").text == "first" + assert chat.send_message("two").text == "second" + assert len(chat.get_history()) == 4 + assert mock_gemini.call_count == 3 + + +@pytest.mark.asyncio +async def test_async_generation_stream_and_chat(mock_gemini): + mock_gemini.add_responses("hello", "streamed response", "chat reply") + async with genai.Client(api_key="fixture-only").aio as client: + assert ( + await client.models.generate_content(model="gemini-2.5-flash", contents="hi") + ).text == "hello" + stream = await client.models.generate_content_stream( + model="gemini-2.5-flash", contents="hi" + ) + chunks = [chunk async for chunk in stream] + assert all(isinstance(c, types.GenerateContentResponse) for c in chunks) + assert "".join(c.text for c in chunks) == "streamed response" + chat = client.chats.create(model="gemini-2.5-flash") + assert (await chat.send_message("hi")).text == "chat reply" + with pytest.raises(RuntimeError, match="Gemini network blocked"): + await client.models.get(model="unsupported") + + +@pytest.mark.parametrize( + "kind,code", + [ + ("rate_limit", 429), + ("auth", 401), + ("timeout", 504), + ("server", 500), + ("invalid_request", 400), + ], +) +def test_error_types_strict_and_recovery(mock_gemini, kind, code): + mock_gemini.simulate_error(kind, times=1) + mock_gemini.add_response("recovered") + with genai.Client(api_key="fixture-only") as client: + with pytest.raises(errors.APIError) as error: + client.models.generate_content(model="gemini-2.5-flash", contents="hi") + assert error.value.code == code + assert ( + client.models.generate_content(model="gemini-2.5-flash", contents="hi").text + == "recovered" + ) + mock_gemini.set_strict_mode() + with pytest.raises(RuntimeError, match="No mock response"): + client.models.generate_content(model="gemini-2.5-flash", contents="hi") + + +def test_function_call_and_structured_output(mock_gemini): + class Answer(BaseModel): + answer: int + + mock_gemini.add_response( + "", tool_calls=[{"function": {"name": "weather", "arguments": {"city": "Paris"}}}] + ) + mock_gemini.add_response('{"answer":42}') + with genai.Client(api_key="fixture-only") as client: + response = client.models.generate_content(model="gemini-2.5-flash", contents="weather?") + assert response.function_calls[0].name == "weather" + assert response.function_calls[0].args == {"city": "Paris"} + response = client.models.generate_content( + model="gemini-2.5-flash", + contents="answer?", + config=types.GenerateContentConfig( + response_mime_type="application/json", response_schema=Answer + ), + ) + assert response.parsed == Answer(answer=42) + + +def test_chat_stream_uses_real_chat_session(mock_gemini): + mock_gemini.add_response("hello", stream_chunks=["hel", "lo"]) + with genai.Client(api_key="fixture-only") as client: + chat = client.chats.create(model="gemini-2.5-flash") + assert "".join(c.text for c in chat.send_message_stream("hi")) == "hello" + history = chat.get_history() + assert history[0].role == "user" + assert "".join(part.text for content in history[1:] for part in content.parts) == "hello" diff --git a/tests/test_gemini_legacy.py b/tests/test_gemini_legacy.py new file mode 100644 index 0000000..87e828e --- /dev/null +++ b/tests/test_gemini_legacy.py @@ -0,0 +1,40 @@ +"""Gemini provider integration contracts.""" + +import pytest + +genai = pytest.importorskip("google.generativeai") + + +def test_module_imported_before_fixture_is_intercepted(mock_gemini): + """A normal module-level SDK import must not bypass the active fixture.""" + mock_gemini.add_response("offline") + + with pytest.warns(DeprecationWarning, match="google.generativeai support"): + model = genai.GenerativeModel("gemini-1.5-pro") + response = model.generate_content("Tell me about AI") + + assert response.text == "offline" + assert mock_gemini.call_count == 1 + + +def test_legacy_stream_and_provider_error(mock_gemini): + from google.api_core.exceptions import ResourceExhausted + + mock_gemini.add_response("hello world") + with pytest.warns(DeprecationWarning): + model = genai.GenerativeModel("gemini-1.5-pro") + assert "".join(c.text for c in model.generate_content("hi", stream=True)) == "hello world" + assert mock_gemini.last_call["contents"] == "hi" + mock_gemini.simulate_error("rate_limit") + with pytest.raises(ResourceExhausted): + model.generate_content("hi") + + +@pytest.mark.asyncio +async def test_legacy_async_generation(mock_gemini): + mock_gemini.add_responses("hello", "streamed answer") + with pytest.warns(DeprecationWarning): + model = genai.GenerativeModel("gemini-1.5-pro") + assert (await model.generate_content_async("hi")).text == "hello" + stream = await model.generate_content_async("hi", stream=True) + assert "".join([c.text async for c in stream]) == "streamed answer" diff --git a/tests/test_langchain.py b/tests/test_langchain.py index 4e0fc13..1c8467d 100644 --- a/tests/test_langchain.py +++ b/tests/test_langchain.py @@ -90,3 +90,15 @@ def test_with_structured_output_include_raw_uses_langchain_envelope(): assert result["raw"].content == _model_json(response_model) assert isinstance(result["parsed"], HabitantsResponse) assert result["parsing_error"] is None + +@pytest.mark.asyncio +async def test_langchain_typed_async_and_stream_results(mock_langchain): + from langchain_core.messages import AIMessage, AIMessageChunk + mock_langchain.add_responses('async answer', 'streamed answer') + model = ChatOpenAI(model='gpt-4o-mini', api_key='fixture-only') + result = await model.ainvoke('hi') + assert isinstance(result, AIMessage) + assert result.content == 'async answer' + chunks = [chunk async for chunk in model.astream('hi')] + assert all(isinstance(chunk, AIMessageChunk) for chunk in chunks) + assert ''.join(chunk.content for chunk in chunks) == 'streamed answer' diff --git a/tests/test_package.py b/tests/test_package.py new file mode 100644 index 0000000..d162e0e --- /dev/null +++ b/tests/test_package.py @@ -0,0 +1,11 @@ +"""Published metadata must agree with the importable package.""" + +from importlib.metadata import version +from pathlib import Path + +import pytest_mockllm + + +def test_version_and_typing_marker(): + assert version("pytest-mockllm") == pytest_mockllm.__version__ == "0.4.0" + assert Path(pytest_mockllm.__file__).with_name("py.typed").exists() diff --git a/tests/test_plugin.py b/tests/test_plugin.py index dd972c2..9f209bc 100644 --- a/tests/test_plugin.py +++ b/tests/test_plugin.py @@ -4,6 +4,7 @@ """ from importlib.metadata import requires +from unittest.mock import Mock import pytest @@ -25,7 +26,7 @@ def test_mock_llm_available(mock_llm): assert mock_llm is not None """ ) - + result = testdir.runpytest("-v") result.assert_outcomes(passed=4) @@ -42,7 +43,7 @@ def test_openai_mock(mock_openai): assert mock_openai.call_count == 1 """ ) - + result = testdir.runpytest("-v") result.assert_outcomes(passed=1) @@ -63,7 +64,7 @@ def test_multiple_responses(mock_openai): assert mock_openai._get_next_response().content == "Third" """ ) - + result = testdir.runpytest("-v") result.assert_outcomes(passed=1) @@ -81,7 +82,7 @@ def test_strict_mode(mock_openai): mock_openai._get_next_response() """ ) - + result = testdir.runpytest("-v") result.assert_outcomes(passed=1) @@ -89,7 +90,7 @@ def test_strict_mode(mock_openai): def test_cli_options_registered(testdir): """Verify CLI options are available.""" result = testdir.runpytest("--help") - + assert "--llm-record" in result.stdout.str() assert "--llm-cassette-dir" in result.stdout.str() assert "--llm-strict" in result.stdout.str() @@ -117,7 +118,7 @@ def test_unconfigured_call_fails(mock_openai): def test_markers_registered(testdir): """Verify markers are registered.""" result = testdir.runpytest("--markers") - + assert "llm_mock" in result.stdout.str() assert "llm_record" in result.stdout.str() assert "llm_replay" in result.stdout.str() @@ -131,3 +132,117 @@ def test_langchain_extra_installs_documented_provider_package(): requirement.startswith("langchain-openai") and "extra == 'langchain'" in requirement for requirement in requirements ) + + +@pytest.mark.parametrize("flag", ["--llm-record=once", "--llm-record"]) +def test_recording_cli_and_offline_default(pytester, flag): + pytester.makepyfile(""" + import httpx + from openai import OpenAI + def test_record(llm_recorder): + def handle(request): + assert llm_recorder.is_recording, "replay reached transport" + return httpx.Response(200, json={ + "id": "chat_1", "object": "chat.completion", "created": 1, + "model": "test", "choices": [{"index": 0, "finish_reason": "stop", + "message": {"role": "assistant", "content": "hello"}}]}) + with OpenAI(api_key="fake", max_retries=0, http_client=httpx.Client(transport=httpx.MockTransport(handle))) as client: + response = client.chat.completions.create(model="test", messages=[]) + assert response.choices[0].message.content == "hello" + """) + pytester.runpytest("-q", flag).assert_outcomes(passed=1) + files = list((pytester.path / "tests" / "llm_cassettes").glob("*.yaml")) + assert len(files) == 1 + before = files[0].read_bytes() + pytester.runpytest("-q").assert_outcomes(passed=1) + assert files[0].read_bytes() == before + + +def test_record_marker_activates_offline_recorder_and_replay_overrides_cli(pytester): + pytester.makepyfile(""" + import pytest + from openai import OpenAI + from pytest_mockllm.recording import CassetteMissError + @pytest.mark.llm_record + @pytest.mark.llm_replay + def test_offline(): + with OpenAI(api_key="fake", max_retries=0) as client: + with pytest.raises(CassetteMissError): + client.chat.completions.create(model="test", messages=[]) + """) + pytester.runpytest("-q", "--llm-record=all").assert_outcomes(passed=1) + assert not list(pytester.path.rglob("*.yaml")) + + +def test_recorder_rejects_mixed_mock_fixtures(pytester): + pytester.makepyfile("def test_conflict(llm_recorder, mock_openai): pass") + result = pytester.runpytest("-q") + result.assert_outcomes(errors=1) + assert "separately" in result.stdout.str() + + +def test_cassette_names_include_module_and_parameter_ids(pytester): + source = """ + import pytest + @pytest.mark.parametrize("value", ["a/b", "a?b"]) + def test_same(llm_recorder, value): + llm_recorder.record_interaction({"body": {}}, {"body": value}, "test", "test") + """ + pytester.makepyfile(test_one=source, test_two=source) + pytester.runpytest("-q", "--llm-record=once").assert_outcomes(passed=4) + files = list((pytester.path / "tests" / "llm_cassettes").glob("*.yaml")) + assert len(files) == 4 + + +@pytest.mark.parametrize( + "args,ini,expected", + [ + ([], False, False), + (["--llm-stats"], False, True), + ([], True, True), + (["--no-llm-stats"], True, False), + (["--llm-stats", "--no-llm-stats"], False, False), + ], +) +def test_stats_are_opt_in(pytester, args, ini, expected): + pytester.makeini("[pytest]\nllm_stats = " + str(ini).lower()) + pytester.makepyfile(""" + def test_call(mock_openai): + mock_openai.add_response("hi") + mock_openai.client.chat.completions.create(model="test", messages=[]) + """) + result = pytester.runpytest("-q", *args) + result.assert_outcomes(passed=1) + assert ("Total Estimated Cost Saved" in result.stdout.str()) is expected + + +@pytest.mark.parametrize("cost,display", [ + (0, "$0.0000"), + (0.0000001, "<$0.0001"), + (0.000099, "<$0.0001"), + (0.0001, "$0.0001"), + (1.23456, "$1.2346"), +]) +def test_stats_cost_precision(monkeypatch, cost, display): + from pytest_mockllm.plugin import pytest_terminal_summary + from pytest_mockllm.stats import GLOBAL_STATS + + monkeypatch.setattr(GLOBAL_STATS, "total_calls", 1) + monkeypatch.setattr(GLOBAL_STATS, "total_cost_saved", cost) + config = Mock() + config.getoption.side_effect = lambda name: name == "--llm-stats" + reporter = Mock() + pytest_terminal_summary(reporter, 0, config) + reporter.write_sep.assert_called_once_with( + "=", f"Total Estimated Cost Saved: {display}", bold=True, blue=True, + ) + + +def test_global_strict_applies_to_langchain(pytester): + pytester.makepyfile(""" + import pytest + def test_strict(mock_langchain): + with pytest.raises(RuntimeError, match="No mock response"): + mock_langchain.model.invoke("hello") + """) + pytester.runpytest("-q", "--llm-strict").assert_outcomes(passed=1) diff --git a/tests/test_readme.py b/tests/test_readme.py new file mode 100644 index 0000000..f8cedfe --- /dev/null +++ b/tests/test_readme.py @@ -0,0 +1,24 @@ +"""Execute every offline Python example from the README as a pytest test module.""" + +import ast +import re +from pathlib import Path + + +def test_readme_examples(pytester): + readme = (Path(__file__).resolve().parents[1] / "README.md").read_text(encoding="utf-8-sig") + examples = [] + count = 0 + for source in re.findall(r"```python\n(.*?)\n```", readme, re.S): + tree = ast.parse(source) + functions = [ + node for node in tree.body if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) + ] + # The live recorder example is covered by SDK/CLI transport tests instead. + if any(any(arg.arg == "llm_recorder" for arg in node.args.args) for node in functions): + continue + count += sum(node.name.startswith("test_") for node in functions) + examples.append(source) + assert count >= 9 + pytester.makepyfile("\n\n".join(examples)) + pytester.runpytest("-q").assert_outcomes(passed=count) diff --git a/tests/test_recording.py b/tests/test_recording.py index 327586e..2e1bd32 100644 --- a/tests/test_recording.py +++ b/tests/test_recording.py @@ -1,20 +1,775 @@ -"""Safety contracts for unfinished recording and replay support.""" +"""Record through real SDK parsers; replay must never reach the HTTP transport.""" +from __future__ import annotations + +import importlib +import json from pathlib import Path import pytest +import yaml + +from pytest_mockllm.recording import CassetteMissError, LLMRecorder, RecordingError + + +def http_module(provider): + sdk = importlib.import_module(f"{provider}._base_client") + return getattr(sdk, "httpx2", getattr(sdk, "httpx", None)) + + +def payload(provider, text="hello"): + if provider == "openai": + return { + "id": "chat_1", + "object": "chat.completion", + "created": 1, + "model": "test-model", + "choices": [ + { + "index": 0, + "message": {"role": "assistant", "content": text}, + "finish_reason": "stop", + } + ], + "usage": {"prompt_tokens": 3, "completion_tokens": 2, "total_tokens": 5}, + } + return { + "id": "msg_1", + "type": "message", + "role": "assistant", + "model": "test-model", + "content": [{"type": "text", "text": text}], + "stop_reason": "end_turn", + "stop_sequence": None, + "usage": {"input_tokens": 3, "output_tokens": 2}, + } + + +def streaming_body(provider, tool=False): + if provider == "openai": + chunks = [] + for index, text in enumerate(["hel", "lo"] if not tool else ['{"city":', '"Paris"}']): + delta = {"content": text} + if tool: + delta = { + "tool_calls": [ + { + "index": 0, + "id": "call_1" if index == 0 else None, + "type": "function", + "function": { + "name": "weather" if index == 0 else None, + "arguments": text, + }, + } + ] + } + chunks.append( + { + "id": "chat_1", + "object": "chat.completion.chunk", + "created": 1, + "model": "test-model", + "choices": [ + { + "index": 0, + "delta": delta, + "finish_reason": ("tool_calls" if tool else "stop") if index else None, + } + ], + } + ) + return "".join("data: " + json.dumps(c) + "\n\n" for c in chunks) + "data: [DONE]\n\n" + message = payload(provider) + message.update(content=[], stop_reason=None) + events = [ + {"type": "message_start", "message": message}, + { + "type": "content_block_start", + "index": 0, + "content_block": {"type": "tool_use", "id": "tool_1", "name": "weather", "input": {}} + if tool + else {"type": "text", "text": ""}, + }, + ] + for text in ['{"city":', '"Paris"}'] if tool else ["hel", "lo"]: + events.append( + { + "type": "content_block_delta", + "index": 0, + "delta": {"type": "input_json_delta", "partial_json": text} + if tool + else {"type": "text_delta", "text": text}, + } + ) + events += [ + {"type": "content_block_stop", "index": 0}, + { + "type": "message_delta", + "delta": {"stop_reason": "tool_use" if tool else "end_turn", "stop_sequence": None}, + "usage": {"output_tokens": 2}, + }, + {"type": "message_stop"}, + ] + return "".join("event: " + e["type"] + "\ndata: " + json.dumps(e) + "\n\n" for e in events) + + +def make_client(provider, handler, asynchronous=False, key="private-record-key", base_url=None): + http = http_module(provider) + sdk = importlib.import_module(provider) + cls = getattr( + sdk, ("Async" if asynchronous else "") + ("OpenAI" if provider == "openai" else "Anthropic") + ) + http_client = (http.AsyncClient if asynchronous else http.Client)( + transport=http.MockTransport(handler) + ) + return cls( + api_key=key, + http_client=http_client, + max_retries=0, + base_url=base_url or "https://provider.invalid/v1", + ) + + +def call(client, provider, prompt="hi", **kwargs): + params = {"model": "test-model", "messages": [{"role": "user", "content": prompt}], **kwargs} + if provider == "anthropic": + return client.messages.create(max_tokens=64, **params) + return client.chat.completions.create(**params) + + +def dump(result): + return result.model_dump(mode="json") + + +def forbidden(request): + pytest.fail("Replay reached HTTP transport") + + +@pytest.mark.parametrize("provider", ["openai", "anthropic"]) +@pytest.mark.parametrize("mode", ["once", "new_episodes", "all"]) +def test_sync_record_replay_offline(tmp_path, provider, mode): + path = tmp_path / "test.yaml" + requests = [] + http = http_module(provider) + + def handle(request): + requests.append(request) + return http.Response( + 200, + json=payload(provider), + headers={"x-request-id" if provider == "openai" else "request-id": "req_1"}, + ) + + with make_client(provider, handle) as client, LLMRecorder(path, record_mode=mode): + original = call(client, provider) + assert len(requests) == 1 + assert "private-record-key" not in path.read_text() + assert "hello" in path.read_text() + with make_client(provider, forbidden, key="different-offline-key") as client, LLMRecorder(path): + replay = call(client, provider) + assert type(original) is type(replay) + assert dump(original) == dump(replay) + assert replay._request_id == "req_1" + + +@pytest.mark.parametrize("provider", ["openai", "anthropic"]) +@pytest.mark.parametrize("tool", [False, True]) +@pytest.mark.asyncio +async def test_async_stream_roundtrip(tmp_path, provider, tool): + path = tmp_path / "stream.yaml" + http = http_module(provider) + + def handle(request): + return http.Response( + 200, + content=streaming_body(provider, tool), + headers={"content-type": "text/event-stream"}, + ) + + async with make_client(provider, handle, asynchronous=True) as client: + with LLMRecorder(path, record_mode="once"): + stream = await call(client, provider, stream=True) + original = [dump(event) async for event in stream] + await stream.close() + async with make_client(provider, forbidden, asynchronous=True) as client: + with LLMRecorder(path): + stream = await call(client, provider, stream=True) + replay = [dump(event) async for event in stream] + await stream.close() + assert original == replay + assert len(replay) >= 2 + assert yaml.safe_load(path.read_text())["interactions"][0]["response"]["encoding"] == "text" + + +@pytest.mark.parametrize("provider", ["openai", "anthropic"]) +@pytest.mark.parametrize("tool", [False, True]) +def test_sync_stream_roundtrip(tmp_path, provider, tool): + path = tmp_path / "stream.yaml" + http = http_module(provider) + + def handle(request): + return http.Response( + 200, + content=streaming_body(provider, tool), + headers={"content-type": "text/event-stream"}, + ) + + with make_client(provider, handle) as client, LLMRecorder(path, record_mode="once"): + with call(client, provider, stream=True) as stream: + original = [dump(event) for event in stream] + with make_client(provider, forbidden) as client, LLMRecorder(path): + with call(client, provider, stream=True) as stream: + assert [dump(event) for event in stream] == original + + +@pytest.mark.parametrize("provider", ["openai", "anthropic"]) +@pytest.mark.asyncio +async def test_async_nonstream_roundtrip(tmp_path, provider): + http = http_module(provider) + path = tmp_path / "async.yaml" + async with make_client( + provider, lambda r: http.Response(200, json=payload(provider)), True + ) as client: + with LLMRecorder(path, record_mode="once"): + original = await call(client, provider) + async with make_client(provider, forbidden, True) as client: + with LLMRecorder(path): + assert dump(await call(client, provider)) == dump(original) + + +@pytest.mark.parametrize("provider", ["openai", "anthropic"]) +def test_modes_matching_and_repeated_calls(tmp_path, provider): + path = tmp_path / "modes.yaml" + http = http_module(provider) + requests = [] + + def handle(request): + requests.append(request) + return http.Response(200, json=payload(provider, str(len(requests)))) + + with make_client(provider, handle) as client: + with LLMRecorder(path, record_mode="once"): + first = dump(call(client, provider, extra_body={"temperature": 0.1})) + second = dump(call(client, provider, extra_body={"temperature": 0.1})) + before = path.read_bytes() + with LLMRecorder(path, record_mode="once"): + assert dump(call(client, provider, extra_body={"temperature": 0.1})) == first + assert dump(call(client, provider, extra_body={"temperature": 0.1})) == second + with pytest.raises(CassetteMissError): + call(client, provider, extra_body={"temperature": 0.1}) + with pytest.raises(CassetteMissError): + call(client, provider, extra_body={"temperature": 0.2}) + assert path.read_bytes() == before + assert len(requests) == 2 + with LLMRecorder(path, record_mode="new_episodes"): + assert dump(call(client, provider, extra_body={"temperature": 0.1})) == first + call(client, provider, prompt="new") + assert len(requests) == 3 + assert len(yaml.safe_load(path.read_text())["interactions"]) == 3 + with LLMRecorder(path, record_mode="all"): + call(client, provider, prompt="replacement") + assert len(requests) == 4 + assert len(yaml.safe_load(path.read_text())["interactions"]) == 1 + + +@pytest.mark.parametrize("provider", ["openai", "anthropic"]) +def test_missing_mismatched_and_unsupported_fail_closed(tmp_path, provider): + path = tmp_path / "missing.yaml" + with make_client(provider, forbidden) as client, LLMRecorder(path): + with pytest.raises(CassetteMissError, match="Network access refused"): + call(client, provider) + with pytest.raises(RecordingError, match="Unsupported"): + client.models.list() + assert not path.exists() + + +@pytest.mark.parametrize("provider", ["openai", "anthropic"]) +def test_errors_replay_as_official_sdk_errors(tmp_path, provider): + http = http_module(provider) + sdk = importlib.import_module(provider) + path = tmp_path / "error.yaml" + + def handle(request): + return http.Response( + 429, + json={"type": "error", "error": {"type": "rate_limit_error", "message": "slow down"}}, + headers={"retry-after": "0"}, + ) + + with ( + make_client(provider, handle) as client, + LLMRecorder(path, record_mode="once"), + pytest.raises(sdk.RateLimitError) as original, + ): + call(client, provider) + with ( + make_client(provider, forbidden) as client, + LLMRecorder(path), + pytest.raises(sdk.RateLimitError) as replay, + ): + call(client, provider) + assert replay.value.status_code == original.value.status_code == 429 + assert replay.value.body == original.value.body + + +@pytest.mark.parametrize("provider", ["openai", "anthropic"]) +def test_redaction_in_headers_query_nested_body_and_echo(tmp_path, provider): + http = http_module(provider) + path = tmp_path / "secret.yaml" + + def handle(request): + data = payload(provider, "echo private-record-key and Bearer short-token") + data["metadata"] = { + "api_key": "sk-ant-" + "a" * 35, + "nested": [{"Authorization": "private-record-key"}], + } + return http.Response( + 200, + json=data, + headers={"Set-Cookie": "session=private", "authorization": "response-auth"}, + ) + + with make_client(provider, handle, base_url="https://provider.invalid/v1") as client: + # Query params on SDK base URLs vary; explicit extra_query is stable. + with LLMRecorder(path, record_mode="once"): + call( + client, + provider, + extra_query={"key": "query-secret"}, + extra_body={"metadata": {"api_key": "sk-" + "b" * 35}}, + extra_headers={"X-Api-Key": "custom-secret"}, + ) + text = path.read_text() + for secret in [ + "private-record-key", + "short-token", + "sk-ant-" + "a" * 35, + "session=private", + "response-auth", + "query-secret", + "sk-" + "b" * 35, + "custom-secret", + ]: + assert secret not in text + assert "[REDACTED]" in text + assert "[REDACTED_ANTHROPIC_KEY]" in text + + +@pytest.mark.parametrize("provider", ["openai", "anthropic"]) +def test_tool_schema_and_body_fields_survive_record_replay(tmp_path, provider): + http = http_module(provider) + path = tmp_path / "schema.yaml" + schema = { + "type": "object", + "properties": { + name: {"type": "string", "description": f"The {name} value"} + for name in ("token", "key", "auth", "api_key", "authorization") + }, + "required": ["token"], + } + tool = {"name": "inspect", "input_schema": schema} if provider == "anthropic" else { + "type": "function", "function": {"name": "inspect", "parameters": schema}, + } + body_fields = {"token": "word", "key": "lookup", "auth": "method", + "nested": {"headers": {"Authorization": "schema description"}}} + + def handle(request): + assert json.loads(request.content)["tools"] == [tool] + data = payload(provider) + data["metadata"] = body_fields + return http.Response(200, json=data) + + with make_client(provider, handle) as client, LLMRecorder(path, record_mode="once"): + original = dump(call(client, provider, tools=[tool])) + interaction = yaml.safe_load(path.read_text())["interactions"][0] + assert interaction["request"]["body"]["tools"] == [tool] + assert interaction["response"]["body"]["metadata"] == body_fields + assert "private-record-key" not in path.read_text() + with make_client(provider, forbidden, key="different-credential") as client, LLMRecorder(path): + schema["properties"]["token"]["type"] = "integer" + with pytest.raises(CassetteMissError): + call(client, provider, tools=[tool]) + schema["properties"]["token"]["type"] = "string" + assert dump(call(client, provider, tools=[tool])) == original + + +@pytest.mark.parametrize("provider", ["openai", "anthropic"]) +def test_failed_stream_does_not_create_cassette(tmp_path, provider): + http = http_module(provider) + path = tmp_path / "incomplete.yaml" + + class Broken(http.SyncByteStream): + def __iter__(self): + yield b"data: partial\n\n" + raise http.ReadError("disconnected") + + def handle(request): + return http.Response(200, stream=Broken(), headers={"content-type": "text/event-stream"}) + + with ( + make_client(provider, handle) as client, + LLMRecorder(path, record_mode="once"), + pytest.raises(Exception), + ): + call(client, provider, stream=True) + assert not path.exists() + + +def test_cleanup_nested_context_and_legacy_modes(tmp_path): + import openai + + original = openai._base_client.SyncAPIClient.request + with pytest.raises(ValueError, match="body failed"), LLMRecorder(tmp_path / "empty.yaml"): + with pytest.raises(RecordingError, match="Nested"): + with LLMRecorder(tmp_path / "nested.yaml"): + pass + raise ValueError("body failed") + assert openai._base_client.SyncAPIClient.request is original + with ( + pytest.warns(DeprecationWarning), + LLMRecorder(tmp_path / "legacy.yaml", mode="none") as recorder, + ): + assert not recorder.is_recording + assert openai._base_client.SyncAPIClient.request is original + for old, new in [("auto", "once"), ("record", "all"), ("replay", "none")]: + with pytest.warns(DeprecationWarning): + assert LLMRecorder(tmp_path / "legacy.yaml", mode=old).record_mode == new + with pytest.raises(ValueError): + LLMRecorder(tmp_path / "bad.yaml", record_mode="typo") + + +def test_corrupt_file_does_not_enable_network_and_restores_patches(tmp_path): + path = tmp_path / "corrupt.yaml" + path.write_text('name: old\nversion: "1.0"\ninteractions: []\n') + with pytest.raises(RecordingError, match="version"), LLMRecorder(path, record_mode="once"): + pytest.fail("must not enter") + with LLMRecorder(tmp_path / "other.yaml"): + pass + + +def test_unrelated_http_passes_through(tmp_path): + import httpx + + with httpx.Client( + transport=httpx.MockTransport(lambda r: httpx.Response(200, text="unrelated")) + ) as client: + with LLMRecorder(tmp_path / "empty.yaml"): + assert client.get("https://unrelated.invalid").text == "unrelated" + + +@pytest.mark.parametrize("provider", ["openai", "anthropic"]) +def test_sdk_retry_sequence_roundtrips(tmp_path, provider): + http = http_module(provider) + path = tmp_path / "retry.yaml" + count = 0 + + def handle(request): + nonlocal count + count += 1 + if count == 1: + return http.Response( + 429, + json={"type": "error", "error": {"type": "rate_limit_error", "message": "retry"}}, + headers={"retry-after-ms": "1"}, + ) + return http.Response(200, json=payload(provider)) + + with ( + make_client(provider, handle).with_options(max_retries=1) as client, + LLMRecorder(path, record_mode="once"), + ): + original = call(client, provider) + assert count == 2 + with make_client(provider, forbidden).with_options(max_retries=1) as client, LLMRecorder(path): + assert dump(call(client, provider)) == dump(original) + assert len(yaml.safe_load(path.read_text())["interactions"]) == 2 + + +@pytest.mark.parametrize("asynchronous", [False, True]) +@pytest.mark.asyncio +async def test_anthropic_stream_manager(tmp_path, asynchronous): + http = http_module("anthropic") + path = tmp_path / "manager.yaml" + + def handle(request): + return http.Response( + 200, content=streaming_body("anthropic"), headers={"content-type": "text/event-stream"} + ) + + results = [] + for mode, handler in [("once", handle), ("none", forbidden)]: + client = make_client("anthropic", handler, asynchronous) + with LLMRecorder(path, record_mode=mode): + manager = client.messages.stream( + model="test-model", messages=[{"role": "user", "content": "hi"}], max_tokens=64 + ) + if asynchronous: + async with manager as stream: + text = "".join([part async for part in stream.text_stream]) + message = await stream.get_final_message() + await client.close() + else: + with manager as stream: + text = "".join(stream.text_stream) + message = stream.get_final_message() + client.close() + results.append((text, dump(message))) + assert results[0] == results[1] + assert results[0][0] == "hello" + + +def response_payload(text="hello"): + return { + "id": "resp_1", + "object": "response", + "created_at": 1, + "model": "test-model", + "status": "completed", + "error": None, + "incomplete_details": None, + "instructions": None, + "max_output_tokens": None, + "parallel_tool_calls": False, + "tool_choice": "auto", + "tools": [], + "temperature": 1, + "top_p": 1, + "output": [ + { + "type": "message", + "id": "msg_1", + "role": "assistant", + "status": "completed", + "content": [{"type": "output_text", "text": text, "annotations": []}], + } + ], + "usage": { + "input_tokens": 3, + "output_tokens": 2, + "total_tokens": 5, + "input_tokens_details": {"cached_tokens": 0}, + "output_tokens_details": {"reasoning_tokens": 0}, + }, + } + + +@pytest.mark.parametrize("asynchronous", [False, True]) +@pytest.mark.asyncio +async def test_openai_responses_parse_roundtrip(tmp_path, asynchronous): + from pydantic import BaseModel + + class Answer(BaseModel): + answer: str + + path = tmp_path / "responses.yaml" + http = http_module("openai") + results = [] + for mode, handler in [ + ("once", lambda r: http.Response(200, json=response_payload('{"answer":"hello"}'))), + ("none", forbidden), + ]: + client = make_client("openai", handler, asynchronous) + with LLMRecorder(path, record_mode=mode): + response = client.responses.parse(model="test-model", input="hi", text_format=Answer) + if asynchronous: + response = await response + assert response.output_parsed == Answer(answer="hello") + results.append(dump(response)) + if asynchronous: + await client.close() + else: + client.close() + assert results[0] == results[1] + + +@pytest.mark.parametrize("asynchronous", [False, True]) +@pytest.mark.asyncio +async def test_openai_responses_sse_roundtrip(tmp_path, asynchronous): + path = tmp_path / "responses-stream.yaml" + http = http_module("openai") + events = [ + { + "type": "response.output_text.delta", + "sequence_number": 0, + "output_index": 0, + "content_index": 0, + "item_id": "msg_1", + "delta": "hello", + "logprobs": [], + }, + {"type": "response.completed", "sequence_number": 1, "response": response_payload()}, + ] + + def handle(request): + return http.Response( + 200, + content="".join( + "event: " + e["type"] + "\ndata: " + json.dumps(e) + "\n\n" for e in events + ), + headers={"content-type": "text/event-stream"}, + ) + + results = [] + for mode, handler in [("once", handle), ("none", forbidden)]: + client = make_client("openai", handler, asynchronous) + with LLMRecorder(path, record_mode=mode): + stream = client.responses.create(model="test-model", input="hi", stream=True) + if asynchronous: + stream = await stream + result = [dump(e) async for e in stream] + await client.close() + else: + result = [dump(e) for e in stream] + client.close() + results.append(result) + assert results[0] == results[1] + assert results[1][0]["delta"] == "hello" + + +@pytest.mark.parametrize("provider", ["openai", "anthropic"]) +def test_actual_local_http_roundtrip(tmp_path, provider, monkeypatch): + from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer + from threading import Thread + import socket + + requests = [] + + class Handler(BaseHTTPRequestHandler): + def do_POST(self): + requests.append(json.loads(self.rfile.read(int(self.headers["Content-Length"])))) + body = json.dumps(payload(provider)).encode() + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def log_message(self, *args): + pass + + server = ThreadingHTTPServer(("127.0.0.1", 0), Handler) + thread = Thread(target=server.serve_forever, daemon=True) + thread.start() + base_url = f"http://127.0.0.1:{server.server_port}/v1" + sdk = importlib.import_module(provider) + cls = getattr(sdk, "OpenAI" if provider == "openai" else "Anthropic") + path = tmp_path / "local.yaml" + try: + with cls(api_key="local-only", base_url=base_url, max_retries=0) as client: + with LLMRecorder(path, record_mode="once"): + original = call(client, provider) + finally: + server.shutdown() + server.server_close() + thread.join(timeout=3) + assert len(requests) == 1 + + def no_connect(*args, **kwargs): + pytest.fail("Replay attempted a socket connection") + + monkeypatch.setattr(socket.socket, "connect", no_connect) + with cls(api_key="offline-only", base_url=base_url, max_retries=0) as client: + with LLMRecorder(path): + assert dump(call(client, provider)) == dump(original) + + +@pytest.mark.parametrize("provider", ["openai", "anthropic"]) +def test_custom_send_override_fails_before_network(tmp_path, provider): + http = http_module(provider) + sdk = importlib.import_module(provider) + + class Custom(http.Client): + def send(self, *args, **kwargs): + pytest.fail("custom send bypassed the recorder") + + cls = getattr(sdk, "OpenAI" if provider == "openai" else "Anthropic") + with cls(api_key="fake", http_client=Custom(), max_retries=0) as client: + with LLMRecorder(tmp_path / "unsupported.yaml", record_mode="once"): + with pytest.raises(RecordingError, match="overriding send"): + call(client, provider) + + +def test_atomic_save_preserves_previous_file_on_failure(tmp_path, monkeypatch): + import pytest_mockllm.recording as recording + + path = tmp_path / "existing.yaml" + path.write_text("previous contents") + cassette = recording.Cassette(name="test") + + def failure(*args): + raise OSError("replacement failed") + + monkeypatch.setattr(recording.os, "replace", failure) + with pytest.raises(OSError, match="replacement failed"): + cassette.save(path) + assert path.read_text() == "previous contents" + assert list(tmp_path.glob("*.tmp")) == [] -from pytest_mockllm.recording import LLMRecorder +@pytest.mark.parametrize("provider", ["openai", "anthropic"]) +def test_stream_credentials_are_redacted(tmp_path, provider): + http = http_module(provider) + path = tmp_path / "secret-stream.yaml" + body = streaming_body(provider).replace('"hel"', '"private-record-key"') + schema = {"token": {"type": "string"}, "key": "lookup", "auth": "method"} + lines = [] + for line in body.splitlines(keepends=True): + if line.startswith("data: {"): + data = json.loads(line[5:]) + data["metadata"] = schema + line = "data: " + json.dumps(data) + "\n" + lines.append(line) + body = "".join(lines) -@pytest.mark.parametrize("mode", ["auto", "record", "replay"]) -def test_recorder_fails_closed_before_provider_calls(tmp_path: Path, mode: str) -> None: - recorder = LLMRecorder(tmp_path / "cassette.yaml", mode=mode) + def handle(request): + return http.Response( + 200, + content=body, + headers={"content-type": "text/event-stream", "authorization": "response-secret"}, + ) - with pytest.raises(RuntimeError, match="refuses to fall through to a live API"): - recorder.__enter__() + with make_client(provider, handle) as client, LLMRecorder(path, record_mode="once"): + list(call(client, provider, stream=True)) + contents = path.read_text() + assert "private-record-key" not in contents + assert "response-secret" not in contents + recorded_body = yaml.safe_load(contents)["interactions"][0]["response"]["body"] + for line in recorded_body.splitlines(): + if line.startswith("data: {"): + assert json.loads(line[5:])["metadata"] == schema + with make_client(provider, forbidden) as client, LLMRecorder(path): + events = list(call(client, provider, stream=True)) + assert "[REDACTED]" in json.dumps([dump(e) for e in events]) -def test_explicit_none_mode_allows_pass_through(tmp_path: Path) -> None: - with LLMRecorder(tmp_path / "cassette.yaml", mode="none") as recorder: - assert recorder.is_recording is False +@pytest.mark.parametrize("asynchronous", [False, True]) +@pytest.mark.asyncio +async def test_embedding_record_replay(tmp_path, asynchronous): + http = http_module("openai") + path = tmp_path / "embedding.yaml" + body = { + "object": "list", + "model": "text-embedding-3-small", + "data": [ + {"object": "embedding", "index": 0, "embedding": [0.1, 0.2]}, + {"object": "embedding", "index": 1, "embedding": [0.3, 0.4]}, + ], + "usage": {"prompt_tokens": 2, "total_tokens": 2}, + } + results = [] + for mode, handler in [("once", lambda r: http.Response(200, json=body)), ("none", forbidden)]: + client = make_client("openai", handler, asynchronous) + with LLMRecorder(path, record_mode=mode): + result = client.embeddings.create( + model="text-embedding-3-small", input=["one", "two"], dimensions=2 + ) + if asynchronous: + result = await result + results.append(dump(result)) + if asynchronous: + await client.close() + else: + client.close() + assert results[0] == results[1] + assert results[1]["data"][1]["embedding"] == [0.3, 0.4] diff --git a/tests/unit/test_pii_redactor.py b/tests/unit/test_pii_redactor.py index 836a2b6..0bf6513 100644 --- a/tests/unit/test_pii_redactor.py +++ b/tests/unit/test_pii_redactor.py @@ -1,47 +1,63 @@ +from urllib.parse import parse_qs, urlsplit + +import pytest + from pytest_mockllm.recording import PIIRedactor -def test_redact_openai_key(): - text = "The key is sk-1234567890abcdef1234567890abcdef." - redacted = PIIRedactor.redact_text(text) - assert "[REDACTED_OPENAI_KEY]" in redacted - assert "sk-" not in redacted + +@pytest.mark.parametrize(("prefix", "label"), [ + ("sk-", "OPENAI"), + ("sk-proj-", "OPENAI"), + ("sk-ant-api03-", "ANTHROPIC"), + ("sk-ant-", "ANTHROPIC"), + ("ant-api-", "ANTHROPIC"), + ("AIza", "GCP"), +]) +def test_known_key_patterns_and_labels(prefix, label): + key = prefix + "a" * 35 + assert PIIRedactor.redact_text("Key: " + key) == f"Key: [REDACTED_{label}_KEY]" + def test_redact_bearer_token(): - text = "Authorization: Bearer my-secret-token-1234567890" - redacted = PIIRedactor.redact_text(text) - assert "Bearer [REDACTED]" in redacted + assert PIIRedactor.redact_text("Bearer credential") == "Bearer [REDACTED]" + def test_redact_azure_key(): - text = "Connecting with api-key: 1234567890abcdef1234567890abcdef" - redacted = PIIRedactor.redact_text(text) - assert "api-key: [REDACTED]" in redacted - assert "1234567890" not in redacted - -def test_redact_gcp_key(): - # Using a fake pattern that matches our regex but doesn't look real to GitHub - text = "Key: AIza_MOCK_GOOGLE_API_KEY_FOR_TESTING_123" - redacted = PIIRedactor.redact_text(text) - assert "[REDACTED_GCP_KEY]" in redacted - assert "AIza" not in redacted - -def test_redact_dict(): + assert PIIRedactor.redact_text("api-key: " + "a" * 32) == "api-key: [REDACTED]" + + +def test_json_field_names_do_not_identify_secrets(): data = { - "api_key": "sk-12345", - "api-key": "azure-secret", - "nested": { - "token": "secret-token", - "msg": "Hello sk-1234567890abcdef12345" - }, - "list": ["AIza_MOCK_GOOGLE_API_KEY_FOR_TESTING_123", {"auth": "Bearer token"}] + "properties": {name: {"type": "string"} for name in [ + "key", "token", "auth", "api_key", "authorization", "secret", "password", + ]}, + "nested": [{"headers": {"Authorization": "a schema value"}}], + "token": "a word in a sentence", + "auth": "authentication method", } - redacted = PIIRedactor.redact_dict(data) - - assert redacted["api_key"] == "[REDACTED]" - assert redacted["api-key"] == "[REDACTED]" - assert redacted["nested"]["token"] == "[REDACTED]" - assert "[REDACTED_OPENAI_KEY]" in redacted["nested"]["msg"] - assert "[REDACTED_GCP_KEY]" in redacted["list"][0] - assert redacted["list"][1]["auth"] == "[REDACTED]" - # Note: 'auth' is not in SENSITIVE_KEYS, but it might contain a Bearer token - # Wait, 'auth' is not in my SENSITIVE_KEYS set. Let me check. - # PIIRedactor.SENSITIVE_KEYS = {"api_key", "api-key", "authorization", "x-api-key", "token", "access_token", "secret"} + assert PIIRedactor.redact_dict(data) == data + assert PIIRedactor.redact_http({"body": data})["body"] == data + + +def test_secret_values_are_redacted_regardless_of_json_field_name(): + data = {"nested": [{"ordinary": "sk-ant-" + "a" * 35}], "auth": "Bearer credential"} + assert PIIRedactor.redact_dict(data) == { + "nested": [{"ordinary": "[REDACTED_ANTHROPIC_KEY]"}], "auth": "Bearer [REDACTED]", + } + + +@pytest.mark.parametrize("header", [ + "Authorization", "X-Api-Key", "api-key", "api_key", "X-Goog-Api-Key", + "Proxy-Authorization", "Cookie", "Set-Cookie", +]) +def test_credentials_are_redacted_in_header_context(header): + assert PIIRedactor.redact_headers({header: "unrecognizable-credential", "X-Request-ID": "req_1"}) == { + header: "[REDACTED]", "X-Request-ID": "req_1", + } + + +def test_google_query_key_is_redacted_without_touching_other_parameters(): + url = PIIRedactor.redact_url( + "https://generativelanguage.googleapis.com/v1beta/models?key=opaque-credential&alt=sse" + ) + assert parse_qs(urlsplit(url).query) == {"key": ["[REDACTED]"], "alt": ["sse"]} diff --git a/tests/unit/test_token_counter.py b/tests/unit/test_token_counter.py index ad31db3..1ea182b 100644 --- a/tests/unit/test_token_counter.py +++ b/tests/unit/test_token_counter.py @@ -1,20 +1,22 @@ from pytest_mockllm.core import TokenCounter import tiktoken -def test_token_counter_tiktoken(): - text = "Hello, world! This is a test." +def test_token_counter_tiktoken(monkeypatch): + import tiktoken.model + import tiktoken.registry + + # A real, locally constructed encoder proves preloaded encoders are used + # without making the unit suite download third-party tokenizer data. + enc = tiktoken.Encoding( + name="offline-test", pat_str=r".", + mergeable_ranks={bytes([i]): i for i in range(256)}, special_tokens={}, + ) model = "gpt-4o" - - # Expected count using tiktoken directly - enc = tiktoken.encoding_for_model(model) - expected = len(enc.encode(text)) - - # Counter count - actual = TokenCounter.count_tokens(text, model) - - print(f"DEBUG: Expected {expected}, Actual {actual}") - assert actual == expected - assert actual > 0 + name = tiktoken.model.encoding_name_for_model(model) + monkeypatch.setitem(tiktoken.registry.ENCODINGS, name, enc) + text = "Hello, world! This is a test." + assert TokenCounter.count_tokens(text, model) == len(enc.encode(text)) + def test_token_counter_fallback(): # Model that doesn't exist in tiktoken mappings should fallback to cl100k_base or char-count @@ -22,3 +24,13 @@ def test_token_counter_fallback(): actual = TokenCounter.count_tokens(text, "non-existent-model") # Should be roughly len/4 + 1 if tiktoken fails or cl100k_base if tiktoken handles it via fallback assert actual > 0 + + +def test_token_counter_does_not_download_an_uncached_tokenizer(monkeypatch): + import tiktoken.registry + monkeypatch.setattr(tiktoken.registry, 'ENCODINGS', {}) + def forbidden(*args, **kwargs): + raise AssertionError('Implicit token counting must not load/download encoders') + monkeypatch.setattr(tiktoken, 'encoding_for_model', forbidden) + monkeypatch.setattr(tiktoken, 'get_encoding', forbidden) + assert TokenCounter.count_tokens('hello', 'gpt-4o') == 2