Repository navigation
feat: prepare 0.4.0 with LLM cassettes, modern Gemini, and SDK contracts - #8
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Recording in 0.3.1 was disabled, Gemini used the retired SDK, and fresh SDK releases exposed gaps in error and streaming behavior. This prepares 0.4.0 with tested OpenAI/Anthropic recording, modern Gemini fixtures, quieter defaults, and launch documentation. It does not publish to PyPI.
What changed
Implemented YAML record/replay for OpenAI Chat Completions, Responses, embeddings, and Anthropic Messages: sync/async JSON and SSE, text/tool fragments, SDK stream helpers, error responses, and HTTP retry sequences.
Added
none(offline default),once,new_episodes, andallthrough--llm-record, the existingllm_recorderfixture, and automatic markers. Matching includes provider, method, URL/query, JSON body, and selected semantic headers; identical calls consume recordings in order. Offline misses and unsupported endpoints fail closed.Added credential/header redaction, atomic writes, and cassette paths derived from full pytest IDs. Legacy recorder arguments remain with deprecation warnings, including legacy
mode='none'pass-through.Migrated
[google]/[all]togoogle-genai, with typed generation, streams, chat state, function calls, structured output, and errors. Retained the deprecated adapter under[google-legacy], with sync/async/stream/error regression coverage.Made statistics opt-in with
--llm-stats/llm_stats;--no-llm-statsoverrides them.Fixed SDK timeout/server error construction, HTTPX/HTTPX2 compatibility, async Responses delay, Responses text/tool streams and final parsed output, and tool streams without explicit fragments. Applied global strict mode to LangChain.
Rewrote the README around the testing problem, runnable quickstart, provider/stream/tool/error/recording examples, and a sourced comparison with unittest.mock, respx, vcrpy, pytest-recording, deepeval, and promptfoo.
Added an actual asciinema recording and regeneration script, SDK compatibility notes, minimum-SDK CI,
py.typed, and consistent 0.4.0 metadata/CHANGELOG.Fixed an offline-testing issue found during review: implicit token estimation no longer downloads tokenizers. It uses preloaded tiktoken encoders or character estimates.
Validation
Read all tracked files and ran the unchanged baseline first: 60 passed on Python 3.12.
Initial local full-suite results on Windows (before review fixes):
| Environment | Result |
| --- | --- |
| Python 3.9.25, newest compatible SDKs | 150 passed; optional legacy Gemini module skipped |
| Python 3.10.21, current SDKs | 150 passed; optional legacy Gemini module skipped |
| Python 3.11.16, current SDKs | 150 passed; optional legacy Gemini module skipped |
| Python 3.12.13, current SDKs + legacy Gemini | 153 passed |
| Python 3.13.15, current SDKs | 150 passed; optional legacy Gemini module skipped |
| Python 3.12.13, minimum SDK set + legacy Gemini | 153 passed |
Current SDKs checked against PyPI and tested: OpenAI 3.35.1, Anthropic 1.15.0, google-genai 2.31.0, langchain-core 1.6.12, langchain-openai 1.7.2. Minimum set: 1.109.1 / 0.75.0 / 1.47.0 / 0.3.80 / 0.3.35, respectively. Legacy google-generativeai: 0.8.6.
Final commit 7eef29b: all CI checks passed, including the full Python 3.9-3.14 matrix on Linux/macOS/Windows.
Also passed:
Real local HTTP-server recording, then replay after server shutdown with socket connections blocked, for both providers.
Controlled-transport contracts for modes, exhaustion, changed credentials, redaction, streams, parsed output, retries, errors, incomplete reads, cleanup, and atomic replacement.
All nine offline README Python examples and all three demo tests.
Ruff, mypy, and
git diff --check.Wheel/source distribution build and
twine check.Clean built-wheel smoke test without provider SDKs; demo tests with only the OpenAI extra; verified packaged
py.typed.The Python 3.9 dependency set also emits upstream Google/Pydantic warnings; all applicable tests pass.
The pytester integration tests emit pytest-asyncio's unset fixture-loop-scope deprecation warning in their temporary projects. They pass; this is not a warning from ordinary plugin usage.
Not included
Gemini recording, LiteLLM-specific integration, new assertion helpers, or an environment-gated live-test marker.
A complete typing overhaul: the marker and public fixture annotations are included, but some internals still use
Any.Paid/live provider calls, model-quality evaluation, a PyPI release, or launch posts.
Limits and review focus
Recording buffers full UTF-8 JSON/SSE responses before returning them. Events/tool fragments replay; arrival timing and partial HTTP disconnects do not. Memory scales with response size.
Interception uses private SDK request-building hooks and HTTPX/HTTPX2 send hooks. Minimum/current suites passed; every intervening SDK release has not been tested.
WebSockets, non-JSON requests, unsupported endpoints, HTTP clients overriding
send, overlapping recorder contexts, and concurrent writers to one cassette file are unsupported. Unrelated HTTP traffic is not globally blocked.Redaction is credential-focused, not general PII removal. Prompts/answers remain readable and need review before committing cassettes.
The OpenAI-compatible
base_urltest proves the OpenAI SDK path; it does not certify other vendors' live APIs.Local validation is Windows-only. Final-commit CI passed Python 3.9-3.14 on Linux, macOS, and Windows (18 matrix jobs), plus minimum-SDK tests, lint, and package build/smoke checks. Publishing was skipped. No claims about live API acceptance or account/model availability are made.
See SDK compatibility and evidence and demo recording.