OAI Lens

OAI Lens is a small, container-first black-box conformance runner for servers that claim compatibility with the OpenAI API. It uses the unmodified official OpenAI Python SDK and reports each API area separately rather than inventing a compatibility percentage.

This is the first usable version of the project outlined in docs/outline.md. The name reflects its purpose: inspecting externally observable API behavior without assuming anything about the implementation behind the endpoint.

Run in Docker

Copy the example configuration and edit the target and model:

cp config.example.yaml config.yaml
make target

The machine-readable report is written to reports/report.json. host.docker.internal is mapped to the Docker host by Compose, so the example configuration can reach a locally running API server. Every probe runs on every invocation, and the process exits nonzero if any probe fails.

The JSON report includes an ordered exchanges array containing every prepared SDK request and received response, including retries, URLs, duplicate headers, status codes, complete JSON bodies, complete SSE streams, and elapsed time. Text bodies are stored as UTF-8; other bodies are losslessly base64-encoded. Authentication, cookies, and API-key headers are always replaced with <redacted>.

To run a prebuilt image without Compose:

docker build -t oai-lens .
docker run --rm \
  --add-host host.docker.internal:host-gateway \
  -v "$PWD/config.yaml:/config/config.yaml:ro" \
  -v "$PWD/reports:/output" \
  oai-lens

Candidate credentials are independent from OpenAI credentials. Set target.api_key_env in YAML and pass that environment variable into the container when authentication is needed. Do not put secrets directly in committed configuration.

config.yaml

Configuration is strict: unknown keys are errors. See config.example.yaml for every current setting. Every invocation runs all v1 suites:

• responses: core calls, strict structured parsing, stateless output-item replay, deliberate incomplete responses, and repeated longer-context/cache accounting;
• streaming: text and function-argument deltas, ordered item lifecycle events, terminal states, and usage;
• tools: single and multiple function continuation, web search, hosted shell/container execution, generated JavaScript execution, and file search cleanup;
• vision: repository-owned image fixtures sent as data URLs, remote URLs, uploaded file IDs, and mixed multi-image input;
• errors: invalid prior-response IDs and invalid strict tool schemas, including SDK exception and wire-envelope observations;
• chat: messages, strict structured parsing, streaming, and repeated longer context.

Every ordinary call targets only target.base_url. Live OpenAI access is used exclusively by capture.

Capture an OpenAI reference

The reference authentication method and model are configured independently from the candidate target:

capture:
  # auto, api_key, or codex_oauth
  auth: auto
  model: gpt-5.6-luna
  # Captures are not latency-sensitive.
  service_tier: flex

The capture target runs locally through the uv-managed environment because OAuth credentials remain on the host. auto uses OPENAI_API_KEY when it is set, then falls back to the oai-lens ChatGPT login:

# OpenAI Platform API key with capture.auth: auto or api_key
OPENAI_API_KEY=... make capture

# ChatGPT Plus/Pro/Codex subscription with capture.auth: auto or codex_oauth
uv run oai-lens login
make capture

oai-lens login implements the ChatGPT PKCE flow directly; the Codex CLI is not required. It prints an authorization URL, then asks for the complete localhost callback URL. This also works on a headless or remote server: open the URL on your workstation, complete login, copy the final URL from the browser address bar (the localhost page may fail to load), and paste it into the remote terminal. Use --no-browser to suppress the attempt to open a browser on the server.

API-key capture sends the complete corpus directly to https://api.openai.com/v1. It uses the configured processing tier; the example and default are flex because reference capture is not latency-sensitive. Hosted-tool probes may create temporary files, vector stores, and containers; corpus-owned file and vector-store resources are deleted before the probe returns, including after response failures.

ChatGPT capture calls https://chatgpt.com/backend-api/codex with the required account header through an in-process compatibility transport. The transport converts string inputs to list form, aggregates mandatory upstream streams for non-streaming calls, forces store: false, reconstructs function-tool continuations without server-side storage, and presents Chat Completions over the Responses backend. A translated capture remains limited to capabilities exposed by that Codex endpoint; hosted-tool and resource APIs may therefore require direct API-key capture. Translated references carry "profile": "codex_translated"; direct API-key references carry "profile": "openai_api" so their provenance is never ambiguous.

OAuth credentials are stored with mode 0600 at ${OAI_LENS_AUTH_FILE:-~/.config/oai-lens/auth.json} and refreshed automatically. They are never written to reports, references, command arguments, or logs.

The default output is references/openai/YYYY-MM-DD.json; override it with CAPTURE_OUTPUT=path. Published snapshots are immutable, and capture refuses to write a partial reference if any probe fails.

For comparison, mount a reference read-only and set reference in config.yaml. Normal target runs require neither OpenAI credentials nor network access to OpenAI. Normalized IDs and timestamps are replaced while meaningful structure, ordering, event types, statuses, and content remain reviewable.

Development with uv

uv creates and manages the repository-local .venv; no manual python -m venv step is needed:

uv sync --frozen
uv run pytest
uv run ruff format --check .
uv run ruff check .
uv run pylint oai_lens tests

Both Ruff and Pylint are required quality gates. Ruff enables its full rule set, with only incompatible formatter rules and narrowly justified per-file exceptions. Pylint retains its defaults except for duplicate-code warnings on deliberately independent probe payloads.

The equivalent local target command is:

uv run oai-lens --config config.yaml test

Current scope

The corpus currently contains twenty-two independently reported SDK-level probes spanning core Responses and Chat Completions, strict structured output parsing, text and tool-call streaming, incomplete responses, stateless reasoning/output-item replay, single and multiple function continuation, negative error protocol behavior, hosted web/container/file tools, generated JavaScript execution, and vision transport variants. Reports preserve complete usage-detail schemas and derive non-negative, total-consistency, detail-bound, and cached-token-ratio observations without requiring candidate token counts to exactly equal OpenAI's tokenizer-dependent counts. Bulky text and binary inputs live in fixtures/; probes load and encode them at runtime rather than embedding generated data in source code.

The long-term target is first-class support for two intentionally distinct compatibility profiles: the classic public OpenAI API and the ChatGPT subscription-backed Codex protocol. A server will be testable against either profile explicitly, with each profile retaining its own request rules, supported capabilities, reference captures, and expected wire behaviour rather than being reduced to a misleading common subset. The current codex_translated capture is a bridge that produces OpenAI-shaped reference data from a subscription model; it remains labeled separately from direct openai_api captures while native Codex-profile candidate testing is developed.