diff --git a/README.md b/README.md index 504efdb..a277d4c 100644 --- a/README.md +++ b/README.md @@ -7,15 +7,15 @@ # TOKENFOLD -

More context. Fewer tokens. Exact by default.

+

Select what matters. Fit more context.

- Deterministic context compression for LLMs. Cut JSON tokens by up to 68% with exact recovery, or exceed 90% with opt-in, recoverable pruning. + Tokenfold Select brings query-aware relevance ranking to your LLM context. Fit complete context groups to a token budget, backed by Core's structural compression and inspectable receipts.

-

Tokenfold Core delivers model-free structural compression.

-

-Tokenfold Select ranks the context that matters most when token budgets get tight.

+

Python-first SDK. TypeScript alongside it. Rust at the core.

+

One fit() workflow for text, JSON, grouped context, and provider requests.

+

Select for relevance. Core for structural efficiency. One integrated workflow.


@@ -25,204 +25,246 @@ Tokenfold Select ranks the context that matters most when token budgets get tigh -[Install](#installation) | [Quick start](#quick-start) | [How it works](#how-it-works) | [MCP & agents](#coding-agents-and-mcp-integration) | [Core](#tokenfold-core) | [Select](#tokenfold-select) | [Benchmarks](#measured-results) | [Safety](#safety-and-auditability) +![Terminal demo: Python fit() code and verified 2,366 to 1,376 token savings from the first frame, followed by Core-first processing and optional Tokenfold Select. Select illustration only.](https://raw.githubusercontent.com/snchimata/tokenfold/main/docs/assets/tokenfold-demo.gif) -
- -## Installation +
+Demo scope and regeneration + +The 13-second FIT -> CORE -> SELECT loop returns directly to the code, with no +closing slide. It opens on the executed Python SDK `fit()` example and its +verified result: explicit folding reduces already-compact JSON from 2,366 to +1,376 tokens (**41.8%**), meeting a 1,500-token budget without dropping any of +the 30 records. The compact baseline exceeds that same budget. + +The following Core frame starts from the formatted API-response fixture: +3,812 to 1,376 tokens (**63.9%**), with exact JSON-value recovery. These are +different input representations, not interchangeable SDK and CLI baselines. +The workflow builds the Core baseline first; Select requires a still-over-budget, +eligible request, an approved runtime and explicit per-fit opt-in. Select is an +**illustration, not live inference**, and remains experimental and unqualified. +Fixture savings are not universal, and reversible data is not model-answer proof. + +The [renderer](scripts/render-demo.py) requires Pillow, a locally built CLI and +the current SDK installed in a Python >=3.12 environment. It checks the fixture +hash, Core counts, decode equality and SDK receipt before rendering: ```bash -pip install tokenfold # Python library -npm install tokenfold # Node.js / TypeScript library -cargo install tokenfold-cli # Standalone CLI -cargo add tokenfold-core # Rust library +python scripts/render-demo.py --sdk-python /path/to/sdk-env/python ``` -These installs provide Core. For optional query-aware model scoring, see -[Select setup](#tokenfold-select); its models and runtime setup are separate. - - - - - - - -
-
- Up to 96.3% fewer tokens (showcase) -

- Opt-in, recoverable pruning for heterogeneous array feeds - every dropped row stays fetchable. -

- 		-
- 45.6-67.6% on three fixtures, lossless -

- Reversible structural folding of repeated keys and columns on three bundled fixtures - verified by exact JSON-value recovery. -

- 		-
- Model-free & deterministic -

- No GPU, no semantic guesswork, no model in the Core path. A Rust engine with Python, Node.js, CLI and proxy interfaces. -

-
+The [VHS tape](docs/tokenfold-demo.tape) is a separate, Core-only terminal +recording and writes to `tmp/tokenfold-core-terminal.gif`. -> [!NOTE] -> **Built for:** Developers and AI teams sending structured JSON, tool definitions, and RAG feeds to LLMs. Start with a representative payload, inspect the receipt, and validate the result in your application. +
-
+## Table of contents -## How it works +[Python quickstart](#quick-start) | [Python fit workflow](#native-sdk-fitting) | [Select](#tokenfold-select) | [Benchmark highlights](#benchmark-highlights) | [Other integrations](#other-integrations) | [How it works](#how-it-works) | [Core](#tokenfold-core) | [Full results](#measured-results) | [Extended tooling](#extended-tooling) | [Safety and auditability](#safety-and-auditability) | [Reproduce the results](#reproduce-the-results) | [Contributing](#contributing) | [License](#license) -- **Core (default):** folds repeated structure without removing data. The - compact payload decodes to the original JSON value. -- **[Recoverable pruning](#recoverable-lossy-pruning) (opt-in):** model-free - rules rank eligible JSON rows; omissions are stored locally before replacement - with `$tf_ref` handles. This is the path behind the **96.3% showcase**, not Select. -- **[Select](#tokenfold-select) (separate):** a model scores groups against your - query; your allocator keeps selected original text under a token budget. - Select does not automatically store omissions or create recovery handles. +> **Built for:** Python application developers and AI teams fitting JSON, tool +> definitions, and grouped context into LLM requests. TypeScript, coding agents, +> CLI, proxy, and Rust integrations use the same underlying Core engine. -### What the model sees +## Quick start -```jsonc -// Input (conceptual illustration - 2 rows shown; small inputs like this -// pass through unchanged under the never-larger guard) -[{"id":1,"role":"admin"},{"id":2,"role":"member"}] +### Published Python SDK -// Tokenfold output on larger uniform arrays -> still JSON -{"__tf_cols__":["id","role"],"__tf_rows__":[[1,"admin"],[2,"member"]]} +Python >=3.12. Install the current SDK release; the base workflow needs no +repository checkout, model, GPU, or ML framework: + +```bash +python -m pip install "tokenfold==0.6.0" ``` -![Terminal demo: Tokenfold reduces the bundled API response from 3,812 to 1,376 tokens.](https://raw.githubusercontent.com/snchimata/tokenfold/main/docs/assets/tokenfold-demo.gif) + -
-How this demo was generated (VHS tape) +### Python fit workflow -The [VHS tape](docs/tokenfold-demo.tape) runs `inspect`, `compress`, and -`decode` against the bundled API response, including the **63.9%** receipt. -Run `vhs docs/tokenfold-demo.tape` to regenerate the GIF; the tape requires -`tokenfold` on `PATH`. +The SDK fits text, JSON, or grouped context to an explicit token ceiling. +Fitting is offline and model-free unless you explicitly configure and consent to +Select. No automatic sending, model downloads, or recoverable pruning occurs. -
+Default `Data(records)` renders standard compact JSON; opt into the folded +representation only for consumers you have validated: -Send the folded JSON directly to the model: column names label each value's -position in every row. Decode only when your application needs the original -object shape again. +```python +from tokenfold import Data, fit + +records = [ + {"id": i, "service": "checkout-api", "region": "us-east-1", + "status": 200, "latency_ms": 42 + i} + for i in range(1, 31) +] +result = fit(Data(records, representation="folded"), target_tokens=500) +print(f"{result.original_tokens} -> {result.final_tokens} tokens") +payload_text = result.require_met() +print(result.receipt.to_json()) +``` -#### Why Tokenfold, not LLMLingua / prompt compressors? +`require_met()` returns the payload only when the exact budget is met; otherwise +it raises `BudgetUnmetError` or `BudgetIndeterminateError` with the receipt. +Text/Data/Context return text; full provider requests return JSON objects. +An exact encoding count is not an exact provider-framing or billing count. +Explicit best-effort can accept an estimate without calling it exact. -- **Targets structured JSON, not prose.** Folds repeated keys and columns - mechanically rather than using semantic guesswork. -- **Model-free and deterministic.** Zero models running in Core and zero GPU - overhead. -- **Verified by exact decode.** Every lossless transform is verified by an - in-memory round trip before emitting; lossy pruning is strictly opt-in and - recoverable. +See the [Python example](examples/sdk_fit.py), [migration guide](docs/sdk-migration.md), +and [validation status](docs/sdk-validation-0.6.0.md). Advanced compression lives +under `tokenfold.lowlevel`. -#### How Tokenfold decides what to fold +For callers using the lower-level compression API, import it from +`tokenfold.lowlevel`: -```mermaid -%%{init: {"theme": "base", "themeVariables": {"primaryColor": "#f8fafc", "primaryTextColor": "#0f172a", "primaryBorderColor": "#94a3b8", "lineColor": "#64748b", "edgeLabelBackground": "#f8fafc", "fontFamily": "Arial", "fontSize": "15px"}, "flowchart": {"curve": "linear", "padding": 16, "nodeSpacing": 40, "rankSpacing": 32}}}%% -flowchart LR - accTitle: Lossless first, pruning only when opted in - accDescr: Build a verified lossless baseline. Consider pruning only when opted in, eligible, and still needed. Store omissions before removal and use pruning only if it beats the lossless baseline; otherwise keep that baseline. - Lossless["Build lossless JSON baseline
Verify folds; retain savings"] --> OptIn{"Pruning enabled,
eligible and needed?"} - OptIn -->|No| Keep("Lossless payload
+ receipt") - OptIn -->|Yes| Prune["Rank eligible rows
Store before removal"] - Prune -->|No further savings| Keep - Prune -->|Fewer tokens than baseline| Emit("Pruned payload
+ markers and receipt") +```python +import json +from tokenfold.lowlevel import InputFormat, compress, decode + +records = [ + {"id": i, "service": "checkout-api", "region": "us-east-1", + "status": 200, "latency_ms": 42 + i} + for i in range(1, 31) +] +raw_bytes = json.dumps(records).encode("utf-8") +result = compress(raw_bytes, format=InputFormat.JSON) +print(f"{result.report.original_tokens} -> {result.report.compressed_tokens} tokens") +assert json.loads(decode(result.payload)) == records +``` - classDef core fill:#eff6ff,stroke:#2563eb,stroke-width:2px,color:#1e3a8a - classDef optional fill:#f8fafc,stroke:#64748b,stroke-dasharray:5 4,color:#334155 - class Lossless,Keep core - class Prune optional + + +## Tokenfold Select + +**When structure ends, rank what matters.** + +[Tokenfold Select v3][tokenfold-select] is an Apache-2.0, **149.6M-parameter** +joint-group Ettin scorer, integrated into the SDK's `fit()` workflow. +Core remains model-free and neither loads nor invokes Select. +The SDK uses the lightweight maintained [tokenfold-select-runtime](packages/tokenfold-select-runtime/README.md), +not the superseded v2 package or an implicit direct-bundle loader. Model files are +acquired separately at an immutable revision; they do not ship in a clone or base SDK. +Use explicit CPU/FP32 or CUDA/BF16 setup and per-fit selection consent. Import and +already-fitting Core requests never load weights. Selection is experimental and +unqualified: grouping and required flags preserve caller-declared dependencies, +not independent proof of answer quality. No automatic recoverable pruning occurs. +See [provisioning, accounting, and lifecycle](docs/sdk-migration.md). + +### Python Select workflow + +After [explicit runtime provisioning](docs/sdk-migration.md#explicit-select-provisioning-and-consent), +supply the resulting config and allow selection on each fit. Keep dependent +content in one group or mark it required: + +```python +from tokenfold import Context, Group, SelectRuntime, Tokenfold + +context = Context( + query="Which change caused the cache failure?", + groups=[ + Group("constraints", "Use only the supplied release evidence.", required=True), + Group("changes", "Commit a41f9c changed the Redis TTL to 0ms."), + Group("logs", "After that release, cache evictions spiked on shard-02."), + Group("notes", "Weekly planning notes cover unrelated roadmap items. " * 40), + ], +) +with Tokenfold(select=SelectRuntime("select-runtime.json", experimental=True)) as client: + result = client.fit(context, target_tokens=96, allow_selection=True) + print(result.receipt.to_json()) + fitted_context = result.require_met() + print("Selection applied:", result.selection_applied) + print("Kept:", result.groups_kept, "Dropped:", result.groups_dropped) ``` -
-Text version (screen readers / no-image fallback) +The full context deliberately exceeds the ceiling. A successful fit requires +selection; the example does not promise particular kept IDs or answer accuracy. +If scoring fails or cannot meet the budget, checked access rejects the overflow +rather than silently returning a successful fit. Already-fitting requests skip +scoring even when consent is present. -1. Build the lossless baseline using eligible transforms. Verify structural - folds with an exact decode round trip and keep only token reductions. -2. Consider pruning only when explicitly opted in, the payload is eligible, - and the lossless baseline has not already met a requested token target. -3. Rank eligible rows and store omissions locally before removing them. - Rows refused by storage stay inline. -4. Use the pruned payload only if it has fewer tokens than the lossless - baseline; otherwise keep the lossless baseline. -5. Emit the chosen payload and compression receipt. Pruned payloads include - retrieval markers for stored omissions. +## Benchmark highlights -
+Saved fixture results, not universal savings or downstream quality guarantees: -| Workload | Type | Core mode | Savings | -| --- | --- | --- | --- | -| API responses | 30-record JSON payload | Lossless | **63.9% fewer tokens** | -| Tool schemas | OpenAI tool-schema fixture (Criterion bench) | Lossless | **45.6% fewer tokens** | -| Repetitive JSON | 50-record payload | Lossless | **67.6% fewer tokens** | +| Scope | Measured result | Limits and evidence | +| --- | --- | --- | +| Core structural compression | **45.6-63.9% on two fixtures, lossless** | Tool schemas and the 30-record API response; exact JSON-value recovery | +| Core vs. Headroom | **6/6 fixture wins**, 13,071 -> 8,715 tokens (**33.3% saved**) | Default local generic-JSON paths; Tokenfold recovered 6/6 exactly, Headroom matched 4/6 | +| Tokenfold JSON vs. TOON 4.1.1 | **7/7 fixture wins**, 12,776 vs. 14,841 tokens | 13.9% fewer tokens than TOON in this corpus; both paths round-trip | +| Opt-in, recoverable pruning | **Up to 96.3% fewer tokens (showcase)** | A 100-row heterogeneous feed; omitted rows stored behind local `$tf_ref` handles, not lossless inline compression | -These figures use exact `o200k_base` counts in the balanced preset. Versioned -inputs and provenance live in [`tests/fixtures/readme_metrics.json`](tests/fixtures/readme_metrics.json). -[Recoverable pruning](#recoverable-lossy-pruning) is separate, opt-in extended tooling. +These structural counts use exact `o200k_base` accounting. Inputs and provenance +live in [the metric manifest](tests/fixtures/readme_metrics.json); see +[full comparisons and methods](#measured-results) and the +[pruning showcase](#recoverable-lossy-pruning). The historical **67.6%** result +is a 50-record CHANGELOG measurement without a versioned source fixture; it was +not reproduced in the SDK rerun and is not used in the headline. -
+The [saved downstream-answer diagnostic](#sdk-downstream-answer-diagnostic) +recorded **8/8 correct baseline answers vs. 6/8 with folded JSON**; Select matched +16/16 baseline answers on eight synthetic tasks with two trials each. These are +small development diagnostics, not representative quality or Select qualification. -## Quick start +## Other integrations -### Native SDK fitting (0.6.0) +Python is the primary SDK onboarding path. TypeScript is supported alongside it; +CLI, MCP, proxy, and Rust are alternative entry points, not Python prerequisites. -Python >=3.12 (`pip install tokenfold==0.6.0`) and Node >=22 -(`npm install tokenfold@0.6.0`). See the migration guide for breaking import changes. -Base fitting is offline and model-free, with no auto-send, model download or storage. + -```python -from tokenfold import fit -result = fit("Keep this text unchanged.", target_tokens=32) -payload = result.require_met() -print(result.receipt.to_json()) +### Interfaces + +| Interface | Install / setup | Primary use case | +| --- | --- | --- | +| Python | `python -m pip install "tokenfold==0.6.0"` | Native fitting SDK; start above | +| TypeScript / Node.js | `npm install tokenfold@0.6.0` | Native fitting; CLI utilities live under `tokenfold/cli` | +| MCP | `tokenfold mcp serve` | Coding agents and editors; install the CLI first | +| CLI | `cargo install tokenfold-cli` | Files, stdin, and shell workflows | +| Proxy | `cargo install tokenfold-proxy` | Existing OpenAI-compatible clients | +| Rust | `cargo add tokenfold-core` | Native embedding | + +### TypeScript / Node.js + +Node >=22. Install the published SDK package. Core fitting runs in-process; +the base install needs neither Python nor a source build: + +```sh +npm install tokenfold@0.6.0 ``` +See the [Node example](examples/sdk_fit.mjs) and +[migration guide](docs/sdk-migration.md): + ```js import { fit } from "tokenfold"; const result = await fit("Keep this text unchanged.", { targetTokens: 32 }); const payload = result.requireMet(); ``` -Text/Data/Context return text; full provider requests return JSON objects. Provider -framing is not an encoding count: exact checked access rejects unavailable/estimated -budgets. Explicit best-effort can accept an estimate without calling it exact. -See [SDK migration](docs/sdk-migration.md), [Python example](examples/sdk_fit.py), -[Node example](examples/sdk_fit.mjs), and [validation status](docs/sdk-validation-0.6.0.md). -Advanced compression remains under `tokenfold.lowlevel` / `tokenfold/cli`. - -### Python: compress JSON directly +### Coding agents and MCP integration -After installing Tokenfold, try this self-contained example. Thirty uniform -records provide enough repeated structure to demonstrate lossless folding: +Coding agents can exhaust context limits on large shell outputs. Tokenfold runs as a +stdio MCP server exposing `tokenfold_compress`, `tokenfold_inspect`, +`tokenfold_retrieve`, and `tokenfold_stats`, with pre-configured filters for Git +status, diffs, build output, and test logs before they consume agent context. +`tokenfold init` currently supports Claude Code; Codex uses manual MCP +registration: -```python -import json -from tokenfold.lowlevel import InputFormat, compress +```bash +tokenfold init --agent claude-code +tokenfold doctor --agent claude-code -records = [{"id": i, "role": "member", "status": "active"} for i in range(30)] -result = compress(json.dumps(records), format=InputFormat.JSON) -print(f"{result.report.original_tokens} -> {result.report.compressed_tokens} tokens " - f"({result.saved_pct():.1f}% saved)") +# Or connect manually in Codex: +codex mcp add tokenfold -- tokenfold mcp serve ``` -Send `result.payload` (bytes) to the model. Use `tokenfold.lowlevel.decode` when your -application needs the original JSON shape again. Small or irregular inputs may -pass through unchanged under the never-larger guard. - -### CLI: inspect and compress a file - -Install the CLI with `cargo install tokenfold-cli` or use a -[checksummed GitHub Release](https://github.com/snchimata/tokenfold/releases/latest), then: +`tokenfold init --agent claude-code` merges a project-scoped `.mcp.json` entry without replacing +other servers; `tokenfold doctor --agent claude-code` verifies it. `tokenfold_inspect` +is side-effect-free (never returns a modified payload); `tokenfold_retrieve` +retrieves by hash or marker (report references are reserved but not yet +resolvable). See +[`docs/configuration.md`](docs/configuration.md) for tested MCP JSON/TOML and every environment +override. Trusted filters for Git, build, and test output +are available through `tokenfold filters list`. -```bash -tokenfold inspect payload.json --format json -tokenfold compress payload.json --format json --output payload.compact.json -``` +
### Drop-in proxy for OpenAI-compatible clients @@ -294,212 +336,166 @@ sequenceDiagram P-->>-C: Response ``` -### Interfaces +### CLI and low-level Python -| Interface | 1-line install / command | Primary use case | -| --- | --- | --- | -| Proxy | `cargo install tokenfold-proxy` | Existing OpenAI-compatible clients | -| CLI | `cargo install tokenfold-cli` | Files, stdin, and shell workflows | -| Python | `pip install tokenfold` | Application pipelines | -| TypeScript | `npm install tokenfold` | Node.js applications and automation | -| Rust | `cargo add tokenfold-core` | Native embedding | -| MCP | `tokenfold mcp serve` | Agents and editors | +For files and stdin, install the CLI with `cargo install tokenfold-cli` or use a +[checksummed GitHub Release](https://github.com/snchimata/tokenfold/releases/latest): -### Presets +```bash +tokenfold inspect payload.json --format json +tokenfold compress payload.json --format json --output payload.compact.json +``` -| Preset | Behavior | -| --- | --- | -| `conservative` | Minification only; no restructuring of what the model sees. | -| `balanced` (default) | Minification plus reversible columnar folding (`json_field_fold`, `json_value_dict`, `log_field_fold`) on eligible formats. | -| `aggressive` | Same default transform set as `balanced` today. | +For provider-shaped JSON, use the low-level Python API from `tokenfold.lowlevel`: -Provider message formats (`openai_json`, `anthropic_json`) use minification -only; columnar folding applies to generic JSON (`--format json`). The default -proxy path follows this rule. Explicit `--observations` enables lossless -compression of eligible JSON tool-result strings on `/v1/chat/completions` -without folding the provider envelope; see [observation compression](docs/configuration.md#restart-safe-observation-commitments). +```python +from tokenfold.lowlevel import InputFormat, compress -> [!NOTE] -> `schema_compaction` is not enabled by any preset. Unlike the default -> transforms, it truncates `examples` arrays and is not reversible. +# Accepts raw JSON bytes or str (messages, tools, or schemas) +request_json_bytes = b'{"messages": [{"role": "user", "content": "hello"}]}' +result = compress(request_json_bytes, format=InputFormat.OPENAI_JSON) + +# For standalone JSON records, use: format=InputFormat.JSON +compressed_payload = result.payload # bytes +print(f"Saved {result.report.saved_tokens} tokens ({result.saved_pct():.1f}%)") +``` ### Runnable examples (Python >=3.12, Node.js >=22, Rust >=1.98) -Runnable examples, one per surface, all under [`examples/`](examples): +After building and installing the SDK, run the examples under [`examples/`](examples): ```bash -python examples/sdk_fit.py # Python: native fit, checked payload, fit receipt -node examples/sdk_fit.mjs # Node: native fit, checked payload, fit receipt -python examples/quickstart.py # Advanced Python lowlevel compression and decoding -node examples/quickstart.mjs # Advanced Node CLI compression and decoding +python examples/quickstart.py # Python: compress messages, request bodies, JSON data +node examples/quickstart.mjs # TypeScript/Node: compress, inspect, read the receipt cargo run -p tokenfold-core --example quickstart # Rust: the embedded core API python examples/lossy_pruning.py # CLI: opt-in recoverable pruning, end to end -python examples/contextdoctor_report.py # source checkout: offline outcome/regression controls ``` [`examples/quickstart.ipynb`](examples/quickstart.ipynb) is the guided tour of the whole Python surface: everything `quickstart.py` shows, plus previews, budgets and presets, -provider payloads, recoverable pruning previews, and the offline ContextDoctor controls. -Its separately provisioned Select v3 CUDA example is explicitly enabled; set -`RUN_SELECT_V3 = False` for a core-only tour. See the [evaluation guide](eval/README.md) -for ContextDoctor's source-only interface and evidence limits. +provider payloads, recoverable pruning with retrieval, and where Tokenfold Select fits. -### Coding agents and MCP integration +## How it works -Coding agents can exhaust context limits on large shell outputs. Tokenfold runs as a -stdio MCP server exposing `tokenfold_compress`, `tokenfold_inspect`, -`tokenfold_retrieve`, and `tokenfold_stats`, with pre-configured filters for Git -status, diffs, build output, and test logs before they consume agent context. -`tokenfold init` currently supports Claude Code; Codex uses manual MCP -registration: +### What the model sees -```bash -tokenfold init --agent claude-code -tokenfold doctor --agent claude-code +```json +// Input (conceptual illustration - 2 rows shown; small inputs like this +// pass through unchanged under the never-larger guard) +[{"id":1,"role":"admin"},{"id":2,"role":"member"}] -# Or connect manually in Codex: -codex mcp add tokenfold -- tokenfold mcp serve +// Tokenfold output on larger uniform arrays -> still JSON +{"__tf_cols__":["id","role"],"__tf_rows__":[[1,"admin"],[2,"member"]]} ``` -`tokenfold init --agent claude-code` merges a project-scoped `.mcp.json` entry without replacing -other servers; `tokenfold doctor --agent claude-code` verifies it. `tokenfold_inspect` -is side-effect-free (never returns a modified payload); `tokenfold_retrieve` -retrieves by hash or marker (report references are reserved but not yet -resolvable). See -[`docs/configuration.md`](docs/configuration.md) for tested MCP JSON/TOML and every environment -override. Trusted filters for Git, build, and test output -are available through `tokenfold filters list`. - -
- -## Tokenfold Core - -### Structural, not semantic compression - -Semantic prompt compressors such as [LLMLingua](https://github.com/microsoft/LLMLingua) use a smaller language model to identify and remove less-important prompt tokens. -Tokenfold solves a different problem first: repeated keys, values, schemas, rows, and other structural redundancy. Core needs no model, is deterministic, and verifies an exact decode before accepting a lossless transform. Core does not inject prompt guidance or fine-tune models. +Column names label each value's position in every row. Use folded JSON only +with a consumer you have validated for that representation; decode when your +application needs the original object shape. The SDK's default `Data` +representation keeps standard compact JSON. -[Tokenfold Select](#tokenfold-select) is Tokenfold's complementary query-aware offering: when structure ends, it ranks what matters under a token budget. +#### Why Tokenfold Core for structured payloads? -
- - - -## Tokenfold Select - -**When structure ends, rank what matters.** - -[Tokenfold Select v3][tokenfold-select] is an Apache-2.0, **149.6M-parameter** -joint-group Ettin scorer. Core remains model-free and neither loads nor invokes Select. -The 0.6 SDK uses the lightweight maintained [tokenfold-select-runtime](packages/tokenfold-select-runtime/README.md), -not the superseded v2 package or an implicit direct-bundle loader. Model files are -acquired separately at an immutable revision; they do not ship in a clone or base SDK. -Use explicit CPU/FP32 or CUDA/BF16 setup and per-fit selection consent. Import and -already-fitting Core requests never load weights. Selection is experimental and -unqualified: grouping and required flags preserve caller-declared dependencies, -not independent proof of answer quality. No automatic recoverable pruning occurs. -See [provisioning, accounting, and lifecycle](docs/sdk-migration.md). - -### Tokenfold Select v3 benchmarks +- **Targets structured JSON, not prose.** Folds repeated keys and columns + mechanically rather than using semantic guesswork. +- **Model-free and deterministic.** Zero models running in Core and zero GPU + overhead. +- **Verified by exact decode.** Every lossless transform is verified by an + in-memory round trip before emitting; lossy pruning is strictly opt-in and + recoverable. -Saved results from the [published v3 benchmark snapshot](https://huggingface.co/snchimata/tokenfold-select-v3/blob/317d3424d3de78c1a708c42a47a7812ab877f1a8/README.md), -pinned to model-card revision `317d3424d3de78c1a708c42a47a7812ab877f1a8`. -These measurements include the scorer **and whole-group allocator**, not the -scoring model alone. Core's lossless benchmarks remain separate below. +#### How Tokenfold decides what to fold -#### Context-token reduction at a 50% budget +```mermaid +%%{init: {"theme": "base", "themeVariables": {"primaryColor": "#f8fafc", "primaryTextColor": "#0f172a", "primaryBorderColor": "#94a3b8", "lineColor": "#64748b", "edgeLabelBackground": "#f8fafc", "fontFamily": "Arial", "fontSize": "15px"}, "flowchart": {"curve": "linear", "padding": 16, "nodeSpacing": 40, "rankSpacing": 32}}}%% +flowchart LR + accTitle: Lossless first, pruning only when opted in + accDescr: Build a verified lossless baseline. Consider pruning only when opted in, eligible, and still needed. Store omissions before removal and use pruning only if it beats the lossless baseline; otherwise keep that baseline. + Lossless["Build lossless JSON baseline
Verify folds; retain savings"] --> OptIn{"Pruning enabled,
eligible and needed?"} + OptIn -->|No| Keep("Lossless payload
+ receipt") + OptIn -->|Yes| Prune["Rank eligible rows
Store before removal"] + Prune -->|No further savings| Keep + Prune -->|Fewer tokens than baseline| Emit("Pruned payload
+ markers and receipt") -Select v3 removed **50.0-60.0%** of the original context tokens in these five -saved development examples. Both models had the same exact `o200k_base` output -cap for each workload; retained V3 groups were copied without rewriting. + classDef core fill:#eff6ff,stroke:#2563eb,stroke-width:2px,color:#1e3a8a + classDef optional fill:#f8fafc,stroke:#64748b,stroke-dasharray:5 4,color:#334155 + class Lossless,Keep core + class Prune optional +``` -| Workload | Original tokens | Select v3 output | V3 reduction | Kompress v2 output | -| --- | ---: | ---: | ---: | ---: | -| Code | 4,411 | 2,205 | **50.0%** | 2,149 | -| JSON | 1,303 | 651 | **50.0%** | Refused; 1,303 raw | -| Logs | 1,103 | 503 | **54.4%** | Refused; 1,103 raw | -| Multi-turn | 587 | 235 | **60.0%** | 284 | -| Retrieval | 4,498 | 2,217 | **50.7%** | 2,219 | +
+Text version (screen readers / no-image fallback) -One exposed source per workload; JSON and logs share a source population. -Token reduction does not establish answer accuracy or semantic preservation. -Six of ten historical Kompress cells refused the budget across the 25%/50% -screen; refusals and raw fallbacks are **inconclusive, not V3 wins**. The -[full ten-cell history](https://huggingface.co/snchimata/tokenfold-select-v3/blob/317d3424d3de78c1a708c42a47a7812ab877f1a8/README.md#benchmark-methodology) -retains every result. +1. Build the lossless baseline using eligible transforms. Verify structural + folds with an exact decode round trip and keep only token reductions. +2. Consider pruning only when explicitly opted in, the payload is eligible, + and the lossless baseline has not already met a requested token target. +3. Rank eligible rows and store omissions locally before removing them. + Rows refused by storage stay inline. +4. Use the pruned payload only if it has fewer tokens than the lossless + baseline; otherwise keep the lossless baseline. +5. Emit the chosen payload and compression receipt. Pruned payloads include + retrieval markers for stored omissions. -#### Resident-warm complete-request latency at a 50% budget +
-| Workload | Select v3 ms | Kompress v2 ms | V3 time difference | Measurement basis | -| --- | ---: | ---: | ---: | --- | -| Code | 146.088 | 200.304 | -27.1% | One historical request per model | -| Retrieval | 139.291 | 201.564 | -30.9% | One historical request per model | -| Multi-turn | 66.572 | 77.751 | -14.4% | Median of 5 requests per model | +## Tokenfold Core -Resident-warm RTX 5080 research pipelines, FP32 parameters/BF16 compute; -startup excluded. Separate time blocks, not controlled-idle or paired trials. -These are descriptive development observations, **not representative quality, population p50/p95, or superiority qualification**. +### Structural, not semantic compression -
-Benchmark methodology and preserved history +Core removes structural redundancy, not prose selected by a language model. +See [what the model sees](#what-the-model-sees) for the columnar JSON format and +the verified lossless-first pipeline. Unlike semantic prompt compressors such +as [LLMLingua](https://github.com/microsoft/LLMLingua), Core needs no model or GPU, +does not inject prompt guidance, and does not fine-tune models. -RTX 5080, FP32 parameters/BF16 compute, cache off, TF32 off, one Torch thread. -V3 used the serialized SDPA research runner; Kompress v2 used native EAGER. -Complete-request clocks include grouping, counting, IPC, scoring, allocation, -final recount and applicable teardown, but **exclude process/model startup and -conditioning**. They are not cold-CLI or quickstart-SDK latency measurements. +[Tokenfold Select](#tokenfold-select) complements Core when query-aware group +selection is explicitly enabled. -The models ran in separate time blocks. The five-request multi-turn repeat was -not controlled-idle: the GPU-busy admission gate was explicitly waived, with -pre-model busy observations of 25% and 38%. All five samples per model are -retained. The earlier multi-turn result was **69.362 ms vs 64.411 ms** (V3 time -7.7% higher), and remains in the history rather than being replaced by the -repeat's sample medians. +### Presets -The shared synthetic publisher-data pilot has incomplete comparator/answer -evaluation; its literal-substring checks are not answer accuracy. The separate -[partial answer-lookup evaluation](https://huggingface.co/snchimata/tokenfold-select-v3/blob/317d3424d3de78c1a708c42a47a7812ab877f1a8/QUALITY-EVAL.json) -also remains incomplete, with no overall quality score or qualified 90% -comparability claim. +| Preset | Behavior | +| --- | --- | +| `conservative` | Minification only; no restructuring of what the model sees. | +| `balanced` (default) | Minification plus reversible columnar folding (`json_field_fold`, `json_value_dict`, `log_field_fold`) on eligible formats. | +| `aggressive` | Same default transform set as `balanced` today. | -Sources: [exact counts, timings, all repeat samples and refusals](https://huggingface.co/snchimata/tokenfold-select-v3/blob/317d3424d3de78c1a708c42a47a7812ab877f1a8/COMPARISON.json), -[validation](https://huggingface.co/snchimata/tokenfold-select-v3/blob/317d3424d3de78c1a708c42a47a7812ab877f1a8/VALIDATION.json), and -[full methodology](https://huggingface.co/snchimata/tokenfold-select-v3/blob/317d3424d3de78c1a708c42a47a7812ab877f1a8/README.md#benchmark-methodology). -
+Provider message formats (`openai_json`, `anthropic_json`) use minification +only; columnar folding applies to generic JSON (`--format json`). The default +proxy path follows this rule. Explicit `--observations` enables lossless +compression of eligible JSON tool-result strings on `/v1/chat/completions` +without folding the provider envelope; see [observation compression](docs/configuration.md#restart-safe-observation-commitments). -
-Historical Select v1 benchmarks (not v3 results) +> [!NOTE] +> `schema_compaction` is not enabled by any preset. Unlike the default +> transforms, it truncates `examples` arrays and is not reversible. -The original [Tokenfold Select v1][tokenfold-select-v1] model card reports the -following literal-answer-retention results against Kompress-v2 and BM25. -These are source-reported external results, not Core CI or v3 results. +## Measured results -| Tokens kept | Tokenfold Select | Kompress-v2 (Headroom) native | Kompress-v2 (Headroom) relevance | BM25 | vs. best baseline | -| ---: | ---: | ---: | ---: | ---: | ---: | -| 50% | **86.3%** | 66.5% | 80.5% | 79.4% | **+5.8 pp** | -| 25% | **70.3%** | 47.2% | 61.5% | 60.7% | **+8.8 pp** | -| 10% | **39.9%** | 30.4% | 37.7% | 37.7% | **+2.2 pp** | +### SDK downstream-answer diagnostic -In this v1 evaluation, Select beat every measured baseline at every budget. -At a 25% budget, the fine-tuned ranker keeps the answer **70.3% of the time**: -an **8.8 percentage-point** lead over the strongest baseline (Kompress-v2 relevance, 61.5%), and a -**15.8% relative lift** over BM25 (60.7% -> 70.3%). Critical-content survival is -**100% at every measured budget** through allocator force-keep. Results are -three-seed repeated subsampling over roughly 73,000 training and 24,000 -held-out fixtures per run; the [v1 model card][tokenfold-select-v1] publishes the -training recipe and full baseline set. Here, task success means the literal -gold answer survived selection under the stated budget. -These are source-reported external results, pinned by model revision in the -[metric manifest](tests/fixtures/readme_metrics.json); Core CI does not reproduce them. +On 2026-11-10, one free hosted model (`nvidia/nemotron-3.5-super-vl:free`, +Nvidia provider) answered 12 predeclared tasks twice per arm, with counterbalanced +order and exact typed-JSON grading. The 48 completed quality calls measured +answers, not just whether answer text survived compression: -See the [Select v1 model card][tokenfold-select-v1] for its setup, evaluation, -training data, and limitations. +| Path | Distinct tasks x trials | Baseline correct | Candidate correct | Provider prompt tokens | +| --- | ---: | ---: | ---: | ---: | +| Explicit folded JSON vs. compact JSON | 4 x 2 | 8/8 | **6/8** | 24,496 -> 16,600 (**32.2% fewer**) | +| Select at a 50% rendered-text budget vs. full context | 8 x 2 | 16/16 | 16/16 | 5,706 -> 3,416 (**40.1% fewer**) | -
+**Exact recovery did not guarantee correct model answers:** both folded-JSON +failures misread the same nested API record's plan tier and seat count. Default +SDK `Data` uses compact JSON; folding requires explicit opt-in. Select matched +its baseline on these eight synthetic tasks, not a representative accuracy test. +Repeated trials are not additional independent tasks; Select remains unqualified. -
+The first model scope stopped on a rate-limited probe, with all 48 quality cells +unexecuted; it is retained, not scored as 0% accuracy or silently retried. +See the [full method, failures and limits](docs/sdk-benchmarks-0.6.0.md), +[frozen prompts and oracles](eval/research/sdk_quality_corpus.json), and +[all-attempt results](eval/research/sdk_quality_results.json). -## Measured results +### Lossless corpus counts Exact `o200k_base` token counts, original input versus Tokenfold's lossless output across the six-fixture Headroom corpus: @@ -604,6 +600,106 @@ and [reproduction command](eval/research/README.md).
+### Tokenfold Select v3 benchmarks + +Saved results from the [published v3 benchmark snapshot](https://huggingface.co/snchimata/tokenfold-select-v3/blob/317d3424d3de78c1a708c42a47a7812ab877f1a8/README.md), +pinned to model-card revision `317d3424d3de78c1a708c42a47a7812ab877f1a8`. +These measurements include the scorer **and whole-group allocator**, not the +scoring model alone. Core's lossless benchmarks are separate above. + +#### Context-token reduction at a 50% budget + +Select v3 removed **50.0-60.0%** of the original context tokens in these five +saved development examples. Both models had the same exact `o200k_base` output +cap for each workload; retained V3 groups were copied without rewriting. + +| Workload | Original tokens | Select v3 output | V3 reduction | Kompress v2 output | +| --- | ---: | ---: | ---: | ---: | +| Code | 4,411 | 2,205 | **50.0%** | 2,149 | +| JSON | 1,303 | 651 | **50.0%** | Refused; 1,303 raw | +| Logs | 1,103 | 503 | **54.4%** | Refused; 1,103 raw | +| Multi-turn | 587 | 235 | **60.0%** | 284 | +| Retrieval | 4,498 | 2,217 | **50.7%** | 2,219 | + +One exposed source per workload; JSON and logs share a source population. +Token reduction does not establish answer accuracy or semantic preservation. +Six of ten historical Kompress cells refused the budget across the 25%/50% +screen; refusals and raw fallbacks are **inconclusive, not V3 wins**. The +[full ten-cell history](https://huggingface.co/snchimata/tokenfold-select-v3/blob/317d3424d3de78c1a708c42a47a7812ab877f1a8/README.md#benchmark-methodology) +retains every result. + +#### Resident-warm complete-request latency at a 50% budget + +| Workload | Select v3 ms | Kompress v2 ms | V3 time difference | +| --- | ---: | ---: | ---: | +| Code | 146.088 | 200.304 | -27.1% | +| Retrieval | 139.291 | 201.564 | -30.9% | +| Multi-turn | 66.572 | 77.751 | -14.4% | + +Resident-warm RTX 5080 research pipelines, FP32 parameters/BF16 compute; +startup excluded. Code and retrieval each use one historical request per model; +multi-turn uses the median of five. Separate time blocks, not controlled-idle or paired trials. +These are descriptive development observations, **not representative quality, population p50/p95, or superiority qualification**. + +
+Benchmark methodology and preserved history + +RTX 5080, FP32 parameters/BF16 compute, cache off, TF32 off, one Torch thread. +V3 used the serialized SDPA research runner; Kompress v2 used native EAGER. +Complete-request clocks include grouping, counting, IPC, scoring, allocation, +final recount and applicable teardown, but **exclude process/model startup and +conditioning**. They are not cold-CLI or quickstart-SDK latency measurements. + +The models ran in separate time blocks. The five-request multi-turn repeat was +not controlled-idle: the GPU-busy admission gate was explicitly waived, with +pre-model busy observations of 25% and 38%. All five samples per model are +retained. The earlier multi-turn result was **69.362 ms vs 64.411 ms** (V3 time +7.7% higher), and remains in the history rather than being replaced by the +repeat's sample medians. + +The shared synthetic publisher-data pilot has incomplete comparator/answer +evaluation; its literal-substring checks are not answer accuracy. The separate +[partial answer-lookup evaluation](https://huggingface.co/snchimata/tokenfold-select-v3/blob/317d3424d3de78c1a708c42a47a7812ab877f1a8/QUALITY-EVAL.json) +also remains incomplete, with no overall quality score or qualified 90% +comparability claim. + +Sources: [exact counts, timings, all repeat samples and refusals](https://huggingface.co/snchimata/tokenfold-select-v3/blob/317d3424d3de78c1a708c42a47a7812ab877f1a8/COMPARISON.json), +[validation](https://huggingface.co/snchimata/tokenfold-select-v3/blob/317d3424d3de78c1a708c42a47a7812ab877f1a8/VALIDATION.json), and +[full methodology](https://huggingface.co/snchimata/tokenfold-select-v3/blob/317d3424d3de78c1a708c42a47a7812ab877f1a8/README.md#benchmark-methodology). +
+ +
+Historical Select v1 benchmarks (not v3 results) + +The original [Tokenfold Select v1][tokenfold-select-v1] model card reports the +following literal-answer-retention results against Kompress-v2 and BM25. +These are source-reported external results, not Core CI or v3 results. + +| Tokens kept | Tokenfold Select | Kompress-v2 (Headroom) native | Kompress-v2 (Headroom) relevance | BM25 | vs. best baseline | +| ---: | ---: | ---: | ---: | ---: | ---: | +| 50% | **86.3%** | 66.5% | 80.5% | 79.4% | **+5.8 pp** | +| 25% | **70.3%** | 47.2% | 61.5% | 60.7% | **+8.8 pp** | +| 10% | **39.9%** | 30.4% | 37.7% | 37.7% | **+2.2 pp** | + +In this v1 evaluation, Select beat every measured baseline at every budget. +At a 25% budget, the fine-tuned ranker keeps the answer **70.3% of the time**: +an **8.8 percentage-point** lead over the strongest baseline (Kompress-v2 relevance, 61.5%), and a +**15.8% relative lift** over BM25 (60.7% -> 70.3%). Critical-content survival is +**100% at every measured budget** through allocator force-keep. Results are +three-seed repeated subsampling over roughly 73,000 training and 24,000 +held-out fixtures per run; the [v1 model card][tokenfold-select-v1] publishes the +training recipe and full baseline set. Here, task success means the literal +gold answer survived selection under the stated budget. +These are source-reported external results, pinned by model revision in the +[metric manifest](tests/fixtures/readme_metrics.json); Core CI does not reproduce them. + +See the [Select v1 model card][tokenfold-select-v1] for its setup, evaluation, +training data, and limitations. + +
+ +
+ ## Extended tooling These additional capabilities extend Core with recoverable pruning and an @@ -624,7 +720,7 @@ pruning when strict context budgets apply: | --- | --- | --- | | **What it does** | Minifies, folds repeated keys into columns, and stores repeated values once | Adds deterministic ranking and retrieval markers for selected array rows | | **Best on** | Uniform records, schemas, logs, diffs | Search results, mixed event feeds, agent traces, long arrays | -| **Measured here** | **45.6-67.6%** fewer tokens on three bundled fixtures | **51.7-96.3%** fewer tokens (40-event feed at 51.7-68.2%; 100-row showcase at 96.3%) | +| **Measured here** | **45.6-63.9%** fewer tokens on two versioned fixtures | **51.7-96.3%** fewer tokens (40-event feed at 51.7-68.2%; 100-row showcase at 96.3%) | | **Recovery** | All data remains in the payload | Every emitted marker resolves through the local retrieval store | @@ -691,8 +787,9 @@ tokenfold inspect examples/incident_feed.json --format json \ --prune --keep-ratio 0.35 ``` -The same flags, and the same fail-closed contract, from Python and TypeScript. -Use a throwaway store directory while experimenting and reuse it for retrieval: +The same flags and fail-closed contract are available through the low-level +bindings. Use a throwaway store directory while experimenting and reuse it for +retrieval: ```python import json diff --git a/docs/assets/tokenfold-demo.gif b/docs/assets/tokenfold-demo.gif index 648a5e9..d34fa0f 100644 Binary files a/docs/assets/tokenfold-demo.gif and b/docs/assets/tokenfold-demo.gif differ diff --git a/docs/sdk-benchmarks-0.6.0.md b/docs/sdk-benchmarks-0.6.0.md new file mode 100644 index 0000000..9b91d93 --- /dev/null +++ b/docs/sdk-benchmarks-0.6.0.md @@ -0,0 +1,154 @@ +# SDK 0.6.0 benchmark follow-up - 2026-11-10 + +**Development-only; Select remains unqualified.** These are local/development +observations, not representative accuracy, noninferiority, production readiness, +or a controlled before/after SDK speed comparison. + +## Downstream answers, not answer retention + +We froze four exposed repository JSON tasks and eight hand-authored synthetic +Select tasks before hosted answers. Two repetitions reversed baseline/candidate +order, using temperature 0 and seed 271. This is **12 distinct tasks**, not 24 +independent tasks. No judge model or generated code was executed: one final JSON +object had to match the predeclared oracle, including recursive value types. +Duplicate keys, extra fields, markdown, truncation and refusals fail grading. + +The hosted model ID was `nvidia/nemotron-3.5-super-vl:free`, restricted to the +`nvidia` provider with fallback disabled and zero price ceilings. Responses +reported provider `Nvidia`. The endpoint advertised a 2026-10-15 model name, but +served checkpoint bytes, fingerprint and service tier were unavailable. This is +an exact model-ID/provider pin, **not an immutable downstream checkpoint pin**. + +| Diagnostic | Baseline correct | Candidate correct | Baseline-only losses | Provider prompt tokens | +| --- | ---: | ---: | ---: | ---: | +| Core: 4 fixtures x 2 trials, compact vs. explicit folded JSON | 8/8 | 6/8 | 2 | 24,496 -> 16,600 (-32.2%) | +| Select: 8 synthetic tasks x 2 trials, full vs. 50% rendered-text cap | 16/16 | 16/16 | 0 | 5,706 -> 3,416 (-40.1%) | + +Both Core losses occurred on the same API-record lookup, once in each repetition: +the baseline correctly returned plan `free`, seats `1`; the folded-input answer +returned plan `team`, seats `25`. The name and active flag were correct. The +folded JSON still decoded exactly to the original value. **Codec reversibility +is not model-answer equivalence**, even with a shared explanation of column and +dictionary frames. Default SDK `Data` remains compact JSON; folding is opt-in. + +Select's tasks cover code, logs, retrieval, a flattened conversation and an +unknown-value abstention. Supporting groups occupy positions 0, 1, 2, 3, 4, 0, +1, 2 rather than always appearing first. Dependencies were explicitly bundled by +the fixture author. All eight candidates scored, selected, reconstructed complete +groups in source order and met their exact local targets. This does not test +dependency discovery, required-group protection or provider-role conversation +compaction; all groups were optional. It does not establish general quality. + +Before any hosted answers, a separate review pass corrected an omitted schema +oracle field and a first-group placement bias. Earlier local preparations were +retained. These development revisions are another reason not to label this an +untouched held-out qualification set. No prompt, oracle, target or selected +payload was changed after the hosted answers, including the two failures. + +## Accounting and failed scopes + +There were **50 actual generation attempts**, with no retries or automatic +provider fallback: + +1. A `google/gemma-4-31b-it:free` / `google-ai-studio` connectivity probe returned + HTTP 429 from the shared upstream pool. That scope stopped; all 48 planned + quality cells remained unexecuted. Its usage and charge are unknown, not zero. +2. A separately frozen and announced NVIDIA scope completed one connectivity + probe plus all 48 quality calls. All 49 replies ended with `stop`, with no + BYOK, refusal or nonzero reported charge. The probe is excluded from accuracy + and paired token tables, but included in all-call accounting. + +NVIDIA reported 50,256 prompt + 749 completion = **51,005 tokens including its +probe**. This is not a complete campaign total because the failed probe's usage +is unknown. Request pricing was unreported, and provider-reported zero charges +are not audited invoices or complete billing savings. No dollar-saving ratio is +claimed. The older 12 HTTP-400 attempts described in +[SDK validation](sdk-validation-0.6.0.md) remain separate historical evidence. + +Provider prompt counts include the full submitted messages and shared formatting +instructions. They are not local `o200k_base` counts: + +| Quality-call totals | Local rendered baseline -> candidate | Provider completion baseline -> candidate | +| --- | ---: | ---: | +| Core | 18,988 -> 12,044 (-36.6%) | 202 -> 204 | +| Select | 3,786 -> 1,694 (-55.3%) | 168 -> 169 | + +Completion tokens did not decrease. All wrong answers remain in these totals. +Core reported 4,336 cached input tokens per arm; Select reported zero. Cache +and free-provider load were not controlled. Client-call medians, including the +isolated HTTP worker but excluding SDK preparation, were 761.95 -> 849.92 ms +for Core and 608.27 -> 603.04 ms for Select. **No latency improvement is claimed.** +The 900-second limit was a dispatch deadline plus child cleanup, not a strict +whole-process lifetime guarantee. + +## Other benchmark and correctness checks + +The preceding Core rerun reproduced all seven published structural output counts, +with exact JSON-value recovery; the six-fixture corpus remained 13,071 -> 8,715 +tokens. Tokenfold vs. TOON remained 12,776 vs. 14,841 tokens, both round-tripping +all seven fixtures. These local JSON paths are not hosted provider comparisons. +Fresh pinned-source Headroom output totaled 12,819 rather than historical 12,807 +tokens (+10 incident-feed, +2 OpenAI payload), still value-equal on 4/6 cases. +The README's existing comparator tables retain their historical source snapshot; +this small environment-dependent drift is not attributed to the SDK. + +Explicit SDK folded Data totaled 12,772 instead of 12,776: four fewer tokens from +two `0.0` -> `0` normalizations in one report fixture, not better structural +folding. The opt-in pruning showcase remained 304,032 -> 11,364 tokens (96.3%), +with retrieval/preserve/fallback assertions passing; it remains unvalidated +lossy evidence. Tool-schema savings remained 45.63%. The historical 67.6% +repetitive-JSON claim lacks a versioned input and was not reconstructed. + +The release-Rust benchmark gate passed: command p95 449.972 ms, structured JSON +p95 1469.547 ms, total allocation 1,580,507,977 bytes/call, savings ratio 0.4563. +Historical threshold baselines used another compiler/time block; lower observed +latency/allocation is not causal SDK speedup evidence. Total allocation is not +peak resident memory. Folded SDK performance was mixed across fixtures, including +a slower incident-feed path; there is no universal speedup claim. + +Fresh local checks on Windows x64, Rust 1.98, Python 3.12 and Node 22 passed: +`cargo fmt --all --check`, workspace Clippy with denied warnings, 773 Rust tests +(two explicit real-worker/model entrypoints ignored), the installed Python suite, +43 Node tests with zero skips, and the deterministic fidelity smoke. These are +not cross-platform or exact-head hosted CI receipts. Initial test runs lacked +Pyright and explicit CLI/Python test paths; those failures were retained and +resolved by supplying the existing test prerequisites, without product edits. + +## Evidence and reproduction + +- [Frozen corpus](../eval/research/sdk_quality_corpus.json): questions, typed + oracles, original source paths/hashes, balanced groups, exact submitted message + arrays, common request template, targets and selection exposure. +- [Curated result ledger](../eval/research/sdk_quality_results.json): every + quality reply including both failures, probe outcomes, prior unexecuted cells, + provider usage, cached tokens, timings, charge limitations and source/method + hashes. Private request/account IDs and credentials are not published. +- [README metric pins](../tests/fixtures/readme_metrics.json): canonical-JSON + SHA-256 digests and the exact displayed quality counts. + +Offline evidence reconciliation (no model, weights or network): + +```sh +python -m pytest python-tests/test_sdk_quality_evidence.py -q +``` + +To replay hosted answers, use a **fresh, explicitly bounded scope**, verify the +current endpoint's zero prices and supported parameters, and compose each body +from the corpus's `protocol.request_template` plus that arm's `messages`. +Follow its two-trial order and 48-call maximum, with at most one separate probe, +1024 output tokens, no retry/fallback, durable dispatch/terminal accounting and +the same exact grading. The existing +[free-provider adapter](../eval/contextdoctor_openrouter.py) supplies catalog +validation, guarded HTTP and receipt normalization. A changed model/provider or +configuration is a new scope, not a resumed or relabeled old run. Future hosted +outputs are not guaranteed to reproduce because checkpoint/cache/load are not +immutable. + +Frozen message replay tests interpretation of the measured SDK outputs, not a +fresh execution of Select. To regenerate SDK inputs, install 0.6 artifacts and +use `Data(value)` / `Data(value, representation="folded")` for Core. For Select, +build `Context(query, groups)` in the corpus order, use an explicitly approved +v3 CUDA/BF16 runtime, and fit with `allow_selection=True` at +`floor(full_rendered_baseline / 2)`. Model acquisition and runtime approval remain +explicit; this corpus/test does not download weights, provision environments or +make hosted calls automatically. See [SDK migration](sdk-migration.md). diff --git a/docs/sdk-validation-0.6.0.md b/docs/sdk-validation-0.6.0.md index f508618..fb7f1f7 100644 --- a/docs/sdk-validation-0.6.0.md +++ b/docs/sdk-validation-0.6.0.md @@ -94,7 +94,9 @@ so its runtime source digest is `df523862e73e3de07000a62a0e78cb133bfaf68cb5af71e It requires a **new explicit approval**, not reuse/relabeling of the previous config. Clone-built artifacts and their functional receipts are distinct from the timings above. -## Bounded downstream attempt: inconclusive + + +## Earlier bounded downstream attempt: inconclusive Three synthetic tasks, two counterbalanced repetitions, baseline/candidate arms, 1024 output tokens per call, exact predeclared JSON oracles, no retry or provider @@ -111,6 +113,11 @@ Local candidate preparation did apply selection: rendered-text counts 173 -> 75 `o200k_base` rendered-text counts, **not provider-request savings, billed usage, answer-quality evidence or superiority qualification**. +A subsequent [2026-11-10 downstream diagnostic](sdk-benchmarks-0.6.0.md) completed +48 quality calls: compact JSON 8/8 vs. explicit folded JSON 6/8, and full-context +16/16 vs. Select 16/16 on eight synthetic tasks. Those limited observations do +not replace the earlier failed attempts or establish general quality preservation. + ## Actual user notebook The ignored local `tokenfold_run/quickstart.ipynb` is distinct from the tracked diff --git a/docs/tokenfold-demo.tape b/docs/tokenfold-demo.tape index 1372ffa..678799a 100644 --- a/docs/tokenfold-demo.tape +++ b/docs/tokenfold-demo.tape @@ -1,4 +1,6 @@ -Output docs/assets/tokenfold-demo.gif +# Legacy Core-only terminal recording. The README marketing GIF is generated +# by scripts/render-demo.py; keep this tape's output separate. +Output tmp/tokenfold-core-terminal.gif # Window geometry optimized for GitHub README width (prevents browser blur & empty void) Set Shell "bash" diff --git a/eval/research/sdk_quality_corpus.json b/eval/research/sdk_quality_corpus.json new file mode 100644 index 0000000..1cca97c --- /dev/null +++ b/eval/research/sdk_quality_corpus.json @@ -0,0 +1,1184 @@ +{ + "schema_version": 1, + "kind": "tokenfold.sdk-quality.frozen-corpus", + "date": "2026-11-10", + "protocol": { + "request_template": { + "model": "nvidia/nemotron-3.5-super-vl:free", + "stream": false, + "max_tokens": 1024, + "temperature": 0.0, + "provider": { + "only": [ + "nvidia" + ], + "allow_fallbacks": false, + "require_parameters": true, + "max_price": { + "prompt": 0, + "completion": 0, + "request": 0, + "image": 0 + } + }, + "plugins": [], + "reasoning": { + "enabled": false + }, + "seed": 271 + }, + "trials": 2, + "ordering": "trial 0 baseline,candidate; trial 1 candidate,baseline; fixed task order", + "grading": "one JSON object, exact recursive types/values, no duplicate keys/markdown; only final nonrefused stop responses eligible", + "max_quality_calls": 48, + "max_probe_calls_per_scope": 1, + "per_call_timeout_seconds": 45, + "batch_dispatch_deadline_seconds": 900, + "retries": false, + "fallbacks": false + }, + "tasks": [ + { + "id": "core-api-lookup", + "family": "core", + "workload": "json", + "source": "examples/api_response.json", + "source_sha256": "91653e687b399362c766760a69bc260c8ef0a1e805825c71602f968cd1c6ef27", + "query": "For record id 1017, return name, active, plan tier and plan seats as JSON fields name (string), active (boolean), tier (string), seats (integer).", + "oracle": { + "name": "Niklaus Wirth", + "active": true, + "tier": "free", + "seats": 1 + }, + "arms": { + "baseline": { + "messages": [ + { + "role": "user", + "content": "{\"status\":\"ok\",\"page\":1,\"per_page\":30,\"total\":30,\"results\":[{\"id\":1001,\"name\":\"Ada Lovelace\",\"role\":\"admin\",\"active\":true,\"email_verified\":true,\"last_login\":\"2026-08-01T09:00:00Z\",\"plan\":{\"tier\":\"pro\",\"seats\":5,\"region\":\"us-east-1\",\"billing_cycle\":\"annual\",\"support_level\":\"priority\"},\"permissions\":[\"read\",\"write\",\"billing\",\"admin\"]},{\"id\":1002,\"name\":\"Alan Turing\",\"role\":\"member\",\"active\":true,\"email_verified\":true,\"last_login\":\"2026-08-02T09:01:00Z\",\"plan\":{\"tier\":\"free\",\"seats\":1,\"region\":\"us-east-1\",\"billing_cycle\":\"monthly\",\"support_level\":\"community\"},\"permissions\":[\"read\",\"write\"]},{\"id\":1003,\"name\":\"Grace Hopper\",\"role\":\"member\",\"active\":true,\"email_verified\":true,\"last_login\":\"2026-08-03T09:02:00Z\",\"plan\":{\"tier\":\"team\",\"seats\":25,\"region\":\"eu-west-1\",\"billing_cycle\":\"annual\",\"support_level\":\"priority\"},\"permissions\":[\"read\",\"write\"]},{\"id\":1004,\"name\":\"Katherine Johnson\",\"role\":\"member\",\"active\":false,\"email_verified\":true,\"last_login\":\"2026-08-04T09:03:00Z\",\"plan\":{\"tier\":\"pro\",\"seats\":5,\"region\":\"us-east-1\",\"billing_cycle\":\"annual\",\"support_level\":\"priority\"},\"permissions\":[\"read\",\"write\"]},{\"id\":1005,\"name\":\"Edsger Dijkstra\",\"role\":\"member\",\"active\":true,\"email_verified\":true,\"last_login\":\"2026-08-05T09:04:00Z\",\"plan\":{\"tier\":\"free\",\"seats\":1,\"region\":\"us-east-1\",\"billing_cycle\":\"monthly\",\"support_level\":\"community\"},\"permissions\":[\"read\",\"write\"]},{\"id\":1006,\"name\":\"Barbara Liskov\",\"role\":\"member\",\"active\":true,\"email_verified\":true,\"last_login\":\"2026-08-06T09:05:00Z\",\"plan\":{\"tier\":\"pro\",\"seats\":5,\"region\":\"us-east-1\",\"billing_cycle\":\"annual\",\"support_level\":\"priority\"},\"permissions\":[\"read\",\"write\"]},{\"id\":1007,\"name\":\"Donald Knuth\",\"role\":\"member\",\"active\":true,\"email_verified\":true,\"last_login\":\"2026-08-07T09:06:00Z\",\"plan\":{\"tier\":\"pro\",\"seats\":5,\"region\":\"us-east-1\",\"billing_cycle\":\"annual\",\"support_level\":\"priority\"},\"permissions\":[\"read\",\"write\"]},{\"id\":1008,\"name\":\"Radia Perlman\",\"role\":\"admin\",\"active\":true,\"email_verified\":true,\"last_login\":\"2026-08-08T09:07:00Z\",\"plan\":{\"tier\":\"free\",\"seats\":1,\"region\":\"us-east-1\",\"billing_cycle\":\"monthly\",\"support_level\":\"community\"},\"permissions\":[\"read\",\"write\",\"billing\",\"admin\"]},{\"id\":1009,\"name\":\"John Backus\",\"role\":\"member\",\"active\":true,\"email_verified\":true,\"last_login\":\"2026-08-09T09:08:00Z\",\"plan\":{\"tier\":\"team\",\"seats\":25,\"region\":\"eu-west-1\",\"billing_cycle\":\"annual\",\"support_level\":\"priority\"},\"permissions\":[\"read\",\"write\"]},{\"id\":1010,\"name\":\"Frances Allen\",\"role\":\"member\",\"active\":false,\"email_verified\":true,\"last_login\":\"2026-08-10T09:09:00Z\",\"plan\":{\"tier\":\"pro\",\"seats\":5,\"region\":\"us-east-1\",\"billing_cycle\":\"annual\",\"support_level\":\"priority\"},\"permissions\":[\"read\",\"write\"]},{\"id\":1011,\"name\":\"Ken Thompson\",\"role\":\"member\",\"active\":true,\"email_verified\":true,\"last_login\":\"2026-08-11T09:10:00Z\",\"plan\":{\"tier\":\"pro\",\"seats\":5,\"region\":\"us-east-1\",\"billing_cycle\":\"annual\",\"support_level\":\"priority\"},\"permissions\":[\"read\",\"write\"]},{\"id\":1012,\"name\":\"Adele Goldberg\",\"role\":\"member\",\"active\":true,\"email_verified\":true,\"last_login\":\"2026-08-12T09:11:00Z\",\"plan\":{\"tier\":\"team\",\"seats\":25,\"region\":\"eu-west-1\",\"billing_cycle\":\"annual\",\"support_level\":\"priority\"},\"permissions\":[\"read\",\"write\"]},{\"id\":1013,\"name\":\"Dennis Ritchie\",\"role\":\"member\",\"active\":true,\"email_verified\":true,\"last_login\":\"2026-08-13T09:12:00Z\",\"plan\":{\"tier\":\"pro\",\"seats\":5,\"region\":\"us-east-1\",\"billing_cycle\":\"annual\",\"support_level\":\"priority\"},\"permissions\":[\"read\",\"write\"]},{\"id\":1014,\"name\":\"Margaret Hamilton\",\"role\":\"member\",\"active\":true,\"email_verified\":true,\"last_login\":\"2026-08-14T09:13:00Z\",\"plan\":{\"tier\":\"free\",\"seats\":1,\"region\":\"us-east-1\",\"billing_cycle\":\"monthly\",\"support_level\":\"community\"},\"permissions\":[\"read\",\"write\"]},{\"id\":1015,\"name\":\"Tony Hoare\",\"role\":\"admin\",\"active\":true,\"email_verified\":true,\"last_login\":\"2026-08-15T09:14:00Z\",\"plan\":{\"tier\":\"team\",\"seats\":25,\"region\":\"eu-west-1\",\"billing_cycle\":\"annual\",\"support_level\":\"priority\"},\"permissions\":[\"read\",\"write\",\"billing\",\"admin\"]},{\"id\":1016,\"name\":\"Lynn Conway\",\"role\":\"member\",\"active\":false,\"email_verified\":true,\"last_login\":\"2026-08-16T09:15:00Z\",\"plan\":{\"tier\":\"pro\",\"seats\":5,\"region\":\"us-east-1\",\"billing_cycle\":\"annual\",\"support_level\":\"priority\"},\"permissions\":[\"read\",\"write\"]},{\"id\":1017,\"name\":\"Niklaus Wirth\",\"role\":\"member\",\"active\":true,\"email_verified\":true,\"last_login\":\"2026-08-17T09:16:00Z\",\"plan\":{\"tier\":\"free\",\"seats\":1,\"region\":\"us-east-1\",\"billing_cycle\":\"monthly\",\"support_level\":\"community\"},\"permissions\":[\"read\",\"write\"]},{\"id\":1018,\"name\":\"Shafi Goldwasser\",\"role\":\"member\",\"active\":true,\"email_verified\":true,\"last_login\":\"2026-08-18T09:17:00Z\",\"plan\":{\"tier\":\"team\",\"seats\":25,\"region\":\"eu-west-1\",\"billing_cycle\":\"annual\",\"support_level\":\"priority\"},\"permissions\":[\"read\",\"write\"]},{\"id\":1019,\"name\":\"Leslie Lamport\",\"role\":\"member\",\"active\":true,\"email_verified\":true,\"last_login\":\"2026-08-19T09:18:00Z\",\"plan\":{\"tier\":\"pro\",\"seats\":5,\"region\":\"us-east-1\",\"billing_cycle\":\"annual\",\"support_level\":\"priority\"},\"permissions\":[\"read\",\"write\"]},{\"id\":1020,\"name\":\"Anita Borg\",\"role\":\"member\",\"active\":true,\"email_verified\":true,\"last_login\":\"2026-08-20T09:19:00Z\",\"plan\":{\"tier\":\"free\",\"seats\":1,\"region\":\"us-east-1\",\"billing_cycle\":\"monthly\",\"support_level\":\"community\"},\"permissions\":[\"read\",\"write\"]},{\"id\":1021,\"name\":\"John McCarthy\",\"role\":\"member\",\"active\":true,\"email_verified\":true,\"last_login\":\"2026-08-21T09:20:00Z\",\"plan\":{\"tier\":\"pro\",\"seats\":5,\"region\":\"us-east-1\",\"billing_cycle\":\"annual\",\"support_level\":\"priority\"},\"permissions\":[\"read\",\"write\"]},{\"id\":1022,\"name\":\"Karen Sparck Jones\",\"role\":\"admin\",\"active\":false,\"email_verified\":true,\"last_login\":\"2026-08-22T09:21:00Z\",\"plan\":{\"tier\":\"pro\",\"seats\":5,\"region\":\"us-east-1\",\"billing_cycle\":\"annual\",\"support_level\":\"priority\"},\"permissions\":[\"read\",\"write\",\"billing\",\"admin\"]},{\"id\":1023,\"name\":\"Claude Shannon\",\"role\":\"member\",\"active\":true,\"email_verified\":true,\"last_login\":\"2026-08-23T09:22:00Z\",\"plan\":{\"tier\":\"free\",\"seats\":1,\"region\":\"us-east-1\",\"billing_cycle\":\"monthly\",\"support_level\":\"community\"},\"permissions\":[\"read\",\"write\"]},{\"id\":1024,\"name\":\"Jean Bartik\",\"role\":\"member\",\"active\":true,\"email_verified\":true,\"last_login\":\"2026-08-24T09:23:00Z\",\"plan\":{\"tier\":\"team\",\"seats\":25,\"region\":\"eu-west-1\",\"billing_cycle\":\"annual\",\"support_level\":\"priority\"},\"permissions\":[\"read\",\"write\"]},{\"id\":1025,\"name\":\"Vint Cerf\",\"role\":\"member\",\"active\":true,\"email_verified\":true,\"last_login\":\"2026-08-25T09:24:00Z\",\"plan\":{\"tier\":\"pro\",\"seats\":5,\"region\":\"us-east-1\",\"billing_cycle\":\"annual\",\"support_level\":\"priority\"},\"permissions\":[\"read\",\"write\"]},{\"id\":1026,\"name\":\"Sophie Wilson\",\"role\":\"member\",\"active\":true,\"email_verified\":true,\"last_login\":\"2026-08-26T09:25:00Z\",\"plan\":{\"tier\":\"pro\",\"seats\":5,\"region\":\"us-east-1\",\"billing_cycle\":\"annual\",\"support_level\":\"priority\"},\"permissions\":[\"read\",\"write\"]},{\"id\":1027,\"name\":\"Bjarne Stroustrup\",\"role\":\"member\",\"active\":true,\"email_verified\":true,\"last_login\":\"2026-08-27T09:26:00Z\",\"plan\":{\"tier\":\"team\",\"seats\":25,\"region\":\"eu-west-1\",\"billing_cycle\":\"annual\",\"support_level\":\"priority\"},\"permissions\":[\"read\",\"write\"]},{\"id\":1028,\"name\":\"Gladys West\",\"role\":\"member\",\"active\":false,\"email_verified\":true,\"last_login\":\"2026-08-28T09:27:00Z\",\"plan\":{\"tier\":\"pro\",\"seats\":5,\"region\":\"us-east-1\",\"billing_cycle\":\"annual\",\"support_level\":\"priority\"},\"permissions\":[\"read\",\"write\"]},{\"id\":1029,\"name\":\"Guido van Rossum\",\"role\":\"admin\",\"active\":true,\"email_verified\":true,\"last_login\":\"2026-08-01T09:28:00Z\",\"plan\":{\"tier\":\"free\",\"seats\":1,\"region\":\"us-east-1\",\"billing_cycle\":\"monthly\",\"support_level\":\"community\"},\"permissions\":[\"read\",\"write\",\"billing\",\"admin\"]},{\"id\":1030,\"name\":\"Hedy Lamarr\",\"role\":\"member\",\"active\":true,\"email_verified\":true,\"last_login\":\"2026-08-02T09:29:00Z\",\"plan\":{\"tier\":\"team\",\"seats\":25,\"region\":\"eu-west-1\",\"billing_cycle\":\"annual\",\"support_level\":\"priority\"},\"permissions\":[\"read\",\"write\"]}]}" + }, + { + "role": "user", + "content": "For record id 1017, return name, active, plan tier and plan seats as JSON fields name (string), active (boolean), tier (string), seats (integer).\nReturn only one JSON object, with exactly the requested fields and types; no markdown. Use only the supplied evidence. Folded JSON uses __tf_cols__ as column names and __tf_rows__ as row values. A __tf_dict__ array stores values referenced by {\"__tf_ref__\":i} inside __tf_data__." + } + ], + "rendered_tokens": 2366, + "target_tokens": 100000, + "budget_status": "met", + "accounting": { + "counter_id": "encoding:o200k_base:v1", + "scope": "rendered_text", + "original_basis": "exact", + "baseline_basis": "exact", + "final_basis": "exact" + }, + "core_transforms": [], + "scorer_succeeded": false, + "selection_applied": false, + "groups_kept": [], + "groups_dropped": [] + }, + "candidate": { + "messages": [ + { + "role": "user", + "content": "{\"__tf_dict__\":[{\"tier\":\"pro\",\"seats\":5,\"region\":\"us-east-1\",\"billing_cycle\":\"annual\",\"support_level\":\"priority\"},[\"read\",\"write\",\"billing\",\"admin\"],{\"tier\":\"free\",\"seats\":1,\"region\":\"us-east-1\",\"billing_cycle\":\"monthly\",\"support_level\":\"community\"},{\"tier\":\"team\",\"seats\":25,\"region\":\"eu-west-1\",\"billing_cycle\":\"annual\",\"support_level\":\"priority\"}],\"__tf_data__\":{\"status\":\"ok\",\"page\":1,\"per_page\":30,\"total\":30,\"results\":{\"__tf_cols__\":[\"id\",\"name\",\"role\",\"active\",\"email_verified\",\"last_login\",\"plan\",\"permissions\"],\"__tf_rows__\":[[1001,\"Ada Lovelace\",\"admin\",true,true,\"2026-08-01T09:00:00Z\",{\"__tf_ref__\":0},{\"__tf_ref__\":1}],[1002,\"Alan Turing\",\"member\",true,true,\"2026-08-02T09:01:00Z\",{\"__tf_ref__\":2},[\"read\",\"write\"]],[1003,\"Grace Hopper\",\"member\",true,true,\"2026-08-03T09:02:00Z\",{\"__tf_ref__\":3},[\"read\",\"write\"]],[1004,\"Katherine Johnson\",\"member\",false,true,\"2026-08-04T09:03:00Z\",{\"__tf_ref__\":0},[\"read\",\"write\"]],[1005,\"Edsger Dijkstra\",\"member\",true,true,\"2026-08-05T09:04:00Z\",{\"__tf_ref__\":2},[\"read\",\"write\"]],[1006,\"Barbara Liskov\",\"member\",true,true,\"2026-08-06T09:05:00Z\",{\"__tf_ref__\":0},[\"read\",\"write\"]],[1007,\"Donald Knuth\",\"member\",true,true,\"2026-08-07T09:06:00Z\",{\"__tf_ref__\":0},[\"read\",\"write\"]],[1008,\"Radia Perlman\",\"admin\",true,true,\"2026-08-08T09:07:00Z\",{\"__tf_ref__\":2},{\"__tf_ref__\":1}],[1009,\"John Backus\",\"member\",true,true,\"2026-08-09T09:08:00Z\",{\"__tf_ref__\":3},[\"read\",\"write\"]],[1010,\"Frances Allen\",\"member\",false,true,\"2026-08-10T09:09:00Z\",{\"__tf_ref__\":0},[\"read\",\"write\"]],[1011,\"Ken Thompson\",\"member\",true,true,\"2026-08-11T09:10:00Z\",{\"__tf_ref__\":0},[\"read\",\"write\"]],[1012,\"Adele Goldberg\",\"member\",true,true,\"2026-08-12T09:11:00Z\",{\"__tf_ref__\":3},[\"read\",\"write\"]],[1013,\"Dennis Ritchie\",\"member\",true,true,\"2026-08-13T09:12:00Z\",{\"__tf_ref__\":0},[\"read\",\"write\"]],[1014,\"Margaret Hamilton\",\"member\",true,true,\"2026-08-14T09:13:00Z\",{\"__tf_ref__\":2},[\"read\",\"write\"]],[1015,\"Tony Hoare\",\"admin\",true,true,\"2026-08-15T09:14:00Z\",{\"__tf_ref__\":3},{\"__tf_ref__\":1}],[1016,\"Lynn Conway\",\"member\",false,true,\"2026-08-16T09:15:00Z\",{\"__tf_ref__\":0},[\"read\",\"write\"]],[1017,\"Niklaus Wirth\",\"member\",true,true,\"2026-08-17T09:16:00Z\",{\"__tf_ref__\":2},[\"read\",\"write\"]],[1018,\"Shafi Goldwasser\",\"member\",true,true,\"2026-08-18T09:17:00Z\",{\"__tf_ref__\":3},[\"read\",\"write\"]],[1019,\"Leslie Lamport\",\"member\",true,true,\"2026-08-19T09:18:00Z\",{\"__tf_ref__\":0},[\"read\",\"write\"]],[1020,\"Anita Borg\",\"member\",true,true,\"2026-08-20T09:19:00Z\",{\"__tf_ref__\":2},[\"read\",\"write\"]],[1021,\"John McCarthy\",\"member\",true,true,\"2026-08-21T09:20:00Z\",{\"__tf_ref__\":0},[\"read\",\"write\"]],[1022,\"Karen Sparck Jones\",\"admin\",false,true,\"2026-08-22T09:21:00Z\",{\"__tf_ref__\":0},{\"__tf_ref__\":1}],[1023,\"Claude Shannon\",\"member\",true,true,\"2026-08-23T09:22:00Z\",{\"__tf_ref__\":2},[\"read\",\"write\"]],[1024,\"Jean Bartik\",\"member\",true,true,\"2026-08-24T09:23:00Z\",{\"__tf_ref__\":3},[\"read\",\"write\"]],[1025,\"Vint Cerf\",\"member\",true,true,\"2026-08-25T09:24:00Z\",{\"__tf_ref__\":0},[\"read\",\"write\"]],[1026,\"Sophie Wilson\",\"member\",true,true,\"2026-08-26T09:25:00Z\",{\"__tf_ref__\":0},[\"read\",\"write\"]],[1027,\"Bjarne Stroustrup\",\"member\",true,true,\"2026-08-27T09:26:00Z\",{\"__tf_ref__\":3},[\"read\",\"write\"]],[1028,\"Gladys West\",\"member\",false,true,\"2026-08-28T09:27:00Z\",{\"__tf_ref__\":0},[\"read\",\"write\"]],[1029,\"Guido van Rossum\",\"admin\",true,true,\"2026-08-01T09:28:00Z\",{\"__tf_ref__\":2},{\"__tf_ref__\":1}],[1030,\"Hedy Lamarr\",\"member\",true,true,\"2026-08-02T09:29:00Z\",{\"__tf_ref__\":3},[\"read\",\"write\"]]]}}}" + }, + { + "role": "user", + "content": "For record id 1017, return name, active, plan tier and plan seats as JSON fields name (string), active (boolean), tier (string), seats (integer).\nReturn only one JSON object, with exactly the requested fields and types; no markdown. Use only the supplied evidence. Folded JSON uses __tf_cols__ as column names and __tf_rows__ as row values. A __tf_dict__ array stores values referenced by {\"__tf_ref__\":i} inside __tf_data__." + } + ], + "rendered_tokens": 1376, + "target_tokens": 100000, + "budget_status": "met", + "accounting": { + "counter_id": "encoding:o200k_base:v1", + "scope": "rendered_text", + "original_basis": "exact", + "baseline_basis": "exact", + "final_basis": "exact" + }, + "core_transforms": [ + "json_field_fold", + "json_value_dict" + ], + "scorer_succeeded": false, + "selection_applied": false, + "groups_kept": [], + "groups_dropped": [] + } + } + }, + { + "id": "core-users-lookup", + "family": "core", + "workload": "json", + "source": "eval/research/toon_corpus/flat_users.json", + "source_sha256": "2ae2b2f59a851c6b79efc8a1d27102b113dc1424007c64f80ae0909bc37f578c", + "query": "For user id 1009, return JSON fields name (string), role (string), active (boolean).", + "oracle": { + "name": "John Backus", + "role": "member", + "active": true + }, + "arms": { + "baseline": { + "messages": [ + { + "role": "user", + "content": "[{\"id\":1001,\"name\":\"Ada Lovelace\",\"role\":\"admin\",\"active\":true},{\"id\":1002,\"name\":\"Alan Turing\",\"role\":\"member\",\"active\":true},{\"id\":1003,\"name\":\"Grace Hopper\",\"role\":\"member\",\"active\":true},{\"id\":1004,\"name\":\"Katherine Johnson\",\"role\":\"member\",\"active\":false},{\"id\":1005,\"name\":\"Edsger Dijkstra\",\"role\":\"member\",\"active\":true},{\"id\":1006,\"name\":\"Barbara Liskov\",\"role\":\"member\",\"active\":true},{\"id\":1007,\"name\":\"Donald Knuth\",\"role\":\"member\",\"active\":true},{\"id\":1008,\"name\":\"Radia Perlman\",\"role\":\"admin\",\"active\":true},{\"id\":1009,\"name\":\"John Backus\",\"role\":\"member\",\"active\":true},{\"id\":1010,\"name\":\"Frances Allen\",\"role\":\"member\",\"active\":false},{\"id\":1011,\"name\":\"Ken Thompson\",\"role\":\"member\",\"active\":true},{\"id\":1012,\"name\":\"Adele Goldberg\",\"role\":\"member\",\"active\":true},{\"id\":1013,\"name\":\"Dennis Ritchie\",\"role\":\"member\",\"active\":true},{\"id\":1014,\"name\":\"Margaret Hamilton\",\"role\":\"member\",\"active\":true},{\"id\":1015,\"name\":\"Tony Hoare\",\"role\":\"admin\",\"active\":true},{\"id\":1016,\"name\":\"Lynn Conway\",\"role\":\"member\",\"active\":false},{\"id\":1017,\"name\":\"Niklaus Wirth\",\"role\":\"member\",\"active\":true},{\"id\":1018,\"name\":\"Shafi Goldwasser\",\"role\":\"member\",\"active\":true},{\"id\":1019,\"name\":\"Leslie Lamport\",\"role\":\"member\",\"active\":true},{\"id\":1020,\"name\":\"Anita Borg\",\"role\":\"member\",\"active\":true},{\"id\":1021,\"name\":\"John McCarthy\",\"role\":\"member\",\"active\":true},{\"id\":1022,\"name\":\"Karen Sparck Jones\",\"role\":\"admin\",\"active\":false},{\"id\":1023,\"name\":\"Claude Shannon\",\"role\":\"member\",\"active\":true},{\"id\":1024,\"name\":\"Jean Bartik\",\"role\":\"member\",\"active\":true},{\"id\":1025,\"name\":\"Vint Cerf\",\"role\":\"member\",\"active\":true},{\"id\":1026,\"name\":\"Sophie Wilson\",\"role\":\"member\",\"active\":true},{\"id\":1027,\"name\":\"Bjarne Stroustrup\",\"role\":\"member\",\"active\":true},{\"id\":1028,\"name\":\"Gladys West\",\"role\":\"member\",\"active\":false},{\"id\":1029,\"name\":\"Guido van Rossum\",\"role\":\"admin\",\"active\":true},{\"id\":1030,\"name\":\"Hedy Lamarr\",\"role\":\"member\",\"active\":true}]" + }, + { + "role": "user", + "content": "For user id 1009, return JSON fields name (string), role (string), active (boolean).\nReturn only one JSON object, with exactly the requested fields and types; no markdown. Use only the supplied evidence. Folded JSON uses __tf_cols__ as column names and __tf_rows__ as row values. A __tf_dict__ array stores values referenced by {\"__tf_ref__\":i} inside __tf_data__." + } + ], + "rendered_tokens": 586, + "target_tokens": 100000, + "budget_status": "met", + "accounting": { + "counter_id": "encoding:o200k_base:v1", + "scope": "rendered_text", + "original_basis": "exact", + "baseline_basis": "exact", + "final_basis": "exact" + }, + "core_transforms": [], + "scorer_succeeded": false, + "selection_applied": false, + "groups_kept": [], + "groups_dropped": [] + }, + "candidate": { + "messages": [ + { + "role": "user", + "content": "{\"__tf_cols__\":[\"id\",\"name\",\"role\",\"active\"],\"__tf_rows__\":[[1001,\"Ada Lovelace\",\"admin\",true],[1002,\"Alan Turing\",\"member\",true],[1003,\"Grace Hopper\",\"member\",true],[1004,\"Katherine Johnson\",\"member\",false],[1005,\"Edsger Dijkstra\",\"member\",true],[1006,\"Barbara Liskov\",\"member\",true],[1007,\"Donald Knuth\",\"member\",true],[1008,\"Radia Perlman\",\"admin\",true],[1009,\"John Backus\",\"member\",true],[1010,\"Frances Allen\",\"member\",false],[1011,\"Ken Thompson\",\"member\",true],[1012,\"Adele Goldberg\",\"member\",true],[1013,\"Dennis Ritchie\",\"member\",true],[1014,\"Margaret Hamilton\",\"member\",true],[1015,\"Tony Hoare\",\"admin\",true],[1016,\"Lynn Conway\",\"member\",false],[1017,\"Niklaus Wirth\",\"member\",true],[1018,\"Shafi Goldwasser\",\"member\",true],[1019,\"Leslie Lamport\",\"member\",true],[1020,\"Anita Borg\",\"member\",true],[1021,\"John McCarthy\",\"member\",true],[1022,\"Karen Sparck Jones\",\"admin\",false],[1023,\"Claude Shannon\",\"member\",true],[1024,\"Jean Bartik\",\"member\",true],[1025,\"Vint Cerf\",\"member\",true],[1026,\"Sophie Wilson\",\"member\",true],[1027,\"Bjarne Stroustrup\",\"member\",true],[1028,\"Gladys West\",\"member\",false],[1029,\"Guido van Rossum\",\"admin\",true],[1030,\"Hedy Lamarr\",\"member\",true]]}" + }, + { + "role": "user", + "content": "For user id 1009, return JSON fields name (string), role (string), active (boolean).\nReturn only one JSON object, with exactly the requested fields and types; no markdown. Use only the supplied evidence. Folded JSON uses __tf_cols__ as column names and __tf_rows__ as row values. A __tf_dict__ array stores values referenced by {\"__tf_ref__\":i} inside __tf_data__." + } + ], + "rendered_tokens": 366, + "target_tokens": 100000, + "budget_status": "met", + "accounting": { + "counter_id": "encoding:o200k_base:v1", + "scope": "rendered_text", + "original_basis": "exact", + "baseline_basis": "exact", + "final_basis": "exact" + }, + "core_transforms": [ + "json_field_fold" + ], + "scorer_succeeded": false, + "selection_applied": false, + "groups_kept": [], + "groups_dropped": [] + } + } + }, + { + "id": "core-wide-lookup", + "family": "core", + "workload": "json", + "source": "examples/toon_metrics.json", + "source_sha256": "a5057abc36a7cc8e63f48101a3649f0578f82c1415a14dd33cf0ec8d4d8472de", + "query": "For timestamp 2026-09-02T12:00:13Z, return JSON fields service (string), request_count (integer), error_count (integer), p95_ms (integer).", + "oracle": { + "service": "billing", + "request_count": 1369, + "error_count": 2, + "p95_ms": 158 + }, + "arms": { + "baseline": { + "messages": [ + { + "role": "user", + "content": "[{\"timestamp\":\"2026-09-02T12:00:00Z\",\"service\":\"api-gateway\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"degraded\",\"request_count\":1200,\"error_count\":0,\"p95_ms\":145},{\"timestamp\":\"2026-09-02T12:00:01Z\",\"service\":\"billing\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":1213,\"error_count\":1,\"p95_ms\":146},{\"timestamp\":\"2026-09-02T12:00:02Z\",\"service\":\"catalog\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":1226,\"error_count\":2,\"p95_ms\":147},{\"timestamp\":\"2026-09-02T12:00:03Z\",\"service\":\"search\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":1239,\"error_count\":3,\"p95_ms\":148},{\"timestamp\":\"2026-09-02T12:00:04Z\",\"service\":\"api-gateway\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":1252,\"error_count\":4,\"p95_ms\":149},{\"timestamp\":\"2026-09-02T12:00:05Z\",\"service\":\"billing\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":1265,\"error_count\":5,\"p95_ms\":150},{\"timestamp\":\"2026-09-02T12:00:06Z\",\"service\":\"catalog\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":1278,\"error_count\":6,\"p95_ms\":151},{\"timestamp\":\"2026-09-02T12:00:07Z\",\"service\":\"search\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":1291,\"error_count\":7,\"p95_ms\":152},{\"timestamp\":\"2026-09-02T12:00:08Z\",\"service\":\"api-gateway\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":1304,\"error_count\":8,\"p95_ms\":153},{\"timestamp\":\"2026-09-02T12:00:09Z\",\"service\":\"billing\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":1317,\"error_count\":9,\"p95_ms\":154},{\"timestamp\":\"2026-09-02T12:00:10Z\",\"service\":\"catalog\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":1330,\"error_count\":10,\"p95_ms\":155},{\"timestamp\":\"2026-09-02T12:00:11Z\",\"service\":\"search\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":1343,\"error_count\":0,\"p95_ms\":156},{\"timestamp\":\"2026-09-02T12:00:12Z\",\"service\":\"api-gateway\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":1356,\"error_count\":1,\"p95_ms\":157},{\"timestamp\":\"2026-09-02T12:00:13Z\",\"service\":\"billing\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":1369,\"error_count\":2,\"p95_ms\":158},{\"timestamp\":\"2026-09-02T12:00:14Z\",\"service\":\"catalog\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":1382,\"error_count\":3,\"p95_ms\":159},{\"timestamp\":\"2026-09-02T12:00:15Z\",\"service\":\"search\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":1395,\"error_count\":4,\"p95_ms\":160},{\"timestamp\":\"2026-09-02T12:00:16Z\",\"service\":\"api-gateway\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":1408,\"error_count\":5,\"p95_ms\":161},{\"timestamp\":\"2026-09-02T12:00:17Z\",\"service\":\"billing\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"degraded\",\"request_count\":1421,\"error_count\":6,\"p95_ms\":162},{\"timestamp\":\"2026-09-02T12:00:18Z\",\"service\":\"catalog\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":1434,\"error_count\":7,\"p95_ms\":163},{\"timestamp\":\"2026-09-02T12:00:19Z\",\"service\":\"search\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":1447,\"error_count\":8,\"p95_ms\":164},{\"timestamp\":\"2026-09-02T12:00:20Z\",\"service\":\"api-gateway\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":1460,\"error_count\":9,\"p95_ms\":165},{\"timestamp\":\"2026-09-02T12:00:21Z\",\"service\":\"billing\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":1473,\"error_count\":10,\"p95_ms\":166},{\"timestamp\":\"2026-09-02T12:00:22Z\",\"service\":\"catalog\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":1486,\"error_count\":0,\"p95_ms\":167},{\"timestamp\":\"2026-09-02T12:00:23Z\",\"service\":\"search\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":1499,\"error_count\":1,\"p95_ms\":145},{\"timestamp\":\"2026-09-02T12:00:24Z\",\"service\":\"api-gateway\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":1512,\"error_count\":2,\"p95_ms\":146},{\"timestamp\":\"2026-09-02T12:00:25Z\",\"service\":\"billing\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":1525,\"error_count\":3,\"p95_ms\":147},{\"timestamp\":\"2026-09-02T12:00:26Z\",\"service\":\"catalog\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":1538,\"error_count\":4,\"p95_ms\":148},{\"timestamp\":\"2026-09-02T12:00:27Z\",\"service\":\"search\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":1551,\"error_count\":5,\"p95_ms\":149},{\"timestamp\":\"2026-09-02T12:00:28Z\",\"service\":\"api-gateway\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":1564,\"error_count\":6,\"p95_ms\":150},{\"timestamp\":\"2026-09-02T12:00:29Z\",\"service\":\"billing\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":1577,\"error_count\":7,\"p95_ms\":151},{\"timestamp\":\"2026-09-02T12:00:30Z\",\"service\":\"catalog\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":1590,\"error_count\":8,\"p95_ms\":152},{\"timestamp\":\"2026-09-02T12:00:31Z\",\"service\":\"search\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":1603,\"error_count\":9,\"p95_ms\":153},{\"timestamp\":\"2026-09-02T12:00:32Z\",\"service\":\"api-gateway\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":1616,\"error_count\":10,\"p95_ms\":154},{\"timestamp\":\"2026-09-02T12:00:33Z\",\"service\":\"billing\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":1629,\"error_count\":0,\"p95_ms\":155},{\"timestamp\":\"2026-09-02T12:00:34Z\",\"service\":\"catalog\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"degraded\",\"request_count\":1642,\"error_count\":1,\"p95_ms\":156},{\"timestamp\":\"2026-09-02T12:00:35Z\",\"service\":\"search\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":1655,\"error_count\":2,\"p95_ms\":157},{\"timestamp\":\"2026-09-02T12:00:36Z\",\"service\":\"api-gateway\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":1668,\"error_count\":3,\"p95_ms\":158},{\"timestamp\":\"2026-09-02T12:00:37Z\",\"service\":\"billing\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":1681,\"error_count\":4,\"p95_ms\":159},{\"timestamp\":\"2026-09-02T12:00:38Z\",\"service\":\"catalog\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":1694,\"error_count\":5,\"p95_ms\":160},{\"timestamp\":\"2026-09-02T12:00:39Z\",\"service\":\"search\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":1707,\"error_count\":6,\"p95_ms\":161},{\"timestamp\":\"2026-09-02T12:00:40Z\",\"service\":\"api-gateway\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":1720,\"error_count\":7,\"p95_ms\":162},{\"timestamp\":\"2026-09-02T12:00:41Z\",\"service\":\"billing\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":1733,\"error_count\":8,\"p95_ms\":163},{\"timestamp\":\"2026-09-02T12:00:42Z\",\"service\":\"catalog\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":1746,\"error_count\":9,\"p95_ms\":164},{\"timestamp\":\"2026-09-02T12:00:43Z\",\"service\":\"search\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":1759,\"error_count\":10,\"p95_ms\":165},{\"timestamp\":\"2026-09-02T12:00:44Z\",\"service\":\"api-gateway\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":1772,\"error_count\":0,\"p95_ms\":166},{\"timestamp\":\"2026-09-02T12:00:45Z\",\"service\":\"billing\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":1785,\"error_count\":1,\"p95_ms\":167},{\"timestamp\":\"2026-09-02T12:00:46Z\",\"service\":\"catalog\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":1798,\"error_count\":2,\"p95_ms\":145},{\"timestamp\":\"2026-09-02T12:00:47Z\",\"service\":\"search\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":1811,\"error_count\":3,\"p95_ms\":146},{\"timestamp\":\"2026-09-02T12:00:48Z\",\"service\":\"api-gateway\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":1824,\"error_count\":4,\"p95_ms\":147},{\"timestamp\":\"2026-09-02T12:00:49Z\",\"service\":\"billing\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":1837,\"error_count\":5,\"p95_ms\":148},{\"timestamp\":\"2026-09-02T12:00:50Z\",\"service\":\"catalog\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":1850,\"error_count\":6,\"p95_ms\":149},{\"timestamp\":\"2026-09-02T12:00:51Z\",\"service\":\"search\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"degraded\",\"request_count\":1863,\"error_count\":7,\"p95_ms\":150},{\"timestamp\":\"2026-09-02T12:00:52Z\",\"service\":\"api-gateway\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":1876,\"error_count\":8,\"p95_ms\":151},{\"timestamp\":\"2026-09-02T12:00:53Z\",\"service\":\"billing\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":1889,\"error_count\":9,\"p95_ms\":152},{\"timestamp\":\"2026-09-02T12:00:54Z\",\"service\":\"catalog\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":1902,\"error_count\":10,\"p95_ms\":153},{\"timestamp\":\"2026-09-02T12:00:55Z\",\"service\":\"search\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":1915,\"error_count\":0,\"p95_ms\":154},{\"timestamp\":\"2026-09-02T12:00:56Z\",\"service\":\"api-gateway\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":1928,\"error_count\":1,\"p95_ms\":155},{\"timestamp\":\"2026-09-02T12:00:57Z\",\"service\":\"billing\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":1941,\"error_count\":2,\"p95_ms\":156},{\"timestamp\":\"2026-09-02T12:00:58Z\",\"service\":\"catalog\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":1954,\"error_count\":3,\"p95_ms\":157},{\"timestamp\":\"2026-09-02T12:00:59Z\",\"service\":\"search\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":1967,\"error_count\":4,\"p95_ms\":158},{\"timestamp\":\"2026-09-02T12:01:00Z\",\"service\":\"api-gateway\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":1980,\"error_count\":5,\"p95_ms\":159},{\"timestamp\":\"2026-09-02T12:01:01Z\",\"service\":\"billing\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":1993,\"error_count\":6,\"p95_ms\":160},{\"timestamp\":\"2026-09-02T12:01:02Z\",\"service\":\"catalog\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":2006,\"error_count\":7,\"p95_ms\":161},{\"timestamp\":\"2026-09-02T12:01:03Z\",\"service\":\"search\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":2019,\"error_count\":8,\"p95_ms\":162},{\"timestamp\":\"2026-09-02T12:01:04Z\",\"service\":\"api-gateway\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":2032,\"error_count\":9,\"p95_ms\":163},{\"timestamp\":\"2026-09-02T12:01:05Z\",\"service\":\"billing\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":2045,\"error_count\":10,\"p95_ms\":164},{\"timestamp\":\"2026-09-02T12:01:06Z\",\"service\":\"catalog\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":2058,\"error_count\":0,\"p95_ms\":165},{\"timestamp\":\"2026-09-02T12:01:07Z\",\"service\":\"search\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":2071,\"error_count\":1,\"p95_ms\":166},{\"timestamp\":\"2026-09-02T12:01:08Z\",\"service\":\"api-gateway\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"degraded\",\"request_count\":2084,\"error_count\":2,\"p95_ms\":167},{\"timestamp\":\"2026-09-02T12:01:09Z\",\"service\":\"billing\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":2097,\"error_count\":3,\"p95_ms\":145},{\"timestamp\":\"2026-09-02T12:01:10Z\",\"service\":\"catalog\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":2110,\"error_count\":4,\"p95_ms\":146},{\"timestamp\":\"2026-09-02T12:01:11Z\",\"service\":\"search\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":2123,\"error_count\":5,\"p95_ms\":147},{\"timestamp\":\"2026-09-02T12:01:12Z\",\"service\":\"api-gateway\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":2136,\"error_count\":6,\"p95_ms\":148},{\"timestamp\":\"2026-09-02T12:01:13Z\",\"service\":\"billing\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":2149,\"error_count\":7,\"p95_ms\":149},{\"timestamp\":\"2026-09-02T12:01:14Z\",\"service\":\"catalog\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":2162,\"error_count\":8,\"p95_ms\":150},{\"timestamp\":\"2026-09-02T12:01:15Z\",\"service\":\"search\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":2175,\"error_count\":9,\"p95_ms\":151},{\"timestamp\":\"2026-09-02T12:01:16Z\",\"service\":\"api-gateway\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":2188,\"error_count\":10,\"p95_ms\":152},{\"timestamp\":\"2026-09-02T12:01:17Z\",\"service\":\"billing\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":2201,\"error_count\":0,\"p95_ms\":153},{\"timestamp\":\"2026-09-02T12:01:18Z\",\"service\":\"catalog\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":2214,\"error_count\":1,\"p95_ms\":154},{\"timestamp\":\"2026-09-02T12:01:19Z\",\"service\":\"search\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":2227,\"error_count\":2,\"p95_ms\":155},{\"timestamp\":\"2026-09-02T12:01:20Z\",\"service\":\"api-gateway\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":2240,\"error_count\":3,\"p95_ms\":156},{\"timestamp\":\"2026-09-02T12:01:21Z\",\"service\":\"billing\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":2253,\"error_count\":4,\"p95_ms\":157},{\"timestamp\":\"2026-09-02T12:01:22Z\",\"service\":\"catalog\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":2266,\"error_count\":5,\"p95_ms\":158},{\"timestamp\":\"2026-09-02T12:01:23Z\",\"service\":\"search\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":2279,\"error_count\":6,\"p95_ms\":159},{\"timestamp\":\"2026-09-02T12:01:24Z\",\"service\":\"api-gateway\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":2292,\"error_count\":7,\"p95_ms\":160},{\"timestamp\":\"2026-09-02T12:01:25Z\",\"service\":\"billing\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"degraded\",\"request_count\":2305,\"error_count\":8,\"p95_ms\":161},{\"timestamp\":\"2026-09-02T12:01:26Z\",\"service\":\"catalog\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":2318,\"error_count\":9,\"p95_ms\":162},{\"timestamp\":\"2026-09-02T12:01:27Z\",\"service\":\"search\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":2331,\"error_count\":10,\"p95_ms\":163},{\"timestamp\":\"2026-09-02T12:01:28Z\",\"service\":\"api-gateway\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":2344,\"error_count\":0,\"p95_ms\":164},{\"timestamp\":\"2026-09-02T12:01:29Z\",\"service\":\"billing\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":2357,\"error_count\":1,\"p95_ms\":165},{\"timestamp\":\"2026-09-02T12:01:30Z\",\"service\":\"catalog\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":2370,\"error_count\":2,\"p95_ms\":166},{\"timestamp\":\"2026-09-02T12:01:31Z\",\"service\":\"search\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":2383,\"error_count\":3,\"p95_ms\":167},{\"timestamp\":\"2026-09-02T12:01:32Z\",\"service\":\"api-gateway\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":2396,\"error_count\":4,\"p95_ms\":145},{\"timestamp\":\"2026-09-02T12:01:33Z\",\"service\":\"billing\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":2409,\"error_count\":5,\"p95_ms\":146},{\"timestamp\":\"2026-09-02T12:01:34Z\",\"service\":\"catalog\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":2422,\"error_count\":6,\"p95_ms\":147},{\"timestamp\":\"2026-09-02T12:01:35Z\",\"service\":\"search\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":2435,\"error_count\":7,\"p95_ms\":148},{\"timestamp\":\"2026-09-02T12:01:36Z\",\"service\":\"api-gateway\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":2448,\"error_count\":8,\"p95_ms\":149},{\"timestamp\":\"2026-09-02T12:01:37Z\",\"service\":\"billing\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":2461,\"error_count\":9,\"p95_ms\":150},{\"timestamp\":\"2026-09-02T12:01:38Z\",\"service\":\"catalog\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":2474,\"error_count\":10,\"p95_ms\":151},{\"timestamp\":\"2026-09-02T12:01:39Z\",\"service\":\"search\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":2487,\"error_count\":0,\"p95_ms\":152},{\"timestamp\":\"2026-09-02T12:01:40Z\",\"service\":\"api-gateway\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":2500,\"error_count\":1,\"p95_ms\":153},{\"timestamp\":\"2026-09-02T12:01:41Z\",\"service\":\"billing\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":2513,\"error_count\":2,\"p95_ms\":154},{\"timestamp\":\"2026-09-02T12:01:42Z\",\"service\":\"catalog\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"degraded\",\"request_count\":2526,\"error_count\":3,\"p95_ms\":155},{\"timestamp\":\"2026-09-02T12:01:43Z\",\"service\":\"search\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":2539,\"error_count\":4,\"p95_ms\":156},{\"timestamp\":\"2026-09-02T12:01:44Z\",\"service\":\"api-gateway\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":2552,\"error_count\":5,\"p95_ms\":157},{\"timestamp\":\"2026-09-02T12:01:45Z\",\"service\":\"billing\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":2565,\"error_count\":6,\"p95_ms\":158},{\"timestamp\":\"2026-09-02T12:01:46Z\",\"service\":\"catalog\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":2578,\"error_count\":7,\"p95_ms\":159},{\"timestamp\":\"2026-09-02T12:01:47Z\",\"service\":\"search\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":2591,\"error_count\":8,\"p95_ms\":160},{\"timestamp\":\"2026-09-02T12:01:48Z\",\"service\":\"api-gateway\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":2604,\"error_count\":9,\"p95_ms\":161},{\"timestamp\":\"2026-09-02T12:01:49Z\",\"service\":\"billing\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":2617,\"error_count\":10,\"p95_ms\":162},{\"timestamp\":\"2026-09-02T12:01:50Z\",\"service\":\"catalog\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":2630,\"error_count\":0,\"p95_ms\":163},{\"timestamp\":\"2026-09-02T12:01:51Z\",\"service\":\"search\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":2643,\"error_count\":1,\"p95_ms\":164},{\"timestamp\":\"2026-09-02T12:01:52Z\",\"service\":\"api-gateway\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":2656,\"error_count\":2,\"p95_ms\":165},{\"timestamp\":\"2026-09-02T12:01:53Z\",\"service\":\"billing\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":2669,\"error_count\":3,\"p95_ms\":166},{\"timestamp\":\"2026-09-02T12:01:54Z\",\"service\":\"catalog\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":2682,\"error_count\":4,\"p95_ms\":167},{\"timestamp\":\"2026-09-02T12:01:55Z\",\"service\":\"search\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":2695,\"error_count\":5,\"p95_ms\":145},{\"timestamp\":\"2026-09-02T12:01:56Z\",\"service\":\"api-gateway\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"ok\",\"request_count\":2708,\"error_count\":6,\"p95_ms\":146},{\"timestamp\":\"2026-09-02T12:01:57Z\",\"service\":\"billing\",\"environment\":\"production\",\"region\":\"us-east-1\",\"status\":\"ok\",\"request_count\":2721,\"error_count\":7,\"p95_ms\":147},{\"timestamp\":\"2026-09-02T12:01:58Z\",\"service\":\"catalog\",\"environment\":\"production\",\"region\":\"us-west-2\",\"status\":\"ok\",\"request_count\":2734,\"error_count\":8,\"p95_ms\":148},{\"timestamp\":\"2026-09-02T12:01:59Z\",\"service\":\"search\",\"environment\":\"production\",\"region\":\"eu-west-1\",\"status\":\"degraded\",\"request_count\":2747,\"error_count\":9,\"p95_ms\":149}]" + }, + { + "role": "user", + "content": "For timestamp 2026-09-02T12:00:13Z, return JSON fields service (string), request_count (integer), error_count (integer), p95_ms (integer).\nReturn only one JSON object, with exactly the requested fields and types; no markdown. Use only the supplied evidence. Folded JSON uses __tf_cols__ as column names and __tf_rows__ as row values. A __tf_dict__ array stores values referenced by {\"__tf_ref__\":i} inside __tf_data__." + } + ], + "rendered_tokens": 6310, + "target_tokens": 100000, + "budget_status": "met", + "accounting": { + "counter_id": "encoding:o200k_base:v1", + "scope": "rendered_text", + "original_basis": "exact", + "baseline_basis": "exact", + "final_basis": "exact" + }, + "core_transforms": [], + "scorer_succeeded": false, + "selection_applied": false, + "groups_kept": [], + "groups_dropped": [] + }, + "candidate": { + "messages": [ + { + "role": "user", + "content": "{\"__tf_cols__\":[\"timestamp\",\"service\",\"environment\",\"region\",\"status\",\"request_count\",\"error_count\",\"p95_ms\"],\"__tf_rows__\":[[\"2026-09-02T12:00:00Z\",\"api-gateway\",\"production\",\"us-east-1\",\"degraded\",1200,0,145],[\"2026-09-02T12:00:01Z\",\"billing\",\"production\",\"us-west-2\",\"ok\",1213,1,146],[\"2026-09-02T12:00:02Z\",\"catalog\",\"production\",\"eu-west-1\",\"ok\",1226,2,147],[\"2026-09-02T12:00:03Z\",\"search\",\"production\",\"us-east-1\",\"ok\",1239,3,148],[\"2026-09-02T12:00:04Z\",\"api-gateway\",\"production\",\"us-west-2\",\"ok\",1252,4,149],[\"2026-09-02T12:00:05Z\",\"billing\",\"production\",\"eu-west-1\",\"ok\",1265,5,150],[\"2026-09-02T12:00:06Z\",\"catalog\",\"production\",\"us-east-1\",\"ok\",1278,6,151],[\"2026-09-02T12:00:07Z\",\"search\",\"production\",\"us-west-2\",\"ok\",1291,7,152],[\"2026-09-02T12:00:08Z\",\"api-gateway\",\"production\",\"eu-west-1\",\"ok\",1304,8,153],[\"2026-09-02T12:00:09Z\",\"billing\",\"production\",\"us-east-1\",\"ok\",1317,9,154],[\"2026-09-02T12:00:10Z\",\"catalog\",\"production\",\"us-west-2\",\"ok\",1330,10,155],[\"2026-09-02T12:00:11Z\",\"search\",\"production\",\"eu-west-1\",\"ok\",1343,0,156],[\"2026-09-02T12:00:12Z\",\"api-gateway\",\"production\",\"us-east-1\",\"ok\",1356,1,157],[\"2026-09-02T12:00:13Z\",\"billing\",\"production\",\"us-west-2\",\"ok\",1369,2,158],[\"2026-09-02T12:00:14Z\",\"catalog\",\"production\",\"eu-west-1\",\"ok\",1382,3,159],[\"2026-09-02T12:00:15Z\",\"search\",\"production\",\"us-east-1\",\"ok\",1395,4,160],[\"2026-09-02T12:00:16Z\",\"api-gateway\",\"production\",\"us-west-2\",\"ok\",1408,5,161],[\"2026-09-02T12:00:17Z\",\"billing\",\"production\",\"eu-west-1\",\"degraded\",1421,6,162],[\"2026-09-02T12:00:18Z\",\"catalog\",\"production\",\"us-east-1\",\"ok\",1434,7,163],[\"2026-09-02T12:00:19Z\",\"search\",\"production\",\"us-west-2\",\"ok\",1447,8,164],[\"2026-09-02T12:00:20Z\",\"api-gateway\",\"production\",\"eu-west-1\",\"ok\",1460,9,165],[\"2026-09-02T12:00:21Z\",\"billing\",\"production\",\"us-east-1\",\"ok\",1473,10,166],[\"2026-09-02T12:00:22Z\",\"catalog\",\"production\",\"us-west-2\",\"ok\",1486,0,167],[\"2026-09-02T12:00:23Z\",\"search\",\"production\",\"eu-west-1\",\"ok\",1499,1,145],[\"2026-09-02T12:00:24Z\",\"api-gateway\",\"production\",\"us-east-1\",\"ok\",1512,2,146],[\"2026-09-02T12:00:25Z\",\"billing\",\"production\",\"us-west-2\",\"ok\",1525,3,147],[\"2026-09-02T12:00:26Z\",\"catalog\",\"production\",\"eu-west-1\",\"ok\",1538,4,148],[\"2026-09-02T12:00:27Z\",\"search\",\"production\",\"us-east-1\",\"ok\",1551,5,149],[\"2026-09-02T12:00:28Z\",\"api-gateway\",\"production\",\"us-west-2\",\"ok\",1564,6,150],[\"2026-09-02T12:00:29Z\",\"billing\",\"production\",\"eu-west-1\",\"ok\",1577,7,151],[\"2026-09-02T12:00:30Z\",\"catalog\",\"production\",\"us-east-1\",\"ok\",1590,8,152],[\"2026-09-02T12:00:31Z\",\"search\",\"production\",\"us-west-2\",\"ok\",1603,9,153],[\"2026-09-02T12:00:32Z\",\"api-gateway\",\"production\",\"eu-west-1\",\"ok\",1616,10,154],[\"2026-09-02T12:00:33Z\",\"billing\",\"production\",\"us-east-1\",\"ok\",1629,0,155],[\"2026-09-02T12:00:34Z\",\"catalog\",\"production\",\"us-west-2\",\"degraded\",1642,1,156],[\"2026-09-02T12:00:35Z\",\"search\",\"production\",\"eu-west-1\",\"ok\",1655,2,157],[\"2026-09-02T12:00:36Z\",\"api-gateway\",\"production\",\"us-east-1\",\"ok\",1668,3,158],[\"2026-09-02T12:00:37Z\",\"billing\",\"production\",\"us-west-2\",\"ok\",1681,4,159],[\"2026-09-02T12:00:38Z\",\"catalog\",\"production\",\"eu-west-1\",\"ok\",1694,5,160],[\"2026-09-02T12:00:39Z\",\"search\",\"production\",\"us-east-1\",\"ok\",1707,6,161],[\"2026-09-02T12:00:40Z\",\"api-gateway\",\"production\",\"us-west-2\",\"ok\",1720,7,162],[\"2026-09-02T12:00:41Z\",\"billing\",\"production\",\"eu-west-1\",\"ok\",1733,8,163],[\"2026-09-02T12:00:42Z\",\"catalog\",\"production\",\"us-east-1\",\"ok\",1746,9,164],[\"2026-09-02T12:00:43Z\",\"search\",\"production\",\"us-west-2\",\"ok\",1759,10,165],[\"2026-09-02T12:00:44Z\",\"api-gateway\",\"production\",\"eu-west-1\",\"ok\",1772,0,166],[\"2026-09-02T12:00:45Z\",\"billing\",\"production\",\"us-east-1\",\"ok\",1785,1,167],[\"2026-09-02T12:00:46Z\",\"catalog\",\"production\",\"us-west-2\",\"ok\",1798,2,145],[\"2026-09-02T12:00:47Z\",\"search\",\"production\",\"eu-west-1\",\"ok\",1811,3,146],[\"2026-09-02T12:00:48Z\",\"api-gateway\",\"production\",\"us-east-1\",\"ok\",1824,4,147],[\"2026-09-02T12:00:49Z\",\"billing\",\"production\",\"us-west-2\",\"ok\",1837,5,148],[\"2026-09-02T12:00:50Z\",\"catalog\",\"production\",\"eu-west-1\",\"ok\",1850,6,149],[\"2026-09-02T12:00:51Z\",\"search\",\"production\",\"us-east-1\",\"degraded\",1863,7,150],[\"2026-09-02T12:00:52Z\",\"api-gateway\",\"production\",\"us-west-2\",\"ok\",1876,8,151],[\"2026-09-02T12:00:53Z\",\"billing\",\"production\",\"eu-west-1\",\"ok\",1889,9,152],[\"2026-09-02T12:00:54Z\",\"catalog\",\"production\",\"us-east-1\",\"ok\",1902,10,153],[\"2026-09-02T12:00:55Z\",\"search\",\"production\",\"us-west-2\",\"ok\",1915,0,154],[\"2026-09-02T12:00:56Z\",\"api-gateway\",\"production\",\"eu-west-1\",\"ok\",1928,1,155],[\"2026-09-02T12:00:57Z\",\"billing\",\"production\",\"us-east-1\",\"ok\",1941,2,156],[\"2026-09-02T12:00:58Z\",\"catalog\",\"production\",\"us-west-2\",\"ok\",1954,3,157],[\"2026-09-02T12:00:59Z\",\"search\",\"production\",\"eu-west-1\",\"ok\",1967,4,158],[\"2026-09-02T12:01:00Z\",\"api-gateway\",\"production\",\"us-east-1\",\"ok\",1980,5,159],[\"2026-09-02T12:01:01Z\",\"billing\",\"production\",\"us-west-2\",\"ok\",1993,6,160],[\"2026-09-02T12:01:02Z\",\"catalog\",\"production\",\"eu-west-1\",\"ok\",2006,7,161],[\"2026-09-02T12:01:03Z\",\"search\",\"production\",\"us-east-1\",\"ok\",2019,8,162],[\"2026-09-02T12:01:04Z\",\"api-gateway\",\"production\",\"us-west-2\",\"ok\",2032,9,163],[\"2026-09-02T12:01:05Z\",\"billing\",\"production\",\"eu-west-1\",\"ok\",2045,10,164],[\"2026-09-02T12:01:06Z\",\"catalog\",\"production\",\"us-east-1\",\"ok\",2058,0,165],[\"2026-09-02T12:01:07Z\",\"search\",\"production\",\"us-west-2\",\"ok\",2071,1,166],[\"2026-09-02T12:01:08Z\",\"api-gateway\",\"production\",\"eu-west-1\",\"degraded\",2084,2,167],[\"2026-09-02T12:01:09Z\",\"billing\",\"production\",\"us-east-1\",\"ok\",2097,3,145],[\"2026-09-02T12:01:10Z\",\"catalog\",\"production\",\"us-west-2\",\"ok\",2110,4,146],[\"2026-09-02T12:01:11Z\",\"search\",\"production\",\"eu-west-1\",\"ok\",2123,5,147],[\"2026-09-02T12:01:12Z\",\"api-gateway\",\"production\",\"us-east-1\",\"ok\",2136,6,148],[\"2026-09-02T12:01:13Z\",\"billing\",\"production\",\"us-west-2\",\"ok\",2149,7,149],[\"2026-09-02T12:01:14Z\",\"catalog\",\"production\",\"eu-west-1\",\"ok\",2162,8,150],[\"2026-09-02T12:01:15Z\",\"search\",\"production\",\"us-east-1\",\"ok\",2175,9,151],[\"2026-09-02T12:01:16Z\",\"api-gateway\",\"production\",\"us-west-2\",\"ok\",2188,10,152],[\"2026-09-02T12:01:17Z\",\"billing\",\"production\",\"eu-west-1\",\"ok\",2201,0,153],[\"2026-09-02T12:01:18Z\",\"catalog\",\"production\",\"us-east-1\",\"ok\",2214,1,154],[\"2026-09-02T12:01:19Z\",\"search\",\"production\",\"us-west-2\",\"ok\",2227,2,155],[\"2026-09-02T12:01:20Z\",\"api-gateway\",\"production\",\"eu-west-1\",\"ok\",2240,3,156],[\"2026-09-02T12:01:21Z\",\"billing\",\"production\",\"us-east-1\",\"ok\",2253,4,157],[\"2026-09-02T12:01:22Z\",\"catalog\",\"production\",\"us-west-2\",\"ok\",2266,5,158],[\"2026-09-02T12:01:23Z\",\"search\",\"production\",\"eu-west-1\",\"ok\",2279,6,159],[\"2026-09-02T12:01:24Z\",\"api-gateway\",\"production\",\"us-east-1\",\"ok\",2292,7,160],[\"2026-09-02T12:01:25Z\",\"billing\",\"production\",\"us-west-2\",\"degraded\",2305,8,161],[\"2026-09-02T12:01:26Z\",\"catalog\",\"production\",\"eu-west-1\",\"ok\",2318,9,162],[\"2026-09-02T12:01:27Z\",\"search\",\"production\",\"us-east-1\",\"ok\",2331,10,163],[\"2026-09-02T12:01:28Z\",\"api-gateway\",\"production\",\"us-west-2\",\"ok\",2344,0,164],[\"2026-09-02T12:01:29Z\",\"billing\",\"production\",\"eu-west-1\",\"ok\",2357,1,165],[\"2026-09-02T12:01:30Z\",\"catalog\",\"production\",\"us-east-1\",\"ok\",2370,2,166],[\"2026-09-02T12:01:31Z\",\"search\",\"production\",\"us-west-2\",\"ok\",2383,3,167],[\"2026-09-02T12:01:32Z\",\"api-gateway\",\"production\",\"eu-west-1\",\"ok\",2396,4,145],[\"2026-09-02T12:01:33Z\",\"billing\",\"production\",\"us-east-1\",\"ok\",2409,5,146],[\"2026-09-02T12:01:34Z\",\"catalog\",\"production\",\"us-west-2\",\"ok\",2422,6,147],[\"2026-09-02T12:01:35Z\",\"search\",\"production\",\"eu-west-1\",\"ok\",2435,7,148],[\"2026-09-02T12:01:36Z\",\"api-gateway\",\"production\",\"us-east-1\",\"ok\",2448,8,149],[\"2026-09-02T12:01:37Z\",\"billing\",\"production\",\"us-west-2\",\"ok\",2461,9,150],[\"2026-09-02T12:01:38Z\",\"catalog\",\"production\",\"eu-west-1\",\"ok\",2474,10,151],[\"2026-09-02T12:01:39Z\",\"search\",\"production\",\"us-east-1\",\"ok\",2487,0,152],[\"2026-09-02T12:01:40Z\",\"api-gateway\",\"production\",\"us-west-2\",\"ok\",2500,1,153],[\"2026-09-02T12:01:41Z\",\"billing\",\"production\",\"eu-west-1\",\"ok\",2513,2,154],[\"2026-09-02T12:01:42Z\",\"catalog\",\"production\",\"us-east-1\",\"degraded\",2526,3,155],[\"2026-09-02T12:01:43Z\",\"search\",\"production\",\"us-west-2\",\"ok\",2539,4,156],[\"2026-09-02T12:01:44Z\",\"api-gateway\",\"production\",\"eu-west-1\",\"ok\",2552,5,157],[\"2026-09-02T12:01:45Z\",\"billing\",\"production\",\"us-east-1\",\"ok\",2565,6,158],[\"2026-09-02T12:01:46Z\",\"catalog\",\"production\",\"us-west-2\",\"ok\",2578,7,159],[\"2026-09-02T12:01:47Z\",\"search\",\"production\",\"eu-west-1\",\"ok\",2591,8,160],[\"2026-09-02T12:01:48Z\",\"api-gateway\",\"production\",\"us-east-1\",\"ok\",2604,9,161],[\"2026-09-02T12:01:49Z\",\"billing\",\"production\",\"us-west-2\",\"ok\",2617,10,162],[\"2026-09-02T12:01:50Z\",\"catalog\",\"production\",\"eu-west-1\",\"ok\",2630,0,163],[\"2026-09-02T12:01:51Z\",\"search\",\"production\",\"us-east-1\",\"ok\",2643,1,164],[\"2026-09-02T12:01:52Z\",\"api-gateway\",\"production\",\"us-west-2\",\"ok\",2656,2,165],[\"2026-09-02T12:01:53Z\",\"billing\",\"production\",\"eu-west-1\",\"ok\",2669,3,166],[\"2026-09-02T12:01:54Z\",\"catalog\",\"production\",\"us-east-1\",\"ok\",2682,4,167],[\"2026-09-02T12:01:55Z\",\"search\",\"production\",\"us-west-2\",\"ok\",2695,5,145],[\"2026-09-02T12:01:56Z\",\"api-gateway\",\"production\",\"eu-west-1\",\"ok\",2708,6,146],[\"2026-09-02T12:01:57Z\",\"billing\",\"production\",\"us-east-1\",\"ok\",2721,7,147],[\"2026-09-02T12:01:58Z\",\"catalog\",\"production\",\"us-west-2\",\"ok\",2734,8,148],[\"2026-09-02T12:01:59Z\",\"search\",\"production\",\"eu-west-1\",\"degraded\",2747,9,149]]}" + }, + { + "role": "user", + "content": "For timestamp 2026-09-02T12:00:13Z, return JSON fields service (string), request_count (integer), error_count (integer), p95_ms (integer).\nReturn only one JSON object, with exactly the requested fields and types; no markdown. Use only the supplied evidence. Folded JSON uses __tf_cols__ as column names and __tf_rows__ as row values. A __tf_dict__ array stores values referenced by {\"__tf_ref__\":i} inside __tf_data__." + } + ], + "rendered_tokens": 4061, + "target_tokens": 100000, + "budget_status": "met", + "accounting": { + "counter_id": "encoding:o200k_base:v1", + "scope": "rendered_text", + "original_basis": "exact", + "baseline_basis": "exact", + "final_basis": "exact" + }, + "core_transforms": [ + "json_field_fold" + ], + "scorer_succeeded": false, + "selection_applied": false, + "groups_kept": [], + "groups_dropped": [] + } + } + }, + { + "id": "core-schema-lookup", + "family": "core", + "workload": "json", + "source": "tests/golden/schema_compaction/openai_tool_schema.in.json", + "source_sha256": "056e766b6c7a099cb71e00477fe52f04331b90df81976e1432078f9ad877b081", + "query": "Return the schema name as name, the origin and destination of the fourth parameters.examples entry as origin and destination, and the top-level required array as required. Use strings and an array of strings.", + "oracle": { + "name": "search_flights", + "origin": "NRT", + "destination": "SIN", + "required": [ + "origin", + "destination", + "departure_date" + ] + }, + "arms": { + "baseline": { + "messages": [ + { + "role": "user", + "content": "{\"name\":\"search_flights\",\"description\":\"Searches available commercial flights between two airports on a given travel date.\",\"parameters\":{\"type\":\"object\",\"properties\":{\"origin\":{\"type\":\"string\",\"description\":\"IATA airport code for the departure airport.\"},\"destination\":{\"type\":\"string\",\"description\":\"IATA airport code for the arrival airport.\"},\"cabin_class\":{\"type\":\"string\",\"enum\":[\"economy\",\"premium_economy\",\"business\",\"first\"],\"description\":\"Desired cabin class for the itinerary.\"},\"departure_date\":{\"type\":\"string\",\"description\":\"Departure date in YYYY-MM-DD format.\"}},\"examples\":[{\"origin\":\"JFK\",\"destination\":\"LAX\",\"departure_date\":\"2026-08-01\"},{\"origin\":\"SFO\",\"destination\":\"ORD\",\"departure_date\":\"2026-09-15\"},{\"origin\":\"LHR\",\"destination\":\"CDG\",\"departure_date\":\"2026-10-03\"},{\"origin\":\"NRT\",\"destination\":\"SIN\",\"departure_date\":\"2026-11-20\"},{\"origin\":\"SYD\",\"destination\":\"AKL\",\"departure_date\":\"2026-12-05\"}]},\"required\":[\"origin\",\"destination\",\"departure_date\"]}" + }, + { + "role": "user", + "content": "Return the schema name as name, the origin and destination of the fourth parameters.examples entry as origin and destination, and the top-level required array as required. Use strings and an array of strings.\nReturn only one JSON object, with exactly the requested fields and types; no markdown. Use only the supplied evidence. Folded JSON uses __tf_cols__ as column names and __tf_rows__ as row values. A __tf_dict__ array stores values referenced by {\"__tf_ref__\":i} inside __tf_data__." + } + ], + "rendered_tokens": 232, + "target_tokens": 100000, + "budget_status": "met", + "accounting": { + "counter_id": "encoding:o200k_base:v1", + "scope": "rendered_text", + "original_basis": "exact", + "baseline_basis": "exact", + "final_basis": "exact" + }, + "core_transforms": [], + "scorer_succeeded": false, + "selection_applied": false, + "groups_kept": [], + "groups_dropped": [] + }, + "candidate": { + "messages": [ + { + "role": "user", + "content": "{\"name\":\"search_flights\",\"description\":\"Searches available commercial flights between two airports on a given travel date.\",\"parameters\":{\"type\":\"object\",\"properties\":{\"origin\":{\"type\":\"string\",\"description\":\"IATA airport code for the departure airport.\"},\"destination\":{\"type\":\"string\",\"description\":\"IATA airport code for the arrival airport.\"},\"cabin_class\":{\"type\":\"string\",\"enum\":[\"economy\",\"premium_economy\",\"business\",\"first\"],\"description\":\"Desired cabin class for the itinerary.\"},\"departure_date\":{\"type\":\"string\",\"description\":\"Departure date in YYYY-MM-DD format.\"}},\"examples\":{\"__tf_cols__\":[\"origin\",\"destination\",\"departure_date\"],\"__tf_rows__\":[[\"JFK\",\"LAX\",\"2026-08-01\"],[\"SFO\",\"ORD\",\"2026-09-15\"],[\"LHR\",\"CDG\",\"2026-10-03\"],[\"NRT\",\"SIN\",\"2026-11-20\"],[\"SYD\",\"AKL\",\"2026-12-05\"]]}},\"required\":[\"origin\",\"destination\",\"departure_date\"]}" + }, + { + "role": "user", + "content": "Return the schema name as name, the origin and destination of the fourth parameters.examples entry as origin and destination, and the top-level required array as required. Use strings and an array of strings.\nReturn only one JSON object, with exactly the requested fields and types; no markdown. Use only the supplied evidence. Folded JSON uses __tf_cols__ as column names and __tf_rows__ as row values. A __tf_dict__ array stores values referenced by {\"__tf_ref__\":i} inside __tf_data__." + } + ], + "rendered_tokens": 219, + "target_tokens": 100000, + "budget_status": "met", + "accounting": { + "counter_id": "encoding:o200k_base:v1", + "scope": "rendered_text", + "original_basis": "exact", + "baseline_basis": "exact", + "final_basis": "exact" + }, + "core_transforms": [ + "json_field_fold" + ], + "scorer_succeeded": false, + "selection_applied": false, + "groups_kept": [], + "groups_dropped": [] + } + } + }, + { + "id": "code-branch", + "family": "select", + "workload": "code", + "query": "Using checkout.py, what does shipping_fee(105) return? Return JSON with integer shipping_usd.", + "oracle": { + "shipping_usd": 9 + }, + "supporting_group_position": 0, + "groups": [ + { + "id": "g0", + "content": "Current checkout.py, production implementation:\ndef shipping_fee(subtotal):\n if subtotal >= 120:\n return 0\n return 9\nThe function accepts whole USD and returns whole USD. No other shipping modifier applies.", + "required": false + }, + { + "id": "g1", + "content": "Historical checkout_v1.py, not imported in production:\ndef shipping_fee(subtotal):\n return 4\nThis archived function predates the new threshold. Its fixed fee must not be used for checkout.py.", + "required": false + }, + { + "id": "g2", + "content": "Warehouse notes: order picking begins after payment capture. The warehouse prints one label per parcel. Oversized products use a separate freight workflow. Nothing here overrides checkout.py's ordinary shipping calculation.", + "required": false + }, + { + "id": "g3", + "content": "Discount reference: coupon SPRING takes ten dollars off eligible baskets. The shipping_fee function is passed a subtotal directly, not a coupon name. There is no coupon in this question.", + "required": false + }, + { + "id": "g4", + "content": "Returns reference: return labels may be subsidized by the merchant. A return is not an outbound purchase. The returns subsystem has its own accounting and is not called by shipping_fee.", + "required": false + } + ], + "arms": { + "baseline": { + "messages": [ + { + "role": "user", + "content": "Using checkout.py, what does shipping_fee(105) return? Return JSON with integer shipping_usd.\n\nCurrent checkout.py, production implementation:\ndef shipping_fee(subtotal):\n if subtotal >= 120:\n return 0\n return 9\nThe function accepts whole USD and returns whole USD. No other shipping modifier applies.\n\nHistorical checkout_v1.py, not imported in production:\ndef shipping_fee(subtotal):\n return 4\nThis archived function predates the new threshold. Its fixed fee must not be used for checkout.py.\n\nWarehouse notes: order picking begins after payment capture. The warehouse prints one label per parcel. Oversized products use a separate freight workflow. Nothing here overrides checkout.py's ordinary shipping calculation.\n\nDiscount reference: coupon SPRING takes ten dollars off eligible baskets. The shipping_fee function is passed a subtotal directly, not a coupon name. There is no coupon in this question.\n\nReturns reference: return labels may be subsidized by the merchant. A return is not an outbound purchase. The returns subsystem has its own accounting and is not called by shipping_fee." + }, + { + "role": "user", + "content": "Answer the question stated in the supplied context.\nReturn only one JSON object, with exactly the requested fields and types; no markdown. Use only the supplied evidence. Folded JSON uses __tf_cols__ as column names and __tf_rows__ as row values. A __tf_dict__ array stores values referenced by {\"__tf_ref__\":i} inside __tf_data__." + } + ], + "rendered_tokens": 218, + "target_tokens": 100000, + "budget_status": "met", + "accounting": { + "counter_id": "encoding:o200k_base:v1", + "scope": "rendered_text", + "original_basis": "exact", + "baseline_basis": "exact", + "final_basis": "exact" + }, + "core_transforms": [], + "scorer_succeeded": false, + "selection_applied": false, + "groups_kept": [ + "g0", + "g1", + "g2", + "g3", + "g4" + ], + "groups_dropped": [] + }, + "candidate": { + "messages": [ + { + "role": "user", + "content": "Using checkout.py, what does shipping_fee(105) return? Return JSON with integer shipping_usd.\n\nCurrent checkout.py, production implementation:\ndef shipping_fee(subtotal):\n if subtotal >= 120:\n return 0\n return 9\nThe function accepts whole USD and returns whole USD. No other shipping modifier applies.\n\nHistorical checkout_v1.py, not imported in production:\ndef shipping_fee(subtotal):\n return 4\nThis archived function predates the new threshold. Its fixed fee must not be used for checkout.py." + }, + { + "role": "user", + "content": "Answer the question stated in the supplied context.\nReturn only one JSON object, with exactly the requested fields and types; no markdown. Use only the supplied evidence. Folded JSON uses __tf_cols__ as column names and __tf_rows__ as row values. A __tf_dict__ array stores values referenced by {\"__tf_ref__\":i} inside __tf_data__." + } + ], + "rendered_tokens": 109, + "target_tokens": 109, + "budget_status": "met", + "accounting": { + "counter_id": "encoding:o200k_base:v1", + "scope": "rendered_text", + "original_basis": "exact", + "baseline_basis": "exact", + "final_basis": "exact" + }, + "core_transforms": [], + "scorer_succeeded": true, + "selection_applied": true, + "groups_kept": [ + "g0", + "g1" + ], + "groups_dropped": [ + "g2", + "g3", + "g4" + ] + } + } + }, + { + "id": "code-dependency", + "family": "select", + "workload": "code", + "query": "Using current pricing.py, what is total(17)? Return JSON with integer total_usd.", + "oracle": { + "total_usd": 37 + }, + "supporting_group_position": 1, + "groups": [ + { + "id": "g0", + "content": "Shipping notes: a separate shipping function may return a fee for physical deliveries. The total function shown in current pricing.py contains no shipping call. Do not invent an additional shipping charge.", + "required": false + }, + { + "id": "g1", + "content": "Atomic current pricing.py dependency group (helper and caller belong together):\ndef service_fee():\n return 3\ndef total(subtotal):\n return 2 * subtotal + service_fee()\nBoth functions use whole USD. No rounding, tax or other fees apply.", + "required": false + }, + { + "id": "g2", + "content": "Archived pricing_old.py:\ndef service_fee():\n return 8\ndef total(subtotal):\n return subtotal + service_fee()\nThis file is not imported by current pricing.py and is retained only for migration history.", + "required": false + }, + { + "id": "g3", + "content": "Invoice renderer notes: totals are displayed as whole USD. The receipt footer contains the merchant name and customer support address. Rendering does not recalculate totals or invoke a different fee implementation.", + "required": false + }, + { + "id": "g4", + "content": "Payment settlement notes: the settlement batch runs nightly after order capture. Settlement may split transactions by region. The batch does not change the amount returned by pricing.py during checkout.", + "required": false + } + ], + "arms": { + "baseline": { + "messages": [ + { + "role": "user", + "content": "Using current pricing.py, what is total(17)? Return JSON with integer total_usd.\n\nShipping notes: a separate shipping function may return a fee for physical deliveries. The total function shown in current pricing.py contains no shipping call. Do not invent an additional shipping charge.\n\nAtomic current pricing.py dependency group (helper and caller belong together):\ndef service_fee():\n return 3\ndef total(subtotal):\n return 2 * subtotal + service_fee()\nBoth functions use whole USD. No rounding, tax or other fees apply.\n\nArchived pricing_old.py:\ndef service_fee():\n return 8\ndef total(subtotal):\n return subtotal + service_fee()\nThis file is not imported by current pricing.py and is retained only for migration history.\n\nInvoice renderer notes: totals are displayed as whole USD. The receipt footer contains the merchant name and customer support address. Rendering does not recalculate totals or invoke a different fee implementation.\n\nPayment settlement notes: the settlement batch runs nightly after order capture. Settlement may split transactions by region. The batch does not change the amount returned by pricing.py during checkout." + }, + { + "role": "user", + "content": "Answer the question stated in the supplied context.\nReturn only one JSON object, with exactly the requested fields and types; no markdown. Use only the supplied evidence. Folded JSON uses __tf_cols__ as column names and __tf_rows__ as row values. A __tf_dict__ array stores values referenced by {\"__tf_ref__\":i} inside __tf_data__." + } + ], + "rendered_tokens": 220, + "target_tokens": 100000, + "budget_status": "met", + "accounting": { + "counter_id": "encoding:o200k_base:v1", + "scope": "rendered_text", + "original_basis": "exact", + "baseline_basis": "exact", + "final_basis": "exact" + }, + "core_transforms": [], + "scorer_succeeded": false, + "selection_applied": false, + "groups_kept": [ + "g0", + "g1", + "g2", + "g3", + "g4" + ], + "groups_dropped": [] + }, + "candidate": { + "messages": [ + { + "role": "user", + "content": "Using current pricing.py, what is total(17)? Return JSON with integer total_usd.\n\nShipping notes: a separate shipping function may return a fee for physical deliveries. The total function shown in current pricing.py contains no shipping call. Do not invent an additional shipping charge.\n\nAtomic current pricing.py dependency group (helper and caller belong together):\ndef service_fee():\n return 3\ndef total(subtotal):\n return 2 * subtotal + service_fee()\nBoth functions use whole USD. No rounding, tax or other fees apply." + }, + { + "role": "user", + "content": "Answer the question stated in the supplied context.\nReturn only one JSON object, with exactly the requested fields and types; no markdown. Use only the supplied evidence. Folded JSON uses __tf_cols__ as column names and __tf_rows__ as row values. A __tf_dict__ array stores values referenced by {\"__tf_ref__\":i} inside __tf_data__." + } + ], + "rendered_tokens": 107, + "target_tokens": 110, + "budget_status": "met", + "accounting": { + "counter_id": "encoding:o200k_base:v1", + "scope": "rendered_text", + "original_basis": "exact", + "baseline_basis": "exact", + "final_basis": "exact" + }, + "core_transforms": [], + "scorer_succeeded": true, + "selection_applied": true, + "groups_kept": [ + "g0", + "g1" + ], + "groups_dropped": [ + "g2", + "g3", + "g4" + ] + } + } + }, + { + "id": "logs-causal", + "family": "select", + "workload": "logs", + "query": "For request r-731, identify the failing dependency and its HTTP status. Return JSON with dependency (string) and status_code (integer).", + "oracle": { + "dependency": "inventory", + "status_code": 503 + }, + "supporting_group_position": 2, + "groups": [ + { + "id": "g0", + "content": "Service telemetry: routine liveness probes returned status 200 during this interval. Probe identifiers start with health-. Probe results are not transaction dependency results and do not identify the failing upstream for a customer request.", + "required": false + }, + { + "id": "g1", + "content": "Operator reference: gateway_status is the externally returned gateway response. status_code on a dependency event is the upstream response. Always retain the request identifier when correlating events from multiple services.", + "required": false + }, + { + "id": "g2", + "content": "Atomic trace r-731:\n10:21:00 request=r-731 dependency=inventory attempt=1 status_code=503 message=backend_unavailable\n10:21:01 request=r-731 gateway_status=502 cause=inventory_upstream_failure\nThe gateway status is not the dependency's HTTP status.", + "required": false + }, + { + "id": "g3", + "content": "Trace r-730:\n10:20:00 request=r-730 dependency=payments attempt=1 status_code=200\n10:20:01 request=r-730 gateway_status=200 message=completed\nThis trace belongs to another request and is unrelated to r-731.", + "required": false + }, + { + "id": "g4", + "content": "Trace r-732:\n10:22:00 request=r-732 dependency=inventory attempt=1 status_code=200\n10:22:01 request=r-732 gateway_status=200 message=completed\nA later successful request does not rewrite the earlier r-731 failure.", + "required": false + } + ], + "arms": { + "baseline": { + "messages": [ + { + "role": "user", + "content": "For request r-731, identify the failing dependency and its HTTP status. Return JSON with dependency (string) and status_code (integer).\n\nService telemetry: routine liveness probes returned status 200 during this interval. Probe identifiers start with health-. Probe results are not transaction dependency results and do not identify the failing upstream for a customer request.\n\nOperator reference: gateway_status is the externally returned gateway response. status_code on a dependency event is the upstream response. Always retain the request identifier when correlating events from multiple services.\n\nAtomic trace r-731:\n10:21:00 request=r-731 dependency=inventory attempt=1 status_code=503 message=backend_unavailable\n10:21:01 request=r-731 gateway_status=502 cause=inventory_upstream_failure\nThe gateway status is not the dependency's HTTP status.\n\nTrace r-730:\n10:20:00 request=r-730 dependency=payments attempt=1 status_code=200\n10:20:01 request=r-730 gateway_status=200 message=completed\nThis trace belongs to another request and is unrelated to r-731.\n\nTrace r-732:\n10:22:00 request=r-732 dependency=inventory attempt=1 status_code=200\n10:22:01 request=r-732 gateway_status=200 message=completed\nA later successful request does not rewrite the earlier r-731 failure." + }, + { + "role": "user", + "content": "Answer the question stated in the supplied context.\nReturn only one JSON object, with exactly the requested fields and types; no markdown. Use only the supplied evidence. Folded JSON uses __tf_cols__ as column names and __tf_rows__ as row values. A __tf_dict__ array stores values referenced by {\"__tf_ref__\":i} inside __tf_data__." + } + ], + "rendered_tokens": 278, + "target_tokens": 100000, + "budget_status": "met", + "accounting": { + "counter_id": "encoding:o200k_base:v1", + "scope": "rendered_text", + "original_basis": "exact", + "baseline_basis": "exact", + "final_basis": "exact" + }, + "core_transforms": [], + "scorer_succeeded": false, + "selection_applied": false, + "groups_kept": [ + "g0", + "g1", + "g2", + "g3", + "g4" + ], + "groups_dropped": [] + }, + "candidate": { + "messages": [ + { + "role": "user", + "content": "For request r-731, identify the failing dependency and its HTTP status. Return JSON with dependency (string) and status_code (integer).\n\nOperator reference: gateway_status is the externally returned gateway response. status_code on a dependency event is the upstream response. Always retain the request identifier when correlating events from multiple services.\n\nAtomic trace r-731:\n10:21:00 request=r-731 dependency=inventory attempt=1 status_code=503 message=backend_unavailable\n10:21:01 request=r-731 gateway_status=502 cause=inventory_upstream_failure\nThe gateway status is not the dependency's HTTP status." + }, + { + "role": "user", + "content": "Answer the question stated in the supplied context.\nReturn only one JSON object, with exactly the requested fields and types; no markdown. Use only the supplied evidence. Folded JSON uses __tf_cols__ as column names and __tf_rows__ as row values. A __tf_dict__ array stores values referenced by {\"__tf_ref__\":i} inside __tf_data__." + } + ], + "rendered_tokens": 126, + "target_tokens": 139, + "budget_status": "met", + "accounting": { + "counter_id": "encoding:o200k_base:v1", + "scope": "rendered_text", + "original_basis": "exact", + "baseline_basis": "exact", + "final_basis": "exact" + }, + "core_transforms": [], + "scorer_succeeded": true, + "selection_applied": true, + "groups_kept": [ + "g1", + "g2" + ], + "groups_dropped": [ + "g0", + "g3", + "g4" + ] + } + } + }, + { + "id": "logs-state", + "family": "select", + "workload": "logs", + "query": "What listener port was last successfully activated for search? Return JSON with integer port.", + "oracle": { + "port": 9443 + }, + "supporting_group_position": 3, + "groups": [ + { + "id": "g0", + "content": "Archived search staging configuration: the staging listener used port 8080 in an old environment. This is not a successful activation event for the search service in the supplied live reload history.", + "required": false + }, + { + "id": "g1", + "content": "Health-check notes: probes use whichever listener is currently active. A requested configuration is not proof that the listener changed. The system logs requested_port separately from active_port to make failed activations visible.", + "required": false + }, + { + "id": "g2", + "content": "Operations notes: reload requests are serialized per service. Each success replaces the previous configuration; each failure retains it. There are no later search events beyond the ones in the atomic search history.", + "required": false + }, + { + "id": "g3", + "content": "Atomic search reload history:\n11:00 service=search requested_port=8443 reload=success active_port=8443\n11:05 service=search requested_port=9443 reload=success active_port=9443\n11:09 service=search requested_port=10443 reload=failed active_port=9443\nFailed reloads leave the previous listener active.", + "required": false + }, + { + "id": "g4", + "content": "Billing reload history:\n11:02 service=billing requested_port=7443 reload=success active_port=7443\n11:10 service=billing requested_port=7543 reload=success active_port=7543\nBilling is a separate service with its own listener.", + "required": false + } + ], + "arms": { + "baseline": { + "messages": [ + { + "role": "user", + "content": "What listener port was last successfully activated for search? Return JSON with integer port.\n\nArchived search staging configuration: the staging listener used port 8080 in an old environment. This is not a successful activation event for the search service in the supplied live reload history.\n\nHealth-check notes: probes use whichever listener is currently active. A requested configuration is not proof that the listener changed. The system logs requested_port separately from active_port to make failed activations visible.\n\nOperations notes: reload requests are serialized per service. Each success replaces the previous configuration; each failure retains it. There are no later search events beyond the ones in the atomic search history.\n\nAtomic search reload history:\n11:00 service=search requested_port=8443 reload=success active_port=8443\n11:05 service=search requested_port=9443 reload=success active_port=9443\n11:09 service=search requested_port=10443 reload=failed active_port=9443\nFailed reloads leave the previous listener active.\n\nBilling reload history:\n11:02 service=billing requested_port=7443 reload=success active_port=7443\n11:10 service=billing requested_port=7543 reload=success active_port=7543\nBilling is a separate service with its own listener." + }, + { + "role": "user", + "content": "Answer the question stated in the supplied context.\nReturn only one JSON object, with exactly the requested fields and types; no markdown. Use only the supplied evidence. Folded JSON uses __tf_cols__ as column names and __tf_rows__ as row values. A __tf_dict__ array stores values referenced by {\"__tf_ref__\":i} inside __tf_data__." + } + ], + "rendered_tokens": 256, + "target_tokens": 100000, + "budget_status": "met", + "accounting": { + "counter_id": "encoding:o200k_base:v1", + "scope": "rendered_text", + "original_basis": "exact", + "baseline_basis": "exact", + "final_basis": "exact" + }, + "core_transforms": [], + "scorer_succeeded": false, + "selection_applied": false, + "groups_kept": [ + "g0", + "g1", + "g2", + "g3", + "g4" + ], + "groups_dropped": [] + }, + "candidate": { + "messages": [ + { + "role": "user", + "content": "What listener port was last successfully activated for search? Return JSON with integer port.\n\nArchived search staging configuration: the staging listener used port 8080 in an old environment. This is not a successful activation event for the search service in the supplied live reload history.\n\nAtomic search reload history:\n11:00 service=search requested_port=8443 reload=success active_port=8443\n11:05 service=search requested_port=9443 reload=success active_port=9443\n11:09 service=search requested_port=10443 reload=failed active_port=9443\nFailed reloads leave the previous listener active." + }, + { + "role": "user", + "content": "Answer the question stated in the supplied context.\nReturn only one JSON object, with exactly the requested fields and types; no markdown. Use only the supplied evidence. Folded JSON uses __tf_cols__ as column names and __tf_rows__ as row values. A __tf_dict__ array stores values referenced by {\"__tf_ref__\":i} inside __tf_data__." + } + ], + "rendered_tokens": 126, + "target_tokens": 128, + "budget_status": "met", + "accounting": { + "counter_id": "encoding:o200k_base:v1", + "scope": "rendered_text", + "original_basis": "exact", + "baseline_basis": "exact", + "final_basis": "exact" + }, + "core_transforms": [], + "scorer_succeeded": true, + "selection_applied": true, + "groups_kept": [ + "g0", + "g3" + ], + "groups_dropped": [ + "g1", + "g2", + "g4" + ] + } + } + }, + { + "id": "rag-service", + "family": "select", + "workload": "retrieval", + "query": "For production ledger in eu-west-2, return JSON fields port (integer) and health_path (string).", + "oracle": { + "port": 8717, + "health_path": "/ready-ledger" + }, + "supporting_group_position": 4, + "groups": [ + { + "id": "g0", + "content": "Deployment ledger/staging/eu-west-2: listener port 8081 and readiness endpoint /health-stage. Staging is used for integration tests. Its listener settings must not be copied to the production deployment.", + "required": false + }, + { + "id": "g1", + "content": "Deployment ledger/prod/us-east-1: TLS listener port 9727 and readiness endpoint /ready-us. The US deployment is managed independently from eu-west-2 and has different network settings.", + "required": false + }, + { + "id": "g2", + "content": "Deployment journal/prod/eu-west-2: listener port 8818 and readiness endpoint /ready-journal. Journal and ledger are different services despite sharing a region and an infrastructure team.", + "required": false + }, + { + "id": "g3", + "content": "Operations guide: the deployment identifier combines service, environment and region. Match all three fields before selecting a listener or readiness path. Load balancers route to the deployment-specific listener.", + "required": false + }, + { + "id": "g4", + "content": "Deployment ledger/prod/eu-west-2, current approved configuration: TLS listener port 8717. Readiness endpoint /ready-ledger. This configuration is the authoritative production deployment record for this service and region.", + "required": false + } + ], + "arms": { + "baseline": { + "messages": [ + { + "role": "user", + "content": "For production ledger in eu-west-2, return JSON fields port (integer) and health_path (string).\n\nDeployment ledger/staging/eu-west-2: listener port 8081 and readiness endpoint /health-stage. Staging is used for integration tests. Its listener settings must not be copied to the production deployment.\n\nDeployment ledger/prod/us-east-1: TLS listener port 9727 and readiness endpoint /ready-us. The US deployment is managed independently from eu-west-2 and has different network settings.\n\nDeployment journal/prod/eu-west-2: listener port 8818 and readiness endpoint /ready-journal. Journal and ledger are different services despite sharing a region and an infrastructure team.\n\nOperations guide: the deployment identifier combines service, environment and region. Match all three fields before selecting a listener or readiness path. Load balancers route to the deployment-specific listener.\n\nDeployment ledger/prod/eu-west-2, current approved configuration: TLS listener port 8717. Readiness endpoint /ready-ledger. This configuration is the authoritative production deployment record for this service and region." + }, + { + "role": "user", + "content": "Answer the question stated in the supplied context.\nReturn only one JSON object, with exactly the requested fields and types; no markdown. Use only the supplied evidence. Folded JSON uses __tf_cols__ as column names and __tf_rows__ as row values. A __tf_dict__ array stores values referenced by {\"__tf_ref__\":i} inside __tf_data__." + } + ], + "rendered_tokens": 219, + "target_tokens": 100000, + "budget_status": "met", + "accounting": { + "counter_id": "encoding:o200k_base:v1", + "scope": "rendered_text", + "original_basis": "exact", + "baseline_basis": "exact", + "final_basis": "exact" + }, + "core_transforms": [], + "scorer_succeeded": false, + "selection_applied": false, + "groups_kept": [ + "g0", + "g1", + "g2", + "g3", + "g4" + ], + "groups_dropped": [] + }, + "candidate": { + "messages": [ + { + "role": "user", + "content": "For production ledger in eu-west-2, return JSON fields port (integer) and health_path (string).\n\nDeployment ledger/staging/eu-west-2: listener port 8081 and readiness endpoint /health-stage. Staging is used for integration tests. Its listener settings must not be copied to the production deployment.\n\nDeployment ledger/prod/eu-west-2, current approved configuration: TLS listener port 8717. Readiness endpoint /ready-ledger. This configuration is the authoritative production deployment record for this service and region." + }, + { + "role": "user", + "content": "Answer the question stated in the supplied context.\nReturn only one JSON object, with exactly the requested fields and types; no markdown. Use only the supplied evidence. Folded JSON uses __tf_cols__ as column names and __tf_rows__ as row values. A __tf_dict__ array stores values referenced by {\"__tf_ref__\":i} inside __tf_data__." + } + ], + "rendered_tokens": 107, + "target_tokens": 109, + "budget_status": "met", + "accounting": { + "counter_id": "encoding:o200k_base:v1", + "scope": "rendered_text", + "original_basis": "exact", + "baseline_basis": "exact", + "final_basis": "exact" + }, + "core_transforms": [], + "scorer_succeeded": true, + "selection_applied": true, + "groups_kept": [ + "g0", + "g4" + ], + "groups_dropped": [ + "g1", + "g2", + "g3" + ] + } + } + }, + { + "id": "rag-invoice", + "family": "select", + "workload": "retrieval", + "query": "For invoice INV-274, compute subtotal minus discount plus shipping. Return JSON with integer total_usd. No tax or other fees apply.", + "oracle": { + "total_usd": 87 + }, + "supporting_group_position": 0, + "groups": [ + { + "id": "g0", + "content": "Atomic INV-274 invoice and adjustments: subtotal 100 USD; discount 20 USD; shipping 7 USD. These values belong to the same invoice and must be used together. There is no tax or additional fee. The discount is subtracted once, then shipping is added once.", + "required": false + }, + { + "id": "g1", + "content": "Invoice INV-273: subtotal 120 USD; discount 10 USD; shipping 4 USD. This is a separate paid invoice. It supplies no amounts for INV-274 and must not be included in its final total.", + "required": false + }, + { + "id": "g2", + "content": "General shipping guide: some customers qualify for free shipping, while other orders have a default shipping estimate of 12 USD. The actual shipping amount recorded on an invoice overrides these general estimates.", + "required": false + }, + { + "id": "g3", + "content": "Refund policy: approved refunds are paid to the original payment method. A refund is distinct from an invoice discount. No refund has been recorded for INV-274 in the supplied material.", + "required": false + }, + { + "id": "g4", + "content": "Billing support guide: quote the invoice identifier when asking about totals. Support agents may explain charges but do not silently modify posted invoice amounts. Only the stated adjustments apply here.", + "required": false + } + ], + "arms": { + "baseline": { + "messages": [ + { + "role": "user", + "content": "For invoice INV-274, compute subtotal minus discount plus shipping. Return JSON with integer total_usd. No tax or other fees apply.\n\nAtomic INV-274 invoice and adjustments: subtotal 100 USD; discount 20 USD; shipping 7 USD. These values belong to the same invoice and must be used together. There is no tax or additional fee. The discount is subtracted once, then shipping is added once.\n\nInvoice INV-273: subtotal 120 USD; discount 10 USD; shipping 4 USD. This is a separate paid invoice. It supplies no amounts for INV-274 and must not be included in its final total.\n\nGeneral shipping guide: some customers qualify for free shipping, while other orders have a default shipping estimate of 12 USD. The actual shipping amount recorded on an invoice overrides these general estimates.\n\nRefund policy: approved refunds are paid to the original payment method. A refund is distinct from an invoice discount. No refund has been recorded for INV-274 in the supplied material.\n\nBilling support guide: quote the invoice identifier when asking about totals. Support agents may explain charges but do not silently modify posted invoice amounts. Only the stated adjustments apply here." + }, + { + "role": "user", + "content": "Answer the question stated in the supplied context.\nReturn only one JSON object, with exactly the requested fields and types; no markdown. Use only the supplied evidence. Folded JSON uses __tf_cols__ as column names and __tf_rows__ as row values. A __tf_dict__ array stores values referenced by {\"__tf_ref__\":i} inside __tf_data__." + } + ], + "rendered_tokens": 237, + "target_tokens": 100000, + "budget_status": "met", + "accounting": { + "counter_id": "encoding:o200k_base:v1", + "scope": "rendered_text", + "original_basis": "exact", + "baseline_basis": "exact", + "final_basis": "exact" + }, + "core_transforms": [], + "scorer_succeeded": false, + "selection_applied": false, + "groups_kept": [ + "g0", + "g1", + "g2", + "g3", + "g4" + ], + "groups_dropped": [] + }, + "candidate": { + "messages": [ + { + "role": "user", + "content": "For invoice INV-274, compute subtotal minus discount plus shipping. Return JSON with integer total_usd. No tax or other fees apply.\n\nAtomic INV-274 invoice and adjustments: subtotal 100 USD; discount 20 USD; shipping 7 USD. These values belong to the same invoice and must be used together. There is no tax or additional fee. The discount is subtracted once, then shipping is added once." + }, + { + "role": "user", + "content": "Answer the question stated in the supplied context.\nReturn only one JSON object, with exactly the requested fields and types; no markdown. Use only the supplied evidence. Folded JSON uses __tf_cols__ as column names and __tf_rows__ as row values. A __tf_dict__ array stores values referenced by {\"__tf_ref__\":i} inside __tf_data__." + } + ], + "rendered_tokens": 85, + "target_tokens": 118, + "budget_status": "met", + "accounting": { + "counter_id": "encoding:o200k_base:v1", + "scope": "rendered_text", + "original_basis": "exact", + "baseline_basis": "exact", + "final_basis": "exact" + }, + "core_transforms": [], + "scorer_succeeded": true, + "selection_applied": true, + "groups_kept": [ + "g0" + ], + "groups_dropped": [ + "g1", + "g2", + "g3", + "g4" + ] + } + } + }, + { + "id": "conversation-override", + "family": "select", + "workload": "multi-turn", + "query": "What deployment region did the user finally approve for canary c-52? Return JSON with string region.", + "oracle": { + "region": "eu-west-2" + }, + "supporting_group_position": 1, + "groups": [ + { + "id": "g0", + "content": "Intermediate exchange: user: Change the proposed region for c-52 to eu-central-1. assistant: I have updated the proposal to eu-central-1. This proposal had not yet received final approval.", + "required": false + }, + { + "id": "g1", + "content": "Final approval exchange, latest c-52 decision: user: The final approved region for canary c-52 is eu-west-2, not either earlier region. assistant: Confirmed, the final approved c-52 region is eu-west-2.", + "required": false + }, + { + "id": "g2", + "content": "Unrelated exchange: user: Keep canary c-51 in ap-south-1. assistant: Confirmed, canary c-51 remains in ap-south-1. This is a different canary and must not override the c-52 decision.", + "required": false + }, + { + "id": "g3", + "content": "Runbook: a later explicit final approval supersedes earlier proposals for the same canary. Keep the canary identifier attached to each decision. An unrelated canary's configuration is not an update to c-52.", + "required": false + }, + { + "id": "g4", + "content": "Earlier exchange: user: Start canary c-52 in us-east-1. assistant: I have recorded the initial region as us-east-1. This was an early proposal, before the later corrections.", + "required": false + } + ], + "arms": { + "baseline": { + "messages": [ + { + "role": "user", + "content": "What deployment region did the user finally approve for canary c-52? Return JSON with string region.\n\nIntermediate exchange: user: Change the proposed region for c-52 to eu-central-1. assistant: I have updated the proposal to eu-central-1. This proposal had not yet received final approval.\n\nFinal approval exchange, latest c-52 decision: user: The final approved region for canary c-52 is eu-west-2, not either earlier region. assistant: Confirmed, the final approved c-52 region is eu-west-2.\n\nUnrelated exchange: user: Keep canary c-51 in ap-south-1. assistant: Confirmed, canary c-51 remains in ap-south-1. This is a different canary and must not override the c-52 decision.\n\nRunbook: a later explicit final approval supersedes earlier proposals for the same canary. Keep the canary identifier attached to each decision. An unrelated canary's configuration is not an update to c-52.\n\nEarlier exchange: user: Start canary c-52 in us-east-1. assistant: I have recorded the initial region as us-east-1. This was an early proposal, before the later corrections." + }, + { + "role": "user", + "content": "Answer the question stated in the supplied context.\nReturn only one JSON object, with exactly the requested fields and types; no markdown. Use only the supplied evidence. Folded JSON uses __tf_cols__ as column names and __tf_rows__ as row values. A __tf_dict__ array stores values referenced by {\"__tf_ref__\":i} inside __tf_data__." + } + ], + "rendered_tokens": 251, + "target_tokens": 100000, + "budget_status": "met", + "accounting": { + "counter_id": "encoding:o200k_base:v1", + "scope": "rendered_text", + "original_basis": "exact", + "baseline_basis": "exact", + "final_basis": "exact" + }, + "core_transforms": [], + "scorer_succeeded": false, + "selection_applied": false, + "groups_kept": [ + "g0", + "g1", + "g2", + "g3", + "g4" + ], + "groups_dropped": [] + }, + "candidate": { + "messages": [ + { + "role": "user", + "content": "What deployment region did the user finally approve for canary c-52? Return JSON with string region.\n\nIntermediate exchange: user: Change the proposed region for c-52 to eu-central-1. assistant: I have updated the proposal to eu-central-1. This proposal had not yet received final approval.\n\nFinal approval exchange, latest c-52 decision: user: The final approved region for canary c-52 is eu-west-2, not either earlier region. assistant: Confirmed, the final approved c-52 region is eu-west-2." + }, + { + "role": "user", + "content": "Answer the question stated in the supplied context.\nReturn only one JSON object, with exactly the requested fields and types; no markdown. Use only the supplied evidence. Folded JSON uses __tf_cols__ as column names and __tf_rows__ as row values. A __tf_dict__ array stores values referenced by {\"__tf_ref__\":i} inside __tf_data__." + } + ], + "rendered_tokens": 113, + "target_tokens": 125, + "budget_status": "met", + "accounting": { + "counter_id": "encoding:o200k_base:v1", + "scope": "rendered_text", + "original_basis": "exact", + "baseline_basis": "exact", + "final_basis": "exact" + }, + "core_transforms": [], + "scorer_succeeded": true, + "selection_applied": true, + "groups_kept": [ + "g0", + "g1" + ], + "groups_dropped": [ + "g2", + "g3", + "g4" + ] + } + } + }, + { + "id": "json-abstention", + "family": "select", + "workload": "json", + "query": "What retention_days is configured for dataset cobalt? If it is not stated, return JSON with retention_days set to null. Never substitute another dataset's value.", + "oracle": { + "retention_days": null + }, + "supporting_group_position": 2, + "groups": [ + { + "id": "g0", + "content": "Catalog documentation: absence of a retention field means the value is unknown in this snapshot. Do not invent a default. Operational policies may exist elsewhere but are not supplied here.", + "required": false + }, + { + "id": "g1", + "content": "Storage documentation: encryption, region and owner fields do not determine retention duration. A dataset's retention setting must be explicitly provided for that dataset, not inferred from another record.", + "required": false + }, + { + "id": "g2", + "content": "Current cobalt record: {\"dataset\":\"cobalt\",\"region\":\"eu-west-2\",\"owner\":\"analytics\",\"encryption\":\"enabled\"}. This record contains no retention_days field and no other cobalt retention setting is provided.", + "required": false + }, + { + "id": "g3", + "content": "Separate amber record: {\"dataset\":\"amber\",\"retention_days\":30,\"region\":\"us-east-1\"}. Amber is not cobalt; its retention period applies only to amber.", + "required": false + }, + { + "id": "g4", + "content": "Separate jade record: {\"dataset\":\"jade\",\"retention_days\":90,\"region\":\"eu-west-2\"}. Sharing a region with cobalt does not imply sharing its retention policy.", + "required": false + } + ], + "arms": { + "baseline": { + "messages": [ + { + "role": "user", + "content": "What retention_days is configured for dataset cobalt? If it is not stated, return JSON with retention_days set to null. Never substitute another dataset's value.\n\nCatalog documentation: absence of a retention field means the value is unknown in this snapshot. Do not invent a default. Operational policies may exist elsewhere but are not supplied here.\n\nStorage documentation: encryption, region and owner fields do not determine retention duration. A dataset's retention setting must be explicitly provided for that dataset, not inferred from another record.\n\nCurrent cobalt record: {\"dataset\":\"cobalt\",\"region\":\"eu-west-2\",\"owner\":\"analytics\",\"encryption\":\"enabled\"}. This record contains no retention_days field and no other cobalt retention setting is provided.\n\nSeparate amber record: {\"dataset\":\"amber\",\"retention_days\":30,\"region\":\"us-east-1\"}. Amber is not cobalt; its retention period applies only to amber.\n\nSeparate jade record: {\"dataset\":\"jade\",\"retention_days\":90,\"region\":\"eu-west-2\"}. Sharing a region with cobalt does not imply sharing its retention policy." + }, + { + "role": "user", + "content": "Answer the question stated in the supplied context.\nReturn only one JSON object, with exactly the requested fields and types; no markdown. Use only the supplied evidence. Folded JSON uses __tf_cols__ as column names and __tf_rows__ as row values. A __tf_dict__ array stores values referenced by {\"__tf_ref__\":i} inside __tf_data__." + } + ], + "rendered_tokens": 214, + "target_tokens": 100000, + "budget_status": "met", + "accounting": { + "counter_id": "encoding:o200k_base:v1", + "scope": "rendered_text", + "original_basis": "exact", + "baseline_basis": "exact", + "final_basis": "exact" + }, + "core_transforms": [], + "scorer_succeeded": false, + "selection_applied": false, + "groups_kept": [ + "g0", + "g1", + "g2", + "g3", + "g4" + ], + "groups_dropped": [] + }, + "candidate": { + "messages": [ + { + "role": "user", + "content": "What retention_days is configured for dataset cobalt? If it is not stated, return JSON with retention_days set to null. Never substitute another dataset's value.\n\nCurrent cobalt record: {\"dataset\":\"cobalt\",\"region\":\"eu-west-2\",\"owner\":\"analytics\",\"encryption\":\"enabled\"}. This record contains no retention_days field and no other cobalt retention setting is provided." + }, + { + "role": "user", + "content": "Answer the question stated in the supplied context.\nReturn only one JSON object, with exactly the requested fields and types; no markdown. Use only the supplied evidence. Folded JSON uses __tf_cols__ as column names and __tf_rows__ as row values. A __tf_dict__ array stores values referenced by {\"__tf_ref__\":i} inside __tf_data__." + } + ], + "rendered_tokens": 74, + "target_tokens": 107, + "budget_status": "met", + "accounting": { + "counter_id": "encoding:o200k_base:v1", + "scope": "rendered_text", + "original_basis": "exact", + "baseline_basis": "exact", + "final_basis": "exact" + }, + "core_transforms": [], + "scorer_succeeded": true, + "selection_applied": true, + "groups_kept": [ + "g2" + ], + "groups_dropped": [ + "g0", + "g1", + "g3", + "g4" + ] + } + } + } + ] +} diff --git a/eval/research/sdk_quality_results.json b/eval/research/sdk_quality_results.json new file mode 100644 index 0000000..48558bf --- /dev/null +++ b/eval/research/sdk_quality_results.json @@ -0,0 +1,2135 @@ +{ + "schema_version": 1, + "kind": "tokenfold.sdk-quality.diagnostic-results", + "date": "2026-11-10", + "corpus": "eval/research/sdk_quality_corpus.json", + "corpus_canonical_json_sha256": "c09c30b004b0bd4405de30b51ac2f215eaad26b5fb3288f8d790ae31fda0474c", + "sdk_version": "0.6.0", + "source_commit": "3304fe2340b29ab72f96783283d7af0eebc9840a", + "working_tree_dirty": true, + "frozen_scope_sha256": "4915e484e1ab80cf1cc0a503f06ca91b2deb431a7e80d80a20ae3881504cd87a", + "frozen_prepared_sha256": "c4c87bda9e295ffc965d511e0c3f33cebdf2eec6f23efef8b2378a081bdbf9e6", + "frozen_executor_sha256": "00ae054faeac1e3ab4af9684abb8980f09796725c97380b936163147c2b35679", + "frozen_adapter_sha256": "9235d8badab1b94d43c6855083cbf101f68523e3620e0d12d005a39dd5db6ff6", + "model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "nvidia", + "observed_provider": "Nvidia", + "provider_endpoint_name": "Nvidia | nvidia/nemotron-3.5-super-vl-20261015:free", + "immutable_downstream_checkpoint": null, + "system_fingerprint": null, + "qualification": false, + "selection_runtime": { + "model_revision": "fd8231d3cdc1efc46f7a6d7d83ca88b336a953591d63ea431407c8b13b990309", + "device": "cuda:0", + "precision": "fp32-parameters-bf16-autocast", + "config_sha256": "a5f91285e6823915313e67bb0b500cd074af253e890c3762a77a948df538897a" + }, + "preparation_history": [ + "Schema required-field oracle corrected before hosted answers; original scope retained.", + "First-group positional bias corrected by balanced supporting-group positions before hosted answers; original local preparations retained.", + "Scope is exposed/development diagnostic, not untouched held-out qualification." + ], + "prior_scopes": [ + { + "model": "google/gemma-4-31b-it:free", + "provider": "google-ai-studio", + "probe_attempts": 1, + "probe_http_status": 429, + "probe_usage": null, + "probe_cost_usd": null, + "quality_calls_planned": 48, + "quality_calls_attempted": 0, + "quality_calls_unexecuted": 48, + "reason": "upstream shared pool rate limited; terminal scope, no retry" + } + ], + "nvidia_probe": { + "attempts": 1, + "finish_reason": "stop", + "oracle_pass": true, + "usage": { + "prompt_tokens": 38, + "completion_tokens": 6, + "total_tokens": 44, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 638.2947, + "cost_coverage_reason": "request_price_unreported" + }, + "all_attempt_accounting": { + "actual_generation_attempts": 50, + "successful_quality_replies": 48, + "successful_probe_replies": 1, + "failed_probe_attempts": 1, + "unknown_usage_attempts": 1, + "unknown_reported_cost_attempts": 1, + "nvidia_observed_prompt_tokens": 50256, + "nvidia_observed_completion_tokens": 749, + "nvidia_known_reported_charge_usd": "0", + "complete_campaign_spend_usd": null, + "complete_cost_savings_pct": null, + "reason": "Failed Gemma probe usage/cost unknown; NVIDIA request pricing unreported; provider receipts are not audited invoices." + }, + "families": { + "core": { + "tasks": 4, + "trials_per_task": 2, + "arms": { + "baseline": { + "trials": 8, + "oracle_passes": 8, + "provider_prompt_tokens": 24496, + "provider_completion_tokens": 202, + "provider_cached_tokens": 4336, + "local_rendered_tokens": 18988, + "provider_call_wall_median_ms": 761.94675 + }, + "candidate": { + "trials": 8, + "oracle_passes": 6, + "provider_prompt_tokens": 16600, + "provider_completion_tokens": 204, + "provider_cached_tokens": 4336, + "local_rendered_tokens": 12044, + "provider_call_wall_median_ms": 849.91875 + } + }, + "paired": { + "both_correct": 6, + "baseline_only": 2, + "candidate_only": 0, + "neither": 0, + "incomplete": 0 + }, + "provider_prompt_reduction_pct": 32.233834 + }, + "select": { + "tasks": 8, + "trials_per_task": 2, + "arms": { + "baseline": { + "trials": 16, + "oracle_passes": 16, + "provider_prompt_tokens": 5706, + "provider_completion_tokens": 168, + "provider_cached_tokens": 0, + "local_rendered_tokens": 3786, + "provider_call_wall_median_ms": 608.2652 + }, + "candidate": { + "trials": 16, + "oracle_passes": 16, + "provider_prompt_tokens": 3416, + "provider_completion_tokens": 169, + "provider_cached_tokens": 0, + "local_rendered_tokens": 1694, + "provider_call_wall_median_ms": 603.0378 + } + }, + "paired": { + "both_correct": 16, + "baseline_only": 0, + "candidate_only": 0, + "neither": 0, + "incomplete": 0 + }, + "provider_prompt_reduction_pct": 40.133193 + } + }, + "calls": [ + { + "id": "core-api-lookup/0/baseline", + "task": "core-api-lookup", + "family": "core", + "repeat": 0, + "arm": "baseline", + "body_sha256": "10f8eabb0bc897da50c268e138a541cf941cf6bf72e26594b6e09c9255e029c5", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"name\":\"Niklaus Wirth\",\"active\":true,\"tier\":\"free\",\"seats\":1}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 2990, + "completion_tokens": 24, + "total_tokens": 3014, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 759.0706, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "2090f72a2da6db27cd14990a1173e2c531c3d995f3446dffb86e02ed81b8d22f" + }, + { + "id": "core-api-lookup/0/candidate", + "task": "core-api-lookup", + "family": "core", + "repeat": 0, + "arm": "candidate", + "body_sha256": "e58f4fe9b1e8d6a7858f18ec009ec5ce5935d56cb60f36e4af2537d0ede511b7", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"name\":\"Niklaus Wirth\",\"active\":true,\"tier\":\"team\",\"seats\":25}", + "oracle_pass": false, + "usage": { + "prompt_tokens": 1800, + "completion_tokens": 25, + "total_tokens": 1825, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 816.7523, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "561cdfbd6c18a36f21ac304d85882f0c77c09c6adfa967bc66e80a6bd0dead6e" + }, + { + "id": "core-users-lookup/0/baseline", + "task": "core-users-lookup", + "family": "core", + "repeat": 0, + "arm": "baseline", + "body_sha256": "d5d8332ab358f178cd7dc4ee9b2204ccc12512436e46352421e686851bd9bbc1", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"name\":\"John Backus\",\"role\":\"member\",\"active\":true}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 791, + "completion_tokens": 16, + "total_tokens": 807, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 688.2292, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "c39e1162eac0722f1660f150262fe2c3f4904f1fb72addf9633d5556974a1642" + }, + { + "id": "core-users-lookup/0/candidate", + "task": "core-users-lookup", + "family": "core", + "repeat": 0, + "arm": "candidate", + "body_sha256": "27e89c0cbc3c7f566d2e03f1764d83c9707c8d05ff9c21ec8402706a346363cf", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"name\":\"John Backus\",\"role\":\"member\",\"active\":true}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 543, + "completion_tokens": 16, + "total_tokens": 559, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 2527.2407, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "f9523c9f3d9739bf71da43d5f34f6f452690934589e06d374d35aab57a41f86f" + }, + { + "id": "core-wide-lookup/0/baseline", + "task": "core-wide-lookup", + "family": "core", + "repeat": 0, + "arm": "baseline", + "body_sha256": "fb44422d53464ee199f11d507fca6e01b8d7b6cfba89659df6da236cb0b8de4a", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"service\":\"billing\",\"request_count\":1369,\"error_count\":2,\"p95_ms\":158}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 8053, + "completion_tokens": 29, + "total_tokens": 8082, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 1119.8069, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "c88e946aff8137f718f70f32bc80133d8b04de06e041acd84981bfd16f8c1456" + }, + { + "id": "core-wide-lookup/0/candidate", + "task": "core-wide-lookup", + "family": "core", + "repeat": 0, + "arm": "candidate", + "body_sha256": "88b19761daa5cd338ba59799544a7267ca9078215a7bc089455d70692596af26", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"service\":\"billing\",\"request_count\":1369,\"error_count\":2,\"p95_ms\":158}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 5567, + "completion_tokens": 29, + "total_tokens": 5596, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 1520.8215, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "0f21b53142b4efb31d0b458b5448cdca4251f1fcc82e4b81cfb17473aab3a1ec" + }, + { + "id": "core-schema-lookup/0/baseline", + "task": "core-schema-lookup", + "family": "core", + "repeat": 0, + "arm": "baseline", + "body_sha256": "9db8779caf97fc08243fbe65874a0b080c172269fca02ee82712a65866d69b87", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"name\":\"search_flights\",\"origin\":\"NRT\",\"destination\":\"SIN\",\"required\":[\"origin\",\"destination\",\"departure_date\"]}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 414, + "completion_tokens": 32, + "total_tokens": 446, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 697.1284, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "f3873a9b14a6aa37d0f7a4d112080bc28ffe58900108ea75c31f3cfbfe140706" + }, + { + "id": "core-schema-lookup/0/candidate", + "task": "core-schema-lookup", + "family": "core", + "repeat": 0, + "arm": "candidate", + "body_sha256": "f895ba46bcd7ac89e162ce339675e6ed752d3007eca911fa07525f533ffcd086", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"name\":\"search_flights\",\"origin\":\"NRT\",\"destination\":\"SIN\",\"required\":[\"origin\",\"destination\",\"departure_date\"]}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 390, + "completion_tokens": 32, + "total_tokens": 422, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 883.0852, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "5acf39307ae26dee164834a4e975857747e2b7f5e73343922927b7a56bc00bb6" + }, + { + "id": "code-branch/0/baseline", + "task": "code-branch", + "family": "select", + "repeat": 0, + "arm": "baseline", + "body_sha256": "1fdc22ad29a779925c1c731688e0ba4c0df3015527339af3e3d3c69ac8fba8f1", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"shipping_usd\": 9}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 332, + "completion_tokens": 9, + "total_tokens": 341, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 646.3815, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "bce46ab3de904819bc9403487c48686d4c57f817c96dd1e74fa6f7003057c3a5" + }, + { + "id": "code-branch/0/candidate", + "task": "code-branch", + "family": "select", + "repeat": 0, + "arm": "candidate", + "body_sha256": "956af1c1196dfaf3b6112aa1e860d4fb4b629dbea94a72af89758cc8e17698bb", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"shipping_usd\": 9}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 216, + "completion_tokens": 9, + "total_tokens": 225, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 788.7132, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "4db07cc6e1c886138c70213657450ae66e38ce98436257e3ec12e5b56779a959" + }, + { + "id": "code-dependency/0/baseline", + "task": "code-dependency", + "family": "select", + "repeat": 0, + "arm": "baseline", + "body_sha256": "aa3afa369b7bb010f1b27953cbc33a42c285ca7a4d890130df0ca47c5adbc358", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"total_usd\": 37}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 328, + "completion_tokens": 10, + "total_tokens": 338, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 533.292, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "9a2d16b2e882f8f605fff6018a7affc644d5b5bbaa5ceb78dec30c132ccee63e" + }, + { + "id": "code-dependency/0/candidate", + "task": "code-dependency", + "family": "select", + "repeat": 0, + "arm": "candidate", + "body_sha256": "6e86e36ca9c718670a598b8b616fb8c1969961cbc7ffed5bf67ce9bfd98e72bb", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"total_usd\": 37}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 209, + "completion_tokens": 10, + "total_tokens": 219, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 562.4004, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "b1bc563daedf6a80da966a0f270e9810635cd46c3528a147d7f2ce6288608ef3" + }, + { + "id": "logs-causal/0/baseline", + "task": "logs-causal", + "family": "select", + "repeat": 0, + "arm": "baseline", + "body_sha256": "055faa4591653929fb164fbbf39b4efd2dc44f2991e4089ef831d4ff1d0bc445", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"dependency\":\"inventory\",\"status_code\":503}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 435, + "completion_tokens": 13, + "total_tokens": 448, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 577.1023, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "99c5934afb6c1699986473505e94b091a38dae4ba80585f11f554c2c797c1995" + }, + { + "id": "logs-causal/0/candidate", + "task": "logs-causal", + "family": "select", + "repeat": 0, + "arm": "candidate", + "body_sha256": "9fe221606c6e225006218363d0a2f734bd4624e06c9577517f1a3e3462961941", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"dependency\":\"inventory\",\"status_code\":503}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 243, + "completion_tokens": 13, + "total_tokens": 256, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 597.045, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "5721a7a7a3bc82c073cd629c68953e06ac53350e34aff98c3e88202e6a14c9be" + }, + { + "id": "logs-state/0/baseline", + "task": "logs-state", + "family": "select", + "repeat": 0, + "arm": "baseline", + "body_sha256": "30b543e2ef56bc0691afa68a97e3cb63bad075f939115b877d2a2274cce3f5af", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"port\":9443}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 388, + "completion_tokens": 9, + "total_tokens": 397, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 591.1002, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "7a21c41a4f3011420eb49a0af5c06213f1f5cfe52148ce4c9a9031696b731711" + }, + { + "id": "logs-state/0/candidate", + "task": "logs-state", + "family": "select", + "repeat": 0, + "arm": "candidate", + "body_sha256": "5da1023cd0441e276452e28e3d5177d70f0b4b18efcc7df30ca35ce0a007e1d1", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"port\":9443}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 246, + "completion_tokens": 9, + "total_tokens": 255, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 553.255, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "a8ed54064558b7ce60c9309e0ca351d614ecf10897f447cda10efe83cac9dde1" + }, + { + "id": "rag-service/0/baseline", + "task": "rag-service", + "family": "select", + "repeat": 0, + "arm": "baseline", + "body_sha256": "4d721eb1aec96def187744f16e0b50597065ad6975bc588d35f81e6be02c49cd", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"port\":8717,\"health_path\":\"/ready-ledger\"}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 330, + "completion_tokens": 17, + "total_tokens": 347, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 643.9423, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "adf9c0bf2443b3f79dc97bb2990c300e0fb908149f543c9e750deb9604bf6889" + }, + { + "id": "rag-service/0/candidate", + "task": "rag-service", + "family": "select", + "repeat": 0, + "arm": "candidate", + "body_sha256": "2493d80b5174b11cbcdb67102c81dbaad3dad3e7cdc536082cf176db08835f43", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"port\":8717,\"health_path\":\"/ready-ledger\"}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 211, + "completion_tokens": 17, + "total_tokens": 228, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 602.4317, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "b5a8b7eb1c7a6a98e08bdffe17bc87629e62a90d56dde9e0a79a39a65f7f76cf" + }, + { + "id": "rag-invoice/0/baseline", + "task": "rag-invoice", + "family": "select", + "repeat": 0, + "arm": "baseline", + "body_sha256": "09978a15bec792f62cc676b8aa7373d72c299ac713b2d41e179acd81509dc806", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"total_usd\":87}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 361, + "completion_tokens": 9, + "total_tokens": 370, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 588.4601, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "42aee1fdc0b0347ddfc75b5a35fa3ba9f45dd8c256b8aa40a363688002044594" + }, + { + "id": "rag-invoice/0/candidate", + "task": "rag-invoice", + "family": "select", + "repeat": 0, + "arm": "candidate", + "body_sha256": "702f4d818e63c2a157b5aa78945347ba7a306cd2f50466eeefe68cddf18abbd9", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"total_usd\":87}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 193, + "completion_tokens": 9, + "total_tokens": 202, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 603.6439, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "67bec2e7930240557caf168919a0f0e2c12db7108bde98bd1222e11b66986571" + }, + { + "id": "conversation-override/0/baseline", + "task": "conversation-override", + "family": "select", + "repeat": 0, + "arm": "baseline", + "body_sha256": "624efa44defb893827cb2ca485e2d6dfe267ed18f13adb44aefe67292b5a297c", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"region\":\"eu-west-2\"}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 362, + "completion_tokens": 9, + "total_tokens": 371, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 602.4025, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "e6132205c80510e835f1938b345ea9347d2809b0c50608a8eb2f3524ee2fba56" + }, + { + "id": "conversation-override/0/candidate", + "task": "conversation-override", + "family": "select", + "repeat": 0, + "arm": "candidate", + "body_sha256": "204937bab72147868290a449fbda1d67f828a1c19ed967d143cfd192819472cb", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"region\":\"eu-west-2\"}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 218, + "completion_tokens": 9, + "total_tokens": 227, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 519.6564, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "aa4b7ea2b7c5a5c909867d71094dc012493446d60b0c6940373c23d905305d46" + }, + { + "id": "json-abstention/0/baseline", + "task": "json-abstention", + "family": "select", + "repeat": 0, + "arm": "baseline", + "body_sha256": "d078048b4938108f632b1fdda758f2a43434fb59b41258a4fc804368da9c1288", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"retention_days\": null}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 317, + "completion_tokens": 8, + "total_tokens": 325, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 697.7447, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "2192f1c9aa79fc4311635bfd8e50dfbb22146ac7b9306a8419163733b3aae6d6" + }, + { + "id": "json-abstention/0/candidate", + "task": "json-abstention", + "family": "select", + "repeat": 0, + "arm": "candidate", + "body_sha256": "a1f861838ffcecce818b0395a2e34d2431b0c0cc972a6f352e1c1745196afa59", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"retention_days\": null}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 172, + "completion_tokens": 8, + "total_tokens": 180, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 527.7811, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "2fc82870ac179f731f03dcfeeffd19b3b65c18d69c29043cdf6c9d28d1a106e2" + }, + { + "id": "core-api-lookup/1/candidate", + "task": "core-api-lookup", + "family": "core", + "repeat": 1, + "arm": "candidate", + "body_sha256": "e58f4fe9b1e8d6a7858f18ec009ec5ce5935d56cb60f36e4af2537d0ede511b7", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"name\":\"Niklaus Wirth\",\"active\":true,\"tier\":\"team\",\"seats\":25}", + "oracle_pass": false, + "usage": { + "prompt_tokens": 1800, + "completion_tokens": 25, + "total_tokens": 1825, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 696.3879, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "72e4146641dd509dd8c530691a5525382212319d52435afff331c6442a82e544" + }, + { + "id": "core-api-lookup/1/baseline", + "task": "core-api-lookup", + "family": "core", + "repeat": 1, + "arm": "baseline", + "body_sha256": "10f8eabb0bc897da50c268e138a541cf941cf6bf72e26594b6e09c9255e029c5", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"name\":\"Niklaus Wirth\",\"active\":true,\"tier\":\"free\",\"seats\":1}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 2990, + "completion_tokens": 24, + "total_tokens": 3014, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 24042.8133, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "a03e77f1b24210156d8496c56c84b5e51f1f68c09658ac3f407cd511a3a566bd" + }, + { + "id": "core-users-lookup/1/candidate", + "task": "core-users-lookup", + "family": "core", + "repeat": 1, + "arm": "candidate", + "body_sha256": "27e89c0cbc3c7f566d2e03f1764d83c9707c8d05ff9c21ec8402706a346363cf", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"name\":\"John Backus\",\"role\":\"member\",\"active\":true}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 543, + "completion_tokens": 16, + "total_tokens": 559, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 647.4851, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "c5b796762e1a31ead16933358b7e9a6dbb3e4b4ca3340aec499836b539b8b265" + }, + { + "id": "core-users-lookup/1/baseline", + "task": "core-users-lookup", + "family": "core", + "repeat": 1, + "arm": "baseline", + "body_sha256": "d5d8332ab358f178cd7dc4ee9b2204ccc12512436e46352421e686851bd9bbc1", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"name\":\"John Backus\",\"role\":\"member\",\"active\":true}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 791, + "completion_tokens": 16, + "total_tokens": 807, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 708.6065, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "8008f1cd5ee786bfc3fbaf9d146d9524279c7573fa90b029e63716065ac199c4" + }, + { + "id": "core-wide-lookup/1/candidate", + "task": "core-wide-lookup", + "family": "core", + "repeat": 1, + "arm": "candidate", + "body_sha256": "88b19761daa5cd338ba59799544a7267ca9078215a7bc089455d70692596af26", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"service\":\"billing\",\"request_count\":1369,\"error_count\":2,\"p95_ms\":158}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 5567, + "completion_tokens": 29, + "total_tokens": 5596, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 4336, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 4266.7013, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "d26ebaad8c1bcf78a2ec4927e7963188e2e641eee5dceebbe6f50f63c7c43b63" + }, + { + "id": "core-wide-lookup/1/baseline", + "task": "core-wide-lookup", + "family": "core", + "repeat": 1, + "arm": "baseline", + "body_sha256": "fb44422d53464ee199f11d507fca6e01b8d7b6cfba89659df6da236cb0b8de4a", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"service\":\"billing\",\"request_count\":1369,\"error_count\":2,\"p95_ms\":158}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 8053, + "completion_tokens": 29, + "total_tokens": 8082, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 4336, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 1041.8348, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "9c52273865ac6320b2a03d630594946e083250952fd1360979ea892c25a5eba9" + }, + { + "id": "core-schema-lookup/1/candidate", + "task": "core-schema-lookup", + "family": "core", + "repeat": 1, + "arm": "candidate", + "body_sha256": "f895ba46bcd7ac89e162ce339675e6ed752d3007eca911fa07525f533ffcd086", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"name\":\"search_flights\",\"origin\":\"NRT\",\"destination\":\"SIN\",\"required\":[\"origin\",\"destination\",\"departure_date\"]}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 390, + "completion_tokens": 32, + "total_tokens": 422, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 673.8335, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "f4f44d95bd5d6608e9876bea1080b9e880609c371590d6c1a74cecc1d1cceeba" + }, + { + "id": "core-schema-lookup/1/baseline", + "task": "core-schema-lookup", + "family": "core", + "repeat": 1, + "arm": "baseline", + "body_sha256": "9db8779caf97fc08243fbe65874a0b080c172269fca02ee82712a65866d69b87", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"name\":\"search_flights\",\"origin\":\"NRT\",\"destination\":\"SIN\",\"required\":[\"origin\",\"destination\",\"departure_date\"]}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 414, + "completion_tokens": 32, + "total_tokens": 446, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 764.8229, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "c105a02bc5fb020ba4f07d24de310ae7159d3954ff3460fe211a46e0fe7c1c64" + }, + { + "id": "code-branch/1/candidate", + "task": "code-branch", + "family": "select", + "repeat": 1, + "arm": "candidate", + "body_sha256": "956af1c1196dfaf3b6112aa1e860d4fb4b629dbea94a72af89758cc8e17698bb", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"shipping_usd\": 9}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 216, + "completion_tokens": 9, + "total_tokens": 225, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 679.8772, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "78a72be6b28b3dbe1d97d87653a947d91651d3b698dcf2b7daa0bad75058ca88" + }, + { + "id": "code-branch/1/baseline", + "task": "code-branch", + "family": "select", + "repeat": 1, + "arm": "baseline", + "body_sha256": "1fdc22ad29a779925c1c731688e0ba4c0df3015527339af3e3d3c69ac8fba8f1", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"shipping_usd\": 9}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 332, + "completion_tokens": 9, + "total_tokens": 341, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 4666.9517, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "f139fc26a089f3e6a749def2ca19f733b8d7c4a0702797c15f406e5c529cd9d2" + }, + { + "id": "code-dependency/1/candidate", + "task": "code-dependency", + "family": "select", + "repeat": 1, + "arm": "candidate", + "body_sha256": "6e86e36ca9c718670a598b8b616fb8c1969961cbc7ffed5bf67ce9bfd98e72bb", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"total_usd\": 37}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 209, + "completion_tokens": 10, + "total_tokens": 219, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 610.3172, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "277db0338f80c0fa34577cb4f31d38336a076a074597b24ed44657a36895d648" + }, + { + "id": "code-dependency/1/baseline", + "task": "code-dependency", + "family": "select", + "repeat": 1, + "arm": "baseline", + "body_sha256": "aa3afa369b7bb010f1b27953cbc33a42c285ca7a4d890130df0ca47c5adbc358", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"total_usd\": 37}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 328, + "completion_tokens": 10, + "total_tokens": 338, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 684.4183, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "fd0acb4dacd3417f5982d005b57e309cae68b67702790cd04f96d16fcea38d3e" + }, + { + "id": "logs-causal/1/candidate", + "task": "logs-causal", + "family": "select", + "repeat": 1, + "arm": "candidate", + "body_sha256": "9fe221606c6e225006218363d0a2f734bd4624e06c9577517f1a3e3462961941", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"dependency\":\"inventory\",\"status_code\":503}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 243, + "completion_tokens": 13, + "total_tokens": 256, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 650.6396, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "aa3fbe0b12dba5c5e71f35b64fa4afda5b92c80d8a4dd1a068b1769942ace3ae" + }, + { + "id": "logs-causal/1/baseline", + "task": "logs-causal", + "family": "select", + "repeat": 1, + "arm": "baseline", + "body_sha256": "055faa4591653929fb164fbbf39b4efd2dc44f2991e4089ef831d4ff1d0bc445", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"dependency\":\"inventory\",\"status_code\":503}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 435, + "completion_tokens": 13, + "total_tokens": 448, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 573.6413, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "dbb3428a584df9e2d5017ee4ae55c69a10428f86a39197736905c27df54d47b1" + }, + { + "id": "logs-state/1/candidate", + "task": "logs-state", + "family": "select", + "repeat": 1, + "arm": "candidate", + "body_sha256": "5da1023cd0441e276452e28e3d5177d70f0b4b18efcc7df30ca35ce0a007e1d1", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"port\":9443}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 246, + "completion_tokens": 9, + "total_tokens": 255, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 568.4316, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "bbd34ec770b1dda22a3533b38a147b5f0287a39866b06f1fbe97011e18b24590" + }, + { + "id": "logs-state/1/baseline", + "task": "logs-state", + "family": "select", + "repeat": 1, + "arm": "baseline", + "body_sha256": "30b543e2ef56bc0691afa68a97e3cb63bad075f939115b877d2a2274cce3f5af", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"port\":9443}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 388, + "completion_tokens": 9, + "total_tokens": 397, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 668.4017, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "67494d2f7c7d6a43686b37a860098fa728fe3d78f8aa9b712f58e0bd71881b32" + }, + { + "id": "rag-service/1/candidate", + "task": "rag-service", + "family": "select", + "repeat": 1, + "arm": "candidate", + "body_sha256": "2493d80b5174b11cbcdb67102c81dbaad3dad3e7cdc536082cf176db08835f43", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"port\":8717,\"health_path\":\"/ready-ledger\"}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 211, + "completion_tokens": 17, + "total_tokens": 228, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 702.4962, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "0864fb5c158ff5cec423bff1a144b7c61f374d74d87180aa4743bc092c244511" + }, + { + "id": "rag-service/1/baseline", + "task": "rag-service", + "family": "select", + "repeat": 1, + "arm": "baseline", + "body_sha256": "4d721eb1aec96def187744f16e0b50597065ad6975bc588d35f81e6be02c49cd", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"port\":8717,\"health_path\":\"/ready-ledger\"}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 330, + "completion_tokens": 17, + "total_tokens": 347, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 650.882, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "5f19f85c5df378df058572b4ed80caad7a2d1417baef7f05b9e6ae8f26abd30a" + }, + { + "id": "rag-invoice/1/candidate", + "task": "rag-invoice", + "family": "select", + "repeat": 1, + "arm": "candidate", + "body_sha256": "702f4d818e63c2a157b5aa78945347ba7a306cd2f50466eeefe68cddf18abbd9", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"total_usd\": 87}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 193, + "completion_tokens": 10, + "total_tokens": 203, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 1015.9318, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "25ce2ca27b6b00a31ebaa85afd58176ab3ecca6719530dcc93eb4720415cd034" + }, + { + "id": "rag-invoice/1/baseline", + "task": "rag-invoice", + "family": "select", + "repeat": 1, + "arm": "baseline", + "body_sha256": "09978a15bec792f62cc676b8aa7373d72c299ac713b2d41e179acd81509dc806", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"total_usd\":87}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 361, + "completion_tokens": 9, + "total_tokens": 370, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 579.2868, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "42b36a4f53a2a59c78407023cc803b810c661b8fb3069c2d38537a674705d8f7" + }, + { + "id": "conversation-override/1/candidate", + "task": "conversation-override", + "family": "select", + "repeat": 1, + "arm": "candidate", + "body_sha256": "204937bab72147868290a449fbda1d67f828a1c19ed967d143cfd192819472cb", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"region\":\"eu-west-2\"}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 218, + "completion_tokens": 9, + "total_tokens": 227, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 582.4784, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "8b3d7d76844e9c3cbcd5ad9b514967bbebbed7cfa08bfac027eb7aee15506f3e" + }, + { + "id": "conversation-override/1/baseline", + "task": "conversation-override", + "family": "select", + "repeat": 1, + "arm": "baseline", + "body_sha256": "624efa44defb893827cb2ca485e2d6dfe267ed18f13adb44aefe67292b5a297c", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"region\":\"eu-west-2\"}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 362, + "completion_tokens": 9, + "total_tokens": 371, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 589.0127, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "6ba4b111dc0c4e985a075e48a5f4448f10e1e7febf114e682e55eae5390298ea" + }, + { + "id": "json-abstention/1/candidate", + "task": "json-abstention", + "family": "select", + "repeat": 1, + "arm": "candidate", + "body_sha256": "a1f861838ffcecce818b0395a2e34d2431b0c0cc972a6f352e1c1745196afa59", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"retention_days\": null}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 172, + "completion_tokens": 8, + "total_tokens": 180, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 646.9794, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "92c0986baf3a00b53f67635fabee979e56bc3d6f626c33ec99ec390e52616862" + }, + { + "id": "json-abstention/1/baseline", + "task": "json-abstention", + "family": "select", + "repeat": 1, + "arm": "baseline", + "body_sha256": "d078048b4938108f632b1fdda758f2a43434fb59b41258a4fc804368da9c1288", + "status": "completed", + "returned_model": "nvidia/nemotron-3.5-super-vl:free", + "provider": "Nvidia", + "finish_reason": "stop", + "refusal": null, + "output": "{\"retention_days\": null}", + "oracle_pass": true, + "usage": { + "prompt_tokens": 317, + "completion_tokens": 8, + "total_tokens": 325, + "cost": 0, + "is_byok": false, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 0, + "audio_tokens": 0, + "video_tokens": 0 + }, + "cost_details": { + "upstream_inference_cost": 0, + "upstream_inference_prompt_cost": 0, + "upstream_inference_completions_cost": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "image_tokens": 0, + "audio_tokens": 0 + } + }, + "wall_ms": 614.1279, + "cost_coverage_reason": "request_price_unreported", + "request_id_sha256": "28d9f738f3b1992fff5c0fc7da10301672448d916569d3057ed4921a9e7bc27f" + } + ], + "limits": [ + "12 distinct tasks, not 24 independent tasks; same seed and temperature across repeated trials.", + "Four exposed repository JSON fixtures and eight hand-authored synthetic contexts; no representative accuracy/noninferiority claim.", + "Dependencies were manually bundled; no dependency discovery, required-flag protection or provider-role history compaction was tested.", + "Local rendered o200k_base counts and full provider prompt usage are separate accounting scopes.", + "Core cached-token observations retained; free-provider latency/load/cache uncontrolled; no latency superiority claim.", + "Exact model ID/provider frozen, but served model weights/fingerprint unavailable; no immutable checkpoint claim.", + "Raw IDs/account details/credentials omitted; every failed, unexecuted and completed attempt remains represented." + ] +} diff --git a/examples/quickstart.ipynb b/examples/quickstart.ipynb index 4ac7b9b..6f48fdd 100644 --- a/examples/quickstart.ipynb +++ b/examples/quickstart.ipynb @@ -2,12 +2,13 @@ "cells": [ { "cell_type": "markdown", + "id": "66477405", "metadata": {}, "source": [ "# Tokenfold 0.6.0: fit first, inspect the receipt\n", "\n", - "Python >=3.12. Install `tokenfold==0.6.0` in your environment before opening this\n", - "notebook. Use a clean kernel outside the existing ML environment.\n", + "Python >=3.12. Install `tokenfold==0.6.0` before opening this notebook. Use a\n", + "clean kernel outside the existing ML environment.\n", "No cell installs packages, downloads models, sends provider requests or writes stores.\n", "The optional maintained worker must be separately provisioned and approved; see\n", "[SDK migration](../docs/sdk-migration.md). Selection remains unqualified.\n" @@ -15,7 +16,10 @@ }, { "cell_type": "code", + "execution_count": null, + "id": "d34d566d", "metadata": {}, + "outputs": [], "source": [ "import tokenfold as tf\n", "from importlib.metadata import version\n", @@ -39,12 +43,11 @@ " pass\n", "else:\n", " raise AssertionError(\"an estimate must not become exact\")\n" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", + "id": "54b4d6f5", "metadata": {}, "source": [ "## Advanced lowlevel recovery and redaction\n", @@ -59,10 +62,10 @@ "id": "setup", "metadata": { "execution": { - "iopub.execute_input": "2026-08-21T00:06:53.355564Z", - "iopub.status.busy": "2026-08-21T00:06:53.355564Z", - "iopub.status.idle": "2026-08-21T00:06:53.697647Z", - "shell.execute_reply": "2026-08-21T00:06:53.697144Z" + "iopub.execute_input": "2026-11-09T17:29:11.771306Z", + "iopub.status.busy": "2026-11-09T17:29:11.771306Z", + "iopub.status.idle": "2026-11-09T17:29:11.781528Z", + "shell.execute_reply": "2026-11-09T17:29:11.780306Z" } }, "outputs": [], @@ -144,10 +147,10 @@ "id": "compress-json", "metadata": { "execution": { - "iopub.execute_input": "2026-08-21T00:06:53.701653Z", - "iopub.status.busy": "2026-08-21T00:06:53.701653Z", - "iopub.status.idle": "2026-08-21T00:06:54.220012Z", - "shell.execute_reply": "2026-08-21T00:06:54.219007Z" + "iopub.execute_input": "2026-11-09T17:29:11.785528Z", + "iopub.status.busy": "2026-11-09T17:29:11.785528Z", + "iopub.status.idle": "2026-11-09T17:29:11.917121Z", + "shell.execute_reply": "2026-11-09T17:29:11.916117Z" } }, "outputs": [], @@ -189,10 +192,10 @@ "id": "inspect", "metadata": { "execution": { - "iopub.execute_input": "2026-08-21T00:06:54.223012Z", - "iopub.status.busy": "2026-08-21T00:06:54.223012Z", - "iopub.status.idle": "2026-08-21T00:06:54.707569Z", - "shell.execute_reply": "2026-08-21T00:06:54.706564Z" + "iopub.execute_input": "2026-11-09T17:29:11.920427Z", + "iopub.status.busy": "2026-11-09T17:29:11.920427Z", + "iopub.status.idle": "2026-11-09T17:29:11.937090Z", + "shell.execute_reply": "2026-11-09T17:29:11.936087Z" } }, "outputs": [], @@ -239,10 +242,10 @@ "id": "modes", "metadata": { "execution": { - "iopub.execute_input": "2026-08-21T00:06:54.710569Z", - "iopub.status.busy": "2026-08-21T00:06:54.710569Z", - "iopub.status.idle": "2026-08-21T00:06:57.374190Z", - "shell.execute_reply": "2026-08-21T00:06:57.373186Z" + "iopub.execute_input": "2026-11-09T17:29:11.940650Z", + "iopub.status.busy": "2026-11-09T17:29:11.940650Z", + "iopub.status.idle": "2026-11-09T17:29:11.974756Z", + "shell.execute_reply": "2026-11-09T17:29:11.973654Z" } }, "outputs": [], @@ -266,10 +269,10 @@ "id": "policy-and-budget", "metadata": { "execution": { - "iopub.execute_input": "2026-08-21T00:06:57.377349Z", - "iopub.status.busy": "2026-08-21T00:06:57.377349Z", - "iopub.status.idle": "2026-08-21T00:06:58.831453Z", - "shell.execute_reply": "2026-08-21T00:06:58.830449Z" + "iopub.execute_input": "2026-11-09T17:29:11.977755Z", + "iopub.status.busy": "2026-11-09T17:29:11.977755Z", + "iopub.status.idle": "2026-11-09T17:29:11.998866Z", + "shell.execute_reply": "2026-11-09T17:29:11.997863Z" } }, "outputs": [], @@ -311,10 +314,10 @@ "id": "compress-messages", "metadata": { "execution": { - "iopub.execute_input": "2026-08-21T00:06:58.834452Z", - "iopub.status.busy": "2026-08-21T00:06:58.834452Z", - "iopub.status.idle": "2026-08-21T00:06:59.236628Z", - "shell.execute_reply": "2026-08-21T00:06:59.235542Z" + "iopub.execute_input": "2026-11-09T17:29:12.002423Z", + "iopub.status.busy": "2026-11-09T17:29:12.002423Z", + "iopub.status.idle": "2026-11-09T17:29:12.007240Z", + "shell.execute_reply": "2026-11-09T17:29:12.006236Z" } }, "outputs": [], @@ -359,10 +362,10 @@ "id": "provider-helpers", "metadata": { "execution": { - "iopub.execute_input": "2026-08-21T00:06:59.239628Z", - "iopub.status.busy": "2026-08-21T00:06:59.239628Z", - "iopub.status.idle": "2026-08-21T00:07:00.060554Z", - "shell.execute_reply": "2026-08-21T00:07:00.060554Z" + "iopub.execute_input": "2026-11-09T17:29:12.010239Z", + "iopub.status.busy": "2026-11-09T17:29:12.010239Z", + "iopub.status.idle": "2026-11-09T17:29:12.015597Z", + "shell.execute_reply": "2026-11-09T17:29:12.015597Z" } }, "outputs": [], @@ -395,10 +398,10 @@ "id": "protected-floor", "metadata": { "execution": { - "iopub.execute_input": "2026-08-21T00:07:00.064573Z", - "iopub.status.busy": "2026-08-21T00:07:00.064573Z", - "iopub.status.idle": "2026-08-21T00:07:00.469458Z", - "shell.execute_reply": "2026-08-21T00:07:00.469458Z" + "iopub.execute_input": "2026-11-09T17:29:12.021111Z", + "iopub.status.busy": "2026-11-09T17:29:12.020111Z", + "iopub.status.idle": "2026-11-09T17:29:12.024570Z", + "shell.execute_reply": "2026-11-09T17:29:12.023567Z" } }, "outputs": [], @@ -440,10 +443,10 @@ "id": "format-routing", "metadata": { "execution": { - "iopub.execute_input": "2026-08-21T00:07:00.473556Z", - "iopub.status.busy": "2026-08-21T00:07:00.473556Z", - "iopub.status.idle": "2026-08-21T00:07:02.606136Z", - "shell.execute_reply": "2026-08-21T00:07:02.605132Z" + "iopub.execute_input": "2026-11-09T17:29:12.028569Z", + "iopub.status.busy": "2026-11-09T17:29:12.027570Z", + "iopub.status.idle": "2026-11-09T17:29:12.054583Z", + "shell.execute_reply": "2026-11-09T17:29:12.053578Z" } }, "outputs": [], @@ -504,7 +507,14 @@ "cell_type": "code", "execution_count": null, "id": "toon-hard-targets-code", - "metadata": {}, + "metadata": { + "execution": { + "iopub.execute_input": "2026-11-09T17:29:12.057583Z", + "iopub.status.busy": "2026-11-09T17:29:12.057583Z", + "iopub.status.idle": "2026-11-09T17:29:12.071786Z", + "shell.execute_reply": "2026-11-09T17:29:12.070783Z" + } + }, "outputs": [], "source": [ "toon_input = json.dumps({\"users\": [{\"id\": 1}, {\"id\": 2}]}).encode(\"utf-8\")\n", @@ -551,10 +561,10 @@ "id": "lossy-preview", "metadata": { "execution": { - "iopub.execute_input": "2026-08-21T00:07:02.609136Z", - "iopub.status.busy": "2026-08-21T00:07:02.609136Z", - "iopub.status.idle": "2026-08-21T00:07:03.680513Z", - "shell.execute_reply": "2026-08-21T00:07:03.679509Z" + "iopub.execute_input": "2026-11-09T17:29:12.074787Z", + "iopub.status.busy": "2026-11-09T17:29:12.074787Z", + "iopub.status.idle": "2026-11-09T17:29:12.154199Z", + "shell.execute_reply": "2026-11-09T17:29:12.153195Z" } }, "outputs": [], @@ -582,10 +592,10 @@ "id": "lossy-compress", "metadata": { "execution": { - "iopub.execute_input": "2026-08-21T00:07:03.683513Z", - "iopub.status.busy": "2026-08-21T00:07:03.683513Z", - "iopub.status.idle": "2026-08-21T00:07:05.653050Z", - "shell.execute_reply": "2026-08-21T00:07:05.652047Z" + "iopub.execute_input": "2026-11-09T17:29:12.157198Z", + "iopub.status.busy": "2026-11-09T17:29:12.157198Z", + "iopub.status.idle": "2026-11-09T17:29:12.159668Z", + "shell.execute_reply": "2026-11-09T17:29:12.159668Z" } }, "outputs": [], @@ -599,10 +609,10 @@ "id": "lossy-retrieve", "metadata": { "execution": { - "iopub.execute_input": "2026-08-21T00:07:05.656051Z", - "iopub.status.busy": "2026-08-21T00:07:05.656051Z", - "iopub.status.idle": "2026-08-21T00:07:06.248082Z", - "shell.execute_reply": "2026-08-21T00:07:06.247077Z" + "iopub.execute_input": "2026-11-09T17:29:12.163673Z", + "iopub.status.busy": "2026-11-09T17:29:12.163673Z", + "iopub.status.idle": "2026-11-09T17:29:12.167720Z", + "shell.execute_reply": "2026-11-09T17:29:12.166716Z" } }, "outputs": [], @@ -664,13 +674,11 @@ "| Retrieval and realized-savings operations | CLI: `retrieve`, `stats`, `gain`, `session`, `output-savings` |\n", "| Agent/editor tools | `tokenfold mcp serve` for MCP and `doctor` for diagnostics; `init`/`uninit` exist, but no host integration is supported yet |\n", "| Transparent provider traffic | `tokenfold-proxy` |\n", - "| Advanced lowlevel Python tour | [`quickstart.py`](quickstart.py) |\n", - "| Advanced Node CLI tour | [`quickstart.mjs`](quickstart.mjs) |\n", + "| Plain-script version of this tour | [`quickstart.py`](quickstart.py) |\n", + "| Node application | [`quickstart.mjs`](quickstart.mjs) |\n", "| Native embedding | `tokenfold-core` and [`quickstart.rs`](quickstart.rs) |\n", "| Deeper pruning behavior | [`lossy_pruning.py`](lossy_pruning.py) |\n", "| Independent recorded-outcome comparison | Section 11, [`contextdoctor_report.py`](contextdoctor_report.py), and the [evaluation guide](../eval/README.md) |\n", - "| Native fitting | [`sdk_fit.py`](sdk_fit.py) and [`sdk_fit.mjs`](sdk_fit.mjs) |\n", - "| Query-aware learned ranking | Optional maintained Select runtime below and the [Tokenfold Select v3 model card](https://huggingface.co/snchimata/tokenfold-select-v3) |\n", "\n", "### Which path is yours?\n", "\n", @@ -687,6 +695,7 @@ }, { "cell_type": "markdown", + "id": "66067dd1", "metadata": {}, "source": [ "## Optional maintained Select runtime\n", @@ -700,7 +709,10 @@ }, { "cell_type": "code", + "execution_count": null, + "id": "f9edc984", "metadata": {}, + "outputs": [], "source": [ "import os\n", "from pathlib import Path\n", @@ -728,9 +740,7 @@ " selected.require_met()\n", " # Independent fixture oracle, not a general semantic-quality claim.\n", " assert \"8443\" in selected.payload, \"payments evidence was lost\"\n" - ], - "execution_count": null, - "outputs": [] + ] } ], "metadata": { @@ -749,7 +759,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.14.6" + "version": "3.12.14" } }, "nbformat": 4, diff --git a/python-tests/test_notebook.py b/python-tests/test_notebook.py index 9fe1ca6..59636b8 100644 --- a/python-tests/test_notebook.py +++ b/python-tests/test_notebook.py @@ -22,7 +22,12 @@ def select_cell(): def test_notebook_sources_compile_and_select_requires_explicit_approval(): - for i, cell in enumerate(cells()): + notebook_cells = cells() + intro = "".join(notebook_cells[0]["source"]) + assert "tokenfold==0.6.0" in intro + assert "unreleased" not in intro.lower() + assert "locally built wheel" not in intro.lower() + for i, cell in enumerate(notebook_cells): if cell["cell_type"] == "code": ast.parse("".join(cell["source"]), filename=f"notebook-cell-{i}") assert cell["outputs"] == [] diff --git a/python-tests/test_readme_claims.py b/python-tests/test_readme_claims.py index 1667266..28a1b39 100644 --- a/python-tests/test_readme_claims.py +++ b/python-tests/test_readme_claims.py @@ -105,8 +105,16 @@ def test_every_headline_metric_has_versioned_provenance(): manifest = json.loads(METRICS.read_text(encoding="utf-8")) readme = README.read_text(encoding="utf-8") compression = manifest["compression"] - for name in ("search_rag", "repetitive_json", "api_responses", "tool_schemas"): + for name in ("search_rag",): assert f"{compression[name]['reduction_pct']:.1f}% fewer tokens" in readme + tool_schema_pct = compression["tool_schemas"]["reduction_pct"] + api_response_pct = compression["api_responses"]["reduction_pct"] + assert ( + f"**{tool_schema_pct:.1f}-{api_response_pct:.1f}% on two fixtures, lossless**" + in readme + ) + historical = compression["repetitive_json"]["reduction_pct"] + assert f"historical **{historical:.1f}%** result" in readme for budget, values in manifest["select"]["budgets"].items(): ratio = str(int(float(budget) * 100)) row = re.compile( diff --git a/python-tests/test_readme_layout.py b/python-tests/test_readme_layout.py index f778ad4..cc34dba 100644 --- a/python-tests/test_readme_layout.py +++ b/python-tests/test_readme_layout.py @@ -19,7 +19,7 @@ def test_common_path_and_visible_limits(self): self.assertTrue(self.text.isascii()) self.assertEqual(self.text.count('Exact decode proves data recovery'), 1) self.assertIn('Up to 96.3% fewer tokens (showcase)', self.text) - self.assertIn('45.6-67.6% on three fixtures, lossless', self.text) + self.assertIn('45.6-63.9% on two fixtures, lossless', self.text) self.assertEqual(self.text.count('```mermaid'), 2) self.assertIn('flowchart LR\n accTitle: Lossless first', self.text) self.assertLess(self.text.index('![Terminal demo:'), self.text.index('#### Why Tokenfold')) @@ -37,10 +37,84 @@ def test_common_path_and_visible_limits(self): self.assertEqual(depth, 0) def test_fit_onboarding_and_migration_are_visible(self): - for text in ('0.6.0', 'Python >=3.12', 'from tokenfold import fit', 'require_met()', - 'docs/sdk-migration.md', 'pip install tokenfold==0.6.0', - 'npm install tokenfold@0.6.0', 'tokenfold-select-runtime'): + for text in ('tokenfold==0.6.0', 'Python >=3.12', 'from tokenfold import Data, fit', 'require_met()', + 'docs/sdk-migration.md', 'tokenfold-select-runtime'): self.assertIn(text, self.text) + visible = re.sub(r'\]\([^)]+\)', ']', self.text) + self.assertNotIn('The 0.6 SDK', visible) + self.assertNotIn('Measurement basis', self.text) + intro = self.text.split('## Table of contents', 1)[0] + self.assertIn('Select what matters. Fit more context.', intro) + self.assertIn('Tokenfold Select brings query-aware relevance ranking', intro) + self.assertIn('Select for relevance. Core for structural efficiency.', intro) + self.assertLess(intro.index('Tokenfold Select'), intro.index('Core')) + self.assertNotIn('Start model-free with Core.', intro) + + def test_python_first_release_tracks_are_separate(self): + published = self.text.split('### Published Python SDK\n', 1)[1] + published, fitting = published.split('### Python fit workflow\n', 1) + self.assertIn('python -m pip install "tokenfold==0.6.0"', published) + self.assertIn('from tokenfold import Data, fit', fitting) + self.assertIn('from tokenfold.lowlevel import InputFormat, compress, decode', self.text) + self.assertNotIn('Path(', published) + self.assertIn('Python >=3.12', published) + self.assertIn('representation="folded"', fitting) + self.assertLess(self.text.index('### Published Python SDK'), self.text.index('```js')) + self.assertLess(self.text.index('## Tokenfold Select'), self.text.index('## Benchmark highlights')) + self.assertLess(self.text.index('## Benchmark highlights'), self.text.index('## Other integrations')) + self.assertNotIn('google.com/search', self.text) + self.assertNotIn('unreleased', self.text.lower()) + self.assertNotIn('checkout preview', self.text.lower()) + + def test_node_sdk_quickstart_uses_the_published_in_process_package(self): + self.assertIn('npm install tokenfold@0.6.0', self.text) + self.assertIn('Core fitting runs in-process', self.text) + self.assertNotIn('native fitting uses the SDK build', self.text) + + def test_lowlevel_python_example_preserves_source_value(self): + import tokenfold.lowlevel as tokenfold + + if not hasattr(tokenfold, 'compress'): + self.skipTest('installed package does not expose tokenfold.lowlevel') + code = next(block for block in re.findall(r'```python\n(.*?)```', self.text, re.S) + if 'from tokenfold.lowlevel import InputFormat, compress, decode' in block) + namespace = {} + with redirect_stdout(StringIO()) as output: + exec(compile(code, 'README.md:lowlevel-python', 'exec'), namespace) + result = namespace['result'] + self.assertIsInstance(result.payload, bytes) + self.assertGreater(result.report.saved_tokens, 0) + self.assertEqual(json.loads(tokenfold.decode(result.payload)), namespace['records']) + self.assertIn(f'{result.report.original_tokens} -> {result.report.compressed_tokens} tokens', output.getvalue()) + + def test_python_fit_preview_meets_budget(self): + code = next(block for block in re.findall(r'```python\n(.*?)```', self.text, re.S) + if 'from tokenfold import Data, fit' in block) + namespace = {} + with redirect_stdout(StringIO()): + exec(compile(code, 'README.md:fit-preview', 'exec'), namespace) + result = namespace['result'] + self.assertEqual(result.budget_status, 'met') + self.assertEqual(result.receipt.accounting.final_basis, 'exact') + self.assertLess(result.final_tokens, result.original_tokens) + self.assertLessEqual(result.final_tokens, result.target_tokens) + from tokenfold.lowlevel import decode + self.assertEqual(json.loads(decode(namespace['payload_text'])), namespace['records']) + + def test_select_example_requires_selection_to_fit(self): + from tokenfold import fit + + code = next(block for block in re.findall(r'```python\n(.*?)```', self.text, re.S) + if 'from tokenfold import Context, Group, SelectRuntime, Tokenfold' in block) + self.assertIn('experimental=True', code) + self.assertIn('allow_selection=True', code) + self.assertNotIn('from_config', code) + namespace = {} + exec(compile(code.split('with Tokenfold(', 1)[0], 'README.md:select-input', 'exec'), namespace) + baseline = fit(namespace['context'], target_tokens=96) + self.assertEqual(baseline.budget_status, 'unmet') + self.assertGreater(baseline.final_tokens, baseline.target_tokens) + self.assertFalse(baseline.selection_applied) def test_local_links_resolve(self): targets = re.findall(r'\]\(([^)]+)\)|src="([^"]+)"', self.text) @@ -106,7 +180,9 @@ def test_select_v3_claims_match_versioned_provenance(self): for row in metrics['latency_rows']: change = 100 * (row['v3_ms'] / row['kompress_ms'] - 1) self.assertIn(f"| {row['workload']} | {row['v3_ms']:.3f} | {row['kompress_ms']:.3f} | " - f"{change:.1f}% | {row['basis']} |", visible) + f"{change:.1f}% |", visible) + self.assertIn('one historical request per model', visible) + self.assertIn('median of five', visible) old = metrics['historical_multiturn_ms'] self.assertIn(f"{old['v3']:.3f} ms vs {old['kompress']:.3f} ms", section) self.assertEqual((metrics['historical_refusals'], metrics['historical_cells']), (6, 10)) diff --git a/python-tests/test_sdk_quality_evidence.py b/python-tests/test_sdk_quality_evidence.py new file mode 100644 index 0000000..388b562 --- /dev/null +++ b/python-tests/test_sdk_quality_evidence.py @@ -0,0 +1,141 @@ +"""Reconcile frozen answer-quality evidence offline; never invoke a scorer/provider.""" +from decimal import Decimal +import hashlib +import json +from pathlib import Path +import statistics + +import pytest + +ROOT = Path(__file__).resolve().parents[1] +METRICS = ROOT / "tests/fixtures/readme_metrics.json" +pytestmark = pytest.mark.skipif(not METRICS.exists(), reason="repository evidence absent") + + +def canonical(value): + return json.dumps(value, sort_keys=True, ensure_ascii=True, separators=(",", ":"), allow_nan=False) + + +def digest(value): + return hashlib.sha256(canonical(value).encode()).hexdigest() + + +def evidence(): + pins = json.loads(METRICS.read_text())["sdk_0_6_quality"] + return pins, json.loads((ROOT / pins["source"]).read_text()), json.loads((ROOT / pins["corpus"]).read_text()) + + +def correct(row, oracle): + def unique(pairs): + result = dict(pairs) + if len(result) != len(pairs): + raise ValueError("duplicate key") + return result + try: + actual = json.loads(row["output"], object_pairs_hook=unique) + return (row["finish_reason"] == "stop" and not row["refusal"] and type(actual) is dict + and canonical(actual) == canonical(oracle)) + except (ValueError, TypeError): + return False + + +def test_quality_pins_and_visible_readme_claims(): + pins, report, corpus = evidence() + assert digest(report) == pins["source_canonical_json_sha256"] + assert digest(corpus) == pins["corpus_canonical_json_sha256"] == report["corpus_canonical_json_sha256"] + assert report["model"] == pins["model"] == corpus["protocol"]["request_template"]["model"] + assert not pins["qualification"] and not report["qualification"] + text = (ROOT / "README.md").read_text() + assert "Exact recovery did not guarantee correct model answers" in text + assert "45.6-63.9% on two fixtures, lossless" in text + assert "without a versioned source" in text + assert "| Explicit folded JSON vs. compact JSON | 4 x 2 | 8/8 | **6/8** | 24,496 -> 16,600 (**32.2% fewer**) |" in text + assert "| Select at a 50% rendered-text budget vs. full context | 8 x 2 | 16/16 | 16/16 | 5,706 -> 3,416 (**40.1% fewer**) |" in text + + +def test_every_quality_reply_is_independently_regraded_and_bound_to_its_request(): + _, report, corpus = evidence() + tasks = {t["id"]: t for t in corpus["tasks"]} + assert len(tasks) == 12 and len(report["calls"]) == 48 + ids = set() + for row in report["calls"]: + task = tasks[row["task"]] + key = row["task"], row["repeat"], row["arm"] + assert key not in ids + ids.add(key) + assert row["oracle_pass"] == correct(row, task["oracle"]) + body = {**corpus["protocol"]["request_template"], "messages": task["arms"][row["arm"]]["messages"]} + assert hashlib.sha256(json.dumps(body, sort_keys=True).encode()).hexdigest() == row["body_sha256"] + assert row["returned_model"] == report["model"] and row["provider"] == "Nvidia" + usage = row["usage"] + assert all(type(usage[k]) is int and usage[k] >= 0 for k in ("prompt_tokens", "completion_tokens", "total_tokens")) + assert usage["prompt_tokens"] + usage["completion_tokens"] == usage["total_tokens"] + assert usage["completion_tokens"] <= body["max_tokens"] + assert usage["is_byok"] is False and Decimal(str(usage["cost"])) == 0 + assert ids == {(id, repeat, arm) for id in tasks for repeat in range(2) for arm in ("baseline", "candidate")} + base = {"output": '{"v":1}', "finish_reason": "stop", "refusal": None} + assert correct(base, {"v": 1}) + for output in ('{"v":true}', '{"v":1.0}', '{"v":0,"v":1}', '{"v":NaN}', '```json\n{"v":1}\n```'): + assert not correct({**base, "output": output}, {"v": 1}) + assert not correct({**base, "refusal": "refused"}, {"v": 1}) + + +def test_quality_totals_retain_losses_cached_tokens_and_failed_probe(): + pins, report, _ = evidence() + for family in ("core", "select"): + summary = report["families"][family] + for arm in ("baseline", "candidate"): + rows = [r for r in report["calls"] if r["family"] == family and r["arm"] == arm] + totals = summary["arms"][arm] + assert totals["trials"] == len(rows) == pins[family]["trials"] + assert totals["oracle_passes"] == sum(r["oracle_pass"] for r in rows) == pins[family][f"{arm}_correct"] + for name in ("prompt", "completion"): + assert totals[f"provider_{name}_tokens"] == sum(r["usage"][f"{name}_tokens"] for r in rows) + assert totals["provider_cached_tokens"] == sum(r["usage"]["prompt_tokens_details"]["cached_tokens"] for r in rows) + assert totals["provider_call_wall_median_ms"] == round(statistics.median(r["wall_ms"] for r in rows), 6) + before, after = [summary["arms"][a]["provider_prompt_tokens"] for a in ("baseline", "candidate")] + assert round(100 * (before - after) / before, 6) == summary["provider_prompt_reduction_pct"] + assert round(summary["provider_prompt_reduction_pct"], 1) == pins[family]["provider_prompt_reduction_pct"] + losses = [r for r in report["calls"] if not r["oracle_pass"]] + assert [(r["task"], r["arm"]) for r in losses] == [("core-api-lookup", "candidate")] * 2 + assert report["families"]["core"]["paired"]["baseline_only"] == len(losses) + first = report["prior_scopes"][0] + assert (first["probe_http_status"], first["quality_calls_attempted"], first["quality_calls_unexecuted"]) == (429, 0, 48) + assert first["probe_usage"] is None and first["probe_cost_usd"] is None + accounting = report["all_attempt_accounting"] + assert accounting["actual_generation_attempts"] == first["probe_attempts"] + report["nvidia_probe"]["attempts"] + len(report["calls"]) == 50 + assert accounting["unknown_usage_attempts"] == accounting["unknown_reported_cost_attempts"] == 1 + assert accounting["complete_campaign_spend_usd"] is None + for name in ("prompt", "completion"): + assert accounting[f"nvidia_observed_{name}_tokens"] == sum(r["usage"][f"{name}_tokens"] for r in report["calls"]) + report["nvidia_probe"]["usage"][f"{name}_tokens"] + + +def test_frozen_sdk_payloads_recount_recover_and_preserve_group_boundaries(): + from tokenfold import EncodingCounter, RenderedRequest + from tokenfold.lowlevel import decode + _, _, corpus = evidence() + counter = EncodingCounter() + positions = [] + for task in corpus["tasks"]: + for arm in task["arms"].values(): + payload = arm["messages"][0]["content"] + count = counter.count(RenderedRequest("text", json.dumps(payload), "rendered_text")) + assert count.basis == "exact" and count.tokens == arm["rendered_tokens"] + assert arm["accounting"]["scope"] == "rendered_text" + baseline, candidate = task["arms"]["baseline"], task["arms"]["candidate"] + if task["family"] == "core": + source = ROOT / task["source"] + assert hashlib.sha256(source.read_bytes()).hexdigest() == task["source_sha256"] + assert json.loads(decode(candidate["messages"][0]["content"])) == json.loads(source.read_text()) + else: + groups = task["groups"] + kept = candidate["groups_kept"] + assert kept == [g["id"] for g in groups if g["id"] in kept] + assert candidate["groups_dropped"] == [g["id"] for g in groups if g["id"] not in kept] + assert candidate["messages"][0]["content"] == task["query"] + "".join("\n\n" + g["content"] for g in groups if g["id"] in kept) + assert candidate["scorer_succeeded"] and candidate["selection_applied"] + assert candidate["budget_status"] == "met" + assert candidate["target_tokens"] == baseline["rendered_tokens"] // 2 + assert candidate["rendered_tokens"] <= candidate["target_tokens"] + positions.append(task["supporting_group_position"]) + assert positions == [0, 1, 2, 3, 4, 0, 1, 2] diff --git a/scripts/render-demo.py b/scripts/render-demo.py new file mode 100644 index 0000000..a8b741e --- /dev/null +++ b/scripts/render-demo.py @@ -0,0 +1,246 @@ +"""Render the README GIF. Requires Pillow, a local CLI and the installed SDK. + +python scripts/render-demo.py --sdk-python /path/to/sdk-env/python +Select is a labeled illustration; only Core and base SDK checks execute. +""" +import argparse +from functools import lru_cache +import hashlib +import json +import os +from pathlib import Path +import shutil +import subprocess +import sys +import tempfile + +from PIL import Image, ImageDraw, ImageFont + + +ROOT = Path(__file__).resolve().parents[1] +OUT = ROOT / "docs/assets/tokenfold-demo.gif" +W, H, SCALE, FPS = 1280, 720, 2, 10 +BG, PANEL, LINE = "#0b1118", "#121e29", "#293b4b" +WHITE, MUTED, MINT, BLUE = "#f3f7fb", "#afc1d0", "#b8f58b", "#79d8eb" +LABELS = ("FIT", "CORE", "TOKENFOLD SELECT") +DURATIONS = (5, 3.5, 4.5) +FIT_CODE = '''from tokenfold import Data, fit + +result = fit( + Data(records, representation="folded"), + target_tokens=1500, +) +payload = result.require_met()''' +with Image.open(ROOT / "docs/assets/tokenfold-logo.png") as source: + LOGO = source.convert("RGBA") +LOGO = LOGO.crop(LOGO.getbbox()) +LOGO.thumbnail((220 * SCALE, 82 * SCALE), Image.Resampling.LANCZOS) + + +def evidence(cli, sdk_python): + """Fail before rendering if the fixture, round trip or shown SDK call drifts.""" + fixture = ROOT / "examples/api_response.json" + pins = json.loads((ROOT / "tests/fixtures/readme_metrics.json").read_text()) + assert hashlib.sha256(fixture.read_bytes()).hexdigest() == pins["compression"]["api_responses"]["sha256"] + with tempfile.TemporaryDirectory(prefix="tokenfold-demo-") as directory: + packed, restored = (Path(directory) / name for name in ("packed.json", "restored.json")) + run = subprocess.run([cli, "compress", str(fixture), "-o", str(packed), + "--receipt-format", "json"], check=True, capture_output=True, text=True) + report = json.loads(run.stderr) + subprocess.run([cli, "decode", str(packed), "-o", str(restored)], + check=True, capture_output=True) + assert json.loads(restored.read_bytes()) == json.loads(fixture.read_bytes()) + assert report["estimator"]["is_exact"] and report["estimator"]["model"] == "o200k_base" + assert report["preset"] == "balanced" and report["pruning"] is None + assert (report["original_tokens"], report["compressed_tokens"]) == (3812, 1376) + assert round(report["savings_pct"], 1) == pins["compression"]["api_responses"]["reduction_pct"] + code = ('import json\nfrom pathlib import Path\nfrom tokenfold.lowlevel import decode\n' + 'records = json.loads(Path("examples/api_response.json").read_bytes())\n' + + FIT_CODE + '\nassert json.loads(decode(payload.encode("utf-8"))) == records\n' + 'assert len(records["results"]) == 30\n' + 'baseline = fit(Data(records), target_tokens=result.target_tokens)\n' + 'assert baseline.budget_status == "unmet"\n' + 'assert baseline.final_tokens == result.original_tokens\n' + 'print(result.receipt.to_json())') + receipt = json.loads(subprocess.run([sdk_python, "-c", code], cwd=ROOT, check=True, + capture_output=True, text=True).stdout) + assert receipt["budget_status"] == "met" and receipt["target_tokens"] == 1500 + assert (receipt["original_tokens"], receipt["final_tokens"]) == (2366, 1376) + assert receipt["original_tokens"] > receipt["target_tokens"] >= receipt["final_tokens"] + assert receipt["accounting"]["original_basis"] == receipt["accounting"]["final_basis"] == "exact" + assert receipt["accounting"]["scope"] == "rendered_text" and not receipt["scorer_attempted"] + return report, receipt + + +@lru_cache(maxsize=None) +def font(size, style="regular"): + names = { + "regular": ("segoeui.ttf", "DejaVuSans.ttf", "Arial.ttf"), + "bold": ("segoeuib.ttf", "DejaVuSans-Bold.ttf", "Arial Bold.ttf"), + "mono": ("consola.ttf", "DejaVuSansMono.ttf", "Andale Mono.ttf"), + }[style] + directories = (Path(os.environ.get("WINDIR", "C:/Windows")) / "Fonts", + Path("/usr/share/fonts/truetype/dejavu"), Path("/System/Library/Fonts/Supplemental")) + for directory in directories: + for name in names: + if (directory / name).exists(): + return ImageFont.truetype(str(directory / name), size * SCALE) + raise RuntimeError("Install Segoe UI/Consolas, DejaVu Sans/Mono or Arial/Andale Mono") + + +def text(draw, x, y, value, size=24, color=WHITE, style="regular"): + face = font(size, style) + box = draw.textbbox((x * SCALE, y * SCALE), value, font=face) + assert box[2] <= W * SCALE and box[3] <= H * SCALE, value + draw.text((x * SCALE, y * SCALE), value, fill=color, font=face) + + +def rect(draw, box, fill=PANEL, outline=LINE, radius=16): + draw.rounded_rectangle(tuple(round(v * SCALE) for v in box), radius=radius * SCALE, + fill=fill, outline=outline, width=SCALE) + + +def line(draw, points, color=LINE, width=2): + draw.line([(round(x * SCALE), round(y * SCALE)) for x, y in points], fill=color, width=width * SCALE) + + +def ease(t): + t = max(0, min(t, 1)) + return t * t * (3 - 2 * t) + + +def base(scene): + im = Image.new("RGB", (W * SCALE, H * SCALE), BG) + d = ImageDraw.Draw(im) + # A quiet drafting grid, rather than a stock terminal window. + for x in range(640, W, 48): + line(d, [(x, 96), (x, 620)], "#101a24", 1) + for y in range(96, 620, 48): + line(d, [(640, y), (W - 64, y)], "#101a24", 1) + im.paste(LOGO, (64 * SCALE, 8 * SCALE), LOGO) + label = "PYTHON + TYPESCRIPT" + text(d, 1216 - font(20).getlength(label) / SCALE, 36, label, 20, MUTED) + line(d, [(64, 91), (1216, 91)]) + for i, label in enumerate(LABELS): + x = 64 + i * 392 + line(d, [(x, 655), (x + 368, 655)], MINT if i == scene else LINE, 3) + text(d, x, 672, f"0{i + 1} {label}", 18, MINT if i == scene else MUTED, "bold") + return im, d + + +def render(scene, t, report, receipt): + im, d = base(scene) + if scene == 2: + text(d, 64, 118, "03 / QUERY-AWARE SELECTION", 20, BLUE, "bold") + text(d, 60, 152, "Tokenfold Select", 64, WHITE, "bold") + text(d, 64, 236, "Why did checkout fail?", 26, MINT) + text(d, 64, 272, "CONTEXT", 18, MUTED, "bold") + text(d, 735, 272, "SELECTED", 18, MINT, "bold") + groups = ("Required rules", "Checkout: HTTP 504", + "Inventory timeout", "Unrelated changelog") + for i, title in enumerate(groups): + y = 310 + i * 68 + chosen = i < 3 and t >= 0.6 + i * 0.75 + dim = i == 3 and t >= 3 + rect(d, (64, y, 615, y + 56), fill=BG if dim else PANEL, + outline=MINT if chosen else LINE, radius=10) + text(d, 80, y + 15, title, 23, MUTED if dim else WHITE) + if chosen: + text(d, 570, y + 15, "+", 24, MINT, "bold") + progress = ease((t - 0.6 - i * 0.75) / 0.6) + line(d, [(631, y + 28), (631 + 88 * progress, y + 28)], MINT) + rect(d, (735, y, 1216, y + 56), outline=MINT, radius=10) + text(d, 751, y + 15, title, 23, MINT) + if dim: + text(d, 735, y + 15, "Omitted", 22, MUTED) + text(d, 64, 579, "Core first · Over budget? · allow_selection=True", 20, BLUE) + text(d, 64, 612, "Approved runtime · Experimental / unqualified · Illustration (not live)", 18, MUTED) + + elif scene == 1: + text(d, 64, 118, "02 / CORE FIRST / STRUCTURAL EFFICIENCY", 20, BLUE, "bold") + text(d, 60, 154, "Less repetition. Same JSON value.", 51, WHITE, "bold") + text(d, 64, 227, "Formatted JSON fixture / balanced preset / exact o200k_base counts", 23, MUTED) + before, after = report["original_tokens"], report["compressed_tokens"] + rect(d, (64, 292, 813, 553)) + text(d, 90, 314, "FORMATTED JSON", 19, MUTED, "bold") + text(d, 615, 303, f"{before:,}", 38, WHITE, "bold") + rect(d, (90, 366, 779, 390), BLUE, BLUE, 7) + text(d, 90, 422, "CORE LOSSLESS", 19, MINT, "bold") + text(d, 615, 410, f"{after:,}", 38, MINT, "bold") + rect(d, (90, 475, 90 + 689 * after / before, 499), MINT, MINT, 7) + text(d, 859, 299, f"{report['savings_pct']:.1f}%", 76, MINT, "bold") + text(d, 863, 397, "fewer tokens", 29, WHITE, "bold") + text(d, 863, 450, "Fixture result,", 22, MUTED) + text(d, 863, 482, "not a universal promise.", 22, MUTED) + text(d, 64, 573, "VERIFIED / Exact JSON-value recovery after decode", 23, MINT, "bold") + text(d, 64, 608, "examples/api_response.json · Recovery is not proof of model-answer equivalence.", 20, MUTED) + + elif scene == 0: + before, after, target = receipt["original_tokens"], receipt["final_tokens"], receipt["target_tokens"] + text(d, 64, 118, "01 / FIT MORE WITHOUT DROPPING RECORDS", 20, BLUE, "bold") + text(d, 60, 154, "Fit the budget. Keep every record.", 53, WHITE, "bold") + text(d, 64, 229, f"Compact JSON: {before:,} tokens / Input budget: {target:,} tokens", 24, MUTED) + rect(d, (64, 294, 791, 574)) + text(d, 88, 311, "PYTHON / records = bundled API fixture", 18, MUTED, "mono") + line(d, [(88, 347), (767, 347)]) + for i, row in enumerate(FIT_CODE.splitlines()): + color = MINT if "require_met" in row else BLUE if "from " in row else WHITE + text(d, 88, 364 + i * 27, row, 23, color, "mono") + rect(d, (819, 294, 1216, 574), outline=MINT) + text(d, 845, 311, "FITS THE INPUT BUDGET", 18, MUTED, "bold") + text(d, 841, 345, f"{after:,}", 58, MINT, "bold") + text(d, 845, 431, f"of {target:,} tokens", 29, WHITE, "bold") + text(d, 845, 481, f"{100 * (1 - after / before):.1f}% fewer tokens", 24, MINT) + text(d, 845, 530, "30 records. None dropped.", 22, MINT) + text(d, 64, 608, "Explicit folded representation / Verified JSON-value recovery / Local o200k_base counts", 20, MUTED) + return im.resize((W, H), Image.Resampling.LANCZOS) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--sdk-python", default=sys.executable, help="Python with the current SDK installed") + local_cli = ROOT / "target/release" / ("tokenfold.exe" if os.name == "nt" else "tokenfold") + parser.add_argument("--cli", default=str(local_cli) if local_cli.exists() else shutil.which("tokenfold")) + args = parser.parse_args() + if not args.cli: + parser.error("Build the CLI first: cargo build --release -p tokenfold-cli --locked") + report, receipt = evidence(args.cli, args.sdk_python) + previews = [render(i, duration - 0.5, report, receipt) for i, duration in enumerate(DURATIONS)] + sheet = Image.new("RGB", (W, H * len(previews)), BG) + for i, preview in enumerate(previews): + sheet.paste(preview, (0, i * H)) + # Reserve brand colors so the small gradient logo survives GIF quantization. + brand = Image.new("RGB", LOGO.size, BG) + brand.paste(LOGO, (0, 0), LOGO) + palette = sheet.quantize(colors=128, method=Image.Quantize.MEDIANCUT) + palette.putpalette(palette.getpalette()[:384] + brand.quantize(colors=128).getpalette()[:384]) + frames = [] + for scene, duration in enumerate(DURATIONS): + for tick in range(round(duration * FPS)): + frame = render(scene, tick / FPS, report, receipt) + frames.append(frame.quantize(palette=palette, dither=Image.Dither.NONE)) + OUT.parent.mkdir(exist_ok=True) + frames[0].save(OUT, save_all=True, append_images=frames[1:], duration=1000 // FPS, + loop=0, optimize=True, disposal=1) + with Image.open(OUT) as gif: + assert gif.size == (W, H) and gif.info["loop"] == 0 + elapsed = 0 + for i in range(gif.n_frames): + gif.seek(i) + expected = frames[elapsed // (1000 // FPS)].convert("RGB") + assert gif.convert("RGB").tobytes() == expected.tobytes(), f"GIF artifacts in frame {i}" + elapsed += gif.info["duration"] + assert elapsed == len(frames) * (1000 // FPS) + assert OUT.stat().st_size < 4_000_000, "Keep the README GIF below 4 MB" + scratch = ROOT / "tmp/tokenfold-demo" + scratch.mkdir(parents=True, exist_ok=True) + sheet.resize((960, 540 * len(previews)), Image.Resampling.LANCZOS).save(scratch / "storyboard.png") + for i, preview in enumerate(previews): + preview.save(scratch / f"scene-{i + 1}.png") + print(f"Core verified: {report['original_tokens']} -> {report['compressed_tokens']}; exact JSON recovery") + print(f"SDK verified: {receipt['original_tokens']} -> {receipt['final_tokens']} / {receipt['target_tokens']}; exact rendered text; met") + print(f"{OUT}: {OUT.stat().st_size:,} bytes; {len(frames) / FPS:.1f}s; Select illustration only") + + +if __name__ == "__main__": + main() diff --git a/scripts/test_render_demo.py b/scripts/test_render_demo.py new file mode 100644 index 0000000..26bae34 --- /dev/null +++ b/scripts/test_render_demo.py @@ -0,0 +1,89 @@ +"""Offline layout regressions; no SDK, model or network calls.""" +import importlib.util +from pathlib import Path +import unittest +from unittest.mock import patch + +from PIL import Image, ImageChops + + +SPEC = importlib.util.spec_from_file_location("render_demo", Path(__file__).with_name("render-demo.py")) +demo = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(demo) +REPORT = {"original_tokens": 3812, "compressed_tokens": 1376, "savings_pct": 63.90346274921301} +RECEIPT = {"budget_status": "met", "original_tokens": 2366, "final_tokens": 1376, "target_tokens": 1500} + + +class DemoLayout(unittest.TestCase): + def test_every_scene_uses_the_repository_logo(self): + with Image.open(demo.ROOT / "docs/assets/tokenfold-logo.png") as source: + logo = source.convert("RGBA") + logo = logo.crop(logo.getbbox()) + logo.thumbnail((220 * demo.SCALE, 82 * demo.SCALE), Image.Resampling.LANCZOS) + expected = Image.new("RGB", logo.size, demo.BG) + expected.paste(logo, (0, 0), logo) + x, y = 64 * demo.SCALE, 8 * demo.SCALE + for scene in range(len(demo.LABELS)): + frame, _ = demo.base(scene) + with self.subTest(scene=scene): + self.assertEqual(frame.crop((x, y, x + logo.width, y + logo.height)).tobytes(), + expected.tobytes()) + + def test_sdk_scene_demonstrates_savings_not_just_a_cap(self): + with patch.object(demo, "text", wraps=demo.text) as rendered: + demo.render(0, 0, REPORT, RECEIPT) + copy = "\n".join(call.args[3] for call in rendered.call_args_list) + for value in ('Data(records, representation="folded")', "2,366", "1,376", "1,500", + "41.8% fewer tokens", "30 records. None dropped."): + with self.subTest(value=value): + self.assertIn(value, copy) + self.assertNotIn("63.9%", copy) + + def test_short_loop_core_precedes_opt_in_select(self): + self.assertEqual(demo.LABELS, ("FIT", "CORE", "TOKENFOLD SELECT")) + self.assertEqual(len(demo.DURATIONS), len(demo.LABELS)) + self.assertEqual(sum(demo.DURATIONS), 13) + with patch.object(demo, "text", wraps=demo.text) as rendered: + demo.render(2, 0, REPORT, RECEIPT) + copy = "\n".join(call.args[3] for call in rendered.call_args_list) + self.assertIn("allow_selection=True", copy) + self.assertIn("Over budget?", copy) + self.assertIn("Approved runtime", copy) + self.assertIn("Experimental / unqualified", copy) + + def test_select_brand_is_prominent_and_copy_is_short(self): + with patch.object(demo, "text", wraps=demo.text) as rendered: + demo.render(2, demo.DURATIONS[2] - 0.5, REPORT, RECEIPT) + calls = rendered.call_args_list + headlines = [call for call in calls if call.args[3] == "Tokenfold Select"] + self.assertTrue(headlines, "Use the product name as the headline") + self.assertGreaterEqual(headlines[0].args[4], 60) + copy = [call.args[3] for call in calls] + self.assertIn("03 TOKENFOLD SELECT", copy) + self.assertIn("CONTEXT", copy) + self.assertIn("SELECTED", copy) + self.assertIn("not live", " ".join(copy)) + self.assertLessEqual(len(" ".join(copy).split()), 65) + + def test_selection_animation_never_covers_source_text(self): + baseline = demo.render(2, 0, REPORT, RECEIPT) + for t in (0.6, 0.8, 1, 1.35, 1.6, 2.1, 2.3, 3.1): + frame = demo.render(2, t, REPORT, RECEIPT) + for row in range(3): + y = 310 + row * 68 + box = (78, y + 10, 550, y + 50) + with self.subTest(t=t, row=row): + self.assertIsNone(ImageChops.difference(baseline.crop(box), frame.crop(box)).getbbox()) + + def test_release_copy_and_chapter_numbers(self): + for scene, duration in enumerate(demo.DURATIONS): + with patch.object(demo, "text", wraps=demo.text) as rendered: + demo.render(scene, duration - 0.5, REPORT, RECEIPT) + copy = [call.args[3] for call in rendered.call_args_list] + with self.subTest(scene=scene): + self.assertNotRegex("\n".join(copy).lower(), r"\b(preview|unreleased)\b") + self.assertTrue(any(value.startswith(f"0{scene + 1} / ") for value in copy)) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/fixtures/readme_metrics.json b/tests/fixtures/readme_metrics.json index 50c4ac0..25ffbe6 100644 --- a/tests/fixtures/readme_metrics.json +++ b/tests/fixtures/readme_metrics.json @@ -120,5 +120,16 @@ "v3": 69.3616, "kompress": 64.4106 } + }, + "sdk_0_6_quality": { + "source": "eval/research/sdk_quality_results.json", + "corpus": "eval/research/sdk_quality_corpus.json", + "source_canonical_json_sha256": "603a988c61d0863bbf0b5be0079bf338e8ecbbda18e7374d5be295da892dc347", + "corpus_canonical_json_sha256": "c09c30b004b0bd4405de30b51ac2f215eaad26b5fb3288f8d790ae31fda0474c", + "model": "nvidia/nemotron-3.5-super-vl:free", + "distinct_tasks": 12, + "core": { "baseline_correct": 8, "candidate_correct": 6, "trials": 8, "provider_prompt_reduction_pct": 32.2 }, + "select": { "baseline_correct": 16, "candidate_correct": 16, "trials": 16, "provider_prompt_reduction_pct": 40.1 }, + "qualification": false } }