Add GPT-5.6 Sol high TensorCircuit benchmark results

[QingyunQian]

Summary

• add the complete 12-task TensorCircuit benchmark run for Codex gpt-5.6-sol at reasoning effort high
• record the functional, static-policy, and GPT-5.6 Sol high audit artifacts for every challenge
• add reproducible GPT-5.5 high versus GPT-5.6 Sol high runtime and agent-resource comparison datasets and figures
• correct a documented challenge-05 LLM-audit false negative using the pinned TensorCircuit package source and direct container API checks

Results

• functional checks: 12/12
• static policy: 12/12
• original LLM-only audit / reward: 9/12
• final adjudicated audit / reward: 10/12
• final audit failures: challenges 01 and 08
• passed-task geometric-mean slowdown: 1.428x for GPT-5.6 Sol high versus 2.197x for the GPT-5.5 high paper run
• agent resources: 26.071M total solving-side tokens, USD 25.527 total recorded cost, 3h 17m 41.7s total agent wall time
• per valid solution after adjudication: USD 2.55 and 19.77 minutes

The two model runs have the same 10/12 validity count. GPT-5.6 Sol high's valid artifacts are about 35% closer to the same-run expert runtime baseline by the aggregate slowdown metric.

Challenge 05 adjudication

The original GPT-5.6 Sol audit claimed that TensorCircuit exp1 implements exp(-i * theta * U / 2) and therefore rejected the candidate for a missing factor of two. In the pinned tensorcircuit-nightly==1.8.0.dev20260726 image, exp1 defaults to half=False and implements exp(-i * theta * U); only named rotations such as rxx and rzz opt into the half-angle convention. The candidate's theta=1j*a and theta=1j*b therefore implement exactly exp(a*X) and exp(b*ZZ).

Direct container checks matched the required X and ZZ exponentials at approximately 2.77e-08 and 5.88e-08 Frobenius error. The candidate also satisfies the specified layer structure, normalization, optimizer, history, and return requirements. Challenge 05 is therefore adjudicated from reward 0.0 to 1.0.

The original audit-details.json and Harbor job-result.json remain unchanged for provenance. The final reward.json, stamp-info.json, and new audit-adjudication.json make the correction machine-readable and auditable.

Protocol and provenance

• solver and verifier ran through Harbor in the TensorCircuit Docker image
• solver model and audit model were both held at gpt-5.6-sol, reasoning effort high
• one official trial per task, with no Harbor retries
• canonical tasks, templates, adapters, and benchmark tests were not modified
• command-log review found no solver network/download commands and no reads of benchmark tests, hidden oracles, or expert solutions
• expert reference artifacts used for runtime comparison were executed only after candidate files were frozen and committed; their source was not opened
• all eleven other audit decisions were source-reviewed; no other clear misclassification was found

Validation

• validated all final reward, stamp, adjudication, resource, and comparison JSON
• independently recomputed pass counts, failure rates, geometric means, token totals, wall-time totals, and cost-per-valid metrics from task-level data
• verified TensorCircuit exp1 semantics in the pinned Docker package for both X and ZZ filters
• regenerated and visually inspected both committed figures
• confirmed both figure scripts reproduce byte-identical PNGs across consecutive runs
• ran Python compilation checks and git diff --check
• confirmed the PR contains no changes under tasks/, templates/, or adapters/
• rebuilt this PR branch directly from sxzgroup/ORBIT-Q:main so no fork-only Fable/scorer history is included

Caveats

This is not a fully controlled model-only comparison: the GPT-5.5 paper run and this run used different hosts and benchmark revisions, and this run uses GPT-5.6 Sol high semantic audit plus documented API adjudication instead of the paper's GPT-5.5 audit plus human adjudication. Each runtime ratio is measured against an expert reference on the same machine within its run, reducing but not eliminating those confounders. Recorded model-service costs are not normalized to a common provider price schedule.

Result figures

Final task outcomes

Agent-side resource use

[QingyunQian]

Challenge 05 Audit Correction

Here is an explanation of the Challenge 05 correction in the latest PR update, as it exposes an interesting benchmark-evaluation issue.

Original audit decision

The original GPT-5.6 Sol audit rejected Challenge 05 because of a supposed factor-of-two error.

It claimed that TensorCircuit's exp1 implements:

exp(-1j * theta * U / 2)

Based on this assumption, the audit concluded that the candidate's

theta = 1j * a
theta = 1j * b

applied filters that were only half as strong as required.

Why the rejection was incorrect

This was an API-level false rejection.

In the exact pinned environment used by the benchmark:

tensorcircuit-nightly==1.8.0.dev20260726

exp1 defaults to:

half=False

and therefore implements:

exp(-1j * theta * U)

The half-angle convention applies to named rotation gates such as:

• rxx

• ryy

• rzz

These gates explicitly set:

half=True

That convention does not apply to the candidate's direct exp1 calls.

Therefore:

exp1(theta=1j * a, U=X) = exp(aX)

and:

exp1(theta=1j * b, U=ZZ) = exp(bZZ)

These are exactly the filters required by the problem.

Numerical verification

I also verified the behavior numerically inside the pinned Docker image.

The generated X and ZZ matrices matched the required exponentials with Frobenius errors of approximately:

Operator | Error against required exponential | Error against incorrect half-angle exponential -- | -- | -- X | 2.77e-08 | 0.145 ZZ | 5.88e-08 | 0.205

These results confirm that the submitted implementation used the correct full-angle convention.

Independent verification of the submission

The rest of the submission was independently checked as well. It:

• uses the required ten cooling layers;

• uses the required brickwork bonds;

• initializes all filter strengths to 0.01;

• normalizes the state after every layer;

• differentiates through the normalization;

• performs exactly 600 Adam updates;

• returns the required pre-update history;

• returns parameters with the required shapes.

The functional evaluator also passed.

Reward adjudication

For these reasons, I treated Challenge 05 as an LLM-audit false negative and adjudicated its reward from:

0.0 → 1.0

This changes the final benchmark result from:

9/12 → 10/12

The derived runtime and resource figures have been updated accordingly.

Transparency and retained artifacts

To keep the correction fully transparent, I did not overwrite the original LLM audit or the Harbor aggregate output.

The original files remain unchanged:

audit-details.json
job-result.json

A separate file records the source-level and numerical evidence:

audit-adjudication.json

The following outputs now reflect the adjudicated result:

reward.json
summary data
documentation
figures

Broader benchmark-design observation

More broadly, this correction highlights an interesting benchmark-design issue.

An LLM judge can correctly understand the overall algorithm while still confidently confusing two closely related framework API conventions. Because the audit score is binary and multiplicative, one localized API hallucination can change the full benchmark result.

For framework-specific semantic judgments, it therefore seems important to use at least one of the following:

1. Check the source code of the exact installed package version.

2. Run a small numerical API-level verification.

3. Retain a lightweight human or API-grounded adjudication step.

Review of the remaining challenges

I also re-reviewed the other eleven audit decisions and did not find another comparably clear factual misclassification.

Challenge 08 remains a more interpretive policy question because it uses correlated Sobol quasi-sampling. Challenge 05, by contrast, involves an objectively testable API-semantics error.

[refraction-ray]

sounds reaonable that problem 5 solution is correct

[refraction-ray]

this random number generation is weird, no idea why the agent choose this over a uniform distrubution