Add GPT-5.6 Sol ultra TensorCircuit benchmark results

[QingyunQian]

Summary

• Add the complete 12-task TensorCircuit benchmark archive for gpt-5.6-sol at ultra solver effort, with the audit held at high.
• Add a task-by-task comparison against the existing high run, including solver semantics and audit findings.
• Add three reproducible figures covering pass outcomes, resource use, and per-task agent resource use.
• Preserve candidate solutions, functional output, raw reward/audit details, job summaries, and run metadata under results/gpt56sol-ultra/.

Results

• Final adjudicated validity: 10/12.
• Raw verifier reward and audit result: 10/12.
• Functional and static-policy checks: 12/12.
• Final failures: Challenges 01 and 08.
• Challenge 07's original audit failure is retained in the raw artifacts but overturned by a later exact source-level adjudication.
• high and ultra both score 10/12 after final expert adjudication; both fail Challenges 01 and 08.
• Compared with high, ultra used 26.8% more solving-side tokens and cost 17.6% more, while agent solve wall time was 7.5% lower.
• Across the ten tasks passed by both runs, the geometric mean of ultra_runtime / high_runtime is 0.815. Relative to shared expert references, the geometric-mean runtime ratios are 1.428× for high and 1.165× for ultra.

Effort findings

• Challenge 01: both efforts cap the MPS bond dimension after two-qubit gates, so both share the same semantic shortcut and fail audit.
• Challenge 07: both efforts pass after final adjudication. The raw audit rejected ultra for reusing analytically sampled branches, but the exact reduction study shows that the fixed-discrete-sample pathwise gradient of the ancilla sampling angles is zero. Those angles and realized branches therefore remain unchanged, making branch reuse exact for the published workload rather than a changed optimization trajectory.
• Challenge 08: the raw ultra verifier pass is preserved, but final human expert adjudication rejects it under the intended sampling contract; both efforts are therefore counted as failures.
• Challenge 05: both implementations are semantically valid; the earlier high audit result was an API-semantics false negative rather than a solver difference.

Protocol and validation

• Solver: gpt-5.6-sol, reasoning effort ultra
• Audit: gpt-5.6-sol, reasoning effort high
• Framework/image: TensorCircuit, challenge-benchmark-quantum-tensorcircuit:py311
• One clean Harbor trial per challenge, using isolated temporary task copies
• Original reward and audit artifacts remain unchanged; the Challenge 07 post-hoc adjudication is documented separately.
• Parsed all 12 candidate files, validated archived reward/resource/comparison JSON, regenerated all three figures byte-identically, visually inspected the figures, and ran git diff --check

Result figures

Final task outcomes

Agent-side resource use

[refraction-ray]

solution7 might not be wrong, deserve a detailed look. maybe the problem design is not that sufficient that a simple workaround exists

[refraction-ray]

solution7 might not be wrong, deserve a detailed look. maybe the problem design is not that sufficient that a simple workaround exists

actually this may lead to a very smart way to accelerate challenge07 by somehow exploiting the loophole of the challenge design (might have unexpected reduction structure)

[QingyunQian]

solution7 might not be wrong, deserve a detailed look. maybe the problem design is not that sufficient that a simple workaround exists

actually this may lead to a very smart way to accelerate challenge07 by somehow exploiting the loophole of the challenge design (might have unexpected reduction structure)

i am trying to reformulate and solve it

[refraction-ray]

this somehow lie between success and fail, as it has the tendency of bypassing the framework