Skip to content

Add a complete agent-axis run (Fable 5 / Cursor agent × TensorCircuit-NG): 12/12 official reward 1.0, runtime-ratio protocol, and two infrastructure fixes - #4

Draft
QingyunQian wants to merge 46 commits into
sxzgroup:v2from
QingyunQian:main

Conversation

@QingyunQian

Copy link
Copy Markdown

What this PR does

1. A complete agent-axis benchmark run (results/fable5/)

Adds a full 12-task agent-axis data row: solver = Fable 5 via Cursor agent, framework fixed at TensorCircuit-NG. Every task passes the unmodified official verifier — functional evaluator, static policy, and Codex LLM audit (model gpt-5.6-sol, recorded per task) — with reward = 1.0 on all 12 tasks. Per-task artifacts include the solution, verifier outputs, audit provenance, and a same-machine runtime comparison against the publication reference. A full report (results/fable5/REPORT.md) documents the pipeline, per-task observations, and reproducible summary figures.

Harness caveat, stated upfront: the solver ran outside the Harbor agent container (Cursor agent operating on the workspace, verification through the documented verifier-only candidate-check path). Artifact-side metrics (pass components, runtime, T/T_ref) follow the standard verifier and are comparable; agent-side metrics (tokens, solve wall time) are not comparable with in-container rows. Solving-integrity protocol: only instruction.md and the framework prompt were read; tests/ and solution/ were never opened during solving; references were executed for timing only, after each candidate was frozen.

2. Fix: reward formula desync between docs and scorer

Since ea088db, AGENTS.md/README declare reward = functional × static_policy × llm_audit ("runtime … must not reduce pass reward"), but score_submission.py (template + all 12 task copies) still multiplied runtime_score into reward. Observed impact: a run with all three gating components at 1.0 and runtime_sec = 196.25 scored reward = 0.8645833… — exactly (300−196.25)/(300−180). This PR syncs the code with the declared policy (runtime_score/runtime_sec still computed and reported unchanged). Also available as a standalone atomic PR if preferred.

3. Fix: framework pin predates the omeco integration

frameworks/tensorcircuit/requirements.txt pinned tensorcircuit-nightly==1.7.0.dev20260618, which predates set_contractor("omeco") landing in master, so the publication references for tasks 01/05 crash on the public image (the omeco package itself was already listed). Bumped to 1.8.0.dev20260726: all 12 references now run unmodified; the four candidates already measured were re-verified on the new pin (all still functional 1.0).

What we found

  • Artifact efficiency (T/T_ref, same machine & image): official-record median 2.39. At matched precision the candidate beats the expert reference on two tasks (06: 0.76, 11: 0.74), is within 1.3× on two more (05/07), sits at 1.7–3.6× on six (mostly XLA-compile/contraction-path overhead), and shows two large gaps (04: 11.9, 10: 18.1 — the 18-qubit CZ hyperedge via multicontrol-MPO is ~18× slower than the expert treatment).
  • Precision attribution: all 12 references use complex64 (checked post-freeze, configuration lines only). Five candidates ran complex128; their complex64 controls were re-run through the official evaluator. Notably, task 04's control fails functionally at single precision (the 2e-4 channel-fit tolerance is unreachable with the vectorized-DM approach) while the reference passes at c64 — so that ratio partly encodes an algorithmic precision-requirement gap, not just speed.
  • LLM-audit boundary calibration: task 04's first implementation (a numerically-validated hand-built Kraus-ladder network on raw tensornetwork nodes) passed functional+static but was rejected by the Codex audit as a raw-simulator bypass; the framework-native rewrite on tc.MPSCircuit (numerically identical to 3e-15) passed and runs 4× slower — a concrete quantification of the "compliance tax" the benchmark is designed to surface.
  • Framework observations collected in the report: default-contractor pitfalls (full-wavefunction materialization at 49 qubits; MPO-sandwich hangs), JIT constant-capture of a 2.9 GB sparse Pauli-sum at 22 qubits, cond_measure/QuditCircuit/multicontrol working well as framework-native primitives, and a measurement-insensitive protocol emerging from trajectory-averaged feedback optimization (task 07).

Note for maintainers

  • The default audit model in AGENTS.md/README/conf.toml was updated from gpt-5 to gpt-5.6-sol during the run (the ChatGPT-auth catalog no longer serves the bare gpt-5 slug). Keep or revert as you prefer.
  • results/fable5/ placement and naming are proposals; happy to move/rename if you want a different results layout.

cursoragent and others added 30 commits July 27, 2026 11:37
…, verify tools

Co-authored-by: QingyunQian <QingyunQian@users.noreply.github.com>
…d stamp provenance

Co-authored-by: QingyunQian <QingyunQian@users.noreply.github.com>
… cloud precheck

Co-authored-by: QingyunQian <QingyunQian@users.noreply.github.com>
…(01, 02)

Co-authored-by: QingyunQian <QingyunQian@users.noreply.github.com>
…gyunQian/ORBIT-Q into cursor/fable5-benchmark-7148

Co-authored-by: QingyunQian <QingyunQian@users.noreply.github.com>
Co-authored-by: QingyunQian <QingyunQian@users.noreply.github.com>
Co-authored-by: QingyunQian <QingyunQian@users.noreply.github.com>
…time ratio; challenge-03 stamp in summary

Co-authored-by: QingyunQian <QingyunQian@users.noreply.github.com>
…est environment

Co-authored-by: QingyunQian <QingyunQian@users.noreply.github.com>
…co integrated); re-measure all candidates and references

Co-authored-by: QingyunQian <QingyunQian@users.noreply.github.com>
…udit rejected Kraus-ladder v1

Co-authored-by: QingyunQian <QingyunQian@users.noreply.github.com>
…k, runtime ratio 1.19

Co-authored-by: QingyunQian <QingyunQian@users.noreply.github.com>
…ltiplier from reward (doc changed in ea088db, code never followed); template and all 12 task copies patched identically (upstream task generator source unavailable in this workspace)

Co-authored-by: QingyunQian <QingyunQian@users.noreply.github.com>
…04's stamped reward

Co-authored-by: QingyunQian <QingyunQian@users.noreply.github.com>
…DE evolution), precheck, ratio 1.09

Co-authored-by: QingyunQian <QingyunQian@users.noreply.github.com>
…ltiplier from reward (doc changed in ea088db, code never followed); template and all 12 task copies patched identically (upstream task generator source unavailable in this workspace)

Co-authored-by: QingyunQian <QingyunQian@users.noreply.github.com>
… 1.28; record 04 re-stamp and 06 stamp

Co-authored-by: QingyunQian <QingyunQian@users.noreply.github.com>
…io 2.64; record 07 stamp

Co-authored-by: QingyunQian <QingyunQian@users.noreply.github.com>
…ing-7148

Fix verifier/doc desync: remove deprecated runtime_score multiplier from reward in score_submission.py (policy declared in ea088db)
…analytic max), ratio 3.63; record 08 stamp

Co-authored-by: QingyunQian <QingyunQian@users.noreply.github.com>
QingyunQian and others added 14 commits July 28, 2026 02:20
…), ratio 18.06; record 09 stamp

Co-authored-by: QingyunQian <QingyunQian@users.noreply.github.com>
…, ratio 0.74; record 10 stamp

Co-authored-by: QingyunQian <QingyunQian@users.noreply.github.com>
… record 11 stamp

Co-authored-by: QingyunQian <QingyunQian@users.noreply.github.com>
…summary figures and full report

Co-authored-by: QingyunQian <QingyunQian@users.noreply.github.com>
…l 1.11x); annotate summary and report

Co-authored-by: QingyunQian <QingyunQian@users.noreply.github.com>
…y post-freeze check); 0.74 is a matched-precision win; document per-task dtype matchups

Co-authored-by: QingyunQian <QingyunQian@users.noreply.github.com>
…ts the reference (0.76); c04's method requires double precision; report and figures updated

Co-authored-by: QingyunQian <QingyunQian@users.noreply.github.com>
…ntime-ratio protocol, matched-precision controls, report and figures

Co-authored-by: QingyunQian <QingyunQian@users.noreply.github.com>
Co-authored-by: QingyunQian <QingyunQian@users.noreply.github.com>
…el control diamonds; user-approved push to main

Co-authored-by: QingyunQian <QingyunQian@users.noreply.github.com>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this figure has no extra info than the other one?

cursoragent and others added 2 commits July 28, 2026 11:48
…le agent resource-use figure (solve wall time + artifact runtime); record solve-time data with method note

Co-authored-by: QingyunQian <QingyunQian@users.noreply.github.com>
…ol / Fable 5) with dataset and script; restore ratio-bars title

Co-authored-by: QingyunQian <QingyunQian@users.noreply.github.com>
@QingyunQian
QingyunQian changed the base branch from main to v2 August 3, 2026 06:06
@QingyunQian

Copy link
Copy Markdown
Author

As it is Cursor agent ,which means different harness for the task, so transfer to draft and dont merge now

@QingyunQian
QingyunQian marked this pull request as draft August 3, 2026 06:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants