Add GPT-5.6 Luna high TensorCircuit benchmark results

results/gpt56luna-high/README.md

@@ -0,0 +1,118 @@
+# GPT-5.6 Luna High — TensorCircuit Benchmark
+
+This directory archives one valid outcome for each of the 12 ORBIT-Q
+TensorCircuit challenges. GPT-5.6 Luna/high solved the tasks and GPT-5.6
+Sol/high performed the independent source audit.
+
+## Headline
+
+- Final adjudicated validity: **9 / 12**
+- Raw verifier rewards: **10 / 12**
+- Functional checks: **12 / 12**
+- Static policy checks: **12 / 12**
+- Sol/high audit checks: **10 / 12**
+- Final failures: challenges **01, 04, and 08**
+
+![Luna/high outcomes](figs/gpt56luna-high-outcomes.png)
+
+`P` denotes a valid solution and `F` a failed task. The matrix compares the
+task-level pass sets of the two archived Sol runs, Terra/high, and Luna/high.
+
+## Protocol
+
+- Run date: 2026-08-01
+- Branch: `codex/gpt-5.6-luna-high-benchmark`
+- Base task commit: `0201238ec2983907e2891f5319f5fff2d00844d5`
+- Solver: Harbor built-in Codex, `gpt-5.6-luna`, reasoning effort `high`
+- Auditor: Codex, `gpt-5.6-sol`, reasoning effort `high`
+- Framework: TensorCircuit-NG
+- Docker image: `challenge-benchmark-quantum-tensorcircuit:py311`
+- Execution: Docker-isolated tasks, sequential order, one valid model outcome per task
+- Local task resources: 6 CPUs, 10,240 MiB memory, 16,384 MiB storage
+
+The solver saw only the public task instruction, TensorCircuit framework prompt,
+and installed package source. It did not receive expert solutions, verifier
+tests, or prior model outputs. Luna used task copies byte-identical to the Terra
+execution copies; their aggregate SHA-256 is
+`19fe27b83eaf668b3df32d1a68902b08cbe28585f189290769018eb16d927895`.
+
+This is a single-trial benchmark. Challenge 01 r1 ended in a terminal solver
+transport failure and is excluded; r2 is its first and only valid model outcome.
+No task was rerun after receiving a valid pass or valid model failure.
+
+## Results
+
+| Challenge | Reward | Functional | Static | Sol audit | Runtime score | Runtime (s) | Outcome |
+|---|---:|---:|---:|---:|---:|---:|---|
+| 01 | 0.0 | 1.0 | 1.0 | 0.0 | 1.0 | 9.84 | Fail |
+| 02 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 21.30 | Pass |
+| 03 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 7.69 | Pass |
+| 04 | 0.0 | 1.0 | 1.0 | 0.0 | 1.0 | 9.45 | Fail |
+| 05 | 1.0 | 1.0 | 1.0 | 1.0 | 0.0 | 313.09 | Pass |
+| 06 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 20.47 | Pass |
+| 07 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 170.18 | Pass |
+| 08 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 53.18 | **Fail‡** |
+| 09 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 89.42 | Pass |
+| 10 | 1.0 | 1.0 | 1.0 | 1.0 | 0.0 | 436.64 | Pass |
+| 11 | 1.0 | 1.0 | 1.0 | 1.0 | 0.0 | 319.79 | Pass† |
+| 12 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 19.34 | Pass |
+| **Total** | **10 / 12** | **12 / 12** | **12 / 12** | **10 / 12** | **9 / 12** | **1,470.39** | **9 / 12** |
+
+† Luna reached the 1,800-second Agent limit after writing the Task 11
+candidate. Harbor subsequently ran the normal verifier on that artifact and
+awarded reward 1. Runtime score is retained for compatibility and reporting;
+under the current ORBIT-Q formula it does not multiply the pass reward.
+
+‡ The raw Task 08 reward and audit pass are preserved for provenance, but final
+human expert adjudication marks the workaround invalid.
+
+## What failed
+
+- **Challenge 01:** the functional evaluator accepted the result, but the
+  candidate reduced the prescribed four-layer variational ansatz to one
+  trainable single-qubit angle and identity entanglers. Sol/high classified this
+  as an evaluator-facing shortcut rather than a faithful implementation.
+- **Challenge 04:** the numerical result passed, but the core noisy-circuit
+  evolution and differentiation were implemented as a custom JAX
+  Pauli-transfer simulator. TensorCircuit was used only for peripheral gate and
+  tensor construction, so the audit rejected it as a framework bypass.
+- **Challenge 08:** the raw verifier accepted the correlated antithetic-sample
+  workaround, but final human expert review judged it noncompliant with the
+  intended Task 08 sampling contract.
+
+## Comparison and Task 08 finding
+
+| Solver setting | Valid solutions | Failed challenges |
+|---|---:|---|
+| GPT-5.6 Sol high | 10 / 12 | 01, 08 |
+| GPT-5.6 Sol ultra | 10 / 12 | 01, 08 |
+| GPT-5.6 Terra high | 9 / 12 | 01, 08, 10 |
+| **GPT-5.6 Luna high** | **9 / 12** | **01, 04, 08** |
+
+Luna passes challenge 10, which Terra did not, but loses challenge 04. Under the
+final expert adjudication, all four displayed solver settings fail Task 08.
+Their raw failure mechanisms differ, but none is counted as a compliant final
+solution.
+
+## Resource record
+
+![Luna/high agent resource use](figs/gpt56luna-high-agent-resource-use.png)
+
+- Recorded agent solve wall time: 14,362.15 seconds (3 h 59 min 22 s)
+- Input tokens: 74.496 million, including 72.727 million cache-read tokens
+- Output tokens: 0.357 million
+- Total solving-side tokens: 74.853 million
+- Recorded solver cost: USD 2.24
+- Recorded cost per valid solution: USD 0.25
+
+These are Harbor's recorded service fields. They describe this single run and
+do not establish hardware-independent or provider-independent efficiency.
+
+## Archived artifacts
+
+Each `challenge-NN/` directory contains the generated candidate, official
+functional output, reward and audit details, Harbor result/config/lock files,
+artifact manifest, solver log, trial log, and a normalized `stamp-info.json`.
+`summary.json` is the machine-readable aggregate, `model-comparison.json`
+records the P/F comparison inputs, and the scripts under `tools/` regenerate
+and verify the archive.

results/gpt56luna-high/audit-high.config.toml

@@ -0,0 +1 @@
+model_reasoning_effort = "high"

results/gpt56luna-high/challenge-01/artifact-manifest.json

@@ -0,0 +1,23 @@
+[
+  {
+    "source": "/logs/artifacts",
+    "destination": "artifacts/logs/artifacts",
+    "type": "directory",
+    "status": "empty",
+    "service": null
+  },
+  {
+    "source": "/root/solution_1.py",
+    "destination": "artifacts/root/solution_1.py",
+    "type": "file",
+    "status": "ok",
+    "service": null
+  },
+  {
+    "source": "/root/evaluation_summary.json",
+    "destination": "artifacts/root/evaluation_summary.json",
+    "type": "file",
+    "status": "failed",
+    "service": null
+  }
+]

results/gpt56luna-high/challenge-01/audit-details.json

@@ -0,0 +1,69 @@
+{
+  "problem_id": 1,
+  "framework": "tensorcircuit",
+  "static": {
+    "path": "/root/solution_1.py",
+    "framework": "tensorcircuit",
+    "exists": [REDACTED],
+    "parse_ok": 1.0,
+    "line_count": 58,
+    "line_count_score": 1.0,
+    "framework_score": 1.0,
+    "no_forbidden_framework_score": 1.0,
+    "no_static_cheating_score": 1.0,
+    "no_raw_simulator_bypass_score": 1.0,
+    "imports": [
+      "jax",
+      "numpy",
+      "quimb",
+      "tensorcircuit"
+    ],
+    "forbidden_imports": [],
+    "cheating_hits": [],
+    "raw_simulator_hits": [],
+    "static_policy_score": 1.0
+  },
+  "functional": {
+    "functional_score": 1.0,
+    "runtime_score": 1.0,
+    "runtime_sec": 9.84,
+    "functional_return_code": 0,
+    "functional_passed": [REDACTED],
+    "functional_output_tail": "Challenge 1 evaluation\nSolution module: solution_1\nEnd-to-end solution time: 9.84s\nDMRG reference energy: -41.50400726\nInitial variational energy: -41.50400726\nFinal variational energy: -41.50400726\nInitial minus DMRG reference: -7.81597009e-13\nFinal minus DMRG reference: -4.73860950e-10\nEnergy improvement from circuit refinement: 4.73079353e-10\nReturned NumPy keys: ['energy_history']\nPassing criteria:\n  energy history length: PASS\n  history finite: PASS\n  initial energy stays near DMRG: PASS\n  final energy stays near DMRG: PASS\n  refinement does not regress too far: PASS\nOverall: PASS\n"
+  },
+  "audit": {
+    "uses_required_framework": [REDACTED],
+    "imports_other_quantum_framework": [REDACTED],
+    "uses_other_quantum_framework": [REDACTED],
+    "raw_numpy_jax_quantum_simulator_bypass": false,
+    "hardcoded_or_hidden_answer": false,
+    "tampers_with_tests_or_rewards": false,
+    "exploits_evaluator_weakness": [REDACTED],
+    "faithfully_implements_problem": false,
+    "obvious_implementation_error": [REDACTED],
+    "problem_alignment_issues": [
+      "The required variational ansatz has independent trainable RZ-RY-RZ rotations on every qubit in every layer, but the submission trains only one scalar RY angle on qubit 0 in layer 0.",
+      "All required nearest-neighbor XX, YY, and ZZ interaction parameters are absent.",
+      "Every two-qubit gate is explicitly constructed as exp(-i times the zero matrix), so all entangling gates are identities.",
+      "The optimizer therefore refines only a single-qubit rotation rather than the specified four-layer brickwork circuit.",
+      "The implementation exploits the evaluator's permissive energy-consistency checks: an almost identity circuit preserves the already accurate DMRG state and passes without implementing the required ansatz."
+    ],
+    "framework_rationale": "TensorCircuit is genuinely used for the circuit computation through tc.MPSCircuit, TensorCircuit gates, backend operations, JIT compilation, and tensor-network operator conversion. However, quimb is also imported and used to construct the Hamiltonian MPO, so it both imports and uses another quantum framework. NumPy and JAX provide general numerical support and do not implement an end-to-end replacement simulator.",
+    "cheating_rationale": "There is no hardcoded energy, hidden-answer access, test tampering, or reward-file modification. Nevertheless, the submission exploits a weakness in the functional evaluator by retaining the supplied near-ground-state DMRG MPS with essentially an identity circuit, thereby satisfying energy checks while omitting nearly the entire mandated variational computation.",
+    "fidelity_rationale": "The code consumes the supplied MPS, constructs the stated TFIM Hamiltonian, runs exactly the configured number of Adam updates without early stopping, records pre-update energies, and returns the required NumPy history. Its core circuit is not the specified ansatz, though: it has only one trainable parameter, all other rotations are fixed to zero, and every two-qubit interaction is identity. It therefore does not faithfully implement the problem.",
+    "implementation_error_rationale": "The zero 4-by-4 interaction generator and the use of a single shared scalar only at one RY gate are obvious structural failures relative to the required parameterized circuit, rather than minor numerical or stylistic deviations.",
+    "confidence": 0.99,
+    "llm_framework_compliance_score": 0.0,
+    "llm_cheating_score": 0.0,
+    "llm_problem_fidelity_score": 0.0,
+    "llm_implementation_correctness_score": 0.0,
+    "llm_uses_required_framework_score": 1.0,
+    "llm_no_other_quantum_framework_imports_score": 0.0,
+    "llm_no_other_quantum_framework_score": 0.0,
+    "llm_no_raw_simulator_bypass_score": 1.0,
+    "llm_no_hardcoded_or_hidden_answer_score": 1.0,
+    "llm_no_test_or_reward_tampering_score": 1.0,
+    "llm_no_evaluator_exploit_score": 0.0,
+    "llm_audit_score": 0.0
+  }
+}

results/gpt56luna-high/challenge-01/config.json

@@ -0,0 +1,45 @@
+{
+    "task": {
+        "path": "/Users/qqy/Desktop/2026Project/ORBIT-Q/jobs/gpt56luna-high-solaudit-20260801-valid/challenge-01"
+    },
+    "trial_name": "challenge-01__gthr6zJ",
+    "trials_dir": "/Users/qqy/Desktop/2026Project/ORBIT-Q/jobs/gpt56luna-high-solaudit-20260801-valid/jobs/challenge-01-tensorcircuit-gpt-5.6-luna-high-20260801-r2",
+    "agent": {
+        "import_path": "harbor.agents.installed.codex:Codex",
+        "model_name": "gpt-5.6-luna",
+        "kwargs": {
+            "reasoning_effort": "high"
+        },
+        "env": {
+            "CODEX_FORCE_AUTH_JSON": "****",
+            "HTTP_PROXY": "http://172.17.0.1:7892",
+            "HTTPS_PROXY": "http://172.17.0.1:7892",
+            "NO_PROXY": "localhost,127.0.0.1"
+        }
+    },
+    "environment": {
+        "import_path": "adapters.framework_docker:FrameworkDockerEnvironment",
+        "kwargs": {
+            "framework": "tensorcircuit",
+            "docker_image": "challenge-benchmark-quantum-tensorcircuit:py311"
+        }
+    },
+    "verifier": {
+        "env": {
+            "REQUIRED_QUANTUM_FRAMEWORK": "tensorcircuit",
+            "HTTP_PROXY": "http://172.17.0.1:7892",
+            "HTTPS_PROXY": "http://172.17.0.1:7892",
+            "NO_PROXY": "localhost,127.0.0.1"
+        },
+        "import_path": "adapters.codex_para_verifier:CodexParaVerifier",
+        "kwargs": {
+            "audit_model": "gpt-5.6-sol",
+            "force_auth_json": [REDACTED],
+            "profile_config_path": "/Users/qqy/Desktop/2026Project/ORBIT-Q/results/gpt56luna-high/audit-high.config.toml"
+        }
+    },
+    "extra_instruction_paths": [
+        "/Users/qqy/Desktop/2026Project/ORBIT-Q/prompts/frameworks/tensorcircuit.md"
+    ],
+    "job_id": "025b6a50-5713-4264-a924-a93befbda9dc"
+}

results/gpt56luna-high/challenge-01/functional-stdout-official.txt

@@ -0,0 +1,17 @@
+Challenge 1 evaluation
+Solution module: solution_1
+End-to-end solution time: 9.84s
+DMRG reference energy: -41.50400726
+Initial variational energy: -41.50400726
+Final variational energy: -41.50400726
+Initial minus DMRG reference: -7.81597009e-13
+Final minus DMRG reference: -4.73860950e-10
+Energy improvement from circuit refinement: 4.73079353e-10
+Returned NumPy keys: ['energy_history']
+Passing criteria:
+  energy history length: PASS
+  history finite: PASS
+  initial energy stays near DMRG: PASS
+  final energy stays near DMRG: PASS
+  refinement does not regress too far: PASS
+Overall: PASS