The failure mode. An under-relaxed forward Picard GS solve that starts on its own fixed point has no gradient to descend. Instead of crossing nl_tol it settles into a small periodic orbit just above it, and burns every iteration up to maxits before raising Exceeded "maxits". The state it is sitting on is, physically, converged; the solver reports a hard failure.
#22 fixed the one fatal, unwrapped instance: verify_sigma0_consistency's state anchor (run.py). This issue is about the rest of the class.
Measured signature ( λ=0.25 β-scan, production defaults urf=0.2 / maxits=800 / nl_tol=1e-6):
|
residual cycle |
axis, frozen for 800 its |
vcontrol_val |
rel. |
| α=1.00 |
2.06e-6 / 4.62e-6 / 4.03e-6 |
R 1.7449, Z 3.1658e-2 |
−1.0200e4 / −1.0206e4 / −1.0204e4 |
5.9e-4 |
| α=1.36 |
1.71e-6 / 4.68e-6 / 4.01e-6 |
R 1.7842, Z 3.5443e-2 |
−3.7827e3 … −3.7870e3 |
1.14e-3 |
Period 3, monotone descent first and then dead-flat parking; every printed digit of R_axis and Z_axis identical across hundreds of iterations; the only moving quantity is vcontrol_val. Cold-starting the identical solve converges in ~20 iterations to the same fixed point (q0 0.9125506 cold vs 0.9125762 warm, 3.8e-5 relative).
Traces: CTM-processing/bouquet_paper/beta_scan_<case>/results/anchorprobe_080909/logs/B0_pfast_self_recon.log, results/reinitval_081009/logs/C_pm_pfast_a1.360.log. Cold/warm pair: anchorprobe_080909/{B1_pfast_self_cold,B4_orig_self_recon,B5_orig_self_cold}/probe_*.json.
(i) Upstream OFT enhancement — TRACKED here
Exceeded "maxits" conflates two states a caller must treat differently:
- genuine non-convergence — the residual is still large, or wandering, and the answer is unusable;
- converged on entry / periodic stall — the residual descended monotonically to a floor a few ×
nl_tol, then locked into a period-N orbit with the axis frozen. The state is usable; the iteration is degenerate.
The second is cheaply detectable inside the nonlinear loop: monotone descent, followed by N consecutive iterations whose residual sequence repeats to within a small relative tolerance while the axis position does not move. Reporting it distinctly (a different error, or a status flag on the exception) would let every OFT caller respond correctly — accept the state, or restart from a different initial guess — instead of guessing.
Same handling as #18: this issue tracks the upstream item. Whether to file it on the OpenFUSIONToolkit repo, and in what form, is @d-burg's call. Measured on OFT base v26.6, branch fix/bootstrap-pchip-derivatives @ 19e08b2, DIII-D mesh — same environment as #18.
(ii) bouquet-side — two decisions
The per-draw anchors share the pattern and are only surviving by masking.
init_psi is called in exactly three places (reconstruct_equilibrium at TokaMaker_interface.py:5196, the IMAS forward-solve init at run.py:613, and the new guard re-init at run.py:975). The per-draw anchor calls it nowhere — it inherits whatever state the previous draw left, then solves the same forward jphi-linterp problem. That is the identical warm-start-onto-a-nearby-fixed-point geometry, differing only in that a draw's perturbation usually displaces it far enough to have a gradient.
When it does fail, nothing counts it:
- TokaMaker_interface.py:~2243 —
except (ValueError, RuntimeError): prints [recon-anchor] WARN: solve failed, then falls back to SWB's total_j_phi;
- the fallback re-solve immediately below is wrapped in a bare
except Exception: pass, so if that fails too the draw simply continues on an unconverged state, silently;
- TokaMaker_interface.py:~3974 — the DIFF_BS state anchor prints
state-anchor solve failed (...); SWB may inherit stale state and carries on.
So a campaign can have any number of draws whose anchor never converged, and the archive records nothing about it.
Decision 1 — should per-draw anchors also re-init ψ? Deliberately not done in #22. It would change the starting point of every draw's anchor solve, therefore every draw trajectory, therefore every golden and every archive: a new campaign and a golden regeneration, not a bugfix. It needs to be decided on its merits, with a seeded before/after, rather than smuggled in behind a guard fix.
Decision 2 — count the masked failures, regardless. This part is cheap and non-behavioural. The three sites above should increment a counter and record it (per-draw attribute plus an ensemble-level total), so that "N of 60 draws fell back to SWB total_j_phi" is visible in the archive rather than only in stdout that nobody reads. If decision 1 later says the anchors are fine, the counter is what proves it. If it says otherwise, the counter is how we find out which archives are affected.
Related: #22 (fixes the one fatal instance; this issue is the rest of the class), #18 (same upstream-tracking pattern), #23 and #25 (the other two residuals surfaced by #22).
The failure mode. An under-relaxed forward Picard GS solve that starts on its own fixed point has no gradient to descend. Instead of crossing
nl_tolit settles into a small periodic orbit just above it, and burns every iteration up tomaxitsbefore raisingExceeded "maxits". The state it is sitting on is, physically, converged; the solver reports a hard failure.#22 fixed the one fatal, unwrapped instance:
verify_sigma0_consistency's state anchor (run.py). This issue is about the rest of the class.Measured signature ( λ=0.25 β-scan, production defaults
urf=0.2/maxits=800/nl_tol=1e-6):vcontrol_valPeriod 3, monotone descent first and then dead-flat parking; every printed digit of R_axis and Z_axis identical across hundreds of iterations; the only moving quantity is
vcontrol_val. Cold-starting the identical solve converges in ~20 iterations to the same fixed point (q0 0.9125506 cold vs 0.9125762 warm, 3.8e-5 relative).Traces:
CTM-processing/bouquet_paper/beta_scan_<case>/results/anchorprobe_080909/logs/B0_pfast_self_recon.log,results/reinitval_081009/logs/C_pm_pfast_a1.360.log. Cold/warm pair:anchorprobe_080909/{B1_pfast_self_cold,B4_orig_self_recon,B5_orig_self_cold}/probe_*.json.(i) Upstream OFT enhancement — TRACKED here
Exceeded "maxits"conflates two states a caller must treat differently:nl_tol, then locked into a period-N orbit with the axis frozen. The state is usable; the iteration is degenerate.The second is cheaply detectable inside the nonlinear loop: monotone descent, followed by N consecutive iterations whose residual sequence repeats to within a small relative tolerance while the axis position does not move. Reporting it distinctly (a different error, or a status flag on the exception) would let every OFT caller respond correctly — accept the state, or restart from a different initial guess — instead of guessing.
Same handling as #18: this issue tracks the upstream item. Whether to file it on the OpenFUSIONToolkit repo, and in what form, is @d-burg's call. Measured on OFT base v26.6, branch
fix/bootstrap-pchip-derivatives@ 19e08b2, DIII-D mesh — same environment as #18.(ii) bouquet-side — two decisions
The per-draw anchors share the pattern and are only surviving by masking.
init_psiis called in exactly three places (reconstruct_equilibriumat TokaMaker_interface.py:5196, the IMAS forward-solve init at run.py:613, and the new guard re-init at run.py:975). The per-draw anchor calls it nowhere — it inherits whatever state the previous draw left, then solves the same forward jphi-linterp problem. That is the identical warm-start-onto-a-nearby-fixed-point geometry, differing only in that a draw's perturbation usually displaces it far enough to have a gradient.When it does fail, nothing counts it:
except (ValueError, RuntimeError): prints[recon-anchor] WARN: solve failed, then falls back to SWB'stotal_j_phi;except Exception: pass, so if that fails too the draw simply continues on an unconverged state, silently;state-anchor solve failed (...); SWB may inherit stale stateand carries on.So a campaign can have any number of draws whose anchor never converged, and the archive records nothing about it.
Decision 1 — should per-draw anchors also re-init ψ? Deliberately not done in #22. It would change the starting point of every draw's anchor solve, therefore every draw trajectory, therefore every golden and every archive: a new campaign and a golden regeneration, not a bugfix. It needs to be decided on its merits, with a seeded before/after, rather than smuggled in behind a guard fix.
Decision 2 — count the masked failures, regardless. This part is cheap and non-behavioural. The three sites above should increment a counter and record it (per-draw attribute plus an ensemble-level total), so that "N of 60 draws fell back to SWB
total_j_phi" is visible in the archive rather than only in stdout that nobody reads. If decision 1 later says the anchors are fine, the counter is what proves it. If it says otherwise, the counter is how we find out which archives are affected.Related: #22 (fixes the one fatal instance; this issue is the rest of the class), #18 (same upstream-tracking pattern), #23 and #25 (the other two residuals surfaced by #22).