Load-balance, active-box windowing, block-structured AMR - #1628
Open
sbryngelson wants to merge 922 commits into
Open
Load-balance, active-box windowing, block-structured AMR#1628sbryngelson wants to merge 922 commits into
sbryngelson wants to merge 922 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces an opt-in (“default-off”) family of performance/diagnostic features (load-weight and SFC partition diagnostics, weighted init-time decomposition, rank timing), plus major simulation capabilities (active-box RHS windowing and block-structured AMR) and corresponding post-processing support and documentation/validation updates.
Changes:
- Adds new runtime parameters and toolchain metadata/validation hooks for the experimental performance/AMR feature family.
- Extends the simulation code with new modules for active-box restriction, load-weight diagnostics, SFC partition reporting, rank timing, and AMR integration points (including restart/output plumbing).
- Updates post_process to read/write AMR fine-block overlays and adds/updates golden metadata plus documentation/indexing.
Reviewed changes
Copilot reviewed 82 out of 94 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| toolchain/mfc/params/descriptions.py | Adds user-facing descriptions for new experimental/performance parameters. |
| toolchain/mfc/params/definitions.py | Registers new parameters (AMR, hybrid sensors, load-balance diagnostics) and target applicability. |
| toolchain/mfc/lint_docs.py | Treats new validator checks as non-physics doc checks. |
| tests/F980C769/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/ECABA006/golden-metadata.txt | Adds golden metadata for active-box test coverage. |
| tests/DD4CD8F3/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/CC4213FD/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/BD21A5C0/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/BCBA6E74/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/ACE05393/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/987D9025/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/852CCB81/golden-metadata.txt | Adds golden metadata for AMR-related golden tests. |
| tests/65C375B4/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/4DADE04B/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/454C565F/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/3A474BEE/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/2FC423D3/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/13945217/golden-metadata.txt | Adds golden metadata for moving-IB under AMR test coverage. |
| src/simulation/m_viscous.fpp | Clamps FD coefficient indexing to avoid ghost-region coefficient OOB in IB drag gradient evaluation. |
| src/simulation/m_time_steppers.fpp | Integrates active-box bounds into RK update loops and interleaves AMR fine-stage/subcycle operations. |
| src/simulation/m_start_up.fpp | Wires up new modules (rank timing, active-box, load-weight, SFC partition, AMR) into init/timestep/finalize and restart I/O. |
| src/simulation/m_sfc_partition.fpp | Adds analysis-only SFC tiling + weighted partition prediction and reporting. |
| src/simulation/m_rank_timing.fpp | Adds per-rank wall-time imbalance measurement helpers and reporting. |
| src/simulation/m_load_weight.fpp | Adds per-cell load-weight field construction and rank-level imbalance reporting. |
| src/simulation/m_hypoelastic.fpp | Refactors FD coefficient setup into a callable update routine (supporting AMR grid swaps). |
| src/simulation/m_global_parameters.fpp | Adds AMR working-state mirrors and slot selection helper plus defaults for new parameters. |
| src/simulation/m_data_output.fpp | Adds output/report hooks for load-weight, SFC partition, and rank-time diagnostics. |
| src/simulation/m_checker.fpp | Adds input validation/prohibits for active-box, hybrid sensors, load-balance, and AMR configurations. |
| src/simulation/m_active_box.fpp | Adds active-box initialization/growth and debug envelope checking. |
| src/simulation/m_acoustic_src.fpp | Adds AMR-aware handling of acoustic source support (bounding boxes and overlap abort). |
| src/post_process/m_start_up.fpp | Calls AMR fine-data reader and AMR overlay writer when amr is enabled. |
| src/post_process/m_global_parameters.fpp | Adds default-off amr flag for post_process overlay behavior. |
| src/post_process/m_data_output.fpp | Implements AMR fine-block overlay mesh/variables output (Silo/binary) and multimesh registration. |
| src/common/m_phase_change.fpp | Exposes per-cell Newton iteration count and threads it through relaxation to support load-weighting. |
| src/common/m_global_parameters_common.fpp | Adjusts start_idx lifecycle/allocation and makes load_weight_wrt visible to GPU macros. |
| src/common/m_derived_types.fpp | Introduces a simple t_box type used by new partitioning infrastructure. |
| src/common/m_box.fpp | Adds box/partition arithmetic helpers (equal/weighted splits, box-from-splits). |
| src/common/m_boundary_common.fpp | Skips BC buffer population during AMR fine advance to rely on coarse-driven ghost fill. |
| docs/module_categories.json | Registers new modules under documentation categories. |
| docs/documentation/readme.md | Adds AMR section link to the documentation index. |
| .typos.toml | Adds project-specific abbreviations to the spelling allowlist. |
| D = ((gs_min(lp) - 1.0_wp)*cvs(lp))/((gs_min(vp) - 1.0_wp)*cvs(vp)) | ||
|
|
||
| #ifdef MFC_SIMULATION | ||
| if (relax .and. load_weight_wrt) then |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #1628 +/- ##
==========================================
+ Coverage 62.81% 63.10% +0.28%
==========================================
Files 84 96 +12
Lines 21975 30211 +8236
Branches 3224 4943 +1719
==========================================
+ Hits 13804 19065 +5261
- Misses 5949 7846 +1897
- Partials 2222 3300 +1078 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Member
Author
|
Upstream latent gap found during the MHD+AMR investigation (independent of this PR): Also for the record: MHD+AMR was attempted and re-gated on measured evidence rather than assumption — the coarse/fine seam is a continuous O(1) div(B) source that cleaning spreads but cannot remove (details in the amr.md support matrix row and commit ac203b1). |
sbryngelson
changed the title
Member
Author
|
2 tasks
fahnab666
added a commit
to fahnab666/mfc-nabid
that referenced
this pull request
Aug 29, 2026
…isting lint debt Merges the 20 commits added to sbryngelson/up-mega since ee7758b: the CCE descriptor and GPU_DECLARE fixes for the Frontier AMR abort, the NVHPC declare-ordering fix, restart format v2 (per-block owner/extents), amr_blocking_factor defaulting to 4, and the level-order clustering reductions. The delta is confined to AMR: m_amr, m_amr_regrid, m_amr_restart, m_constants, m_global_parameters, cmake/GPU.cmake, 14 AMR goldens, and two AMR-only toolchain hunks. No JWL, IBM, Riemann, or EOS source file is touched and no non-AMR golden changes. Merge was conflict-free; the local static-AMR GPU allocation cap for IBM survives intact. Also clears lint debt that predates this merge and blocked any commit: formatter line-wrapping in six JWL/IBM files, the duplicated rxn_val PROHIBITs in pre_process/m_checker.fpp (already enforced in case_validator.check_stiffened_eos, as its own docstring states), the stale hyperelasticity row in case.md (removed upstream), the missing sec-mhd anchor, and m_jwl/m_jwl_sources missing from module_categories.json.
…ntier CCE gpu-acc) A declare-create on a module allocatable binds a present-table entry to the descriptor at program init; the move_alloc at the allocation site then swaps that descriptor out, so every later kernel lookup misses. All 36 AMR tests on the CCE gpu-acc lane died with 'find_in_present_table failed' for amr_cons_br (432 hits) and amr_scr_prim (9). amr_cg has always used move_alloc + GPU_ENTER_DATA with no declare and works on every lane; this makes the other two match. The declares were added for the lib-4425 uninitialized-descriptor abort, but the move_alloc is what actually fixed that, so they were redundant as well as harmful. amdflang's OpenMP runtime creates the mapping implicitly, which is why no local gate could see this.
s_populate_variables_buffers sits inside s_compute_rhs, which PH_COARSE wraps whole, so a uniform (amr=F) run reported 95% 'coarse' and no communication at all. That made the one comparison that matters impossible: how much of AMR's ~31% communication share is AMR's own versus the solver's baseline.
Halo probe counts distinct block slots this rank's metadata reads touch between regrids, reduced with MPI_MAX over ranks rather than reported from rank 0 (which owns a domain corner and is the least connected rank). The reduction is entered by EVERY rank: an earlier draft placed it inside the proc_rank == 0 guard, so one rank called ALLREDUCE while the rest ran ahead into other collectives and the job died with MPI_ERR_TRUNCATE. Grid efficiency reports tagged versus covered coarse cells, which prices amr_tag_eps and amr_buf and bounds the quoted payoff.
…reshold A histogram of the tagging ratio g/(2*r0) over ~49 regrids of this case shows the distribution is continuous from 0 to 0.5 with no gap, and amr_tag_eps = 0.05 already sits at a local minimum, so there is nowhere better to move it. Near the threshold the density is ~9,400 cells per unit ratio per regrid, so one flip across 49 regrids needs a perturbation of ~2e-6; roundoff is 1e-12 to 1e-14 and cannot reach that. The flipped tag is therefore a consequence of divergence, not its cause: roundoff is amplified by a chaotic configuration until it reaches ~1e-6, and only then do tags flip. t_step_stop 100 -> 25 keeps 12 regrids and the store still grows 25 -> 36 -> 49 blocks with the cap never binding, while giving roundoff far less room to grow. override_tol = 1e-11 covers the residual toolchain roundoff (CCE measured 1.06e-12) without being loose enough to hide a mesh divergence.
The allocation is guarded by hypo_nc_mode == hypo_nc_mode_dual_pass alone, but the teardown sat nested inside if (.not. igr), so any configuration with igr and dual_pass together leaked flux_n, flux_src_n and flux_gsrc_n along with their %sf payloads and device maps. Nothing couples igr to hypo_nc_mode in the validator, so the combination is reachable. Hoisted the teardown block out of the igr guard so the two are textually identical.
Nothing in the tree writes this array, but s_finalize_riemann_solver_hatR copies all of 1:sys_size out of it into flux_gsrc_n(id), which m_rhs folds into the RHS. Under cyl_coord with the dual-pass path that fed uninitialized memory into the solution. Mirrors the zeroing its twin flux_gsrc_rsx_vf already gets for the same reason.
s_compute_load_weight reads it on the first s_write_data_files, which for a run saving at t_step_start precedes any relaxation sweep, so the first load-weight diagnostic and load_weight.*.dat came from uninitialized memory. The device copy is the one that matters: the reader is a GPU_PARALLEL_LOOP and the only writer sits inside a device region, so a host-only assignment would have reached neither.
s_write_amr_restart marks format v2 with a NEGATIVE rank count, and the simulation reader was taught both formats, but the post_process reader was not. Its rank-count check therefore fires unconditionally (a negative can never equal num_procs) and reports a rank-count mismatch even when the counts match, which sends the reader after the wrong problem; it also still reads the 3*num_procs extents record that v2 does not write. Detect the negative marker in both the serial and MPI branches and abort saying what is actually wrong. This is a stopgap: reading v2 in post needs the whole-block ownership layout (one contiguous chunk per block, ownership from the file's 4-int record rather than derived geometrically), which is a separate change. Nothing in the default suite runs AMR with post_process as a target, which is why the break shipped unnoticed.
Only the parallel_io branch of s_write_amr_restart writes the negative rank count that marks format v2; the serial branch still writes a positive num_procs to p_all/p<rank>/<t_step>/amr_fine.dat. The guard added to the serial reader could therefore never fire, and its comment claimed a failure mode that path does not have. Verified against a real 8-rank AMR run: the parallel file's header is (-8, 224, 6), so the parallel guard is the one that matters.
The simulation path raises MFCException on a non-zero return code; the --test-all post_process path captured stdout and ignored the result entirely (common.system runs with check=False). post_process could abort, segfault or fail outright and the test still reported PASS as long as the simulation goldens matched. That is how a total break of the AMR post-process reader shipped unnoticed: --test-all ran post_process on every AMR case and threw the answer away. Verified both directions: an AMR test now fails where it previously passed, and non-AMR cases are unaffected.
Two defects, both in the AMR fine-block reader, both invisible until the test suite started checking post_process's exit code. First: format v2 was never taught to post. The writer marks v2 with a negative rank count, the simulation reader handles both formats, and post did not, so its rank-count check fired unconditionally (a negative can never equal num_procs) and blamed a rank-count mismatch on runs that used the right number of ranks. v2 stores one contiguous chunk per block written by that block's single owner, plus a 4-int (owner + 1, m, n, p) record. Ownership now comes from that record rather than from the reader's own geometric intersection, which also fixes a case where a rank overlapping a block it does not own would read a -1 extent and abort. The whole chunk is read and this rank's intersection sub-box copied out, so s_setup_amr_block still reconstructs coordinates from a valid local coarse index. v1 files stay readable. Second: post runs with a LARGER sys_size than the simulation for 5eq Lagrange bubbles, because it appends beta_idx as a post-only output slot. Comparing the file's conserved-variable count against post's inflated sys_size rejected every valid AMR + Lagrange-bubbles file. Size and read by the count the writer actually used.
…oarse RHS between them (ledger 137 experiment, amr_rf_overlap parameter)
…rain it after -- single-node excess 0.63 -> 0.40 s/step (1.11x AMReX), byte-identical; the reflux wait row barely moves and the fine-RHS and coarse rows carry the gain (mechanism not yet named)
…s worth +0.019 s/step (ledger 138 measures ledger 137's gain as ledger 131's EOS fix)
…m both predated the EOS fix, so the 0.63 to 0.40 was ledger 131 expressed wrongly as an excess; the single-node excess stands at 0.583 and item 7 is not met; the reflux post/drain reorder is worth nothing measurable and is reverted
…7 cut (21/51 to 20/32), and what the cut did not buy
…reproduce ledger 117 at 1.92-1.93x on both nodes and disagree with ledger 132 -- the whole 0.583-to-0.696 gap is a level shift in the uniform arm, the term ledger 120 measured as 96 percent of this protocol variance, and about half the ratio move is AMReX own excess falling on an unchanged binary
…ext (per-advanced-cell 2.69x not 1.96x, ledger 120 quote splice, range-vs-sd, start-up hedge, 1.6x-5x noise range, within-window variance)
…w cuts the uniform arm's share of the excess variance from 96 percent to 52 and shows the short window was biased as well as noisy; the excess on the primary deck reads 0.649 and 1.75x AMReX on one node
…ountered outside a parallel region is not deferred on this stack but runs inline, the launch loop returning in 102 percent of the kernel time, so it overlaps 11 to 14 percent of a concurrent exchange and the rescaled prediction falls below the detection threshold
….23x then 1.32x per doubling, 1.28x over the span, against a matched cross-node AMReX bar of 1.23x; and the growth is the base-grid advance and the base halo, not the regrid family the campaign expected
… zero, and two of the three rows are reflux
…the whole buffered box
…eX on one node in one window, the non-case-opt arm reproduces ledger 140's 0.649 beside it, and the excess moves only -0.033 +/- 0.037 because it is a difference and case optimization speeds both of its terms -- the goal's own prediction that it would not be a wash is falsified
… ms per step floor every rank pays, so 40 percent of it is structural serialization rather than skew; the base solve is identical between the arms; and the migration-path store growth this campaign called the largest AMR item is 6.6 ms per step on the deck that owns statement 2, not the 144.3 measured on the rung deck
…model, port its damage flux and solid-partial-density private-list entries into the restructured HLL and HLLC solvers, apply its explicit num_species chemistry bounds, and migrate our AMR and GPU pitfalls into the docs files that replaced the deleted rules page
…und in the restructured Riemann solvers
Lines of Code
|
…urement retractions
…s docs linkcheck)
…s energy The master merge brought upstream's damage model, in which the energy uses the undamaged modulus, and upstream regenerated all six of its own cont_damage goldens in that merge (0BAA2F42 32A3A936 574636EE 5A137315 6CFBCF4A EF3E7C79). D731AB7A is the seventh and is AMR-only, so upstream could not reach it. It is the only test the merge changed: a clean 15-test rerun on the merged tree fails it and otherwise only the ten non-Newtonian tests that also fail on upstream master alone on this machine. The new value matches the candidate the merged code produced on both CPU and GPU (999.0098931133879).
…the merge dropped The merge took MFlowCode#1852's species bound (10 -> 60) everywhere but left U_L/U_R, F_L/F_R/F_star_L/F_star_R/F_HLLC and F_HLL at dimension(20), while the guard it also brought admits sys_size up to AMD_SYS_SIZE_MAX = 70. Any amdflang non-case-optimized run with 20 < sys_size <= 70 overran them silently. The three sites now read exactly as upstream's.
…amage golden; the ten non-Newtonian failures are upstream's on this toolchain
…code span (fixes docs linkcheck)
…p-level rows The RESIDUAL was wall minus the sum of every row, including rows nested inside others, so it read strongly negative (-9.9 % on a run whose true residual is +2.3 %). s_phase_tic now keeps a stack of open brackets and records the shallowest and deepest depth each phase opens at; s_phase_toc counts interleaved and orphan closes. The report appends a tier column (T1, T2, or T1-2 when a phase opens at two depths), sums only rows that were top-level on every rank, and says whether every bracket closed innermost-first. The column is appended, never inserted, so existing positional parsers read the same fields. Verified: AMR goldens F57C3A5B and 09E0D257 pass; the budget is valid on both on CPU and on the amdflang OpenMP-offload build, with an identical tier map on each.
…ment tools were rebuilt and proven able to fail
…generated golden (Phase 0 closed)
…block-shape diversity, so Phase 2 is retargeted to an amr_bat_pad probe
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An opt-in, default-off family of performance features and the measurement infrastructure they rest on. With all flags at their defaults the only touched production path is
s_mpi_decompose_computational_domain, refactored through the newm_boxmodule (byte-identical; covered by the existing suite).m_box(partition arithmetic),m_load_weight/load_weight_wrt(per-cell load-weight field + imbalance metric),m_sfc_partition/sfc_partition_wrt(Morton-SFC predicted-imbalance diagnostic),m_load_balance/load_balance(weighted static decomposition at init; AMR-fine-work-aware),m_rank_timing/rank_time_wrt(per-rank compute-time diagnostic).m_active_box/active_box: restricts reconstruction/Riemann/RK windows to a light-cone-grown box around non-ambient flow; strict-subset golden-tested.hybrid_wenoandhybrid_riemann(+hybrid_weno_eps,hybrid_smooth_flux): linear-optimal weights / central-or-Rusanov flux in smooth cells, full WENO/HLLC at flagged discontinuities (Jameson sensor, stencil-dilated, per-level under AMR).m_amr+m_amr_registers: two-level 2:1 refined block hierarchy; conservative restriction and conservative-linear prolongation with physics-specific closures; per-stage flux registers with Berger–Colella refluxing; Berger–Rigoutsos multi-block dynamic regrid; optional dt/2 subcycling; multi-rank (single-owner blocks assigned by Morton-SFC work balancing at each regrid, with migration; blocks may span rank seams via P2P coarse↔fine gather/scatter; same-level seam halo; distributed registers); restart (both IO modes, regridded-layout persistence); AMR-aware post-processing (fine blocks visualizable as Silo overlay domains); GPU-resident fine level on both OpenACC and OpenMP offload.Full algorithm and user documentation:
docs/documentation/amr.md(support matrix enforced at runtime by the checker — unsupported combinations abort with named messages, never silently).AMR physics support matrix (abridged; authoritative table in amr.md)
Supported and golden-tested: single- and multi-fluid (5-eq,
mpp_lim) · 6-eq with per-block pressure relaxation · viscous (refluxed) · phase change (relax) · chemistry incl. species diffusion · Euler–Euler bubbles (polytropic/non-polytropic, mono/polydisperse, QBMM incl. non-polytropic with per-blockpb/mvside-state; dynamic regrid + subcycle) · acoustic sources (coarse-grid support with regrid exclusion) · immersed boundaries (multi-body, static or prescribed-motion, incl. dynamic regrid with body-containment expansion and per-substage guards) · 2D axisymmetric (per-block WENO-coefficient recompute) · stretched grids (exact parent-bisection ghost coordinates + per-swap coefficient recompute) · hybrid WENO/Riemann sensors (per-level) · Lagrangian bubbles (cloud excluded from blocks; two-way coupling on the coarse grid; regrid clips around the moving cloud) ·active_box(blocks contained in the growing window; agrees with plain AMR to ~1e-14) · IGR (restriction-only coupling: fine sigma solve seeded/Dirichlet-bounded by the coarse solve; documented truncation-order seam, exact free-stream) · 1D MHD/RMHD (div(B)=0 by construction in 1D; HLL and HLLD, incl. relativistic).Gated with named aborts (documented rationale): surface tension (seam force imbalance is structural — three fixes attempted and diagnosed in amr.md) · 2D/3D MHD (attempted and measured: the c/f seam is a continuous O(1) div(B) source GLM cleaning cannot remove — needs constrained-transport-class B prolongation/reflux) · hyperelasticity · 3D cylindrical (global azimuthal filter) · force-driven IB (
moving_ibm=2) · STL bodies · Riemann-extrapolation BCs (bc=-4) ·amr_subcycleunder IGR · stretched grids with Lagrangian/IB-regrid (uniform-spacing index conversions).Validation evidence
Performance
Measured on AMD MI210 (gfx90a), OpenMP offload, one MPI rank per GPU, double precision throughout — MFC and the reference are compared at the same precision. Every figure is a per-step wall differenced between two from-scratch runs of different length, so start-up and mesh ramp-in cancel, and is the mean of three repetitions with the two codes interleaved inside one node reservation.
Reference. AMReX's
Tests/GPU/CNSblob test, rebuilt withNUM_GROW = 4to match MFC's WENO5 ghost width, on a deck matched to MFC's: unit periodic box, entropy blob advected at u=1, gamma 1.4, inviscid, 400^3 base,max_level = 2,ref_ratio = 2,regrid_int = 20, no subcycling, reflux on, density-gradient tagging thresholded to tag the same cells.AMR overhead on the matched deck (8 GPUs, 400^3 base + 2 levels)
The two codes do not run the same numerics, so the comparable quantity is each code's excess over its own uniform-grid arm at the same advanced-cell count:
MFC pays 1.71x AMReX's AMR overhead on this deck. AMR does not slow the base solve: the AMR and uniform arms agree on the base-solve phases to -6.9 ms/step, inside noise.
Where MFC's 627 ms/step goes: 521 ms is AMR machinery, of which 352 ms (56%) is MPI wait — but the minimum over ranks is 251 ms, so most of that is a genuine communication floor, not load imbalance. Largest phases: reflux 116, coarse/fine gather 114, restriction 107 ms/step. Reducing that wait is the active work; it is not a correctness or an API concern for this PR.
Scaling
Weak ladder, fixed work per rank, 8 -> 48 GPUs (multi-node), on the lower-order scaling deck with the same geometry and refinement settings:
a fitted 1.222x per rank-doubling, against 1.234x for AMReX on the matched ladder. The scaling slope is at parity — the gap above is a constant factor, not a worsening one.
Notes for anyone reproducing these
--case-optimization. On this deck it is worth a paired -12.4% on the AMR step (-0.581 s/step, sd 0.084, n=3), 78% of it in the RHS. The excess figures above are case-optimized; without it the same comparison gives 1.80x rather than 1.71x, since case optimization largely cancels in the difference.Known issues (all non-gating or in progress)
continue-on-error): an intermittent post-detected NaN on the two Lagrangian+AMR goldens. Exhaustively unreproducible off GitHub's runners — the exact failing stack (NVHPC 24.3 SDK,-tp=px -Kieee, HPC-X MPI, and the CI docker image itself under apptainer) passes elsewhere, as do native/zen2 builds; 24.5+ green. Documented at the golden definitions.Review guide
The commit history is arc-ordered (active-box → load-weight → SFC → weighted decomposition → rank timing → hybrid → m_box → AMR rungs → physics envelope → CI/GPU hardening); reviewing by arc is much easier than by file. The AMR arc builds stepwise: static hierarchy → restriction/prolongation → fine advance → refluxing → regrid → subcycling → multi-rank → GPU → each physics rung with its own validation. Commit messages carry the validation evidence for their change (measured defects, golden UUIDs, repro details for CI fixes).
All parameters ship default-off with
case_validatorentries, runtime checker gates, andcase.md/amr.mddocumentation.