diff --git a/AMR_AMD_FLANG_NOTES.md b/AMR_AMD_FLANG_NOTES.md deleted file mode 100644 index f522c2a5b1..0000000000 --- a/AMR_AMD_FLANG_NOTES.md +++ /dev/null @@ -1,101 +0,0 @@ -# AMR on AMD flang (OpenMP target offload) — root cause and fix - -**Status: RESOLVED** (2026-07-14, verified on AMD MI250X / gfx90a, ROCm flang, OpenMP offload). -The block-structured AMR GPU path now runs correctly on the AMD flang OpenMP-target-offload -backend, alongside the pre-existing NVHPC (OpenACC / OpenMP) and Cray CCE (OpenMP) support. - -This note supersedes the original hypothesis (which pinned the failure on the empty AMD -`defaultmap` in `omp_macros.fpp`). Live reproduction on an MI250X **refuted** that: see below. - -## Symptom (AMD `gpu-omp`, np ≥ 2, 2D+) -Four AMR tests failed only on the AMD offload lane, passing on every other backend: -- **Self-abort** — `244B1E42` (2D single-level subcycle np=2), `ADA042A2` (2D multi-level static - np=2): NaN appears at a seam-region cell after a few steps and the NaN guard calls `MPI_Abort`. -- **Tolerance drift** — 2D dynamic-regrid cases (`660FFBFE` IGR, `B7704247` stretched grid): the - field diverges beyond golden tolerance. - -## Root cause (confirmed by isolation) -The empty AMD `defaultmap` and the Cray-only `ACC_SETUP_SFs`/`ACC_TEARDOWN_SFs` are **not** the -cause. Isolation on the MI250X shows the failure is **2D-and-higher, np ≥ 2 only**: -- `21C71558` (1D static np=1) — passes. -- `5EFB3277` (1D dynamic regrid np=2) — passes. -- `244B1E42` (2D subcycle np=2) — NaN → abort. - -A backend-wide defaultmap/descriptor problem would break 1D and np=1 too; those pass. The one -code path that is 2D-and-higher-np≥2-specific is the **fine-fine seam halo** -`s_amr_fine_slice` (`src/simulation/m_amr.fpp`). It packed/unpacked on the host and moved the -near-seam slab with a `target update` map clause of a **strided array section of a doubly-nested -derived-type member**: - -``` -$:GPU_UPDATE(host='[amr_slots(slot)%q_cons(i)%sf(dlo:dhi, 0:fm(2), 0:fm(3))]') -``` - -flang miscomputes the offset/stride of that strided section (seam dim `d < num_dims`) in the map -clause. In 1D the section is contiguous, so flang gets it right — which is exactly why 1D np≥2 -passes and 2D+ np≥2 does not. `LIBOMPTARGET_INFO=-1` on a failing run showed **no** mapping error -(no present-table miss): the update silently moves the wrong bytes, corrupting the seam ghosts → NaN. - -The base-grid MPI halo (`s_mpi_sendrecv_variables_buffers`) never hits this: it **device-packs** -into a contiguous buffer and only ever transfers the contiguous buffer. - -## Fix -Rewrite `s_amr_fine_slice` to mirror the base-grid halo — pack/unpack on the **device** straight -into the contiguous buffer, and move only that contiguous buffer host↔device (per-kernel -`copyin`/`copyout`, no strided map-clause section). Two consequences that the live run surfaced and -that the fix accounts for: -1. **`buf` must be mapped.** It is not `declare target`, and AMD's `default='present'` expands to an - empty defaultmap, so the pack/unpack kernels map it explicitly (`copyout` on pack, `copyin` on - unpack). Without this the kernel faults on a null `buf` device pointer. -2. **`amr_slots` is not `GPU_DECLARE`d.** Indexing the module array `amr_slots(slot)%q_cons(i)%sf` - *inside a kernel* dereferences a null outer descriptor. The slot's `q_cons` is passed in as a - `scalar_field` array argument and the kernel indexes the dummy — the same pattern every other AMR - kernel already uses (e.g. `s_amr_fill_fine_ghosts`). - -The change is numerically identical (same pack/unpack ordering and casts), so goldens are unchanged -on all backends; only the transfer mechanism differs. - -## Verification (AMD MI250X, OpenMP offload, np=2) -All four previously-failing cases pass, no regression on the passing cases: -`244B1E42`, `ADA042A2`, `660FFBFE`, `B7704247` (fixed); `21C71558`, `5EFB3277` (still pass). - -## Second, separate bug: AMR+IB fine-marker swap/restore hangs the offload runtime - -Found while sweeping the suite. **Symptom:** AMR+IB tests hang (99% host CPU, GPU idle, no timestep -output) inside `s_ibm_swap_to_fine` / `s_ibm_restore_from_fine` (`m_ibm.fpp`), the AMR-fine IB -ghost-point park/pull kernels. `rocgdb` on the live hang shows the AMD offload runtime busy-looping in -a recursive `targetDataBegin → targetDataMapper → targetDataBegin` (cycling through `free`, -`SourceInfo::getSubstring`, `std::string::find`). - -**Root cause:** those kernels access the device-resident `allocatable` derived-type arrays -`ghost_points` / `gp_park` with only `default='present'` — which on AMD emits **no** defaultmap, so -flang generates a map *entry* for them and lowers it to a per-element custom mapper (the same amdflang -failure the code already dodged for the whole-array `ghost_points` `GPU_UPDATE`). The runtime then -busy-loops processing that entry. Both an implicit `tofrom` map and an explicit `map(present,alloc:…)` -still create the entry and still hang; only **no entry at all** avoids it. - -**Fix:** add an AMD-only `defaultmap(present:allocatable)` (via `extraOmpArgs`, gated on -`MFC_COMPILER == "LLVMFlang"`) to the four swap/restore kernels — asserts them present with no map -entry, so no mapper is generated. CCE already gets this via its `default='present'`; NVHPC/CPU -unchanged. `05A8C23C` is the only *multi-level* AMR+IB test (largest `gp_park`), which is why it -surfaced first; the fix is on the shared kernels so it covers all AMR+IB cases. - -**Not minimally reproducible:** a standalone kernel implicitly mapping the same flat derived-type array -completes instantly even at MFC's element count and present-table size — the per-element-mapper path -only triggers in MFC's full offload context. Worth a flang/ROCm report with the `rocgdb` backtraces. - -**Verification:** all AMR+IB tests pass on the MI250X: `05A8C23C` (multi-level, was hanging), -`2854A102`, `F980C769`, `7FC2F9F8`, `13945217`, `43AF9F25`. - -## Reproduce / debug env -``` -source ./mfc.sh load -c amdfund -m g # must be chained (&&) with the command below in ONE shell -export OMP_TARGET_OFFLOAD=MANDATORY # abort if offload cannot reach a GPU -export OFFLOAD_TRACK_NUM_KERNEL_LAUNCH_TRACES=8 # name the faulting kernel on a GPU memory fault -export LIBOMPTARGET_INFO=-1 # full host<->device mapping trace (very verbose) -./mfc.sh test --only 244B1E42 -- -b mpirun -n 2 -``` -For a *hang* (no error text), attach to the spinning rank to see where the runtime is stuck: -``` -rocgdb -p $(pgrep -n simulation) -batch -ex 'bt 12' -``` diff --git a/docs/documentation/amr.md b/docs/documentation/amr.md index fdaacca5e9..90996f7308 100644 --- a/docs/documentation/amr.md +++ b/docs/documentation/amr.md @@ -10,9 +10,9 @@ Block-structured adaptive mesh refinement (AMR) concentrates resolution where the flow demands it — around shocks, interfaces, and bubble clouds — while leaving the rest of the -domain at the coarser base-grid resolution. MFC implements a two-level hierarchy: the -unmodified base (level-0) solve runs as usual, and one or more 2:1 refined rectangular -blocks advance alongside it on a finer grid. +domain at the coarser base-grid resolution. MFC implements a multi-level block hierarchy: the +unmodified base (level-0) solve runs as usual, and one or more refined rectangular blocks advance +alongside it on a finer grid — nested recursively to `amr_max_level` levels when enabled. The fine blocks are dynamically repositioned every `amr_regrid_int` coarse steps using a gradient-based cell tagger and Berger–Rigoutsos block clustering, so they follow moving @@ -27,24 +27,29 @@ mid-run. ## The Block-Structured Model {#amr-model} -The hierarchy has exactly two levels: +The hierarchy spans levels `0` through `amr_max_level` (default `1`, i.e. two levels): - **Level 0** — the base grid with cell spacing `dx`, `dy`, `dz`. The ordinary MFC solver advances this level every step. - **Level 1** — a list of up to `amr_max_blocks` rectangular refined blocks, each covering - a sub-region of the level-0 domain at 2:1 refinement (half the cell spacing in every - active direction). + a sub-region of the level-0 domain at `ref_ratio`:1 refinement (`ref_ratio` = 2 or 4; + the cell spacing shrinks by `ref_ratio` in every active direction). +- **Levels 2 … `amr_max_level`** — when `amr_max_level > 1`, blocks nest recursively: a + level-`l` block refines a region of its parent level-(`l-1`) block by a further + `ref_ratio`, tracking a moving feature to arbitrary depth. Multi-level nesting requires + `ref_ratio = 2`. See @ref amr_multilevel for the nesting and reflux details. Each block is described by its bounding box in level-0 cell-index space -(`amr_block_beg(1:num_dims)` to `amr_block_end(1:num_dims)`). The fine-block extents must -satisfy: +(`amr_block_beg(1:num_dims)` to `amr_block_end(1:num_dims)`). The initial (level-1) +fine-block extents must satisfy: ``` -2*(amr_block_end(i) - amr_block_beg(i) + 1) - 1 <= N_i +ref_ratio*(amr_block_end(i) - amr_block_beg(i) + 1) - 1 <= N_i ``` where `N_i` is the global cell count in direction `i`. This ensures the fine scratch -(which is sized to the base grid) is never overflowed. +(which is sized to the base grid) is never overflowed. A level-`l` block's fine extent +grows as `ref_ratio**l`, so the nested boxes are sized accordingly. **Fixed-slot storage.** All block slots are pre-allocated at init to the maximum possible block size (half the per-rank subdomain in each dimension). Setting `amr_max_blocks = N` @@ -225,7 +230,6 @@ a diagnostic message for unsupported combinations. | Grid stretching (`stretch_x[y,z] = T`) | Supported | Fine ghost-shell coordinates extend by exact parent-cell bisection, and the spacing-dependent WENO coefficients are recomputed for the active grid on every block swap/restore (`amr_weno_coef_recompute`, armed automatically when the grid is nonuniform); prolongation stays conservative but its slope estimate is first-order on nonuniform parents. Stretched grids do NOT combine with Lagrangian bubbles or dynamic regrid with immersed bodies (their position-to-cell-index conversions assume uniform spacing; init abort) | | Riemann-extrapolation BCs (`bc = -4`) | **Not supported** | Boundary-adjusted WENO coefficient rows cannot be inherited by interior blocks (checker gate) | | `active_box` | Supported (single-rank, per active_box's own MPI gate) | Blocks must sit strictly inside the monotonically-growing active window (init abort + regrid clamp: the windowed coarse update would drop reflux corrections at faces outside it); the fine advance disables the coarse-indexed windowing and treats its whole block as active; the frozen exterior is valid ambient data for ghost prolongation | -| `hybrid_weno` / `hybrid_riemann` | Supported | The sensor arrays are sized to the coarse ghost-inclusive bounds (the fine extent guard keeps fine indices inside them) and the sensor is recomputed from the live bounds every RHS call, so each level evaluates its own sensor; conservative full-WENO defaults at buffer edges | | `acoustic_source` | Supported | The source acts on the coarse grid only: its support must not overlap the initial block (startup abort), and dynamic regrid keeps its boxes clear of the support (tags suppressed, candidate boxes clipped); emitted waves enter blocks through the coarse/fine coupling | **Mandatory solver settings.** @@ -252,6 +256,8 @@ default values, and cross-parameter constraints see @ref case section 7.1. | `amr_buf` | Integer | 3 | Coarse-cell padding around tagged cells; required `>= 1` when `amr_regrid_int > 0` | | `amr_subcycle` | Logical | F | Advance fine level at `dt/2` (two substeps per coarse step) with Berger-Colella refluxing | | `amr_max_blocks` | Integer | 4 | Number of fixed refined-block slots preallocated; each slot is max-block sized (~N times device memory for N slots) | +| `amr_max_level` | Integer | 1 | Maximum refinement depth: `1` = single refined level, `> 1` = recursive multi-level nesting (needs `amr_max_blocks >= 2` and `ref_ratio = 2`). See @ref amr_multilevel | +| `ref_ratio` | Integer | 2 | Cell-refinement ratio between adjacent levels; must be 2 or 4. `ref_ratio = 4` is single-level only (no nesting, no subcycling) | | `amr_cluster_eff` | Real | 0.7 | Berger-Rigoutsos min tag efficiency a clustered box reaches before splitting stops; must satisfy `0 < amr_cluster_eff <= 1` | --- @@ -299,13 +305,15 @@ For multi-fluid (5-equation), additionally set: - **Fixed-slot memory.** All `amr_max_blocks` slots are allocated at init at maximum size. More blocks = more device memory. Compact per-block memory pools are future work. -- **Same-rank-count restart.** The AMR restart file encodes the block geometry and fine - solution per rank. Restarting with a different `num_procs` is not supported and aborts - with a clear message; np-flexible restart is future work. +- **Restart across rank counts.** `parallel_io` restart repartitions the fine blocks + across any `num_procs` (each block is one contiguous region under whole-block ownership). + The serial (per-rank-file) restart path requires the same `num_procs` and aborts with a + clear message otherwise. - **Half-subdomain limit.** Each block may span at most half of any rank's local subdomain per dimension, because the fine advance reuses the rank-local solver scratch. -- **Single refinement level.** Only one refined level is supported. Multi-level AMR - (levels 2, 3, ...) is not implemented. +- **Multi-level constraints.** Recursive multi-level nesting (`amr_max_level > 1`) requires + `ref_ratio = 2`; with immersed boundaries it is single-rank only, and a moving body is + not yet supported. `ref_ratio = 4` is single-level only. - **Level-0 output only.** Standard visualization output (HDF5/SILO) is written at level-0 resolution; the restricted fine solution is already folded into the coarse fields over the block region, so existing visualization workflows are unchanged. diff --git a/docs/documentation/case.md b/docs/documentation/case.md index 544dfcc6b4..0f2ee5c47b 100644 --- a/docs/documentation/case.md +++ b/docs/documentation/case.md @@ -687,10 +687,6 @@ To restart the simulation from $k$-th time step, see @ref running "Restarting Ca | `amr_max_level` | Integer | Maximum AMR refinement depth (number of refined levels above L0); must be >= 1 (default 1). Multi-level nesting (>= 2) is supported: static AMR (`amr_regrid_int = 0`) nests up to level 2, dynamic regrid (`amr_regrid_int > 0`) nests deeper (see @ref amr_multilevel) | | `amr_cluster_eff` | Real | Berger-Rigoutsos min tag efficiency a clustered block box reaches before splitting stops; must satisfy 0 < eff <= 1 (default 0.7) | | `ref_ratio` | Integer | AMR refinement ratio between coarse and fine levels; must be 2 or 4 (default 2). Only ref_ratio = 2 is supported with multi-level AMR or subcycling (v1). | -| `hybrid_weno` | Logical | Use linear-optimal reconstruction in smooth cells, full WENO only at flagged discontinuities (requires WENO reconstruction) | -| `hybrid_weno_eps` | Real | Smoothness threshold for hybrid WENO shock flagging; must be > 0 (default 1e-2) | -| `hybrid_riemann` | Logical | Use a cheap central/Rusanov flux in smooth cells, full HLLC only at flagged discontinuities (requires HLLC, 5eq/6eq) | -| `hybrid_smooth_flux` | Integer | Smooth-region flux for hybrid Riemann: 1 = central, 2 = Rusanov (default 2) | | `partition_tile_size` | Integer | Tile side for the SFC partitioner (default 8) | | `alpha_rho_wrt(i)` | Logical | Add the partial density of the fluid $i$ to the database \| | `rho_wrt` | Logical | Add the mixture density to the database | @@ -865,9 +861,7 @@ AMR is incompatible with surface tension, 3D cylindrical coordinates (2D axisymmetric IS supported), 2D/3D MHD (measured: the coarse/fine seam is a continuous div(B) source that GLM cleaning cannot remove; 1D MHD/RMHD IS supported since div(B) = 0 by construction there), hyperelasticity, and Riemann-extrapolation -boundaries (bc = -4). `active_box` is supported (single-rank): blocks must sit strictly inside the growing active window (init abort + regrid clamp), and the fine advance treats its whole block as active. `hybrid_weno`/`hybrid_riemann` are supported: each -level recomputes the smoothness sensor over its own (swapped) bounds every RHS call. -Nonuniform grids ARE supported (grid stretching and the axisymmetric axis half-cell): the fine +boundaries (bc = -4). `active_box` is supported (single-rank): blocks must sit strictly inside the growing active window (init abort + regrid clamp), and the fine advance treats its whole block as active.Nonuniform grids ARE supported (grid stretching and the axisymmetric axis half-cell): the fine ghost-shell coordinates extend by exact parent-cell bisection and the spacing-dependent WENO coefficients are recomputed for the active grid on every block swap/restore, armed automatically when the grid is detected nonuniform at startup. diff --git a/docs/superpowers/plans/2026-07-13-amr-banked-increments.md b/docs/superpowers/plans/2026-07-13-amr-banked-increments.md deleted file mode 100644 index f599443116..0000000000 --- a/docs/superpowers/plans/2026-07-13-amr-banked-increments.md +++ /dev/null @@ -1,129 +0,0 @@ -# Banked AMR increments — design + plan (for fresh-session execution) - -Three independent, conservation-critical AMR follow-ups, deferred from the multi-level-IB -session (2026-07-13) to be executed **fresh, one at a time** (silent-wrong-answer risk + -context fatigue). Each is its own brainstorm-refine → SDD cycle. **Execution order: #29 → -#30a → #30b** (self-contained first; ref_ratio=4 broadest last). Branch `amr-multilevel` -(PR #6). Each currently **fail-closed** in `m_checker.fpp`. - -Ground rules (all three): golden-file validation on CPU **and** GPU (V100, targeted -`--only -- -b mpirun`, NOT the full-suite sbatch which flakes on SIGILL); existing -goldens byte-identical where the feature is inactive; `./mfc.sh format` → build → precheck-on-commit. - ---- - -## #29 — Distributed pb/mv coupling for non-polytropic QBMM at np≥2 - -**Problem.** Non-polytropic QBMM carries a per-cell quadrature side-state `pb` (bubble -pressure) and `mv` (vapor mass), `nnode × nb` per cell. It **evolves cell-locally** (no -face flux), but its AMR coarse↔fine coupling is done LOCALLY, exact only at np=1. At np≥2 -the SFC block owner needn't hold the block's coarse cells, so pb/mv couple to the wrong -rank's coarse side-state = silent wrong answer. Gated: `m_checker.fpp:106` -`@:PROHIBIT(qbmm .and. .not. polytropic .and. num_procs > 1, ...)`. - -**Current local sites (`m_amr.fpp`):** -- Prolong (coarse→fine): `s_amr_prolong_pbmv` (~1884), called ~1347. Reads coarse pb/mv locally. -- Restrict (fine→coarse fold-back): `s_restrict_pbmv` (~1884 def; called 1509/1590). Local; comment 1589 "np>=2 QBMM fold-back is not yet distributed." - -**Approach: mirror the `q_cons` P2P distribution for pb/mv (no reflux — pb/mv aren't fluxed).** -The pattern to copy: `s_amr_gather_coarse_patch` (the coarse-patch P2P gather: -`f_amr_rank_coarse_range` + `MPI_IRECV`/`MPI_ISEND`, owner assembles `amr_cg`) and the -restrict scatter in `s_restrict_fine_to_coarse` (owner restricts, scatters covered coarse -slices to coarse-cell owners). Two differences from q_cons: (a) per-cell payload is -`nnode*nb` (both pb and mv), so buffer sizes scale by that; (b) NO Berger-Colella reflux -(pb/mv have no C/F flux correction — prolong sets the fine ghost/interior, restrict folds -back; that's the whole coupling). - -**Task plan (SDD):** -1. **Distributed pb/mv gather** — assemble the block's coarse pb/mv patch on the owner via - P2P (mirror `s_amr_gather_coarse_patch`; a parallel `amr_cg`-like pb/mv buffer, or extend - the gather to carry pb/mv alongside q_cons). Prolong reads that instead of local coarse. - Gate: np=1 byte-identical (single owner → local copy path unchanged); build. -2. **Distributed pb/mv restrict scatter** — owner restricts fine pb/mv → coarse averages, - scatters covered coarse slices to coarse-cell owners (mirror the q_cons restrict P2P). - Gate: np=1 byte-identical. -3. **Lift the gate + golden** — drop the `num_procs > 1` term from `m_checker.fpp:106`; add a - np=2 non-polytropic-QBMM + AMR golden. Validate: per-node pb/mv moments conserved - (machine-zero for the conservative moments; pb/mv are a side-state so define the gate as - "np=2 trajectory == np=1 trajectory" on a bit-uniform grid, mirroring the q_cons np-cross - check). CPU + GPU. - -**Open questions for the fresh brainstorm:** whether to fold pb/mv into the existing q_cons -gather/scatter buffers (one MPI round) or a separate exchange (simpler, more messages); -device-buffer handling for the larger payload (pb/mv are `pres_field`, GPU_DECLARE'd). -**Validation oracle:** np=2 == np=1 trajectory on a bit-uniform grid (the WENO-table-ulp -finding means non-bit-uniform grids diverge at ulp). - ---- - -## #30a — Lazy owned-only IB marker sizing - -**Problem.** The multi-level-IB increment sized the declare-target `ib_markers` (and the park -slots) to `2**amr_max_level * base_block_extent` — the **global** deepest extent, right at -np=1. At np≥2 (once multi-level IB is un-gated there — a separate future item) that -over-allocates the never-realloc'd device field on every rank to the global deepest, even -ranks owning only shallow/no fine blocks. Task #30's "lazy owned-only sizing" = size the -marker field to the deepest fine block **a given rank actually owns**, allocated lazily. - -**Current state:** `s_ibm_marker_bounds` (m_ibm.fpp, added 2026-07-13) computes the deepest -bound from `amr_max_level` + `amr_block_beg/end` (global). Multi-level IB is currently np=1 -only, so this is not yet a live cost — **#30a is a memory optimization that pairs with -un-gating multi-level IB at np>1** (itself deferred). Low priority until np>1 IB lands. - -**Approach.** Replace the global `2**amr_max_level` bound with the per-rank owned deepest -level. Because `ib_markers` is a device declare-target that must NOT be reallocated after -mapping, "lazy" means: at init, size to the deepest level the rank's INITIAL decomposition -could own; if a later regrid would need deeper, that's the same never-realloc constraint — -so either (a) size to the rank's static owned-region deepest possible, or (b) accept the -global bound at np=1 (status quo) and only optimize once np>1 IB + repartition-on-restart -lands. **Recommend deferring #30a until np>1 multi-level IB exists** — optimizing an -allocation for a configuration the checker doesn't yet admit is speculative. - -**Task plan:** small — parameterize `s_ibm_marker_bounds` by a per-rank owned-level input; -validate memory footprint drops at np>1 with no golden change. Blocked on np>1 IB. - ---- - -## #30b — ref_ratio = 4 - -**Problem.** Refinement ratio is hard-coded 2:1 per level. One 4:1 level reaches the -resolution of two 2:1 levels with one fewer coupling layer. The `2*`/`2**level` extent -factors are threaded through every coupling kernel. - -**Approach.** Introduce a runtime `ref_ratio ∈ {2,4}` (param, default 2 = byte-identical). -De-hardcode the fine-extent, prolong, restrict, halo, and reflux stencils from `2` to -`ref_ratio` (and `2**level` to `ref_ratio**level`). The prolong stencil widens (4:1 injection -+ the multi-fluid/species closure over a 4-wide child block); restrict averages `ref_ratio^d` -children; the fine-fine seam halo and reflux child-face counts scale by `ref_ratio`. - -**Highest-risk feature of the three** — it touches every coupling kernel's stencil, and a -wrong factor conserves-to-machine-zero while being physically wrong. - -**Task plan (SDD):** -1. **Param + gate** — add `ref_ratio` (definitions.py + descriptions.py + checker default 2); - gate `ref_ratio ∉ {2,4}` and any unsupported combos fail-closed. Byte-identical at - ref_ratio=2. `amr_slots(:)%ref_ratio` already exists (per-block) — audit that it's - populated from the param, not a `2` literal. -2. **De-hardcode extents** — sweep `2*(...)`/`2**level` → `ref_ratio*(...)`/`ref_ratio**level` - in fine-geometry sizing, marker sizing, `old_ext`, tower weight, `s_amr_fine_fine_halo` - (the level-aware `2**level` from #35), reflux child-face counts. Gate: ref_ratio=2 - byte-identical (every site reduces to `2`). -3. **Prolong/restrict stencils** — generalize `s_prolong_one_var` (injection/interp for a - `ref_ratio`-wide child), `s_restrict_one_var` (`ref_ratio^d` child sum), and the alpha/ - species closures. Gate: ref_ratio=2 byte-identical. -4. **ref_ratio=4 golden** — a single-level ref_ratio=4 case; validate conservation - machine-zero (inviscid) + a resolution check vs two 2:1 levels. CPU + GPU. - -**Open questions:** buff_size / stencil reach for a 4-wide prolong (does the coarse patch -margin `amr_cpat_mar` suffice?); interaction with multi-level (ref_ratio=4 AND amr_max_level>1 -= 16:1 two levels — likely gate to one at a time first); whether `ref_ratio` is global or -per-level. - ---- - -## Cross-cutting notes -- All three keep existing goldens byte-identical when inactive (param defaults / gated). -- GPU validate on V100 via targeted `-- -b mpirun` (the full-suite sbatch SIGILL-flakes on - arch-mismatched nodes; see the `phoenix-srun-mpirun` memory + the hpcx-bin-on-PATH note). -- Conservation gates: q_cons machine-zero; pb/mv and any non-conservative side-state use the - "np=2 == np=1 trajectory on a bit-uniform grid" oracle. diff --git a/docs/superpowers/plans/2026-07-13-amr-qbmm-pbmv-np2.md b/docs/superpowers/plans/2026-07-13-amr-qbmm-pbmv-np2.md deleted file mode 100644 index dee049fafb..0000000000 --- a/docs/superpowers/plans/2026-07-13-amr-qbmm-pbmv-np2.md +++ /dev/null @@ -1,158 +0,0 @@ -# Distributed pb/mv coupling for non-polytropic QBMM at np≥2 — Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Make the non-polytropic QBMM pb/mv quadrature side-state couple correctly across MPI ranks under AMR, so `qbmm .and. .not. polytropic` runs on more than one rank (currently fail-closed in the checker). - -**Architecture:** pb/mv (bubble pressure, vapor mass; `nnode × nb` per cell) evolve cell-locally — no face flux, therefore **no Berger-Colella reflux**. Their only coarse↔fine coupling is prolong (coarse→fine ghosts+interior) and restrict (fine→coarse fold-back). Today both read/write the block **owner's LOCAL** coarse pb/mv, which is exact only at np=1. This plan mirrors the proven q_cons P2P distribution for pb/mv as a **separate exchange** (q_cons buffers untouched → q_cons stays byte-identical by construction): a gathered coarse pb/mv patch (`amr_cg_pb`/`amr_cg_mv`, the analogue of `amr_cg`) that the prolong/ghost-fill read, and a restrict scatter that folds fine averages back to the coarse-cell owners. - -**Tech Stack:** Fortran 2008 + Fypp, OpenACC/OpenMP-offload GPU, MPI P2P (`MPI_ISEND`/`MPI_IRECV`), MFC AMR module `src/simulation/m_amr.fpp`. - -## Global Constraints - -- **Scope = single-level np≥2 only.** Mirror `s_amr_gather_coarse_patch` (the L0↔L1 gather), NOT the parent gather. Multi-level QBMM np≥2 (`amr_max_level>1`) stays fail-closed. -- **np=1 gate goldens (Tasks 1-3):** `BCBA6E74` (AMR → 1D → bubbles QBMM → nonpolytropic) is the primary gate; `B0A5D230` (same + regrid subcycle) covers the subcycle ghost-fill path. Run both at np=1 after each task. -- **np=1 byte-identical.** Every task's np=1 path must reduce to the current local read/write (single owner holds every covered/coarse cell). Existing goldens unchanged. This is the primary per-task gate. -- **No reflux for pb/mv.** pb/mv have no flux; do NOT add any reflux/flux-correction. Prolong sets fine, restrict folds back — that is the entire coupling. -- **GPU macros only** (`GPU_*` Fypp), precision via `wp`/`stp`, abort via `s_mpi_abort`/`@:PROHIBIT`, `@:ALLOCATE`↔`@:DEALLOCATE` paired. MPI type `mpi_p` ↔ `wp` on the wire (messages carry `wp`, cast to `stp` on unpack — mirror q_cons at `m_amr.fpp:615`, `:1575`). -- **Validation oracle for pb/mv** (a non-conservative side-state): **np=2 trajectory == np=1 trajectory on a bit-uniform grid** — the same np-cross check q_cons uses. Non-bit-uniform grids diverge at ulp (WENO-table finding), so the golden grid must be bit-uniform. -- Reference frames (memorize both): - - **Local coarse frame:** `ci = amr_isect_lo(d) + f/rr - start_idx(d)` (what pb/mv reads TODAY — owner's own coarse array). - - **Patch-local frame:** `ci = amr_isect_lo(d) + f/rr - amr_cpat_off(d)` (what q_cons prolong reads from `amr_cg` — the gathered patch; see `s_prolong_one_var` `m_amr.fpp:1149-1159`). - The whole change is: read/write the gathered patch in the patch-local frame instead of the local coarse array. -- Workflow: `./mfc.sh format` → build (`-t simulation --gpu acc` for GPU tasks) → precheck-on-commit (never `--no-verify`) → targeted golden. GPU validate on V100 via `--only -- -b mpirun` (NOT full-suite sbatch — it SIGILL-flakes on arch-mismatched nodes; see `phoenix-srun-mpirun` memory, prepend hpcx bin to PATH after `source ./mfc.sh load -c p -m g`). - ---- - -## Task 1: Gathered coarse pb/mv patch + distributed prolong (init/regrid path) - -**Files:** -- Modify: `src/simulation/m_amr.fpp` — module state (near `amr_cg` decl ~line 153), allocation (~368-375), finalize dealloc, new gather routine (after `s_amr_gather_coarse_patch` ~651), `s_amr_prolong_pbmv` (~1887), its callers (1340 init, 4316/4355/4799 regrid). -- Test: existing QBMM+AMR golden (find via `./mfc.sh test -l | grep -i qbmm` — the non-polytropic AMR case) run at np=1. - -**Interfaces:** -- Produces: `s_amr_gather_coarse_patch_pbmv(pb_coarse, mv_coarse, pull_host)` — gathers the current block's coarse pb/mv patch into module arrays `amr_cg_pb`/`amr_cg_mv` (block-local/patch frame, cell 0 = global `amr_cpat_off(d)`). `pull_host=.false.` ⇒ host copy current (init/regrid host prolong); `.true.` ⇒ device current (runtime). Signature/semantics mirror `s_amr_gather_coarse_patch(q_coarse, pull_host)`. -- Consumes: `f_amr_rank_coarse_range`, `amr_cpat_off`, `amr_cpat_hi`, `amr_isect_lo`, `amr_slots(amr_cur)`, `nnode`, `nb` (all existing). - -- [ ] **Step 1: Add module state.** After the `amr_cg` declaration block (~line 153-156), add: -```fortran - !! Gathered coarse pb/mv patch for non-polytropic QBMM (analogue of amr_cg): the block's coarse-side pb/mv side-state, - !! P2P-gathered from the coarse-cell owners into the block owner in the amr_cg patch-local frame (cell 0 == amr_cpat_off). - !! Read by the pb/mv prolong + ghost-fill so np>=2 couples to the correct coarse rank. Allocated only for non-polytropic QBMM. - real(stp), allocatable, dimension(:,:,:,:,:) :: amr_cg_pb, amr_cg_mv - $:GPU_DECLARE(create='[amr_cg_pb, amr_cg_mv]') -``` - -- [ ] **Step 2: Allocate/map** next to `amr_cg` (after line 375), gated on non-polytropic QBMM, sized to the same patch footprint as `amr_cg` plus the pb/mv `(nnode, nb)` trailing dims: -```fortran - if (qbmm .and. .not. polytropic) then - @:ALLOCATE(amr_cg_pb(0:amr_cpat_hi(1), 0:amr_cpat_hi(2), 0:amr_cpat_hi(3), 1:nnode, 1:nb)) - @:ALLOCATE(amr_cg_mv(0:amr_cpat_hi(1), 0:amr_cpat_hi(2), 0:amr_cpat_hi(3), 1:nnode, 1:nb)) - amr_cg_pb = 0._stp; amr_cg_mv = 0._stp - end if -``` -NOTE for implementer: `amr_cg_pb`/`amr_cg_mv` are plain 5D arrays, NOT `scalar_field`s. Follow the **`amr_rhs_pb_f` idiom EXACTLY** (`m_amr.fpp:115-116` module `GPU_DECLARE(create=...)` + `m_amr.fpp:349` `@:ALLOCATE`): the module `GPU_DECLARE` (Step 1) + `@:ALLOCATE` handle device mapping. Do NOT use `@:ACC_SETUP_SFs` (that is only for `scalar_field` `%sf` pointers, e.g. `amr_cg(i)`). Do NOT invent a scalar_field wrapper. Note the payload is `stp` (matches `pb_f%sf`/`pb_ts`), whereas `amr_rhs_pb_f` is `wp` — use `real(stp)` for `amr_cg_pb`/`amr_cg_mv`. - -- [ ] **Step 3: Finalize dealloc.** Where `amr_rhs_pb_f`/`amr_cg` are deallocated in `s_finalize_amr_module`, add the paired `@:DEALLOCATE(amr_cg_pb)` / `@:DEALLOCATE(amr_cg_mv)` under the same `qbmm .and. .not. polytropic` guard (grep the finalize routine for `amr_rhs_pb_f` to co-locate). - -- [ ] **Step 4: Write the gather routine.** Add `s_amr_gather_coarse_patch_pbmv(pb_coarse, mv_coarse, pull_host)` after `s_amr_gather_coarse_patch` (~651). Structure it as a **verbatim structural mirror** of `s_amr_gather_coarse_patch` (read `m_amr.fpp:505-651` first), with these payload differences: - - Dummies: `real(stp), dimension(idwbuff(1)%beg:,idwbuff(2)%beg:,idwbuff(3)%beg:,1:,1:), intent(in) :: pb_coarse, mv_coarse` (the coarse-level pb/mv, e.g. `pb_ts(1)%sf`). - - Assert single-level: this routine is single-level only. Early-guard `if (amr_block_level(amr_cur) >= 2) return` — multi-level QBMM np≥2 is gated in the checker; the routine must never be reached at level≥2 (Task 4 gate enforces it). - - Per-cell payload = `2*nnode*nb` (pb then mv). Buffer sizes: replace `sys_size` with `2*nnode*nb` in `boxsz`/`maxsz`; pack loop order `do ib_=1,nb; do q=1,nnode` for pb, then the same for mv (or interleave — keep sender/receiver order identical, as q_cons does over `i=1,sys_size`). - - np=1 device-kernel local-copy path: mirror lines 543-566 (`num_procs == 1` branch) — a `GPU_PARALLEL_LOOP` copying `pb_coarse`→`amr_cg_pb`, `mv_coarse`→`amr_cg_mv` over the in-domain patch, same index map `amr_cg_pb(g1-coff1, g2-coff2, g3-coff3, q, ib_) = pb_coarse(g1-o1, g2-o2, g3-o3, q, ib_)`. Hoist `coff*`/`o*` like the q_cons kernel. Device-current; sync host only when `.not. pull_host` (mirror the `to_host` gate at 625/726). - - np≥2 P2P path: mirror 577-651 — own-slice local unpack + `MPI_IRECV`/`MPI_ISEND` over `f_amr_rank_coarse_range`, `wp` on the wire, cast to `stp` into `amr_cg_pb`/`amr_cg_mv`. `GPU_UPDATE(device=...)` after receive; `GPU_UPDATE(host=...)` before the host-consumer path when `.not. pull_host`. - -- [ ] **Step 5: Rewrite `s_amr_prolong_pbmv`** (1887) to read the gathered patch. Two edits: - - Add a gather call at the top: `call s_amr_gather_coarse_patch_pbmv(pb_ts(1)%sf, mv_ts(1)%sf, .false.)` (host-current: this is a host loop). - - Change the coarse read from local `pb_ts(1)%sf(ci,cj,ck,q,ib_)` to `amr_cg_pb(ci,cj,ck,q,ib_)` and the index origin from `ox = start_idx(1)` (etc.) to `ox = amr_cpat_off(1); oy = amr_cpat_off(2); oz = amr_cpat_off(3)` (patch-local frame, matching `s_prolong_one_var:1151`). Keep `ci = lo1 + fi/rr - ox` shape. - -- [ ] **Step 6: Build** (`./mfc.sh build -t simulation --gpu acc -j 8`). Expected: compiles clean. - -- [ ] **Step 7: np=1 golden byte-identical.** Run the existing non-polytropic-QBMM+AMR golden at np=1 (`./mfc.sh test --only -- -b mpirun`). Expected: PASS unchanged (single owner ⇒ gather's np=1 branch copies local coarse→patch, prolong reads the same values it read before, just via the patch frame). - -- [ ] **Step 8: Commit** `amr(qbmm): gather coarse pb/mv patch + distributed prolong (init/regrid, np=1 byte-identical)`. - ---- - -## Task 2: Distributed pb/mv ghost-fill (runtime advance path) - -**Files:** -- Modify: `src/simulation/m_amr.fpp` — `s_amr_fill_fine_ghosts_pbmv` (~1925) index frame, and its callers (2864, 2959, 2960, 3053) to gather-then-read. -- Test: same np=1 QBMM+AMR golden. - -**Interfaces:** -- Consumes: `s_amr_gather_coarse_patch_pbmv` (Task 1), `amr_cg_pb`/`amr_cg_mv`, `amr_cpat_off`. - -- [ ] **Step 1: Change `s_amr_fill_fine_ghosts_pbmv` index origin** (1925). Its dummies `pb_c`/`mv_c` will now receive `amr_cg_pb`/`amr_cg_mv`. Change the dummy declaration bounds to the patch array bounds `real(stp), dimension(0:,0:,0:,1:,1:), intent(in) :: pb_c, mv_c` (the gathered patch is 0-based; DO NOT keep the `idwbuff(1)%beg:` bounds — the patch has no ghost origin). Change `ox = start_idx(1)` (etc.) to `ox = amr_cpat_off(1); oy = amr_cpat_off(2); oz = amr_cpat_off(3)`. Keep the floor-based `ci = lo1 + floor(fi/rr) - ox` map. - -- [ ] **Step 2: Gather before each runtime ghost-fill.** At each caller, insert a device-current pb/mv gather immediately before the fill, reading the SAME coarse snapshot the paired q_cons gather uses, and pass the patch: - - Line 2864: after the q_cons `s_amr_fill_fine_ghosts(amr_cg, ...)` at 2858, replace `call s_amr_fill_fine_ghosts_pbmv(pb_in, mv_in, ...)` with `call s_amr_gather_coarse_patch_pbmv(pb_in, mv_in, .true.)` then `call s_amr_fill_fine_ghosts_pbmv(amr_cg_pb, amr_cg_mv, ...)`. - - Lines 2959/2960: pair with the q_cons gathers at 2951 (`q_old`) / 2953 (`q_new`) — gather `(pb_old,mv_old,.true.)` before the 2959 fill, `(pb_in,mv_in,.true.)` before the 2960 fill. - - Line 3053 (`s_amr_lerp_fine_ghosts_pbmv`): **CONFIRMED reads pre-filled ghost shells `pb_ghost_a/b`, NOT a coarse array** (verified `m_amr.fpp:3051-3055`). NO new gather here — leave it unchanged. - -- [ ] **Step 3: Build.** Expected: clean. - -- [ ] **Step 4: np=1 golden byte-identical.** Same UUID, `-- -b mpirun`. Expected: PASS unchanged. - -- [ ] **Step 5: Commit** `amr(qbmm): distributed pb/mv ghost-fill on the runtime advance path (np=1 byte-identical)`. - ---- - -## Task 3: Distributed pb/mv restrict scatter (fine→coarse fold-back) - -**Files:** -- Modify: `src/simulation/m_amr.fpp` — replace the two local `s_restrict_pbmv` calls (1509 np=1 branch, 1590 np≥2 branch) with a distributed scatter; add a `s_amr_scatter_pbmv` routine (or extend `s_restrict_pbmv` to scatter). -- Test: same np=1 QBMM+AMR golden. - -**Interfaces:** -- Consumes: `f_amr_rank_interior`, `s_amr_box_isect`, `amr_region_lo_all`/`amr_region_hi_all`, `amr_block_owner`, the restrict child-average arithmetic in `s_restrict_pbmv`. - -- [ ] **Step 1: Read the q_cons scatter** (`s_restrict_fine_to_coarse:1479-1587`) — the owner restricts covered cells locally + sends each other coarse-owner its covered slice; the coarse-owner receives and overwrites local coarse. This is the exact pattern to mirror for pb/mv. - -- [ ] **Step 2: Add `s_amr_scatter_pbmv(pb_coarse, mv_coarse, pb_fin, mv_fin)`** mirroring the q_cons scatter structure but with the pb/mv child-average (from `s_restrict_pbmv:2059-2070`) as the per-cell restrict value and `2*nnode*nb` payload: - - Owner branch: over covered cells `[region_lo:region_hi]`, for cells this rank owns (`f_amr_rank_interior(owner) ∩ region`), overwrite `pb_coarse`/`mv_coarse` locally with the child average (device kernel, mirror `s_amr_restrict_overwrite_device`; or reuse the existing `s_restrict_pbmv` for the owner-local covered box). For every OTHER coarse-owner, pack its covered slice (child averages, `wp`) and `MPI_ISEND`. - - Coarse-owner branch: if it holds covered cells, `MPI_RECV` its slice, cast `wp`→`stp` into local `pb_coarse`/`mv_coarse`, `GPU_UPDATE(device=...)` only the covered slice (mirror 1583-1585 — never whole-array, which clobbers device-advanced non-covered coarse). - - np=1: owner owns every covered cell, sends nothing, overwrites locally ⇒ **identical child-sum to the current `s_restrict_pbmv`** (bit-identical). - -- [ ] **Step 3: Wire it in.** Replace line 1509 (`call s_restrict_pbmv(...)`, np=1 device branch) and line 1590 (np≥2 branch) with `call s_amr_scatter_pbmv(pb_ts(1)%sf, mv_ts(1)%sf, amr_slots(amr_cur)%pb_f%sf, amr_slots(amr_cur)%mv_f%sf)`. Remove the `amr_rank_owns_block` guard from the call site — the scatter must run on ALL ranks (owner sends, coarse-owners receive); do the owner/non-owner split INSIDE the routine (like `s_restrict_fine_to_coarse`, which is called unconditionally and branches on `proc_rank == owner`). Keep the level≥2 early-return upstream (1473-1477) unchanged — multi-level fold-back stays gated. - -- [ ] **Step 4: Build.** Expected: clean. - -- [ ] **Step 5: np=1 golden byte-identical.** Same UUID, `-- -b mpirun`. Expected: PASS unchanged. - -- [ ] **Step 6: Commit** `amr(qbmm): distributed pb/mv restrict scatter (np=1 byte-identical)`. - ---- - -## Task 4: Lift the single-level gate + np=2 golden - -**Files:** -- Modify: `src/simulation/m_checker.fpp` (~106) — narrow the gate to multi-level only. -- Modify: `toolchain/mfc/test/cases.py` — add a np=2 non-polytropic-QBMM + single-level-AMR golden on a bit-uniform grid. -- Test: new golden UUID (CPU + GPU). - -- [ ] **Step 1: Narrow the checker gate** (`m_checker.fpp:106`). Change: -```fortran -@:PROHIBIT(qbmm .and. (.not. polytropic) .and. num_procs > 1, "...") -``` -to gate only the still-unsupported MULTI-level case: -```fortran -@:PROHIBIT(qbmm .and. (.not. polytropic) .and. amr_max_level > 1 .and. num_procs > 1, & - & "amr with non-polytropic QBMM on more than one MPI rank is only supported at amr_max_level = 1: the multi-level (parent-side) pb/mv coarse/fine coupling is not yet distributed. Run multi-level non-polytropic QBMM on a single rank.") -``` -Update the explanatory comment above it (101-105) to say single-level np≥2 is now distributed; multi-level remains TODO. - -- [ ] **Step 2: Add the np=2 golden** in `cases.py`, mirroring the `BCBA6E74` case (trace `"AMR -> 1D -> bubbles nonpolytropic"` region at `cases.py:3440`; find the QBMM-nonpolytropic sibling near it). Requirements: `amr_max_level=1`, run at **np=2** with a domain (raise `m`) that decomposes so the AMR block **straddles the rank boundary** — a coarse cell under the block owned by a DIFFERENT rank than the block owner. This is what exercises the gather/scatter; if the block sits entirely on one rank the P2P path is dead and the test proves nothing. **Bit-uniform grid** (uniform `dx`, no stretching) so the np-cross oracle holds. Distinct trace noting np=2. - -- [ ] **Step 3: Generate + validate on CPU.** Build CPU (`./mfc.sh build -t simulation -j 8`), `./mfc.sh test --generate --only -- -b mpirun`. Then **prove the oracle by hand**: run the same case at np=1 and np=2, confirm the pb/mv moments (and q_cons) match to machine-zero (bit-uniform ⇒ exact). If np=2 ≠ np=1, the scatter/gather is wrong — do NOT accept the golden; return to Task 1-3. - -- [ ] **Step 4: Validate on GPU (V100).** `source ./mfc.sh load -c p -m g`, prepend hpcx bin to PATH, `./mfc.sh build -t simulation --gpu acc -j 8`, `./mfc.sh test --only -- -b mpirun` at np=2. Expected: PASS (GPU np=2 == CPU golden). Also re-run the pre-existing QBMM+AMR np=1 golden on GPU to confirm no regression. - -- [ ] **Step 5: Commit** `amr(qbmm): admit single-level non-polytropic QBMM at np>=2 + np=2 golden`. - ---- - -## Cross-cutting notes -- The three coupling edits (gather+prolong, ghost-fill, scatter) are each independently np=1-byte-identical, so a regression in any one is caught by the existing QBMM+AMR golden before Task 4 ever runs at np≥2. -- If the existing QBMM+AMR golden's block does not straddle a rank boundary at np=2, Task 4's new case MUST arrange straddle — otherwise the np≥2 paths are dead-code-covered and the "distributed" claim is unproven. -- Do NOT touch `s_amr_restrict_to_parent` / `s_amr_gather_from_parent` (multi-level) — out of scope, kept gated. diff --git a/docs/superpowers/plans/2026-07-13-amr-ref-ratio-4.md b/docs/superpowers/plans/2026-07-13-amr-ref-ratio-4.md deleted file mode 100644 index c89145a78f..0000000000 --- a/docs/superpowers/plans/2026-07-13-amr-ref-ratio-4.md +++ /dev/null @@ -1,94 +0,0 @@ -# Runtime ref_ratio ∈ {2,4} for single-level AMR — Implementation Plan (#30b) - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax. This is the HIGHEST-RISK banked AMR increment — a wrong refinement factor conserves-to-machine-zero while being physically misplaced. Every task gates on "existing goldens byte-identical at ref_ratio=2 (the default)". Start FRESH with context headroom. - -**Goal:** Make the AMR refinement ratio a runtime parameter `ref_ratio ∈ {2,4}` (default 2), so one fine level can reach 4:1 resolution instead of 2:1 — de-hardcoding every 2:1 assumption threaded through the coupling kernels. - -**Architecture:** The per-block field `amr_slots%ref_ratio` already exists and is used in ~49 sites, but it is pinned `= 2` at one assignment (`m_amr.fpp:5418`) and many kernels still bake in the literal 2. Drive the field from a new param, then fix every hardcoded-2 site. Restrict to **single-level, global, lock-step** for v1 (see Global Constraints); multi-level 4:1 (16:1 towers) and subcycled 4:1 are deferred follow-ups. - -**Tech Stack:** Fortran 2008 + Fypp, OpenACC/OpenMP-offload GPU, `src/simulation/m_amr.fpp` + `m_amr_registers.fpp`, param system (`toolchain/mfc/params/definitions.py`). - -## Global Constraints -- **Scope = single-level, global, lock-step, {2,4}.** ref_ratio applies to every level (global); v1 gates `ref_ratio ≠ 2 ⇒ amr_max_level = 1 .and. .not. amr_subcycle`. Multi-level 4:1 and subcycled 4:1 are separate follow-ups. -- **Default ref_ratio = 2 ⇒ every changed site reduces to the current literal.** Existing goldens MUST stay byte-identical after EVERY task. This is the primary per-task gate and the safety net for this high-risk work. -- **Prolong offset is conservation-visible.** Unlike most factors (which conserve-but-misplace when wrong), the prolong child-offset must keep `mean over children = 0` or the fine children don't average to the coarse value → conservation breaks. So the ref_ratio=4 golden's conservation check catches an offset bug. Use this. -- MFC discipline: GPU via `GPU_*` Fypp macros only; precision `wp`/`stp`; abort via `s_mpi_abort`/`@:PROHIBIT`; `@:ALLOCATE`↔`@:DEALLOCATE`; new param via `definitions.py` `_r()`+`_nv()` (+ `case_validator.py` if physics-constrained). Build: `./mfc.sh format` → build → precheck-on-commit. -- **Ops (this env):** builds need the Bash sandbox OFF (else SIGTERM/143); CPU mpirun via `source ./mfc.sh load -c p -m c`; GPU via `load -c p -m g` + `clean --gpu acc` before an incremental GPU build (nvlink "newer than"); do NOT run interactive builds while a batch GPU build shares `build/`. Golden validate `-- -b mpirun`. - ---- - -## The hardcoded-2 inventory (exhaustive sweep, 2026-07-13) -The change point is `amr_slots(islot)%ref_ratio = 2` (`m_amr.fpp:5418`) → `= ref_ratio`. Every site below assumes 2:1 and MUST be generalized. `rr` = `amr_slots(amr_cur)%ref_ratio` (or the local `rr`/`ref_ratio` already in scope). - -**Cat 1 — fine-extent sizing (`2*E-1` → `rr*E-1`):** `m_amr.fpp` 265, 267, 268 (`max_f{1,2,3}`), 1289, 1291, 1292 (`%m/%n/%p`), 1599–1601 (L2-fits-parent guard — multi-level, gated, but generalize for correctness). `m_amr_registers.fpp` 147, 149, 150 (freg sizing — MUST match m_amr's max_f). - -**Cat 2 — `2**amr_block_level` → `rr**amr_block_level`:** `m_amr.fpp` 1146–1148 (tower weight), 3210 + 3211–3216 (`fmul` fine-fine halo extent), 4580 + 4581–4583 (`old_ext` regrid stash). - -**Cat 3 — fine→coarse index `/2` and `maxc /2` → `/rr`:** `m_amr.fpp` 244, 246, 247 (`amr_maxc = (glb+1)/2`), 253 (fit cap allreduce), 459 (`c = lo + fi/2` in `s_build_level_coords`), 2609, 2620, 2632, 2643, 2656, 2667 (`floor(jg/2.)` ghost-coord maps in `s_amr_swap_to_fine`), 2757, 2759, 2760 (`floor(j/2.)` in `s_amr_igr_swap_sigma` — IGR only). - -**Cat 4 — prolong child-offset `(mod(f,rr)-0.5)*0.5` → general (see Task 2 for exact form):** `m_amr.fpp` 1363, 1366, 1369 (`s_prolong_one_var`), 1437, 1440, 1443 + 1484, 1487, 1490 (`s_interpolate_coarse_to_fine` alpha + bubble paths), 2871, 2876, 2879 + 2920, 2925, 2928 (`s_amr_fill_fine_ghosts` GPU kernels). **15 sites.** CONSERVATION-CRITICAL. - -**Cat 7 — coarse-patch margin:** `m_amr.fpp` 362 (`amr_cpat_mar = (buff_size+1)/2+1`), 1331 (`nmar`, same). → `(buff_size + rr - 1)/rr + 1` (byte-identical at rr=2). The `2*amr_cpat_mar` two-sided additions (364–366, 542–545, 692–695, 896–899) are rr-INDEPENDENT — leave them. - -**Cat 8 — fine-grid COORDINATE generation (NOT mechanical — redesign):** `m_amr.fpp` 463 (`mod(fi,2)==0` left/right-half branch in `s_build_level_coords`), 2610, 2621, 2633, 2644, 2657, 2668 (`mod(jg,2)` coarse-midpoint-vs-edge branch in `s_amr_swap_to_fine`). For rr=2 a fine boundary is either the coarse midpoint (even) or coarse edge (odd); for rr general there are `rr` uniform sub-boundaries per coarse cell at fractions `k/rr`. The even/odd branch must become a general `k = mod(idx, rr)` fractional placement `xcb(c-1) + (k/rr)*(xcb(c)-xcb(c-1))`. - -**Cat 5/6 — coarse-RHS reflux (`s_amr_apply_reflux`, `m_amr_registers.fpp`):** 549–552, 560–561 (x-faces `nch`/`dd*_hi`/`f*0 = 2*c`), 582–584, 592–593, 596 (y-faces), 614, 622–623, 625–626 (z-faces, `nch=4`, `do dd=0,1`). This routine has its OWN hardcoded-2 loops SEPARATE from the already-general shared kernel `s_amr_reflux_apply_faces` (which takes `rr`). Preferred fix: **route `s_amr_apply_reflux` through the shared kernel** (completes the #39 dedup); fallback: thread `rr` into its loops. - -**Subcycle (DEFERRED — v1 gates lock-step):** 182 (`amr_dt_fine = 0.5*dt`), 3426, 3566 (`do sub = 1, 2`), 3428, 3568 (`th = ...*0.5`), 3620 (`dt_sub*0.5`). Out of v1 scope (gated `.not. amr_subcycle`); a follow-up increment generalizes these to `ref_ratio` substeps. - -**Multi-level inset (OUT of scope — gated):** 1585–1587, 4539–4541 (`/4` L2 placement heuristic). amr_max_level=1 in v1. - -**Confirmed SAFE (do not touch):** the shared reflux kernel `s_amr_reflux_apply_faces` + capture routines (already `rr`-general); `nchild`/child-loops in the restrict + pbmv + tagger paths (already `rr`); Morton `iand` (SFC); the `0.5*(xcb+xcb)` cell-center formulas in coord gen (mesh-spacing, not ratio); the `/(2*r0)` gradient sensor (finite-difference, not ratio); the `2*amr_cpat_mar` two-sided margin adds. - ---- - -## Task 1: Param + gate + mechanical de-hardcode (Cat 1, 2, 3, 7) - -**Files:** `toolchain/mfc/params/definitions.py`, `src/simulation/m_amr.fpp`, `src/simulation/m_amr_registers.fpp`, `src/common/m_derived_types.fpp` (if a scalar decl is needed — likely not, ref_ratio is a scalar namelist var), `src/simulation/m_checker.fpp`, `src/common/m_global_parameters_common.fpp` (default assignment). - -- [ ] **Add the param.** `definitions.py`: `_r('ref_ratio')` + `_nv()` NAMELIST_VARS registration; default 2 (in `s_assign_default_values_to_user_inputs`). Re-run cmake so the Fortran decl + namelist binding are generated. -- [ ] **Checker gate** (`m_checker.fpp`, AMR block): `@:PROHIBIT(ref_ratio /= 2 .and. ref_ratio /= 4, "ref_ratio must be 2 or 4")` and `@:PROHIBIT(ref_ratio /= 2 .and. (amr_max_level > 1 .or. amr_subcycle), "ref_ratio /= 2 is only supported at amr_max_level = 1 without subcycling (v1)")`. -- [ ] **Drive the per-block field:** `m_amr.fpp:5418` `amr_slots(islot)%ref_ratio = 2` → `= ref_ratio`. -- [ ] **Cat 1/2/3/7 swaps** at every line listed above: `2*` → `ref_ratio*`, `2**amr_block_level` → `ref_ratio**amr_block_level`, `/2`/`/2._wp` → `/ref_ratio`/`/real(ref_ratio,wp)`, `(m_glb+1)/2` → `(m_glb+1)/ref_ratio`, margin `(buff_size+1)/2+1` → `(buff_size+ref_ratio-1)/ref_ratio+1`. In `m_amr_registers.fpp` use the module `ref_ratio` (add `use` if needed) or the level-1 value. **Do NOT touch Cat 8 (Task 3), Cat 5/6 reflux (Task 4), or subcycle sites.** -- [ ] **Build (sandbox off) + gate:** existing AMR goldens byte-identical at ref_ratio=2 (default). Run a representative subset (`BCBA6E74`, a core single-level AMR golden, a multi-fluid AMR golden) `-- -b mpirun`. Commit `amr(ref_ratio): param + gate + mechanical de-hardcode of the 2:1 extent/index/margin factors (ref_ratio=2 byte-identical)`. - -## Task 2: Prolong child-offset generalization (Cat 4 — 15 sites) - -**Files:** `src/simulation/m_amr.fpp`. - -Current (rr=2): `xi = (real(mod(f, rr), wp) - 0.5_wp)*0.5_wp` → children at ±0.25. -General: `xi = (real(mod(f, rr), wp) - real(rr - 1, wp)*0.5_wp) / real(rr, wp)`. -Verify byte-identity at rr=2: `(mod-0.5)/2` = `(0-0.5)/2=-0.25`, `(1-0.5)/2=0.25` — bit-identical to the old form (0.5, 0.25 exact in binary). For rr=4: `(0,1,2,3 - 1.5)/4` = `-0.375,-0.125,0.125,0.375` — symmetric (mean 0 ⇒ conservative), correct. - -- [ ] Replace the offset at all 15 sites (1363, 1366, 1369, 1437, 1440, 1443, 1484, 1487, 1490, 2871, 2876, 2879, 2920, 2925, 2928) with the general form, using the `rr`/`ref_ratio` already in scope at each site. The QBMM `inject`/`pw_const` path zeroes the slopes so it is offset-independent — but fix it anyway for the non-inject variables that share the loop. -- [ ] **Build + gate:** existing goldens byte-identical (the algebra is exact at rr=2 — verify with a golden diff, not just PASS). Commit `amr(ref_ratio): generalize prolong child-offset stencil (rr=2 bit-identical, symmetric+conservative at rr=4)`. - -## Task 3: Fine-grid coordinate generation (Cat 8 — redesign) - -**Files:** `src/simulation/m_amr.fpp` — `s_build_level_coords` (~459–472), `s_amr_swap_to_fine` (~2609–2668). - -- [ ] **`s_build_level_coords`:** the `c = lo + fi/2` map is Task 1 (`/ref_ratio`); the `mod(fi,2)` left/right-half branch (463) needs the general sub-boundary placement. A fine cell-boundary index `fi` sits at coarse cell `c = lo + fi/rr`, sub-index `k = mod(fi, rr)`, fractional position `k/rr` within `[xcb(c-1), xcb(c)]`: `x_cb(fi) = xcb(c-1) + (real(k,wp)/real(rr,wp))*(xcb(c) - xcb(c-1))`. Read the full routine and replace the even/odd branch with this. rr=2: k∈{0,1} → fractions 0, 0.5 → coarse edge, coarse midpoint — matches the old branch (verify the index convention). -- [ ] **`s_amr_swap_to_fine` ghost-coord maps (2609–2668):** same generalization for the 6 ghost-extension branches (hi/lo × x/y/z) — replace `mod(jg,2)` midpoint-vs-edge with the `k=mod(jg,rr)` fractional placement. -- [ ] **Build + gate:** existing goldens byte-identical (rr=2 reduces to bisection). This is the trickiest task — diff the `x_cb`/grid output of a golden to confirm bit-identity, not just physics PASS. Commit `amr(ref_ratio): generalize fine-grid coordinate subdivision to rr-way (rr=2 bit-identical)`. - -## Task 4: Coarse-RHS reflux generalization (Cat 5/6) - -**Files:** `src/simulation/m_amr_registers.fpp` — `s_amr_apply_reflux` (~519–639). - -- [ ] Read `s_amr_apply_reflux` AND the shared `s_amr_reflux_apply_faces` (~721–818, already `rr`-general). **Preferred:** refactor `s_amr_apply_reflux` to call the shared kernel (completing the #39 dedup) so there is ONE reflux averaging path. **Fallback if the call shapes don't line up:** thread `rr` into `s_amr_apply_reflux`'s own loops — `nch` products (549–552, 582–584, 614) use `rr`; `f*0 = 2*c` (560–561, 592–593, 622–623) → `rr*c`; `do dd = 0, 1` (596, 625–626) → `do dd = 0, rr-1`. -- [ ] **Build + gate:** existing goldens byte-identical + per-fluid mass conservation ~machine-zero (reflux is conservation-critical). Commit `amr(ref_ratio): generalize coarse-RHS reflux child-averaging to rr (dedup into shared kernel)`. - -## Task 5: ref_ratio=4 golden + validation - -**Files:** `toolchain/mfc/test/cases.py`, new `tests//`. - -- [ ] Add a **single-level, lock-step, ref_ratio=4** golden: a 1D (or 2D) inviscid case with `amr=T, amr_max_level=1, amr_subcycle=F, ref_ratio=4`, a static block over a smooth feature, bit-uniform grid. Mirror an existing single-level AMR golden's params + set `ref_ratio=4`. -- [ ] **Generate + validate CPU:** (a) conservation machine-zero (inviscid, no IB) — the primary gate; a prolong-offset or reflux bug shows here. (b) A **resolution check**: the 4:1 block's fine solution should match a two-level 2:1 tower (amr_max_level=2, ref_ratio=2) to discretization order on the same feature — confirms the 4:1 refinement is physically placed, not just conservative. (c) Optionally an amr_max_level=1 ref_ratio=2 vs a coarser ref_ratio=4-of-half-the-blocks equivalence. -- [ ] **Validate GPU (V100):** `clean --gpu acc` build, run the new golden + a ref_ratio=2 regression at np=1 `-- -b mpirun`. Commit `amr(ref_ratio): single-level ref_ratio=4 golden + conservation/resolution validation`. - ---- - -## Cross-cutting notes -- After EACH task, a golden DIFF (not just PASS) is the byte-identity proof for the rr=2 default — this high-risk work relies on that invariant holding at every step. -- The conservation-visible prolong offset (Task 2) + per-fluid mass check (Task 4) are the two places a silent ratio bug becomes loud — lean on them. -- Deferred follow-ups (own increments): subcycled ref_ratio=4 (subcycle site generalization + 4-substep ghost-lerp/reflux conservation); multi-level ref_ratio=4 (16:1 towers — the `/4` inset heuristics, `rr**level` cumulative factors, fine-fine seam at mixed depth). Both gated closed by Task 1's checker. diff --git a/src/common/m_global_parameters_common.fpp b/src/common/m_global_parameters_common.fpp index 2f87a4913a..e98128a6a7 100644 --- a/src/common/m_global_parameters_common.fpp +++ b/src/common/m_global_parameters_common.fpp @@ -105,7 +105,6 @@ module m_global_parameters_common $:GPU_DECLARE(create='[mapped_weno, wenoz, teno, wenoz_q, mhd, relativity]') $:GPU_DECLARE(create='[igr_iter_solver, igr_order, viscous, igr_pres_lim, igr]') $:GPU_DECLARE(create='[recon_type, muscl_order, muscl_polyn, muscl_lim]') - $:GPU_DECLARE(create='[hybrid_weno, hybrid_riemann, hybrid_weno_eps, hybrid_smooth_flux]') #:endif #endif diff --git a/src/simulation/m_amr.fpp b/src/simulation/m_amr.fpp index 24a54c0448..4221afb850 100644 --- a/src/simulation/m_amr.fpp +++ b/src/simulation/m_amr.fpp @@ -17,7 +17,7 @@ module m_amr use m_global_parameters use m_constants, only: num_fluids_max, model_eqns_6eq, mapCells use m_pressure_relaxation, only: s_pressure_relaxation_procedure - use m_mpi_proxy, only: s_mpi_abort, s_initialize_amr_mpi_buffers + use m_mpi_proxy, only: s_mpi_abort use m_mpi_common, only: s_mpi_allreduce_integer_min, s_mpi_allreduce_integer_max, s_mpi_allreduce_sum, s_mpi_allreduce_min, & & s_mpi_allreduce_max, s_mpi_allreduce_integer_sum, s_mpi_sendrecv_variables_buffers, s_mpi_allreduce_array_max use m_rhs, only: s_compute_rhs @@ -37,11 +37,10 @@ module m_amr private public :: t_level, amr_maxc, amr_maxc_fit, amr_dt_fine, s_initialize_amr_module, s_populate_amr_fine, & - & s_interpolate_coarse_to_fine, s_restrict_fine_to_coarse, s_amr_conservation_check, s_finalize_amr_module, & - & s_amr_swap_to_fine, s_amr_restore_coarse, s_amr_fill_fine_ghosts, s_amr_operator_checks, s_amr_fine_stage_fill, & - & s_amr_fine_stage_advance, s_amr_fine_fine_halo, s_amr_advance_fine_subcycle_all, s_amr_conservation_defect, & - & s_set_amr_fine_geometry, s_amr_regrid, s_write_amr_restart, s_read_amr_restart, s_amr_relax_fine, s_amr_setup_ib, & - & s_amr_check_active_box_containment, s_amr_p2p_reflux_faces, s_amr_reflux_to_parent + & s_interpolate_coarse_to_fine, s_restrict_fine_to_coarse, s_finalize_amr_module, s_amr_swap_to_fine, & + & s_amr_restore_coarse, s_amr_fill_fine_ghosts, s_amr_fine_stage_fill, s_amr_fine_stage_advance, s_amr_fine_fine_halo, & + & s_amr_advance_fine_subcycle_all, s_set_amr_fine_geometry, s_amr_regrid, s_write_amr_restart, s_read_amr_restart, & + & s_amr_relax_fine, s_amr_setup_ib, s_amr_check_active_box_containment, s_amr_p2p_reflux_faces, s_amr_reflux_to_parent !> Fine-level time step for subcycling (= 0.5*dt after init; 0 when amr is off). real(wp) :: amr_dt_fine = 0._wp @@ -91,6 +90,14 @@ module m_amr integer :: max_f1, max_f2, max_f3 integer :: mbuf1_lo, mbuf1_hi, mbuf2_lo, mbuf2_hi, mbuf3_lo, mbuf3_hi logical, allocatable :: amr_slot_live(:) + !! fine-fine seam pack buffers, hoisted out of the per-seam s_amr_fine_fine_halo loop (allocated once at max seam extent) + real(wp), allocatable :: amr_seambuf_x(:), amr_seambuf_y(:) + !! cached same-level adjacent-seam list (3, npairs) = (xb, yb, seam-dim), so s_amr_fine_fine_halo iterates O(#seams) + !! instead of rescanning all O(nblocks^2) pairs every RK stage. Block topology (regions/levels/count) changes only at + !! regrid/restart, so the list is rebuilt only when amr_seam_pairs_dirty is set (or the block count changes - a tripwire). + integer, allocatable :: amr_seam_pairs(:,:) + integer :: amr_num_seam_pairs, amr_seam_pairs_nblk + logical :: amr_seam_pairs_dirty !> Regrid box size cap per dim (fixed for the run, identical on all ranks; 1 in collapsed dims): a box of at most min over ranks !! of (local extent + 1)/2 cells intersects EVERY rank in at most (its extent + 1)/2 cells, so the per-rank scratch constraint @@ -136,7 +143,6 @@ module m_amr real(wp), allocatable :: sw_z_cb(:), sw_z_cc(:), sw_dz(:) !> Conservation-defect baselines (level-0 interior integrals at init; per-fluid masses + energy) - real(wp) :: amr_mass0(num_fluids_max) = 0._wp, amr_energy0 = 0._wp !> True (identically on all ranks) iff some rank's fine ghost-fill stencil reads its coarse GHOST cells - the solver populates !! only PRIM ghosts, so the CONS ghosts the fill prolongs from must be halo-exchanged first. Never true at np=1 (block faces sit @@ -172,7 +178,7 @@ contains !! (sys_size/buff_size set). Per-slot fine arrays are allocated lazily (s_amr_reconcile_slots) - only the blocks a rank owns. impure subroutine s_initialize_amr_module() - integer :: i, d, islot + integer :: i, d, islot, tcap integer :: sidx(3), ext(3), maxc_loc(3), bad_loc, bad_glb, fit_d integer :: blk_lo(3), blk_hi(3) type(scalar_field), allocatable :: tmp_cg(:) @@ -280,8 +286,17 @@ contains if (n_glb > 0) max_f2 = ref_ratio*maxc_loc(2) - 1 if (p_glb > 0) max_f3 = ref_ratio*maxc_loc(3) - 1 - ! MPI exchange buffers for the fine halo (all ranks; no-op without MFC_MPI) - call s_initialize_amr_mpi_buffers(max_f1, max_f2, max_f3) + ! fine-fine seam pack buffers: sized ONCE to the largest possible seam (sys_size*buff_size * max transverse fine + ! face), so s_amr_fine_fine_halo reuses them instead of allocating per seam per stage. Transverse cap = the largest + ! fine face over the choice of seam dimension: 3D -> max pairwise product, 2D -> the larger single dim, 1D -> 1. + tcap = 1 + if (n_glb > 0 .and. p_glb > 0) then + tcap = max((max_f1 + 1)*(max_f2 + 1), (max_f2 + 1)*(max_f3 + 1), (max_f1 + 1)*(max_f3 + 1)) + else if (n_glb > 0) then + tcap = max(max_f1, max_f2) + 1 + end if + allocate (amr_seambuf_x(sys_size*buff_size*tcap), amr_seambuf_y(sys_size*buff_size*tcap)) + amr_seam_pairs_dirty = .true.; amr_seam_pairs_nblk = -1 ! force a seam-list build on the first fine-fine halo mbuf1_lo = -buff_size; mbuf1_hi = max_f1 + buff_size mbuf2_lo = 0; mbuf2_hi = 0; mbuf3_lo = 0; mbuf3_hi = 0 if (n_glb > 0) then; mbuf2_lo = -buff_size; mbuf2_hi = max_f2 + buff_size; end if @@ -1149,7 +1164,6 @@ contains integer :: k, kk, r, a, lev, maxlev, ord(amr_num_blocks) integer(kind=8) :: wt(amr_num_blocks), twt(amr_num_blocks), key(amr_num_blocks), tmpk, cum, tgt, total integer :: tmpo - real(wp) :: rank_load(0:num_procs - 1), imbal if (amr_num_blocks < 1) return @@ -1196,14 +1210,12 @@ contains do k = 1, amr_num_blocks if (amr_block_level(k) == 1) total = total + twt(k) end do - rank_load = 0._wp r = 0; cum = 0_8 do k = 1, amr_num_blocks if (amr_block_level(ord(k)) /= 1) cycle tgt = (int(r + 1, 8)*total)/int(num_procs, 8) if (cum >= tgt .and. r < num_procs - 1) r = r + 1 amr_block_owner(ord(k)) = r - rank_load(r) = rank_load(r) + real(twt(ord(k)), wp) cum = cum + twt(ord(k)) end do @@ -1215,12 +1227,6 @@ contains end do end do - if (proc_rank == 0) then - imbal = maxval(rank_load)/max(real(total, wp)/real(num_procs, wp), 1._wp) - print '(A,I0,A,F6.2,A)', ' [amr] fine-dist map: ', amr_num_blocks, ' block(s), predicted fine-work imbalance ', & - & imbal, 'x (1.00 = perfect; map computed, not yet applied)' - end if - end subroutine s_amr_assign_block_owners !> 3D Morton (Z-order) key from global coarse indices; collapsed dims contribute 0. 21 bits/dim (fits a 64-bit key for grids up @@ -1706,8 +1712,8 @@ contains if (rank_time_wrt .and. amr_rank_owns_block) call s_rank_time_tic() ! multi-level: a level>=2 block folds back into its PARENT block's fine array (the coarse side of level l is level l-1), - ! not the L0 coarse_tgt. Same restriction kernel, targeted at the parent in the parent-fine frame. np=1 local; np>=2 P2P - ! TODO. + ! not the L0 coarse_tgt. Same restriction kernel, targeted at the parent in the parent-fine frame. Co-located towers + ! keep a level>=2 block and its parent on the same rank, so the restrict is always local (no cross-rank P2P needed). if (amr_block_level(amr_cur) >= 2) then if (amr_rank_owns_block) call s_amr_restrict_to_parent() if (rank_time_wrt .and. amr_rank_owns_block) call s_rank_time_toc() @@ -2525,50 +2531,6 @@ contains end subroutine s_amr_scatter_pbmv - !> SP2 gate: restrict(prolong(coarse)) must reproduce coarse over the block interior (conservation). Init-only diagnostic; - !! allocates a scratch coarse target, never touches level-0 or the solve. - impure subroutine s_amr_conservation_check(q_cons_base) - - type(scalar_field), dimension(sys_size), intent(in) :: q_cons_base - type(scalar_field), dimension(:), allocatable :: scratch - integer :: i, ci, cj, ck - real(wp) :: err, e - - if (.not. amr) return - ! this self-test compares restrict(fine) to the gathered patch amr_cg for the CURRENT block; amr_cg holds only the LAST - ! block s_populate_amr_fine gathered, so it is meaningful only when there is a single block (the untiled np=1 case) - if (amr_num_blocks > 1) return - if (.not. amr_rank_owns_block) return - ! fine-level distribution: the block's coarse cells live in the gathered patch amr_cg (set by s_populate_amr_fine just - ! before this), not the owner's local q_cons_base. Compare restrict(fine) to amr_cg in the block-local patch frame. - allocate (scratch(1:sys_size)) - do i = 1, sys_size - allocate (scratch(i)%sf(0:amr_cpat_hi(1),0:amr_cpat_hi(2),0:amr_cpat_hi(3))) - end do - ! host restriction path: the scratch target is host-only and the fine state is host-current at init - do i = 1, sys_size - call s_restrict_one_var(amr_slots(amr_cur)%q_cons(i), scratch(i)) - end do - err = 0._wp - do i = 1, sys_size - do ck = amr_isect_lo(3), merge(amr_isect_hi(3), amr_isect_lo(3), p_glb > 0) - do cj = amr_isect_lo(2), merge(amr_isect_hi(2), amr_isect_lo(2), n_glb > 0) - do ci = amr_isect_lo(1), amr_isect_hi(1) - e = abs(real(scratch(i)%sf(ci - amr_cpat_off(1), cj - amr_cpat_off(2), ck - amr_cpat_off(3)), & - & wp) - real(amr_cg(i)%sf(ci - amr_cpat_off(1), cj - amr_cpat_off(2), ck - amr_cpat_off(3)), wp)) - if (e > err) err = e - end do - end do - end do - end do - print '(A,ES12.4)', ' [amr] restrict-prolong conservation err = ', err ! every rank with fine cells prints - do i = 1, sys_size - deallocate (scratch(i)%sf) - end do - deallocate (scratch) - - end subroutine s_amr_conservation_check - !> Swap the global grid state to the fine block. MUST be paired with s_amr_restore_coarse. impure subroutine s_amr_swap_to_fine() @@ -3215,6 +3177,53 @@ contains end subroutine s_amr_fine_slice + !> Seam dimension of the ordered pair (xb, yb): the dim d in which yb is the immediate high-face neighbour of xb at matched + !! resolution (same level), or 0 if they are not a same-level fine-fine seam. Face adjacency requires transverse overlap, so a + !! pair is adjacent in at most one dim; the last-true assignment reproduces the original inline scan in s_amr_fine_fine_halo. + pure integer function f_amr_seam_dim(xb, yb) result(d) + + integer, intent(in) :: xb, yb + + d = 0 + if (xb == yb) return + if (amr_block_level(xb) /= amr_block_level(yb)) return + if (f_amr_seam(xb, yb, 1)) d = 1 + if (n_glb > 0) then; if (f_amr_seam(xb, yb, 2)) d = 2; end if + if (p_glb > 0) then; if (f_amr_seam(xb, yb, 3)) d = 3; end if + + end function f_amr_seam_dim + + !> Rebuild the cached same-level seam-pair list (amr_seam_pairs): one O(nblocks^2) scan per regrid/restart in place of the same + !! scan every RK stage (6x per fine step). Same (xb, yb) nested-loop order on all ranks (replicated region metadata) so the + !! paired MPI_SENDRECVs in s_amr_fine_fine_halo stay matched. Count then fill for an exact-size list (no cap, no overflow). + impure subroutine s_amr_build_seam_pairs() + + integer :: xb, yb, d, np + + if (allocated(amr_seam_pairs)) deallocate (amr_seam_pairs) + np = 0 + do xb = 1, amr_num_blocks + do yb = 1, amr_num_blocks + if (f_amr_seam_dim(xb, yb) > 0) np = np + 1 + end do + end do + amr_num_seam_pairs = np + allocate (amr_seam_pairs(3, max(np, 1))) + np = 0 + do xb = 1, amr_num_blocks + do yb = 1, amr_num_blocks + d = f_amr_seam_dim(xb, yb) + if (d > 0) then + np = np + 1 + amr_seam_pairs(1, np) = xb; amr_seam_pairs(2, np) = yb; amr_seam_pairs(3, np) = d + end if + end do + end do + amr_seam_pairs_nblk = amr_num_blocks + amr_seam_pairs_dirty = .false. + + end subroutine s_amr_build_seam_pairs + !> Block-to-block fine-fine halo (max_grid_size tiling): overwrite each sub-block's seam ghost cells (faces shared with an !! ADJACENT sub-block) with the neighbour's stage-entry fine interior, so the shared fine flux matches on both sides !! (coarse-prolonged seam ghosts would be non-conservative). For each seam pair (xb below, yb above, dim d) the two owners @@ -3222,69 +3231,64 @@ contains !! stp on unpack (identity for stp fields). No-op with a single block / no adjacent pairs (incl. every np=1 case, untiled). impure subroutine s_amr_fine_fine_halo() - integer :: xb, yb, d, rX, rY, cnt, xm(3), ym(3), tsz, ierr, fmul - real(wp), allocatable :: xbuf(:), ybuf(:) + integer :: xb, yb, d, rX, rY, cnt, xm(3), ym(3), tsz, ierr, fmul, idx if (.not. amr) return if (amr_num_blocks < 2) return + ! iterate the cached same-level seam list (rebuilt only when the topology changes) instead of the old O(nblocks^2) + ! f_amr_seam rescan every stage; the list preserves the original (xb, yb) order so paired MPI_SENDRECVs still match. + if (amr_seam_pairs_dirty .or. amr_seam_pairs_nblk /= amr_num_blocks) call s_amr_build_seam_pairs() ! device<->host of the fine state is done per-seam inside s_amr_fine_slice, moving only the buff_size-deep near-seam ! slab each pack/unpack touches (not the whole block) - a large PCIe saving since this runs per stage (6x per fine step) - do xb = 1, amr_num_blocks - do yb = 1, amr_num_blocks - if (xb == yb) cycle - if (amr_block_level(xb) /= amr_block_level(yb)) cycle ! fine-fine halo is same-level only (matched resolution) - d = 0 - if (f_amr_seam(xb, yb, 1)) d = 1 - if (n_glb > 0) then; if (f_amr_seam(xb, yb, 2)) d = 2; end if - if (p_glb > 0) then; if (f_amr_seam(xb, yb, 3)) d = 3; end if - if (d == 0) cycle - rX = amr_block_owner(xb); rY = amr_block_owner(yb) - if (proc_rank /= rX .and. proc_rank /= rY) cycle - ! fine extents from the REPLICATED region metadata (not amr_slots%m/n/p: at np>1 this rank may own only one of the - ! pair, and the transverse size (used for the buffer count) must be valid for both). A level-L block's region is in - ! L0-coarse cells but its own grid is 2**L finer (each level halves dx), so fine = 2**L*(coarse extent)-1; xb, yb - ! share the level (same-level seam). 2**1 keeps L1 byte-identical; L2 tiles need 2**2 (an L1-frame 2* mislocates the - ! seam slice to half the block, filling the seam ghost from the wrong cells - the source of the L2-L2 leak). - fmul = ref_ratio**amr_block_level(xb) - xm(1) = fmul*(amr_region_hi_all(1, xb) - amr_region_lo_all(1, xb) + 1) - 1 - xm(2) = merge(fmul*(amr_region_hi_all(2, xb) - amr_region_lo_all(2, xb) + 1) - 1, 0, n_glb > 0) - xm(3) = merge(fmul*(amr_region_hi_all(3, xb) - amr_region_lo_all(3, xb) + 1) - 1, 0, p_glb > 0) - ym(1) = fmul*(amr_region_hi_all(1, yb) - amr_region_lo_all(1, yb) + 1) - 1 - ym(2) = merge(fmul*(amr_region_hi_all(2, yb) - amr_region_lo_all(2, yb) + 1) - 1, 0, n_glb > 0) - ym(3) = merge(fmul*(amr_region_hi_all(3, yb) - amr_region_lo_all(3, yb) + 1) - 1, 0, p_glb > 0) - ! transverse fine size (dims /= d); xb and yb share it (exact-match seam) - tsz = 1 - if (d /= 1) tsz = tsz*(xm(1) + 1) - if (d /= 2 .and. n_glb > 0) tsz = tsz*(xm(2) + 1) - if (d /= 3 .and. p_glb > 0) tsz = tsz*(xm(3) + 1) - cnt = sys_size*buff_size*tsz - allocate (xbuf(cnt), ybuf(cnt)) - if (rX == rY) then ! same rank owns both: pack each near-seam interior, unpack into the other's seam ghost - call s_amr_fine_slice(xb, amr_slots(xb)%q_cons, d, xm(d) - buff_size + 1, xm(d), xbuf, 1) ! xb high interior - call s_amr_fine_slice(yb, amr_slots(yb)%q_cons, d, 0, buff_size - 1, ybuf, 1) ! yb low interior - call s_amr_fine_slice(yb, amr_slots(yb)%q_cons, d, -buff_size, -1, xbuf, -1) ! -> yb low ghost - call s_amr_fine_slice(xb, amr_slots(xb)%q_cons, d, xm(d) + 1, xm(d) + buff_size, ybuf, -1) ! -> xb high ghost - else if (proc_rank == rX) then - ! send xb high interior - call s_amr_fine_slice(xb, amr_slots(xb)%q_cons, d, xm(d) - buff_size + 1, xm(d), xbuf, 1) + do idx = 1, amr_num_seam_pairs + xb = amr_seam_pairs(1, idx); yb = amr_seam_pairs(2, idx); d = amr_seam_pairs(3, idx) + rX = amr_block_owner(xb); rY = amr_block_owner(yb) + if (proc_rank /= rX .and. proc_rank /= rY) cycle + ! fine extents from the REPLICATED region metadata (not amr_slots%m/n/p: at np>1 this rank may own only one of the + ! pair, and the transverse size (used for the buffer count) must be valid for both). A level-L block's region is in + ! L0-coarse cells but its own grid is 2**L finer (each level halves dx), so fine = 2**L*(coarse extent)-1; xb, yb + ! share the level (same-level seam). 2**1 keeps L1 byte-identical; L2 tiles need 2**2 (an L1-frame 2* mislocates the + ! seam slice to half the block, filling the seam ghost from the wrong cells - the source of the L2-L2 leak). + fmul = ref_ratio**amr_block_level(xb) + xm(1) = fmul*(amr_region_hi_all(1, xb) - amr_region_lo_all(1, xb) + 1) - 1 + xm(2) = merge(fmul*(amr_region_hi_all(2, xb) - amr_region_lo_all(2, xb) + 1) - 1, 0, n_glb > 0) + xm(3) = merge(fmul*(amr_region_hi_all(3, xb) - amr_region_lo_all(3, xb) + 1) - 1, 0, p_glb > 0) + ym(1) = fmul*(amr_region_hi_all(1, yb) - amr_region_lo_all(1, yb) + 1) - 1 + ym(2) = merge(fmul*(amr_region_hi_all(2, yb) - amr_region_lo_all(2, yb) + 1) - 1, 0, n_glb > 0) + ym(3) = merge(fmul*(amr_region_hi_all(3, yb) - amr_region_lo_all(3, yb) + 1) - 1, 0, p_glb > 0) + ! transverse fine size (dims /= d); xb and yb share it (exact-match seam) + tsz = 1 + if (d /= 1) tsz = tsz*(xm(1) + 1) + if (d /= 2 .and. n_glb > 0) tsz = tsz*(xm(2) + 1) + if (d /= 3 .and. p_glb > 0) tsz = tsz*(xm(3) + 1) + cnt = sys_size*buff_size*tsz + if (rX == rY) then ! same rank owns both: pack each near-seam interior, unpack into the other's seam ghost + ! xb high interior + call s_amr_fine_slice(xb, amr_slots(xb)%q_cons, d, xm(d) - buff_size + 1, xm(d), amr_seambuf_x(1:cnt), 1) + call s_amr_fine_slice(yb, amr_slots(yb)%q_cons, d, 0, buff_size - 1, amr_seambuf_y(1:cnt), 1) ! yb low interior + call s_amr_fine_slice(yb, amr_slots(yb)%q_cons, d, -buff_size, -1, amr_seambuf_x(1:cnt), -1) ! -> yb low ghost + ! -> xb high ghost + call s_amr_fine_slice(xb, amr_slots(xb)%q_cons, d, xm(d) + 1, xm(d) + buff_size, amr_seambuf_y(1:cnt), -1) + else if (proc_rank == rX) then + ! send xb high interior + call s_amr_fine_slice(xb, amr_slots(xb)%q_cons, d, xm(d) - buff_size + 1, xm(d), amr_seambuf_x(1:cnt), 1) #ifdef MFC_MPI - call MPI_SENDRECV(xbuf, cnt, mpi_p, rY, 4200, ybuf, cnt, mpi_p, rY, 4201, MPI_COMM_WORLD, MPI_STATUS_IGNORE, & - & ierr) + call MPI_SENDRECV(amr_seambuf_x(1:cnt), cnt, mpi_p, rY, 4200, amr_seambuf_y(1:cnt), cnt, mpi_p, rY, 4201, & + & MPI_COMM_WORLD, MPI_STATUS_IGNORE, ierr) #endif - ! recv yb low interior -> xb high ghost - call s_amr_fine_slice(xb, amr_slots(xb)%q_cons, d, xm(d) + 1, xm(d) + buff_size, ybuf, -1) - else ! proc_rank == rY - call s_amr_fine_slice(yb, amr_slots(yb)%q_cons, d, 0, buff_size - 1, ybuf, 1) ! send yb low interior + ! recv yb low interior -> xb high ghost + call s_amr_fine_slice(xb, amr_slots(xb)%q_cons, d, xm(d) + 1, xm(d) + buff_size, amr_seambuf_y(1:cnt), -1) + else ! proc_rank == rY + ! send yb low interior + call s_amr_fine_slice(yb, amr_slots(yb)%q_cons, d, 0, buff_size - 1, amr_seambuf_y(1:cnt), 1) #ifdef MFC_MPI - call MPI_SENDRECV(ybuf, cnt, mpi_p, rX, 4201, xbuf, cnt, mpi_p, rX, 4200, MPI_COMM_WORLD, MPI_STATUS_IGNORE, & - & ierr) + call MPI_SENDRECV(amr_seambuf_y(1:cnt), cnt, mpi_p, rX, 4201, amr_seambuf_x(1:cnt), cnt, mpi_p, rX, 4200, & + & MPI_COMM_WORLD, MPI_STATUS_IGNORE, ierr) #endif - ! recv xb high interior -> yb low ghost - call s_amr_fine_slice(yb, amr_slots(yb)%q_cons, d, -buff_size, -1, xbuf, -1) - end if - deallocate (xbuf, ybuf) - end do + ! recv xb high interior -> yb low ghost + call s_amr_fine_slice(yb, amr_slots(yb)%q_cons, d, -buff_size, -1, amr_seambuf_x(1:cnt), -1) + end if end do call s_amr_select_slot(1) @@ -3669,26 +3673,24 @@ contains end subroutine s_amr_advance_children - !> Shrink box [blo:bhi] to the tight bounding box of the tagged (gtag==1) cells inside it. ok=.false. if no tagged cell. - !! Collapsed dims (lo=hi=0) survive unchanged. Deterministic (integer scan of the identical global tag field). - impure subroutine s_amr_trim_box(gtag, blo, bhi, ok) + !> Shrink box [blo:bhi] to the tight bounding box of the tagged cells inside it. ok=.false. if no tagged cell. Collapsed dims + !! (lo=hi=0) survive unchanged. Deterministic (integer scan of the identical sparse tag list). + impure subroutine s_amr_trim_box(tags, ts, te, blo, bhi, ok) - integer, intent(in) :: gtag(0:,0:,0:) + integer, intent(in) :: tags(:,:), ts, te integer, intent(inout) :: blo(3), bhi(3) logical, intent(out) :: ok - integer :: tlo(3), thi(3), i, j, k + integer :: tlo(3), thi(3), t, i, j, k tlo = huge(1); thi = -huge(1) - do k = blo(3), bhi(3) - do j = blo(2), bhi(2) - do i = blo(1), bhi(1) - if (gtag(i, j, k) == 1) then - tlo(1) = min(tlo(1), i); thi(1) = max(thi(1), i) - tlo(2) = min(tlo(2), j); thi(2) = max(thi(2), j) - tlo(3) = min(tlo(3), k); thi(3) = max(thi(3), k) - end if - end do - end do + do t = ts, te + i = tags(1, t); j = tags(2, t); k = tags(3, t) + if (i < blo(1) .or. i > bhi(1)) cycle + if (j < blo(2) .or. j > bhi(2)) cycle + if (k < blo(3) .or. k > bhi(3)) cycle + tlo(1) = min(tlo(1), i); thi(1) = max(thi(1), i) + tlo(2) = min(tlo(2), j); thi(2) = max(thi(2), j) + tlo(3) = min(tlo(3), k); thi(3) = max(thi(3), k) end do ok = thi(1) >= tlo(1) if (ok) then; blo = tlo; bhi = thi; end if @@ -3698,14 +3700,14 @@ contains !> Berger-Rigoutsos bisection of one (already tagged-trimmed) candidate box on the global tag field: pick the longest splittable !! axis, prefer a zero-signature hole (widest interior run), else the strongest signature inflection (Laplacian sign change). !! ok=.false. if no axis admits a split leaving both children >= 2 cells. Integer-only => identical on all ranks. - impure subroutine s_amr_find_split(gtag, blo, bhi, sax, spos, ok) + impure subroutine s_amr_find_split(tags, ts, te, blo, bhi, sax, spos, ok) - integer, intent(in) :: gtag(0:,0:,0:) + integer, intent(in) :: tags(:,:), ts, te integer, intent(in) :: blo(3), bhi(3) integer, intent(out) :: sax, spos logical, intent(out) :: ok integer, parameter :: min_child = 2 - integer :: axord(3), ext(3), d, ax, t, u, v, s + integer :: axord(3), ext(3), d, ax, t, i, j, k, s integer :: run, run_start, best_run, best_start, lap, prevlap, bestmag, bestpos integer, allocatable :: sig(:) @@ -3724,12 +3726,19 @@ contains ax = axord(d) if (ax > num_dims) cycle if (ext(ax) < 2*min_child) cycle + ! 1D signature sig(t) = count of in-box tagged cells at axis-position t (sum over the two transverse dims) do t = blo(ax), bhi(ax) sig(t) = 0 + end do + do t = ts, te + i = tags(1, t); j = tags(2, t); k = tags(3, t) + if (i < blo(1) .or. i > bhi(1)) cycle + if (j < blo(2) .or. j > bhi(2)) cycle + if (k < blo(3) .or. k > bhi(3)) cycle select case (ax) - case (1); do v = blo(3), bhi(3); do u = blo(2), bhi(2); sig(t) = sig(t) + gtag(t, u, v); end do; end do - case (2); do v = blo(3), bhi(3); do u = blo(1), bhi(1); sig(t) = sig(t) + gtag(u, t, v); end do; end do - case (3); do v = blo(2), bhi(2); do u = blo(1), bhi(1); sig(t) = sig(t) + gtag(u, v, t); end do; end do + case (1); sig(i) = sig(i) + 1 + case (2); sig(j) = sig(j) + 1 + case (3); sig(k) = sig(k) + 1 end select end do ! (1) widest interior zero run (box is trimmed => sig(blo)>0 and sig(bhi)>0, so any run is interior) @@ -4083,82 +4092,193 @@ contains end subroutine s_amr_tile_box - !> Cluster the local per-cell tag field into a LIST of separated block boxes (global level-0 cell indices), identically on every - !! rank. Gathers the tags into a global field (allreduce MAX), runs Berger-Rigoutsos recursive bisection until each box's tag - !! efficiency reaches amr_cluster_eff (or it is atomic / the amr_max_blocks cap is reached), then merges any two boxes whose - !! amr_buf-padded extents come within buff_size (guaranteeing no fine-fine adjacency: separated boxes stay >= buff_size apart, - !! nearby ones collapse to a single box == the legacy bounding box). Boxes are the raw tagged extents; the caller pads, clamps - !! and size-caps each one. - impure subroutine s_amr_cluster(tag_grid, boxes, nboxes) + !> Rank-invariant SPARSE tag list for level clustering (SP7a): all-gathers each rank's tagged-cell global linear indices, then + !! decodes them into a coordinate list tags(1:3, 1:ntag). Replaces the O(global-grid) dense tag field entirely, so per-rank + !! memory scales with the number of tagged cells. At np=1 the list is built directly from tag_grid (no allgather). + !! Deterministic: every rank decodes the same gathered index set into the same list, so the bisection is rank-invariant. + impure subroutine s_amr_union_gtag(tags, ntag, tag_grid, mg, ng, pg, sidx) + + integer, allocatable, intent(out) :: tags(:,:) + integer, intent(out) :: ntag + logical, intent(in) :: tag_grid(0:,0:,0:) + integer, intent(in) :: mg, ng, pg, sidx(3) + integer :: ci, cj, ck, gi, gj, gk + +#ifdef MFC_MPI + integer :: i, jrem, nloc, ierr + integer, allocatable :: rcnt(:), rdsp(:) + integer(8), allocatable :: locidx(:), allidx(:) + + if (num_procs > 1) then + allocate (locidx((m + 1)*(n + 1)*(p + 1)), rcnt(num_procs), rdsp(num_procs)) + nloc = 0 + do ck = 0, p; do cj = 0, n; do ci = 0, m + if (tag_grid(ci, cj, ck)) then + gi = ci + sidx(1); gj = 0; gk = 0 + if (n_glb > 0) gj = cj + sidx(2) + if (p_glb > 0) gk = ck + sidx(3) + nloc = nloc + 1 + locidx(nloc) = int(gi, 8) + int(mg + 1, 8)*(int(gj, 8) + int(ng + 1, 8)*int(gk, 8)) + end if + end do; end do; end do + call MPI_ALLGATHER(nloc, 1, MPI_INTEGER, rcnt, 1, MPI_INTEGER, MPI_COMM_WORLD, ierr) + rdsp(1) = 0 + do i = 2, num_procs; rdsp(i) = rdsp(i - 1) + rcnt(i - 1); end do + ntag = rdsp(num_procs) + rcnt(num_procs) + allocate (allidx(max(ntag, 1)), tags(3, max(ntag, 1))) + call MPI_ALLGATHERV(locidx, nloc, MPI_INTEGER8, allidx, rcnt, rdsp, MPI_INTEGER8, MPI_COMM_WORLD, ierr) + do i = 1, ntag + gk = int(allidx(i)/(int(mg + 1, 8)*int(ng + 1, 8))) + jrem = int(allidx(i) - int(gk, 8)*int(mg + 1, 8)*int(ng + 1, 8)) + gj = jrem/(mg + 1) + gi = jrem - gj*(mg + 1) + tags(1, i) = gi; tags(2, i) = gj; tags(3, i) = gk + end do + deallocate (locidx, rcnt, rdsp, allidx) + return + end if +#endif + ! np=1: the whole global tag field is local; decode tag_grid directly into the list + ntag = 0 + do ck = 0, p; do cj = 0, n; do ci = 0, m + if (tag_grid(ci, cj, ck)) ntag = ntag + 1 + end do; end do; end do + allocate (tags(3, max(ntag, 1))) + ntag = 0 + do ck = 0, p; do cj = 0, n; do ci = 0, m + if (tag_grid(ci, cj, ck)) then + gi = ci + sidx(1); gj = 0; gk = 0 + if (n_glb > 0) gj = cj + sidx(2) + if (p_glb > 0) gk = ck + sidx(3) + ntag = ntag + 1 + tags(1, ntag) = gi; tags(2, ntag) = gj; tags(3, ntag) = gk + end if + end do; end do; end do + + end subroutine s_amr_union_gtag + + !> Grow the per-level pack buffers sidx(:) (int8 linear index) / skb(:) (parent box id) geometrically so at least nloc+extra + !! slots fit; preserves the first nloc entries. Amortized O(1) append for s_amr_pack_gwin_pairs. + impure subroutine s_amr_grow_pack(sidx, skb, nloc, extra) + + integer(8), allocatable, intent(inout) :: sidx(:) + integer, allocatable, intent(inout) :: skb(:) + integer, intent(in) :: nloc, extra + integer :: cap, newcap + integer(8), allocatable :: t8(:) + integer, allocatable :: ti(:) + + cap = 0 + if (allocated(sidx)) cap = size(sidx) + if (nloc + extra <= cap) return + newcap = max(2*cap, max(nloc + extra, 1024)) + allocate (t8(newcap), ti(newcap)) + if (nloc > 0) then + t8(1:nloc) = sidx(1:nloc) + ti(1:nloc) = skb(1:nloc) + end if + call move_alloc(t8, sidx) + call move_alloc(ti, skb) + + end subroutine s_amr_grow_pack + + !> Pack this rank's OWNED tagged cells of the child window [mlo:mhi] as (linear-index, kb) pairs, appended to the per-level send + !! arrays sidx(:) (int8 linear index) / skb(:) (parent box id). The int8 encode matches the decode in s_amr_regrid's pass 2, so + !! gathering these pairs across ranks and setting them into a per-parent dense window reproduces the old per-parent dense-window + !! dedup (replicated/overlapping tags collapse) and the (k,j,i) extraction order exactly -> byte-identical child boxes. Batching + !! all parents of a level into one allgatherv (in the caller) drops the collective count from O(#parent-boxes) to O(#levels). + !! gwin is read here (not modified). + impure subroutine s_amr_pack_gwin_pairs(gwin, mlo, mhi, mg, ng, kb, sidx, skb, nloc) + + integer, intent(in) :: mlo(3), mhi(3), mg, ng, kb + logical, intent(in) :: gwin(mlo(1):,mlo(2):,mlo(3):) + integer(8), allocatable, intent(inout) :: sidx(:) + integer, allocatable, intent(inout) :: skb(:) + integer, intent(inout) :: nloc + integer :: gi, gj, gk + + do gk = mlo(3), mhi(3) + do gj = mlo(2), mhi(2) + do gi = mlo(1), mhi(1) + if (.not. gwin(gi, gj, gk)) cycle + call s_amr_grow_pack(sidx, skb, nloc, 1) + nloc = nloc + 1 + sidx(nloc) = int(gi, 8) + int(mg + 1, 8)*(int(gj, 8) + int(ng + 1, 8)*int(gk, 8)) + skb(nloc) = kb + end do + end do + end do + + end subroutine s_amr_pack_gwin_pairs + + !> Cluster a rank-invariant SPARSE tag list (global level-0 cell coords, tags(1:3, 1:ntag_in)) into a LIST of separated block + !! boxes, identically on every rank. The caller builds the list (s_amr_union_gtag / s_amr_pack_gwin_pairs). Per-rank memory is + !! O(#tagged), not O(global grid). Runs Berger-Rigoutsos recursive bisection until each box's tag efficiency reaches + !! amr_cluster_eff (or it is atomic / the amr_max_blocks cap is reached), then merges any two boxes whose amr_buf-padded extents + !! come within buff_size (guaranteeing no fine-fine adjacency: separated boxes stay >= buff_size apart, nearby ones collapse to + !! a single box == the legacy bounding box). Boxes are the raw tagged extents; the caller pads, clamps and size-caps each one. + impure subroutine s_amr_cluster(tags, ntag_in, boxes, nboxes) - logical, intent(in) :: tag_grid(0:,0:,0:) + integer, intent(in) :: tags(:,:), ntag_in type(t_box), allocatable, intent(out) :: boxes(:) integer, intent(out) :: nboxes - integer, allocatable :: gtag(:,:,:), slo(:,:), shi(:,:), alo(:,:), ahi(:,:) - integer :: mg, ng, pg, sidx(3), ci, cj, ck, gi, gj, gk + integer, allocatable :: slo(:,:), shi(:,:), alo(:,:), ahi(:,:) + integer, allocatable :: sts(:), ste(:), wt(:,:) + integer :: mg, ng, pg, t integer :: cap, nwork, nacc, i, j, d, sax, spos, thr, ntag, vol - integer :: blo(3), bhi(3) + integer :: blo(3), bhi(3), ts, te, lo, hi, tmp(3) logical :: ok, force, capped, changed, tooclose real(wp) :: eff -#ifdef MFC_MPI - integer :: ierr -#endif - nboxes = 0 + if (ntag_in == 0) return mg = m_glb; ng = 0; pg = 0 if (n_glb > 0) ng = n_glb if (p_glb > 0) pg = p_glb - allocate (gtag(0:mg,0:ng,0:pg)); gtag = 0 - sidx = 0; sidx(1) = start_idx(1) - if (n_glb > 0) sidx(2) = start_idx(2) - if (p_glb > 0) sidx(3) = start_idx(3) - do ck = 0, p - do cj = 0, n - do ci = 0, m - if (tag_grid(ci, cj, ck)) then - gi = ci + sidx(1); gj = 0; gk = 0 - if (n_glb > 0) gj = cj + sidx(2) - if (p_glb > 0) gk = ck + sidx(3) - gtag(gi, gj, gk) = 1 - end if - end do - end do - end do -#ifdef MFC_MPI - ! every rank ORs in its local tags => an identical global tag field, so the bisection below is rank-invariant (SP7a) - if (num_procs > 1) call MPI_ALLREDUCE(MPI_IN_PLACE, gtag, (mg + 1)*(ng + 1)*(pg + 1), MPI_INTEGER, MPI_MAX, & - & MPI_COMM_WORLD, ierr) -#endif - if (sum(gtag) == 0) then; deallocate (gtag); return; end if cap = amr_max_blocks allocate (slo(3, 4*cap + 8), shi(3, 4*cap + 8), alo(3, cap), ahi(3, cap)) + allocate (sts(4*cap + 8), ste(4*cap + 8), wt(3, ntag_in)) + ! working copy of the tag list, partitioned in place as the tree descends so each node scans only its tags + do t = 1, ntag_in + wt(:,t) = tags(:,t) + end do nwork = 1; slo(:,1) = [0, 0, 0]; shi(:,1) = [mg, ng, pg] ! first pop trims to the global tagged bbox + sts(1) = 1; ste(1) = ntag_in nacc = 0; capped = .false. do while (nwork > 0) - blo = slo(:,nwork); bhi = shi(:,nwork); nwork = nwork - 1 - call s_amr_trim_box(gtag, blo, bhi, ok) + blo = slo(:,nwork); bhi = shi(:,nwork); ts = sts(nwork); te = ste(nwork); nwork = nwork - 1 + call s_amr_trim_box(wt, ts, te, blo, bhi, ok) if (.not. ok) cycle - ntag = 0 - do ck = blo(3), bhi(3); do cj = blo(2), bhi(2); do ci = blo(1), bhi(1) - ntag = ntag + gtag(ci, cj, ck) - end do; end do; end do + ! invariant: [ts:te] holds exactly the tags in this box, and trim only shrinks to their bbox => count is the range size + ntag = te - ts + 1 vol = 1 do d = 1, num_dims; vol = vol*(bhi(d) - blo(d) + 1); end do eff = real(ntag, wp)/real(max(vol, 1), wp) - call s_amr_find_split(gtag, blo, bhi, sax, spos, ok) + call s_amr_find_split(wt, ts, te, blo, bhi, sax, spos, ok) force = (nacc + nwork + 1 >= cap) ! splitting now could overflow the amr_max_blocks cap if (eff >= amr_cluster_eff .or. .not. ok .or. force) then if (nacc < cap) then; nacc = nacc + 1; alo(:,nacc) = blo; ahi(:,nacc) = bhi; end if if (force .and. ok .and. eff < amr_cluster_eff) capped = .true. else + ! partition wt(:, ts:te) in place: coord(sax) < spos to the front (low child), >= spos to the back (high) + lo = ts; hi = te + do while (lo <= hi) + if (wt(sax, lo) < spos) then + lo = lo + 1 + else + tmp = wt(:,lo); wt(:,lo) = wt(:,hi); wt(:,hi) = tmp + hi = hi - 1 + end if + end do + ! low child = [ts:lo-1], high child = [lo:te]; every parent tag lands in exactly one (box just trimmed+split) slo(:,nwork + 1) = blo; shi(:,nwork + 1) = bhi; shi(sax, nwork + 1) = spos - 1 + sts(nwork + 1) = ts; ste(nwork + 1) = lo - 1 slo(:,nwork + 2) = blo; shi(:,nwork + 2) = bhi; slo(sax, nwork + 2) = spos + sts(nwork + 2) = lo; ste(nwork + 2) = te nwork = nwork + 2 end if end do - deallocate (gtag) ! min-separation merge: two boxes are separated only if some active dim's gap reaches thr; else fuse to their bounding box thr = buff_size + 2*amr_buf @@ -4187,7 +4307,7 @@ contains do i = 1, nboxes boxes(i)%lo = alo(:,i); boxes(i)%hi = ahi(:,i) end do - deallocate (slo, shi, alo, ahi) + deallocate (slo, shi, alo, ahi, sts, ste, wt) end subroutine s_amr_cluster @@ -4207,6 +4327,8 @@ contains logical :: old_owns(amr_max_blocks), any_xchg, same, merged integer :: ci, cj, ck, fi, fj, fk, ofi, ofj, ofk, i integer :: sidx(3), tg_lo(3), tg_hi(3), nboxes, box_level(amr_max_blocks) + integer :: mg0, ng0, pg0, ntag + integer, allocatable :: tags(:,:) real(wp) :: r0, g ! valid coarse CONS ghosts at internal rank boundaries: the tag sweep reads +/-1 across seams and the rebuild @@ -4235,8 +4357,10 @@ contains do ck = tg_lo(3), tg_hi(3) do cj = tg_lo(2), tg_hi(2) do ci = tg_lo(1), tg_hi(1) - ! total density gradient (sum of the continuity variables): degenerates to the single-fluid tagger and is - ! immune to trace-fluid noise. Matched-density composition-only interfaces are invisible (documented limit). + ! total density gradient (sum of the continuity variables): degenerates to the single-fluid tagger + ! and is + ! immune to trace-fluid noise. Matched-density composition-only interfaces are invisible (documented + ! limit). r0 = max(abs(f_amr_rho_tot(q_cons_base, ci, cj, ck)), 1.e-30_wp) g = abs(f_amr_rho_tot(q_cons_base, ci + 1, cj, ck) - f_amr_rho_tot(q_cons_base, ci - 1, cj, ck)) if (n_glb > 0) g = max(g, abs(f_amr_rho_tot(q_cons_base, ci, cj + 1, ck) - f_amr_rho_tot(q_cons_base, ci, & @@ -4258,9 +4382,14 @@ contains end do end do - ! 2) cluster into a list of separated boxes (deterministic on all ranks) - call s_amr_cluster(tag_grid, boxes, nboxes) + ! 2) build the rank-invariant sparse global tag list, then cluster into a list of separated boxes + mg0 = m_glb; ng0 = 0; pg0 = 0 + if (n_glb > 0) ng0 = n_glb + if (p_glb > 0) pg0 = p_glb + call s_amr_union_gtag(tags, ntag, tag_grid, mg0, ng0, pg0, sidx) deallocate (tag_grid) + call s_amr_cluster(tags, ntag, boxes, nboxes) + deallocate (tags) if (nboxes == 0) return ! nothing tagged on any rank; keep the current blocks ! 3) pad + clamp + size-cap each box (amr_maxc_fit lets each box move freely across rank boundaries); drop margin-only boxes @@ -4268,8 +4397,10 @@ contains do kk = 1, nboxes lo = boxes(kk)%lo; hi = boxes(kk)%hi lo(1) = max(lo(1) - amr_buf, buff_size); hi(1) = min(hi(1) + amr_buf, m_glb - buff_size) - ! IB keeps the size-cap CLAMP (a body needs one contiguous block; splitting a body across tiles is untested); the - ! general path leaves boxes full-size and TILES them (below) into <= amr_maxc_fit sub-blocks with a fine-fine halo + ! IB keeps the size-cap CLAMP (a body needs one contiguous block; splitting a body across tiles is + ! untested); the + ! general path leaves boxes full-size and TILES them (below) into <= amr_maxc_fit sub-blocks with a + ! fine-fine halo if (ib .and. hi(1) - lo(1) + 1 > amr_maxc_fit(1)) hi(1) = lo(1) + amr_maxc_fit(1) - 1 if (n_glb > 0) then lo(2) = max(lo(2) - amr_buf, buff_size); hi(2) = min(hi(2) + amr_buf, n_glb - buff_size) @@ -4306,9 +4437,12 @@ contains nboxes = k if (nboxes == 0) return - ! max_grid_size tiling (non-IB): split any box larger than amr_maxc_fit into contiguous <= amr_maxc_fit sub-blocks so a - ! whole block fits a rank's local solver scratch. Tiles are adjacent; the block-to-block fine-fine halo (s_amr_fine_ - ! fine_halo) makes the seams conservative and the reflux skips fine-fine faces. (IB keeps the clamp - see above.) + ! max_grid_size tiling (non-IB): split any box larger than amr_maxc_fit into contiguous <= amr_maxc_fit + ! sub-blocks so a + ! whole block fits a rank's local solver scratch. Tiles are adjacent; the block-to-block fine-fine halo + ! (s_amr_fine_ + ! fine_halo) makes the seams conservative and the reflux skips fine-fine faces. (IB keeps the clamp - see + ! above.) if (.not. ib) then block type(t_box), allocatable :: tiled(:) @@ -4376,20 +4510,26 @@ contains ! 3b) multi-level nesting: hierarchically append a box at level l nested inside each level-(l-1) box, for l = ! 2..amr_max_ - ! level. Parents-first ordering (every level-(l-1) box precedes its level-l children) so the build loop fills a parent - ! before its child's gather-from-parent reads it. SENSOR-ON-FINE: each child's extent is the density-gradient sensor run + ! level. Parents-first ordering (every level-(l-1) box precedes its level-l children) so the build loop fills a + ! parent + ! before its child's gather-from-parent reads it. SENSOR-ON-FINE: each child's extent is the density-gradient + ! sensor run ! on the parent-level FINE solution (the still-live OLD level-(l-1) blocks, read here BEFORE the step-5 stash), ! coarsened - ! to L0-cell granularity and clustered - so children track features inside the parent instead of a fixed centre. A - ! brand-new region with no old fine data falls back to a centred inset (the sensor takes over next regrid); a parent + ! to L0-cell granularity and clustered - so children track features inside the parent instead of a fixed centre. + ! A + ! brand-new region with no old fine data falls back to a centred inset (the sensor takes over next regrid); a + ! parent ! whose - ! fine solution is smooth gets no child. Tagging only places boxes - conservation (restrict/reflux) is independent of + ! fine solution is smooth gets no child. Tagging only places boxes - conservation (restrict/reflux) is + ! independent of ! where they sit. np=1 + non-IB (multi-level distribution / IB nesting are future work). Regions stay in L0 cell ! indices. box_level(1:nboxes) = 1 if (amr_max_level >= 2) then ! the nesting loop below APPENDS level-l child boxes into `boxes` (up to amr_max_blocks total). The non-IB - ! path already grew `boxes` to amr_max_blocks via the tiling move_alloc; the IB path (which only merges, never + ! path already grew `boxes` to amr_max_blocks via the tiling move_alloc; the IB path (which only merges, + ! never ! grows) leaves `boxes` at the cluster count, so grow it here or the child appends overrun the allocation. if (size(boxes) < amr_max_blocks) then block @@ -4401,16 +4541,22 @@ contains end if block integer :: kb, ins(3), clo(3), chi(3), lev, plo, phi, newlo, ob, obi, ncb, kc, mlo(3), mhi(3) - integer :: mg, ng, pg, ci, cj, ck, sidx(3) - logical, allocatable :: ctag(:,:,:), gctag(:,:,:) - logical :: covered, any_tag + integer :: mg, ng, pg, nct, np_lev, nloc_send, gi, gj, gk, jrem, ntot_g + integer, allocatable :: ctags(:,:), skb(:), gkb(:) + integer(8), allocatable :: sidx(:), gidx(:) + logical, allocatable :: gwin(:,:,:), covered(:) + integer, allocatable :: mlo_all(:,:), mhi_all(:,:) + logical :: any_tag type(t_box), allocatable :: cboxes(:) #ifdef MFC_MPI - integer :: ierr + integer :: ierr, ip + integer, allocatable :: rcnt(:), rdsp(:) #endif - ! host-refresh the live (old) blocks' continuity fields: the fine sensor below reads amr_slots(ob)%q_cons on the - ! host, but the GPU_UPDATE host that the step-5 stash does runs AFTER this nesting - so the host copy is stale + ! host-refresh the live (old) blocks' continuity fields: the fine sensor below reads + ! amr_slots(ob)%q_cons on the + ! host, but the GPU_UPDATE host that the step-5 stash does runs AFTER this nesting - so the host copy is + ! stale ! here do ob = 1, amr_num_blocks if (.not. amr_owns_all(ob)) cycle ! np>1: only the owner holds this old block's fine q_cons @@ -4418,35 +4564,49 @@ contains $:GPU_UPDATE(host='[amr_slots(ob)%q_cons(obi)%sf]') end do end do - ! Fine-sensor tags accumulate in a GLOBAL L0 frame: at np>1 an old block is read only by its owner, but its - ! tag footprint can fall in ANOTHER rank's subdomain, so the local (0:m) frame the clusterer uses cannot hold - ! it. Each owner ORs its tags into gctag; an ALLREDUCE unions them; the clusterer then consumes the local slice. + ! Fine-sensor tags accumulate in a GLOBAL L0 frame: at np>1 an old block is read only by its owner, but + ! its + ! tag footprint can fall in ANOTHER rank's subdomain. Each parent's nesting window [mlo:mhi] is small vs + ! the global grid, so a WINDOW-LOCAL dense field gwin (allocated per parent below) holds each owner's + ! tags; s_amr_pack_gwin_pairs extracts them as (linear-index, kb) pairs, one per-level allgatherv unions all + ! parents' pairs across ranks, and pass 2 rebuilds each parent's window from them (no O(global-grid) tag + ! field, no local slice; the clusterer consumes the sparse per-parent list directly). mg = m_glb; ng = 0; pg = 0 if (n_glb > 0) ng = n_glb if (p_glb > 0) pg = p_glb - sidx = 0; sidx(1) = start_idx(1) - if (n_glb > 0) sidx(2) = start_idx(2) - if (p_glb > 0) sidx(3) = start_idx(3) - allocate (ctag(0:m,0:n,0:p), gctag(0:mg,0:ng,0:pg)) plo = 1; phi = nboxes ! [plo:phi] = the boxes at the previous level (lev-1) to nest inside do lev = 2, amr_max_level newlo = nboxes + 1 + ! COLLECT -> ONE COMMUNICATE -> PROCESS, per level: the per-parent cross-rank union (the old per-(lev,kb) + ! allgather) is batched into a SINGLE allgatherv per level, so the collective count is O(#levels) not + ! O(#parent-boxes). Pass 1 tags each parent's window from OWNED obs and appends this rank's tagged cells as + ! (linear-index, parent-kb) pairs; one allgatherv unions them; Pass 2 rebuilds each parent's dense window from + ! the gathered pairs whose gkb==kb, which reproduces the old per-parent dense-window dedup and the (k,j,i) + ! extraction order exactly -> each parent's ctags set (and thus its child boxes) is byte-identical. + np_lev = phi - plo + 1 + if (np_lev < 1) exit ! nothing nested at the previous level -> no deeper levels possible + allocate (covered(plo:phi), mlo_all(3,plo:phi), mhi_all(3,plo:phi)) + covered = .false. + nloc_send = 0 + ! Pass 1: collect (no comm) do kb = plo, phi - if (nboxes + 1 > amr_max_blocks) exit ! pool full - stop nesting ! nesting window: children keep an amr_cpat_mar margin from the parent boundary so their ghost ! prolongation reads valid parent interior cells mlo = boxes(kb)%lo; mhi = boxes(kb)%hi mlo(1) = mlo(1) + amr_cpat_mar; mhi(1) = mhi(1) - amr_cpat_mar if (n_glb > 0) then; mlo(2) = mlo(2) + amr_cpat_mar; mhi(2) = mhi(2) - amr_cpat_mar; end if if (p_glb > 0) then; mlo(3) = mlo(3) + amr_cpat_mar; mhi(3) = mhi(3) - amr_cpat_mar; end if + mlo_all(:,kb) = mlo; mhi_all(:,kb) = mhi if (mhi(1) < mlo(1)) cycle ! too small to nest a child in x if (n_glb > 0 .and. mhi(2) < mlo(2)) cycle if (p_glb > 0 .and. mhi(3) < mlo(3)) cycle - ! sensor-on-fine: tag from every OLD level-(lev-1) block overlapping this parent window (amr_block_level + ! sensor-on-fine: tag from every OLD level-(lev-1) block overlapping this parent window + ! (amr_block_level ! still holds the old levels here - it is reset to box_level at step 5b, below) - gctag = .false.; covered = .false.; any_tag = .false. + allocate (gwin(mlo(1):mhi(1),mlo(2):mhi(2),mlo(3):mhi(3))) + gwin = .false.; any_tag = .false. do ob = 1, amr_num_blocks if (amr_block_level(ob) /= lev - 1) cycle if (boxes(kb)%lo(1) > amr_region_hi_all(1, ob) .or. boxes(kb)%hi(1) < amr_region_lo_all(1, ob)) cycle @@ -4458,60 +4618,125 @@ contains if (boxes(kb)%lo(3) > amr_region_hi_all(3, ob) .or. boxes(kb)%hi(3) < amr_region_lo_all(3, & & ob)) cycle end if - covered = .true. ! replicated (metadata) - identical on every rank regardless of ownership - if (amr_owns_all(ob)) call s_amr_tag_child_from_fine(ob, mlo, mhi, gctag, any_tag) + covered(kb) = .true. ! replicated (metadata) - identical on every rank regardless of ownership + if (amr_owns_all(ob)) call s_amr_tag_child_from_fine(ob, mlo, mhi, gwin, any_tag) end do - ! IB: always refine the body region at this level, even where the density sensor is quiet - mark the - ! body's L0-frame bbox into gctag so it is clustered into a child (mirrors the L1 expand at :3836). - ! Containment margin = max(amr_buf, 4) + amr_cpat_mar: the child window (mlo:mhi) is the parent inset by - ! amr_cpat_mar, and clamping the tag to that window can eat up to amr_cpat_mar of the body's stencil + ! IB: always refine the body region at this level, even where the density sensor is quiet - mark + ! the + ! body's L0-frame bbox into gwin so it is clustered into a child (mirrors the L1 expand at + ! :3836). + ! Containment margin = max(amr_buf, 4) + amr_cpat_mar: the child window (mlo:mhi) is the parent + ! inset by + ! amr_cpat_mar, and clamping the tag to that window can eat up to amr_cpat_mar of the body's + ! stencil ! margin at the parent-adjacent side. The parent (widened in s_amr_expand_box_over_bodies by - ! (amr_max_level-1)*amr_cpat_mar) now clears the body by enough that this window contains the body plus - ! max(amr_buf, 4), so the tag survives the inset with a full image-point stencil of fluid on every side: - ! the body SURFACE is refined at every level and the C/F boundary sits a full stencil off it, in fluid. + ! (amr_max_level-1)*amr_cpat_mar) now clears the body by enough that this window contains the + ! body plus + ! max(amr_buf, 4), so the tag survives the inset with a full image-point stencil of fluid on + ! every side: + ! the body SURFACE is refined at every level and the C/F boundary sits a full stencil off it, in + ! fluid. if (ib) then block - integer :: ib_i, bb_lo(3), bb_hi(3), gi, gj, gk + integer :: ib_i, bb_lo(3), bb_hi(3), gii, gjj, gkk do ib_i = 1, num_ibs call s_amr_body_bbox(ib_i, max(amr_buf, 4) + amr_cpat_mar, bb_lo, bb_hi) - ! clamp the body bbox to this parent's nesting window (global L0 frame - s_amr_body_bbox + ! clamp the body bbox to this parent's nesting window (global L0 frame - + ! s_amr_body_bbox ! returns GLOBAL L0 cell indices, same frame as mlo/mhi) bb_lo = max(bb_lo, mlo); bb_hi = min(bb_hi, mhi) if (bb_hi(1) < bb_lo(1)) cycle if (n_glb > 0 .and. bb_hi(2) < bb_lo(2)) cycle if (p_glb > 0 .and. bb_hi(3) < bb_lo(3)) cycle - covered = .true. - do gk = bb_lo(3), bb_hi(3) - do gj = bb_lo(2), bb_hi(2) - do gi = bb_lo(1), bb_hi(1) - gctag(gi, gj, gk) = .true. + covered(kb) = .true. + do gkk = bb_lo(3), bb_hi(3) + do gjj = bb_lo(2), bb_hi(2) + do gii = bb_lo(1), bb_hi(1) + gwin(gii, gjj, gkk) = .true. end do end do end do end do end block end if + ! extract THIS rank's OWNED tagged cells as (linear-index, kb) pairs into the per-level send + ! arrays. The int8 linear index matches the decode in pass 2, so the gathered pairs reproduce the + ! same window coords. gwin is read here, then freed. + call s_amr_pack_gwin_pairs(gwin, mlo, mhi, mg, ng, kb, sidx, skb, nloc_send) + deallocate (gwin) + end do + + ! COMMUNICATE: one allgatherv per level (np>1) + if (.not. allocated(sidx)) then + allocate (sidx(0), skb(0)) ! this rank owned no tags at this level + end if #ifdef MFC_MPI - ! union the distributed owners' fine tags so every rank clusters the SAME child boxes (regrid must be - ! deterministic); no-op at np=1 (the single owner already holds the whole global tag field) - if (num_procs > 1) call MPI_ALLREDUCE(MPI_IN_PLACE, gctag, (mg + 1)*(ng + 1)*(pg + 1), MPI_LOGICAL, & - & MPI_LOR, MPI_COMM_WORLD, ierr) + if (num_procs > 1) then + allocate (rcnt(num_procs), rdsp(num_procs)) + call MPI_ALLGATHER(nloc_send, 1, MPI_INTEGER, rcnt, 1, MPI_INTEGER, MPI_COMM_WORLD, ierr) + rdsp(1) = 0 + do ip = 2, num_procs + rdsp(ip) = rdsp(ip - 1) + rcnt(ip - 1) + end do + ntot_g = rdsp(num_procs) + rcnt(num_procs) + allocate (gidx(max(ntot_g, 1)), gkb(max(ntot_g, 1))) + call MPI_ALLGATHERV(sidx, nloc_send, MPI_INTEGER8, gidx, rcnt, rdsp, MPI_INTEGER8, MPI_COMM_WORLD, ierr) + call MPI_ALLGATHERV(skb, nloc_send, MPI_INTEGER, gkb, rcnt, rdsp, MPI_INTEGER, MPI_COMM_WORLD, ierr) + deallocate (rcnt, rdsp) + else + call move_alloc(sidx, gidx); call move_alloc(skb, gkb) + ntot_g = nloc_send + end if +#else + call move_alloc(sidx, gidx); call move_alloc(skb, gkb) + ntot_g = nloc_send #endif - any_tag = any(gctag) ! recompute from the reduced field (a rank's local any_tag saw only its own obs) + if (allocated(sidx)) deallocate (sidx) + if (allocated(skb)) deallocate (skb) - if (covered .and. .not. any_tag) cycle ! parent's fine solution is smooth here - no child + ! Pass 2: process (no comm) + do kb = plo, phi + if (nboxes + 1 > amr_max_blocks) exit ! pool full - stop nesting + mlo = mlo_all(:,kb); mhi = mhi_all(:,kb) + if (mhi(1) < mlo(1)) cycle ! too small to nest a child in x + if (n_glb > 0 .and. mhi(2) < mlo(2)) cycle + if (p_glb > 0 .and. mhi(3) < mlo(3)) cycle - if (covered) then - ! slice the reduced global tag field into this rank's local (0:m) frame for the clusterer - do ck = 0, p - do cj = 0, n - do ci = 0, m - ctag(ci, cj, ck) = gctag(ci + sidx(1), cj + sidx(2), ck + sidx(3)) - end do - end do - end do + ! rebuild this parent's dense window from the gathered pairs whose gkb==kb: setting .true. once per + ! gathered cell reproduces the old per-parent dedup (replicated/overlapping tags collapse), and the + ! (k,j,i) sparse extract below matches the old scan order -> byte-identical ctags. + allocate (gwin(mlo(1):mhi(1),mlo(2):mhi(2),mlo(3):mhi(3))) + gwin = .false. + do i = 1, ntot_g + if (gkb(i) /= kb) cycle + gk = int(gidx(i)/(int(mg + 1, 8)*int(ng + 1, 8))) + jrem = int(gidx(i) - int(gk, 8)*int(mg + 1, 8)*int(ng + 1, 8)) + gj = jrem/(mg + 1) + gi = jrem - gj*(mg + 1) + gwin(gi, gj, gk) = .true. + end do + nct = 0 + do gk = mlo(3), mhi(3); do gj = mlo(2), mhi(2); do gi = mlo(1), mhi(1) + if (gwin(gi, gj, gk)) nct = nct + 1 + end do; end do; end do + allocate (ctags(3, max(nct, 1))) + nct = 0 + do gk = mlo(3), mhi(3); do gj = mlo(2), mhi(2); do gi = mlo(1), mhi(1) + if (gwin(gi, gj, gk)) then + nct = nct + 1 + ctags(1, nct) = gi; ctags(2, nct) = gj; ctags(3, nct) = gk + end if + end do; end do; end do + deallocate (gwin) + any_tag = nct > 0 + + ! smooth here - no child + if (covered(kb) .and. .not. any_tag) then; deallocate (ctags); cycle; end if + + if (covered(kb)) then ! cluster the fine-tagged L0 cells into child boxes, pad by amr_buf, clamp into the nesting window - call s_amr_cluster(ctag, cboxes, ncb) + call s_amr_cluster(ctags, nct, cboxes, ncb) + deallocate (ctags) do kc = 1, ncb if (nboxes + 1 > amr_max_blocks) exit clo = cboxes(kc)%lo; chi = cboxes(kc)%hi @@ -4526,28 +4751,40 @@ contains else clo(3) = 0; chi(3) = 0 end if - ! IB: a child clustered from the (widened) body tag must fully contain every overlapping body - - ! expand over bodies (mirrors the L1 expand at :3836), then re-clamp to the nesting window so + ! IB: a child clustered from the (widened) body tag must fully contain every overlapping + ! body - + ! expand over bodies (mirrors the L1 expand at :3836), then re-clamp to the nesting + ! window so ! the - ! child stays nested. Because the parent was widened by (amr_max_level-1)*amr_cpat_mar, its - ! nesting window (mlo:mhi) already contains the body plus max(amr_buf, 4), so the re-clamp does - ! NOT cut the body's stencil: the child CONTAINS the body bbox and the C/F boundary lands a full - ! image-point stencil off the surface, in fluid (surface refined, not just the interior). + ! child stays nested. Because the parent was widened by (amr_max_level-1)*amr_cpat_mar, + ! its + ! nesting window (mlo:mhi) already contains the body plus max(amr_buf, 4), so the + ! re-clamp does + ! NOT cut the body's stencil: the child CONTAINS the body bbox and the C/F boundary + ! lands a full + ! image-point stencil off the surface, in fluid (surface refined, not just the + ! interior). if (ib) then call s_amr_expand_box_over_bodies(clo, chi) clo(1) = max(clo(1), mlo(1)); chi(1) = min(chi(1), mhi(1)) if (n_glb > 0) then; clo(2) = max(clo(2), mlo(2)); chi(2) = min(chi(2), mhi(2)); end if if (p_glb > 0) then; clo(3) = max(clo(3), mlo(3)); chi(3) = min(chi(3), mhi(3)); end if end if - ! slot cap: a level->=2 block's fine grid spans 4*(its L0 extent) cells while the slot holds - ! 2*amr_maxc_fit fine cells, so a child's L0 extent must be <= amr_maxc_fit/2. In LOCK-STEP a - ! feature wider than that TILES into adjacent <= amr_maxc_fit/2 sub-blocks (like the L1 tiling): - ! the per-stage fine-fine halo (s_amr_fine_fine_halo, level-aware) matches the shared seam flux + ! slot cap: a level->=2 block's fine grid spans 4*(its L0 extent) cells while the slot + ! holds + ! 2*amr_maxc_fit fine cells, so a child's L0 extent must be <= amr_maxc_fit/2. In + ! LOCK-STEP a + ! feature wider than that TILES into adjacent <= amr_maxc_fit/2 sub-blocks (like the L1 + ! tiling): + ! the per-stage fine-fine halo (s_amr_fine_fine_halo, level-aware) matches the shared + ! seam flux ! and the L2->L1 reflux skips those fine-fine faces. SUBCYCLE advances level-2 children ! per-block - ! (s_amr_advance_children) with no L2-L2 halo, so it keeps ONE capped child (adjacent tiles + ! (s_amr_advance_children) with no L2-L2 halo, so it keeps ONE capped child (adjacent + ! tiles ! would - ! leak at their seam there - transposing that path is future work); a wide feature is under- + ! leak at their seam there - transposing that path is future work); a wide feature is + ! under- ! refined rather than non-conservative. if (amr_subcycle) then chi(1) = min(chi(1), clo(1) + amr_maxc_fit(1)/2 - 1) @@ -4572,6 +4809,7 @@ contains end do if (allocated(cboxes)) deallocate (cboxes) else + deallocate (ctags) ! brand-new region: no fine tags to cluster ! brand-new region (no old fine data yet): centred inset so the child still appears this regrid ins = 0 ins(1) = max((boxes(kb)%hi(1) - boxes(kb)%lo(1) + 1)/4, amr_cpat_mar) @@ -4585,16 +4823,17 @@ contains boxes(nboxes)%lo = clo; boxes(nboxes)%hi = chi; box_level(nboxes) = lev end if end do + deallocate (gidx, gkb, covered, mlo_all, mhi_all) ! per-level scratch - freed every level (no leak) plo = newlo; phi = nboxes ! the boxes just appended are the parents for the next level if (phi < plo) exit ! nothing nested at this level -> no deeper levels possible end do - deallocate (ctag, gctag) if (nboxes >= amr_max_blocks .and. proc_rank == 0) print '(A)', & & ' [amr] NOTE: block pool full during multi-level nesting; some boxes were not refined further' end block end if - ! 4) unchanged? (same count, boxes AND levels as the live slots -> keep them; a rebuild would reproduce them exactly + ! 4) unchanged? (same count, boxes AND levels as the live slots -> keep them; a rebuild would reproduce them + ! exactly ! anyway). The level must be compared too: a box that keeps its coordinates but changes refinement level would ! otherwise slip through with a stale amr_block_level, corrupting the level-aware coupling. if (nboxes == amr_num_blocks) then @@ -4609,11 +4848,14 @@ contains ! 5) stash every live slot's fine interior (dead-between-steps q_cons_stor bounce), keeping its old intersection origin old_np = amr_num_blocks do k = 1, old_np - ! GLOBAL block origin + extents (replicated, valid on every rank - not the owner-only isect), so the cross-rank - ! migration below and the overlap-copy's index shift are correct even where this rank did not own the old block + ! GLOBAL block origin + extents (replicated, valid on every rank - not the owner-only isect), so the + ! cross-rank + ! migration below and the overlap-copy's index shift are correct even where this rank did not own the old + ! block old_ilo(:,k) = amr_region_lo_all(:,k) old_chi(:,k) = amr_region_hi_all(:,k) ! old COARSE hi (for the P2P migration overlap test below) - ! fine extent = (2**level)*footprint - 1: a level-2 block is 4x its L0 footprint, so stashing/migrating it with the + ! fine extent = (2**level)*footprint - 1: a level-2 block is 4x its L0 footprint, so stashing/migrating it + ! with the ! level-1 factor (2x) truncates half its fine cells. Level-1 blocks (2**1 = 2) are byte-identical to before. old_ext(1, k) = (ref_ratio**amr_block_level(k))*(amr_region_hi_all(1, k) - amr_region_lo_all(1, k) + 1) - 1 old_ext(2, k) = merge((ref_ratio**amr_block_level(k))*(amr_region_hi_all(2, k) - amr_region_lo_all(2, k) + 1) - 1, 0, & @@ -4621,7 +4863,8 @@ contains old_ext(3, k) = merge((ref_ratio**amr_block_level(k))*(amr_region_hi_all(3, k) - amr_region_lo_all(3, k) + 1) - 1, 0, & & p_glb > 0) old_owner(k) = amr_block_owner(k) - old_level(k) = amr_block_level(k) ! overlap-copy must match levels: an old L2's stash is in the 4x parent-fine frame + ! overlap-copy must match levels: an old L2's stash is in the 4x parent-fine frame + old_level(k) = amr_block_level(k) old_owns(k) = amr_owns_all(k) if (old_owns(k)) then do i = 1, sys_size @@ -4652,13 +4895,18 @@ contains amr_num_blocks = nboxes do k = 1, nboxes amr_region_lo_all(:,k) = boxes(k)%lo; amr_region_hi_all(:,k) = boxes(k)%hi - ! box_level(k) is the refinement level assigned during the hierarchical nesting above (1 for the L0->L1 boxes, l for - ! a box nested at level l). Setting it every regrid resets a stale level when a slot is reused across levels. + ! box_level(k) is the refinement level assigned during the hierarchical nesting above (1 for the L0->L1 + ! boxes, l for + ! a box nested at level l). Setting it every regrid resets a stale level when a slot is reused across + ! levels. amr_block_level(k) = box_level(k) end do - ! Proper-nesting guard: each level>=2 block must be covered by EXACTLY ONE parent-level block. f_amr_parent_block (and - ! the gather/reflux that key off it) take the FIRST overlap, so a fine tile straddling two parent tiles - an internal - ! parent-level tile seam crossed by a nested feature - would silently couple to only one parent (wrong coarse BC + a + ! Proper-nesting guard: each level>=2 block must be covered by EXACTLY ONE parent-level block. + ! f_amr_parent_block (and + ! the gather/reflux that key off it) take the FIRST overlap, so a fine tile straddling two parent tiles - an + ! internal + ! parent-level tile seam crossed by a nested feature - would silently couple to only one parent (wrong coarse BC + ! + a ! conservation leak on the other). Abort fail-closed instead. Replicated boxes -> every rank aborts together. block integer :: bk, bkk, npar @@ -4678,11 +4926,16 @@ contains call s_amr_assign_block_owners() #ifdef MFC_MPI - ! Cross-rank fine-state migration: the overlap-copy below preserves each covering old block's fine detail by reading - ! amr_slots(kk)%q_cons_stor, but an old block may be owned by a rank OTHER than the one now owning a covering new block. - ! POINT-TO-POINT (mirrors s_amr_gather_coarse_patch): each old owner sends its stashed fine state ONLY to the distinct - ! new-block owners whose region overlaps that old block. A rank that did not receive old block kk never reads it - the - ! overlap-copy's per-(k,kk) index guard skips every cell of a non-overlapping pair. No-op at np=1 (single owner, local). + ! Cross-rank fine-state migration: the overlap-copy below preserves each covering old block's fine detail by + ! reading + ! amr_slots(kk)%q_cons_stor, but an old block may be owned by a rank OTHER than the one now owning a covering + ! new block. + ! POINT-TO-POINT (mirrors s_amr_gather_coarse_patch): each old owner sends its stashed fine state ONLY to the + ! distinct + ! new-block owners whose region overlaps that old block. A rank that did not receive old block kk never reads it + ! - the + ! overlap-copy's per-(k,kk) index guard skips every cell of a non-overlapping pair. No-op at np=1 (single owner, + ! local). if (num_procs > 1) then block integer :: kk, k2, ii, gi, gj, gk, idx2, ierr2, rr, maxcnt, nrq @@ -4764,14 +5017,12 @@ contains ! 6) build each new slot: geometry (collective on all ranks), prolong, then overlap-copy from every covering old slot any_xchg = .false. - if (proc_rank == 0) print '(A,I0,A)', ' [amr] regrid: ', nboxes, ' block(s)' do k = 1, nboxes amr_cur = k - if (amr_block_owner(k) == proc_rank) call s_amr_alloc_slot(k) ! owned slot needs its arrays before geometry/prolong + ! owned slot needs its arrays before geometry/prolong + if (amr_block_owner(k) == proc_rank) call s_amr_alloc_slot(k) call s_set_amr_fine_geometry(boxes(k)%lo, boxes(k)%hi) any_xchg = any_xchg .or. amr_xchg_coarse_ghosts - if (proc_rank == 0) print '(A,I0,A,I0,A,I0,A,I0,A)', ' [amr] block ', k, ': box x ', boxes(k)%lo(1), ':', & - & boxes(k)%hi(1), ' (', (boxes(k)%hi(1) - boxes(k)%lo(1) + 1), ' coarse cells)' ! fine-level distribution: gather this new block's coarse patch (collective - before the owner-only cycle; ! q_cons_base is host-current with valid ghosts from the exchange at the top of s_amr_regrid) call s_amr_gather_coarse_patch(q_cons_base, .false.) @@ -4779,11 +5030,15 @@ contains if (qbmm .and. .not. polytropic) call s_amr_gather_coarse_patch_pbmv(pb_ts(1)%sf, mv_ts(1)%sf, .false.) if (.not. amr_rank_owns_block) cycle call s_interpolate_coarse_to_fine() - ! every old block's stashed fine state is now replicated in amr_slots(kk)%q_cons_stor (migration above), so copy - ! the overlap from EVERY covering old block regardless of who owned it - sh is the old->new LOCAL fine index shift. - ! A level>=2 block SKIPS this: old_ilo/sh are the L0 index frame, but a child's amr_isect_lo is its PARENT-fine + ! every old block's stashed fine state is now replicated in amr_slots(kk)%q_cons_stor (migration above), so + ! copy + ! the overlap from EVERY covering old block regardless of who owned it - sh is the old->new LOCAL fine index + ! shift. + ! A level>=2 block SKIPS this: old_ilo/sh are the L0 index frame, but a child's amr_isect_lo is its + ! PARENT-fine ! frame, - ! so the shift is wrong. It re-prolongs from its (freshly-built, parents-first) parent each regrid instead; the + ! so the shift is wrong. It re-prolongs from its (freshly-built, parents-first) parent each regrid instead; + ! the ! coupling ! keeps conservation. Detail-preserving same-level L2 migration (parent-fine overlap) is a later increment. if (amr_block_level(amr_cur) < 2) then @@ -4844,12 +5099,14 @@ contains end do amr_xchg_coarse_ghosts = any_xchg ! coarse halo exchanged once per step if ANY block needs it ! lazy sizing: free the transient regrid slots (old blocks this rank stashed/received but does not now own); the - ! new-owned slots were allocated in the build loop, so this only frees - a rank keeps just its owned blocks' fine arrays + ! new-owned slots were allocated in the build loop, so this only frees - a rank keeps just its owned blocks' + ! fine arrays call s_amr_reconcile_slots() ! rebuild every block's fine-grid IB state for the NEW geometry (markers/ghost points/ ! image points recomputed from the body definitions; no state carries across regrids) if (ib) call s_amr_setup_ib() call s_amr_select_slot(1) + amr_seam_pairs_dirty = .true. ! block set changed: the cached seam-pair list must be rebuilt end subroutine s_amr_regrid @@ -4861,7 +5118,7 @@ contains impure subroutine s_amr_tag_child_from_fine(ob, win_lo, win_hi, ctag, any_tag) integer, intent(in) :: ob, win_lo(3), win_hi(3) - logical, intent(inout) :: ctag(0:,0:,0:) + logical, intent(inout) :: ctag(win_lo(1):,win_lo(2):,win_lo(3):) logical, intent(inout) :: any_tag integer :: rr, ci, cj, ck, fi, fj, fk, d1, d2, d3, fm1, fm2, fm3, olo(3), lo(3), hi(3) real(wp) :: r0, g @@ -4910,7 +5167,7 @@ contains !> Write the fine-level restart file for save step t_step alongside the level-0 restart (whose format stays untouched): the !! writing rank count, the active-block count, and for EACH block its box + each rank's intersection-local fine conservative !! state. Serial mode: one unformatted file per rank inside its level-0 step directory. Parallel mode: one shared MPI-IO file - !! (3-int global header [np, nboxes, sys_size], then per block a 6-int box header, a 3*np-int per-rank fine-extents record + !! (3-int global header [np, nboxes, sys_size], then per block a 7-int box+level header, a 3*np-int per-rank fine-extents record !! [m,n,p per rank, 0s for non-owners; validated on read], followed by the ranks' fine blocks concatenated in rank order). Same !! rank count + decomposition required to restart (enforced by the extents record). impure subroutine s_write_amr_restart(t_step) @@ -4920,13 +5177,14 @@ contains integer :: i, k #ifdef MFC_MPI - integer :: ifile, ierr, cnt, idx, fi, fj, fk, reg(6), ibytes, sbytes - integer :: myext(3) - integer, allocatable :: wext(:) - integer, dimension(MPI_STATUS_SIZE) :: status - integer(kind=MPI_OFFSET_KIND) :: my_cnt, my_off, tot_cnt, disp0, ddisp - logical :: file_exist - real(stp), allocatable :: buf(:) + integer :: ifile, ierr, cnt, idx, fi, fj, fk, reg(6), bhdr(7), ibytes, sbytes + integer :: myext(3) + integer, allocatable :: wext(:), myext_all(:), wext_all(:) + integer, dimension(MPI_STATUS_SIZE) :: status + integer(kind=MPI_OFFSET_KIND) :: my_cnt, my_off, disp0, ddisp + integer(kind=MPI_OFFSET_KIND), allocatable :: my_cnt_vec(:), my_off_vec(:), tot_cnt_vec(:) + logical :: file_exist + real(stp), allocatable :: buf(:) #endif if (.not. amr) return @@ -4945,7 +5203,10 @@ contains open (2, FILE=trim(file_loc), form='unformatted', STATUS='new') write (2) num_procs, amr_num_blocks, sys_size do k = 1, amr_num_blocks - write (2) amr_slots(k)%region%lo, amr_slots(k)%region%hi, amr_slots(k)%m, amr_slots(k)%n, amr_slots(k)%p + ! per-block header: region box, refinement LEVEL (a level-l block's fine extent is ref_ratio**l, + ! not ref_ratio, of the region - the reader needs the level to rebuild multi-level geometry), extents + write (2) amr_slots(k)%region%lo, amr_slots(k)%region%hi, amr_block_level(k), amr_slots(k)%m, amr_slots(k)%n, & + & amr_slots(k)%p if (amr_owns_all(k)) then do i = 1, sys_size write (2) amr_slots(k)%q_cons(i)%sf(0:amr_slots(k)%m,0:amr_slots(k)%n,0:amr_slots(k)%p) @@ -4968,31 +5229,45 @@ contains if (proc_rank == 0) call MPI_FILE_WRITE_AT(ifile, int(0, MPI_OFFSET_KIND), [num_procs, amr_num_blocks, sys_size], 3, & & MPI_INTEGER, status, ierr) disp0 = int(3*ibytes, MPI_OFFSET_KIND) ! running byte offset past the 3-int global header + ! hoist per-block metadata collectives: one EXSCAN/ALLREDUCE/ALLGATHER over ALL blocks + allocate (my_cnt_vec(amr_num_blocks), my_off_vec(amr_num_blocks), tot_cnt_vec(amr_num_blocks)) + allocate (myext_all(3*amr_num_blocks), wext_all(3*num_procs*amr_num_blocks)) do k = 1, amr_num_blocks cnt = sys_size*(amr_slots(k)%m + 1)*(amr_slots(k)%n + 1)*(amr_slots(k)%p + 1) if (.not. amr_owns_all(k)) cnt = 0 - my_cnt = int(cnt, MPI_OFFSET_KIND) - my_off = int(0, MPI_OFFSET_KIND) - call MPI_EXSCAN(my_cnt, my_off, 1, MPI_OFFSET, MPI_SUM, MPI_COMM_WORLD, ierr) - if (proc_rank == 0) my_off = int(0, MPI_OFFSET_KIND) - call MPI_ALLREDUCE(my_cnt, tot_cnt, 1, MPI_OFFSET, MPI_SUM, MPI_COMM_WORLD, ierr) + my_cnt_vec(k) = int(cnt, MPI_OFFSET_KIND) + myext_all(3*(k - 1) + 1:3*(k - 1) + 3) = 0 + if (amr_owns_all(k)) myext_all(3*(k - 1) + 1:3*(k - 1) + 3) = [amr_slots(k)%m, amr_slots(k)%n, amr_slots(k)%p] + end do + my_off_vec = int(0, MPI_OFFSET_KIND) + call MPI_EXSCAN(my_cnt_vec, my_off_vec, amr_num_blocks, MPI_OFFSET, MPI_SUM, MPI_COMM_WORLD, ierr) + if (proc_rank == 0) my_off_vec = int(0, MPI_OFFSET_KIND) + call MPI_ALLREDUCE(my_cnt_vec, tot_cnt_vec, amr_num_blocks, MPI_OFFSET, MPI_SUM, MPI_COMM_WORLD, ierr) + ! per-rank fine extents (0s for non-owning ranks): readers rebuild this vector + ! from their own decomposition and abort on mismatch - a different rank count, + ! ownership pattern, or load_balance split would otherwise silently misalign + ! the concatenated per-rank data slices below + call MPI_ALLGATHER(myext_all, 3*amr_num_blocks, MPI_INTEGER, wext_all, 3*amr_num_blocks, MPI_INTEGER, MPI_COMM_WORLD, & + & ierr) + if (.not. allocated(wext)) allocate (wext(3*num_procs)) + do k = 1, amr_num_blocks + cnt = int(my_cnt_vec(k), kind(cnt)) + my_off = my_off_vec(k) if (proc_rank == 0) then - reg(1:3) = amr_slots(k)%region%lo; reg(4:6) = amr_slots(k)%region%hi - call MPI_FILE_WRITE_AT(ifile, disp0, reg, 6, MPI_INTEGER, status, ierr) + ! 7-int per-block header: region box (6) + refinement LEVEL (a level-l block's fine extent is + ! ref_ratio**l, not ref_ratio, of the region - the reader needs the level to rebuild multi-level geometry) + bhdr(1:3) = amr_slots(k)%region%lo; bhdr(4:6) = amr_slots(k)%region%hi; bhdr(7) = amr_block_level(k) + call MPI_FILE_WRITE_AT(ifile, disp0, bhdr, 7, MPI_INTEGER, status, ierr) end if - ! per-rank fine extents (0s for non-owning ranks): readers rebuild this vector - ! from their own decomposition and abort on mismatch - a different rank count, - ! ownership pattern, or load_balance split would otherwise silently misalign - ! the concatenated per-rank data slices below - if (.not. allocated(wext)) allocate (wext(3*num_procs)) - myext = 0 - if (amr_owns_all(k)) myext = [amr_slots(k)%m, amr_slots(k)%n, amr_slots(k)%p] - call MPI_ALLGATHER(myext, 3, MPI_INTEGER, wext, 3, MPI_INTEGER, MPI_COMM_WORLD, ierr) + ! wext_all layout: rank r's extents for block k at wext_all(3*amr_num_blocks*r + 3*(k-1) + 1 : +3) + do i = 0, num_procs - 1 + wext(3*i + 1:3*i + 3) = wext_all(3*amr_num_blocks*i + 3*(k - 1) + 1:3*amr_num_blocks*i + 3*(k - 1) + 3) + end do if (proc_rank == 0) then - call MPI_FILE_WRITE_AT(ifile, disp0 + int(6*ibytes, MPI_OFFSET_KIND), wext, 3*num_procs, MPI_INTEGER, status, & + call MPI_FILE_WRITE_AT(ifile, disp0 + int(7*ibytes, MPI_OFFSET_KIND), wext, 3*num_procs, MPI_INTEGER, status, & & ierr) end if - ddisp = disp0 + int((6 + 3*num_procs)*ibytes, MPI_OFFSET_KIND) + ddisp = disp0 + int((7 + 3*num_procs)*ibytes, MPI_OFFSET_KIND) allocate (buf(max(cnt, 1))) idx = 0 do i = 1, sys_size @@ -5010,8 +5285,9 @@ contains if (ierr /= MPI_SUCCESS) & & call s_mpi_abort('amr restart write: data write failed (disk full/quota?); the file is unusable') deallocate (buf) - disp0 = ddisp + tot_cnt*int(sbytes, MPI_OFFSET_KIND) + disp0 = ddisp + tot_cnt_vec(k)*int(sbytes, MPI_OFFSET_KIND) end do + deallocate (my_cnt_vec, my_off_vec, tot_cnt_vec, myext_all, wext_all) ! the close is where buffered MPI-IO data flushes on many stacks - a failure here truncates the file call MPI_FILE_CLOSE(ifile, ierr) if (ierr /= MPI_SUCCESS) call s_mpi_abort('amr restart write: MPI_FILE_CLOSE failed; the file may be truncated') @@ -5032,16 +5308,16 @@ contains character(LEN=path_len + 3*name_len) :: file_loc character(LEN=300) :: msg logical :: file_exist - integer :: i, k, ts, have_loc, have_glb, ghdr(3), reg(6), rm, rn, rp + integer :: i, k, ts, have_loc, have_glb, ghdr(3), reg(6), lvl, rm, rn, rp logical, allocatable :: had_data(:) #ifdef MFC_MPI - integer :: ifile, ierr, cnt, idx, fi, fj, fk, ibytes, sbytes, np_old + integer :: ifile, ierr, cnt, idx, fi, fj, fk, ibytes, sbytes, np_old, bhdr(7) integer :: myext(3) - integer, allocatable :: wext(:), rext(:) + integer, allocatable :: wext(:), rext(:), myext_all(:), wext_all(:) integer, dimension(MPI_STATUS_SIZE) :: status integer(kind=MPI_OFFSET_KIND) :: my_cnt, my_off, disp0, ddisp, fsz - integer(kind=MPI_OFFSET_KIND), allocatable :: blk_base(:) + integer(kind=MPI_OFFSET_KIND), allocatable :: blk_base(:), my_cnt_vec(:), my_off_vec(:) real(stp), allocatable :: buf(:) #endif @@ -5099,20 +5375,26 @@ contains ! owner's fine state. Whole-block ownership is decomposition-deterministic, so the file's ! data-presence flag drives the read here; the owner map is rebuilt from the regions in pass 2. do k = 1, amr_num_blocks - read (2) reg, rm, rn, rp + read (2) reg, lvl, rm, rn, rp ! corrupt/foreign-file guard: a box outside the global domain would drive the geometry ! build and coordinate reads out of bounds silently in release builds if (reg(1) < 0 .or. reg(4) > m_glb .or. reg(1) > reg(4) .or. (n_glb > 0 .and. (reg(2) < 0 .or. reg(5) > n_glb & & .or. reg(2) > reg(5))) .or. (p_glb > 0 .and. (reg(3) < 0 .or. reg(6) > p_glb .or. reg(3) > reg(6)))) then call s_mpi_abort('amr restart: corrupt block record (box outside the global domain)') end if + if (lvl < 1 .or. lvl > amr_max_level) then + call s_mpi_abort('amr restart: corrupt block record (block level outside 1..amr_max_level)') + end if amr_region_lo_all(:,k) = reg(1:3); amr_region_hi_all(:,k) = reg(4:6) + ! set the level BEFORE the owner/geometry rebuild below: s_amr_assign_block_owners and + ! s_set_amr_fine_geometry key off amr_block_level to place L>=2 blocks under their parent + amr_block_level(k) = lvl had_data(k) = rm >= 0 if (had_data(k)) then - ! whole-block owner extents are region-derived (decomposition-independent); a file whose - ! stored extent disagrees is corrupt/foreign - reject before the direct read - if (rm /= ref_ratio*(reg(4) - reg(1) + 1) - 1 .or. rn /= merge(ref_ratio*(reg(5) - reg(2) + 1) - 1, 0, & - & n_glb > 0) .or. rp /= merge(ref_ratio*(reg(6) - reg(3) + 1) - 1, 0, p_glb > 0)) then + ! whole-block owner extents are region-derived per level (a level-l block covers ref_ratio**l + ! fine cells per L0 cell of its region, not ref_ratio); a stored extent that disagrees is corrupt + if (rm /= (ref_ratio**lvl)*(reg(4) - reg(1) + 1) - 1 .or. rn /= merge((ref_ratio**lvl)*(reg(5) - reg(2) + 1) & + & - 1, 0, n_glb > 0) .or. rp /= merge((ref_ratio**lvl)*(reg(6) - reg(3) + 1) - 1, 0, p_glb > 0)) then call s_mpi_abort('amr restart: block fine extents disagree with the region (corrupt file)') end if ! serial (same rank count): had_data == this run's ownership, so this is the owned slot @@ -5126,7 +5408,8 @@ contains ! PASS 2: rebuild whole-block owners from the regions, then each block's geometry under the ! correct owner; verify the data read (write-owner) matches who owns the block in this run call s_amr_assign_block_owners() - call s_amr_reconcile_slots() ! free any init slots not in the restart set (had_data slots stay: they are owned) + ! free any init slots not in the restart set (had_data slots stay: they are owned) + call s_amr_reconcile_slots() do k = 1, amr_num_blocks amr_cur = k call s_set_amr_fine_geometry(amr_region_lo_all(:,k), amr_region_hi_all(:,k)) @@ -5151,10 +5434,6 @@ contains ! so ANY new rank count can read it - pass 2 re-assigns owners for THIS run and each new owner reads its ! whole blocks. np_old == num_procs is byte-identical to the same-rank path (and keeps the layout check). np_old = ghdr(1) - if (np_old /= num_procs .and. proc_rank == 0) then - print '(A,I0,A,I0,A)', ' [amr] restart: repartitioning a ', np_old, '-rank checkpoint onto ', num_procs, & - & ' ranks (fine blocks re-assigned by this run''s SFC map)' - end if if (ghdr(3) /= sys_size) then write (msg, '(A,I0,A,I0,A)') 'amr restart sys_size mismatch: the AMR restart file has ', ghdr(3), & & ' conserved variables but this run has ', sys_size, & @@ -5173,50 +5452,76 @@ contains ! so all offsets are known before the owner map is rebuilt in pass 2. disp0 = int(3*ibytes, MPI_OFFSET_KIND) do k = 1, amr_num_blocks - call MPI_FILE_READ_AT_ALL(ifile, disp0, reg, 6, MPI_INTEGER, status, ierr) + call MPI_FILE_READ_AT_ALL(ifile, disp0, bhdr, 7, MPI_INTEGER, status, ierr) + reg = bhdr(1:6); lvl = bhdr(7) ! corrupt/foreign-file guard: a box outside the global domain would drive the geometry ! build and coordinate reads out of bounds silently in release builds if (reg(1) < 0 .or. reg(4) > m_glb .or. reg(1) > reg(4) .or. (n_glb > 0 .and. (reg(2) < 0 .or. reg(5) > n_glb & & .or. reg(2) > reg(5))) .or. (p_glb > 0 .and. (reg(3) < 0 .or. reg(6) > p_glb .or. reg(3) > reg(6)))) then call s_mpi_abort('amr restart: corrupt block record (box outside the global domain)') end if + if (lvl < 1 .or. lvl > amr_max_level) then + call s_mpi_abort('amr restart: corrupt block record (block level outside 1..amr_max_level)') + end if amr_region_lo_all(:,k) = reg(1:3); amr_region_hi_all(:,k) = reg(4:6) + ! set the level before the owner/geometry rebuild: s_amr_assign_block_owners and + ! s_set_amr_fine_geometry key off amr_block_level to place L>=2 blocks under their parent + amr_block_level(k) = lvl blk_base(k) = disp0 - cnt = sys_size*(ref_ratio*(reg(4) - reg(1) + 1))*merge(ref_ratio*(reg(5) - reg(2) + 1), 1, & - & n_glb > 0)*merge(ref_ratio*(reg(6) - reg(3) + 1), 1, p_glb > 0) - disp0 = disp0 + int((6 + 3*np_old)*ibytes, MPI_OFFSET_KIND) + int(cnt, MPI_OFFSET_KIND)*int(sbytes, MPI_OFFSET_KIND) + ! data size is region-derived per level: a level-l block covers ref_ratio**l fine cells per L0 cell + cnt = sys_size*((ref_ratio**lvl)*(reg(4) - reg(1) + 1))*merge((ref_ratio**lvl)*(reg(5) - reg(2) + 1), 1, & + & n_glb > 0)*merge((ref_ratio**lvl)*(reg(6) - reg(3) + 1), 1, p_glb > 0) + disp0 = disp0 + int((7 + 3*np_old)*ibytes, MPI_OFFSET_KIND) + int(cnt, MPI_OFFSET_KIND)*int(sbytes, MPI_OFFSET_KIND) end do ! PASS 2: rebuild whole-block owners from the regions, then per block build geometry under the ! correct owner, validate the writer's layout, and read this rank's owned slice at its offset. call s_amr_assign_block_owners() - call s_amr_reconcile_slots() ! allocate this run's owned blocks (frees any stale init slots) before the read below + ! allocate this run's owned blocks (frees any stale init slots) before the read below + call s_amr_reconcile_slots() do k = 1, amr_num_blocks amr_cur = k call s_set_amr_fine_geometry(amr_region_lo_all(:,k), amr_region_hi_all(:,k)) - ! same rank count: validate the writer's per-rank layout against this run's decomposition (a - ! re-derived load_balance split would silently misalign every rank's slice). Repartitioning - ! (np_old /= num_procs) intentionally uses a DIFFERENT decomposition, so the layout cannot match - - ! skip the check; whole-block ownership makes each block one contiguous chunk the new owner reads - ! wholly, and the file-size check below still fails closed on a truncated/corrupt file. + end do + ! hoist per-block metadata collectives: one ALLGATHER/EXSCAN over ALL blocks + allocate (my_cnt_vec(amr_num_blocks), my_off_vec(amr_num_blocks)) + allocate (myext_all(3*amr_num_blocks), wext_all(3*num_procs*amr_num_blocks)) + do k = 1, amr_num_blocks + cnt = sys_size*(amr_slots(k)%m + 1)*(amr_slots(k)%n + 1)*(amr_slots(k)%p + 1) + if (.not. amr_owns_all(k)) cnt = 0 + my_cnt_vec(k) = int(cnt, MPI_OFFSET_KIND) + myext_all(3*(k - 1) + 1:3*(k - 1) + 3) = 0 + if (amr_owns_all(k)) myext_all(3*(k - 1) + 1:3*(k - 1) + 3) = [amr_slots(k)%m, amr_slots(k)%n, amr_slots(k)%p] + end do + my_off_vec = int(0, MPI_OFFSET_KIND) + call MPI_EXSCAN(my_cnt_vec, my_off_vec, amr_num_blocks, MPI_OFFSET, MPI_SUM, MPI_COMM_WORLD, ierr) + if (proc_rank == 0) my_off_vec = int(0, MPI_OFFSET_KIND) + ! same rank count: validate the writer's per-rank layout against this run's decomposition (a + ! re-derived load_balance split would silently misalign every rank's slice). Repartitioning + ! (np_old /= num_procs) intentionally uses a DIFFERENT decomposition, so the layout cannot match - + ! skip the check; whole-block ownership makes each block one contiguous chunk the new owner reads + ! wholly, and the file-size check below still fails closed on a truncated/corrupt file. + if (np_old == num_procs) then + call MPI_ALLGATHER(myext_all, 3*amr_num_blocks, MPI_INTEGER, wext_all, 3*amr_num_blocks, MPI_INTEGER, & + & MPI_COMM_WORLD, ierr) + end if + if (.not. allocated(wext)) allocate (wext(3*np_old)) + if (.not. allocated(rext)) allocate (rext(3*num_procs)) + do k = 1, amr_num_blocks + cnt = int(my_cnt_vec(k), kind(cnt)) + my_off = my_off_vec(k) if (np_old == num_procs) then - call MPI_FILE_READ_AT_ALL(ifile, blk_base(k) + int(6*ibytes, MPI_OFFSET_KIND), wext, 3*np_old, MPI_INTEGER, & + call MPI_FILE_READ_AT_ALL(ifile, blk_base(k) + int(7*ibytes, MPI_OFFSET_KIND), wext, 3*np_old, MPI_INTEGER, & & status, ierr) - myext = 0 - if (amr_rank_owns_block) myext = [amr_slots(k)%m, amr_slots(k)%n, amr_slots(k)%p] - call MPI_ALLGATHER(myext, 3, MPI_INTEGER, rext, 3, MPI_INTEGER, MPI_COMM_WORLD, ierr) + do i = 0, num_procs - 1 + rext(3*i + 1:3*i + 3) = wext_all(3*amr_num_blocks*i + 3*(k - 1) + 1:3*amr_num_blocks*i + 3*(k - 1) + 3) + end do if (any(rext /= wext)) then call s_mpi_abort('amr restart: the per-rank fine-block layout in the file does not match ' & & // 'this run''s decomposition; with the same rank count the ownership and ' & & // '(with load_balance) the weighted splits must match the run that wrote the restart') end if end if - cnt = sys_size*(amr_slots(k)%m + 1)*(amr_slots(k)%n + 1)*(amr_slots(k)%p + 1) - if (.not. amr_rank_owns_block) cnt = 0 - my_cnt = int(cnt, MPI_OFFSET_KIND) - my_off = int(0, MPI_OFFSET_KIND) - call MPI_EXSCAN(my_cnt, my_off, 1, MPI_OFFSET, MPI_SUM, MPI_COMM_WORLD, ierr) - if (proc_rank == 0) my_off = int(0, MPI_OFFSET_KIND) - ddisp = blk_base(k) + int((6 + 3*np_old)*ibytes, MPI_OFFSET_KIND) + ddisp = blk_base(k) + int((7 + 3*np_old)*ibytes, MPI_OFFSET_KIND) allocate (buf(max(cnt, 1))) call MPI_FILE_READ_AT_ALL(ifile, ddisp + my_off*int(sbytes, MPI_OFFSET_KIND), buf, cnt*mpi_io_type, mpi_io_p, & & status, ierr) @@ -5233,7 +5538,7 @@ contains end do deallocate (buf) end do - deallocate (blk_base) + deallocate (blk_base, my_cnt_vec, my_off_vec, myext_all, wext_all) ! disp0 now equals the exact byte count a complete file must have: a truncated file (crashed ! writer, filesystem hiccup) passes every layout check above but returns short reads with ! garbage tails - fail closed instead of restoring uninitialized data as the fine level @@ -5264,153 +5569,11 @@ contains end do end if call s_amr_select_slot(1) + amr_seam_pairs_dirty = .true. ! restored a new block set: the cached seam-pair list must be rebuilt restored = .true. - if (proc_rank == 0) then - print '(A,I0,A)', ' [amr] restart: restored fine level, ', amr_num_blocks, ' block(s)' - end if end subroutine s_read_amr_restart - !> Global Sum(dV*U) for the per-fluid masses (continuity variables) and energy (eqn_idx%E) over the level-0 interior. First call - !! (finalize_report=F) stores the baselines; the finalize call prints the relative drifts (~roundoff with refluxing). - impure subroutine s_amr_conservation_defect(q_cons_base, finalize_report) - - type(scalar_field), dimension(sys_size), intent(in) :: q_cons_base - logical, intent(in) :: finalize_report - real(wp) :: sm(num_fluids_max), se, dv, s_glb - integer :: ci, cj, ck, f - - if (.not. amr) return - ! host consumer: diagnostics (host sum over exactly the summed fields). The init baseline call - ! runs BEFORE s_initialize_gpu_vars pushes the ICs to the device, so it must NOT pull the - ! (uninitialized) device copies. - if (finalize_report) then - do f = 1, num_fluids - $:GPU_UPDATE(host='[q_cons_base(f)%sf]') - end do - $:GPU_UPDATE(host='[q_cons_base(eqn_idx%E)%sf]') - end if - sm = 0._wp; se = 0._wp - do ck = 0, p - do cj = 0, n - do ci = 0, m - dv = dx(ci) - if (n_glb > 0) dv = dv*dy(cj) - if (p_glb > 0) dv = dv*dz(ck) - do f = 1, num_fluids - sm(f) = sm(f) + dv*real(q_cons_base(f)%sf(ci, cj, ck), wp) - end do - se = se + dv*real(q_cons_base(eqn_idx%E)%sf(ci, cj, ck), wp) - end do - end do - end do - if (num_procs > 1) then - do f = 1, num_fluids - call s_mpi_allreduce_sum(sm(f), s_glb); sm(f) = s_glb - end do - call s_mpi_allreduce_sum(se, s_glb); se = s_glb - end if - if (.not. finalize_report) then - amr_mass0(1:num_fluids) = sm(1:num_fluids); amr_energy0 = se - else if (proc_rank == 0) then - do f = 1, num_fluids - print '(A,I0,A,ES12.4)', ' [amr] conservation defect: mass(', f, ') drift = ', & - & abs(sm(f) - amr_mass0(f))/max(abs(amr_mass0(f)), 1.e-30_wp) - end do - print '(A,ES12.4)', ' [amr] conservation defect: energy drift = ', abs(se - amr_energy0)/max(abs(amr_energy0), & - & 1.e-30_wp) - end if - - end subroutine s_amr_conservation_defect - - !> Init-time operator verification: (b) linear reproduction, (c) restriction of an independent field. Uses - !! amr_slots(amr_cur)%q_cons(1) as scratch; called before s_populate_amr_fine overwrites it. - impure subroutine s_amr_operator_checks() - - type(scalar_field), allocatable :: cscr(:) - integer :: fi, fj, fk, ci, cj, ck, l1, l2, l3, g1, g2, g3 - real(wp) :: e, errb, errc, si_f, si_c, dvf, dvc, want, xc, yc, zc - - if (.not. amr) return - if (.not. amr_rank_owns_block) return - ! fine-level distribution: the owner's block need not lie in its coarse subdomain, so operate in the block-local patch - ! frame (the amr_cg frame: cell 0 == region_lo - nmar) and take coarse cell centres/spacings from the GLOBAL boundaries. - amr_cpat_off = 0 - amr_cpat_off(1) = amr_isect_lo(1) - amr_cpat_mar - if (n_glb > 0) amr_cpat_off(2) = amr_isect_lo(2) - amr_cpat_mar - if (p_glb > 0) amr_cpat_off(3) = amr_isect_lo(3) - amr_cpat_mar - - ! (b) fill a coarse-patch scratch with an exactly-linear field (global coords), prolong, compare pointwise - allocate (cscr(1:1)) - allocate (cscr(1)%sf(0:amr_cpat_hi(1),0:amr_cpat_hi(2),0:amr_cpat_hi(3))) - do l3 = 0, amr_cpat_hi(3) - g3 = l3 + amr_cpat_off(3); zc = 0._wp; if (p_glb > 0) zc = 0.5_wp*(amr_gzcb(g3 - 1) + amr_gzcb(g3)) - do l2 = 0, amr_cpat_hi(2) - g2 = l2 + amr_cpat_off(2); yc = 0._wp; if (n_glb > 0) yc = 0.5_wp*(amr_gycb(g2 - 1) + amr_gycb(g2)) - do l1 = 0, amr_cpat_hi(1) - g1 = l1 + amr_cpat_off(1); xc = 0.5_wp*(amr_gxcb(g1 - 1) + amr_gxcb(g1)) - cscr(1)%sf(l1, l2, l3) = 1._wp + 2._wp*xc - if (n_glb > 0) cscr(1)%sf(l1, l2, l3) = cscr(1)%sf(l1, l2, l3) + 3._wp*yc - if (p_glb > 0) cscr(1)%sf(l1, l2, l3) = cscr(1)%sf(l1, l2, l3) + 4._wp*zc - end do - end do - end do - call s_prolong_one_var(cscr(1), amr_slots(amr_cur)%q_cons(1)) - errb = 0._wp - do fk = 0, amr_slots(amr_cur)%p - do fj = 0, amr_slots(amr_cur)%n - do fi = 0, amr_slots(amr_cur)%m - want = 1._wp + 2._wp*amr_slots(amr_cur)%x_cc(fi) - if (n_glb > 0) want = want + 3._wp*amr_slots(amr_cur)%y_cc(fj) - if (p_glb > 0) want = want + 4._wp*amr_slots(amr_cur)%z_cc(fk) - e = abs(real(amr_slots(amr_cur)%q_cons(1)%sf(fi, fj, fk), wp) - want) - if (e > errb) errb = e - end do - end do - end do - - ! (c) fill the fine block with a quadratic (NOT from prolongation), restrict, compare integrals - do fk = 0, amr_slots(amr_cur)%p - do fj = 0, amr_slots(amr_cur)%n - do fi = 0, amr_slots(amr_cur)%m - amr_slots(amr_cur)%q_cons(1)%sf(fi, fj, fk) = amr_slots(amr_cur)%x_cc(fi)**2 - if (n_glb > 0) amr_slots(amr_cur)%q_cons(1)%sf(fi, fj, fk) = amr_slots(amr_cur)%q_cons(1)%sf(fi, fj, & - & fk) + amr_slots(amr_cur)%y_cc(fj)**2 - if (p_glb > 0) amr_slots(amr_cur)%q_cons(1)%sf(fi, fj, fk) = amr_slots(amr_cur)%q_cons(1)%sf(fi, fj, & - & fk) + amr_slots(amr_cur)%z_cc(fk)**2 - end do - end do - end do - call s_restrict_one_var(amr_slots(amr_cur)%q_cons(1), cscr(1)) - si_f = 0._wp; si_c = 0._wp - do fk = 0, amr_slots(amr_cur)%p - do fj = 0, amr_slots(amr_cur)%n - do fi = 0, amr_slots(amr_cur)%m - dvf = amr_slots(amr_cur)%dx(fi) - if (n_glb > 0) dvf = dvf*amr_slots(amr_cur)%dy(fj) - if (p_glb > 0) dvf = dvf*amr_slots(amr_cur)%dz(fk) - si_f = si_f + dvf*real(amr_slots(amr_cur)%q_cons(1)%sf(fi, fj, fk), wp) - end do - end do - end do - do ck = amr_isect_lo(3), merge(amr_isect_hi(3), amr_isect_lo(3), p_glb > 0) - do cj = amr_isect_lo(2), merge(amr_isect_hi(2), amr_isect_lo(2), n_glb > 0) - do ci = amr_isect_lo(1), amr_isect_hi(1) - dvc = amr_gxcb(ci) - amr_gxcb(ci - 1) ! GLOBAL coarse spacing (owner may not hold local dx here) - if (n_glb > 0) dvc = dvc*(amr_gycb(cj) - amr_gycb(cj - 1)) - if (p_glb > 0) dvc = dvc*(amr_gzcb(ck) - amr_gzcb(ck - 1)) - si_c = si_c + dvc*real(cscr(1)%sf(ci - amr_cpat_off(1), cj - amr_cpat_off(2), ck - amr_cpat_off(3)), wp) - end do - end do - end do - errc = abs(si_f - si_c)/max(abs(si_f), 1.e-30_wp) - ! every rank with fine cells prints - print '(A,ES12.4)', ' [amr] prolong linear-reproduction err = ', errb - print '(A,ES12.4)', ' [amr] restrict independent-integral err = ', errc - deallocate (cscr(1)%sf); deallocate (cscr) - - end subroutine s_amr_operator_checks - !> Total density (sum of the continuity variables) at one cell: the regrid tag field. Reduces to variable 1 for one fluid. pure function f_amr_rho_tot(q, ci, cj, ck) result(r) @@ -5585,6 +5748,8 @@ contains @:DEALLOCATE(amr_cg_mv) end if deallocate (amr_slot_live) + if (allocated(amr_seambuf_x)) deallocate (amr_seambuf_x, amr_seambuf_y) + if (allocated(amr_seam_pairs)) deallocate (amr_seam_pairs) do i = 1, sys_size @:DEALLOCATE(amr_cg(i)%sf) end do diff --git a/src/simulation/m_amr_registers.fpp b/src/simulation/m_amr_registers.fpp index adb0673b8c..0039932bc5 100644 --- a/src/simulation/m_amr_registers.fpp +++ b/src/simulation/m_amr_registers.fpp @@ -55,6 +55,14 @@ module m_amr_registers type(t_face_reg) :: freg(3) !< fine flux at the covering fine faces (0-based fine transverse) $:GPU_DECLARE(create='[creg, freg]') + !> Per-slot geometry scratch for the batched creg capture kernels (1:amr_max_blocks): host-filled from the per-slot + !! flags/overlap, then GPU_UPDATE'd so ONE kernel iterates the slot dimension instead of O(blocks) tiny launches. bactive gates + !! the slot; bt1lo/bt1hi/bt2lo/bt2hi are the per-slot transverse window (a slot outside the rectangular max caps is cycled); + !! bjlo/bjhi are the normal-face flux indices; bo1/bo2 the transverse origins; bclo/bchi the per-face capture gates. + integer, allocatable :: bjlo(:), bjhi(:), bo1(:), bo2(:), bt1lo(:), bt1hi(:), bt2lo(:), bt2hi(:) + logical, allocatable :: bclo(:), bchi(:), bactive(:) + $:GPU_DECLARE(create='[bjlo, bjhi, bo1, bo2, bt1lo, bt1hi, bt2lo, bt2hi, bclo, bchi, bactive]') + contains !> Reflux-face participation for THIS rank: own_lo(d)/own_hi(d) = it owns the coarse cell layer just OUTSIDE the block's @@ -166,109 +174,124 @@ contains @:ALLOCATE(freg(3)%lo(1:sys_size,0:max_f1,0:max_f2,1:amr_max_blocks), freg(3)%hi(1:sys_size,0:max_f1,0:max_f2, & & 1:amr_max_blocks)) end if + ! per-slot geometry scratch for the batched capture kernels (device-resident: filled on host, GPU_UPDATE'd before each call) + @:ALLOCATE(bjlo(1:amr_max_blocks), bjhi(1:amr_max_blocks), bo1(1:amr_max_blocks), bo2(1:amr_max_blocks)) + @:ALLOCATE(bt1lo(1:amr_max_blocks), bt1hi(1:amr_max_blocks), bt2lo(1:amr_max_blocks), bt2hi(1:amr_max_blocks)) + @:ALLOCATE(bclo(1:amr_max_blocks), bchi(1:amr_max_blocks), bactive(1:amr_max_blocks)) end subroutine s_initialize_amr_registers - !> Shared creg boundary-flux capture (dense eq range): creg(id)%lo/hi(eq, t1, t2, slot) [+=/=] cf * flux(face, o1+t1, o2+t2) for - !! eq in [eqb:eqe], over transverse [t1lo:t1hi] x [t2lo:t2hi]. acc=.true. accumulates, .false. overwrites (the merge picks the - !! old value or 0 with no arithmetic, so a stage-1 overwrite reads no uninitialized creg). clo/chi gate the low/high face - !! (unowned coarse faces off; child faces always on). Used for the advective (flux_dir, eqb=1..sys_size) and viscous (flux_src, - !! eqb=mom..E) captures on BOTH the coarse-self (islot) and child (kc) sides - see s_amr_capture_boundary_flux. - impure subroutine s_amr_capture_creg_dense(slot, id, flux, cf, acc, clo, chi, jlo, jhi, o1, o2, t1lo, t1hi, t2lo, t2hi, eqb, & - & eqe) - - integer, intent(in) :: slot, id, jlo, jhi, o1, o2, t1lo, t1hi, t2lo, t2hi, eqb, eqe + !> Shared creg boundary-flux capture (dense eq range), BATCHED over the slot dimension: for each active slot in [1:nb], + !! creg(id)%lo/hi(eq, t1, t2, slot) [+=/=] cf * flux(face, bo1(slot)+t1, bo2(slot)+t2) for eq in [eqb:eqe], over the per-slot + !! transverse window [bt1lo:bt1hi] x [bt2lo:bt2hi]. acc=.true. accumulates, .false. overwrites (the merge picks the old value or + !! 0 with no arithmetic, so a stage-1 overwrite reads no uninitialized creg). bclo/bchi gate the low/high face (unowned coarse + !! faces off; child faces always on). The device kernel collapses (slot, t2, t1, eq) over the rectangular caps + !! [0:maxt2]x[0:maxt1] (max over slots) and cycles inactive slots / out-of-window cells - one launch replaces the O(blocks) + !! per-slot launches. The per-slot geometry (bjlo etc.) is filled on host and GPU_UPDATE'd by the caller. Used for the advective + !! (flux_dir, eqb=1..sys_size) and viscous (flux_src, eqb=mom..E) captures on BOTH the coarse-self and child sides. + impure subroutine s_amr_capture_creg_dense_batch(nb, id, flux, cf, acc, maxt1, maxt2, eqb, eqe) + + integer, intent(in) :: nb, id, maxt1, maxt2, eqb, eqe type(vector_field), intent(in) :: flux real(wp), intent(in) :: cf - logical, intent(in) :: acc, clo, chi - integer :: eq, t1, t2 - - $:GPU_PARALLEL_LOOP(collapse=3) - do t2 = t2lo, t2hi - do t1 = t1lo, t1hi - do eq = eqb, eqe - select case (id) - case (1) - if (clo) creg(1)%lo(eq, t1, t2, slot) = merge(creg(1)%lo(eq, t1, t2, slot), 0._wp, & - & acc) + cf*real(flux%vf(eq)%sf(jlo, o1 + t1, o2 + t2), wp) - if (chi) creg(1)%hi(eq, t1, t2, slot) = merge(creg(1)%hi(eq, t1, t2, slot), 0._wp, & - & acc) + cf*real(flux%vf(eq)%sf(jhi, o1 + t1, o2 + t2), wp) - case (2) - if (clo) creg(2)%lo(eq, t1, t2, slot) = merge(creg(2)%lo(eq, t1, t2, slot), 0._wp, & - & acc) + cf*real(flux%vf(eq)%sf(o1 + t1, jlo, o2 + t2), wp) - if (chi) creg(2)%hi(eq, t1, t2, slot) = merge(creg(2)%hi(eq, t1, t2, slot), 0._wp, & - & acc) + cf*real(flux%vf(eq)%sf(o1 + t1, jhi, o2 + t2), wp) - case (3) - if (clo) creg(3)%lo(eq, t1, t2, slot) = merge(creg(3)%lo(eq, t1, t2, slot), 0._wp, & - & acc) + cf*real(flux%vf(eq)%sf(o1 + t1, o2 + t2, jlo), wp) - if (chi) creg(3)%hi(eq, t1, t2, slot) = merge(creg(3)%hi(eq, t1, t2, slot), 0._wp, & - & acc) + cf*real(flux%vf(eq)%sf(o1 + t1, o2 + t2, jhi), wp) - end select + logical, intent(in) :: acc + integer :: eq, t1, t2, slot + + $:GPU_PARALLEL_LOOP(collapse=4) + do slot = 1, nb + do t2 = 0, maxt2 + do t1 = 0, maxt1 + do eq = eqb, eqe + if (.not. bactive(slot)) cycle + if (t1 < bt1lo(slot) .or. t1 > bt1hi(slot) .or. t2 < bt2lo(slot) .or. t2 > bt2hi(slot)) cycle + select case (id) + case (1) + if (bclo(slot)) creg(1)%lo(eq, t1, t2, slot) = merge(creg(1)%lo(eq, t1, t2, slot), 0._wp, & + & acc) + cf*real(flux%vf(eq)%sf(bjlo(slot), bo1(slot) + t1, bo2(slot) + t2), wp) + if (bchi(slot)) creg(1)%hi(eq, t1, t2, slot) = merge(creg(1)%hi(eq, t1, t2, slot), 0._wp, & + & acc) + cf*real(flux%vf(eq)%sf(bjhi(slot), bo1(slot) + t1, bo2(slot) + t2), wp) + case (2) + if (bclo(slot)) creg(2)%lo(eq, t1, t2, slot) = merge(creg(2)%lo(eq, t1, t2, slot), 0._wp, & + & acc) + cf*real(flux%vf(eq)%sf(bo1(slot) + t1, bjlo(slot), bo2(slot) + t2), wp) + if (bchi(slot)) creg(2)%hi(eq, t1, t2, slot) = merge(creg(2)%hi(eq, t1, t2, slot), 0._wp, & + & acc) + cf*real(flux%vf(eq)%sf(bo1(slot) + t1, bjhi(slot), bo2(slot) + t2), wp) + case (3) + if (bclo(slot)) creg(3)%lo(eq, t1, t2, slot) = merge(creg(3)%lo(eq, t1, t2, slot), 0._wp, & + & acc) + cf*real(flux%vf(eq)%sf(bo1(slot) + t1, bo2(slot) + t2, bjlo(slot)), wp) + if (bchi(slot)) creg(3)%hi(eq, t1, t2, slot) = merge(creg(3)%hi(eq, t1, t2, slot), 0._wp, & + & acc) + cf*real(flux%vf(eq)%sf(bo1(slot) + t1, bo2(slot) + t2, bjhi(slot)), wp) + end select + end do end do end do end do $:END_GPU_PARALLEL_LOOP() - end subroutine s_amr_capture_creg_dense + end subroutine s_amr_capture_creg_dense_batch - !> Shared creg boundary-flux capture (chemistry species diffusion): always-accumulate the species mass fluxes, plus the energy - !! flux only when NOT viscous (the viscous pass already captured flux_src(E)). Species use a seq inner loop (a runtime range). - !! Used for the chem capture on BOTH the coarse-self and child sides. - impure subroutine s_amr_capture_creg_chem(slot, id, flux, cf, clo, chi, jlo, jhi, o1, o2, t1lo, t1hi, t2lo, t2hi) + !> Shared creg boundary-flux capture (chemistry species diffusion), BATCHED over the slot dimension: always-accumulate the + !! species mass fluxes, plus the energy flux only when NOT viscous (the viscous pass already captured flux_src(E)). Species use + !! a seq inner loop (a runtime range). The device kernel collapses (slot, t2, t1) over the rectangular caps [0:maxt2]x[0:maxt1] + !! (max over slots) and cycles inactive slots / out-of-window cells. Per-slot geometry is host-filled + GPU_UPDATE'd by the + !! caller. Used for the chem capture on BOTH the coarse-self and child sides. + impure subroutine s_amr_capture_creg_chem_batch(nb, id, flux, cf, maxt1, maxt2) - integer, intent(in) :: slot, id, jlo, jhi, o1, o2, t1lo, t1hi, t2lo, t2hi + integer, intent(in) :: nb, id, maxt1, maxt2 type(vector_field), intent(in) :: flux real(wp), intent(in) :: cf - logical, intent(in) :: clo, chi - integer :: eq, t1, t2 - - $:GPU_PARALLEL_LOOP(collapse=2) - do t2 = t2lo, t2hi - do t1 = t1lo, t1hi - $:GPU_LOOP(parallelism='[seq]') - do eq = eqn_idx%species%beg, eqn_idx%species%end - select case (id) - case (1) - if (clo) creg(1)%lo(eq, t1, t2, slot) = creg(1)%lo(eq, t1, t2, slot) + cf*real(flux%vf(eq)%sf(jlo, & - & o1 + t1, o2 + t2), wp) - if (chi) creg(1)%hi(eq, t1, t2, slot) = creg(1)%hi(eq, t1, t2, slot) + cf*real(flux%vf(eq)%sf(jhi, & - & o1 + t1, o2 + t2), wp) - case (2) - if (clo) creg(2)%lo(eq, t1, t2, slot) = creg(2)%lo(eq, t1, t2, slot) + cf*real(flux%vf(eq)%sf(o1 + t1, & - & jlo, o2 + t2), wp) - if (chi) creg(2)%hi(eq, t1, t2, slot) = creg(2)%hi(eq, t1, t2, slot) + cf*real(flux%vf(eq)%sf(o1 + t1, & - & jhi, o2 + t2), wp) - case (3) - if (clo) creg(3)%lo(eq, t1, t2, slot) = creg(3)%lo(eq, t1, t2, slot) + cf*real(flux%vf(eq)%sf(o1 + t1, & - & o2 + t2, jlo), wp) - if (chi) creg(3)%hi(eq, t1, t2, slot) = creg(3)%hi(eq, t1, t2, slot) + cf*real(flux%vf(eq)%sf(o1 + t1, & - & o2 + t2, jhi), wp) - end select + integer :: eq, t1, t2, slot + + $:GPU_PARALLEL_LOOP(collapse=3) + do slot = 1, nb + do t2 = 0, maxt2 + do t1 = 0, maxt1 + if (.not. bactive(slot)) cycle + if (t1 < bt1lo(slot) .or. t1 > bt1hi(slot) .or. t2 < bt2lo(slot) .or. t2 > bt2hi(slot)) cycle + $:GPU_LOOP(parallelism='[seq]') + do eq = eqn_idx%species%beg, eqn_idx%species%end + select case (id) + case (1) + if (bclo(slot)) creg(1)%lo(eq, t1, t2, slot) = creg(1)%lo(eq, t1, t2, & + & slot) + cf*real(flux%vf(eq)%sf(bjlo(slot), bo1(slot) + t1, bo2(slot) + t2), wp) + if (bchi(slot)) creg(1)%hi(eq, t1, t2, slot) = creg(1)%hi(eq, t1, t2, & + & slot) + cf*real(flux%vf(eq)%sf(bjhi(slot), bo1(slot) + t1, bo2(slot) + t2), wp) + case (2) + if (bclo(slot)) creg(2)%lo(eq, t1, t2, slot) = creg(2)%lo(eq, t1, t2, & + & slot) + cf*real(flux%vf(eq)%sf(bo1(slot) + t1, bjlo(slot), bo2(slot) + t2), wp) + if (bchi(slot)) creg(2)%hi(eq, t1, t2, slot) = creg(2)%hi(eq, t1, t2, & + & slot) + cf*real(flux%vf(eq)%sf(bo1(slot) + t1, bjhi(slot), bo2(slot) + t2), wp) + case (3) + if (bclo(slot)) creg(3)%lo(eq, t1, t2, slot) = creg(3)%lo(eq, t1, t2, & + & slot) + cf*real(flux%vf(eq)%sf(bo1(slot) + t1, bo2(slot) + t2, bjlo(slot)), wp) + if (bchi(slot)) creg(3)%hi(eq, t1, t2, slot) = creg(3)%hi(eq, t1, t2, & + & slot) + cf*real(flux%vf(eq)%sf(bo1(slot) + t1, bo2(slot) + t2, bjhi(slot)), wp) + end select + end do + if (.not. viscous) then + select case (id) + case (1) + if (bclo(slot)) creg(1)%lo(eqn_idx%E, t1, t2, slot) = creg(1)%lo(eqn_idx%E, t1, t2, & + & slot) + cf*real(flux%vf(eqn_idx%E)%sf(bjlo(slot), bo1(slot) + t1, bo2(slot) + t2), wp) + if (bchi(slot)) creg(1)%hi(eqn_idx%E, t1, t2, slot) = creg(1)%hi(eqn_idx%E, t1, t2, & + & slot) + cf*real(flux%vf(eqn_idx%E)%sf(bjhi(slot), bo1(slot) + t1, bo2(slot) + t2), wp) + case (2) + if (bclo(slot)) creg(2)%lo(eqn_idx%E, t1, t2, slot) = creg(2)%lo(eqn_idx%E, t1, t2, & + & slot) + cf*real(flux%vf(eqn_idx%E)%sf(bo1(slot) + t1, bjlo(slot), bo2(slot) + t2), wp) + if (bchi(slot)) creg(2)%hi(eqn_idx%E, t1, t2, slot) = creg(2)%hi(eqn_idx%E, t1, t2, & + & slot) + cf*real(flux%vf(eqn_idx%E)%sf(bo1(slot) + t1, bjhi(slot), bo2(slot) + t2), wp) + case (3) + if (bclo(slot)) creg(3)%lo(eqn_idx%E, t1, t2, slot) = creg(3)%lo(eqn_idx%E, t1, t2, & + & slot) + cf*real(flux%vf(eqn_idx%E)%sf(bo1(slot) + t1, bo2(slot) + t2, bjlo(slot)), wp) + if (bchi(slot)) creg(3)%hi(eqn_idx%E, t1, t2, slot) = creg(3)%hi(eqn_idx%E, t1, t2, & + & slot) + cf*real(flux%vf(eqn_idx%E)%sf(bo1(slot) + t1, bo2(slot) + t2, bjhi(slot)), wp) + end select + end if end do - if (.not. viscous) then - select case (id) - case (1) - if (clo) creg(1)%lo(eqn_idx%E, t1, t2, slot) = creg(1)%lo(eqn_idx%E, t1, t2, & - & slot) + cf*real(flux%vf(eqn_idx%E)%sf(jlo, o1 + t1, o2 + t2), wp) - if (chi) creg(1)%hi(eqn_idx%E, t1, t2, slot) = creg(1)%hi(eqn_idx%E, t1, t2, & - & slot) + cf*real(flux%vf(eqn_idx%E)%sf(jhi, o1 + t1, o2 + t2), wp) - case (2) - if (clo) creg(2)%lo(eqn_idx%E, t1, t2, slot) = creg(2)%lo(eqn_idx%E, t1, t2, & - & slot) + cf*real(flux%vf(eqn_idx%E)%sf(o1 + t1, jlo, o2 + t2), wp) - if (chi) creg(2)%hi(eqn_idx%E, t1, t2, slot) = creg(2)%hi(eqn_idx%E, t1, t2, & - & slot) + cf*real(flux%vf(eqn_idx%E)%sf(o1 + t1, jhi, o2 + t2), wp) - case (3) - if (clo) creg(3)%lo(eqn_idx%E, t1, t2, slot) = creg(3)%lo(eqn_idx%E, t1, t2, & - & slot) + cf*real(flux%vf(eqn_idx%E)%sf(o1 + t1, o2 + t2, jlo), wp) - if (chi) creg(3)%hi(eqn_idx%E, t1, t2, slot) = creg(3)%hi(eqn_idx%E, t1, t2, & - & slot) + cf*real(flux%vf(eqn_idx%E)%sf(o1 + t1, o2 + t2, jhi), wp) - end select - end if end do end do $:END_GPU_PARALLEL_LOOP() - end subroutine s_amr_capture_creg_chem + end subroutine s_amr_capture_creg_chem_batch !> Capture the c/f boundary-face fluxes for direction id from the just-finalized flux array. Runs INSIDE s_compute_rhs: coarse !! call (amr_in_fine_advance false, coarse globals) fills creg at the block boundary faces; fine call (flag true, globals @@ -281,7 +304,7 @@ contains type(vector_field), intent(in) :: flux_src integer, intent(in) :: stage integer :: eq, t1, t2, jlo, jhi, t1_lo, t1_hi, t2_lo, t2_hi, o1, o2, islot, save_cur - integer :: sidx(3), ext(3), tlo(3), thi(3), kc, dch + integer :: sidx(3), ext(3), tlo(3), thi(3), kc, dch, maxt1, maxt2 logical :: own_lo(3), own_hi(3), cap_lo, cap_hi real(wp) :: coef, ccoef logical :: accum, cacc, is_child @@ -439,7 +462,11 @@ contains ! (s_amr_reflux_to_parent). Captures the TOTAL flux - advective (flux_dir), then viscous (flux_src, mom..E), then ! chemistry species+energy - mirroring the coarse-self branch below, so viscous/chemistry multi-level conserves (no ! checker gate). np=1 (children co-owned with the parent); np>=2 P2P delivery is future work. + ! fill the per-slot (per-child) geometry, then issue ONE batched kernel per capture category. Each child is its OWN creg + ! slot (slot=kc); both faces always owned (child co-located), t1lo=t2lo=0. ccoef = rk3_w(stage); cacc = (stage > 1) + bactive = .false. + maxt1 = 0; maxt2 = 0 do kc = 1, amr_num_blocks if (amr_block_level(kc) /= amr_block_level(amr_cur) + 1 .or. .not. amr_owns_all(kc)) cycle is_child = .true. @@ -459,15 +486,20 @@ contains o1 = amr_isect_lo_all(1, kc); t1_hi = amr_isect_hi_all(1, kc) - amr_isect_lo_all(1, kc) o2 = amr_isect_lo_all(2, kc); t2_hi = amr_isect_hi_all(2, kc) - amr_isect_lo_all(2, kc) end select - ! shared capture into this CHILD's creg (parent-fine frame, both faces always owned since child is co-located): - ! advective, then total-flux viscous, then chemistry species+energy. - call s_amr_capture_creg_dense(kc, id, flux_dir, ccoef, cacc, .true., .true., jlo, jhi, o1, o2, 0, t1_hi, 0, & - & t2_hi, 1, sys_size) - if (viscous) call s_amr_capture_creg_dense(kc, id, flux_src, ccoef, .true., .true., .true., jlo, jhi, o1, o2, 0, & - & t1_hi, 0, t2_hi, eqn_idx%mom%beg, eqn_idx%E) - if (chemistry .and. chem_params%diffusion) call s_amr_capture_creg_chem(kc, id, flux_src, ccoef, .true., .true., & - & jlo, jhi, o1, o2, 0, t1_hi, 0, t2_hi) + bactive(kc) = .true.; bclo(kc) = .true.; bchi(kc) = .true. + bjlo(kc) = jlo; bjhi(kc) = jhi; bo1(kc) = o1; bo2(kc) = o2 + bt1lo(kc) = 0; bt1hi(kc) = t1_hi; bt2lo(kc) = 0; bt2hi(kc) = t2_hi + maxt1 = max(maxt1, t1_hi); maxt2 = max(maxt2, t2_hi) end do + if (any(bactive(1:amr_num_blocks))) then + $:GPU_UPDATE(device='[bjlo, bjhi, bo1, bo2, bt1lo, bt1hi, bt2lo, bt2hi, bclo, bchi, bactive]') + ! shared capture into each CHILD's creg (parent-fine frame): advective, then total-flux viscous, then chemistry. + call s_amr_capture_creg_dense_batch(amr_num_blocks, id, flux_dir, ccoef, cacc, maxt1, maxt2, 1, sys_size) + if (viscous) call s_amr_capture_creg_dense_batch(amr_num_blocks, id, flux_src, ccoef, .true., maxt1, maxt2, & + & eqn_idx%mom%beg, eqn_idx%E) + if (chemistry .and. chem_params%diffusion) call s_amr_capture_creg_chem_batch(amr_num_blocks, id, flux_src, & + & ccoef, maxt1, maxt2) + end if else ! coarse branch: a face's capture runs on the rank owning the coarse cells just OUTSIDE it (its ! flux_n covers that face; at a rank-interior face the same rank also holds the inside cells). @@ -477,6 +509,8 @@ contains ! intersection is the block and both flags hold, recovering the single-rank behavior exactly. ! ONE coarse s_compute_rhs pass fills EVERY active block's registers: revisit each slot's region+intersection in turn. save_cur = amr_cur + bactive = .false. + maxt1 = 0; maxt2 = 0 do islot = 1, amr_num_blocks ! a level>=2 block's coarse side is its PARENT (creg captured in the fine branch), not L0 if (amr_block_level(islot) >= 2) cycle @@ -497,17 +531,23 @@ contains t1_lo = tlo(1) - amr_region_lo(1); t1_hi = thi(1) - amr_region_lo(1); o1 = amr_region_lo(1) - sidx(1) t2_lo = tlo(2) - amr_region_lo(2); t2_hi = thi(2) - amr_region_lo(2); o2 = amr_region_lo(2) - sidx(2) end select - ! shared capture into this coarse block's creg (region/sidx frame, per-face ownership gating): advective, then - ! total-flux viscous, then chemistry species+energy. - call s_amr_capture_creg_dense(islot, id, flux_dir, coef, accum, cap_lo, cap_hi, jlo, jhi, o1, o2, t1_lo, & - & t1_hi, t2_lo, t2_hi, 1, sys_size) - if (viscous) call s_amr_capture_creg_dense(islot, id, flux_src, coef, .true., cap_lo, cap_hi, jlo, jhi, o1, & - & o2, t1_lo, t1_hi, t2_lo, t2_hi, eqn_idx%mom%beg, eqn_idx%E) - if (chemistry .and. chem_params%diffusion) call s_amr_capture_creg_chem(islot, id, flux_src, coef, cap_lo, & - & cap_hi, jlo, jhi, o1, o2, t1_lo, t1_hi, t2_lo, t2_hi) + bactive(islot) = .true.; bclo(islot) = cap_lo; bchi(islot) = cap_hi + bjlo(islot) = jlo; bjhi(islot) = jhi; bo1(islot) = o1; bo2(islot) = o2 + bt1lo(islot) = t1_lo; bt1hi(islot) = t1_hi; bt2lo(islot) = t2_lo; bt2hi(islot) = t2_hi + maxt1 = max(maxt1, t1_hi); maxt2 = max(maxt2, t2_hi) end if ! cap_lo .or. cap_hi end do call s_amr_select_slot(save_cur) + if (any(bactive(1:amr_num_blocks))) then + $:GPU_UPDATE(device='[bjlo, bjhi, bo1, bo2, bt1lo, bt1hi, bt2lo, bt2hi, bclo, bchi, bactive]') + ! shared capture into each coarse block's creg (region/sidx frame, per-face ownership gating): advective, then + ! total-flux viscous, then chemistry species+energy. + call s_amr_capture_creg_dense_batch(amr_num_blocks, id, flux_dir, coef, accum, maxt1, maxt2, 1, sys_size) + if (viscous) call s_amr_capture_creg_dense_batch(amr_num_blocks, id, flux_src, coef, .true., maxt1, maxt2, & + & eqn_idx%mom%beg, eqn_idx%E) + if (chemistry .and. chem_params%diffusion) call s_amr_capture_creg_chem_batch(amr_num_blocks, id, flux_src, coef, & + & maxt1, maxt2) + end if end if end subroutine s_amr_capture_boundary_flux @@ -529,8 +569,8 @@ contains if (igr) return ! stage-1 IGR: restriction-only coupling (no captured fluxes) islot = amr_cur ! working block slot (local => captured by value in the device kernels below) rr = ref_ratio - ! per-face participation: each face's correction runs on the rank owning its OUTSIDE cell layer (all faces at np=1); - ! the owner's freg is broadcast to every participant by s_mpi_bcast_amr_reflux_faces before this is called + ! per-face participation: each face's correction runs on the rank owning its OUTSIDE cell layer (all faces at np=1). + ! Under whole-block ownership the block's freg is already resident on its owner, so no broadcast is needed here. call s_amr_reflux_face_flags(sidx, ext, own_lo, own_hi, tlo, thi) if (.not. (any(own_lo) .or. any(own_hi))) return ! device kernels: the coarse rhs stays device-resident for the coarse RK update kernel @@ -831,6 +871,7 @@ contains @:DEALLOCATE(freg(d)%lo, freg(d)%hi) end if end do + @:DEALLOCATE(bjlo, bjhi, bo1, bo2, bt1lo, bt1hi, bt2lo, bt2hi, bclo, bchi, bactive) end subroutine s_finalize_amr_registers diff --git a/src/simulation/m_checker.fpp b/src/simulation/m_checker.fpp index 3a9bbc864b..42007fcc6f 100644 --- a/src/simulation/m_checker.fpp +++ b/src/simulation/m_checker.fpp @@ -43,8 +43,6 @@ contains if (active_box) then @:PROHIBIT(recon_type /= recon_type_weno, "active_box requires WENO reconstruction") @:PROHIBIT(ib, "active_box is incompatible with immersed boundaries") - @:PROHIBIT(hybrid_weno .or. hybrid_riemann, & - & "active_box is incompatible with hybrid_weno/hybrid_riemann: the smoothness sensor sweeps the full domain but the cons-to-prim conversion is narrowed to the active-box footprint, so the sensor would read unconverted primitives outside it") @:PROHIBIT(acoustic_source, "active_box is incompatible with acoustic sources") @:PROHIBIT(bodyForces, "active_box is incompatible with body forces") @:PROHIBIT(bubbles_lagrange, "active_box is incompatible with Lagrangian bubbles") @@ -69,27 +67,6 @@ contains @:PROHIBIT(sfc_partition_wrt .and. partition_tile_size < 1, "partition_tile_size must be >= 1") @:PROHIBIT(load_balance .and. .not. parallel_io, "load_balance requires parallel_io = T") @:PROHIBIT(load_balance .and. num_procs == 1, "load_balance requires more than one MPI rank") - @:PROHIBIT(hybrid_weno .and. recon_type /= recon_type_weno, "hybrid_weno requires WENO reconstruction") - @:PROHIBIT(hybrid_weno .and. weno_order == 1, "hybrid_weno requires weno_order > 1") - @:PROHIBIT(hybrid_weno .and. hybrid_weno_eps <= 0._wp, "hybrid_weno_eps must be > 0") - @:PROHIBIT(hybrid_weno .and. igr, "hybrid_weno is incompatible with the IGR solver") - @:PROHIBIT(hybrid_riemann .and. recon_type /= recon_type_weno, & - & "hybrid_riemann requires WENO reconstruction (the shared sensor lives in the WENO module)") - @:PROHIBIT(hybrid_riemann .and. weno_order == 1, "hybrid_riemann requires weno_order > 1") - @:PROHIBIT(hybrid_riemann .and. .not. (model_eqns == model_eqns_5eq .or. model_eqns == model_eqns_6eq), & - & "hybrid_riemann supports only the 5- and 6-equation models") - @:PROHIBIT(hybrid_riemann .and. (hybrid_smooth_flux < 1 .or. hybrid_smooth_flux > 2), & - & "hybrid_smooth_flux must be 1 (central) or 2 (Rusanov)") - @:PROHIBIT(hybrid_riemann .and. igr, "hybrid_riemann is incompatible with the IGR solver") - @:PROHIBIT(hybrid_riemann .and. (viscous .or. surface_tension .or. hypoelasticity .or. hyperelasticity .or. elasticity), & - & "hybrid_riemann does not support viscous/elastic/surface-tension physics") - @:PROHIBIT(hybrid_riemann .and. (bubbles_euler .or. bubbles_lagrange .or. qbmm), & - & "hybrid_riemann does not support bubble models") - @:PROHIBIT(hybrid_riemann .and. chemistry, "hybrid_riemann does not support chemistry") - @:PROHIBIT(hybrid_riemann .and. cyl_coord, & - & "hybrid_riemann does not support cylindrical/axisymmetric (no smooth-flux geometric source)") - @:PROHIBIT(hybrid_riemann .and. low_Mach /= 0, & - & "hybrid_riemann (cheap central/Rusanov flux) is incompatible with the low_Mach correction") if (amr) then @:PROHIBIT((.not. igr) .and. recon_type /= recon_type_weno, "amr requires WENO reconstruction (or the IGR solver)") @@ -168,10 +145,6 @@ contains ! inside the monotonically-growing active window (init check + regrid clamp; the ! windowed coarse update would drop reflux corrections at faces outside it), and ! the fine advance disables the coarse-indexed windowing on the swapped block grid - ! no hybrid_weno/hybrid_riemann gate: the sensor arrays are sized to the coarse - ! idwbuff (the fine extent guard keeps fine bounds inside them) and the sensor is - ! recomputed from the live (swapped) idwbuff every RHS call, so each level evaluates - ! its own sensor with conservative full-WENO defaults at its buffer edges. ! no acoustic_source gate here: acoustic sources act on the coarse grid only (their spatial support is precomputed as ! coarse cell indices). A startup check aborts if the support overlaps the user-placed ! initial block; the dynamic regrid keeps its own boxes clear of the support (tags are diff --git a/src/simulation/m_global_parameters.fpp b/src/simulation/m_global_parameters.fpp index 6a8d70f514..c9a10873b8 100644 --- a/src/simulation/m_global_parameters.fpp +++ b/src/simulation/m_global_parameters.fpp @@ -498,7 +498,6 @@ contains sfc_partition_wrt = .false. load_balance = .false. rank_time_wrt = .false. - hybrid_weno_eps = 1.0e-2_wp amr = .false. amr_block_beg(:) = 0 amr_block_end(:) = 0 @@ -510,7 +509,6 @@ contains amr_max_level = 1 amr_cluster_eff = 0.7_wp ref_ratio = 2 - hybrid_smooth_flux = 2 partition_tile_size = 8 many_ib_patch_parallelism = .false. @@ -522,8 +520,6 @@ contains #:if not MFC_CASE_OPTIMIZATION nb = 1 muscl_lim = dflt_int - hybrid_weno = .false. - hybrid_riemann = .false. #:endif adv_n = .false. @@ -966,7 +962,6 @@ contains $:GPU_UPDATE(device='[muscl_order, muscl_lim]') $:GPU_UPDATE(device='[igr, igr_order]') $:GPU_UPDATE(device='[num_fluids, num_dims, viscous, num_vels, nb, muscl_lim]') - $:GPU_UPDATE(device='[hybrid_weno, hybrid_riemann, hybrid_weno_eps, hybrid_smooth_flux]') #:endif $:GPU_UPDATE(device='[int_comp, ic_eps, ic_beta]') diff --git a/src/simulation/m_ibm.fpp b/src/simulation/m_ibm.fpp index abb9635d51..ade38be295 100644 --- a/src/simulation/m_ibm.fpp +++ b/src/simulation/m_ibm.fpp @@ -725,6 +725,9 @@ contains ! Sorting on the device is deliberate: a host round-trip here needs a GPU_UPDATE of the declare-target ! ghost_points, which fails Cray OpenACC's present-table lookup and aborts CCE OpenMP-offload with ! lib-4425 in the AMR fine path (this routine runs mid-swap, see s_ibm_swap_to_fine). + ! Non-AMR runs keep the original (unsorted) ghost order - only moving AMR-IB rebuilds the + ! list on-device per substep, so only it needs the deterministic ordering. + if (.not. amr) return $:GPU_PARALLEL_LOOP(private='[a, b, tmp, less]') do local_idx = 1, 1 do a = 2, num_gps @@ -1839,8 +1842,6 @@ contains call s_compute_image_points(ghost_points) call s_compute_interpolation_coeffs(ghost_points) - if (num_gps > 0) print '(A,I0,A,I0)', ' [amr] block ', amr_cur, ': fine IB ghost points = ', num_gps - end subroutine s_ibm_setup_fine !> Finalize the IBM module diff --git a/src/simulation/m_load_balance.fpp b/src/simulation/m_load_balance.fpp index 294483f231..26f64a7f19 100644 --- a/src/simulation/m_load_balance.fpp +++ b/src/simulation/m_load_balance.fpp @@ -91,7 +91,9 @@ contains inquire (FILE=trim(file_loc), EXIST=file_exist) if (.not. file_exist) then +#ifdef MFC_DEBUG if (proc_rank == 0) print *, '[load_balance] probe: restart file missing, using equal decomposition: ' // trim(file_loc) +#endif return end if @@ -226,19 +228,15 @@ contains & amr_block_beg(3), amr_block_end(3)))) exit scale = 0.5_wp*scale end do +#ifdef MFC_DEBUG if (amr .and. scale < 1._wp .and. proc_rank == 0) then print *, '[load_balance] amr weight softened to fit the fine block per rank; scale =', scale end if +#endif changed = f_offsets_differ_from_equal(ox, m_glb + 1, num_procs_x) .or. f_offsets_differ_from_equal(oy, n_glb + 1, & & num_procs_y) .or. f_offsets_differ_from_equal(oz, p_glb + 1, num_procs_z) - if (proc_rank == 0) then - print *, '[load_balance] x-offsets:', ox - if (num_dims >= 2) print *, '[load_balance] y-offsets:', oy - if (num_dims >= 3) print *, '[load_balance] z-offsets:', oz - end if - if (changed) call s_apply_weighted_offsets(ox, oy, oz) deallocate (wx, wy, wz, vx, vy, vz, ox, oy, oz) diff --git a/src/simulation/m_mpi_proxy.fpp b/src/simulation/m_mpi_proxy.fpp index 9ce7a6e8e2..5df02c8bac 100644 --- a/src/simulation/m_mpi_proxy.fpp +++ b/src/simulation/m_mpi_proxy.fpp @@ -27,9 +27,6 @@ module m_mpi_proxy integer :: i_halo_size $:GPU_DECLARE(create='[i_halo_size]') - real(wp), private, allocatable, dimension(:) :: amr_buff_send !< AMR fine-halo send buffer (device-resident) - real(wp), private, allocatable, dimension(:) :: amr_buff_recv !< AMR fine-halo receive buffer (device-resident) - contains !> Initialize the MPI proxy module @@ -55,184 +52,6 @@ contains end subroutine s_initialize_mpi_proxy_module - !> Preallocate the AMR fine-halo pack buffers at this rank's max fine extents (m_amr's preallocation cap). Called from - !! s_initialize_amr_module on all ranks; no-op without MPI or at np=1. - impure subroutine s_initialize_amr_mpi_buffers(max_f1, max_f2, max_f3) - - integer, intent(in) :: max_f1, max_f2, max_f3 - -#ifdef MFC_MPI - integer :: sz - - if (num_procs == 1) return - sz = sys_size*buff_size*(max_f2 + 1)*(max_f3 + 1) - if (n_glb > 0) sz = max(sz, sys_size*buff_size*(max_f1 + 2*buff_size + 1)*(max_f3 + 1)) - if (p_glb > 0) sz = max(sz, sys_size*buff_size*(max_f1 + 2*buff_size + 1)*(max_f2 + 2*buff_size + 1)) - @:ALLOCATE(amr_buff_send(0:sz - 1), amr_buff_recv(0:sz - 1)) -#endif - - end subroutine s_initialize_amr_mpi_buffers - - !> AMR fine-level halo exchange: overwrite the buff_size fine ghost layers of q_fine at CONTINUATION faces (where the block - !! extends past this rank's intersection) with the neighbor rank's true fine data - the coarse-topology neighbor (bc_x/y/z rank - !! encoding) holds matching fine cells by construction (mirror decomposition, lockstep in time). Sequential per direction with - !! the coarse halo's transverse ranges, so corners propagate. Pack/unpack run as device kernels; only the buffers move through - !! the host (no-op macros on CPU builds). Reads the COARSE grid state for participation - call BEFORE s_amr_swap_to_fine. No - !! exchange fires at np=1 or for fully-contained blocks; fm/fn/fp are the LOCAL fine extents. - impure subroutine s_mpi_sendrecv_amr_fine_halo(q_fine, fm, fn, fp) - - type(scalar_field), dimension(1:), intent(inout) :: q_fine - integer, intent(in) :: fm, fn, fp - -#ifdef MFC_MPI - integer :: i, j, k, l, r, cnt, nbr, stag, rtag, pack_off, unpack_off, d, loc, ierr - logical :: go - - if (num_procs == 1) return - if (.not. amr_rank_owns_block) return - do d = 1, num_dims - do loc = -1, 1, 2 - ! a face participates iff the block CONTINUES past this rank's intersection there; the - ! neighbor then participates on its matching face by construction (blocking pairwise - ! sendrecvs in a fixed (d, loc) order cannot deadlock) - select case (d) - case (1) - cnt = sys_size*buff_size*(fn + 1)*(fp + 1) - if (loc == -1) then - go = amr_isect_lo(1) > amr_region_lo(1); nbr = bc_x%beg - pack_off = 0; unpack_off = -buff_size - else - go = amr_isect_hi(1) < amr_region_hi(1); nbr = bc_x%end - pack_off = fm - buff_size + 1; unpack_off = fm + 1 - end if - case (2) - cnt = sys_size*buff_size*(fm + 2*buff_size + 1)*(fp + 1) - if (loc == -1) then - go = amr_isect_lo(2) > amr_region_lo(2); nbr = bc_y%beg - pack_off = 0; unpack_off = -buff_size - else - go = amr_isect_hi(2) < amr_region_hi(2); nbr = bc_y%end - pack_off = fn - buff_size + 1; unpack_off = fn + 1 - end if - case (3) - cnt = sys_size*buff_size*(fm + 2*buff_size + 1)*(fn + 2*buff_size + 1) - if (loc == -1) then - go = amr_isect_lo(3) > amr_region_lo(3); nbr = bc_z%beg - pack_off = 0; unpack_off = -buff_size - else - go = amr_isect_hi(3) < amr_region_hi(3); nbr = bc_z%end - pack_off = fp - buff_size + 1; unpack_off = fp + 1 - end if - end select - if (.not. go) cycle - ! tags by data direction: 2*d = moving to the lower rank, 2*d+1 = moving to the upper rank - if (loc == -1) then - stag = 2*d; rtag = 2*d + 1 - else - stag = 2*d + 1; rtag = 2*d - end if - ! pack the near-face INTERIOR layers (device kernel) - #:for DIR in [1, 2, 3] - if (d == ${DIR}$) then - #:if DIR == 1 - $:GPU_PARALLEL_LOOP(collapse=4, private='[r]') - do l = 0, fp - do k = 0, fn - do j = 0, buff_size - 1 - do i = 1, sys_size - r = (i - 1) + sys_size*(j + buff_size*(k + (fn + 1)*l)) - amr_buff_send(r) = real(q_fine(i)%sf(j + pack_off, k, l), wp) - end do - end do - end do - end do - $:END_GPU_PARALLEL_LOOP() - #:elif DIR == 2 - $:GPU_PARALLEL_LOOP(collapse=4, private='[r]') - do l = 0, fp - do k = 0, buff_size - 1 - do j = -buff_size, fm + buff_size - do i = 1, sys_size - r = (i - 1) + sys_size*((j + buff_size) + (fm + 2*buff_size + 1)*(k + buff_size*l)) - amr_buff_send(r) = real(q_fine(i)%sf(j, k + pack_off, l), wp) - end do - end do - end do - end do - $:END_GPU_PARALLEL_LOOP() - #:else - $:GPU_PARALLEL_LOOP(collapse=4, private='[r]') - do l = 0, buff_size - 1 - do k = -buff_size, fn + buff_size - do j = -buff_size, fm + buff_size - do i = 1, sys_size - r = (i - 1) + sys_size*((j + buff_size) + (fm + 2*buff_size + 1)*((k + buff_size) & - & + (fn + 2*buff_size + 1)*l)) - amr_buff_send(r) = real(q_fine(i)%sf(j, k, l + pack_off), wp) - end do - end do - end do - end do - $:END_GPU_PARALLEL_LOOP() - #:endif - end if - #:endfor - $:GPU_UPDATE(host='[amr_buff_send]') - call MPI_SENDRECV(amr_buff_send, cnt, mpi_p, nbr, stag, amr_buff_recv, cnt, mpi_p, nbr, rtag, MPI_COMM_WORLD, & - & MPI_STATUS_IGNORE, ierr) - $:GPU_UPDATE(device='[amr_buff_recv]') - ! unpack into the near-face GHOST layers (device kernel) - #:for DIR in [1, 2, 3] - if (d == ${DIR}$) then - #:if DIR == 1 - $:GPU_PARALLEL_LOOP(collapse=4, private='[r]') - do l = 0, fp - do k = 0, fn - do j = 0, buff_size - 1 - do i = 1, sys_size - r = (i - 1) + sys_size*(j + buff_size*(k + (fn + 1)*l)) - q_fine(i)%sf(j + unpack_off, k, l) = real(amr_buff_recv(r), stp) - end do - end do - end do - end do - $:END_GPU_PARALLEL_LOOP() - #:elif DIR == 2 - $:GPU_PARALLEL_LOOP(collapse=4, private='[r]') - do l = 0, fp - do k = 0, buff_size - 1 - do j = -buff_size, fm + buff_size - do i = 1, sys_size - r = (i - 1) + sys_size*((j + buff_size) + (fm + 2*buff_size + 1)*(k + buff_size*l)) - q_fine(i)%sf(j, k + unpack_off, l) = real(amr_buff_recv(r), stp) - end do - end do - end do - end do - $:END_GPU_PARALLEL_LOOP() - #:else - $:GPU_PARALLEL_LOOP(collapse=4, private='[r]') - do l = 0, buff_size - 1 - do k = -buff_size, fn + buff_size - do j = -buff_size, fm + buff_size - do i = 1, sys_size - r = (i - 1) + sys_size*((j + buff_size) + (fm + 2*buff_size + 1)*((k + buff_size) & - & + (fn + 2*buff_size + 1)*l)) - q_fine(i)%sf(j, k, l + unpack_off) = real(amr_buff_recv(r), stp) - end do - end do - end do - end do - $:END_GPU_PARALLEL_LOOP() - #:endif - end if - #:endfor - end do - end do -#endif - - end subroutine s_mpi_sendrecv_amr_fine_halo - !> Since only the processor with rank 0 reads and verifies the consistency of user inputs, these are initially not available to !! the other processors. Then, the purpose of this subroutine is to distribute the user inputs to the remaining processors in !! the communicator. @@ -404,9 +223,6 @@ contains if (ib) then @:DEALLOCATE(ib_buff_send, ib_buff_recv) end if - if (allocated(amr_buff_send)) then - @:DEALLOCATE(amr_buff_send, amr_buff_recv) - end if #endif end subroutine s_finalize_mpi_proxy_module diff --git a/src/simulation/m_rhs.fpp b/src/simulation/m_rhs.fpp index 766973d319..319f8f08fa 100644 --- a/src/simulation/m_rhs.fpp +++ b/src/simulation/m_rhs.fpp @@ -619,8 +619,6 @@ contains call nvtxEndRange end if - if (hybrid_weno .or. hybrid_riemann) call s_compute_weno_sensor(q_prim_qp%vf) - ! Loop over coordinate directions for dimensional splitting do id = 1, num_dims if (igr) then diff --git a/src/simulation/m_riemann_solver_hll.fpp b/src/simulation/m_riemann_solver_hll.fpp index 85cfeeb59f..69c5d0150d 100644 --- a/src/simulation/m_riemann_solver_hll.fpp +++ b/src/simulation/m_riemann_solver_hll.fpp @@ -19,7 +19,6 @@ module m_riemann_solver_hll & get_mixture_energy_mass, get_species_specific_heats_r, get_species_enthalpies_rt, get_mixture_specific_heat_cp_mass, & & molecular_weights use m_riemann_state - use m_weno, only: weno_full implicit none @@ -99,7 +98,6 @@ contains type(riemann_states_vec3) :: cm !< Conservative momentum variables integer :: i, j, k, l !< Generic loop iterators integer :: Re_size_loc1, Re_size_loc2 !< host copies of Re_size; amdflang reads the declare-target original stale cross-TU - logical :: face_smooth !< hybrid Riemann: use central/Rusanov flux at a WENO-smooth face ! Populating the buffers of the left and right Riemann problem states variables, based on the choice of boundary conditions call s_populate_riemann_states_variables_buffers(qL_prim_rsx_vf, dqL_prim_dx_vf, dqL_prim_dy_vf, dqL_prim_dz_vf, & @@ -115,16 +113,16 @@ contains #:set SV = STENCIL_VAR #:set SF = lambda offs: COORDS.format(STENCIL_IDX = SV + offs) if (norm_dir == ${NORM_DIR}$) then - $:GPU_PARALLEL_LOOP(collapse=3, private='[face_smooth, i, j, k, l, alpha_rho_L, alpha_rho_R, vel_L, vel_R, & - & alpha_L, alpha_R, tau_e_L, tau_e_R, Re_L, Re_R, s_L, s_R, s_M, s_P, s_S, xi_M, xi_P, Ys_L, & - & Ys_R, xi_field_L, xi_field_R, Cp_iL, Cp_iR, Xs_L, Xs_R, Gamma_iL, Gamma_iR, Yi_avg, & - & Phi_avg, h_iL, h_iR, h_avg_2, c_fast, pres_mag, B, Ga, vdotB, B2, b4, cm, pcorr, zcoef, & - & vel_L_tmp, vel_R_tmp, rho_L, rho_R, pres_L, pres_R, E_L, E_R, H_L, H_R, Cp_avg, Cv_avg, & - & T_avg, eps, c_sum_Yi_Phi, T_L, T_R, Y_L, Y_R, MW_L, MW_R, R_gas_L, R_gas_R, Cp_L, Cp_R, & - & Cv_L, Cv_R, Gamm_L, Gamm_R, gamma_L, gamma_R, pi_inf_L, pi_inf_R, qv_L, qv_R, qv_avg, c_L, & - & c_R, G_L, G_R, damage_L, damage_R, rho_avg, H_avg, c_avg, gamma_avg, ptilde_L, ptilde_R, & - & vel_L_rms, vel_R_rms, vel_avg_rms, Ms_L, Ms_R, pres_SL, pres_SR, alpha_L_sum, alpha_R_sum, & - & flux_tau_L, flux_tau_R]', copyin='[norm_dir]', firstprivate='[Re_size_loc1, Re_size_loc2]') + $:GPU_PARALLEL_LOOP(collapse=3, private='[i, j, k, l, alpha_rho_L, alpha_rho_R, vel_L, vel_R, alpha_L, alpha_R, & + & tau_e_L, tau_e_R, Re_L, Re_R, s_L, s_R, s_M, s_P, s_S, xi_M, xi_P, Ys_L, Ys_R, xi_field_L, & + & xi_field_R, Cp_iL, Cp_iR, Xs_L, Xs_R, Gamma_iL, Gamma_iR, Yi_avg, Phi_avg, h_iL, h_iR, & + & h_avg_2, c_fast, pres_mag, B, Ga, vdotB, B2, b4, cm, pcorr, zcoef, vel_L_tmp, vel_R_tmp, & + & rho_L, rho_R, pres_L, pres_R, E_L, E_R, H_L, H_R, Cp_avg, Cv_avg, T_avg, eps, c_sum_Yi_Phi, & + & T_L, T_R, Y_L, Y_R, MW_L, MW_R, R_gas_L, R_gas_R, Cp_L, Cp_R, Cv_L, Cv_R, Gamm_L, Gamm_R, & + & gamma_L, gamma_R, pi_inf_L, pi_inf_R, qv_L, qv_R, qv_avg, c_L, c_R, G_L, G_R, damage_L, & + & damage_R, rho_avg, H_avg, c_avg, gamma_avg, ptilde_L, ptilde_R, vel_L_rms, vel_R_rms, & + & vel_avg_rms, Ms_L, Ms_R, pres_SL, pres_SR, alpha_L_sum, alpha_R_sum, flux_tau_L, & + & flux_tau_R]', copyin='[norm_dir]', firstprivate='[Re_size_loc1, Re_size_loc2]') do l = ${Z_BND}$%beg, ${Z_BND}$%end do k = ${Y_BND}$%beg, ${Y_BND}$%end do j = ${X_BND}$%beg, ${X_BND}$%end @@ -636,11 +634,6 @@ contains end do end if #:endif - if (hybrid_riemann) then - face_smooth = .not. (weno_full(${SF('')}$) .or. weno_full(${SF(' + 1')}$)) - if (face_smooth) call s_compute_hybrid_smooth_flux(${SF('')}$, alpha_rho_L, alpha_L, alpha_rho_R, & - & alpha_R, vel_L, vel_R, c_L, c_R, rho_L, rho_R, pres_L, pres_R, E_L, E_R, hybrid_smooth_flux) - end if end do end do end do diff --git a/src/simulation/m_riemann_solver_hllc.fpp b/src/simulation/m_riemann_solver_hllc.fpp index 4d78008625..7ee6eca2b6 100644 --- a/src/simulation/m_riemann_solver_hllc.fpp +++ b/src/simulation/m_riemann_solver_hllc.fpp @@ -22,7 +22,6 @@ module m_riemann_solver_hllc & get_mixture_energy_mass, get_species_specific_heats_r, get_species_enthalpies_rt, get_mixture_specific_heat_cp_mass, & & molecular_weights use m_riemann_state - use m_weno, only: weno_full implicit none @@ -123,7 +122,6 @@ contains real(wp) :: zcoef, pcorr !< low Mach number correction integer :: i, j, k, l, q !< Generic loop iterators integer :: Re_size_loc1, Re_size_loc2 !< host copies of Re_size; amdflang reads the declare-target original stale cross-TU - logical :: face_smooth !< hybrid Riemann: use central/Rusanov flux at a WENO-smooth face ! Populating the buffers of the left and right Riemann problem states variables, based on the choice of boundary conditions call s_populate_riemann_states_variables_buffers(qL_prim_rsx_vf, dqL_prim_dx_vf, dqL_prim_dy_vf, dqL_prim_dz_vf, & @@ -145,9 +143,9 @@ contains ! 6-EQUATION MODEL WITH HLLC HLLC star-state flux with contact wave speed s_S if (model_eqns == model_eqns_6eq) then ! 6-equation model (model_eqns=3): separate phasic internal energies - $:GPU_PARALLEL_LOOP(collapse=3, private='[face_smooth, i, j, k, l, vel_L, vel_R, Re_L, Re_R, alpha_L, & - & alpha_R, alpha_rho_L, alpha_rho_R, Ys_L, Ys_R, Xs_L, Xs_R, Gamma_iL, Gamma_iR, Cp_iL, & - & Cp_iR, Yi_avg, Phi_avg, h_iL, h_iR, h_avg_2, tau_e_L, tau_e_R, flux_ene_e, xi_field_L, & + $:GPU_PARALLEL_LOOP(collapse=3, private='[i, j, k, l, vel_L, vel_R, Re_L, Re_R, alpha_L, alpha_R, & + & alpha_rho_L, alpha_rho_R, Ys_L, Ys_R, Xs_L, Xs_R, Gamma_iL, Gamma_iR, Cp_iL, Cp_iR, & + & Yi_avg, Phi_avg, h_iL, h_iR, h_avg_2, tau_e_L, tau_e_R, flux_ene_e, xi_field_L, & & xi_field_R, pcorr, zcoef, rho_L, rho_R, pres_L, pres_R, E_L, E_R, H_L, H_R, Cp_avg, & & Cv_avg, T_avg, eps, c_sum_Yi_Phi, T_L, T_R, Y_L, Y_R, MW_L, MW_R, R_gas_L, R_gas_R, & & Cp_L, Cp_R, Cv_L, Cv_R, Gamm_L, Gamm_R, gamma_L, gamma_R, pi_inf_L, pi_inf_R, qv_L, & @@ -500,12 +498,6 @@ contains flux_gsrc_rsx_vf(${SF('')}$, eqn_idx%mom%end) = flux_rsx_vf(${SF('')}$, eqn_idx%mom%beg + 1) end if #:endif - if (hybrid_riemann) then - face_smooth = .not. (weno_full(${SF('')}$) .or. weno_full(${SF(' + 1')}$)) - if (face_smooth) call s_compute_hybrid_smooth_flux(${SF('')}$, alpha_rho_L, alpha_L, & - & alpha_rho_R, alpha_R, vel_L, vel_R, c_L, c_R, rho_L, rho_R, pres_L, pres_R, E_L, E_R, & - & hybrid_smooth_flux) - end if end do end do end do @@ -1101,16 +1093,15 @@ contains $:END_GPU_PARALLEL_LOOP() else ! 5-equation model (model_eqns=2): mixture total energy, volume fraction advection - $:GPU_PARALLEL_LOOP(collapse=3, private='[face_smooth, i, T_L, T_R, vel_L_rms, vel_R_rms, pres_L, pres_R, & - & rho_L, gamma_L, pi_inf_L, qv_L, rho_R, gamma_R, pi_inf_R, qv_R, alpha_L_sum, & - & alpha_R_sum, E_L, E_R, MW_L, MW_R, R_gas_L, R_gas_R, Cp_L, Cp_R, Cv_L, Cv_R, Gamm_L, & - & Gamm_R, Y_L, Y_R, H_L, H_R, qv_avg, rho_avg, gamma_avg, H_avg, c_L, c_R, c_avg, s_P, & - & s_M, xi_P, xi_M, xi_L, xi_R, xi_L_m1, xi_R_m1, Ms_L, Ms_R, pres_SL, pres_SR, vel_L, & - & vel_R, Re_L, Re_R, alpha_L, alpha_R, alpha_rho_L, alpha_rho_R, alpha_lim_L, & - & alpha_lim_R, s_L, s_R, s_S, vel_avg_rms, pcorr, zcoef, vel_L_tmp, vel_R_tmp, Ys_L, & - & Ys_R, Xs_L, Xs_R, Gamma_iL, Gamma_iR, Cp_iL, Cp_iR, tau_e_L, tau_e_R, xi_field_L, & - & xi_field_R, Yi_avg, Phi_avg, h_iL, h_iR, h_avg_2, G_L, G_R]', copyin='[is1, is2, is3]', & - & firstprivate='[Re_size_loc1, Re_size_loc2]') + $:GPU_PARALLEL_LOOP(collapse=3, private='[i, T_L, T_R, vel_L_rms, vel_R_rms, pres_L, pres_R, rho_L, gamma_L, & + & pi_inf_L, qv_L, rho_R, gamma_R, pi_inf_R, qv_R, alpha_L_sum, alpha_R_sum, E_L, E_R, & + & MW_L, MW_R, R_gas_L, R_gas_R, Cp_L, Cp_R, Cv_L, Cv_R, Gamm_L, Gamm_R, Y_L, Y_R, H_L, & + & H_R, qv_avg, rho_avg, gamma_avg, H_avg, c_L, c_R, c_avg, s_P, s_M, xi_P, xi_M, xi_L, & + & xi_R, xi_L_m1, xi_R_m1, Ms_L, Ms_R, pres_SL, pres_SR, vel_L, vel_R, Re_L, Re_R, & + & alpha_L, alpha_R, alpha_rho_L, alpha_rho_R, alpha_lim_L, alpha_lim_R, s_L, s_R, s_S, & + & vel_avg_rms, pcorr, zcoef, vel_L_tmp, vel_R_tmp, Ys_L, Ys_R, Xs_L, Xs_R, Gamma_iL, & + & Gamma_iR, Cp_iL, Cp_iR, tau_e_L, tau_e_R, xi_field_L, xi_field_R, Yi_avg, Phi_avg, & + & h_iL, h_iR, h_avg_2, G_L, G_R]', copyin='[is1, is2, is3]', firstprivate='[Re_size_loc1, Re_size_loc2]') do l = ${Z_BND}$%beg, ${Z_BND}$%end do k = ${Y_BND}$%beg, ${Y_BND}$%end do j = ${X_BND}$%beg, ${X_BND}$%end @@ -1490,12 +1481,6 @@ contains flux_gsrc_rsx_vf(${SF('')}$, eqn_idx%mom%end) = flux_rsx_vf(${SF('')}$, eqn_idx%mom%beg + 1) end if #:endif - if (hybrid_riemann) then - face_smooth = .not. (weno_full(${SF('')}$) .or. weno_full(${SF(' + 1')}$)) - if (face_smooth) call s_compute_hybrid_smooth_flux(${SF('')}$, alpha_rho_L, alpha_L, & - & alpha_rho_R, alpha_R, vel_L, vel_R, c_L, c_R, rho_L, rho_R, pres_L, pres_R, E_L, E_R, & - & hybrid_smooth_flux) - end if end do end do end do diff --git a/src/simulation/m_riemann_solver_hlld.fpp b/src/simulation/m_riemann_solver_hlld.fpp index a26c6599a2..540726a47b 100644 --- a/src/simulation/m_riemann_solver_hlld.fpp +++ b/src/simulation/m_riemann_solver_hlld.fpp @@ -12,7 +12,6 @@ module m_riemann_solver_hlld use m_global_parameters use m_variables_conversion use m_riemann_state - use m_weno, only: weno_full implicit none @@ -61,7 +60,6 @@ contains real(wp) :: vL_star, vR_star, wL_star, wR_star real(wp) :: v_double, w_double, By_double, Bz_double, E_doubleL, E_doubleR, E_double integer :: i, j, k, l - logical :: face_smooth !< hybrid Riemann: use central/Rusanov flux at a WENO-smooth face call s_populate_riemann_states_variables_buffers(qL_prim_rsx_vf, dqL_prim_dx_vf, dqL_prim_dy_vf, dqL_prim_dz_vf, & & qR_prim_rsx_vf, dqR_prim_dx_vf, dqR_prim_dy_vf, dqR_prim_dz_vf, norm_dir, ix, iy, iz) @@ -75,10 +73,10 @@ contains #:set SV = STENCIL_VAR #:set SF = lambda offs: COORDS.format(STENCIL_IDX = SV + offs) if (norm_dir == ${NORM_DIR}$) then - $:GPU_PARALLEL_LOOP(collapse=3, private='[face_smooth, alpha_rho_L, alpha_rho_R, vel, alpha_L, alpha_R, rho, & - & pres, E, H_no_mag, gamma, pi_inf, qv, vel_rms, B, c, c_fast, pres_mag, U_L, U_R, U_starL, & - & U_starR, U_doubleL, U_doubleR, F_L, F_R, F_starL, F_starR, F_hlld, s_L, s_R, s_M, s_starL, & - & s_starR, pTot_L, pTot_R, p_star, rhoL_star, rhoR_star, E_starL, E_starR, sqrt_rhoL_star, & + $:GPU_PARALLEL_LOOP(collapse=3, private='[alpha_rho_L, alpha_rho_R, vel, alpha_L, alpha_R, rho, pres, E, & + & H_no_mag, gamma, pi_inf, qv, vel_rms, B, c, c_fast, pres_mag, U_L, U_R, U_starL, U_starR, & + & U_doubleL, U_doubleR, F_L, F_R, F_starL, F_starR, F_hlld, s_L, s_R, s_M, s_starL, s_starR, & + & pTot_L, pTot_R, p_star, rhoL_star, rhoR_star, E_starL, E_starR, sqrt_rhoL_star, & & sqrt_rhoR_star, denom_ds, sign_Bx, vL_star, vR_star, wL_star, wR_star, v_double, w_double, & & By_double, Bz_double, E_doubleL, E_doubleR, E_double]', copyin='[norm_dir]') do l = ${Z_BND}$%beg, ${Z_BND}$%end @@ -239,11 +237,6 @@ contains end if ! Hybrid Riemann: overwrite HLLD with a central/Rusanov flux at a WENO-smooth face - if (hybrid_riemann) then - face_smooth = .not. (weno_full(${SF('')}$) .or. weno_full(${SF(' + 1')}$)) - if (face_smooth) F_hlld = 0.5_wp*(F_L + F_R) - real(hybrid_smooth_flux - 1, & - & wp)*0.5_wp*max(abs(vel%L(1)) + c_fast%L, abs(vel%R(1)) + c_fast%R)*(U_R - U_L) - end if ! (12) Write HLLD flux to output arrays flux_rsx_vf(${SF('')}$, 1) = F_hlld(1) ! TODO multi-component diff --git a/src/simulation/m_riemann_solver_lf.fpp b/src/simulation/m_riemann_solver_lf.fpp index 3a699554d0..c8e05f8a8b 100644 --- a/src/simulation/m_riemann_solver_lf.fpp +++ b/src/simulation/m_riemann_solver_lf.fpp @@ -313,17 +313,7 @@ contains s_L = sqrt(s_L) s_R = sqrt(s_R) - if (hybrid_riemann) then - ! For LF, hybrid_riemann simply switches the dissipation to the local (normal- - ! velocity) wave speed, turning LF into local Lax-Friedrichs everywhere. LF does - ! NOT call the shared smooth-flux helper: LF's flux carries extra terms (pcorr, - ! its own advection form) the generic Rusanov helper lacks, so overwriting only - ! smooth faces would inject a sensor-flip discontinuity. Using LF's own flux with - ! the local wave speed keeps it a single consistent, backend-stable scheme. - s_P = max(abs(vel_L(dir_idx(1))) + c_L, abs(vel_R(dir_idx(1))) + c_R) - else - s_P = max(s_L, s_R) + max(c_L, c_R) - end if + s_P = max(s_L, s_R) + max(c_L, c_R) s_M = -s_P s_L = s_M diff --git a/src/simulation/m_riemann_state.fpp b/src/simulation/m_riemann_state.fpp index 8e34f979db..550ac8b972 100644 --- a/src/simulation/m_riemann_state.fpp +++ b/src/simulation/m_riemann_state.fpp @@ -1143,84 +1143,6 @@ contains end function f_compute_hllc_star_momentum_flux - !> Hybrid-Riemann smooth-face flux: overwrite the Riemann flux at a WENO-smooth face with either a central (hybrid_smooth_flux - !! == 1) or a local Lax-Friedrichs / Rusanov (hybrid_smooth_flux == 2) flux, reusing the left/right states the calling solver - !! already reconstructed. Shared by hll/hllc/lf so the smooth-flux path stays identical across solvers; the reshaped flux - !! buffers are module-scope so only the (j,k,l) index and the small per-fluid/per-dim state arrays need to be passed. - subroutine s_compute_hybrid_smooth_flux(j, k, l, alpha_rho_L, alpha_L, alpha_rho_R, alpha_R, vel_L, vel_R, c_L, c_R, rho_L, & - & rho_R, pres_L, pres_R, E_L, E_R, hybrid_smooth_flux) - - $:GPU_ROUTINE(function_name='s_compute_hybrid_smooth_flux', parallelism='[seq]', cray_inline=True) - - integer, intent(in) :: j, k, l - real(wp), dimension(num_fluids), intent(in) :: alpha_rho_L, alpha_L, alpha_rho_R, alpha_R - real(wp), dimension(num_dims), intent(in) :: vel_L, vel_R - real(wp), intent(in) :: c_L, c_R, rho_L, rho_R, pres_L, pres_R, E_L, E_R - ! hybrid_smooth_flux is passed as an arg (not read from the module): CCE's --case-optimization - ! OMP clone of this device routine cannot resolve that common-module global (ftn-7066). - integer, intent(in) :: hybrid_smooth_flux - real(wp) :: lam, FL, FR, UL, UR - integer :: i - - ! Rusanov dissipation coefficient (dropped for the pure-central flux, hybrid_smooth_flux == 1) - lam = max(abs(vel_L(dir_idx(1))) + c_L, abs(vel_R(dir_idx(1))) + c_R) - - ! Continuity - $:GPU_LOOP(parallelism='[seq]') - do i = 1, eqn_idx%cont%end - FL = alpha_rho_L(i)*vel_L(dir_idx(1)) - FR = alpha_rho_R(i)*vel_R(dir_idx(1)) - flux_rsx_vf(j, k, l, i) = 0.5_wp*(FL + FR) - real(hybrid_smooth_flux - 1, & - & wp)*0.5_wp*lam*(alpha_rho_R(i) - alpha_rho_L(i)) - end do - - ! Momentum - $:GPU_LOOP(parallelism='[seq]') - do i = 1, num_dims - FL = rho_L*vel_L(dir_idx(1))*vel_L(dir_idx(i)) + dir_flg(dir_idx(i))*pres_L - FR = rho_R*vel_R(dir_idx(1))*vel_R(dir_idx(i)) + dir_flg(dir_idx(i))*pres_R - UL = rho_L*vel_L(dir_idx(i)) - UR = rho_R*vel_R(dir_idx(i)) - flux_rsx_vf(j, k, l, eqn_idx%cont%end + dir_idx(i)) = 0.5_wp*(FL + FR) - real(hybrid_smooth_flux - 1, & - & wp)*0.5_wp*lam*(UR - UL) - end do - - ! Energy - FL = vel_L(dir_idx(1))*(E_L + pres_L) - FR = vel_R(dir_idx(1))*(E_R + pres_R) - flux_rsx_vf(j, k, l, eqn_idx%E) = 0.5_wp*(FL + FR) - real(hybrid_smooth_flux - 1, wp)*0.5_wp*lam*(E_R - E_L) - - ! Volume fractions (advection) - $:GPU_LOOP(parallelism='[seq]') - do i = 1, num_fluids - FL = alpha_L(i)*vel_L(dir_idx(1)) - FR = alpha_R(i)*vel_R(dir_idx(1)) - flux_rsx_vf(j, k, l, eqn_idx%adv%beg + i - 1) = 0.5_wp*(FL + FR) - real(hybrid_smooth_flux - 1, & - & wp)*0.5_wp*lam*(alpha_R(i) - alpha_L(i)) - end do - - ! Internal energies (6-equation model only) - if (model_eqns == model_eqns_6eq) then - $:GPU_LOOP(parallelism='[seq]') - do i = 1, num_fluids - UL = alpha_L(i)*(gammas(i)*pres_L + pi_infs(i)) + alpha_rho_L(i)*qvs(i) - UR = alpha_R(i)*(gammas(i)*pres_R + pi_infs(i)) + alpha_rho_R(i)*qvs(i) - FL = UL*vel_L(dir_idx(1)) - FR = UR*vel_R(dir_idx(1)) - flux_rsx_vf(j, k, l, eqn_idx%int_en%beg + i - 1) = 0.5_wp*(FL + FR) - real(hybrid_smooth_flux - 1, & - & wp)*0.5_wp*lam*(UR - UL) - end do - end if - - ! Non-conservative advection source: central interface velocity - $:GPU_LOOP(parallelism='[seq]') - do i = 1, num_dims - vel_src_rsx_vf(j, k, l, dir_idx(i)) = 0.5_wp*(vel_L(dir_idx(i)) + vel_R(dir_idx(i))) - end do - flux_src_rsx_vf(j, k, l, eqn_idx%adv%beg) = vel_src_rsx_vf(j, k, l, dir_idx(1)) - - end subroutine s_compute_hybrid_smooth_flux - !> Deallocation and/or disassociation procedures that are needed to finalize the selected Riemann problem solver subroutine s_finalize_riemann_solver(flux_vf, flux_src_vf, flux_gsrc_vf, norm_dir) diff --git a/src/simulation/m_start_up.fpp b/src/simulation/m_start_up.fpp index 07595c7191..50f85997c6 100644 --- a/src/simulation/m_start_up.fpp +++ b/src/simulation/m_start_up.fpp @@ -896,14 +896,9 @@ contains call s_initialize_amr_module() call s_initialize_amr_registers(amr_maxc_fit) - call s_amr_operator_checks() ! restarts restore the saved (possibly regridded) box and fine state; otherwise prolong from coarse call s_read_amr_restart(amr_restored) if (.not. amr_restored) call s_populate_amr_fine(q_cons_ts(1)%vf) - ! the restrict-prolong check reads the gathered coarse patch amr_cg that s_populate_amr_fine just built; on a restart - ! s_populate is skipped (fine state is restored, not prolonged), so the check is both inapplicable and unarmed - if (.not. amr_restored) call s_amr_conservation_check(q_cons_ts(1)%vf) - call s_amr_conservation_defect(q_cons_ts(1)%vf, .false.) if (model_eqns == model_eqns_6eq) call s_initialize_internal_energy_equations(q_cons_ts(1)%vf) if (ib) then @@ -1116,7 +1111,6 @@ contains !> Finalize and deallocate all simulation sub-modules in reverse initialization order impure subroutine s_finalize_modules - call s_amr_conservation_defect(q_cons_ts(1)%vf, .true.) call s_finalize_amr_registers() call s_finalize_amr_module() call s_finalize_time_steppers_module() diff --git a/src/simulation/m_weno.fpp b/src/simulation/m_weno.fpp index fd7b9b48ce..97a7ca71c4 100644 --- a/src/simulation/m_weno.fpp +++ b/src/simulation/m_weno.fpp @@ -16,8 +16,7 @@ module m_weno use m_thinc, only: s_thinc_compression use m_nvtx - private; public :: s_initialize_weno_module, s_finalize_weno_module, s_weno, s_pack_weno_input_arr, s_compute_weno_sensor, & - & s_compute_weno_coefficients, weno_full + private; public :: s_initialize_weno_module, s_finalize_weno_module, s_weno, s_pack_weno_input_arr, s_compute_weno_coefficients !> @name The cell-average variables that will be WENO-reconstructed unpacked into an array for performance !> @{ @@ -70,10 +69,6 @@ module m_weno integer :: v_size !< Number of WENO-reconstructed cell-average variables $:GPU_DECLARE(create='[v_size]') - logical, allocatable, dimension(:,:,:) :: weno_full !< per-cell: use full WENO (discontinuity in stencil) - logical, allocatable, dimension(:,:,:) :: weno_disc !< raw per-cell discontinuity flag (pre-dilation sensor scratch) - $:GPU_DECLARE(create='[weno_full, weno_disc]') - logical :: uniform_grid(3) !< True if grid spacing is uniform in each direction $:GPU_DECLARE(create='[uniform_grid]') @@ -126,11 +121,6 @@ contains @:ALLOCATE(v_rs_weno(is1_weno%beg:is1_weno%end, is2_weno%beg:is2_weno%end, is3_weno%beg:is3_weno%end, 1:sys_size)) - if (hybrid_weno .or. hybrid_riemann) then - @:ALLOCATE(weno_full(idwbuff(1)%beg:idwbuff(1)%end, idwbuff(2)%beg:idwbuff(2)%end, idwbuff(3)%beg:idwbuff(3)%end)) - @:ALLOCATE(weno_disc(idwbuff(1)%beg:idwbuff(1)%end, idwbuff(2)%beg:idwbuff(2)%end, idwbuff(3)%beg:idwbuff(3)%end)) - end if - ! Allocating/Computing WENO Coefficients in y-direction if (n == 0) return @@ -923,9 +913,8 @@ contains real(wp), dimension(0:weno_num_stencils) :: beta real(wp), dimension(0:weno_num_stencils) :: delta #:endif - real(wp), dimension(-3:3) :: v !< temporary field value array for clarity (WENO7 only) + real(wp), dimension(-3:3) :: v !< temporary field value array for clarity (WENO7 only) real(wp) :: tau - logical :: use_central !< hybrid sensor verdict for this cell (central weights vs full nonlinear WENO) integer :: i, j, k, l, q real(wp) :: vp0, vp1, vp2, vp3, vm1, vm2, vm3 @@ -993,7 +982,7 @@ contains #:set SV = STENCIL_VAR #:set SF = lambda offs: COORDS.format(STENCIL_IDX = SV + offs) if (weno_dir == ${WENO_DIR}$) then - $:GPU_PARALLEL_LOOP(collapse=4,private='[beta, dvd, poly, omega, alpha, tau, q, vp0, vp1, vm1, use_central]') + $:GPU_PARALLEL_LOOP(collapse=4,private='[beta, dvd, poly, omega, alpha, tau, q, vp0, vp1, vm1]') do l = ${Z_BND}$%beg, ${Z_BND}$%end do k = ${Y_BND}$%beg, ${Y_BND}$%end do j = ${X_BND}$%beg, ${X_BND}$%end @@ -1015,70 +1004,56 @@ contains beta(0) = beta_coef_${XYZ}$ (${SV}$, 0, 0)*dvd(0)*dvd(0) + weno_eps beta(1) = beta_coef_${XYZ}$ (${SV}$, 1, 0)*dvd(-1)*dvd(-1) + weno_eps - use_central = .false. - if (hybrid_weno) use_central = .not. weno_full(${SF('')}$) - if (use_central) then - vL_rs_vf_x(j, k, l, i) = d_cbL_${XYZ}$ (0, ${SV}$)*poly(0) + d_cbL_${XYZ}$ (1, & - & ${SV}$)*poly(1) - else - if (wenojs) then - do q = 0, weno_num_stencils - alpha(q) = d_cbL_${XYZ}$ (q, ${SV}$)/(beta(q)**2._wp) - end do - else if (mapped_weno) then - do q = 0, weno_num_stencils - alpha(q) = d_cbL_${XYZ}$ (q, ${SV}$)/(beta(q)**2._wp) - end do - omega = alpha/sum(alpha) - do q = 0, weno_num_stencils - alpha(q) = (d_cbL_${XYZ}$ (q, ${SV}$)*(1._wp + d_cbL_${XYZ}$ (q, & - & ${SV}$) - 3._wp*omega(q)) + omega(q)**2._wp)*(omega(q)/(d_cbL_${XYZ}$ (q, & - & ${SV}$)**2._wp + omega(q)*(1._wp - 2._wp*d_cbL_${XYZ}$ (q, ${SV}$)))) - end do - else if (wenoz) then - ! Borges, et al. (2008) - tau = abs(beta(1) - beta(0)) - do q = 0, weno_num_stencils - alpha(q) = d_cbL_${XYZ}$ (q, ${SV}$)*(1._wp + tau/beta(q)) - end do - end if + if (wenojs) then + do q = 0, weno_num_stencils + alpha(q) = d_cbL_${XYZ}$ (q, ${SV}$)/(beta(q)**2._wp) + end do + else if (mapped_weno) then + do q = 0, weno_num_stencils + alpha(q) = d_cbL_${XYZ}$ (q, ${SV}$)/(beta(q)**2._wp) + end do omega = alpha/sum(alpha) - vL_rs_vf_x(j, k, l, i) = omega(0)*poly(0) + omega(1)*poly(1) + do q = 0, weno_num_stencils + alpha(q) = (d_cbL_${XYZ}$ (q, ${SV}$)*(1._wp + d_cbL_${XYZ}$ (q, & + & ${SV}$) - 3._wp*omega(q)) + omega(q)**2._wp)*(omega(q)/(d_cbL_${XYZ}$ (q, & + & ${SV}$)**2._wp + omega(q)*(1._wp - 2._wp*d_cbL_${XYZ}$ (q, ${SV}$)))) + end do + else if (wenoz) then + ! Borges, et al. (2008) + tau = abs(beta(1) - beta(0)) + do q = 0, weno_num_stencils + alpha(q) = d_cbL_${XYZ}$ (q, ${SV}$)*(1._wp + tau/beta(q)) + end do end if + omega = alpha/sum(alpha) + vL_rs_vf_x(j, k, l, i) = omega(0)*poly(0) + omega(1)*poly(1) ! reconstruct from right side poly(0) = vp0 + poly_coef_cbR_${XYZ}$ (${SV}$, 0, 0)*dvd(0) poly(1) = vp0 + poly_coef_cbR_${XYZ}$ (${SV}$, 1, 0)*dvd(-1) - use_central = .false. - if (hybrid_weno) use_central = .not. weno_full(${SF('')}$) - if (use_central) then - vR_rs_vf_x(j, k, l, i) = d_cbR_${XYZ}$ (0, ${SV}$)*poly(0) + d_cbR_${XYZ}$ (1, & - & ${SV}$)*poly(1) - else - if (wenojs) then - do q = 0, weno_num_stencils - alpha(q) = d_cbR_${XYZ}$ (q, ${SV}$)/(beta(q)**2._wp) - end do - else if (mapped_weno) then - do q = 0, weno_num_stencils - alpha(q) = d_cbR_${XYZ}$ (q, ${SV}$)/(beta(q)**2._wp) - end do - omega = alpha/sum(alpha) - do q = 0, weno_num_stencils - alpha(q) = (d_cbR_${XYZ}$ (q, ${SV}$)*(1._wp + d_cbR_${XYZ}$ (q, & - & ${SV}$) - 3._wp*omega(q)) + omega(q)**2._wp)*(omega(q)/(d_cbR_${XYZ}$ (q, & - & ${SV}$)**2._wp + omega(q)*(1._wp - 2._wp*d_cbR_${XYZ}$ (q, ${SV}$)))) - end do - else if (wenoz) then - do q = 0, weno_num_stencils - alpha(q) = d_cbR_${XYZ}$ (q, ${SV}$)*(1._wp + tau/beta(q)) - end do - end if + if (wenojs) then + do q = 0, weno_num_stencils + alpha(q) = d_cbR_${XYZ}$ (q, ${SV}$)/(beta(q)**2._wp) + end do + else if (mapped_weno) then + do q = 0, weno_num_stencils + alpha(q) = d_cbR_${XYZ}$ (q, ${SV}$)/(beta(q)**2._wp) + end do omega = alpha/sum(alpha) - vR_rs_vf_x(j, k, l, i) = omega(0)*poly(0) + omega(1)*poly(1) + do q = 0, weno_num_stencils + alpha(q) = (d_cbR_${XYZ}$ (q, ${SV}$)*(1._wp + d_cbR_${XYZ}$ (q, & + & ${SV}$) - 3._wp*omega(q)) + omega(q)**2._wp)*(omega(q)/(d_cbR_${XYZ}$ (q, & + & ${SV}$)**2._wp + omega(q)*(1._wp - 2._wp*d_cbR_${XYZ}$ (q, ${SV}$)))) + end do + else if (wenoz) then + do q = 0, weno_num_stencils + alpha(q) = d_cbR_${XYZ}$ (q, ${SV}$)*(1._wp + tau/beta(q)) + end do end if + omega = alpha/sum(alpha) + vR_rs_vf_x(j, k, l, i) = omega(0)*poly(0) + omega(1)*poly(1) end do end do end do @@ -1097,7 +1072,7 @@ contains #:set SF = lambda offs: COORDS.format(STENCIL_IDX = SV + offs) if (weno_dir == ${WENO_DIR}$) then $:GPU_PARALLEL_LOOP(collapse=3,private='[dvd, poly, beta, alpha, omega, tau, delta, q, vp0, vm1, vm2, & - & use_central, vp1, vp2]') + & vp1, vp2]') do l = ${Z_BND}$%beg, ${Z_BND}$%end do k = ${Y_BND}$%beg, ${Y_BND}$%end do j = ${X_BND}$%beg, ${X_BND}$%end @@ -1142,65 +1117,57 @@ contains & 1)*dvd(-1)*dvd(-2) + beta_coef_${XYZ}$ (${SV}$, 2, 2)*dvd(-2)*dvd(-2) + weno_eps end if - use_central = .false. - if (hybrid_weno) use_central = .not. weno_full(${SF('')}$) - if (use_central) then - vL_rs_vf_x(j, k, l, i) = d_cbL_${XYZ}$ (0, ${SV}$)*poly(0) + d_cbL_${XYZ}$ (1, & - & ${SV}$)*poly(1) + d_cbL_${XYZ}$ (2, ${SV}$)*poly(2) - else - if (wenojs) then - do q = 0, weno_num_stencils - alpha(q) = d_cbL_${XYZ}$ (q, ${SV}$)/(beta(q)**2._wp) - end do - else if (mapped_weno) then - do q = 0, weno_num_stencils - alpha(q) = d_cbL_${XYZ}$ (q, ${SV}$)/(beta(q)**2._wp) - end do - omega = alpha/sum(alpha) - do q = 0, weno_num_stencils - alpha(q) = (d_cbL_${XYZ}$ (q, ${SV}$)*(1._wp + d_cbL_${XYZ}$ (q, & - & ${SV}$) - 3._wp*omega(q)) + omega(q)**2._wp)*(omega(q) & - & /(d_cbL_${XYZ}$ (q, & - & ${SV}$)**2._wp + omega(q)*(1._wp - 2._wp*d_cbL_${XYZ}$ (q, ${SV}$)))) - end do - else if (wenoz) then - ! Borges, et al. (2008) - - tau = abs(beta(2) - beta(0)) ! Equation 25 - $:GPU_LOOP(parallelism='[seq]') - do q = 0, weno_num_stencils - alpha(q) = d_cbL_${XYZ}$ (q, ${SV}$)*(1._wp + (tau/beta(q))) - ! Equation 28 (note: weno_eps was already added to beta) - end do - else if (teno) then - ! Fu, et al. (2016) Fu''s code: https://dx.doi.org/10.13140/RG.2.2.36250.34247 - tau = abs(beta(2) - beta(0)) - $:GPU_LOOP(parallelism='[seq]') - do q = 0, weno_num_stencils - ! Equation 22 (reuse alpha as gamma; pick C=1 & q=6) - alpha(q) = 1._wp + tau/beta(q) - ! Equation 22 cont. (some CPU compilers cannot optimize x**6.0) - alpha(q) = (alpha(q)**3._wp)**2._wp - end do - omega = alpha/sum(alpha) ! Equation 25 (reuse omega as xi) + if (wenojs) then + do q = 0, weno_num_stencils + alpha(q) = d_cbL_${XYZ}$ (q, ${SV}$)/(beta(q)**2._wp) + end do + else if (mapped_weno) then + do q = 0, weno_num_stencils + alpha(q) = d_cbL_${XYZ}$ (q, ${SV}$)/(beta(q)**2._wp) + end do + omega = alpha/sum(alpha) + do q = 0, weno_num_stencils + alpha(q) = (d_cbL_${XYZ}$ (q, ${SV}$)*(1._wp + d_cbL_${XYZ}$ (q, & + & ${SV}$) - 3._wp*omega(q)) + omega(q)**2._wp)*(omega(q)/(d_cbL_${XYZ}$ (q, & + & ${SV}$)**2._wp + omega(q)*(1._wp - 2._wp*d_cbL_${XYZ}$ (q, ${SV}$)))) + end do + else if (wenoz) then + ! Borges, et al. (2008) - $:GPU_LOOP(parallelism='[seq]') - do q = 0, weno_num_stencils - if (omega(q) < teno_CT) then ! Equation 26 - delta(q) = 0._wp - else - delta(q) = 1._wp - end if - alpha(q) = delta(q)*d_cbL_${XYZ}$ (q, ${SV}$) ! Equation 27 - end do - end if + tau = abs(beta(2) - beta(0)) ! Equation 25 + $:GPU_LOOP(parallelism='[seq]') + do q = 0, weno_num_stencils + alpha(q) = d_cbL_${XYZ}$ (q, ${SV}$)*(1._wp + (tau/beta(q))) + ! Equation 28 (note: weno_eps was already added to beta) + end do + else if (teno) then + ! Fu, et al. (2016) Fu''s code: https://dx.doi.org/10.13140/RG.2.2.36250.34247 + tau = abs(beta(2) - beta(0)) + $:GPU_LOOP(parallelism='[seq]') + do q = 0, weno_num_stencils + ! Equation 22 (reuse alpha as gamma; pick C=1 & q=6) + alpha(q) = 1._wp + tau/beta(q) + ! Equation 22 cont. (some CPU compilers cannot optimize x**6.0) + alpha(q) = (alpha(q)**3._wp)**2._wp + end do + omega = alpha/sum(alpha) ! Equation 25 (reuse omega as xi) - omega(0) = alpha(0)/(alpha(0) + alpha(1) + alpha(2)) - omega(1) = alpha(1)/(alpha(0) + alpha(1) + alpha(2)) - omega(2) = alpha(2)/(alpha(0) + alpha(1) + alpha(2)) - vL_rs_vf_x(j, k, l, i) = omega(0)*poly(0) + omega(1)*poly(1) + omega(2)*poly(2) + $:GPU_LOOP(parallelism='[seq]') + do q = 0, weno_num_stencils + if (omega(q) < teno_CT) then ! Equation 26 + delta(q) = 0._wp + else + delta(q) = 1._wp + end if + alpha(q) = delta(q)*d_cbL_${XYZ}$ (q, ${SV}$) ! Equation 27 + end do end if + omega(0) = alpha(0)/(alpha(0) + alpha(1) + alpha(2)) + omega(1) = alpha(1)/(alpha(0) + alpha(1) + alpha(2)) + omega(2) = alpha(2)/(alpha(0) + alpha(1) + alpha(2)) + vL_rs_vf_x(j, k, l, i) = omega(0)*poly(0) + omega(1)*poly(1) + omega(2)*poly(2) + ! reconstruct from right side poly(0) = vp0 + poly_coef_cbR_${XYZ}$ (${SV}$, 0, & @@ -1210,44 +1177,36 @@ contains poly(2) = vp0 + poly_coef_cbR_${XYZ}$ (${SV}$, 2, & & 0)*dvd(-1) + poly_coef_cbR_${XYZ}$ (${SV}$, 2, 1)*dvd(-2) - use_central = .false. - if (hybrid_weno) use_central = .not. weno_full(${SF('')}$) - if (use_central) then - vR_rs_vf_x(j, k, l, i) = d_cbR_${XYZ}$ (0, ${SV}$)*poly(0) + d_cbR_${XYZ}$ (1, & - & ${SV}$)*poly(1) + d_cbR_${XYZ}$ (2, ${SV}$)*poly(2) - else - if (wenojs) then - do q = 0, weno_num_stencils - alpha(q) = d_cbR_${XYZ}$ (q, ${SV}$)/(beta(q)**2._wp) - end do - else if (mapped_weno) then - do q = 0, weno_num_stencils - alpha(q) = d_cbR_${XYZ}$ (q, ${SV}$)/(beta(q)**2._wp) - end do - omega = alpha/sum(alpha) - do q = 0, weno_num_stencils - alpha(q) = (d_cbR_${XYZ}$ (q, ${SV}$)*(1._wp + d_cbR_${XYZ}$ (q, & - & ${SV}$) - 3._wp*omega(q)) + omega(q)**2._wp)*(omega(q) & - & /(d_cbR_${XYZ}$ (q, & - & ${SV}$)**2._wp + omega(q)*(1._wp - 2._wp*d_cbR_${XYZ}$ (q, ${SV}$)))) - end do - else if (wenoz) then - $:GPU_LOOP(parallelism='[seq]') - do q = 0, weno_num_stencils - alpha(q) = d_cbR_${XYZ}$ (q, ${SV}$)*(1._wp + (tau/beta(q))) - end do - else if (teno) then - $:GPU_LOOP(parallelism='[seq]') - do q = 0, weno_num_stencils - alpha(q) = delta(q)*d_cbR_${XYZ}$ (q, ${SV}$) - end do - end if - - omega(0) = alpha(0)/(alpha(0) + alpha(1) + alpha(2)) - omega(1) = alpha(1)/(alpha(0) + alpha(1) + alpha(2)) - omega(2) = alpha(2)/(alpha(0) + alpha(1) + alpha(2)) - vR_rs_vf_x(j, k, l, i) = omega(0)*poly(0) + omega(1)*poly(1) + omega(2)*poly(2) + if (wenojs) then + do q = 0, weno_num_stencils + alpha(q) = d_cbR_${XYZ}$ (q, ${SV}$)/(beta(q)**2._wp) + end do + else if (mapped_weno) then + do q = 0, weno_num_stencils + alpha(q) = d_cbR_${XYZ}$ (q, ${SV}$)/(beta(q)**2._wp) + end do + omega = alpha/sum(alpha) + do q = 0, weno_num_stencils + alpha(q) = (d_cbR_${XYZ}$ (q, ${SV}$)*(1._wp + d_cbR_${XYZ}$ (q, & + & ${SV}$) - 3._wp*omega(q)) + omega(q)**2._wp)*(omega(q)/(d_cbR_${XYZ}$ (q, & + & ${SV}$)**2._wp + omega(q)*(1._wp - 2._wp*d_cbR_${XYZ}$ (q, ${SV}$)))) + end do + else if (wenoz) then + $:GPU_LOOP(parallelism='[seq]') + do q = 0, weno_num_stencils + alpha(q) = d_cbR_${XYZ}$ (q, ${SV}$)*(1._wp + (tau/beta(q))) + end do + else if (teno) then + $:GPU_LOOP(parallelism='[seq]') + do q = 0, weno_num_stencils + alpha(q) = delta(q)*d_cbR_${XYZ}$ (q, ${SV}$) + end do end if + + omega(0) = alpha(0)/(alpha(0) + alpha(1) + alpha(2)) + omega(1) = alpha(1)/(alpha(0) + alpha(1) + alpha(2)) + omega(2) = alpha(2)/(alpha(0) + alpha(1) + alpha(2)) + vR_rs_vf_x(j, k, l, i) = omega(0)*poly(0) + omega(1)*poly(1) + omega(2)*poly(2) end do end do end do @@ -1271,7 +1230,7 @@ contains #:set SF = lambda offs: COORDS.format(STENCIL_IDX = SV + offs) if (weno_dir == ${WENO_DIR}$) then $:GPU_PARALLEL_LOOP(collapse=3,private='[poly, beta, alpha, omega, tau, delta, dvd, v, q, vp0, vp1, vp2, & - & use_central, vp3, vm1, vm2, vm3]') + & vp3, vm1, vm2, vm3]') do l = ${Z_BND}$%beg, ${Z_BND}$%end do k = ${Y_BND}$%beg, ${Y_BND}$%end do j = ${X_BND}$%beg, ${X_BND}$%end @@ -1377,58 +1336,49 @@ contains #:endif end if - use_central = .false. - if (hybrid_weno) use_central = .not. weno_full(${SF('')}$) - if (use_central) then + if (wenojs) then do q = 0, weno_num_stencils - omega(q) = d_cbL_${XYZ}$ (q, ${SV}$) + alpha(q) = d_cbL_${XYZ}$ (q, ${SV}$)/(beta(q)**2._wp) end do - else - if (wenojs) then - do q = 0, weno_num_stencils - alpha(q) = d_cbL_${XYZ}$ (q, ${SV}$)/(beta(q)**2._wp) - end do - else if (mapped_weno) then - do q = 0, weno_num_stencils - alpha(q) = d_cbL_${XYZ}$ (q, ${SV}$)/(beta(q)**2._wp) - end do + else if (mapped_weno) then + do q = 0, weno_num_stencils + alpha(q) = d_cbL_${XYZ}$ (q, ${SV}$)/(beta(q)**2._wp) + end do + omega = alpha/sum(alpha) + do q = 0, weno_num_stencils + alpha(q) = (d_cbL_${XYZ}$ (q, ${SV}$)*(1._wp + d_cbL_${XYZ}$ (q, & + & ${SV}$) - 3._wp*omega(q)) + omega(q)**2._wp)*(omega(q)/(d_cbL_${XYZ}$ (q, & + & ${SV}$)**2._wp + omega(q)*(1._wp - 2._wp*d_cbL_${XYZ}$ (q, ${SV}$)))) + end do + else if (wenoz) then + ! Castro, et al. (2010) Don & Borges (2013) also helps + tau = abs(beta(3) - beta(0)) ! Equation 50 + $:GPU_LOOP(parallelism='[seq]') + do q = 0, weno_num_stencils + ! wenoz_q = 2,3,4 for stability + alpha(q) = d_cbL_${XYZ}$ (q, ${SV}$)*(1._wp + (tau/beta(q))**wenoz_q) + end do + else if (teno) then + #:if not MFC_CASE_OPTIMIZATION or weno_num_stencils > 3 + tau = abs(beta(4) - beta(3)) ! Note the reordering of stencils + alpha = 1._wp + tau/beta + alpha = (alpha**3._wp)**2._wp ! some CPU compilers cannot optimize x**6.0 omega = alpha/sum(alpha) - do q = 0, weno_num_stencils - alpha(q) = (d_cbL_${XYZ}$ (q, ${SV}$)*(1._wp + d_cbL_${XYZ}$ (q, & - & ${SV}$) - 3._wp*omega(q)) + omega(q)**2._wp)*(omega(q) & - & /(d_cbL_${XYZ}$ (q, & - & ${SV}$)**2._wp + omega(q)*(1._wp - 2._wp*d_cbL_${XYZ}$ (q, ${SV}$)))) - end do - else if (wenoz) then - ! Castro, et al. (2010) Don & Borges (2013) also helps - tau = abs(beta(3) - beta(0)) ! Equation 50 + $:GPU_LOOP(parallelism='[seq]') do q = 0, weno_num_stencils - ! wenoz_q = 2,3,4 for stability - alpha(q) = d_cbL_${XYZ}$ (q, ${SV}$)*(1._wp + (tau/beta(q))**wenoz_q) + if (omega(q) < teno_CT) then ! Equation 26 + delta(q) = 0._wp + else + delta(q) = 1._wp + end if + alpha(q) = delta(q)*d_cbL_${XYZ}$ (q, ${SV}$) ! Equation 27 end do - else if (teno) then - #:if not MFC_CASE_OPTIMIZATION or weno_num_stencils > 3 - tau = abs(beta(4) - beta(3)) ! Note the reordering of stencils - alpha = 1._wp + tau/beta - alpha = (alpha**3._wp)**2._wp ! some CPU compilers cannot optimize x**6.0 - omega = alpha/sum(alpha) - - $:GPU_LOOP(parallelism='[seq]') - do q = 0, weno_num_stencils - if (omega(q) < teno_CT) then ! Equation 26 - delta(q) = 0._wp - else - delta(q) = 1._wp - end if - alpha(q) = delta(q)*d_cbL_${XYZ}$ (q, ${SV}$) ! Equation 27 - end do - #:endif - end if - - omega = alpha/sum(alpha) + #:endif end if + omega = alpha/sum(alpha) + vL_rs_vf_x(j, k, l, & & i) = omega(0)*poly(0) + omega(1)*poly(1) + omega(2)*poly(2) + omega(3)*poly(3) @@ -1461,44 +1411,35 @@ contains #:endif end if - use_central = .false. - if (hybrid_weno) use_central = .not. weno_full(${SF('')}$) - if (use_central) then + if (wenojs) then do q = 0, weno_num_stencils - omega(q) = d_cbR_${XYZ}$ (q, ${SV}$) + alpha(q) = d_cbR_${XYZ}$ (q, ${SV}$)/(beta(q)**2._wp) + end do + else if (mapped_weno) then + do q = 0, weno_num_stencils + alpha(q) = d_cbR_${XYZ}$ (q, ${SV}$)/(beta(q)**2._wp) end do - else - if (wenojs) then - do q = 0, weno_num_stencils - alpha(q) = d_cbR_${XYZ}$ (q, ${SV}$)/(beta(q)**2._wp) - end do - else if (mapped_weno) then - do q = 0, weno_num_stencils - alpha(q) = d_cbR_${XYZ}$ (q, ${SV}$)/(beta(q)**2._wp) - end do - omega = alpha/sum(alpha) - do q = 0, weno_num_stencils - alpha(q) = (d_cbR_${XYZ}$ (q, ${SV}$)*(1._wp + d_cbR_${XYZ}$ (q, & - & ${SV}$) - 3._wp*omega(q)) + omega(q)**2._wp)*(omega(q) & - & /(d_cbR_${XYZ}$ (q, & - & ${SV}$)**2._wp + omega(q)*(1._wp - 2._wp*d_cbR_${XYZ}$ (q, ${SV}$)))) - end do - else if (wenoz) then - $:GPU_LOOP(parallelism='[seq]') - do q = 0, weno_num_stencils - ! wenoz_q = 2,3,4 for stability - alpha(q) = d_cbR_${XYZ}$ (q, ${SV}$)*(1._wp + (tau/beta(q))**wenoz_q) - end do - else if (teno) then - $:GPU_LOOP(parallelism='[seq]') - do q = 0, weno_num_stencils - alpha(q) = delta(q)*d_cbR_${XYZ}$ (q, ${SV}$) - end do - end if - omega = alpha/sum(alpha) + do q = 0, weno_num_stencils + alpha(q) = (d_cbR_${XYZ}$ (q, ${SV}$)*(1._wp + d_cbR_${XYZ}$ (q, & + & ${SV}$) - 3._wp*omega(q)) + omega(q)**2._wp)*(omega(q)/(d_cbR_${XYZ}$ (q, & + & ${SV}$)**2._wp + omega(q)*(1._wp - 2._wp*d_cbR_${XYZ}$ (q, ${SV}$)))) + end do + else if (wenoz) then + $:GPU_LOOP(parallelism='[seq]') + do q = 0, weno_num_stencils + ! wenoz_q = 2,3,4 for stability + alpha(q) = d_cbR_${XYZ}$ (q, ${SV}$)*(1._wp + (tau/beta(q))**wenoz_q) + end do + else if (teno) then + $:GPU_LOOP(parallelism='[seq]') + do q = 0, weno_num_stencils + alpha(q) = delta(q)*d_cbR_${XYZ}$ (q, ${SV}$) + end do end if + omega = alpha/sum(alpha) + vR_rs_vf_x(j, k, l, & & i) = omega(0)*poly(0) + omega(1)*poly(1) + omega(2)*poly(2) + omega(3)*poly(3) @@ -1643,11 +1584,6 @@ contains @:DEALLOCATE(v_rs_weno) - if (hybrid_weno .or. hybrid_riemann) then - @:DEALLOCATE(weno_full) - @:DEALLOCATE(weno_disc) - end if - ! Deallocating WENO coefficients in x-direction @:DEALLOCATE(poly_coef_cbL_x, poly_coef_cbR_x) @:DEALLOCATE(d_cbL_x, d_cbR_x) @@ -1669,160 +1605,4 @@ contains end subroutine s_finalize_weno_module - !> Compute the per-cell discontinuity flag weno_full using the Jameson sensor on density, pressure, velocity, and volume - !! fractions (Pass 1), then dilate by weno_polyn cells along each active direction (Pass 2). Must only be called when - !! hybrid_weno or hybrid_riemann is true. - impure subroutine s_compute_weno_sensor(q_prim_vf) - - type(scalar_field), dimension(sys_size), intent(in) :: q_prim_vf - integer :: j, k, l, jj, kk, ll, rho_i, p_i, ivar - real(wp) :: phi, c0, cm, cp, num, den - - rho_i = eqn_idx%cont%beg; p_i = eqn_idx%E - - ! weno_disc is module-level (allocated once at init) to avoid per-call device malloc/free. - ! Pass 1a: initialise weno_disc to .false. over full idwbuff domain - $:GPU_PARALLEL_LOOP(collapse=3) - do l = idwbuff(3)%beg, idwbuff(3)%end - do k = idwbuff(2)%beg, idwbuff(2)%end - do j = idwbuff(1)%beg, idwbuff(1)%end - weno_disc(j, k, l) = .false. - end do - end do - end do - $:END_GPU_PARALLEL_LOOP() - - ! Pass 1b: Jameson sensor on density and pressure, max over active directions. - ! Bounds shrunk by 1 per active direction so that the +/-1 stencil stays in-bounds. - ! min(1,n)/min(1,p): offset is 0 for collapsed directions (n=0/p=0), 1 for active. - $:GPU_PARALLEL_LOOP(collapse=3, private='[phi, c0, cm, cp, num, den, ivar]') - do l = idwbuff(3)%beg + min(1, p), idwbuff(3)%end - min(1, p) - do k = idwbuff(2)%beg + min(1, n), idwbuff(2)%end - min(1, n) - do j = idwbuff(1)%beg + 1, idwbuff(1)%end - 1 - phi = 0._wp - ! density (eqn_idx%cont%beg) - c0 = real(q_prim_vf(rho_i)%sf(j, k, l), wp) - cm = real(q_prim_vf(rho_i)%sf(j - 1, k, l), wp) - cp = real(q_prim_vf(rho_i)%sf(j + 1, k, l), wp) - num = abs(cp - 2._wp*c0 + cm); den = abs(cp) + 2._wp*abs(c0) + abs(cm) - phi = max(phi, num/max(den, tiny(1._wp))) - if (n > 0) then - cm = real(q_prim_vf(rho_i)%sf(j, k - 1, l), wp) - cp = real(q_prim_vf(rho_i)%sf(j, k + 1, l), wp) - num = abs(cp - 2._wp*c0 + cm); den = abs(cp) + 2._wp*abs(c0) + abs(cm) - phi = max(phi, num/max(den, tiny(1._wp))) - end if - if (p > 0) then - cm = real(q_prim_vf(rho_i)%sf(j, k, l - 1), wp) - cp = real(q_prim_vf(rho_i)%sf(j, k, l + 1), wp) - num = abs(cp - 2._wp*c0 + cm); den = abs(cp) + 2._wp*abs(c0) + abs(cm) - phi = max(phi, num/max(den, tiny(1._wp))) - end if - ! pressure (eqn_idx%E) - c0 = real(q_prim_vf(p_i)%sf(j, k, l), wp) - cm = real(q_prim_vf(p_i)%sf(j - 1, k, l), wp) - cp = real(q_prim_vf(p_i)%sf(j + 1, k, l), wp) - num = abs(cp - 2._wp*c0 + cm); den = abs(cp) + 2._wp*abs(c0) + abs(cm) - phi = max(phi, num/max(den, tiny(1._wp))) - if (n > 0) then - cm = real(q_prim_vf(p_i)%sf(j, k - 1, l), wp) - cp = real(q_prim_vf(p_i)%sf(j, k + 1, l), wp) - num = abs(cp - 2._wp*c0 + cm); den = abs(cp) + 2._wp*abs(c0) + abs(cm) - phi = max(phi, num/max(den, tiny(1._wp))) - end if - if (p > 0) then - cm = real(q_prim_vf(p_i)%sf(j, k, l - 1), wp) - cp = real(q_prim_vf(p_i)%sf(j, k, l + 1), wp) - num = abs(cp - 2._wp*c0 + cm); den = abs(cp) + 2._wp*abs(c0) + abs(cm) - phi = max(phi, num/max(den, tiny(1._wp))) - end if - ! velocity components - do ivar = eqn_idx%mom%beg, eqn_idx%mom%end - c0 = real(q_prim_vf(ivar)%sf(j, k, l), wp) - cm = real(q_prim_vf(ivar)%sf(j - 1, k, l), wp) - cp = real(q_prim_vf(ivar)%sf(j + 1, k, l), wp) - num = abs(cp - 2._wp*c0 + cm); den = abs(cp) + 2._wp*abs(c0) + abs(cm) - phi = max(phi, num/max(den, tiny(1._wp))) - if (n > 0) then - cm = real(q_prim_vf(ivar)%sf(j, k - 1, l), wp) - cp = real(q_prim_vf(ivar)%sf(j, k + 1, l), wp) - num = abs(cp - 2._wp*c0 + cm); den = abs(cp) + 2._wp*abs(c0) + abs(cm) - phi = max(phi, num/max(den, tiny(1._wp))) - end if - if (p > 0) then - cm = real(q_prim_vf(ivar)%sf(j, k, l - 1), wp) - cp = real(q_prim_vf(ivar)%sf(j, k, l + 1), wp) - num = abs(cp - 2._wp*c0 + cm); den = abs(cp) + 2._wp*abs(c0) + abs(cm) - phi = max(phi, num/max(den, tiny(1._wp))) - end if - end do - ! volume fractions - do ivar = eqn_idx%adv%beg, eqn_idx%adv%end - c0 = real(q_prim_vf(ivar)%sf(j, k, l), wp) - cm = real(q_prim_vf(ivar)%sf(j - 1, k, l), wp) - cp = real(q_prim_vf(ivar)%sf(j + 1, k, l), wp) - num = abs(cp - 2._wp*c0 + cm); den = abs(cp) + 2._wp*abs(c0) + abs(cm) - phi = max(phi, num/max(den, tiny(1._wp))) - if (n > 0) then - cm = real(q_prim_vf(ivar)%sf(j, k - 1, l), wp) - cp = real(q_prim_vf(ivar)%sf(j, k + 1, l), wp) - num = abs(cp - 2._wp*c0 + cm); den = abs(cp) + 2._wp*abs(c0) + abs(cm) - phi = max(phi, num/max(den, tiny(1._wp))) - end if - if (p > 0) then - cm = real(q_prim_vf(ivar)%sf(j, k, l - 1), wp) - cp = real(q_prim_vf(ivar)%sf(j, k, l + 1), wp) - num = abs(cp - 2._wp*c0 + cm); den = abs(cp) + 2._wp*abs(c0) + abs(cm) - phi = max(phi, num/max(den, tiny(1._wp))) - end if - end do - weno_disc(j, k, l) = phi > hybrid_weno_eps - end do - end do - end do - $:END_GPU_PARALLEL_LOOP() - - ! Pass 2: initialise weno_full to .true. (conservative default) over full idwbuff - ! domain; cells not covered by the dilation loop below (outermost ghost layers) - ! default to full WENO - safe and negligible cost. - $:GPU_PARALLEL_LOOP(collapse=3) - do l = idwbuff(3)%beg, idwbuff(3)%end - do k = idwbuff(2)%beg, idwbuff(2)%end - do j = idwbuff(1)%beg, idwbuff(1)%end - weno_full(j, k, l) = .true. - end do - end do - end do - $:END_GPU_PARALLEL_LOOP() - ! Dilation: for cells where the full +/-weno_polyn window is within the computed-weno_disc - ! region, evaluate the dilated flag explicitly. Bounds shrunk by weno_polyn per active dir - ! (min(1,n)*weno_polyn: offset is 0 for collapsed dirs, weno_polyn for active dirs). - $:GPU_PARALLEL_LOOP(collapse=3, private='[jj, kk, ll]') - do l = idwbuff(3)%beg + min(1, p)*weno_polyn, idwbuff(3)%end - min(1, p)*weno_polyn - do k = idwbuff(2)%beg + min(1, n)*weno_polyn, idwbuff(2)%end - min(1, n)*weno_polyn - do j = idwbuff(1)%beg + weno_polyn, idwbuff(1)%end - weno_polyn - weno_full(j, k, l) = .false. - $:GPU_LOOP(parallelism='[seq]') - do jj = max(j - weno_polyn, idwbuff(1)%beg), min(j + weno_polyn, idwbuff(1)%end) - if (weno_disc(jj, k, l)) weno_full(j, k, l) = .true. - end do - if (n > 0) then - $:GPU_LOOP(parallelism='[seq]') - do kk = max(k - weno_polyn, idwbuff(2)%beg), min(k + weno_polyn, idwbuff(2)%end) - if (weno_disc(j, kk, l)) weno_full(j, k, l) = .true. - end do - end if - if (p > 0) then - $:GPU_LOOP(parallelism='[seq]') - do ll = max(l - weno_polyn, idwbuff(3)%beg), min(l + weno_polyn, idwbuff(3)%end) - if (weno_disc(j, k, ll)) weno_full(j, k, l) = .true. - end do - end if - end do - end do - end do - $:END_GPU_PARALLEL_LOOP() - - end subroutine s_compute_weno_sensor - end module m_weno diff --git a/tests/01A16919/golden-metadata.txt b/tests/01A16919/golden-metadata.txt deleted file mode 100644 index 1b44d15c9f..0000000000 --- a/tests/01A16919/golden-metadata.txt +++ /dev/null @@ -1,162 +0,0 @@ -This file was created on 2026-07-11 09:13:44.189736. - -mfc.sh: - - Invocation: test --generate --only 01A16919 --no-gpu --no-build -j 1 - Lock: mpi=Yes & gpu=No & debug=No & reldebug=No & gcov=No & unified=No & single=No & mixed=No & fastmath=No - Git: d266ab7873a03abcda0c151915cd3829dca52454 on up/mega (dirty) - -simulation: - - CMake Configuration: - - CMake v3.30.5 on login10 - - C : CrayClang v19.0.0 (/opt/cray/pe/craype/2.7.34/bin/cc) - Fortran : Cray v19.0.0 (/opt/cray/pe/craype/2.7.34/bin/ftn) - - PRE_PROCESS : OFF - SIMULATION : ON - POST_PROCESS : OFF - SYSCHECK : OFF - DOCUMENTATION : OFF - ALL : OFF - - MPI : ON - OpenACC : OFF - OpenMP : OFF - - Fypp : /lustre/orion/cfd154/scratch/sbryngelson/MFC-amr/build/venv/bin/fypp - Doxygen : - - Build Type : Release - - Configuration Environment: - - CC : - CXX : - FC : - OMPI_CC : - OMPI_CXX : - OMPI_FC : - -pre_process: - - CMake Configuration: - - CMake v3.30.5 on login10 - - C : CrayClang v19.0.0 (/opt/cray/pe/craype/2.7.34/bin/cc) - Fortran : Cray v19.0.0 (/opt/cray/pe/craype/2.7.34/bin/ftn) - - PRE_PROCESS : ON - SIMULATION : OFF - POST_PROCESS : OFF - SYSCHECK : OFF - DOCUMENTATION : OFF - ALL : OFF - - MPI : ON - OpenACC : OFF - OpenMP : OFF - - Fypp : /lustre/orion/cfd154/scratch/sbryngelson/MFC-amr/build/venv/bin/fypp - Doxygen : - - Build Type : Release - - Configuration Environment: - - CC : - CXX : - FC : - OMPI_CC : - OMPI_CXX : - OMPI_FC : - -syscheck: - - CMake Configuration: - - CMake v3.30.5 on login10 - - C : CrayClang v19.0.0 (/opt/cray/pe/craype/2.7.34/bin/cc) - Fortran : Cray v19.0.0 (/opt/cray/pe/craype/2.7.34/bin/ftn) - - PRE_PROCESS : OFF - SIMULATION : OFF - POST_PROCESS : OFF - SYSCHECK : ON - DOCUMENTATION : OFF - ALL : OFF - - MPI : ON - OpenACC : OFF - OpenMP : OFF - - Fypp : /lustre/orion/cfd154/scratch/sbryngelson/MFC-amr/build/venv/bin/fypp - Doxygen : - - Build Type : Release - - Configuration Environment: - - CC : - CXX : - FC : - OMPI_CC : - OMPI_CXX : - OMPI_FC : - -CPU: - - CPU Info: - From lscpu - Architecture: x86_64 - CPU op-mode(s): 32-bit, 64-bit - Address sizes: 48 bits physical, 48 bits virtual - Byte Order: Little Endian - CPU(s): 128 - On-line CPU(s) list: 0-127 - Vendor ID: AuthenticAMD - Model name: AMD EPYC 7A53 64-Core Processor - CPU family: 25 - Model: 48 - Thread(s) per core: 2 - Core(s) per socket: 64 - Socket(s): 1 - Stepping: 1 - Frequency boost: enabled - CPU(s) scaling MHz: 56% - CPU max MHz: 3541.0149 - CPU min MHz: 1500.0000 - BogoMIPS: 3992.45 - Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca fsrm ibpb_exit_to_user - Virtualization: AMD-V - L1d cache: 2 MiB (64 instances) - L1i cache: 2 MiB (64 instances) - L2 cache: 32 MiB (64 instances) - L3 cache: 256 MiB (8 instances) - NUMA node(s): 4 - NUMA node0 CPU(s): 0-15,64-79 - NUMA node1 CPU(s): 16-31,80-95 - NUMA node2 CPU(s): 32-47,96-111 - NUMA node3 CPU(s): 48-63,112-127 - Vulnerability Gather data sampling: Not affected - Vulnerability Indirect target selection: Not affected - Vulnerability Itlb multihit: Not affected - Vulnerability L1tf: Not affected - Vulnerability Mds: Not affected - Vulnerability Meltdown: Not affected - Vulnerability Mmio stale data: Not affected - Vulnerability Reg file data sampling: Not affected - Vulnerability Retbleed: Not affected - Vulnerability Spec rstack overflow: Vulnerable - Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl - Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization - Vulnerability Spectre v2: Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected - Vulnerability Srbds: Not affected - Vulnerability Tsa: Vulnerable: No microcode - Vulnerability Tsx async abort: Not affected - Vulnerability Vmscape: Mitigation; IBPB before exit to userspace - diff --git a/tests/01A16919/golden.txt b/tests/01A16919/golden.txt deleted file mode 100644 index 72f9531718..0000000000 --- a/tests/01A16919/golden.txt +++ /dev/null @@ -1,16 +0,0 @@ -D/cons.1.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 -D/cons.1.00.000006.dat 0.99999999981854 0.9999999854378 0.99999901876213 0.99994709456392 0.99782657804113 0.94988378746252 0.54871201474988 0.50353776739815 0.50009197990888 0.50000174727949 0.50000002627029 0.50000000029928 0.49999999998144 0.50000000000204 0.50000000000087 0.49999999999993 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.50000000000007 0.49999999999953 0.49999999999618 0.50000000002044 0.49999999989974 0.49999999113615 0.49999940081457 0.49996753672374 0.49865506895459 0.46065314530066 0.16315543944641 0.12750643398112 0.12506190826174 0.12500106106778 0.12500001428499 0.1250000001217 0.12499999998798 0.12500000000227 0.12500000000042 0.12499999999995 0.125 0.125 0.125 -D/cons.2.00.000000.dat 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -D/cons.2.00.000006.dat 2.106e-10 1.695395e-08 1.12415198e-06 5.869726667e-05 0.00225751784491 0.0442274249734 0.04523800908924 0.00411009384502 0.00010505327361 2.03123479e-06 3.078429e-08 4.1441e-10 -2.185e-11 1.25e-12 1e-12 -8e-14 -0.0 0.0 -0.0 -0.0 0.0 -0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -0.0 0.0 -0.0 -0.0 0.0 0.0 -8e-14 5.6e-13 3.63e-12 -2.613e-11 1.5036e-10 1.031116e-08 6.8569269e-07 3.594508686e-05 0.00139058459327 0.0346384506723 0.03763071618113 0.00303636764032 6.609873053e-05 1.12567478e-06 1.509117e-08 2.1574e-10 -2.095e-11 2.28e-12 4.3e-13 -5e-14 0.0 0.0 -0.0 -D/cons.3.00.000000.dat 2.5 2.5 2.5 2.5 2.5 2.5 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 -D/cons.3.00.000006.dat 2.49999999932969 2.49999994987451 2.49999667471633 2.49982635338563 2.49329953343046 2.37127003020182 1.37376304741356 1.26152767637172 1.25031063456644 1.25000600847883 1.25000009110259 1.25000000109912 1.24999999994878 1.25000000000318 1.250000000003 1.24999999999977 1.24999999999999 1.25000000000001 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.24999999999999 1.25000000000023 1.24999999999834 1.2499999999894 1.25000000006603 1.24999999963281 1.24999996947518 1.24999797166218 1.24989365948843 1.24586774811781 1.14262114294853 0.35423774590675 0.25720407163461 0.25017467246085 0.25000297826912 0.25000004002453 0.25000000034953 0.24999999996923 0.25000000000538 0.25000000000118 0.24999999999987 0.25 0.25 0.25 -D/cons.4.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -D/cons.4.00.000006.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -D/prim.1.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 -D/prim.1.00.000006.dat 0.99999999981854 0.9999999854378 0.99999901876213 0.99994709456392 0.99782657804113 0.94988378746252 0.54871201474988 0.50353776739815 0.50009197990888 0.50000174727949 0.50000002627029 0.50000000029928 0.49999999998144 0.50000000000204 0.50000000000087 0.49999999999993 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.50000000000007 0.49999999999953 0.49999999999618 0.50000000002044 0.49999999989974 0.49999999113615 0.49999940081457 0.49996753672374 0.49865506895459 0.46065314530066 0.16315543944641 0.12750643398112 0.12506190826174 0.12500106106778 0.12500001428499 0.1250000001217 0.12499999998798 0.12500000000227 0.12500000000042 0.12499999999995 0.125 0.125 0.125 -D/prim.2.00.000000.dat 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -D/prim.2.00.000006.dat 2.106e-10 1.695395e-08 1.12415308e-06 5.870037224e-05 0.00226243507098 0.04656087992779 0.08244399224585 0.00816243410351 0.00021006790317 4.06245539e-06 6.156857e-08 8.2881e-10 -4.37e-11 2.51e-12 2.01e-12 -1.6e-13 -1e-14 0.0 -0.0 -0.0 0.0 -0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -0.0 0.0 -0.0 -0.0 0.0 0.0 -1.5e-13 1.11e-12 7.26e-12 -5.226e-11 3.0072e-10 2.062232e-08 1.37138703e-06 7.18948416e-05 0.00278867032513 0.07519421288157 0.2306433442171 0.023813446471 0.00052852808223 9.00532177e-06 1.2072933e-07 1.7259e-09 -1.6759e-10 1.822e-11 3.43e-12 -3.8e-13 0.0 1e-14 -0.0 -D/prim.3.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 -D/prim.3.00.000006.dat 0.99999999973188 0.9999999799498 0.99999866988628 0.99993054066514 0.99731879187468 0.94809615851599 0.54875929855131 0.50460436087465 0.50012424941291 0.50000240338988 0.50000003644103 0.50000000043965 0.49999999997951 0.50000000000127 0.5000000000012 0.49999999999991 0.49999999999999 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.50000000000009 0.49999999999934 0.49999999999576 0.50000000002641 0.49999999985312 0.49999998779007 0.49999918866468 0.49995746327852 0.49834632367072 0.45652753497267 0.13995924351764 0.10286716737819 0.10006986199733 0.10000119130562 0.10000001600981 0.10000000013981 0.09999999998769 0.10000000000215 0.10000000000047 0.09999999999995 0.1 0.1 0.1 -D/prim.4.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -D/prim.4.00.000006.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 \ No newline at end of file diff --git a/tests/053C5DDA/golden-metadata.txt b/tests/053C5DDA/golden-metadata.txt deleted file mode 100644 index b69f6e1cad..0000000000 --- a/tests/053C5DDA/golden-metadata.txt +++ /dev/null @@ -1,193 +0,0 @@ -This file was created on 2026-07-08 14:15:49.281934. - -mfc.sh: - - Invocation: test --generate --only 053C5DDA DDC4BA8A BA4340EA 60739A3E -j 8 -- -b mpirun - Lock: mpi=Yes & gpu=No & debug=No & reldebug=Yes & gcov=No & unified=No & single=No & mixed=No & fastmath=Yes - Git: ace2285a7e72fdce9dc3fbc3a5629e1b9d1a89b7 on amr-hybrid-test-robustness (dirty) - -syscheck: - - CMake Configuration: - - CMake v3.26.5 on atl1-1-02-005-27-0.pace.gatech.edu - - C : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc) - Fortran : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran) - - PRE_PROCESS : OFF - SIMULATION : OFF - POST_PROCESS : OFF - SYSCHECK : ON - DOCUMENTATION : OFF - ALL : OFF - - MPI : ON - OpenACC : OFF - OpenMP : OFF - - Fypp : /storage/project/r-sbryngelson3-0/sbryngelson3/MFC-lomach/build/venv/bin/fypp - Doxygen : - - Build Type : RelDebug - - Configuration Environment: - - CC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc - CXX : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/g++ - FC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran - OMPI_CC : - OMPI_CXX : - OMPI_FC : - -simulation: - - CMake Configuration: - - CMake v3.26.5 on atl1-1-02-005-27-0.pace.gatech.edu - - C : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc) - Fortran : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran) - - PRE_PROCESS : OFF - SIMULATION : ON - POST_PROCESS : OFF - SYSCHECK : OFF - DOCUMENTATION : OFF - ALL : OFF - - MPI : ON - OpenACC : OFF - OpenMP : OFF - - Fypp : /storage/project/r-sbryngelson3-0/sbryngelson3/MFC-lomach/build/venv/bin/fypp - Doxygen : - - Build Type : RelDebug - - Configuration Environment: - - CC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc - CXX : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/g++ - FC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran - OMPI_CC : - OMPI_CXX : - OMPI_FC : - -post_process: - - CMake Configuration: - - CMake v3.26.5 on atl1-1-02-005-27-0.pace.gatech.edu - - C : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc) - Fortran : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran) - - PRE_PROCESS : OFF - SIMULATION : OFF - POST_PROCESS : ON - SYSCHECK : OFF - DOCUMENTATION : OFF - ALL : OFF - - MPI : ON - OpenACC : OFF - OpenMP : OFF - - Fypp : /storage/project/r-sbryngelson3-0/sbryngelson3/MFC-lomach/build/venv/bin/fypp - Doxygen : - - Build Type : RelDebug - - Configuration Environment: - - CC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc - CXX : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/g++ - FC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran - OMPI_CC : - OMPI_CXX : - OMPI_FC : - -pre_process: - - CMake Configuration: - - CMake v3.26.5 on atl1-1-02-005-27-0.pace.gatech.edu - - C : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc) - Fortran : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran) - - PRE_PROCESS : ON - SIMULATION : OFF - POST_PROCESS : OFF - SYSCHECK : OFF - DOCUMENTATION : OFF - ALL : OFF - - MPI : ON - OpenACC : OFF - OpenMP : OFF - - Fypp : /storage/project/r-sbryngelson3-0/sbryngelson3/MFC-lomach/build/venv/bin/fypp - Doxygen : - - Build Type : RelDebug - - Configuration Environment: - - CC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc - CXX : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/g++ - FC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran - OMPI_CC : - OMPI_CXX : - OMPI_FC : - -CPU: - - CPU Info: - From lscpu - Architecture: x86_64 - CPU op-mode(s): 32-bit, 64-bit - Address sizes: 46 bits physical, 48 bits virtual - Byte Order: Little Endian - CPU(s): 24 - On-line CPU(s) list: 0-23 - Vendor ID: GenuineIntel - Model name: Intel(R) Xeon(R) Gold 6226 CPU @ 2.70GHz - CPU family: 6 - Model: 85 - Thread(s) per core: 1 - Core(s) per socket: 12 - Socket(s): 2 - Stepping: 7 - CPU(s) scaling MHz: 100% - CPU max MHz: 2700.0000 - CPU min MHz: 1200.0000 - BogoMIPS: 5400.00 - Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req vnmi pku ospke avx512_vnni md_clear flush_l1d arch_capabilities - Virtualization: VT-x - L1d cache: 768 KiB (24 instances) - L1i cache: 768 KiB (24 instances) - L2 cache: 24 MiB (24 instances) - L3 cache: 38.5 MiB (2 instances) - NUMA node(s): 2 - NUMA node0 CPU(s): 0-11 - NUMA node1 CPU(s): 12-23 - Vulnerability Gather data sampling: Vulnerable - Vulnerability Indirect target selection: Vulnerable - Vulnerability Itlb multihit: KVM: Vulnerable - Vulnerability L1tf: Not affected - Vulnerability Mds: Not affected - Vulnerability Meltdown: Not affected - Vulnerability Mmio stale data: Vulnerable - Vulnerability Reg file data sampling: Not affected - Vulnerability Retbleed: Vulnerable - Vulnerability Spec rstack overflow: Not affected - Vulnerability Spec store bypass: Vulnerable - Vulnerability Spectre v1: Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers - Vulnerability Spectre v2: Vulnerable; IBPB: disabled; STIBP: disabled; PBRSB-eIBRS: Vulnerable; BHI: Vulnerable - Vulnerability Srbds: Not affected - Vulnerability Tsa: Not affected - Vulnerability Tsx async abort: Mitigation; TSX disabled - Vulnerability Vmscape: Vulnerable - diff --git a/tests/053C5DDA/golden.txt b/tests/053C5DDA/golden.txt deleted file mode 100644 index 784c48de44..0000000000 --- a/tests/053C5DDA/golden.txt +++ /dev/null @@ -1,16 +0,0 @@ -D/cons.1.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 -D/cons.1.00.000006.dat 0.99999999617058 0.99999985582993 0.99999663647189 0.99996735266521 0.99908351683917 0.96073412066057 0.53914862171741 0.50102365809854 0.5000421607134 0.50000391685696 0.5000001597969 0.50000000431672 0.50000000009998 0.49999999997689 0.50000000000336 0.50000000000068 0.49999999999993 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.50000000000004 0.49999999999964 0.49999999999727 0.50000000001615 0.49999999993775 0.49999999667003 0.4999999321491 0.49999117663288 0.49916113602137 0.46663030626443 0.1582223114688 0.12597978301782 0.12501524230833 0.12500011073826 0.12500000473267 0.12500000005243 0.12499999999049 0.12500000000228 0.12500000000029 0.12499999999996 0.125 0.125 0.125 -D/cons.2.00.000000.dat 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -D/cons.2.00.000006.dat 4.51027e-09 1.7059395e-07 3.97808351e-06 3.91367436e-05 0.00099561529391 0.04700678678779 0.04675047711094 0.00114978949719 4.922374231e-05 4.62308485e-06 1.8906465e-07 5.12374e-09 1.2606e-10 -2.99e-11 4e-12 8e-13 -9e-14 -0.0 0.0 -0.0 -0.0 -0.0 -0.0 0.0 0.0 0.0 0.0 -0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -0.0 0.0 -0.0 -0.0 0.0 0.0 -5e-14 4.2e-13 3.24e-12 -2.033e-11 7.645e-11 3.94737e-09 8.702671e-08 1.043402465e-05 0.00099039153922 0.03637521810001 0.03832946148979 0.0010781387651 1.613453186e-05 1.2542421e-07 5.00901e-09 9.674e-11 -1.718e-11 2.52e-12 3e-13 -4e-14 0.0 0.0 -0.0 -D/cons.3.00.000000.dat 2.5 2.5 2.5 2.5 2.5 2.5 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 -D/cons.3.00.000006.dat 2.49999998659702 2.49999949540494 2.49998822776409 2.49988605244181 2.49679634072829 2.37447661136589 1.37507856921654 1.25361348366065 1.25014694963717 1.25001370926597 1.25000055928962 1.25000001510851 1.25000000034993 1.24999999991913 1.25000000001177 1.25000000000238 1.24999999999975 1.24999999999999 1.25000000000001 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25000000000015 1.24999999999875 1.24999999999044 1.25000000005654 1.24999999978214 1.24999998834509 1.24999976252195 1.24996911948904 1.24707445137907 1.15164258306816 0.3484790837807 0.25279199644208 0.25004269169899 0.25000031006817 0.25000001325147 0.2500000001468 0.24999999997336 0.25000000000638 0.25000000000082 0.24999999999989 0.25 0.25 0.25 -D/cons.4.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -D/cons.4.00.000006.dat 1.0 1.0 1.0 1.00002707840114 1.00000000002821 1.0 1.0 1.0 1.0 1.00000035150394 1.00000000000007 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.00000053313484 1.00000000291781 0.99999999999995 0.99999999999974 1.0000000000003 1.0 1.0 0.99999690875649 0.99999999999995 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -D/prim.1.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 -D/prim.1.00.000006.dat 0.99999999617058 0.99999985582993 0.99999663647189 0.99996735266521 0.99908351683917 0.96073412066057 0.53914862171741 0.50102365809854 0.5000421607134 0.50000391685696 0.5000001597969 0.50000000431672 0.50000000009998 0.49999999997689 0.50000000000336 0.50000000000068 0.49999999999993 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.50000000000004 0.49999999999964 0.49999999999727 0.50000000001615 0.49999999993775 0.49999999667003 0.4999999321491 0.49999117663288 0.49916113602137 0.46663030626443 0.1582223114688 0.12597978301782 0.12501524230833 0.12500011073826 0.12500000473267 0.12500000005243 0.12499999999049 0.12500000000228 0.12500000000029 0.12499999999996 0.125 0.125 0.125 -D/prim.2.00.000000.dat 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -D/prim.2.00.000006.dat 4.51027e-09 1.7059397e-07 3.97809689e-06 3.913802135e-05 0.00099652859559 0.04892798723071 0.08671166952448 0.00229488064806 9.843918408e-05 9.24609726e-06 3.7812917e-07 1.024747e-08 2.5212e-10 -5.981e-11 7.99e-12 1.6e-12 -1.7e-13 -1e-14 0.0 -0.0 -0.0 -0.0 -0.0 0.0 0.0 0.0 0.0 -0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -0.0 0.0 -0.0 -0.0 0.0 0.0 -1e-13 8.5e-13 6.47e-12 -4.066e-11 1.5289e-10 7.89474e-09 1.7405345e-07 2.086841756e-05 0.00198411187841 0.07795296964573 0.24225067333404 0.00855803002092 0.00012906051746 1.00339282e-06 4.007205e-08 7.7394e-10 -1.3741e-10 2.017e-11 2.39e-12 -3.3e-13 1e-14 1e-14 -0.0 -D/prim.3.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 -D/prim.3.00.000006.dat 0.99999999463881 0.99999979816197 0.99999529110247 0.99992734423664 0.99871833783132 0.94933065505362 0.54922066530235 0.50144486573833 0.50005877888576 0.500005307944 0.5000002237158 0.50000000604341 0.50000000013997 0.49999999996765 0.50000000000471 0.50000000000095 0.4999999999999 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.50000000000006 0.4999999999995 0.49999999999618 0.50000000002262 0.49999999991286 0.49999999533804 0.49999963844155 0.4999876462932 0.49882938754213 0.4600899219729 0.13753456594135 0.10111495322805 0.10001707626313 0.10000043315293 0.10000000530059 0.10000000005872 0.09999999998935 0.10000000000255 0.10000000000033 0.09999999999996 0.1 0.1 0.1 -D/prim.4.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -D/prim.4.00.000006.dat 1.0 1.0 1.0 1.00002707840114 1.00000000002821 1.0 1.0 1.0 1.0 1.00000035150394 1.00000000000007 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.00000053313484 1.00000000291781 0.99999999999995 0.99999999999974 1.0000000000003 1.0 1.0 0.99999690875649 0.99999999999995 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 \ No newline at end of file diff --git a/tests/21272AFB/golden.txt b/tests/21272AFB/golden.txt deleted file mode 100644 index 228bb62995..0000000000 --- a/tests/21272AFB/golden.txt +++ /dev/null @@ -1,16 +0,0 @@ -D/cons.1.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 -D/cons.1.00.000006.dat 0.99999999979591 0.99999998472683 0.99999899243059 0.99994780428952 0.99800633490866 0.96114045213925 0.53777269575953 0.5030507498549 0.5000813962002 0.50000156595549 0.50000002364808 0.50000000028342 0.49999999998468 0.50000000000118 0.50000000000082 0.49999999999993 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.50000000000007 0.49999999999952 0.49999999999697 0.50000000001892 0.49999999989456 0.49999999126629 0.49999942266678 0.49996995381996 0.49884345037773 0.46690023384529 0.15684108925066 0.12738536102708 0.12505946967055 0.12500101444172 0.12500001361476 0.12500000011784 0.12499999998896 0.12500000000198 0.12500000000041 0.12499999999995 0.125 0.125 0.125 -D/cons.2.00.000000.dat 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -D/cons.2.00.000006.dat 2.2435e-10 1.808107e-08 1.1921741e-06 6.175499736e-05 0.00235343211297 0.04637431088153 0.04334870246431 0.00376226359271 9.644425937e-05 1.85291298e-06 2.79646e-08 3.7774e-10 -2.271e-11 1.56e-12 9.6e-13 -8e-14 -0.0 0.0 -0.0 -0.0 -0.0 -0.0 -0.0 0.0 0.0 -0.0 0.0 -0.0 0.0 0.0 -0.0 0.0 0.0 -0.0 -0.0 0.0 0.0 -0.0 -0.0 0.0 0.0 -8e-14 5.6e-13 3.63e-12 -2.622e-11 1.5113e-10 1.032549e-08 6.8310027e-07 3.554859267e-05 0.00136476605923 0.03577661660379 0.03668359552735 0.00287444790368 6.324352877e-05 1.07366847e-06 1.437255e-08 2.0695e-10 -2.094e-11 2.33e-12 4.2e-13 -5e-14 0.0 0.0 -0.0 -D/cons.3.00.000000.dat 2.5 2.5 2.5 2.5 2.5 2.5 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 -D/cons.3.00.000006.dat 2.49999999928568 2.49999994654389 2.4999964735218 2.49981734638575 2.49305675656102 2.37634675700937 1.36967354333457 1.26081856973676 1.25028504291838 1.25000548091579 1.25000008276828 1.25000000099198 1.24999999994637 1.25000000000411 1.25000000000286 1.24999999999976 1.24999999999999 1.25000000000001 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.24999999999999 1.25000000000023 1.24999999999833 1.24999999998939 1.25000000006623 1.24999999963097 1.249999969432 1.24999797934343 1.24989485925863 1.24597556839486 1.15270465024259 0.34422215943072 0.25703510060941 0.25016683463152 0.25000284054476 0.25000003812134 0.25000000032996 0.24999999996909 0.25000000000553 0.25000000000115 0.24999999999987 0.25 0.25 0.25 -D/cons.4.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -D/cons.4.00.000006.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -D/prim.1.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 -D/prim.1.00.000006.dat 0.99999999979591 0.99999998472683 0.99999899243059 0.99994780428952 0.99800633490866 0.96114045213925 0.53777269575953 0.5030507498549 0.5000813962002 0.50000156595549 0.50000002364808 0.50000000028342 0.49999999998468 0.50000000000118 0.50000000000082 0.49999999999993 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.50000000000007 0.49999999999952 0.49999999999697 0.50000000001892 0.49999999989456 0.49999999126629 0.49999942266678 0.49996995381996 0.49884345037773 0.46690023384529 0.15684108925066 0.12738536102708 0.12505946967055 0.12500101444172 0.12500001361476 0.12500000011784 0.12499999998896 0.12500000000198 0.12500000000041 0.12499999999995 0.125 0.125 0.125 -D/prim.2.00.000000.dat 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -D/prim.2.00.000006.dat 2.2435e-10 1.808107e-08 1.1921753e-06 6.175822088e-05 0.00235813344129 0.04824925512011 0.08060785310619 0.0074788947115 0.00019285712307 3.70581434e-06 5.592919e-08 7.5549e-10 -4.541e-11 3.12e-12 1.91e-12 -1.6e-13 -1e-14 0.0 -0.0 -0.0 -0.0 -0.0 -0.0 0.0 0.0 -0.0 0.0 -0.0 0.0 0.0 -0.0 0.0 0.0 -0.0 -0.0 0.0 0.0 -0.0 -0.0 0.0 0.0 -1.5e-13 1.12e-12 7.27e-12 -5.243e-11 3.0225e-10 2.065098e-08 1.36620211e-06 7.1101458e-05 0.00273586043516 0.07662582712616 0.23389021144023 0.02256497827149 0.00050570763599 8.58927807e-06 1.1498037e-07 1.65559e-09 -1.6754e-10 1.86e-11 3.32e-12 -3.8e-13 1e-14 1e-14 -0.0 -D/prim.3.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 -D/prim.3.00.000006.dat 0.99999999971427 0.99999997861756 0.99999858940844 0.99992693779153 0.99722159268301 0.9500911976124 0.54717056816571 0.50432180038005 0.50011401344736 0.50000219236494 0.50000003310731 0.50000000039679 0.49999999997855 0.50000000000165 0.50000000000114 0.4999999999999 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.50000000000009 0.49999999999933 0.49999999999575 0.50000000002649 0.49999999985239 0.4999999877728 0.49999919173719 0.49995794319794 0.49838948059605 0.46053357752923 0.13597287698943 0.10280106787287 0.10006672745606 0.10000113621606 0.10000001524854 0.10000000013198 0.09999999998764 0.10000000000221 0.10000000000046 0.09999999999995 0.1 0.1 0.1 -D/prim.4.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -D/prim.4.00.000006.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 \ No newline at end of file diff --git a/tests/21272AFB/golden-metadata.txt b/tests/4AF96C49/golden-metadata.txt similarity index 60% rename from tests/21272AFB/golden-metadata.txt rename to tests/4AF96C49/golden-metadata.txt index f8fdd416a9..abd81be0cc 100644 --- a/tests/21272AFB/golden-metadata.txt +++ b/tests/4AF96C49/golden-metadata.txt @@ -1,109 +1,109 @@ -This file was created on 2026-07-05 13:30:24.601259. +This file was created on 2026-07-15 14:53:40.911966. mfc.sh: - Invocation: test --generate --only 21272AFB --no-gpu --no-reldebug --no-debug -j 2 - Lock: mpi=No & gpu=No & debug=No & reldebug=No & gcov=No & unified=No & single=No & mixed=No & fastmath=No - Git: 43e4b6c7bf221305f9701ce06d1ea07cccaabd50 on up/mega (dirty) + Invocation: test --generate --only 4AF96C49 --gpu acc --no-build -j 1 + Lock: mpi=Yes & gpu=Acc & debug=No & reldebug=No & gcov=No & unified=No & single=No & mixed=No & fastmath=No + Git: 205d4e6a3c1f2c9a33ce1ad47ab08994dc226cdf on amr-scaling (dirty) -simulation: +pre_process: CMake Configuration: - CMake v3.26.5 on atl1-1-01-003-35-0.pace.gatech.edu + CMake v3.26.5 on atl1-1-01-007-7-0.pace.gatech.edu - C : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc) - Fortran : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran) + C : NVHPC v24.5.0 (/usr/local/pace-apps/manual/packages/nvhpc/24.5/Linux_x86_64/24.5/compilers/bin/nvc) + Fortran : NVHPC v24.5.0 (/usr/local/pace-apps/manual/packages/nvhpc/24.5/Linux_x86_64/24.5/compilers/bin/nvfortran) - PRE_PROCESS : OFF - SIMULATION : ON + PRE_PROCESS : ON + SIMULATION : OFF POST_PROCESS : OFF SYSCHECK : OFF DOCUMENTATION : OFF ALL : OFF - MPI : OFF - OpenACC : OFF + MPI : ON + OpenACC : ON OpenMP : OFF - Fypp : /storage/project/r-sbryngelson3-0/sbryngelson3/MFC-lomach/build/venv/bin/fypp + Fypp : /storage/home/hcoda1/6/sbryngelson3/r-sbryngelson3-0/MFC-lomach/build/venv/bin/fypp Doxygen : Build Type : Release Configuration Environment: - CC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc - CXX : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/g++ - FC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran + CC : /usr/local/pace-apps/manual/packages/nvhpc/24.5/Linux_x86_64/24.5/compilers/bin/nvc + CXX : /usr/local/pace-apps/manual/packages/nvhpc/24.5/Linux_x86_64/24.5/compilers/bin/nvc++ + FC : /usr/local/pace-apps/manual/packages/nvhpc/24.5/Linux_x86_64/24.5/compilers/bin/nvfortran OMPI_CC : OMPI_CXX : OMPI_FC : -syscheck: +simulation: CMake Configuration: - CMake v3.26.5 on atl1-1-01-003-35-0.pace.gatech.edu + CMake v3.26.5 on atl1-1-01-007-7-0.pace.gatech.edu - C : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc) - Fortran : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran) + C : NVHPC v24.5.0 (/usr/local/pace-apps/manual/packages/nvhpc/24.5/Linux_x86_64/24.5/compilers/bin/nvc) + Fortran : NVHPC v24.5.0 (/usr/local/pace-apps/manual/packages/nvhpc/24.5/Linux_x86_64/24.5/compilers/bin/nvfortran) PRE_PROCESS : OFF - SIMULATION : OFF + SIMULATION : ON POST_PROCESS : OFF - SYSCHECK : ON + SYSCHECK : OFF DOCUMENTATION : OFF ALL : OFF - MPI : OFF - OpenACC : OFF + MPI : ON + OpenACC : ON OpenMP : OFF - Fypp : /storage/project/r-sbryngelson3-0/sbryngelson3/MFC-lomach/build/venv/bin/fypp + Fypp : /storage/home/hcoda1/6/sbryngelson3/r-sbryngelson3-0/MFC-lomach/build/venv/bin/fypp Doxygen : Build Type : Release Configuration Environment: - CC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc - CXX : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/g++ - FC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran + CC : /usr/local/pace-apps/manual/packages/nvhpc/24.5/Linux_x86_64/24.5/compilers/bin/nvc + CXX : /usr/local/pace-apps/manual/packages/nvhpc/24.5/Linux_x86_64/24.5/compilers/bin/nvc++ + FC : /usr/local/pace-apps/manual/packages/nvhpc/24.5/Linux_x86_64/24.5/compilers/bin/nvfortran OMPI_CC : OMPI_CXX : OMPI_FC : -pre_process: +syscheck: CMake Configuration: - CMake v3.26.5 on atl1-1-01-003-35-0.pace.gatech.edu + CMake v3.26.5 on atl1-1-01-007-7-0.pace.gatech.edu - C : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc) - Fortran : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran) + C : NVHPC v24.5.0 (/usr/local/pace-apps/manual/packages/nvhpc/24.5/Linux_x86_64/24.5/compilers/bin/nvc) + Fortran : NVHPC v24.5.0 (/usr/local/pace-apps/manual/packages/nvhpc/24.5/Linux_x86_64/24.5/compilers/bin/nvfortran) - PRE_PROCESS : ON + PRE_PROCESS : OFF SIMULATION : OFF POST_PROCESS : OFF - SYSCHECK : OFF + SYSCHECK : ON DOCUMENTATION : OFF ALL : OFF - MPI : OFF - OpenACC : OFF + MPI : ON + OpenACC : ON OpenMP : OFF - Fypp : /storage/project/r-sbryngelson3-0/sbryngelson3/MFC-lomach/build/venv/bin/fypp + Fypp : /storage/home/hcoda1/6/sbryngelson3/r-sbryngelson3-0/MFC-lomach/build/venv/bin/fypp Doxygen : Build Type : Release Configuration Environment: - CC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc - CXX : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/g++ - FC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran + CC : /usr/local/pace-apps/manual/packages/nvhpc/24.5/Linux_x86_64/24.5/compilers/bin/nvc + CXX : /usr/local/pace-apps/manual/packages/nvhpc/24.5/Linux_x86_64/24.5/compilers/bin/nvc++ + FC : /usr/local/pace-apps/manual/packages/nvhpc/24.5/Linux_x86_64/24.5/compilers/bin/nvfortran OMPI_CC : OMPI_CXX : OMPI_FC : diff --git a/tests/71E57E55/golden.txt b/tests/4AF96C49/golden.txt similarity index 59% rename from tests/71E57E55/golden.txt rename to tests/4AF96C49/golden.txt index a5f18c6cbe..686d88593c 100644 --- a/tests/71E57E55/golden.txt +++ b/tests/4AF96C49/golden.txt @@ -1,16 +1,16 @@ D/cons.1.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 -D/cons.1.00.000006.dat 0.99999999979591 0.99999998472683 0.99999899243059 0.99994780428952 0.99800633490866 0.96114045213925 0.53777269575953 0.5030507498549 0.5000813962002 0.50000156595549 0.50000002364808 0.50000000028342 0.49999999998468 0.50000000000118 0.50000000000082 0.49999999999993 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.50000000000007 0.49999999999952 0.49999999999697 0.50000000001892 0.49999999989456 0.49999999126629 0.49999942266678 0.49996995381996 0.49884345037773 0.46690023384529 0.15684108925066 0.12738536102708 0.12505946967055 0.12500101444172 0.12500001361476 0.12500000011784 0.12499999998896 0.12500000000198 0.12500000000041 0.12499999999995 0.125 0.125 0.125 +D/cons.1.00.000006.dat 0.99999999979591 0.99999998472683 0.99999899243059 0.99994780428952 0.99800633490866 0.96114045213925 0.53777269575953 0.5030507498549 0.5000813962002 0.50000156595549 0.50000002364808 0.50000000028342 0.49999999998468 0.50000000000117 0.50000000000082 0.49999999999993 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.49999999999999 0.50000000000002 0.50000000000188 0.49999999946589 0.49999927564897 0.49997010033663 0.49884344004511 0.46690023550826 0.15684109011019 0.12738536104668 0.12505946967076 0.12500101444172 0.12500001361476 0.12500000011784 0.12499999998896 0.12500000000198 0.12500000000041 0.12499999999995 0.125 0.125 0.125 D/cons.2.00.000000.dat 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -D/cons.2.00.000006.dat 2.2435e-10 1.808107e-08 1.1921741e-06 6.175499736e-05 0.00235343211297 0.04637431088153 0.04334870246431 0.00376226359271 9.644425937e-05 1.85291298e-06 2.79646e-08 3.7774e-10 -2.271e-11 1.56e-12 9.6e-13 -8e-14 -0.0 0.0 -0.0 -0.0 -0.0 -0.0 -0.0 0.0 0.0 -0.0 0.0 0.0 -0.0 0.0 0.0 0.0 0.0 0.0 -0.0 -0.0 0.0 -0.0 -0.0 0.0 0.0 -8e-14 5.6e-13 3.63e-12 -2.622e-11 1.5113e-10 1.032549e-08 6.8310027e-07 3.554859267e-05 0.00136476605923 0.03577661660379 0.03668359552735 0.00287444790368 6.324352877e-05 1.07366847e-06 1.437255e-08 2.0695e-10 -2.094e-11 2.33e-12 4.2e-13 -5e-14 0.0 0.0 -0.0 +D/cons.2.00.000006.dat 2.2435e-10 1.808107e-08 1.1921741e-06 6.175499736e-05 0.00235343211297 0.04637431088153 0.04334870246431 0.00376226359271 9.644425937e-05 1.85291298e-06 2.79646e-08 3.7774e-10 -2.271e-11 1.56e-12 9.6e-13 -8e-14 -0.0 -0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -0.0 1e-14 -2e-14 -1.91e-12 6.6191e-10 8.5703665e-07 3.538406673e-05 0.00136476467263 0.03577661786462 0.03668359600978 0.00287444793089 6.324352899e-05 1.07366847e-06 1.437255e-08 2.0695e-10 -2.094e-11 2.33e-12 4.2e-13 -5e-14 0.0 0.0 -0.0 D/cons.3.00.000000.dat 2.5 2.5 2.5 2.5 2.5 2.5 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 -D/cons.3.00.000006.dat 2.49999999928568 2.49999994654389 2.4999964735218 2.49981734638575 2.49305675656102 2.37634675700937 1.36967354333457 1.26081856973676 1.25028504291838 1.25000548091579 1.25000008276828 1.25000000099198 1.24999999994637 1.25000000000411 1.25000000000286 1.24999999999976 1.24999999999999 1.25000000000001 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.24999999999999 1.25000000000023 1.24999999999833 1.24999999998939 1.25000000006623 1.24999999963097 1.249999969432 1.24999797934343 1.24989485925863 1.24597556839486 1.15270465024259 0.34422215943072 0.25703510060941 0.25016683463152 0.25000284054476 0.25000003812134 0.25000000032996 0.24999999996909 0.25000000000553 0.25000000000115 0.24999999999987 0.25 0.25 0.25 +D/cons.3.00.000006.dat 2.49999999928568 2.49999994654389 2.4999964735218 2.49981734638575 2.49305675656102 2.37634675700937 1.36967354333457 1.26081856973676 1.25028504291838 1.25000548091579 1.25000008276828 1.25000000099198 1.24999999994637 1.25000000000411 1.25000000000286 1.24999999999976 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.24999999999998 1.25000000000006 1.25000000000656 1.2499999981306 1.24999746478626 1.2498953720646 1.24597553194693 1.15270465800327 0.34422216078828 0.25703510066963 0.25016683463211 0.25000284054476 0.25000003812134 0.25000000032996 0.24999999996909 0.25000000000553 0.25000000000115 0.24999999999987 0.25 0.25 0.25 D/cons.4.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -D/cons.4.00.000006.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 +D/cons.4.00.000006.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.00000027337241 1.00000000000564 0.99999999999835 1.00000000000128 1.00000000000002 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 D/prim.1.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 -D/prim.1.00.000006.dat 0.99999999979591 0.99999998472683 0.99999899243059 0.99994780428952 0.99800633490866 0.96114045213925 0.53777269575953 0.5030507498549 0.5000813962002 0.50000156595549 0.50000002364808 0.50000000028342 0.49999999998468 0.50000000000118 0.50000000000082 0.49999999999993 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.50000000000007 0.49999999999952 0.49999999999697 0.50000000001892 0.49999999989456 0.49999999126629 0.49999942266678 0.49996995381996 0.49884345037773 0.46690023384529 0.15684108925066 0.12738536102708 0.12505946967055 0.12500101444172 0.12500001361476 0.12500000011784 0.12499999998896 0.12500000000198 0.12500000000041 0.12499999999995 0.125 0.125 0.125 +D/prim.1.00.000006.dat 0.99999999979591 0.99999998472683 0.99999899243059 0.99994780428952 0.99800633490866 0.96114045213925 0.53777269575953 0.5030507498549 0.5000813962002 0.50000156595549 0.50000002364808 0.50000000028342 0.49999999998468 0.50000000000117 0.50000000000082 0.49999999999993 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.49999999999999 0.50000000000002 0.50000000000188 0.49999999946589 0.49999927564897 0.49997010033663 0.49884344004511 0.46690023550826 0.15684109011019 0.12738536104668 0.12505946967076 0.12500101444172 0.12500001361476 0.12500000011784 0.12499999998896 0.12500000000198 0.12500000000041 0.12499999999995 0.125 0.125 0.125 D/prim.2.00.000000.dat 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -D/prim.2.00.000006.dat 2.2435e-10 1.808107e-08 1.1921753e-06 6.175822088e-05 0.00235813344129 0.04824925512011 0.08060785310619 0.0074788947115 0.00019285712307 3.70581434e-06 5.592919e-08 7.5549e-10 -4.541e-11 3.12e-12 1.91e-12 -1.6e-13 -1e-14 0.0 -0.0 -0.0 -0.0 -0.0 -0.0 0.0 0.0 -0.0 0.0 0.0 -0.0 0.0 0.0 0.0 0.0 0.0 -0.0 -0.0 0.0 -0.0 -0.0 0.0 0.0 -1.5e-13 1.12e-12 7.27e-12 -5.243e-11 3.0225e-10 2.065098e-08 1.36620211e-06 7.1101458e-05 0.00273586043516 0.07662582712616 0.23389021144023 0.02256497827149 0.00050570763599 8.58927807e-06 1.1498037e-07 1.65559e-09 -1.6754e-10 1.86e-11 3.32e-12 -3.8e-13 1e-14 1e-14 -0.0 +D/prim.2.00.000006.dat 2.2435e-10 1.808107e-08 1.1921753e-06 6.175822088e-05 0.00235813344129 0.04824925512011 0.08060785310619 0.0074788947115 0.00019285712307 3.70581434e-06 5.592919e-08 7.5549e-10 -4.541e-11 3.12e-12 1.91e-12 -1.6e-13 -0.0 -0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -0.0 1e-14 -4e-14 -3.83e-12 1.32381e-09 1.71407579e-06 7.07723656e-05 0.00273585771221 0.07662582955367 0.23389021323433 0.02256497848161 0.00050570763779 8.58927808e-06 1.1498037e-07 1.65559e-09 -1.6754e-10 1.86e-11 3.32e-12 -3.8e-13 1e-14 1e-14 -0.0 D/prim.3.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 -D/prim.3.00.000006.dat 0.99999999971427 0.99999997861756 0.99999858940844 0.99992693779153 0.99722159268301 0.9500911976124 0.54717056816571 0.50432180038005 0.50011401344736 0.50000219236494 0.50000003310731 0.50000000039679 0.49999999997855 0.50000000000165 0.50000000000114 0.4999999999999 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.50000000000009 0.49999999999933 0.49999999999575 0.50000000002649 0.49999999985239 0.4999999877728 0.49999919173719 0.49995794319794 0.49838948059605 0.46053357752923 0.13597287698943 0.10280106787287 0.10006672745606 0.10000113621606 0.10000001524854 0.10000000013198 0.09999999998764 0.10000000000221 0.10000000000046 0.09999999999995 0.1 0.1 0.1 +D/prim.3.00.000006.dat 0.99999999971427 0.99999997861756 0.99999858940844 0.99992693779152 0.99722159268301 0.9500911976124 0.54717056816571 0.50432180038005 0.50011401344736 0.50000219236494 0.50000003310731 0.50000000039679 0.49999999997855 0.50000000000164 0.50000000000114 0.4999999999999 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.49999999999999 0.50000000000002 0.50000000000263 0.49999999925224 0.49999898591421 0.49995801165027 0.49838946601557 0.46053358059757 0.13597287749655 0.10280106789671 0.1000667274563 0.10000113621606 0.10000001524854 0.10000000013198 0.09999999998764 0.10000000000221 0.10000000000046 0.09999999999995 0.1 0.1 0.1 D/prim.4.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -D/prim.4.00.000006.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 \ No newline at end of file +D/prim.4.00.000006.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.00000027337241 1.00000000000564 0.99999999999835 1.00000000000128 1.00000000000002 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 \ No newline at end of file diff --git a/tests/60739A3E/golden-metadata.txt b/tests/60739A3E/golden-metadata.txt deleted file mode 100644 index 8b420920a4..0000000000 --- a/tests/60739A3E/golden-metadata.txt +++ /dev/null @@ -1,193 +0,0 @@ -This file was created on 2026-07-08 14:15:50.608817. - -mfc.sh: - - Invocation: test --generate --only 053C5DDA DDC4BA8A BA4340EA 60739A3E -j 8 -- -b mpirun - Lock: mpi=Yes & gpu=No & debug=No & reldebug=Yes & gcov=No & unified=No & single=No & mixed=No & fastmath=Yes - Git: ace2285a7e72fdce9dc3fbc3a5629e1b9d1a89b7 on amr-hybrid-test-robustness (dirty) - -syscheck: - - CMake Configuration: - - CMake v3.26.5 on atl1-1-02-005-27-0.pace.gatech.edu - - C : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc) - Fortran : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran) - - PRE_PROCESS : OFF - SIMULATION : OFF - POST_PROCESS : OFF - SYSCHECK : ON - DOCUMENTATION : OFF - ALL : OFF - - MPI : ON - OpenACC : OFF - OpenMP : OFF - - Fypp : /storage/project/r-sbryngelson3-0/sbryngelson3/MFC-lomach/build/venv/bin/fypp - Doxygen : - - Build Type : RelDebug - - Configuration Environment: - - CC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc - CXX : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/g++ - FC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran - OMPI_CC : - OMPI_CXX : - OMPI_FC : - -simulation: - - CMake Configuration: - - CMake v3.26.5 on atl1-1-02-005-27-0.pace.gatech.edu - - C : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc) - Fortran : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran) - - PRE_PROCESS : OFF - SIMULATION : ON - POST_PROCESS : OFF - SYSCHECK : OFF - DOCUMENTATION : OFF - ALL : OFF - - MPI : ON - OpenACC : OFF - OpenMP : OFF - - Fypp : /storage/project/r-sbryngelson3-0/sbryngelson3/MFC-lomach/build/venv/bin/fypp - Doxygen : - - Build Type : RelDebug - - Configuration Environment: - - CC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc - CXX : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/g++ - FC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran - OMPI_CC : - OMPI_CXX : - OMPI_FC : - -post_process: - - CMake Configuration: - - CMake v3.26.5 on atl1-1-02-005-27-0.pace.gatech.edu - - C : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc) - Fortran : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran) - - PRE_PROCESS : OFF - SIMULATION : OFF - POST_PROCESS : ON - SYSCHECK : OFF - DOCUMENTATION : OFF - ALL : OFF - - MPI : ON - OpenACC : OFF - OpenMP : OFF - - Fypp : /storage/project/r-sbryngelson3-0/sbryngelson3/MFC-lomach/build/venv/bin/fypp - Doxygen : - - Build Type : RelDebug - - Configuration Environment: - - CC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc - CXX : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/g++ - FC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran - OMPI_CC : - OMPI_CXX : - OMPI_FC : - -pre_process: - - CMake Configuration: - - CMake v3.26.5 on atl1-1-02-005-27-0.pace.gatech.edu - - C : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc) - Fortran : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran) - - PRE_PROCESS : ON - SIMULATION : OFF - POST_PROCESS : OFF - SYSCHECK : OFF - DOCUMENTATION : OFF - ALL : OFF - - MPI : ON - OpenACC : OFF - OpenMP : OFF - - Fypp : /storage/project/r-sbryngelson3-0/sbryngelson3/MFC-lomach/build/venv/bin/fypp - Doxygen : - - Build Type : RelDebug - - Configuration Environment: - - CC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc - CXX : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/g++ - FC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran - OMPI_CC : - OMPI_CXX : - OMPI_FC : - -CPU: - - CPU Info: - From lscpu - Architecture: x86_64 - CPU op-mode(s): 32-bit, 64-bit - Address sizes: 46 bits physical, 48 bits virtual - Byte Order: Little Endian - CPU(s): 24 - On-line CPU(s) list: 0-23 - Vendor ID: GenuineIntel - Model name: Intel(R) Xeon(R) Gold 6226 CPU @ 2.70GHz - CPU family: 6 - Model: 85 - Thread(s) per core: 1 - Core(s) per socket: 12 - Socket(s): 2 - Stepping: 7 - CPU(s) scaling MHz: 98% - CPU max MHz: 2700.0000 - CPU min MHz: 1200.0000 - BogoMIPS: 5400.00 - Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req vnmi pku ospke avx512_vnni md_clear flush_l1d arch_capabilities - Virtualization: VT-x - L1d cache: 768 KiB (24 instances) - L1i cache: 768 KiB (24 instances) - L2 cache: 24 MiB (24 instances) - L3 cache: 38.5 MiB (2 instances) - NUMA node(s): 2 - NUMA node0 CPU(s): 0-11 - NUMA node1 CPU(s): 12-23 - Vulnerability Gather data sampling: Vulnerable - Vulnerability Indirect target selection: Vulnerable - Vulnerability Itlb multihit: KVM: Vulnerable - Vulnerability L1tf: Not affected - Vulnerability Mds: Not affected - Vulnerability Meltdown: Not affected - Vulnerability Mmio stale data: Vulnerable - Vulnerability Reg file data sampling: Not affected - Vulnerability Retbleed: Vulnerable - Vulnerability Spec rstack overflow: Not affected - Vulnerability Spec store bypass: Vulnerable - Vulnerability Spectre v1: Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers - Vulnerability Spectre v2: Vulnerable; IBPB: disabled; STIBP: disabled; PBRSB-eIBRS: Vulnerable; BHI: Vulnerable - Vulnerability Srbds: Not affected - Vulnerability Tsa: Not affected - Vulnerability Tsx async abort: Mitigation; TSX disabled - Vulnerability Vmscape: Vulnerable - diff --git a/tests/60739A3E/golden.txt b/tests/60739A3E/golden.txt deleted file mode 100644 index 551b7e0966..0000000000 --- a/tests/60739A3E/golden.txt +++ /dev/null @@ -1,16 +0,0 @@ -D/cons.1.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 -D/cons.1.00.000006.dat 0.99999999980033 0.99999998502012 0.99999900704585 0.99994832773634 0.99801804658267 0.96079572985221 0.538078341149 0.50307640429696 0.50008253614088 0.50000159779027 0.50000002428395 0.50000000029376 0.49999999998476 0.50000000000107 0.50000000000084 0.49999999999993 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.50000000000007 0.49999999999952 0.49999999999697 0.50000000001892 0.49999999989456 0.49999999126629 0.49999942266678 0.49996995381996 0.49884345037773 0.46690023384529 0.15684108925066 0.12738536102708 0.12505946967055 0.12500101444172 0.12500001361476 0.12500000011784 0.12499999998896 0.12500000000198 0.12500000000041 0.12499999999995 0.125 0.125 0.125 -D/cons.2.00.000000.dat 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -D/cons.2.00.000006.dat 2.1944e-10 1.773376e-08 1.17488113e-06 6.113572093e-05 0.00233963266291 0.04625888122968 0.043446184847 0.00379325715867 9.779589911e-05 1.89058123e-06 2.871886e-08 3.9039e-10 -2.261e-11 1.44e-12 9.8e-13 -8e-14 -0.0 0.0 -0.0 -0.0 -0.0 -0.0 -0.0 0.0 0.0 0.0 0.0 0.0 -0.0 0.0 -0.0 0.0 0.0 0.0 -0.0 -0.0 0.0 -0.0 -0.0 0.0 0.0 -8e-14 5.6e-13 3.63e-12 -2.622e-11 1.5113e-10 1.032549e-08 6.8310027e-07 3.554859267e-05 0.00136476605923 0.03577661660379 0.03668359552735 0.00287444790368 6.324352877e-05 1.07366847e-06 1.437255e-08 2.0695e-10 -2.094e-11 2.33e-12 4.2e-13 -5e-14 0.0 0.0 -0.0 -D/cons.3.00.000000.dat 2.5 2.5 2.5 2.5 2.5 2.5 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 -D/cons.3.00.000006.dat 2.49999999930116 2.49999994757042 2.49999652467488 2.49981917795075 2.49309746756774 2.37620458649759 1.36967992372711 1.26090765822331 1.25028903609824 1.25000559234001 1.25000008499385 1.25000000102816 1.24999999994667 1.25000000000374 1.25000000000294 1.24999999999976 1.24999999999999 1.25000000000001 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.24999999999999 1.25000000000023 1.24999999999833 1.24999999998939 1.25000000006623 1.24999999963097 1.249999969432 1.24999797934343 1.24989485925863 1.24597556839486 1.15270465024259 0.34422215943072 0.25703510060941 0.25016683463152 0.25000284054476 0.25000003812134 0.25000000032996 0.24999999996909 0.25000000000553 0.25000000000115 0.24999999999987 0.25 0.25 0.25 -D/cons.4.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -D/cons.4.00.000006.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -D/prim.1.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 -D/prim.1.00.000006.dat 0.99999999980033 0.99999998502012 0.99999900704585 0.99994832773634 0.99801804658267 0.96079572985221 0.538078341149 0.50307640429696 0.50008253614088 0.50000159779027 0.50000002428395 0.50000000029376 0.49999999998476 0.50000000000107 0.50000000000084 0.49999999999993 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.50000000000007 0.49999999999952 0.49999999999697 0.50000000001892 0.49999999989456 0.49999999126629 0.49999942266678 0.49996995381996 0.49884345037773 0.46690023384529 0.15684108925066 0.12738536102708 0.12505946967055 0.12500101444172 0.12500001361476 0.12500000011784 0.12499999998896 0.12500000000198 0.12500000000041 0.12499999999995 0.125 0.125 0.125 -D/prim.2.00.000000.dat 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -D/prim.2.00.000006.dat 2.1944e-10 1.773376e-08 1.1748823e-06 6.113888011e-05 0.00234427891451 0.04814642675067 0.0807432329542 0.00754012139363 0.00019555951677 3.78115037e-06 5.743772e-08 7.8077e-10 -4.523e-11 2.87e-12 1.96e-12 -1.6e-13 -1e-14 0.0 -0.0 -0.0 -0.0 -0.0 -0.0 0.0 0.0 0.0 0.0 0.0 -0.0 0.0 -0.0 0.0 0.0 0.0 -0.0 -0.0 0.0 -0.0 -0.0 0.0 0.0 -1.5e-13 1.12e-12 7.27e-12 -5.243e-11 3.0225e-10 2.065098e-08 1.36620211e-06 7.1101458e-05 0.00273586043516 0.07662582712616 0.23389021144023 0.02256497827149 0.00050570763599 8.58927807e-06 1.1498037e-07 1.65559e-09 -1.6754e-10 1.86e-11 3.32e-12 -3.8e-13 1e-14 1e-14 -0.0 -D/prim.3.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 -D/prim.3.00.000006.dat 0.99999999972047 0.99999997902817 0.99999860986967 0.99992767043275 0.99723789007679 0.9500363946317 0.54717037240603 0.50435734296543 0.50011561061431 0.50000223693458 0.50000003399754 0.50000000041127 0.49999999997867 0.5000000000015 0.50000000000118 0.49999999999991 0.49999999999999 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.50000000000009 0.49999999999933 0.49999999999575 0.50000000002649 0.49999999985239 0.4999999877728 0.49999919173719 0.49995794319794 0.49838948059605 0.46053357752923 0.13597287698943 0.10280106787287 0.10006672745606 0.10000113621606 0.10000001524854 0.10000000013198 0.09999999998764 0.10000000000221 0.10000000000046 0.09999999999995 0.1 0.1 0.1 -D/prim.4.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -D/prim.4.00.000006.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 \ No newline at end of file diff --git a/tests/6ABA55B2/golden-metadata.txt b/tests/6ABA55B2/golden-metadata.txt deleted file mode 100644 index a9549f6a36..0000000000 --- a/tests/6ABA55B2/golden-metadata.txt +++ /dev/null @@ -1,162 +0,0 @@ -This file was created on 2026-07-10 21:59:26.720177. - -mfc.sh: - - Invocation: test --generate --only 6ABA55B2 01A16919 78A1FE7C --no-gpu --no-build -j 2 - Lock: mpi=Yes & gpu=No & debug=No & reldebug=No & gcov=No & unified=No & single=No & mixed=No & fastmath=No - Git: 416e414836f4dac0f790ed22f24bedba4ddbc2bb on up/mega (dirty) - -simulation: - - CMake Configuration: - - CMake v3.30.5 on login10 - - C : CrayClang v19.0.0 (/opt/cray/pe/craype/2.7.34/bin/cc) - Fortran : Cray v19.0.0 (/opt/cray/pe/craype/2.7.34/bin/ftn) - - PRE_PROCESS : OFF - SIMULATION : ON - POST_PROCESS : OFF - SYSCHECK : OFF - DOCUMENTATION : OFF - ALL : OFF - - MPI : ON - OpenACC : OFF - OpenMP : OFF - - Fypp : /lustre/orion/cfd154/scratch/sbryngelson/MFC-amr/build/venv/bin/fypp - Doxygen : - - Build Type : Release - - Configuration Environment: - - CC : - CXX : - FC : - OMPI_CC : - OMPI_CXX : - OMPI_FC : - -syscheck: - - CMake Configuration: - - CMake v3.30.5 on login10 - - C : CrayClang v19.0.0 (/opt/cray/pe/craype/2.7.34/bin/cc) - Fortran : Cray v19.0.0 (/opt/cray/pe/craype/2.7.34/bin/ftn) - - PRE_PROCESS : OFF - SIMULATION : OFF - POST_PROCESS : OFF - SYSCHECK : ON - DOCUMENTATION : OFF - ALL : OFF - - MPI : ON - OpenACC : OFF - OpenMP : OFF - - Fypp : /lustre/orion/cfd154/scratch/sbryngelson/MFC-amr/build/venv/bin/fypp - Doxygen : - - Build Type : Release - - Configuration Environment: - - CC : - CXX : - FC : - OMPI_CC : - OMPI_CXX : - OMPI_FC : - -pre_process: - - CMake Configuration: - - CMake v3.30.5 on login10 - - C : CrayClang v19.0.0 (/opt/cray/pe/craype/2.7.34/bin/cc) - Fortran : Cray v19.0.0 (/opt/cray/pe/craype/2.7.34/bin/ftn) - - PRE_PROCESS : ON - SIMULATION : OFF - POST_PROCESS : OFF - SYSCHECK : OFF - DOCUMENTATION : OFF - ALL : OFF - - MPI : ON - OpenACC : OFF - OpenMP : OFF - - Fypp : /lustre/orion/cfd154/scratch/sbryngelson/MFC-amr/build/venv/bin/fypp - Doxygen : - - Build Type : Release - - Configuration Environment: - - CC : - CXX : - FC : - OMPI_CC : - OMPI_CXX : - OMPI_FC : - -CPU: - - CPU Info: - From lscpu - Architecture: x86_64 - CPU op-mode(s): 32-bit, 64-bit - Address sizes: 48 bits physical, 48 bits virtual - Byte Order: Little Endian - CPU(s): 128 - On-line CPU(s) list: 0-127 - Vendor ID: AuthenticAMD - Model name: AMD EPYC 7A53 64-Core Processor - CPU family: 25 - Model: 48 - Thread(s) per core: 2 - Core(s) per socket: 64 - Socket(s): 1 - Stepping: 1 - Frequency boost: enabled - CPU(s) scaling MHz: 56% - CPU max MHz: 3541.0149 - CPU min MHz: 1500.0000 - BogoMIPS: 3992.45 - Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca fsrm ibpb_exit_to_user - Virtualization: AMD-V - L1d cache: 2 MiB (64 instances) - L1i cache: 2 MiB (64 instances) - L2 cache: 32 MiB (64 instances) - L3 cache: 256 MiB (8 instances) - NUMA node(s): 4 - NUMA node0 CPU(s): 0-15,64-79 - NUMA node1 CPU(s): 16-31,80-95 - NUMA node2 CPU(s): 32-47,96-111 - NUMA node3 CPU(s): 48-63,112-127 - Vulnerability Gather data sampling: Not affected - Vulnerability Indirect target selection: Not affected - Vulnerability Itlb multihit: Not affected - Vulnerability L1tf: Not affected - Vulnerability Mds: Not affected - Vulnerability Meltdown: Not affected - Vulnerability Mmio stale data: Not affected - Vulnerability Reg file data sampling: Not affected - Vulnerability Retbleed: Not affected - Vulnerability Spec rstack overflow: Vulnerable - Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl - Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization - Vulnerability Spectre v2: Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected - Vulnerability Srbds: Not affected - Vulnerability Tsa: Vulnerable: No microcode - Vulnerability Tsx async abort: Not affected - Vulnerability Vmscape: Mitigation; IBPB before exit to userspace - diff --git a/tests/6ABA55B2/golden.txt b/tests/6ABA55B2/golden.txt deleted file mode 100644 index 4c7cff1ca7..0000000000 --- a/tests/6ABA55B2/golden.txt +++ /dev/null @@ -1,16 +0,0 @@ -D/cons.1.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 -D/cons.1.00.000006.dat 0.99999999982111 0.99999998570332 0.99999903991531 0.99994847150953 0.99789678449267 0.95088527007054 0.5475456231429 0.50362993491986 0.50009309840553 0.50000176521121 0.50000002649825 0.50000000030166 0.49999999998152 0.50000000000202 0.50000000000087 0.49999999999993 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.50000000000007 0.49999999999954 0.49999999999617 0.50000000002037 0.49999999990136 0.49999999134169 0.49999941782215 0.49996870849419 0.49871974790903 0.46225850514467 0.16142357381688 0.12756716492856 0.1250618165077 0.12500105973488 0.12500001427054 0.12500000012158 0.12499999998798 0.12500000000227 0.12500000000042 0.12499999999995 0.125 0.125 0.125 -D/cons.2.00.000000.dat 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -D/cons.2.00.000006.dat 2.0759e-10 1.664336e-08 1.09968768e-06 5.714273583e-05 0.00218240079764 0.04332316997632 0.04619064008818 0.0041369380963 0.00010650681008 2.05352181e-06 3.106094e-08 4.172e-10 -2.175e-11 1.23e-12 1.01e-12 -8e-14 -0.0 0.0 -0.0 -0.0 0.0 -0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -0.0 0.0 -0.0 -0.0 0.0 0.0 -8e-14 5.5e-13 3.66e-12 -2.612e-11 1.474e-10 1.007325e-08 6.6618912e-07 3.464033672e-05 0.00132299873242 0.03320616660355 0.03920971291733 0.00295865528352 6.60099942e-05 1.12447008e-06 1.507709e-08 2.156e-10 -2.095e-11 2.28e-12 4.3e-13 -5e-14 0.0 0.0 -0.0 -D/cons.3.00.000000.dat 2.5 2.5 2.5 2.5 2.5 2.5 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 -D/cons.3.00.000006.dat 2.49999999933948 2.4999999507923 2.49999674708306 2.49983095272951 2.49352291428832 2.37367244955625 1.37076011864205 1.26189575389282 1.25031494621413 1.25000607440397 1.25000009192071 1.25000000110751 1.24999999994904 1.25000000000312 1.25000000000301 1.24999999999977 1.24999999999998 1.25000000000001 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.24999999999999 1.25000000000023 1.24999999999836 1.24999999998931 1.25000000006609 1.24999999963891 1.24999997018196 1.24999802935464 1.24989752004792 1.24606960095786 1.14673515307452 0.34973878359426 0.2573834355524 0.25017449214931 0.25000297508212 0.25000003998726 0.2500000003492 0.24999999996923 0.25000000000538 0.25000000000118 0.24999999999987 0.25 0.25 0.25 -D/cons.4.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -D/cons.4.00.000006.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -D/prim.1.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 -D/prim.1.00.000006.dat 0.99999999982111 0.99999998570332 0.99999903991531 0.99994847150953 0.99789678449267 0.95088527007054 0.5475456231429 0.50362993491986 0.50009309840553 0.50000176521121 0.50000002649825 0.50000000030166 0.49999999998152 0.50000000000202 0.50000000000087 0.49999999999993 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.50000000000007 0.49999999999954 0.49999999999617 0.50000000002037 0.49999999990136 0.49999999134169 0.49999941782215 0.49996870849419 0.49871974790903 0.46225850514467 0.16142357381688 0.12756716492856 0.1250618165077 0.12500105973488 0.12500001427054 0.12500000012158 0.12499999998798 0.12500000000227 0.12500000000042 0.12499999999995 0.125 0.125 0.125 -D/prim.2.00.000000.dat 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -D/prim.2.00.000006.dat 2.0759e-10 1.664336e-08 1.09968874e-06 5.714568046e-05 0.00218700053107 0.045560880308 0.08435943624761 0.00821424186582 0.00021297396508 4.10702912e-06 6.212187e-08 8.344e-10 -4.35e-11 2.47e-12 2.01e-12 -1.5e-13 -1e-14 0.0 -0.0 -0.0 0.0 -0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -0.0 0.0 -0.0 -0.0 0.0 0.0 -1.5e-13 1.1e-12 7.31e-12 -5.223e-11 2.948e-10 2.01465e-08 1.33237979e-06 6.92850095e-05 0.00265278994459 0.07183462550495 0.24289954676515 0.02319292182415 0.00052781893025 8.99568435e-06 1.2061671e-07 1.72479e-09 -1.6761e-10 1.823e-11 3.43e-12 -3.8e-13 0.0 1e-14 -0.0 -D/prim.3.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 -D/prim.3.00.000006.dat 0.99999999973579 0.99999998031692 0.99999869883298 0.99993238043871 0.99740821113299 0.94907421147013 0.54752472418527 0.5047515051951 0.50012597394902 0.5000024297599 0.50000003676828 0.50000000044301 0.49999999997962 0.50000000000125 0.5000000000012 0.49999999999991 0.49999999999999 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.50000000000009 0.49999999999934 0.49999999999573 0.50000000002644 0.49999999985557 0.49999998807278 0.49999921174168 0.49995900753916 0.4984271384556 0.45821699072132 0.13799070913842 0.10293965024882 0.10006978989146 0.10000119003082 0.1000000159949 0.10000000013968 0.09999999998769 0.10000000000215 0.10000000000047 0.09999999999995 0.1 0.1 0.1 -D/prim.4.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -D/prim.4.00.000006.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 \ No newline at end of file diff --git a/tests/71E57E55/golden-metadata.txt b/tests/78314D65/golden-metadata.txt similarity index 59% rename from tests/71E57E55/golden-metadata.txt rename to tests/78314D65/golden-metadata.txt index 6fcaf1fbbe..7acd37dde4 100644 --- a/tests/71E57E55/golden-metadata.txt +++ b/tests/78314D65/golden-metadata.txt @@ -1,109 +1,109 @@ -This file was created on 2026-07-05 12:53:53.157956. +This file was created on 2026-07-15 18:06:00.792136. mfc.sh: - Invocation: test --generate --only 71E57E55 C6EA340F --no-gpu --no-reldebug --no-debug -j 2 - Lock: mpi=No & gpu=No & debug=No & reldebug=No & gcov=No & unified=No & single=No & mixed=No & fastmath=No - Git: c95567303b89e8f3439f8433e7f4b3b7be716ee3 on up/mega (dirty) + Invocation: test --generate --only 78314D65 --gpu acc --no-build -j 1 + Lock: mpi=Yes & gpu=Acc & debug=No & reldebug=No & gcov=No & unified=No & single=No & mixed=No & fastmath=No + Git: e2d3e41800a226c3ef1d022a7472090cff5b308f on amr-scaling (dirty) -syscheck: +pre_process: CMake Configuration: - CMake v3.26.5 on atl1-1-01-003-35-0.pace.gatech.edu + CMake v3.26.5 on atl1-1-01-007-7-0.pace.gatech.edu - C : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc) - Fortran : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran) + C : NVHPC v24.5.0 (/usr/local/pace-apps/manual/packages/nvhpc/24.5/Linux_x86_64/24.5/compilers/bin/nvc) + Fortran : NVHPC v24.5.0 (/usr/local/pace-apps/manual/packages/nvhpc/24.5/Linux_x86_64/24.5/compilers/bin/nvfortran) - PRE_PROCESS : OFF + PRE_PROCESS : ON SIMULATION : OFF POST_PROCESS : OFF - SYSCHECK : ON + SYSCHECK : OFF DOCUMENTATION : OFF ALL : OFF - MPI : OFF - OpenACC : OFF + MPI : ON + OpenACC : ON OpenMP : OFF - Fypp : /storage/project/r-sbryngelson3-0/sbryngelson3/MFC-lomach/build/venv/bin/fypp + Fypp : /storage/home/hcoda1/6/sbryngelson3/r-sbryngelson3-0/MFC-lomach/build/venv/bin/fypp Doxygen : Build Type : Release Configuration Environment: - CC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc - CXX : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/g++ - FC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran + CC : /usr/local/pace-apps/manual/packages/nvhpc/24.5/Linux_x86_64/24.5/compilers/bin/nvc + CXX : /usr/local/pace-apps/manual/packages/nvhpc/24.5/Linux_x86_64/24.5/compilers/bin/nvc++ + FC : /usr/local/pace-apps/manual/packages/nvhpc/24.5/Linux_x86_64/24.5/compilers/bin/nvfortran OMPI_CC : OMPI_CXX : OMPI_FC : -simulation: +syscheck: CMake Configuration: - CMake v3.26.5 on atl1-1-01-003-35-0.pace.gatech.edu + CMake v3.26.5 on atl1-1-01-007-7-0.pace.gatech.edu - C : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc) - Fortran : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran) + C : NVHPC v24.5.0 (/usr/local/pace-apps/manual/packages/nvhpc/24.5/Linux_x86_64/24.5/compilers/bin/nvc) + Fortran : NVHPC v24.5.0 (/usr/local/pace-apps/manual/packages/nvhpc/24.5/Linux_x86_64/24.5/compilers/bin/nvfortran) PRE_PROCESS : OFF - SIMULATION : ON + SIMULATION : OFF POST_PROCESS : OFF - SYSCHECK : OFF + SYSCHECK : ON DOCUMENTATION : OFF ALL : OFF - MPI : OFF - OpenACC : OFF + MPI : ON + OpenACC : ON OpenMP : OFF - Fypp : /storage/project/r-sbryngelson3-0/sbryngelson3/MFC-lomach/build/venv/bin/fypp + Fypp : /storage/home/hcoda1/6/sbryngelson3/r-sbryngelson3-0/MFC-lomach/build/venv/bin/fypp Doxygen : Build Type : Release Configuration Environment: - CC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc - CXX : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/g++ - FC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran + CC : /usr/local/pace-apps/manual/packages/nvhpc/24.5/Linux_x86_64/24.5/compilers/bin/nvc + CXX : /usr/local/pace-apps/manual/packages/nvhpc/24.5/Linux_x86_64/24.5/compilers/bin/nvc++ + FC : /usr/local/pace-apps/manual/packages/nvhpc/24.5/Linux_x86_64/24.5/compilers/bin/nvfortran OMPI_CC : OMPI_CXX : OMPI_FC : -pre_process: +simulation: CMake Configuration: - CMake v3.26.5 on atl1-1-01-003-35-0.pace.gatech.edu + CMake v3.26.5 on atl1-1-01-007-7-0.pace.gatech.edu - C : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc) - Fortran : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran) + C : NVHPC v24.5.0 (/usr/local/pace-apps/manual/packages/nvhpc/24.5/Linux_x86_64/24.5/compilers/bin/nvc) + Fortran : NVHPC v24.5.0 (/usr/local/pace-apps/manual/packages/nvhpc/24.5/Linux_x86_64/24.5/compilers/bin/nvfortran) - PRE_PROCESS : ON - SIMULATION : OFF + PRE_PROCESS : OFF + SIMULATION : ON POST_PROCESS : OFF SYSCHECK : OFF DOCUMENTATION : OFF ALL : OFF - MPI : OFF - OpenACC : OFF + MPI : ON + OpenACC : ON OpenMP : OFF - Fypp : /storage/project/r-sbryngelson3-0/sbryngelson3/MFC-lomach/build/venv/bin/fypp + Fypp : /storage/home/hcoda1/6/sbryngelson3/r-sbryngelson3-0/MFC-lomach/build/venv/bin/fypp Doxygen : Build Type : Release Configuration Environment: - CC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc - CXX : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/g++ - FC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran + CC : /usr/local/pace-apps/manual/packages/nvhpc/24.5/Linux_x86_64/24.5/compilers/bin/nvc + CXX : /usr/local/pace-apps/manual/packages/nvhpc/24.5/Linux_x86_64/24.5/compilers/bin/nvc++ + FC : /usr/local/pace-apps/manual/packages/nvhpc/24.5/Linux_x86_64/24.5/compilers/bin/nvfortran OMPI_CC : OMPI_CXX : OMPI_FC : @@ -126,7 +126,7 @@ CPU: Core(s) per socket: 12 Socket(s): 2 Stepping: 7 - CPU(s) scaling MHz: 95% + CPU(s) scaling MHz: 65% CPU max MHz: 2700.0000 CPU min MHz: 1200.0000 BogoMIPS: 5400.00 diff --git a/tests/78314D65/golden.txt b/tests/78314D65/golden.txt new file mode 100644 index 0000000000..e69de29bb2 diff --git a/tests/78A1FE7C/golden-metadata.txt b/tests/78A1FE7C/golden-metadata.txt deleted file mode 100644 index 550ebca81b..0000000000 --- a/tests/78A1FE7C/golden-metadata.txt +++ /dev/null @@ -1,162 +0,0 @@ -This file was created on 2026-07-10 21:59:26.563363. - -mfc.sh: - - Invocation: test --generate --only 6ABA55B2 01A16919 78A1FE7C --no-gpu --no-build -j 2 - Lock: mpi=Yes & gpu=No & debug=No & reldebug=No & gcov=No & unified=No & single=No & mixed=No & fastmath=No - Git: 416e414836f4dac0f790ed22f24bedba4ddbc2bb on up/mega (dirty) - -simulation: - - CMake Configuration: - - CMake v3.30.5 on login10 - - C : CrayClang v19.0.0 (/opt/cray/pe/craype/2.7.34/bin/cc) - Fortran : Cray v19.0.0 (/opt/cray/pe/craype/2.7.34/bin/ftn) - - PRE_PROCESS : OFF - SIMULATION : ON - POST_PROCESS : OFF - SYSCHECK : OFF - DOCUMENTATION : OFF - ALL : OFF - - MPI : ON - OpenACC : OFF - OpenMP : OFF - - Fypp : /lustre/orion/cfd154/scratch/sbryngelson/MFC-amr/build/venv/bin/fypp - Doxygen : - - Build Type : Release - - Configuration Environment: - - CC : - CXX : - FC : - OMPI_CC : - OMPI_CXX : - OMPI_FC : - -syscheck: - - CMake Configuration: - - CMake v3.30.5 on login10 - - C : CrayClang v19.0.0 (/opt/cray/pe/craype/2.7.34/bin/cc) - Fortran : Cray v19.0.0 (/opt/cray/pe/craype/2.7.34/bin/ftn) - - PRE_PROCESS : OFF - SIMULATION : OFF - POST_PROCESS : OFF - SYSCHECK : ON - DOCUMENTATION : OFF - ALL : OFF - - MPI : ON - OpenACC : OFF - OpenMP : OFF - - Fypp : /lustre/orion/cfd154/scratch/sbryngelson/MFC-amr/build/venv/bin/fypp - Doxygen : - - Build Type : Release - - Configuration Environment: - - CC : - CXX : - FC : - OMPI_CC : - OMPI_CXX : - OMPI_FC : - -pre_process: - - CMake Configuration: - - CMake v3.30.5 on login10 - - C : CrayClang v19.0.0 (/opt/cray/pe/craype/2.7.34/bin/cc) - Fortran : Cray v19.0.0 (/opt/cray/pe/craype/2.7.34/bin/ftn) - - PRE_PROCESS : ON - SIMULATION : OFF - POST_PROCESS : OFF - SYSCHECK : OFF - DOCUMENTATION : OFF - ALL : OFF - - MPI : ON - OpenACC : OFF - OpenMP : OFF - - Fypp : /lustre/orion/cfd154/scratch/sbryngelson/MFC-amr/build/venv/bin/fypp - Doxygen : - - Build Type : Release - - Configuration Environment: - - CC : - CXX : - FC : - OMPI_CC : - OMPI_CXX : - OMPI_FC : - -CPU: - - CPU Info: - From lscpu - Architecture: x86_64 - CPU op-mode(s): 32-bit, 64-bit - Address sizes: 48 bits physical, 48 bits virtual - Byte Order: Little Endian - CPU(s): 128 - On-line CPU(s) list: 0-127 - Vendor ID: AuthenticAMD - Model name: AMD EPYC 7A53 64-Core Processor - CPU family: 25 - Model: 48 - Thread(s) per core: 2 - Core(s) per socket: 64 - Socket(s): 1 - Stepping: 1 - Frequency boost: enabled - CPU(s) scaling MHz: 56% - CPU max MHz: 3541.0149 - CPU min MHz: 1500.0000 - BogoMIPS: 3992.45 - Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca fsrm ibpb_exit_to_user - Virtualization: AMD-V - L1d cache: 2 MiB (64 instances) - L1i cache: 2 MiB (64 instances) - L2 cache: 32 MiB (64 instances) - L3 cache: 256 MiB (8 instances) - NUMA node(s): 4 - NUMA node0 CPU(s): 0-15,64-79 - NUMA node1 CPU(s): 16-31,80-95 - NUMA node2 CPU(s): 32-47,96-111 - NUMA node3 CPU(s): 48-63,112-127 - Vulnerability Gather data sampling: Not affected - Vulnerability Indirect target selection: Not affected - Vulnerability Itlb multihit: Not affected - Vulnerability L1tf: Not affected - Vulnerability Mds: Not affected - Vulnerability Meltdown: Not affected - Vulnerability Mmio stale data: Not affected - Vulnerability Reg file data sampling: Not affected - Vulnerability Retbleed: Not affected - Vulnerability Spec rstack overflow: Vulnerable - Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl - Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization - Vulnerability Spectre v2: Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected - Vulnerability Srbds: Not affected - Vulnerability Tsa: Vulnerable: No microcode - Vulnerability Tsx async abort: Not affected - Vulnerability Vmscape: Mitigation; IBPB before exit to userspace - diff --git a/tests/78A1FE7C/golden.txt b/tests/78A1FE7C/golden.txt deleted file mode 100644 index 7c518651d2..0000000000 --- a/tests/78A1FE7C/golden.txt +++ /dev/null @@ -1,32 +0,0 @@ -D/cons.1.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 -D/cons.1.00.000020.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.99999999999997 1.00000000000019 0.99999999999894 1.00000000000567 0.99999999997346 1.00000000008067 0.99999999909857 0.99999994608442 0.99999838024113 0.999963441049 0.999402583161 0.99323945942215 0.95476052122168 0.87566101681301 0.78964773123248 0.71189941583957 0.63864010946401 0.64556499307301 0.70974674198244 0.63329696469289 0.59243124110187 0.46257149537151 0.34841324290098 0.30809147940479 0.15973273229222 0.11710340774642 0.11484296578326 0.11485976157285 0.11372050525091 0.11377605576855 0.11619605444442 0.11874687846684 0.1207643235213 0.12266561274431 0.12428543966977 0.12497790310803 0.12499960332571 0.12499999394419 0.12500000013581 0.12500000005357 0.12499999993665 0.12500000002698 0.12499999999436 0.1249999999997 0.12499999999985 0.12500000000181 0.12499999999794 0.12500000000186 0.12499999999889 0.12500000000052 0.12499999999985 0.12500000000001 0.12500000000002 0.12499999999999 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 -D/cons.2.00.000000.dat 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -D/cons.2.00.000020.dat 0.0 0.0 -0.0 0.0 -0.0 -0.0 -0.0 0.0 0.0 0.0 0.0 -0.0 -0.0 -0.0 -0.0 0.0 0.0 -0.0 0.0 -0.0 0.0 0.0 -0.0 0.0 -0.0 -0.0 0.0 -0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -0.0 0.0 -0.0 -0.0 0.0 -0.0 0.0 0.0 0.0 0.0 -0.0 0.0 -0.0 0.0 0.0 0.0 -0.0 -0.0 -0.0 -0.0 -0.0 0.0 0.0 -0.0 0.0 -0.0 -0.0 -0.0 0.0 -1e-14 5e-14 -3.2e-13 1.69e-12 -9.25e-12 4.251e-11 -1.3211e-10 1.44292e-09 8.527453e-08 2.56601812e-06 5.82036942e-05 0.0009572980328 0.01087626124777 0.0710548617324 0.18459307805032 0.29128240355533 0.37075380949852 0.42755376449446 0.46137037259616 0.40661086891355 0.44082185019954 0.42808877528931 0.31556652741097 0.23918105419492 0.20701315583168 0.01740770219482 -0.03270673444573 -0.03428437921979 -0.03365746391926 -0.03876490964424 -0.03767699298238 -0.03047428552334 -0.02179267601206 -0.01491483146223 -0.00842388737377 -0.00241992335533 -7.51910396e-05 -1.34446421e-06 -2.063934e-08 4.5974e-10 1.9475e-10 -2.3322e-10 1.0583e-10 -2.695e-11 5.95e-12 -3.88e-12 7.49e-12 -7.46e-12 6.55e-12 -3.77e-12 1.82e-12 -5.1e-13 3e-14 7e-14 -4e-14 1e-14 0.0 -0.0 0.0 0.0 -0.0 -0.0 -0.0 0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 0.0 -0.0 0.0 0.0 0.0 -0.0 -0.0 0.0 0.0 -0.0 -0.0 0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 0.0 0.0 0.0 -0.0 -0.0 0.0 0.0 0.0 -0.0 -0.0 0.0 0.0 0.0 -0.0 -0.0 0.0 0.0 0.0 -0.0 -0.0 -0.0 0.0 0.0 0.0 -D/cons.3.00.000000.dat 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -D/cons.3.00.000020.dat 0.0 0.0 0.0 0.0 -0.0 0.0 -0.0 -0.0 0.0 0.0 -0.0 -0.0 0.0 -0.0 0.0 0.0 0.0 -0.0 -0.0 0.0 0.0 0.0 0.0 -0.0 0.0 0.0 -0.0 -0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -0.0 -0.0 -0.0 0.0 -0.0 0.0 -0.0 -0.0 0.0 -0.0 0.0 0.0 0.0 -0.0 -0.0 0.0 -0.0 0.0 0.0 -0.0 -0.0 0.0 0.0 0.0 -0.0 0.0 -0.0 0.0 -1e-14 1e-13 -4.9e-13 3.22e-12 -1.259e-11 4.363e-11 -3.2409e-10 -1.937756e-08 -6.3533532e-07 -1.592859793e-05 -0.00028954514117 -0.00360424853675 -0.02551902944823 -0.06968793287859 -0.1158193188623 -0.15462461554705 -0.17958537300683 -0.35944692326244 -1.00825228755043 -1.0005340129398 -0.96699089433848 -0.76524709317683 -0.58726980038044 -0.49127325668011 -0.11456822289113 -0.02648950975604 -0.02469828110995 -0.02442425878003 -0.02847532705236 -0.02776838630454 -0.02205189194511 -0.01554779312437 -0.01046847111188 -0.00580279691122 -0.00150839544076 -3.513492385e-05 -6.0642911e-07 -9.23091e-09 3.2225e-10 1.3675e-10 -1.5526e-10 7.179e-11 -2.196e-11 8.59e-12 -5.3e-12 7.01e-12 -5.48e-12 4.75e-12 -2.25e-12 1.12e-12 -2.9e-13 1e-14 6e-14 -2e-14 0.0 0.0 -0.0 0.0 0.0 -0.0 0.0 -0.0 -0.0 0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 0.0 0.0 -0.0 -0.0 0.0 0.0 -0.0 -0.0 0.0 -0.0 0.0 -0.0 -0.0 -0.0 -0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -D/cons.4.00.000000.dat 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -D/cons.4.00.000020.dat 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -D/cons.5.00.000000.dat 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 -D/cons.5.00.000020.dat 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125 3.28125000000002 3.28124999999985 3.28125000000091 3.28124999999489 3.2812500000266 3.28124999987238 3.28125000037344 3.28124999569102 3.2812497376378 3.28124216260119 3.28107408293095 3.27839088201473 3.24915465630473 3.07246575811094 2.73449773420611 2.40517311448222 2.14285388256343 1.92445947094818 2.01160530426236 2.77339741125324 2.6898209325808 2.63410268456734 2.38007655907256 2.23202150227045 2.0913578410365 1.11398557268874 0.9058181355002 0.89471198520966 0.89612082590725 0.87931837087377 0.88097858078485 0.91001458245355 0.94315214886231 0.97075839336109 0.99681751704262 1.02072949775351 1.03090694152501 1.03124382732673 1.03124990559584 1.03125000215859 1.0312500007703 1.0312499990677 1.03125000041531 1.03124999989214 1.03125000001311 1.03124999998924 1.03125000002289 1.03124999996698 1.03125000002711 1.03124999998342 1.03125000000742 1.03124999999768 1.03125000000008 1.03125000000028 1.03124999999984 1.03125000000003 1.03125 1.03124999999999 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 1.03125 -D/cons.6.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -D/cons.6.00.000020.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -D/cons.7.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -D/cons.7.00.000020.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.00000000000001 0.99999999999997 1.00000000000025 0.99999999999893 1.00000000000742 0.99999999997213 1.00000000011045 0.99999999916988 0.99999995431437 0.99999852623997 0.99996363351568 0.99934929445912 0.99200048331099 0.94318344922261 0.84090765509303 0.72702950088752 0.62229819758412 0.53268274736257 0.30832817952875 -0.3712545824115 -0.55604475370369 -0.54839300472186 -0.52320655211353 -0.50229943219486 -0.56454266114221 -0.81805324601579 -0.87959954012521 -0.87697358334627 -0.87777420584549 -0.85850781079009 -0.86169771540209 -0.89107940225534 -0.92358294168093 -0.94848832672092 -0.971933149448 -0.99248479934016 -0.99982891002826 -0.99999704663568 -0.99999995524124 -1.00000000141176 -1.00000000066016 -0.99999999924868 -1.00000000032906 -0.99999999991509 -1.00000000003014 -0.99999999999282 -1.0000000000344 -0.99999999997755 -1.00000000002192 -0.99999999998778 -1.00000000000614 -0.99999999999878 -1.00000000000018 -1.00000000000022 -0.9999999999999 -1.00000000000003 -1.00000000000001 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -D/cons.8.00.000000.dat 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -D/cons.8.00.000020.dat 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -D/prim.1.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 -D/prim.1.00.000020.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.99999999999997 1.00000000000019 0.99999999999894 1.00000000000567 0.99999999997346 1.00000000008067 0.99999999909857 0.99999994608442 0.99999838024113 0.999963441049 0.999402583161 0.99323945942215 0.95476052122168 0.87566101681301 0.78964773123248 0.71189941583957 0.63864010946401 0.64556499307301 0.70974674198244 0.63329696469289 0.59243124110187 0.46257149537151 0.34841324290098 0.30809147940479 0.15973273229222 0.11710340774642 0.11484296578326 0.11485976157285 0.11372050525091 0.11377605576855 0.11619605444442 0.11874687846684 0.1207643235213 0.12266561274431 0.12428543966977 0.12497790310803 0.12499960332571 0.12499999394419 0.12500000013581 0.12500000005357 0.12499999993665 0.12500000002698 0.12499999999436 0.1249999999997 0.12499999999985 0.12500000000181 0.12499999999794 0.12500000000186 0.12499999999889 0.12500000000052 0.12499999999985 0.12500000000001 0.12500000000002 0.12499999999999 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 -D/prim.2.00.000000.dat 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -D/prim.2.00.000020.dat 0.0 0.0 -0.0 0.0 -0.0 -0.0 -0.0 0.0 0.0 0.0 0.0 -0.0 -0.0 -0.0 -0.0 0.0 0.0 -0.0 0.0 -0.0 0.0 0.0 -0.0 0.0 -0.0 -0.0 0.0 -0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -0.0 0.0 -0.0 -0.0 0.0 -0.0 0.0 0.0 0.0 0.0 -0.0 0.0 -0.0 0.0 0.0 0.0 -0.0 -0.0 -0.0 -0.0 -0.0 0.0 0.0 -0.0 0.0 -0.0 -0.0 -0.0 0.0 -1e-14 5e-14 -3.2e-13 1.69e-12 -9.25e-12 4.251e-11 -1.3211e-10 1.44292e-09 8.527454e-08 2.56602228e-06 5.820582214e-05 0.00095787028064 0.01095029113533 0.07442165878569 0.21080426615558 0.36887638884329 0.52079521523596 0.66947527748185 0.71467687614217 0.57289571739042 0.69607447181326 0.72259655735423 0.68220054752297 0.68648669092894 0.67192106783223 0.1089801817387 -0.2792978878681 -0.29853268753516 -0.29303094015143 -0.3408788024527 -0.33115045804567 -0.26226609560067 -0.18352209585155 -0.12350362281953 -0.06867358492174 -0.01947069070813 -0.00060163467086 -1.075574784e-05 -1.6511475e-07 3.67789e-09 1.55799e-09 -1.86575e-09 8.4666e-10 -2.1557e-10 4.763e-11 -3.102e-11 5.995e-11 -5.969e-11 5.237e-11 -3.017e-11 1.454e-11 -4.11e-12 2.7e-13 5.9e-13 -3e-13 7e-14 1e-14 -1e-14 0.0 0.0 -0.0 -0.0 -0.0 0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 0.0 -0.0 0.0 0.0 0.0 -0.0 -0.0 0.0 0.0 -0.0 -0.0 0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 0.0 0.0 0.0 -0.0 -0.0 0.0 0.0 0.0 -0.0 -0.0 0.0 0.0 0.0 -0.0 -0.0 0.0 0.0 0.0 -0.0 -0.0 -0.0 0.0 0.0 0.0 -D/prim.3.00.000000.dat 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -D/prim.3.00.000020.dat 0.0 0.0 0.0 0.0 -0.0 0.0 -0.0 -0.0 0.0 0.0 -0.0 -0.0 0.0 -0.0 0.0 0.0 0.0 -0.0 -0.0 0.0 0.0 0.0 0.0 -0.0 0.0 0.0 -0.0 -0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -0.0 -0.0 -0.0 0.0 -0.0 0.0 -0.0 -0.0 0.0 -0.0 0.0 0.0 0.0 -0.0 -0.0 0.0 -0.0 0.0 0.0 -0.0 -0.0 0.0 0.0 0.0 -0.0 0.0 -0.0 0.0 -1e-14 1e-13 -4.9e-13 3.22e-12 -1.259e-11 4.363e-11 -3.2409e-10 -1.937756e-08 -6.3533635e-07 -1.592918028e-05 -0.00028971822372 -0.00362878105834 -0.02672819925103 -0.07958323088565 -0.14667213528434 -0.21720008769033 -0.28119964647624 -0.55679432298738 -1.42058036749026 -1.57988127011629 -1.63224156197428 -1.65433257525352 -1.685555335069 -1.5945694364194 -0.71724950326111 -0.22620613922189 -0.21506133128399 -0.21264417099233 -0.25039747220203 -0.24406177659234 -0.1897817619587 -0.13093222596761 -0.08668513023249 -0.04730581604244 -0.01213654185693 -0.00028112908742 -4.8514483e-06 -7.384731e-08 2.57798e-09 1.09396e-09 -1.24207e-09 5.7431e-10 -1.7565e-10 6.875e-11 -4.241e-11 5.605e-11 -4.383e-11 3.8e-11 -1.801e-11 8.99e-12 -2.29e-12 9e-14 4.6e-13 -1.8e-13 3e-14 1e-14 -1e-14 0.0 0.0 -0.0 0.0 -0.0 -0.0 0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 -0.0 0.0 0.0 -0.0 -0.0 0.0 0.0 -0.0 -0.0 0.0 -0.0 0.0 -0.0 -0.0 -0.0 -0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -D/prim.4.00.000000.dat 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -D/prim.4.00.000020.dat 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -D/prim.5.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 -D/prim.5.00.000020.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.00000000000001 0.99999999999995 1.00000000000026 0.99999999999838 1.00000000000767 0.9999999999601 1.00000000010519 0.99999999860846 0.99999991332937 0.99999745454266 0.9999441787733 0.99911635016793 0.99032243529286 0.93737327979115 0.83098215696203 0.71896793316838 0.62185628566743 0.53318640280669 0.56715511986276 0.63624316538574 0.52407726266819 0.50345445569016 0.48853103396608 0.49903358587486 0.47580828901388 0.1824377873107 0.09206278168608 0.08845893028945 0.08883954816221 0.08775135893384 0.08753605073241 0.09026584899902 0.0929527405644 0.09532743685907 0.09712559650024 0.09877349886043 0.09993119572135 0.09999871227119 0.09999998014184 0.10000000029873 0.10000000004406 0.09999999992761 0.1000000000345 0.09999999999082 0.09999999999319 0.09999999999857 0.0999999999954 0.09999999999577 0.10000000000208 0.09999999999825 0.10000000000051 0.09999999999956 0.09999999999996 0.10000000000002 0.09999999999998 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 -D/prim.6.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -D/prim.6.00.000020.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -D/prim.7.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -8.0 -D/prim.7.00.000020.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.00000000000001 0.99999999999997 1.00000000000025 0.99999999999893 1.00000000000742 0.99999999997213 1.00000000011045 0.99999999916988 0.99999995431437 0.99999852623997 0.99996363351568 0.99934929445912 0.99200048331099 0.94318344922261 0.84090765509303 0.72702950088752 0.62229819758412 0.53268274736257 0.30832817952875 -0.3712545824115 -0.55604475370369 -0.54839300472186 -0.52320655211353 -0.50229943219486 -0.56454266114221 -0.81805324601579 -0.87959954012521 -0.87697358334627 -0.87777420584549 -0.85850781079009 -0.86169771540209 -0.89107940225534 -0.92358294168093 -0.94848832672092 -0.971933149448 -0.99248479934016 -0.99982891002826 -0.99999704663568 -0.99999995524124 -1.00000000141176 -1.00000000066016 -0.99999999924868 -1.00000000032906 -0.99999999991509 -1.00000000003014 -0.99999999999282 -1.0000000000344 -0.99999999997755 -1.00000000002192 -0.99999999998778 -1.00000000000614 -0.99999999999878 -1.00000000000018 -1.00000000000022 -0.9999999999999 -1.00000000000003 -1.00000000000001 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -D/prim.8.00.000000.dat 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -D/prim.8.00.000020.dat 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \ No newline at end of file diff --git a/tests/BA4340EA/golden-metadata.txt b/tests/BA4340EA/golden-metadata.txt deleted file mode 100644 index b4814cd764..0000000000 --- a/tests/BA4340EA/golden-metadata.txt +++ /dev/null @@ -1,193 +0,0 @@ -This file was created on 2026-07-08 14:15:45.495752. - -mfc.sh: - - Invocation: test --generate --only 053C5DDA DDC4BA8A BA4340EA 60739A3E -j 8 -- -b mpirun - Lock: mpi=Yes & gpu=No & debug=No & reldebug=Yes & gcov=No & unified=No & single=No & mixed=No & fastmath=Yes - Git: ace2285a7e72fdce9dc3fbc3a5629e1b9d1a89b7 on amr-hybrid-test-robustness (dirty) - -syscheck: - - CMake Configuration: - - CMake v3.26.5 on atl1-1-02-005-27-0.pace.gatech.edu - - C : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc) - Fortran : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran) - - PRE_PROCESS : OFF - SIMULATION : OFF - POST_PROCESS : OFF - SYSCHECK : ON - DOCUMENTATION : OFF - ALL : OFF - - MPI : ON - OpenACC : OFF - OpenMP : OFF - - Fypp : /storage/project/r-sbryngelson3-0/sbryngelson3/MFC-lomach/build/venv/bin/fypp - Doxygen : - - Build Type : RelDebug - - Configuration Environment: - - CC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc - CXX : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/g++ - FC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran - OMPI_CC : - OMPI_CXX : - OMPI_FC : - -simulation: - - CMake Configuration: - - CMake v3.26.5 on atl1-1-02-005-27-0.pace.gatech.edu - - C : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc) - Fortran : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran) - - PRE_PROCESS : OFF - SIMULATION : ON - POST_PROCESS : OFF - SYSCHECK : OFF - DOCUMENTATION : OFF - ALL : OFF - - MPI : ON - OpenACC : OFF - OpenMP : OFF - - Fypp : /storage/project/r-sbryngelson3-0/sbryngelson3/MFC-lomach/build/venv/bin/fypp - Doxygen : - - Build Type : RelDebug - - Configuration Environment: - - CC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc - CXX : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/g++ - FC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran - OMPI_CC : - OMPI_CXX : - OMPI_FC : - -post_process: - - CMake Configuration: - - CMake v3.26.5 on atl1-1-02-005-27-0.pace.gatech.edu - - C : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc) - Fortran : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran) - - PRE_PROCESS : OFF - SIMULATION : OFF - POST_PROCESS : ON - SYSCHECK : OFF - DOCUMENTATION : OFF - ALL : OFF - - MPI : ON - OpenACC : OFF - OpenMP : OFF - - Fypp : /storage/project/r-sbryngelson3-0/sbryngelson3/MFC-lomach/build/venv/bin/fypp - Doxygen : - - Build Type : RelDebug - - Configuration Environment: - - CC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc - CXX : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/g++ - FC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran - OMPI_CC : - OMPI_CXX : - OMPI_FC : - -pre_process: - - CMake Configuration: - - CMake v3.26.5 on atl1-1-02-005-27-0.pace.gatech.edu - - C : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc) - Fortran : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran) - - PRE_PROCESS : ON - SIMULATION : OFF - POST_PROCESS : OFF - SYSCHECK : OFF - DOCUMENTATION : OFF - ALL : OFF - - MPI : ON - OpenACC : OFF - OpenMP : OFF - - Fypp : /storage/project/r-sbryngelson3-0/sbryngelson3/MFC-lomach/build/venv/bin/fypp - Doxygen : - - Build Type : RelDebug - - Configuration Environment: - - CC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc - CXX : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/g++ - FC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran - OMPI_CC : - OMPI_CXX : - OMPI_FC : - -CPU: - - CPU Info: - From lscpu - Architecture: x86_64 - CPU op-mode(s): 32-bit, 64-bit - Address sizes: 46 bits physical, 48 bits virtual - Byte Order: Little Endian - CPU(s): 24 - On-line CPU(s) list: 0-23 - Vendor ID: GenuineIntel - Model name: Intel(R) Xeon(R) Gold 6226 CPU @ 2.70GHz - CPU family: 6 - Model: 85 - Thread(s) per core: 1 - Core(s) per socket: 12 - Socket(s): 2 - Stepping: 7 - CPU(s) scaling MHz: 100% - CPU max MHz: 2700.0000 - CPU min MHz: 1200.0000 - BogoMIPS: 5400.00 - Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req vnmi pku ospke avx512_vnni md_clear flush_l1d arch_capabilities - Virtualization: VT-x - L1d cache: 768 KiB (24 instances) - L1i cache: 768 KiB (24 instances) - L2 cache: 24 MiB (24 instances) - L3 cache: 38.5 MiB (2 instances) - NUMA node(s): 2 - NUMA node0 CPU(s): 0-11 - NUMA node1 CPU(s): 12-23 - Vulnerability Gather data sampling: Vulnerable - Vulnerability Indirect target selection: Vulnerable - Vulnerability Itlb multihit: KVM: Vulnerable - Vulnerability L1tf: Not affected - Vulnerability Mds: Not affected - Vulnerability Meltdown: Not affected - Vulnerability Mmio stale data: Vulnerable - Vulnerability Reg file data sampling: Not affected - Vulnerability Retbleed: Vulnerable - Vulnerability Spec rstack overflow: Not affected - Vulnerability Spec store bypass: Vulnerable - Vulnerability Spectre v1: Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers - Vulnerability Spectre v2: Vulnerable; IBPB: disabled; STIBP: disabled; PBRSB-eIBRS: Vulnerable; BHI: Vulnerable - Vulnerability Srbds: Not affected - Vulnerability Tsa: Not affected - Vulnerability Tsx async abort: Mitigation; TSX disabled - Vulnerability Vmscape: Vulnerable - diff --git a/tests/BA4340EA/golden.txt b/tests/BA4340EA/golden.txt deleted file mode 100644 index ab04568e04..0000000000 --- a/tests/BA4340EA/golden.txt +++ /dev/null @@ -1,16 +0,0 @@ -D/cons.1.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 -D/cons.1.00.000006.dat 0.99999999612368 0.99999985088689 0.99999623507138 0.99993092477978 0.99825678931223 0.9615730003021 0.53726953978334 0.50286743356096 0.50010158763536 0.50000447516482 0.50000016322847 0.50000000429016 0.50000000009912 0.49999999997699 0.50000000000336 0.50000000000068 0.49999999999993 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.50000000000007 0.49999999999952 0.49999999999697 0.50000000001892 0.49999999989456 0.49999999126629 0.49999942266678 0.49996995381996 0.49884345037773 0.46690023384529 0.15684108925066 0.12738536102708 0.12505946967055 0.12500101444172 0.12500001361476 0.12500000011784 0.12499999998896 0.12500000000198 0.12500000000041 0.12499999999995 0.125 0.125 0.125 -D/cons.2.00.000000.dat 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -D/cons.2.00.000006.dat 4.56577e-09 1.7644263e-07 4.45462326e-06 8.166794775e-05 0.00198034905009 0.04660468479445 0.04372766162016 0.00347579656397 0.00011971049077 5.29532092e-06 1.9312417e-07 5.09259e-09 1.2475e-10 -2.973e-11 3.99e-12 8e-13 -9e-14 -0.0 0.0 -0.0 -0.0 -0.0 0.0 -0.0 0.0 0.0 -0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -0.0 -0.0 0.0 -0.0 -0.0 0.0 0.0 -8e-14 5.6e-13 3.63e-12 -2.622e-11 1.5113e-10 1.032549e-08 6.8310027e-07 3.554859267e-05 0.00136476605923 0.03577661660379 0.03668359552735 0.00287444790368 6.324352877e-05 1.07366847e-06 1.437255e-08 2.0695e-10 -2.094e-11 2.33e-12 4.2e-13 -5e-14 0.0 0.0 -0.0 -D/cons.3.00.000000.dat 2.5 2.5 2.5 2.5 2.5 2.5 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 -D/cons.3.00.000006.dat 2.49999998643289 2.49999947810431 2.49998682288242 2.49975857629497 2.49392705148108 2.37761646980249 1.36814889365388 1.26019139737138 1.2503550747231 1.25001566342491 1.2500005713001 1.25000001501556 1.25000000034691 1.24999999991947 1.25000000001176 1.25000000000236 1.24999999999975 1.24999999999999 1.25000000000001 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.24999999999999 1.25000000000023 1.24999999999833 1.24999999998939 1.25000000006623 1.24999999963097 1.249999969432 1.24999797934343 1.24989485925863 1.24597556839486 1.15270465024259 0.34422215943072 0.25703510060941 0.25016683463152 0.25000284054476 0.25000003812134 0.25000000032996 0.24999999996909 0.25000000000553 0.25000000000115 0.24999999999987 0.25 0.25 0.25 -D/cons.4.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -D/cons.4.00.000006.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -D/prim.1.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 -D/prim.1.00.000006.dat 0.99999999612368 0.99999985088689 0.99999623507138 0.99993092477978 0.99825678931223 0.9615730003021 0.53726953978334 0.50286743356096 0.50010158763536 0.50000447516482 0.50000016322847 0.50000000429016 0.50000000009912 0.49999999997699 0.50000000000336 0.50000000000068 0.49999999999993 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.50000000000007 0.49999999999952 0.49999999999697 0.50000000001892 0.49999999989456 0.49999999126629 0.49999942266678 0.49996995381996 0.49884345037773 0.46690023384529 0.15684108925066 0.12738536102708 0.12505946967055 0.12500101444172 0.12500001361476 0.12500000011784 0.12499999998896 0.12500000000198 0.12500000000041 0.12499999999995 0.125 0.125 0.125 -D/prim.2.00.000000.dat 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -D/prim.2.00.000006.dat 4.56577e-09 1.7644265e-07 4.45464003e-06 8.167358937e-05 0.00198380724408 0.04846713123164 0.08138868553351 0.00691195399026 0.00023937234701 1.059054705e-05 3.8624821e-07 1.018518e-08 2.4949e-10 -5.946e-11 7.98e-12 1.59e-12 -1.7e-13 -1e-14 0.0 -0.0 -0.0 -0.0 0.0 -0.0 0.0 0.0 -0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -0.0 -0.0 0.0 -0.0 -0.0 0.0 0.0 -1.5e-13 1.12e-12 7.27e-12 -5.243e-11 3.0225e-10 2.065098e-08 1.36620211e-06 7.1101458e-05 0.00273586043516 0.07662582712616 0.23389021144023 0.02256497827149 0.00050570763599 8.58927807e-06 1.1498037e-07 1.65559e-09 -1.6754e-10 1.86e-11 3.32e-12 -3.8e-13 1e-14 1e-14 -0.0 -D/prim.3.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 -D/prim.3.00.000006.dat 0.99999999457316 0.99999979124172 0.999994729149 0.99990342918396 0.99757003486628 0.95059482884621 0.54654777008141 0.50407175403937 0.50014202415817 0.50000626535875 0.50000022852003 0.50000000600622 0.50000000013876 0.49999999996779 0.5000000000047 0.50000000000095 0.4999999999999 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.50000000000009 0.49999999999933 0.49999999999575 0.50000000002649 0.49999999985239 0.4999999877728 0.49999919173719 0.49995794319794 0.49838948059605 0.46053357752923 0.13597287698943 0.10280106787287 0.10006672745606 0.10000113621606 0.10000001524854 0.10000000013198 0.09999999998764 0.10000000000221 0.10000000000046 0.09999999999995 0.1 0.1 0.1 -D/prim.4.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -D/prim.4.00.000006.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 \ No newline at end of file diff --git a/tests/C6EA340F/golden-metadata.txt b/tests/C6EA340F/golden-metadata.txt deleted file mode 100644 index 80581a302f..0000000000 --- a/tests/C6EA340F/golden-metadata.txt +++ /dev/null @@ -1,159 +0,0 @@ -This file was created on 2026-07-05 12:53:53.157003. - -mfc.sh: - - Invocation: test --generate --only 71E57E55 C6EA340F --no-gpu --no-reldebug --no-debug -j 2 - Lock: mpi=No & gpu=No & debug=No & reldebug=No & gcov=No & unified=No & single=No & mixed=No & fastmath=No - Git: c95567303b89e8f3439f8433e7f4b3b7be716ee3 on up/mega (dirty) - -syscheck: - - CMake Configuration: - - CMake v3.26.5 on atl1-1-01-003-35-0.pace.gatech.edu - - C : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc) - Fortran : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran) - - PRE_PROCESS : OFF - SIMULATION : OFF - POST_PROCESS : OFF - SYSCHECK : ON - DOCUMENTATION : OFF - ALL : OFF - - MPI : OFF - OpenACC : OFF - OpenMP : OFF - - Fypp : /storage/project/r-sbryngelson3-0/sbryngelson3/MFC-lomach/build/venv/bin/fypp - Doxygen : - - Build Type : Release - - Configuration Environment: - - CC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc - CXX : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/g++ - FC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran - OMPI_CC : - OMPI_CXX : - OMPI_FC : - -simulation: - - CMake Configuration: - - CMake v3.26.5 on atl1-1-01-003-35-0.pace.gatech.edu - - C : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc) - Fortran : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran) - - PRE_PROCESS : OFF - SIMULATION : ON - POST_PROCESS : OFF - SYSCHECK : OFF - DOCUMENTATION : OFF - ALL : OFF - - MPI : OFF - OpenACC : OFF - OpenMP : OFF - - Fypp : /storage/project/r-sbryngelson3-0/sbryngelson3/MFC-lomach/build/venv/bin/fypp - Doxygen : - - Build Type : Release - - Configuration Environment: - - CC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc - CXX : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/g++ - FC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran - OMPI_CC : - OMPI_CXX : - OMPI_FC : - -pre_process: - - CMake Configuration: - - CMake v3.26.5 on atl1-1-01-003-35-0.pace.gatech.edu - - C : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc) - Fortran : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran) - - PRE_PROCESS : ON - SIMULATION : OFF - POST_PROCESS : OFF - SYSCHECK : OFF - DOCUMENTATION : OFF - ALL : OFF - - MPI : OFF - OpenACC : OFF - OpenMP : OFF - - Fypp : /storage/project/r-sbryngelson3-0/sbryngelson3/MFC-lomach/build/venv/bin/fypp - Doxygen : - - Build Type : Release - - Configuration Environment: - - CC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc - CXX : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/g++ - FC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran - OMPI_CC : - OMPI_CXX : - OMPI_FC : - -CPU: - - CPU Info: - From lscpu - Architecture: x86_64 - CPU op-mode(s): 32-bit, 64-bit - Address sizes: 46 bits physical, 48 bits virtual - Byte Order: Little Endian - CPU(s): 24 - On-line CPU(s) list: 0-23 - Vendor ID: GenuineIntel - Model name: Intel(R) Xeon(R) Gold 6226 CPU @ 2.70GHz - CPU family: 6 - Model: 85 - Thread(s) per core: 1 - Core(s) per socket: 12 - Socket(s): 2 - Stepping: 7 - CPU(s) scaling MHz: 81% - CPU max MHz: 2700.0000 - CPU min MHz: 1200.0000 - BogoMIPS: 5400.00 - Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req vnmi pku ospke avx512_vnni md_clear flush_l1d arch_capabilities - Virtualization: VT-x - L1d cache: 768 KiB (24 instances) - L1i cache: 768 KiB (24 instances) - L2 cache: 24 MiB (24 instances) - L3 cache: 38.5 MiB (2 instances) - NUMA node(s): 2 - NUMA node0 CPU(s): 0-11 - NUMA node1 CPU(s): 12-23 - Vulnerability Gather data sampling: Vulnerable - Vulnerability Indirect target selection: Vulnerable - Vulnerability Itlb multihit: KVM: Vulnerable - Vulnerability L1tf: Not affected - Vulnerability Mds: Not affected - Vulnerability Meltdown: Not affected - Vulnerability Mmio stale data: Vulnerable - Vulnerability Reg file data sampling: Not affected - Vulnerability Retbleed: Vulnerable - Vulnerability Spec rstack overflow: Not affected - Vulnerability Spec store bypass: Vulnerable - Vulnerability Spectre v1: Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers - Vulnerability Spectre v2: Vulnerable; IBPB: disabled; STIBP: disabled; PBRSB-eIBRS: Vulnerable; BHI: Vulnerable - Vulnerability Srbds: Not affected - Vulnerability Tsa: Not affected - Vulnerability Tsx async abort: Mitigation; TSX disabled - Vulnerability Vmscape: Vulnerable - diff --git a/tests/C6EA340F/golden.txt b/tests/C6EA340F/golden.txt deleted file mode 100644 index 228bb62995..0000000000 --- a/tests/C6EA340F/golden.txt +++ /dev/null @@ -1,16 +0,0 @@ -D/cons.1.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 -D/cons.1.00.000006.dat 0.99999999979591 0.99999998472683 0.99999899243059 0.99994780428952 0.99800633490866 0.96114045213925 0.53777269575953 0.5030507498549 0.5000813962002 0.50000156595549 0.50000002364808 0.50000000028342 0.49999999998468 0.50000000000118 0.50000000000082 0.49999999999993 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.50000000000007 0.49999999999952 0.49999999999697 0.50000000001892 0.49999999989456 0.49999999126629 0.49999942266678 0.49996995381996 0.49884345037773 0.46690023384529 0.15684108925066 0.12738536102708 0.12505946967055 0.12500101444172 0.12500001361476 0.12500000011784 0.12499999998896 0.12500000000198 0.12500000000041 0.12499999999995 0.125 0.125 0.125 -D/cons.2.00.000000.dat 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -D/cons.2.00.000006.dat 2.2435e-10 1.808107e-08 1.1921741e-06 6.175499736e-05 0.00235343211297 0.04637431088153 0.04334870246431 0.00376226359271 9.644425937e-05 1.85291298e-06 2.79646e-08 3.7774e-10 -2.271e-11 1.56e-12 9.6e-13 -8e-14 -0.0 0.0 -0.0 -0.0 -0.0 -0.0 -0.0 0.0 0.0 -0.0 0.0 -0.0 0.0 0.0 -0.0 0.0 0.0 -0.0 -0.0 0.0 0.0 -0.0 -0.0 0.0 0.0 -8e-14 5.6e-13 3.63e-12 -2.622e-11 1.5113e-10 1.032549e-08 6.8310027e-07 3.554859267e-05 0.00136476605923 0.03577661660379 0.03668359552735 0.00287444790368 6.324352877e-05 1.07366847e-06 1.437255e-08 2.0695e-10 -2.094e-11 2.33e-12 4.2e-13 -5e-14 0.0 0.0 -0.0 -D/cons.3.00.000000.dat 2.5 2.5 2.5 2.5 2.5 2.5 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 -D/cons.3.00.000006.dat 2.49999999928568 2.49999994654389 2.4999964735218 2.49981734638575 2.49305675656102 2.37634675700937 1.36967354333457 1.26081856973676 1.25028504291838 1.25000548091579 1.25000008276828 1.25000000099198 1.24999999994637 1.25000000000411 1.25000000000286 1.24999999999976 1.24999999999999 1.25000000000001 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.24999999999999 1.25000000000023 1.24999999999833 1.24999999998939 1.25000000006623 1.24999999963097 1.249999969432 1.24999797934343 1.24989485925863 1.24597556839486 1.15270465024259 0.34422215943072 0.25703510060941 0.25016683463152 0.25000284054476 0.25000003812134 0.25000000032996 0.24999999996909 0.25000000000553 0.25000000000115 0.24999999999987 0.25 0.25 0.25 -D/cons.4.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -D/cons.4.00.000006.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -D/prim.1.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 -D/prim.1.00.000006.dat 0.99999999979591 0.99999998472683 0.99999899243059 0.99994780428952 0.99800633490866 0.96114045213925 0.53777269575953 0.5030507498549 0.5000813962002 0.50000156595549 0.50000002364808 0.50000000028342 0.49999999998468 0.50000000000118 0.50000000000082 0.49999999999993 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.50000000000007 0.49999999999952 0.49999999999697 0.50000000001892 0.49999999989456 0.49999999126629 0.49999942266678 0.49996995381996 0.49884345037773 0.46690023384529 0.15684108925066 0.12738536102708 0.12505946967055 0.12500101444172 0.12500001361476 0.12500000011784 0.12499999998896 0.12500000000198 0.12500000000041 0.12499999999995 0.125 0.125 0.125 -D/prim.2.00.000000.dat 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -D/prim.2.00.000006.dat 2.2435e-10 1.808107e-08 1.1921753e-06 6.175822088e-05 0.00235813344129 0.04824925512011 0.08060785310619 0.0074788947115 0.00019285712307 3.70581434e-06 5.592919e-08 7.5549e-10 -4.541e-11 3.12e-12 1.91e-12 -1.6e-13 -1e-14 0.0 -0.0 -0.0 -0.0 -0.0 -0.0 0.0 0.0 -0.0 0.0 -0.0 0.0 0.0 -0.0 0.0 0.0 -0.0 -0.0 0.0 0.0 -0.0 -0.0 0.0 0.0 -1.5e-13 1.12e-12 7.27e-12 -5.243e-11 3.0225e-10 2.065098e-08 1.36620211e-06 7.1101458e-05 0.00273586043516 0.07662582712616 0.23389021144023 0.02256497827149 0.00050570763599 8.58927807e-06 1.1498037e-07 1.65559e-09 -1.6754e-10 1.86e-11 3.32e-12 -3.8e-13 1e-14 1e-14 -0.0 -D/prim.3.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 -D/prim.3.00.000006.dat 0.99999999971427 0.99999997861756 0.99999858940844 0.99992693779153 0.99722159268301 0.9500911976124 0.54717056816571 0.50432180038005 0.50011401344736 0.50000219236494 0.50000003310731 0.50000000039679 0.49999999997855 0.50000000000165 0.50000000000114 0.4999999999999 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.50000000000009 0.49999999999933 0.49999999999575 0.50000000002649 0.49999999985239 0.4999999877728 0.49999919173719 0.49995794319794 0.49838948059605 0.46053357752923 0.13597287698943 0.10280106787287 0.10006672745606 0.10000113621606 0.10000001524854 0.10000000013198 0.09999999998764 0.10000000000221 0.10000000000046 0.09999999999995 0.1 0.1 0.1 -D/prim.4.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -D/prim.4.00.000006.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 \ No newline at end of file diff --git a/tests/DDC4BA8A/golden-metadata.txt b/tests/DDC4BA8A/golden-metadata.txt deleted file mode 100644 index 949342ed82..0000000000 --- a/tests/DDC4BA8A/golden-metadata.txt +++ /dev/null @@ -1,193 +0,0 @@ -This file was created on 2026-07-08 14:15:53.375534. - -mfc.sh: - - Invocation: test --generate --only 053C5DDA DDC4BA8A BA4340EA 60739A3E -j 8 -- -b mpirun - Lock: mpi=Yes & gpu=No & debug=No & reldebug=Yes & gcov=No & unified=No & single=No & mixed=No & fastmath=Yes - Git: ace2285a7e72fdce9dc3fbc3a5629e1b9d1a89b7 on amr-hybrid-test-robustness (dirty) - -syscheck: - - CMake Configuration: - - CMake v3.26.5 on atl1-1-02-005-27-0.pace.gatech.edu - - C : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc) - Fortran : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran) - - PRE_PROCESS : OFF - SIMULATION : OFF - POST_PROCESS : OFF - SYSCHECK : ON - DOCUMENTATION : OFF - ALL : OFF - - MPI : ON - OpenACC : OFF - OpenMP : OFF - - Fypp : /storage/project/r-sbryngelson3-0/sbryngelson3/MFC-lomach/build/venv/bin/fypp - Doxygen : - - Build Type : RelDebug - - Configuration Environment: - - CC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc - CXX : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/g++ - FC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran - OMPI_CC : - OMPI_CXX : - OMPI_FC : - -simulation: - - CMake Configuration: - - CMake v3.26.5 on atl1-1-02-005-27-0.pace.gatech.edu - - C : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc) - Fortran : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran) - - PRE_PROCESS : OFF - SIMULATION : ON - POST_PROCESS : OFF - SYSCHECK : OFF - DOCUMENTATION : OFF - ALL : OFF - - MPI : ON - OpenACC : OFF - OpenMP : OFF - - Fypp : /storage/project/r-sbryngelson3-0/sbryngelson3/MFC-lomach/build/venv/bin/fypp - Doxygen : - - Build Type : RelDebug - - Configuration Environment: - - CC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc - CXX : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/g++ - FC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran - OMPI_CC : - OMPI_CXX : - OMPI_FC : - -post_process: - - CMake Configuration: - - CMake v3.26.5 on atl1-1-02-005-27-0.pace.gatech.edu - - C : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc) - Fortran : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran) - - PRE_PROCESS : OFF - SIMULATION : OFF - POST_PROCESS : ON - SYSCHECK : OFF - DOCUMENTATION : OFF - ALL : OFF - - MPI : ON - OpenACC : OFF - OpenMP : OFF - - Fypp : /storage/project/r-sbryngelson3-0/sbryngelson3/MFC-lomach/build/venv/bin/fypp - Doxygen : - - Build Type : RelDebug - - Configuration Environment: - - CC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc - CXX : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/g++ - FC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran - OMPI_CC : - OMPI_CXX : - OMPI_FC : - -pre_process: - - CMake Configuration: - - CMake v3.26.5 on atl1-1-02-005-27-0.pace.gatech.edu - - C : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc) - Fortran : GNU v12.3.0 (/usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran) - - PRE_PROCESS : ON - SIMULATION : OFF - POST_PROCESS : OFF - SYSCHECK : OFF - DOCUMENTATION : OFF - ALL : OFF - - MPI : ON - OpenACC : OFF - OpenMP : OFF - - Fypp : /storage/project/r-sbryngelson3-0/sbryngelson3/MFC-lomach/build/venv/bin/fypp - Doxygen : - - Build Type : RelDebug - - Configuration Environment: - - CC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gcc - CXX : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/g++ - FC : /usr/local/pace-apps/spack/packages/linux-rhel9-x86_64_v3/gcc-11.3.1/gcc-12.3.0-ukkkutsxfl5kpnnaxflpkq2jtliwthfz/bin/gfortran - OMPI_CC : - OMPI_CXX : - OMPI_FC : - -CPU: - - CPU Info: - From lscpu - Architecture: x86_64 - CPU op-mode(s): 32-bit, 64-bit - Address sizes: 46 bits physical, 48 bits virtual - Byte Order: Little Endian - CPU(s): 24 - On-line CPU(s) list: 0-23 - Vendor ID: GenuineIntel - Model name: Intel(R) Xeon(R) Gold 6226 CPU @ 2.70GHz - CPU family: 6 - Model: 85 - Thread(s) per core: 1 - Core(s) per socket: 12 - Socket(s): 2 - Stepping: 7 - CPU(s) scaling MHz: 100% - CPU max MHz: 2700.0000 - CPU min MHz: 1200.0000 - BogoMIPS: 5400.00 - Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req vnmi pku ospke avx512_vnni md_clear flush_l1d arch_capabilities - Virtualization: VT-x - L1d cache: 768 KiB (24 instances) - L1i cache: 768 KiB (24 instances) - L2 cache: 24 MiB (24 instances) - L3 cache: 38.5 MiB (2 instances) - NUMA node(s): 2 - NUMA node0 CPU(s): 0-11 - NUMA node1 CPU(s): 12-23 - Vulnerability Gather data sampling: Vulnerable - Vulnerability Indirect target selection: Vulnerable - Vulnerability Itlb multihit: KVM: Vulnerable - Vulnerability L1tf: Not affected - Vulnerability Mds: Not affected - Vulnerability Meltdown: Not affected - Vulnerability Mmio stale data: Vulnerable - Vulnerability Reg file data sampling: Not affected - Vulnerability Retbleed: Vulnerable - Vulnerability Spec rstack overflow: Not affected - Vulnerability Spec store bypass: Vulnerable - Vulnerability Spectre v1: Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers - Vulnerability Spectre v2: Vulnerable; IBPB: disabled; STIBP: disabled; PBRSB-eIBRS: Vulnerable; BHI: Vulnerable - Vulnerability Srbds: Not affected - Vulnerability Tsa: Not affected - Vulnerability Tsx async abort: Mitigation; TSX disabled - Vulnerability Vmscape: Vulnerable - diff --git a/tests/DDC4BA8A/golden.txt b/tests/DDC4BA8A/golden.txt deleted file mode 100644 index 81a0b45a28..0000000000 --- a/tests/DDC4BA8A/golden.txt +++ /dev/null @@ -1,16 +0,0 @@ -D/cons.1.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 -D/cons.1.00.000006.dat 0.99999999984482 0.99999998989396 0.99999942790297 0.99998641325926 0.99884811869203 0.95995746621806 0.5399291230864 0.50125658770351 0.50002267488935 0.50000018958017 0.50000000878297 0.50000000014262 0.49999999997701 0.50000000000289 0.50000000000075 0.49999999999992 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.50000000000004 0.49999999999964 0.49999999999727 0.50000000001615 0.49999999993775 0.49999999667003 0.4999999321491 0.49999117663288 0.49916113602137 0.46663030626443 0.1582223114688 0.12597978301782 0.12501524230833 0.12500011073826 0.12500000473267 0.12500000005243 0.12499999999049 0.12500000000228 0.12500000000029 0.12499999999996 0.125 0.125 0.125 -D/cons.2.00.000000.dat 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -D/cons.2.00.000006.dat 1.7334e-10 1.196103e-08 6.7690433e-07 1.687776508e-05 0.00136037843838 0.04666923450375 0.04642328509974 0.00150246347973 2.68203694e-05 2.4077234e-07 1.038407e-08 2.0241e-10 -3.215e-11 3.51e-12 8.8e-13 -9e-14 -0.0 0.0 -0.0 -0.0 0.0 -0.0 0.0 0.0 0.0 0.0 0.0 -0.0 0.0 -0.0 0.0 0.0 -0.0 0.0 -0.0 -0.0 0.0 -0.0 -0.0 0.0 0.0 -5e-14 4.2e-13 3.24e-12 -2.033e-11 7.645e-11 3.94737e-09 8.702671e-08 1.043402465e-05 0.00099039153922 0.03637521810001 0.03832946148979 0.0010781387651 1.613453186e-05 1.2542421e-07 5.00901e-09 9.674e-11 -1.718e-11 2.52e-12 3e-13 -4e-14 0.0 0.0 -0.0 -D/cons.3.00.000000.dat 2.5 2.5 2.5 2.5 2.5 2.5 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 -D/cons.3.00.000006.dat 2.49999999945686 2.49999996462887 2.49999799766449 2.4999524480715 2.49597806798849 2.37308608589361 1.37648921587959 1.25441615550643 1.25007937012553 1.25000066353147 1.25000003074039 1.25000000049919 1.24999999991953 1.25000000001012 1.25000000000261 1.24999999999972 1.24999999999999 1.25000000000001 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25 1.25000000000015 1.24999999999875 1.24999999999044 1.25000000005654 1.24999999978214 1.24999998834509 1.24999976252195 1.24996911948904 1.24707445137907 1.15164258306816 0.3484790837807 0.25279199644208 0.25004269169899 0.25000031006817 0.25000001325147 0.2500000001468 0.24999999997336 0.25000000000638 0.25000000000082 0.24999999999989 0.25 0.25 0.25 -D/cons.4.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -D/cons.4.00.000006.dat 1.0 1.0 1.0 1.00002854123887 1.0000000000917 1.0 1.0 1.0 1.0 0.99999876553253 0.99999999999999 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.00000053313484 1.00000000291781 0.99999999999995 0.99999999999974 1.0000000000003 1.0 1.0 0.99999690875649 0.99999999999995 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -D/prim.1.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125 -D/prim.1.00.000006.dat 0.99999999984482 0.99999998989396 0.99999942790297 0.99998641325926 0.99884811869203 0.95995746621806 0.5399291230864 0.50125658770351 0.50002267488935 0.50000018958017 0.50000000878297 0.50000000014262 0.49999999997701 0.50000000000289 0.50000000000075 0.49999999999992 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.50000000000004 0.49999999999964 0.49999999999727 0.50000000001615 0.49999999993775 0.49999999667003 0.4999999321491 0.49999117663288 0.49916113602137 0.46663030626443 0.1582223114688 0.12597978301782 0.12501524230833 0.12500011073826 0.12500000473267 0.12500000005243 0.12499999999049 0.12500000000228 0.12500000000029 0.12499999999996 0.125 0.125 0.125 -D/prim.2.00.000000.dat 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -D/prim.2.00.000006.dat 1.7334e-10 1.196103e-08 6.7690472e-07 1.687799439e-05 0.00136194723994 0.04861593992035 0.08598033170423 0.00299739398262 5.363830632e-05 4.8154451e-07 2.076814e-08 4.0482e-10 -6.431e-11 7.02e-12 1.75e-12 -1.9e-13 -0.0 0.0 -0.0 -0.0 0.0 -0.0 0.0 0.0 0.0 0.0 0.0 -0.0 0.0 -0.0 0.0 0.0 -0.0 0.0 -0.0 -0.0 0.0 -0.0 -0.0 0.0 0.0 -1e-13 8.5e-13 6.47e-12 -4.066e-11 1.5289e-10 7.89474e-09 1.7405345e-07 2.086841756e-05 0.00198411187841 0.07795296964573 0.24225067333404 0.00855803002092 0.00012906051746 1.00339282e-06 4.007205e-08 7.7394e-10 -1.3741e-10 2.017e-11 2.39e-12 -3.3e-13 1e-14 1e-14 -0.0 -D/prim.3.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 -D/prim.3.00.000006.dat 0.99999999978274 0.99999998585155 0.9999991990657 0.9999524392902 0.99839085655111 0.9487806606173 0.5497973884615 0.50176556150757 0.50003174776249 0.50000088264739 0.50000001229616 0.50000000019967 0.49999999996781 0.50000000000405 0.50000000000105 0.49999999999989 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.50000000000006 0.4999999999995 0.49999999999618 0.50000000002262 0.49999999991286 0.49999999533804 0.49999963844155 0.4999876462932 0.49882938754213 0.4600899219729 0.13753456594135 0.10111495322805 0.10001707626313 0.10000043315293 0.10000000530059 0.10000000005872 0.09999999998935 0.10000000000255 0.10000000000033 0.09999999999996 0.1 0.1 0.1 -D/prim.4.00.000000.dat 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -D/prim.4.00.000006.dat 1.0 1.0 1.0 1.00002854123887 1.0000000000917 1.0 1.0 1.0 1.0 0.99999876553253 0.99999999999999 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.00000053313484 1.00000000291781 0.99999999999995 0.99999999999974 1.0000000000003 1.0 1.0 0.99999690875649 0.99999999999995 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 \ No newline at end of file diff --git a/toolchain/mfc/case.py b/toolchain/mfc/case.py index 951f0c198f..e770e5b24b 100644 --- a/toolchain/mfc/case.py +++ b/toolchain/mfc/case.py @@ -376,8 +376,6 @@ def __get_sim_fpp(self, print: bool) -> str: viscous = 1 if self.params.get("viscous", "F") == "T" else 0 igr = 1 if self.params.get("igr", "F") == "T" else 0 igr_pres_lim = 1 if self.params.get("igr_pres_lim", "F") == "T" else 0 - hybrid_riemann = 1 if self.params.get("hybrid_riemann", "F") == "T" else 0 - hybrid_weno = 1 if self.params.get("hybrid_weno", "F") == "T" else 0 # Throw error if wenoz_q is required but not set out = f"""\ @@ -406,8 +404,6 @@ def __get_sim_fpp(self, print: bool) -> str: #:set igr_pres_lim = {igr_pres_lim} #:set igr_order = {self.params.get("igr_order", 3)} #:set viscous = {viscous} -#:set hybrid_riemann = {hybrid_riemann} -#:set hybrid_weno = {hybrid_weno} """ else: diff --git a/toolchain/mfc/case_validator.py b/toolchain/mfc/case_validator.py index 860471c565..5bbcc210b2 100644 --- a/toolchain/mfc/case_validator.py +++ b/toolchain/mfc/case_validator.py @@ -216,7 +216,9 @@ "title": "Adaptive Mesh Refinement (AMR)", "category": "Adaptive Mesh Refinement", "explanation": ( - "Block-structured AMR (Experimental) adds up to amr_max_blocks 2:1 refined level-1 blocks. " + "Block-structured AMR (Experimental) adds up to amr_max_blocks refined blocks at ref_ratio:1 " + "refinement (ref_ratio = 2 or 4); with amr_max_level > 1 the blocks nest recursively to that " + "depth (multi-level nesting requires ref_ratio = 2). " "Requires WENO reconstruction (recon_type = 1), SSP-RK3 (time_stepper = 3), " "and the 5- or 6-equation model (model_eqns = 2 or 3; for 6-eq the per-stage pressure " "relaxation also runs on each fine block); num_fluids > 1 additionally requires " @@ -263,12 +265,12 @@ "cleaning spreads but cannot remove; divergence-preserving B prolongation/reflux is future " "work). 1D MHD/RMHD IS supported: div(B) = 0 by construction there (Bx is the uniform Bx0 " "parameter; By/Bz reflux and restrict as ordinary conserved scalars). " - "hybrid_weno/hybrid_riemann are supported: each level recomputes the " - "sensor over its own (swapped) bounds every RHS call. " "active_box is supported (single-rank, per active_box's own MPI gate): blocks must " "sit strictly inside the growing active window (init abort + regrid clamp), and the " "fine advance treats its whole block as active. " "Dynamic regrid (amr_regrid_int > 0) requires amr_tag_eps > 0 and amr_buf >= 1. " + "ref_ratio must be 2 or 4 (ref_ratio = 4 is single-level without subcycling); amr_max_level >= 1, " + "and multi-level (amr_max_level > 1) needs amr_max_blocks >= 2. " "amr_subcycle advances the fine level at dt/2 with Berger-Colella refluxing; " "incompatible with cfl_dt. " "Under MPI the patch may span ranks (each rank holds the fine cells covering its " @@ -1359,10 +1361,6 @@ def check_active_box(self): self.prohibit(recon_type is not None and recon_type != 1, "active_box requires WENO reconstruction (recon_type = 1)") self.prohibit(time_stepper is not None and time_stepper != 3, "active_box requires time_stepper = 3 (SSP-RK3)") self.prohibit(ib, "active_box is incompatible with immersed boundaries") - self.prohibit( - self.get("hybrid_weno", "F") == "T" or self.get("hybrid_riemann", "F") == "T", - "active_box is incompatible with hybrid_weno/hybrid_riemann: " "the smoothness sensor sweeps the full domain but the cons-to-prim conversion is narrowed to the active-box footprint", - ) self.prohibit(acoustic_source, "active_box is incompatible with acoustic sources") self.prohibit(bodyForces, "active_box is incompatible with body forces") self.prohibit(bubbles_lagrange, "active_box is incompatible with Lagrangian bubbles") @@ -1417,8 +1415,20 @@ def check_amr(self): amr_buf = self.get("amr_buf") amr_max_blocks = self.get("amr_max_blocks") amr_cluster_eff = self.get("amr_cluster_eff") + amr_max_level = self.get("amr_max_level") + ref_ratio = self.get("ref_ratio") self.prohibit(amr_max_blocks is not None and amr_max_blocks < 1, "amr_max_blocks must be >= 1") + self.prohibit(amr_max_level is not None and amr_max_level < 1, "amr_max_level must be >= 1") + self.prohibit( + amr_max_level is not None and amr_max_level > 1 and amr_max_blocks is not None and amr_max_blocks < 2, + "multi-level AMR (amr_max_level > 1) needs amr_max_blocks >= 2 " "(at least one level-1 block plus one nested level-2 block)", + ) + self.prohibit(ref_ratio is not None and ref_ratio not in (2, 4), "ref_ratio must be 2 or 4") + self.prohibit( + ref_ratio is not None and ref_ratio != 2 and ((amr_max_level is not None and amr_max_level > 1) or amr_subcycle), + "ref_ratio /= 2 is only supported at amr_max_level = 1 without subcycling (v1)", + ) self.prohibit( amr_cluster_eff is not None and (amr_cluster_eff <= 0 or amr_cluster_eff > 1), "amr_cluster_eff must satisfy 0 < amr_cluster_eff <= 1", diff --git a/toolchain/mfc/params/definitions.py b/toolchain/mfc/params/definitions.py index 565e5a9267..1c61e75873 100644 --- a/toolchain/mfc/params/definitions.py +++ b/toolchain/mfc/params/definitions.py @@ -61,8 +61,6 @@ def _fc(name: str, default: int) -> int: "recon_type", "muscl_order", "muscl_lim", - "hybrid_riemann", - "hybrid_weno", } @@ -668,8 +666,6 @@ def _load(): "sfc_partition_wrt", "load_balance", "rank_time_wrt", - "hybrid_weno", - "hybrid_riemann", "amr", ]: _r(n, LOG, {"output"}) @@ -684,8 +680,6 @@ def _load(): _r("amr_max_level", INT) _r("amr_cluster_eff", REAL) _r("ref_ratio", INT) - _r("hybrid_weno_eps", REAL, {"output"}) - _r("hybrid_smooth_flux", INT, {"output"}) _r("partition_tile_size", INT, {"output"}) for n in [ "schlieren_wrt", @@ -1286,10 +1280,6 @@ def _nv(targets: set, *names: str) -> None: "sfc_partition_wrt", "load_balance", "rank_time_wrt", - "hybrid_weno", - "hybrid_weno_eps", - "hybrid_riemann", - "hybrid_smooth_flux", "partition_tile_size", "alt_soundspeed", "mixture_err", diff --git a/toolchain/mfc/params/descriptions.py b/toolchain/mfc/params/descriptions.py index f5a0ac4ab8..ac18a61ba7 100644 --- a/toolchain/mfc/params/descriptions.py +++ b/toolchain/mfc/params/descriptions.py @@ -127,10 +127,6 @@ "amr_max_level": "Maximum AMR refinement depth (refined levels above L0); >= 1, default 1. Multi-level (>= 2) supported: static (amr_regrid_int=0) up to 2, dynamic regrid (>0) deeper", "amr_cluster_eff": "Berger-Rigoutsos min tag efficiency (tagged/total) a clustered block box must reach before splitting stops (0 < eff <= 1)", "ref_ratio": "AMR refinement ratio between coarse and fine levels (2 or 4; default 2; only 2 supported with multi-level or subcycling in v1)", - "hybrid_weno": "Use linear-optimal reconstruction in smooth cells, full WENO only at flagged discontinuities", - "hybrid_weno_eps": "Smoothness threshold for hybrid WENO shock flagging (must be > 0)", - "hybrid_riemann": "Use a cheap central/Rusanov flux in smooth cells, full HLLC only at flagged discontinuities (requires HLLC, 5eq/6eq)", - "hybrid_smooth_flux": "Smooth-region flux for hybrid Riemann: 1 = central, 2 = Rusanov", "partition_tile_size": "Tile side for the SFC partitioner", "cons_vars_wrt": "Write conservative variables", "probe_wrt": "Write probe data", diff --git a/toolchain/mfc/test/cases.py b/toolchain/mfc/test/cases.py index 344cc86935..45c6edadeb 100644 --- a/toolchain/mfc/test/cases.py +++ b/toolchain/mfc/test/cases.py @@ -2824,17 +2824,6 @@ def amr_golden_tests(): # restart_check on the REGRIDDED layout: unlike the static block, a regridded block set # cannot be reconstructed from the ICs, so the roundtrip proves the restart file itself cases.append(define_case_d(stack, "", {}, restart_check=True)) - # hybrid sensors on the fine level. eps must be chosen so no cell's Jameson phi lands - # NEAR it on any timestep, or a compiler's FP-reordering flips a cell between central and - # WENO -> a percent-level local golden diff (the failure mode that killed eps=0.5, which - # sat right where a shock cell's phi crosses). An eps-sweep of this exact case shows a - # clean phi GAP over [0.20, 0.40] (bit-identical output across the whole range -> no cell - # sits there on any step): eps 0.30 is centred in it with a +/-0.10 margin (~1000x the - # ~1e-4 cross-compiler phi drift), so no compiler can flip a cell, while still moving the - # answer ~4e-4 vs a dead sensor (all-WENO). override_tol 5e-5 sits an order below that - # ~4e-4 dead-sensor signal and ~8x above the ~6e-6 flip-free FP drift. - cases.append(define_case_d(stack, "hybrid_weno sensor", {"hybrid_weno": "T", "hybrid_weno_eps": 0.3}, override_tol=5.0e-5)) - cases.append(define_case_d(stack, "hybrid_riemann sensor", {"hybrid_riemann": "T", "hybrid_weno_eps": 0.3, "hybrid_smooth_flux": 2}, override_tol=5.0e-5)) # 2 MPI ranks + parallel_io: the ONLY test that executes the MPI-IO AMR restart write/read # (EXSCAN offset arithmetic, per-rank-extents validation) and multi-rank dynamic regrid # (coarse-halo exchange before tagging, fine seam halo) - a rank-seam or restart-offset bug @@ -2957,10 +2946,6 @@ def amr_golden_tests(): stack.push("AMR -> 1D -> MHD -> HLLD", {**mhd_1d_base, "amr_regrid_int": 0}) cases.append(define_case_d(stack, "", {})) cases.append(define_case_d(stack, "dynamic regrid", {"amr_regrid_int": 5, "amr_tag_eps": 0.05, "amr_buf": 3})) - # hlld hybrid: the smooth-flux override reuses HLLD's own F_L/F_R physical fluxes (Rusanov at a - # WENO-smooth face). Liveness eps (1e-2, bit-identical to plain by design) confirms the MHD override is - # wired without the consequential-eps threshold-flip fragility (see the HLL/LF note in hybrid_sensor_tests) - cases.append(define_case_d(stack, "hybrid_riemann", {"hybrid_riemann": "T", "hybrid_weno_eps": 1.0e-2, "hybrid_smooth_flux": 2})) stack.pop() stack.push( "AMR -> 1D -> RMHD", @@ -3961,6 +3946,22 @@ def amr_golden_tests(): cases.append(define_case_d(stack, "", {})) stack.pop() + # (h2) multi-level restart roundtrip: same static L2 hierarchy as (h), but restart_check proves the + # AMR restart file round-trips a MULTI-LEVEL block set. A level-2 block's fine extent is ref_ratio**2 + # (not ref_ratio) times its coarse region, so the single-level restart reader rejected every L2 block + # as "corrupt"; the per-block level stored in the file lets the reader rebuild the multi-level geometry. + stack.push( + "AMR -> 1D -> multi-level restart", + { + **amr_1d_base, + "amr_regrid_int": 0, + "amr_max_level": 2, + "amr_max_blocks": 8, + }, + ) + cases.append(define_case_d(stack, "", {}, restart_check=True)) + stack.pop() + # (i) multi-level + dynamic regrid: (h) is a static hierarchy; this arms amr_regrid_int so the # level-2 child is placed by SENSOR-ON-FINE (the density-gradient sensor run on the level-1 fine # solution, coarsened + clustered into a nested child box), not a fixed inset. This is the ONLY @@ -4048,6 +4049,24 @@ def amr_golden_tests(): cases.append(define_case_d(stack, "", {}, ppn=2)) stack.pop() + # (l') multi-level restart via parallel_io (MPI-IO): (l)'s static 2-level hierarchy at np=2, but the restart + # roundtrip goes through the MPI-IO path. Proves the shared restart file round-trips a MULTI-LEVEL block set: + # a level-2 block's fine data is ref_ratio**2 (not ref_ratio) of its region, so the per-block MPI-IO header + # must carry the refinement level. Without it the reader sized L2 blocks at level-1 extents and mislaid every + # downstream block offset (caught only by the total-size tripwire). This is the ONLY multi-level MPI-IO restart. + stack.push( + "AMR -> 1D -> multi-level restart parallel_io np=2", + { + **amr_1d_base, + "amr_regrid_int": 0, + "amr_max_level": 2, + "amr_max_blocks": 8, + "parallel_io": "T", + }, + ) + cases.append(define_case_d(stack, "", {}, ppn=2, restart_check=True, honor_io_keys=True)) + stack.pop() + # (m) multi-level + dynamic regrid at np=2: (l) is a STATIC 2-level hierarchy on two ranks; this arms # amr_regrid_int so the level-2 children are placed by DISTRIBUTED sensor-on-fine nesting. Each rank tags # children only for the level-1 parents it owns (its local fine data), the tags are OR-reduced across ranks @@ -4176,65 +4195,6 @@ def amr_golden_tests(): amr_golden_tests() - def hybrid_sensor_tests(): - """Golden tests for the hybrid WENO/Riemann smoothness sensors: a 1D Sod-type shock so the - sensor genuinely partitions the domain (full nonlinear WENO/upwind flux at the - discontinuities, central weights/flux in the smooth regions). These protect the sensor - plumbing and the shared nonlinear-weight block in m_weno against silent divergence.""" - hybrid_1d_base = { - "m": 63, - "n": 0, - "p": 0, - "dt": 5.0e-4, - "t_step_stop": 6, - "t_step_save": 6, - "x_domain%beg": 0.0, - "x_domain%end": 1.0, - "bc_x%beg": -3, - "bc_x%end": -3, - "patch_icpp(1)%geometry": 1, - "patch_icpp(1)%x_centroid": 0.05, - "patch_icpp(1)%length_x": 0.1, - "patch_icpp(1)%vel(1)": 0.0, - "patch_icpp(2)%geometry": 1, - "patch_icpp(2)%x_centroid": 0.45, - "patch_icpp(2)%length_x": 0.7, - "patch_icpp(2)%vel(1)": 0.0, - "patch_icpp(3)%geometry": 1, - "patch_icpp(3)%x_centroid": 0.9, - "patch_icpp(3)%length_x": 0.2, - "patch_icpp(3)%vel(1)": 0.0, - } - stack.push("Hybrid -> 1D -> WENO sensor", {**hybrid_1d_base, "hybrid_weno": "T", "hybrid_weno_eps": 1.0e-2}) - cases.append(define_case_d(stack, "", {})) - # liveness golden: the eps=1e-2 case above is bitwise-identical to plain by design (only - # constant/eps-dominated cells go central) so it protects no-corruption but cannot detect - # a dead sensor. eps=0.3 puts consequential cells under the sensor (answer moves ~4e-4 vs - # plain WENO). eps is centred in the [0.20,0.40] phi GAP measured by an eps-sweep of this - # case (output bit-identical across that range -> no cell's Jameson phi sits there on any - # timestep), giving a +/-0.10 margin so no compiler's FP reordering can flip a cell across - # the threshold - the fragility that failed eps=0.5 (which sat where a shock cell flips). - cases.append(define_case_d(stack, "consequential eps", {"hybrid_weno_eps": 0.3}, override_tol=5.0e-5)) - stack.pop() - stack.push( - "Hybrid -> 1D -> Riemann sensor", - {**hybrid_1d_base, "hybrid_riemann": "T", "hybrid_weno_eps": 1.0e-2, "hybrid_smooth_flux": 2}, - ) - cases.append(define_case_d(stack, "", {})) - cases.append(define_case_d(stack, "consequential eps", {"hybrid_weno_eps": 0.3}, override_tol=5.0e-5)) - # the central smooth-flux (enum 1) is a distinct flux path from Rusanov (2) - cover both - cases.append(define_case_d(stack, "central flux", {"hybrid_smooth_flux": 1})) - # Cover HLL and Lax-Friedrichs at the liveness eps (inherited 1e-2). HLL shares the smooth-flux helper - # (s_compute_hybrid_smooth_flux). LF instead just switches to the local (normal-velocity) wave speed - # under hybrid_riemann - i.e. LF becomes local-LF everywhere - because LF's own flux carries pcorr and a - # distinct advection form the generic helper lacks; overwriting only smooth faces injected a sensor-flip - # discontinuity that made LF diverge ~2% across compilers. Local-LF is a single consistent scheme. - cases.append(define_case_d(stack, "HLL", {"riemann_solver": 1})) - cases.append(define_case_d(stack, "Lax-Friedrichs", {"riemann_solver": 5})) - stack.pop() - - hybrid_sensor_tests() - def load_balance_tests(): """Golden for the weighted init-time decomposition (load_balance): a two-fluid material interface at x=0.5 makes the alpha marginal asymmetric (fluid-1 volume fraction ~1 left, diff --git a/toolchain/mfc/test/test.py b/toolchain/mfc/test/test.py index e9c0beb2bc..58518684b3 100644 --- a/toolchain/mfc/test/test.py +++ b/toolchain/mfc/test/test.py @@ -156,11 +156,6 @@ def is_uuid(term): # nonpolytropic pair carries override_tol=5e-9, unsatisfiable below single epsilon "AMR -> 1D -> acoustic", "nonpolytropic", - # the consequential-eps hybrid goldens carry override_tol=1e-8 (unscaled by the - # single 1e8 relaxation), below single-precision drift for these cases - "consequential eps", - "hybrid_weno sensor", - "hybrid_riemann sensor", ] if any(label in case.trace for label in skip): cases.remove(case)