AMR scaling: comm/memory-scalable regrid + restart-IO, batched flux-capture, multi-level restart

docs/documentation/amr.md

@@ -10,9 +10,9 @@
 
 Block-structured adaptive mesh refinement (AMR) concentrates resolution where the flow
 demands it — around shocks, interfaces, and bubble clouds — while leaving the rest of the
-domain at the coarser base-grid resolution. MFC implements a two-level hierarchy: the
-unmodified base (level-0) solve runs as usual, and one or more 2:1 refined rectangular
-blocks advance alongside it on a finer grid.
+domain at the coarser base-grid resolution. MFC implements a multi-level block hierarchy: the
+unmodified base (level-0) solve runs as usual, and one or more refined rectangular blocks advance
+alongside it on a finer grid — nested recursively to `amr_max_level` levels when enabled.
 
 The fine blocks are dynamically repositioned every `amr_regrid_int` coarse steps using a
 gradient-based cell tagger and Berger–Rigoutsos block clustering, so they follow moving
@@ -27,24 +27,29 @@ mid-run.
 
 ## The Block-Structured Model {#amr-model}
 
-The hierarchy has exactly two levels:
+The hierarchy spans levels `0` through `amr_max_level` (default `1`, i.e. two levels):
 
 - **Level 0** — the base grid with cell spacing `dx`, `dy`, `dz`. The ordinary MFC solver
   advances this level every step.
 - **Level 1** — a list of up to `amr_max_blocks` rectangular refined blocks, each covering
-  a sub-region of the level-0 domain at 2:1 refinement (half the cell spacing in every
-  active direction).
+  a sub-region of the level-0 domain at `ref_ratio`:1 refinement (`ref_ratio` = 2 or 4;
+  the cell spacing shrinks by `ref_ratio` in every active direction).
+- **Levels 2 … `amr_max_level`** — when `amr_max_level > 1`, blocks nest recursively: a
+  level-`l` block refines a region of its parent level-(`l-1`) block by a further
+  `ref_ratio`, tracking a moving feature to arbitrary depth. Multi-level nesting requires
+  `ref_ratio = 2`. See @ref amr_multilevel for the nesting and reflux details.
 
 Each block is described by its bounding box in level-0 cell-index space
-(`amr_block_beg(1:num_dims)` to `amr_block_end(1:num_dims)`). The fine-block extents must
-satisfy:
+(`amr_block_beg(1:num_dims)` to `amr_block_end(1:num_dims)`). The initial (level-1)
+fine-block extents must satisfy:
 
 ```
-2*(amr_block_end(i) - amr_block_beg(i) + 1) - 1 <= N_i
+ref_ratio*(amr_block_end(i) - amr_block_beg(i) + 1) - 1 <= N_i
 ```
 
 where `N_i` is the global cell count in direction `i`. This ensures the fine scratch
-(which is sized to the base grid) is never overflowed.
+(which is sized to the base grid) is never overflowed. A level-`l` block's fine extent
+grows as `ref_ratio**l`, so the nested boxes are sized accordingly.
 
 **Fixed-slot storage.** All block slots are pre-allocated at init to the maximum possible
 block size (half the per-rank subdomain in each dimension). Setting `amr_max_blocks = N`
@@ -225,7 +230,6 @@ a diagnostic message for unsupported combinations.
 | Grid stretching (`stretch_x[y,z] = T`) | Supported | Fine ghost-shell coordinates extend by exact parent-cell bisection, and the spacing-dependent WENO coefficients are recomputed for the active grid on every block swap/restore (`amr_weno_coef_recompute`, armed automatically when the grid is nonuniform); prolongation stays conservative but its slope estimate is first-order on nonuniform parents. Stretched grids do NOT combine with Lagrangian bubbles or dynamic regrid with immersed bodies (their position-to-cell-index conversions assume uniform spacing; init abort) |
 | Riemann-extrapolation BCs (`bc = -4`) | **Not supported** | Boundary-adjusted WENO coefficient rows cannot be inherited by interior blocks (checker gate) |
 | `active_box` | Supported (single-rank, per active_box's own MPI gate) | Blocks must sit strictly inside the monotonically-growing active window (init abort + regrid clamp: the windowed coarse update would drop reflux corrections at faces outside it); the fine advance disables the coarse-indexed windowing and treats its whole block as active; the frozen exterior is valid ambient data for ghost prolongation |
-| `hybrid_weno` / `hybrid_riemann` | Supported | The sensor arrays are sized to the coarse ghost-inclusive bounds (the fine extent guard keeps fine indices inside them) and the sensor is recomputed from the live bounds every RHS call, so each level evaluates its own sensor; conservative full-WENO defaults at buffer edges |
 | `acoustic_source` | Supported | The source acts on the coarse grid only: its support must not overlap the initial block (startup abort), and dynamic regrid keeps its boxes clear of the support (tags suppressed, candidate boxes clipped); emitted waves enter blocks through the coarse/fine coupling |
 
 **Mandatory solver settings.**
@@ -252,6 +256,8 @@ default values, and cross-parameter constraints see @ref case section 7.1.
 | `amr_buf` | Integer | 3 | Coarse-cell padding around tagged cells; required `>= 1` when `amr_regrid_int > 0` |
 | `amr_subcycle` | Logical | F | Advance fine level at `dt/2` (two substeps per coarse step) with Berger-Colella refluxing |
 | `amr_max_blocks` | Integer | 4 | Number of fixed refined-block slots preallocated; each slot is max-block sized (~N times device memory for N slots) |
+| `amr_max_level` | Integer | 1 | Maximum refinement depth: `1` = single refined level, `> 1` = recursive multi-level nesting (needs `amr_max_blocks >= 2` and `ref_ratio = 2`). See @ref amr_multilevel |
+| `ref_ratio` | Integer | 2 | Cell-refinement ratio between adjacent levels; must be 2 or 4. `ref_ratio = 4` is single-level only (no nesting, no subcycling) |
 | `amr_cluster_eff` | Real | 0.7 | Berger-Rigoutsos min tag efficiency a clustered box reaches before splitting stops; must satisfy `0 < amr_cluster_eff <= 1` |
 
 ---
@@ -299,13 +305,15 @@ For multi-fluid (5-equation), additionally set:
 
 - **Fixed-slot memory.** All `amr_max_blocks` slots are allocated at init at maximum size.
   More blocks = more device memory. Compact per-block memory pools are future work.
-- **Same-rank-count restart.** The AMR restart file encodes the block geometry and fine
-  solution per rank. Restarting with a different `num_procs` is not supported and aborts
-  with a clear message; np-flexible restart is future work.
+- **Restart across rank counts.** `parallel_io` restart repartitions the fine blocks
+  across any `num_procs` (each block is one contiguous region under whole-block ownership).
+  The serial (per-rank-file) restart path requires the same `num_procs` and aborts with a
+  clear message otherwise.
 - **Half-subdomain limit.** Each block may span at most half of any rank's local subdomain
   per dimension, because the fine advance reuses the rank-local solver scratch.
-- **Single refinement level.** Only one refined level is supported. Multi-level AMR
-  (levels 2, 3, ...) is not implemented.
+- **Multi-level constraints.** Recursive multi-level nesting (`amr_max_level > 1`) requires
+  `ref_ratio = 2`; with immersed boundaries it is single-rank only, and a moving body is
+  not yet supported. `ref_ratio = 4` is single-level only.
 - **Level-0 output only.** Standard visualization output (HDF5/SILO) is written at
   level-0 resolution; the restricted fine solution is already folded into the coarse
   fields over the block region, so existing visualization workflows are unchanged.

docs/documentation/case.md

@@ -687,10 +687,6 @@ To restart the simulation from $k$-th time step, see @ref running "Restarting Ca
 | `amr_max_level`        | Integer | Maximum AMR refinement depth (number of refined levels above L0); must be >= 1 (default 1). Multi-level nesting (>= 2) is supported: static AMR (`amr_regrid_int = 0`) nests up to level 2, dynamic regrid (`amr_regrid_int > 0`) nests deeper (see @ref amr_multilevel) |
 | `amr_cluster_eff`       | Real    | Berger-Rigoutsos min tag efficiency a clustered block box reaches before splitting stops; must satisfy 0 < eff <= 1 (default 0.7) |
 | `ref_ratio`             | Integer | AMR refinement ratio between coarse and fine levels; must be 2 or 4 (default 2). Only ref_ratio = 2 is supported with multi-level AMR or subcycling (v1). |
-| `hybrid_weno`           | Logical | Use linear-optimal reconstruction in smooth cells, full WENO only at flagged discontinuities (requires WENO reconstruction) |
-| `hybrid_weno_eps`       | Real    | Smoothness threshold for hybrid WENO shock flagging; must be > 0 (default 1e-2) |
-| `hybrid_riemann`        | Logical | Use a cheap central/Rusanov flux in smooth cells, full HLLC only at flagged discontinuities (requires HLLC, 5eq/6eq) |
-| `hybrid_smooth_flux`    | Integer | Smooth-region flux for hybrid Riemann: 1 = central, 2 = Rusanov (default 2) |
 | `partition_tile_size`   | Integer | Tile side for the SFC partitioner (default 8) |
 | `alpha_rho_wrt(i)`      | Logical | Add the partial density of the fluid $i$ to the database \|
 | `rho_wrt`               | Logical | Add the mixture density to the database	 |
@@ -865,9 +861,7 @@ AMR is incompatible with surface tension, 3D cylindrical
 coordinates (2D axisymmetric IS supported), 2D/3D MHD (measured: the coarse/fine seam is a
 continuous div(B) source that GLM cleaning cannot remove; 1D MHD/RMHD IS supported since
 div(B) = 0 by construction there), hyperelasticity, and Riemann-extrapolation
-boundaries (bc = -4). `active_box` is supported (single-rank): blocks must sit strictly inside the growing active window (init abort + regrid clamp), and the fine advance treats its whole block as active. `hybrid_weno`/`hybrid_riemann` are supported: each
-level recomputes the smoothness sensor over its own (swapped) bounds every RHS call.
-Nonuniform grids ARE supported (grid stretching and the axisymmetric axis half-cell): the fine
+boundaries (bc = -4). `active_box` is supported (single-rank): blocks must sit strictly inside the growing active window (init abort + regrid clamp), and the fine advance treats its whole block as active.Nonuniform grids ARE supported (grid stretching and the axisymmetric axis half-cell): the fine
 ghost-shell coordinates extend by exact parent-cell bisection and the spacing-dependent WENO
 coefficients are recomputed for the active grid on every block swap/restore, armed automatically
 when the grid is detected nonuniform at startup.