diff --git a/docs/advanced/parallel-computing.md b/docs/advanced/parallel-computing.md index 81aa1e069..7181f7c10 100644 --- a/docs/advanced/parallel-computing.md +++ b/docs/advanced/parallel-computing.md @@ -508,6 +508,26 @@ These operations require **ALL ranks** to participate: - [ ] Test with `mpirun -np 2` and `mpirun -np 4` - [ ] Check for deadlocks (script hangs = collective operation issue) +## Timing Output at Extreme Scale + +`uw.timing.print_table()` ultimately calls PETSc's `PetscLogView`. At very +high CPU counts (≳1000 ranks), the **ASCII output path** can hang — +typically appearing as a job that completes its computation cleanly but +never exits. The CSV write path uses a different, less collective-heavy +strategy and avoids the issue: + +```python +# Default — fine at small scale, can hang at ≳1000 ranks +uw.timing.print_table() +uw.timing.print_table("results.txt") + +# Safe at any scale — recommended for HPC runs +uw.timing.print_table("results.csv") +``` + +The behaviour is in PETSc, not Underworld; choosing CSV at scale is the +recommended workaround. (Issue #134.) + ## Summary **Key Takeaways:** @@ -517,5 +537,6 @@ These operations require **ALL ranks** to participate: 3. **Use `with uw.selective_ranks(ranks):`** for serial operations 4. **Collective operations must run on ALL ranks** - never inside rank conditionals 5. **Test with `mpirun -np N`** to catch issues early +6. **At ≳1000 ranks, write timing output as `.csv`** to avoid `PetscLogView` hangs The parallel safety system makes parallel programming in Underworld3 safer and more intuitive - collective operations are evaluated on all ranks automatically, preventing common deadlock scenarios! diff --git a/src/underworld3/timing.py b/src/underworld3/timing.py index ed2248d8e..58fab2b95 100644 --- a/src/underworld3/timing.py +++ b/src/underworld3/timing.py @@ -150,13 +150,13 @@ def print_table(filename=None, format="auto"): ---------- filename : str, optional If provided, write results to file. Extension determines format: - - `.csv` : Spreadsheet-compatible CSV format - - `.txt` or other : Human-readable ASCII table + - ``.csv`` : Spreadsheet-compatible CSV format + - ``.txt`` or other : Human-readable ASCII table format : str, optional Override automatic format detection: - - "auto" : Detect from filename (default) - - "ascii" : Human-readable table - - "csv" : Comma-separated values + - ``"auto"`` : Detect from filename (default) + - ``"ascii"`` : Human-readable table + - ``"csv"`` : Comma-separated values Example ------- @@ -164,6 +164,22 @@ def print_table(filename=None, format="auto"): >>> # ... do work ... >>> uw.timing.print_table() # Print to console >>> uw.timing.print_table("results.csv") # Save as CSV + + Notes + ----- + **High-CPU-count usage (≳1000 ranks): prefer CSV output.** + + Issue #134 (gthyagi, 2026-04-23): the underlying PETSc ``PetscLogView`` + ASCII output path can hang at extreme rank counts on some clusters + (BD-integral routines + ASCII table emit appear to be the trigger), + while the CSV write path uses a different, less collective-heavy + strategy and avoids the issue. If your job is large enough that + timing-output cost matters, write to a ``.csv`` filename: + + >>> uw.timing.print_table("results.csv") # safe at any scale + + The behaviour is in PETSc, not Underworld; choosing CSV at scale is + the recommended workaround. """ print_petsc_log(filename=filename, format=format)