From 79dc253a3297d29df6d6457d4f11a2887e3abd7a Mon Sep 17 00:00:00 2001 From: lmoresi Date: Mon, 4 May 2026 17:02:31 +1000 Subject: [PATCH] Document CSV timing output as the recommended high-rank workaround (#134) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit uw.timing.print_table() ultimately calls PETSc's PetscLogView. At very high CPU counts (≳1000 ranks), the ASCII output path can hang — the job completes its computation cleanly but never exits. The CSV write path uses a different, less collective-heavy strategy and avoids the issue. The behaviour is in PETSc, not Underworld; documenting CSV-at-scale as the recommended workaround rather than working around the PETSc internal in our code (which would diverge from PETSc's own ASCII / CSV separation). Confirmation in #134 from gthyagi: "writing BD integrals to a .txt file can cause the job to hang, whereas using .csv output avoids the problem." Two doc surfaces touched: - uw.timing.print_table docstring — adds a Notes section pointing at #134 and showing the safe call form. - docs/advanced/parallel-computing.md — new "Timing Output at Extreme Scale" section before the Summary, plus a sixth Key-Takeaways bullet. Issue #134 stays open until merge; closing manually after. Underworld development team with AI support from Claude Code --- docs/advanced/parallel-computing.md | 21 +++++++++++++++++++++ src/underworld3/timing.py | 26 +++++++++++++++++++++----- 2 files changed, 42 insertions(+), 5 deletions(-) diff --git a/docs/advanced/parallel-computing.md b/docs/advanced/parallel-computing.md index 81aa1e069..7181f7c10 100644 --- a/docs/advanced/parallel-computing.md +++ b/docs/advanced/parallel-computing.md @@ -508,6 +508,26 @@ These operations require **ALL ranks** to participate: - [ ] Test with `mpirun -np 2` and `mpirun -np 4` - [ ] Check for deadlocks (script hangs = collective operation issue) +## Timing Output at Extreme Scale + +`uw.timing.print_table()` ultimately calls PETSc's `PetscLogView`. At very +high CPU counts (≳1000 ranks), the **ASCII output path** can hang — +typically appearing as a job that completes its computation cleanly but +never exits. The CSV write path uses a different, less collective-heavy +strategy and avoids the issue: + +```python +# Default — fine at small scale, can hang at ≳1000 ranks +uw.timing.print_table() +uw.timing.print_table("results.txt") + +# Safe at any scale — recommended for HPC runs +uw.timing.print_table("results.csv") +``` + +The behaviour is in PETSc, not Underworld; choosing CSV at scale is the +recommended workaround. (Issue #134.) + ## Summary **Key Takeaways:** @@ -517,5 +537,6 @@ These operations require **ALL ranks** to participate: 3. **Use `with uw.selective_ranks(ranks):`** for serial operations 4. **Collective operations must run on ALL ranks** - never inside rank conditionals 5. **Test with `mpirun -np N`** to catch issues early +6. **At ≳1000 ranks, write timing output as `.csv`** to avoid `PetscLogView` hangs The parallel safety system makes parallel programming in Underworld3 safer and more intuitive - collective operations are evaluated on all ranks automatically, preventing common deadlock scenarios! diff --git a/src/underworld3/timing.py b/src/underworld3/timing.py index ed2248d8e..58fab2b95 100644 --- a/src/underworld3/timing.py +++ b/src/underworld3/timing.py @@ -150,13 +150,13 @@ def print_table(filename=None, format="auto"): ---------- filename : str, optional If provided, write results to file. Extension determines format: - - `.csv` : Spreadsheet-compatible CSV format - - `.txt` or other : Human-readable ASCII table + - ``.csv`` : Spreadsheet-compatible CSV format + - ``.txt`` or other : Human-readable ASCII table format : str, optional Override automatic format detection: - - "auto" : Detect from filename (default) - - "ascii" : Human-readable table - - "csv" : Comma-separated values + - ``"auto"`` : Detect from filename (default) + - ``"ascii"`` : Human-readable table + - ``"csv"`` : Comma-separated values Example ------- @@ -164,6 +164,22 @@ def print_table(filename=None, format="auto"): >>> # ... do work ... >>> uw.timing.print_table() # Print to console >>> uw.timing.print_table("results.csv") # Save as CSV + + Notes + ----- + **High-CPU-count usage (≳1000 ranks): prefer CSV output.** + + Issue #134 (gthyagi, 2026-04-23): the underlying PETSc ``PetscLogView`` + ASCII output path can hang at extreme rank counts on some clusters + (BD-integral routines + ASCII table emit appear to be the trigger), + while the CSV write path uses a different, less collective-heavy + strategy and avoids the issue. If your job is large enough that + timing-output cost matters, write to a ``.csv`` filename: + + >>> uw.timing.print_table("results.csv") # safe at any scale + + The behaviour is in PETSc, not Underworld; choosing CSV at scale is + the recommended workaround. """ print_petsc_log(filename=filename, format=format)