Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions docs/advanced/parallel-computing.md
Original file line number Diff line number Diff line change
Expand Up @@ -508,6 +508,26 @@ These operations require **ALL ranks** to participate:
- [ ] Test with `mpirun -np 2` and `mpirun -np 4`
- [ ] Check for deadlocks (script hangs = collective operation issue)

## Timing Output at Extreme Scale

`uw.timing.print_table()` ultimately calls PETSc's `PetscLogView`. At very
high CPU counts (≳1000 ranks), the **ASCII output path** can hang —
typically appearing as a job that completes its computation cleanly but
never exits. The CSV write path uses a different, less collective-heavy
strategy and avoids the issue:

```python
# Default — fine at small scale, can hang at ≳1000 ranks
uw.timing.print_table()
uw.timing.print_table("results.txt")

# Safe at any scale — recommended for HPC runs
uw.timing.print_table("results.csv")
```

The behaviour is in PETSc, not Underworld; choosing CSV at scale is the
recommended workaround. (Issue #134.)

## Summary

**Key Takeaways:**
Expand All @@ -517,5 +537,6 @@ These operations require **ALL ranks** to participate:
3. **Use `with uw.selective_ranks(ranks):`** for serial operations
4. **Collective operations must run on ALL ranks** - never inside rank conditionals
5. **Test with `mpirun -np N`** to catch issues early
6. **At ≳1000 ranks, write timing output as `.csv`** to avoid `PetscLogView` hangs

The parallel safety system makes parallel programming in Underworld3 safer and more intuitive - collective operations are evaluated on all ranks automatically, preventing common deadlock scenarios!
26 changes: 21 additions & 5 deletions src/underworld3/timing.py
Original file line number Diff line number Diff line change
Expand Up @@ -150,20 +150,36 @@ def print_table(filename=None, format="auto"):
----------
filename : str, optional
If provided, write results to file. Extension determines format:
- `.csv` : Spreadsheet-compatible CSV format
- `.txt` or other : Human-readable ASCII table
- ``.csv`` : Spreadsheet-compatible CSV format
- ``.txt`` or other : Human-readable ASCII table
format : str, optional
Override automatic format detection:
- "auto" : Detect from filename (default)
- "ascii" : Human-readable table
- "csv" : Comma-separated values
- ``"auto"`` : Detect from filename (default)
- ``"ascii"`` : Human-readable table
- ``"csv"`` : Comma-separated values

Example
-------
>>> uw.timing.start()
>>> # ... do work ...
>>> uw.timing.print_table() # Print to console
>>> uw.timing.print_table("results.csv") # Save as CSV

Notes
-----
**High-CPU-count usage (≳1000 ranks): prefer CSV output.**

Issue #134 (gthyagi, 2026-04-23): the underlying PETSc ``PetscLogView``
ASCII output path can hang at extreme rank counts on some clusters
(BD-integral routines + ASCII table emit appear to be the trigger),
while the CSV write path uses a different, less collective-heavy
strategy and avoids the issue. If your job is large enough that
timing-output cost matters, write to a ``.csv`` filename:

>>> uw.timing.print_table("results.csv") # safe at any scale
Comment on lines +176 to +179
Comment on lines +173 to +179

The behaviour is in PETSc, not Underworld; choosing CSV at scale is
the recommended workaround.
"""
print_petsc_log(filename=filename, format=format)

Expand Down
Loading