Document CSV timing output as the high-rank workaround (#134) - #165
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Documents a practical workaround for uw.timing.print_table() hangs at very high MPI rank counts by recommending CSV timing output, aligning user guidance with PETSc PetscLogView behavior (Issue #134).
Changes:
- Expanded
uw.timing.print_table()docstring with a “Notes” section describing the high-rank CSV workaround. - Added a new “Timing Output at Extreme Scale” section to the parallel computing guide and updated key takeaways.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
src/underworld3/timing.py |
Docstring updates for print_table() describing the CSV-at-scale workaround. |
docs/advanced/parallel-computing.md |
New guidance section and summary bullet recommending .csv timing output at extreme scale. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+176
to
+179
| strategy and avoids the issue. If your job is large enough that | ||
| timing-output cost matters, write to a ``.csv`` filename: | ||
|
|
||
| >>> uw.timing.print_table("results.csv") # safe at any scale |
Comment on lines
+173
to
+179
| ASCII output path can hang at extreme rank counts on some clusters | ||
| (BD-integral routines + ASCII table emit appear to be the trigger), | ||
| while the CSV write path uses a different, less collective-heavy | ||
| strategy and avoids the issue. If your job is large enough that | ||
| timing-output cost matters, write to a ``.csv`` filename: | ||
|
|
||
| >>> uw.timing.print_table("results.csv") # safe at any scale |
| uw.timing.print_table() | ||
| uw.timing.print_table("results.txt") | ||
|
|
||
| # Safe at any scale — recommended for HPC runs |
| 3. **Use `with uw.selective_ranks(ranks):`** for serial operations | ||
| 4. **Collective operations must run on ALL ranks** - never inside rank conditionals | ||
| 5. **Test with `mpirun -np N`** to catch issues early | ||
| 6. **At ≳1000 ranks, write timing output as `.csv`** to avoid `PetscLogView` hangs |
) uw.timing.print_table() ultimately calls PETSc's PetscLogView. At very high CPU counts (≳1000 ranks), the ASCII output path can hang — the job completes its computation cleanly but never exits. The CSV write path uses a different, less collective-heavy strategy and avoids the issue. The behaviour is in PETSc, not Underworld; documenting CSV-at-scale as the recommended workaround rather than working around the PETSc internal in our code (which would diverge from PETSc's own ASCII / CSV separation). Confirmation in #134 from gthyagi: "writing BD integrals to a .txt file can cause the job to hang, whereas using .csv output avoids the problem." Two doc surfaces touched: - uw.timing.print_table docstring — adds a Notes section pointing at #134 and showing the safe call form. - docs/advanced/parallel-computing.md — new "Timing Output at Extreme Scale" section before the Summary, plus a sixth Key-Takeaways bullet. Issue #134 stays open until merge; closing manually after. Underworld development team with AI support from Claude Code
lmoresi
force-pushed
the
docs/print-table-csv-recommendation-134
branch
from
May 4, 2026 11:46
e6b4388 to
79dc253
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Documents the `.csv`-at-scale workaround for `uw.timing.print_table()` hangs at high CPU counts. No code change.
Closes #134.
Why a doc-only change
`uw.timing.print_table()` ultimately calls PETSc's `PetscLogView`. At ≳1000 ranks the ASCII output path can hang while the CSV write path completes cleanly — confirmed by @gthyagi in the issue. The behaviour is in PETSc, not Underworld; documenting CSV-at-scale as the recommended workaround rather than working around the PETSc internal in our own code (which would diverge from PETSc's own ASCII/CSV separation).
Doc surfaces touched
Test plan
Underworld development team with AI support from Claude Code