-**Instrument (the user's ask of 2026-09-10: line-level, MPI and kernel times, from the same tools, for both codes).** Two ROCm passes on the same 8-rank, 40-step arm of the matched case (amr-bench/prof_mfc2b.sh, prof_amrex2b.sh): (1) ``rocprofv3 --kernel-trace --stats`` per rank (kernel durations; ``-o kern_%%pid%`` so the ranks do not overwrite one file), (2) ``rocprof-sys-sample --profile --inlines`` at 100 Hz with MPIP, ROCm and sampling on (the MPI call table per rank, whole-run, no loop attribution and no MPI-IO rows; host stacks with ``file:line`` when the binary has line tables; ``.kd`` kernel rows carrying module:routine:line for MFC). Both binaries carry line tables: MFC pin 98c2050f-g (``FFLAGS=-gline-tables-only``), AMReX ``Test_GPU_CNS_Blob_3d-g`` (rebuilt in amrex-ref/build_cns_g with ``-gline-tables-only`` in the CXX and HIP flags; no script under amr-bench records that build). Reader: amr-bench/prof_read.py (one directory, both codes; sums ``PMPI_*`` and ``MPI_*`` rows -- AMReX's C API rows have no ``P`` and the first read wrongly reported its MPI table empty); amr-bench/prof_compare.py prints the two side by side; amr-bench/l136_numbers.py prints every number below from a run tag. Environment: ``OMPI_MCA_btl_vader_single_copy_mechanism=none`` was set for the sampler (it cured an errno-14 storm on the first smoke, but the samples show the data path is UCX -- ``uct_mm``, ``uct_cma``, a trace of ``uct_rocm_ipc`` -- so the variable disabled Open MPI's vader single-copy, not UCX's own CMA transport, which stays on the path) and ``UCX_NET_DEVICES`` unset; the ``--trace`` (perfetto) form hung 7 of 8 ranks and inflated the run 25x at 300 Hz, so the tracing form is not used. Node k004-004 (hold 413788). The first two rounds (lines2, lines3) were taken while an orphaned 8-rank MFC step of that hung tracing smoke (slurm step 413788.79, alive 20:36-21:40, invisible to every local process listing) was still on the node; they are kept as notes/prof_*_lines{2,3}_0910.txt and superseded by the clean round below (lines4, taken with ``squeue -s`` showing no other step; mon.sh now lists remote steps per held job and flags orphans).
0 commit comments