Skip to content

Home

Jump to bottom
Jose Gracia edited this page Jun 25, 2025 · 29 revisions

Tips & Tricks

Git workflows

Remote repo naming convention

We are dealing with two remote repositories: SPMT/spmt-lbc.git and your own fork hpcXXX/spmt-lbc.git. For the purpose of this exercise, let us use the following naming convention:

  1. SPMT/spmt-lbc: spmt
  2. hpcXXXX/spmt-lbc.git: hpcXXXX

The default name of the remote repo from which you cloned is origin. Assuming that you just cloned you repo onto Hawk, it will look like this

# git clone git@code.hlrs.de:hpcjgrac/spmt-lbc.git
git remote -v
origin        git@code.hlrs.de:hpcjgrac/spmt-lbc.git (fetch)                 
origin        git@code.hlrs.de:hpcjgrac/spmt-lbc.git (push)

Rename origin to hpcXXX

git remote rename origin hpcjgrac
hpcjgrac        git@code.hlrs.de:hpcjgrac/spmt-lbc.git (fetch)               
hpcjgrac        git@code.hlrs.de:hpcjgrac/spmt-lbc.git (push)

Now, add a second remote labelled spmt for SPMT/spmt-lbc.git

git remote add spmt git@code.hlrs.de:SPMT/spmt-lbc.git
git remote -v 
hpcjgrac        git@code.hlrs.de:hpcjgrac/spmt-lbc.git (fetch)                 
hpcjgrac        git@code.hlrs.de:hpcjgrac/spmt-lbc.git (push)                  
spmt    git@code.hlrs.de:SPMT/spmt-lbc.git (fetch)                             
spmt    git@code.hlrs.de:SPMT/spmt-lbc.git (push)

Synchronize repos

Assume somebody else has done work on the branch spmt/lbc.openmp which you would like to have in your repo as well. We will do this on the command line.

Start by checking out your local branch lbc.openmp (this important):

git checkout lbc.openmp
git tree     # HEAD points to lbc.openmp

See if there is any recent changes on repo spmt:

git fetch spmt
git tree    # spmt/lbc.openmp and lbc.openmp are probably not pointing to the same commit

Pull spmt/lbc.openmp and rebase your changes (if any) on top of it

git pull --rebase
git tree   # spmt/lbc.openmp should point to lbc.openmp or be a ancestor of lbc.openmp
*   a69a64d (HEAD -> lbc.openmp, spmt/lbc.openmp, hpcjgrac/lbc.openmp, hpcjgrac/HEAD) Merge pull request 'Add scorep support to Makefile' (#2) from hpcbison/spmt-lbc:scorep into lbc.openmp

Running the code

Hawk

Hunter

Without offloading:

qsub -I -l select=1:node_type=mi300a:mpiprocs=4:ompthreads=24 -l walltime=1200
NCORES=$OMP_NUM_THREADS

# vanilla
mpiexec -n 4 -ppn 4 ./lbc
Total density:  532686838.951803 at timestep     10   
   Performance [MLUPs]               54.525 MLUPs

# add depth aka cores per process
OMP_NUM_THREADS=24 mpiexec -n 4 -ppn 4 --depth=$NCORES ./lbc
Total density:  532686838.951803 at timestep     10
   Performance [MLUPs]              121.611 MLUPs

# pin threads to cores (!!)
OMP_NUM_THREADS=24 mpiexec -n 4 -ppn 4 --depth=$NCORES --cpu-bind depth ./lbc
Total density:  532686838.951803 at timestep     10
   Performance [MLUPs]              639.157 MLUPs

# reduce OMP threads to utilize BW
OMP_NUM_THREADS=12 mpiexec -n 4 -ppn 4 --depth=$NCORES --cpu-bind depth ./lbc
Total density:  532686838.951803 at timestep     10
   Performance [MLUPs]              989.920 MLUPs

# alternative pinning using hyperthreads
OMP_NUM_THREADS=24 mpiexec -n 4 -ppn 4 --depth=$NCORES --cpu-bind list:0-23,96-119:24-47,120-143:48-71,144-167:72-95,168-191 ./lbc
Total density:  532686838.951803 at timestep     10
   Performance [MLUPs]              629.553 MLUPs

# bind mem
OMP_NUM_THREADS=12 mpiexec -n 4 -ppn 4 --depth=$NCORES --cpu-bind depth --mem-bind local ./lbc
Total density:  532686838.951803 at timestep     10
   Performance [MLUPs]              983.532 MLUPs

# OMP target offloading problem size 480x480x480 per process
OMP_NUM_THREADS=12 mpiexec -n 4 -ppn 4 --depth=$NCORES --cpu-bind depth --gpu-bind per_task:1 ./lbc
Total density:  438691190.957138 at timestep     10
   Performance [MLUPs]             1625.935 MLUPs

DLB/TALP:

NCORES=$OMP_NUM_THREADS

module load dlb
export DLB_TALP_ARGS="--talp --ompt --talp-papi=yes --talp-openmp=yes --talp-output-file=talp-performance-report --talp-summary=pop-metrics"
DLB_TALP_LIB=libdlb_mpi.so

OMP_NUM_THREADS=12 mpiexec -n 4 -ppn 4 --depth=$NCORES --cpu-bind depth env DLB_ARGS="$DLB_TALP_ARGS" LD_PRELOAD=$DLB_TALP_LIB ./lbc

ROCm debug agent:

NCORES=24; HSA_TOOLS_LIB=$ROCM_PATH/lib/librocm-debug-agent.so.2; CRAY_ACC_DEBUG=2 OMP_NUM_THREADS=12 stdbuf -o0 mpiexec -n 4 -ppn 4 --depth=$NCORES --cpu-bind depth ./lbc

Extrae:

NCORES=$OMP_NUM_THREADS

module load extrae
# cp -a $HLRS_EXTRAE_ROOT/share/example/MPI/extrae.xml .
export EXTRAE_CONFIG_FILE=extrae.xml
EXTRAE_LIB=libompitracecf.so

OMP_NUM_THREADS=12 mpiexec -n 4 -ppn 4 --depth=$NCORES --cpu-bind depth env EXTRAE_CONFIG_FILE="$EXTRAE_CONFIG_FILE" LD_PRELOAD=$EXTRAE_LIB ./lbc

Score-P/Scalasca:

NCORES=$OMP_NUM_THREADS

module load scorep
make

OMP_NUM_THREADS=12 mpiexec -n 4 -ppn 4 --depth=$NCORES --cpu-bind depth ./lbc
scorep-score -r profile.cubex > scorep.score

module load scalasca

OMP_NUM_THREADS=12 scalasca -analyze -s mpiexec -n 4 "-ppn 4 --depth=$NCORES --cpu-bind depth" ./lbc
scalasca -examine -s scorep_lbc...sum

OMP_NUM_THREADS=12 scalasca -analyze -t mpiexec -n 4 "-ppn 4 --depth=$NCORES --cpu-bind depth" ./lbc
scalasca -examine -s scorep_lbc...trace

OTF-CPT:

NCORES=$OMP_NUM_THREADS

module load otf-cpt 

export OTFCPT_OPTIONS="verbose=1 log_path=otf-cpt-verbose"
OPTCPT_LIB=libOTFCPT.so

OMP_NUM_THREADS=12 mpiexec -n 4 -ppn 4 --depth=$NCORES --cpu-bind depth env OMP_TOOL_LIBRARIES=$OTPCPT_LIB ./lbc

Perftools-lite:

NCORES=$OMP_NUM_THREADS

module load perftools-lite-gpu

make clean && make && mv lbc lbc-lite-gpu

# increase number of timesteps to 100 at least; first timestep takes a lot of time (in memory transfer?)
OMP_NUM_THREADS=12 mpiexec -n 4 -ppn 4 --depth=$NCORES --cpu-bind depth --gpu-bind per_task:1 ./lbc-lite-gpu

pat_report lbc-lite-gpu+.../
pat_report -O accelerator,accpc,acc_time lbc-lite-gpu+.../

Perftools with pat_build:

NCORES=$OMP_NUM_THREADS

module load perftools

make clean && make
pat_build -g hip,hsa,mpi,omp -o lbc+pat lbc

OMP_NUM_THREADS=12 mpiexec -n 4 -ppn 4 --depth=$NCORES --cpu-bind depth --gpu-bind per_task:1 ./lbc+pat

# increase number of timesteps to 100 at least; first timestep takes a lot of time (in memory transfer?)
pat_report lbc+pat.../
pat_report -O accelerator,accpc,acc_time,acc_kern_stats lbc+pat.../

Omnitrace: start here

NCORES=$OMP_NUM_THREADS

module load omnitrace

omnitrace-avail -G ./omnitrace.cfg  # edit
export OMNITRACE_CONFIG_FILE=$PWD/omnitrace.cfg
omnitrace-instrument -I ".*" -o lbc-apu-omnitrace -- lbc-apu

OMP_NUM_THREADS=12 mpiexec -n 4 -ppn 4 --depth=$NCORES --cpu-bind depth --gpu-bind per_task:1 omnitrace-run ./lbc-apu-omnitrace

Rocprofv3:

NCORES=$OMP_NUM_THREADS

OMP_NUM_THREADS=12 mpiexec -n 4 -ppn 4 --depth=$NCORES --cpu-bind depth --gpu-bind per_task:1 rocprofv3 -d rocprof-output --hip-trace  --hsa-trace  -- ./lbc-apu

Clone this wiki locally