ci: make ccache effective and observable, shallow checkout, add cache… - #7756
ci: make ccache effective and observable, shallow checkout, add cache…#7756MrLi000001 wants to merge 2 commits into
Conversation
… warmer Measured on the self-hosted runners, the Build step of the same workflow ranges from 2.7 min (warm ccache) to 37.5 min (stale ccache), and a full-history checkout took ~12 min on cache-cold runners. - Set ccache --max-size=30G (default ~10G is tight for -O3 -g builds) and print ccache -s before/after the build so hit rates are visible in every run log (if: always()). - fetch-depth: 0 -> 1; CMake only runs 'git log -1' for commit info. - Add cache_warmer.yml: on every push to develop, build (without tests) on both the X64 and gpu pools so /tmp/ccache stays primed with the latest develop. This prevents the first PR runs after a big refactor merge from all paying a full rebuild.
|
The shallow checkout and ccache statistics are useful, but I suggest splitting out the cache warmer. It is not yet clear whether /tmp/ccache persists across jobs and nodes in the dynamic runner pool, so one warmer may only warm a single node while consuming an additional GPU. It also builds the default seven CUDA architectures, while #7753 uses sm_70, so those CUDA cache entries will not be effectively reused. |
|
A few concerns on the 1. The configure flags are copy-pasted and already out of sync. The header comment says to keep them aligned with 2. 3. Neither warmer job sets Given 2 and 3, I'd suggest splitting as @Stardust0831 proposed: land the |
The previous commit (167904f) tried to make the workflow adapt to whatever GPU the runner has by auto-detecting compute capability at CMake configure time. Reviewers (Stardust0831 + 张笑扬) correctly flagged two independent defects and a deeper design issue. Defects in the previous commit: 1. if(COMMAND nvidia-smi) tests for a CMake command, not an executable on PATH. It was always false, so execute_process never ran. The correct check is find_program(NVIDIA_SMI_EXECUTABLE nvidia-smi). 2. The detection block and the historical default list were both inside the same if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES) outer guard. The historical default uses plain set() which shadows the cache entry; all 7 archs were appended regardless of detection. 3. AND USE_CUDA inside the detection block was redundant; the surrounding if(USE_CUDA) at line 419 already guards it. Design issue: Auto-detection is the wrong default for a cluster codebase. ABACUS is configured on whichever host runs cmake (often a login node on an HPC system) but the resulting binary may run on a different compute node. Silent auto-detection produces a binary that fails at run time with 'no kernel image is available for execution on the device' and no warning at configure time. HPC convention is to build on the compute node, where the workflow's hardcoded -D matches the hardware. Furthermore, heterogeneous auto-detection fragments the ccache key per node, which defeats cache_warmer (deepmodeling#7756). An explicit uniform pin across CI is exactly what cache_warmer relies on. This commit: - Reverts the CMakeLists.txt auto-detect block. - Restores -DCMAKE_CUDA_ARCHITECTURES=70 in cuda.yml, with a comment noting that the value assumes a homogeneous V100 pool and should be updated if the pool changes. - Keeps -j $(nproc), which is the real win in e614a28 (~2x parallelism). Note on the measured 33 min -> 18 min result: the reviewer is correct that the savings probably come almost entirely from -j4 -> -j $(nproc), not from the arch cut (7 -> 1 arch). With nvcc being fast and C++ TUs dominating build time, doubling the build parallelism is enough to halve the wall time; the arch reduction's contribution, if any, is small. The arch pin still avoids fatbin bloat and uniform ccache keys, but the headline number in the PR body should not over-claim it.
) * ci(cuda): build only for CI runner GPU arch and raise parallelism The CUDA CI built every .cu file for 7 GPU architectures (60/70/75/80/86/89/90) with a hardcoded -j4, so the Configure & Build step took ~33 min even with a warm ccache. - Pin CMAKE_CUDA_ARCHITECTURES=70: the CI GPU pool is Tesla V100 (sm_70, per nvidia-smi in the run logs and the '16V100' Slurm partition in .ci/slurm/config.ini). This cuts nvcc work by ~7x. - Build with -j $(nproc) instead of -j4; with the arch list reduced, the higher parallelism is memory-safe. Expected: Configure & Build ~33 min -> ~10 min on a cache-cold run. * ci: auto-detect GPU arch via nvidia-smi at CMake configure time Sister commit to e614a28. The previous commit hardcoded -DCMAKE_CUDA_ARCHITECTURES=70 assuming the CI pool is Tesla V100. Reviewers (Stardust0831 + chenmohan) pointed out this couples the workflow to a specific GPU model and breaks if the runner pool is heterogeneous or upgraded. This commit moves the arch selection into CMakeLists.txt: - When CMAKE_CUDA_ARCHITECTURES is unset and nvidia-smi is available, query --query-gpu=compute_cap and set the arch from the result. Map '7.0' -> 70, '8.0' -> 80, '8.9' -> 89, '9.0' -> 90, etc. - Falls back to the historical multi-arch default if nvidia-smi is not present (CPU-only build host) or returns an unrecognized value. - User-provided CMAKE_CUDA_ARCHITECTURES still takes precedence. Workflow file: removed the -DCMAKE_CUDA_ARCHITECTURES=70 line; the Configure step now relies on the CMake-side detection. Verified the plain -DCMAKE_CUDA_ARCHITECTURES=70 still produces a 38 min cold Build on the existing V100 runner. * ci(cuda): revert broken auto-detect; use explicit sm_70 pin The previous commit (167904f) tried to make the workflow adapt to whatever GPU the runner has by auto-detecting compute capability at CMake configure time. Reviewers (Stardust0831 + 张笑扬) correctly flagged two independent defects and a deeper design issue. Defects in the previous commit: 1. if(COMMAND nvidia-smi) tests for a CMake command, not an executable on PATH. It was always false, so execute_process never ran. The correct check is find_program(NVIDIA_SMI_EXECUTABLE nvidia-smi). 2. The detection block and the historical default list were both inside the same if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES) outer guard. The historical default uses plain set() which shadows the cache entry; all 7 archs were appended regardless of detection. 3. AND USE_CUDA inside the detection block was redundant; the surrounding if(USE_CUDA) at line 419 already guards it. Design issue: Auto-detection is the wrong default for a cluster codebase. ABACUS is configured on whichever host runs cmake (often a login node on an HPC system) but the resulting binary may run on a different compute node. Silent auto-detection produces a binary that fails at run time with 'no kernel image is available for execution on the device' and no warning at configure time. HPC convention is to build on the compute node, where the workflow's hardcoded -D matches the hardware. Furthermore, heterogeneous auto-detection fragments the ccache key per node, which defeats cache_warmer (#7756). An explicit uniform pin across CI is exactly what cache_warmer relies on. This commit: - Reverts the CMakeLists.txt auto-detect block. - Restores -DCMAKE_CUDA_ARCHITECTURES=70 in cuda.yml, with a comment noting that the value assumes a homogeneous V100 pool and should be updated if the pool changes. - Keeps -j $(nproc), which is the real win in e614a28 (~2x parallelism). Note on the measured 33 min -> 18 min result: the reviewer is correct that the savings probably come almost entirely from -j4 -> -j $(nproc), not from the arch cut (7 -> 1 arch). With nvcc being fast and C++ TUs dominating build time, doubling the build parallelism is enough to halve the wall time; the arch reduction's contribution, if any, is small. The arch pin still avoids fatbin bloat and uniform ccache keys, but the headline number in the PR body should not over-claim it. * Clean up comments in CUDA workflow Removed comments explaining the CUDA architecture pinning process.
Measured on the self-hosted runners, the Build step of the same workflow ranges from 2.7 min (warm ccache) to 37.5 min (stale ccache), and a full-history checkout took ~12 min on cache-cold runners.
Note: the cache warmer was split out of this PR per review (Stardust0831 / Critsium-xy) - /tmp/ccache persistence across the dynamic runner pool is still unverified, and the warmer's CUDA build flags were out of sync. It will return as its own PR (with timeout-minutes and matching build flags) once the persistence question is answered.
Reminder
AGENTS.mdanddocs/developers_guide/agent_governance.md.source/changes.Linked Issue
Fix #
Unit Tests and/or Case Tests for my changes
What's changed?
Governance Notes