Skip to content

Fix/gpu elecstate pw op spin stride - #7776

Open
chengleizheng wants to merge 4 commits into
deepmodeling:developfrom
chengleizheng:fix/gpu-elecstate-pw-op-spin-stride
Open

Fix/gpu elecstate pw op spin stride#7776
chengleizheng wants to merge 4 commits into
deepmodeling:developfrom
chengleizheng:fix/gpu-elecstate-pw-op-spin-stride

Conversation

@chengleizheng

@chengleizheng chengleizheng commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Reminder

  • I have read AGENTS.md and docs/developers_guide/agent_governance.md.
  • I have linked an issue or explained why this PR does not need one.
  • I have added adequate unit tests and/or case tests, or explained why not.
  • I have listed the exact verification commands run and their results.
  • I have described user-visible behavior changes, including INPUT parameter changes.
  • I have explained core-module impact for ESolver, HSolver, ElecState, Hamilt, Operator, Psi, or other source/ changes.
  • I have requested any needed governance exception below.

Linked Issue

Fix #7773

Case Tests for my changes

  • Commands run:
    • Case test tests/01_PW/007_PW_UPF201_USPP_Fe (PW SCF, USPP, nspin=2, ecutwfc=8, ecutrho=60 — the dense FFT grid is 4× larger per dimension than the smooth grid, which is exactly the configuration that exposed the bug).
    • CPU reference (single process): abacus on 007 → FINAL_ETOT_IS = -673.8349347015964668 eV.
    • GPU (CUDA build, single process): same case → FINAL_ETOT_IS = -673.8349347005279242 eV (GPU kernels active, e.g. PW_Basis_K recip_to_real gpu in timing).
  • Result summary:
    • Post-fix GPU etot agrees with CPU to within 1.1×10⁻⁹ eV (Δ = |−673.8349347005279242 − (−673.8349347015964668)|).

What's changed?

  • Bug: the GPU kernel elecstate_pw in source/source_estate/kernels/cuda/elecstate_op.cu wrote charge density with rho[spin * basis->nrxx + idx], using the wavefunction grid stride (basis->nrxx, the smooth grid). But the rho array is laid out per spin on the density grid (charge->nrxx, the dense grid). For USPP (double grid) the two differ (e.g. 9³ vs 12³ on 007), so the spin-down component was written at a wrong offset and effectively lost → spin-down charge density missing → etot off by 6.07 eV on GPU.
  • Fix: the kernels now take an explicit rho_stride parameter, and every caller passes charge->nrxx:
    • elecstate_op.cu / elecstate_op.h / elecstate_op.cpp: kernel and operator() signatures extended with const int& rho_stride (CPU ignores it; its per-spin pointer indexing was already correct).
    • elecstate_pw.cpp: rho accumulation site (rhoBandK) and kin_r site now pass charge->nrxx.
    • elecstate_pw_cal_tau.cpp: passes charge->nrxx for the kinetic energy density.
    • read_wf2rho_pw.cpp (2 sites) and sto_iter.cpp: pass charge->nrxx.

Governance Notes

  • INPUT/docs changes: none.
  • Core module impact: ElecState GPU path only — the CUDA kernel interface in source_estate/kernels is extended with one parameter; all GPU call sites within this repo are updated in this PR. CPU path behavior is unchanged.
  • Exceptions requested: none.

chengleizheng and others added 2 commits August 4, 2026 23:26
The GPU kernel indexed the spin components of rho with spin*nrxx, where
nrxx is the wavefunction real-space grid size, while rho is allocated
with charge->nrxx (density grid) as the per-spin stride. For USPP tests
using the double grid the two grids differ, so the spin-down density was
written to the wrong offset and effectively lost. Pass an explicit
rho_stride parameter instead.

Fixes 007_PW_UPF201_USPP_Fe GPU SCF etot being off by 6.07 eV.
@chengleizheng chengleizheng added Bugs Bugs that only solvable with sufficient knowledge of DFT GPU & DCU & HPC GPU and DCU and HPC related any issues labels Aug 5, 2026
chengleizheng and others added 2 commits August 5, 2026 09:53
The elecstate_pw_op operator signature gained a rho_stride parameter in
bc21c53, but the unit test still called it with the old signatures,
breaking the test build. Pass this->nrxx, which matches the stride of
the rho layout used in the test.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Bugs Bugs that only solvable with sufficient knowledge of DFT GPU & DCU & HPC GPU and DCU and HPC related any issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GPU PW SCF: spin-down charge density lost due to wrong spin stride in elecstate_pw_op kernel (01_PW/007_PW_UPF201_USPP_Fe etot off by 6.07 eV)

1 participant