Investigated with Claude Code.
What happens
A static sphere at Re 100, Ma 0.1, in uniform flow along x, returns a transverse force larger than its drag on a flow whose wake is axisymmetric. Grid 405×270×270, 64 GPU ranks, body at the center of the domain:
Fx +0.4040 Fy +0.4364 Fz +0.4364
Fy and Fz agree to five digits, and the torques mirror them (Ty −0.028511, Tz +0.028507), which is what a spatially uniform spurious force density in y and z produces through Σ r × F.
It is not a flow quantity
Fy is 0.4367 at time step 1, on a field that is still essentially uniform, and 0.4364 at step 49,793, while Fx relaxes from 1.638 to 0.4040 over those same 50,000 steps. Constant to four significant figures while the flow develops completely.
Where it comes from
Restarting the same field, with the same binary, changing only the rank count:
| ranks |
Fx |
Fy |
Fz |
| 64 |
+0.404403 |
+0.436447 |
+0.436433 |
| 1 |
+0.404403 |
+0.000024 |
+0.000010 |
Fx is identical to six digits. The transverse force collapses by four orders of magnitude. s_communicate_ib_forces returns immediately when num_procs == 1, so the single-rank path skips the reduction entirely — as does every multi-rank effect on the ghost-cell fill.
Independently, reproducing s_compute_ib_forces' volume sum offline from the restart file — same fd_order = 4 stencil, same body mask, pressure and viscous terms — gives Fx +0.40459 against MFC's +0.40399 (0.15 %) and Fy +0.00002. So the per-cell sum itself is right and carries no transverse force; the spurious part enters somewhere between that sum and what lands in patch_ib%force.
That leaves s_communicate_ib_forces and the multi-rank ghost-cell fill as the candidates. The reduction is a hand-rolled dimension-by-dimension prefix sum with a hop count capped at min(2*ib_neighborhood_radius, num_procs_<dir> - 1), which truncates whenever a direction carries more than 2*ib_neighborhood_radius + 1 ranks.
Why it has not been caught
Every immersed-boundary golden file is single-rank, where s_communicate_ib_forces returns on its first line. The force is diagnostic for a static body, so a wrong value is silent; for moving_ibm it drives the motion.
Reproducing
Any static, symmetric body on more than one rank, positioned so it spans a subdomain boundary, with the force written per step (ib_state_wrt). The symmetric body is what makes it visible: the true transverse force is zero, so anything non-zero is the defect. examples/2D_ibm_force_decomposition (on #1859) is the 2D version of that setup and runs in seconds; it shows the smaller, related decomposition dependence in drag that #1859 fixes, which is a different bug — rebuilding with that fix leaves the sphere's 0.4364 unchanged.
Not the same as #1860
#1860 is an out-of-bounds fd_coeff read that makes the force depend on the decomposition at the 0.5 % level. It is real and fixed by #1859, and it is not this. Verified by rebuilding with that fix and re-running the sphere from its own restart: Fy stays at 0.4364.
Investigated with Claude Code.
What happens
A static sphere at Re 100, Ma 0.1, in uniform flow along x, returns a transverse force larger than its drag on a flow whose wake is axisymmetric. Grid 405×270×270, 64 GPU ranks, body at the center of the domain:
Fy and Fz agree to five digits, and the torques mirror them (Ty −0.028511, Tz +0.028507), which is what a spatially uniform spurious force density in y and z produces through Σ r × F.
It is not a flow quantity
Fy is 0.4367 at time step 1, on a field that is still essentially uniform, and 0.4364 at step 49,793, while Fx relaxes from 1.638 to 0.4040 over those same 50,000 steps. Constant to four significant figures while the flow develops completely.
Where it comes from
Restarting the same field, with the same binary, changing only the rank count:
Fx is identical to six digits. The transverse force collapses by four orders of magnitude.
s_communicate_ib_forcesreturns immediately whennum_procs == 1, so the single-rank path skips the reduction entirely — as does every multi-rank effect on the ghost-cell fill.Independently, reproducing
s_compute_ib_forces' volume sum offline from the restart file — samefd_order = 4stencil, same body mask, pressure and viscous terms — gives Fx +0.40459 against MFC's +0.40399 (0.15 %) and Fy +0.00002. So the per-cell sum itself is right and carries no transverse force; the spurious part enters somewhere between that sum and what lands inpatch_ib%force.That leaves
s_communicate_ib_forcesand the multi-rank ghost-cell fill as the candidates. The reduction is a hand-rolled dimension-by-dimension prefix sum with a hop count capped atmin(2*ib_neighborhood_radius, num_procs_<dir> - 1), which truncates whenever a direction carries more than2*ib_neighborhood_radius + 1ranks.Why it has not been caught
Every immersed-boundary golden file is single-rank, where
s_communicate_ib_forcesreturns on its first line. The force is diagnostic for a static body, so a wrong value is silent; formoving_ibmit drives the motion.Reproducing
Any static, symmetric body on more than one rank, positioned so it spans a subdomain boundary, with the force written per step (
ib_state_wrt). The symmetric body is what makes it visible: the true transverse force is zero, so anything non-zero is the defect.examples/2D_ibm_force_decomposition(on #1859) is the 2D version of that setup and runs in seconds; it shows the smaller, related decomposition dependence in drag that #1859 fixes, which is a different bug — rebuilding with that fix leaves the sphere's 0.4364 unchanged.Not the same as #1860
#1860 is an out-of-bounds
fd_coeffread that makes the force depend on the decomposition at the 0.5 % level. It is real and fixed by #1859, and it is not this. Verified by rebuilding with that fix and re-running the sphere from its own restart: Fy stays at 0.4364.