NavierStokesSLCN Gives Error while running with HPC/API

[Chawgai]

Description
A failure occurs when executing the test debug script nstt.py in a distributed environment using mpirun. The issue is reproducible across different operating systems and clusters (macOS and the Kaiju cluster), suggesting the bug is likely related to the code's interaction with the MPI library rather than a local environment quirk. The test script is attached herewith and t

nstt.txt

nstt.txt

he error log is as under,

(underworld3:dev) (base) u808@its-0842 nst % mpirun -n 2 python nstt.py
[Rank 0] Starting with 2 processes
[Rank 0] Creating mesh...
[Rank 1] Starting with 2 processes
[Rank 1] Creating mesh...
[Rank 1] Mesh created
[Rank 0] Mesh created
[Rank 0] Barrier 1 passed
[Rank 0] Creating variables...
[Rank 1] Barrier 1 passed
[Rank 1] Creating variables...
[Rank 0] Variables created
[Rank 0] Initializing variables...
[Rank 1] Variables created
[Rank 1] Initializing variables...
[Rank 0] Variables initialized
[Rank 0] Barrier 2 passed
[Rank 1] Variables initialized
[Rank 1] Barrier 2 passed
[Rank 1] Creating Navier-Stokes solver...
[Rank 0] Creating Navier-Stokes solver...
[Rank 1] Navier-Stokes solver object created
[Rank 0] Navier-Stokes solver object created
[Rank 0] Constitutive model set
[Rank 0] Viscosity set
[Rank 1] Constitutive model set
[Rank 1] Viscosity set
[Rank 0] Body force set
[Rank 0] Adding boundary conditions...
[Rank 1] Body force set
[Rank 1] Adding boundary conditions...
[Rank 1] Boundary conditions added
[Rank 1] Setting solver options...
[Rank 1] Solver options set
[Rank 1] Calculating timestep...
[Rank 0] Boundary conditions added
[Rank 0] Setting solver options...
[Rank 0] Solver options set
[Rank 0] Calculating timestep...
[Rank 0] dt = 2.7829e-04, delta_x = 0.023592
[Rank 1] dt = 2.7829e-04, delta_x = 0.023592
[Rank 1] Barrier 3 passed - ready to solve
[Rank 1] Starting solver...
[Rank 0] Barrier 3 passed - ready to solve
[Rank 0] Starting solver...
[Rank 1] Solver failed: could not broadcast input array from shape (2471,1,1) into shape (2497,1,1)
[Rank 0] Solver failed: could not broadcast input array from shape (2535,1,1) into shape (2509,1,1)
[Rank 1] Traceback (most recent call last):
File "/Users/u8085704/Library/CloudStorage/OneDrive-Personal/uw3d/nst/nstt.py", line 127, in
navier_stokes.solve(timestep=delta_t, zero_init_guess=True)
File "/Users/u8085704/underworld3/.pixi/envs/dev/lib/python3.12/site-packages/underworld3/timing.py", line 310, in timed
result = routine(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/u8085704/underworld3/.pixi/envs/dev/lib/python3.12/site-packages/underworld3/systems/solvers.py", line 3106, in solve
self.DuDt.update_pre_solve(timestep, verbose=verbose, evalf=_evalf)
File "/Users/u8085704/underworld3/.pixi/envs/dev/lib/python3.12/site-packages/underworld3/systems/ddt.py", line 1108, in update_pre_solve
v_mid_result = uw.function.global_evaluate(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/u8085704/underworld3/.pixi/envs/dev/lib/python3.12/site-packages/underworld3/timing.py", line 310, in timed
result = routine(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/u8085704/underworld3/.pixi/envs/dev/lib/python3.12/site-packages/underworld3/function/functions_unit_system.py", line 537, in global_evaluate
raw_result = _global_evaluate_nd(
^^^^^^^^^^^^^^^^^^^^
File "src/underworld3/function/_function.pyx", line 477, in underworld3.function._function.global_evaluate_nd
File "/Users/u8085704/underworld3/.pixi/envs/dev/lib/python3.12/site-packages/underworld3/swarm.py", line 642, in setitem
modified_data[key] = value
~~~~~~~~~~~~~^^^^^
File "/Users/u8085704/underworld3/.pixi/envs/dev/lib/python3.12/site-packages/underworld3/utilities/nd_array_callback.py", line 674, in setitem
super().setitem(key, actual_value)
ValueError: could not broadcast input array from shape (2471,1,1) into shape (2497,1,1)

[Rank 0] Traceback (most recent call last):
File "/Users/u8085704/Library/CloudStorage/OneDrive-Personal/uw3d/nst/nstt.py", line 127, in
navier_stokes.solve(timestep=delta_t, zero_init_guess=True)
File "/Users/u8085704/underworld3/.pixi/envs/dev/lib/python3.12/site-packages/underworld3/timing.py", line 310, in timed
result = routine(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/u8085704/underworld3/.pixi/envs/dev/lib/python3.12/site-packages/underworld3/systems/solvers.py", line 3106, in solve
self.DuDt.update_pre_solve(timestep, verbose=verbose, evalf=_evalf)
File "/Users/u8085704/underworld3/.pixi/envs/dev/lib/python3.12/site-packages/underworld3/systems/ddt.py", line 1108, in update_pre_solve
v_mid_result = uw.function.global_evaluate(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/u8085704/underworld3/.pixi/envs/dev/lib/python3.12/site-packages/underworld3/timing.py", line 310, in timed
result = routine(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/u8085704/underworld3/.pixi/envs/dev/lib/python3.12/site-packages/underworld3/function/functions_unit_system.py", line 537, in global_evaluate
raw_result = _global_evaluate_nd(
^^^^^^^^^^^^^^^^^^^^
File "src/underworld3/function/_function.pyx", line 477, in underworld3.function._function.global_evaluate_nd
File "/Users/u8085704/underworld3/.pixi/envs/dev/lib/python3.12/site-packages/underworld3/swarm.py", line 642, in setitem
modified_data[key] = value
~~~~~~~~~~~~~^^^^^
File "/Users/u8085704/underworld3/.pixi/envs/dev/lib/python3.12/site-packages/underworld3/utilities/nd_array_callback.py", line 674, in setitem
super().setitem(key, actual_value)
ValueError: could not broadcast input array from shape (2535,1,1) into shape (2509,1,1)

[Rank 0] Barrier 4 passed
[Rank 0] Test completed
[Rank 1] Barrier 4 passed
[Rank 1] Test completed
(underworld3:dev) (base) u8085@its-082 nst %

[lmoresi]

Fixed in PR #65 (merged to development).

Root cause: SwarmVariable._canonical_data caches were only invalidated inside the delete_lost_points branch of Swarm.migrate(). When particles moved between ranks without deletion (the common case in global_evaluate_nd), caches became stale — storing the wrong particle count and values. This caused the shape mismatch in global_evaluate when used by NavierStokesSLCN/DDt solvers in parallel.

Two fixes applied:

1. swarm.py: cache invalidation now runs unconditionally after every Swarm.migrate()
2. _function.pyx: added cache invalidation after the bare-bones dm.migrate() in global_evaluate_nd (which bypasses Swarm.migrate() for the return trip)

Parallel regression test added: tests/parallel/test_0760_swarm_cache_migration.py