Skip to content

JIT: two constants of the same name compiled to one constants[] slot - #717

Merged
lmoresi merged 3 commits into
developmentfrom
bugfix/jit-constant-collision
Sep 11, 2026
Merged

JIT: two constants of the same name compiled to one constants[] slot#717
lmoresi merged 3 commits into
developmentfrom
bugfix/jit-constant-collision

Conversation

@lmoresi

@lmoresi lmoresi commented Sep 10, 2026

Copy link
Copy Markdown
Member

Two constants that share a display name were compiled as one constants[] slot, so any solve holding two of them silently used a single value. Every ViscousFlowModel names its viscosity \eta, so this is not a niche case.

The symptom

A layered viscosity built from two ViscousFlowModels returned the uniform-viscosity answer:

L2 vs the exact layered Couette profile 2.819e-1
L2 vs plain linear shear 1.502e-10

The contrast was entirely absent, and nothing upstream looked wrong. unwrap(model.flux[0,0]) gave 2.0*M0*v_0,0 + 2000.0*M1*v_0,0, the level sets held 0..1, the constituent models were distinct objects, and constants_manifest held both values — (0, \eta = 1.0) and (1, \eta = 1000.0). The expression and the manifest were correct. Only the emitted C was wrong.

The cause

_JITConstant (utilities/_jitextension.py) is a plain sympy.Symbol subclass. It never adopted the disambiguation UWexpression uses (docs/developer/design/SYMBOL_DISAMBIGUATION_2025-12.md), and it had neither half of it: no _hashable_content discriminator, and it constructed through Symbol.__new__, which is cached by name.

a, b = _JITConstant(0, name='same'), _JITConstant(1, name='same')
a is b                    # True   <- the same object
a._ccodestr, b._ccodestr  # constants[1], constants[1]

The second placeholder was the first object, and assigning its _ccodestr overwrote the first one's. Every occurrence then rendered as the same slot, so a two-material blend assembled as (phi_0 + phi_1) * constants[k] — one uniform viscosity.

The fix

Follow the same mechanism UWexpression uses: construct via Symbol.__xnew__ to bypass the instance cache, and put the slot index in _hashable_content, so identity is the slot rather than the name. Also __slots__ and __getnewargs_ex__ to match.

Separately, _extract_constants sorted the manifest with a tie-break on str(expr), which for a UWexpression is its current value — so two same-named constants could swap slots when a parameter changed. It now tie-breaks on instance_number, which is creation order and identical on every rank running the same script.

Verification

The composed-model route and the equivalent single-model createMask blend now agree to 2.5e-12 on the same problem (both 8.010e-2 against the exact layered profile, which is the nodal level set's own error, not the solver's), and neither is linear shear.

Two regression tests in tests/test_0103_jit_rampable_constants.py, one at the _extract_constants level and one end-to-end. Both were confirmed to fail with the fix reverted and pass with it.

pytest -m "level_1 and tier_a": see the check run below.

Note for the reviewer

This commit also exists on feature/particle-demos (PR #715), where it was found. The content is identical, so the branches merge cleanly — except tests/test_0103_jit_rampable_constants.py, where the feature branch's end-to-end test uses the MaterialSwarm API introduced there. Take the feature branch's version of that file when the two meet.

Underworld development team with AI support from Claude Code

🤖 Generated with Claude Code

https://claude.ai/code/session_01MSGAFeA7qYXgkuw9ud8F2G

A layered viscosity built from two ViscousFlowModels returned the
UNIFORM-viscosity answer: L2 2.819e-1 against the exact layered Couette
profile, and 1.502e-10 against plain linear shear. The composed flux was
correct and so was the constants manifest ((0, \eta = 1.0),
(1, \eta = 1000.0)); only the emitted C was wrong.

_JITConstant is a plain sympy.Symbol subclass that never adopted the
disambiguation UWexpression uses (docs/developer/design/
SYMBOL_DISAMBIGUATION_2025-12.md), and it has neither half of it. It
constructed through Symbol.__new__, which is cached by NAME, so the
second placeholder was literally the first object and assigning its
_ccodestr overwrote the first one's:

    a, b = _JITConstant(0, name='same'), _JITConstant(1, name='same')
    a is b                    -> True
    a._ccodestr, b._ccodestr  -> constants[1], constants[1]

Every occurrence then rendered as one constants[] slot, so the blend
collapsed to (phi_0 + phi_1) * constants[k]. Every ViscousFlowModel
calls its viscosity \eta, so any model with two of them was affected,
not just multi-material.

Construct via Symbol.__xnew__ to bypass the cache and put the slot index
in _hashable_content, so identity is the slot rather than the name. The
same solve now gives 1.807e-7 against the layered exact solution.

Also: the manifest sort tie-broke on str(expr), which for a UWexpression
is its current VALUE, so two same-named constants could swap slots when
a parameter changed. Tie-break on instance_number instead.

Regression tests at both levels in test_0103_jit_rampable_constants.py.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSGAFeA7qYXgkuw9ud8F2G

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

_extract_constants()’s new sort key can still raise a TypeError when same-named constants mix instance_number=None and integer instance numbers, so the ordering needs a small normalization fix before merge.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Fixes a silent correctness bug in the JIT constant plumbing where two distinct constants sharing the same display name could collapse to a single constants[] slot in generated C, producing incorrect physics (notably multi-material viscosity blends). The PR also stabilizes constant-slot assignment ordering and adds regression coverage to prevent reintroductions.

Changes:

  • Updates _JITConstant to bypass SymPy’s name-based symbol cache (__xnew__) and to disambiguate identity via _hashable_content() using the constants-slot index.
  • Adjusts _extract_constants() ordering so same-named constants don’t swap slots when values change.
  • Adds two regression tests covering both placeholder extraction and an end-to-end two-material Stokes solve.
File summaries
File Description
src/underworld3/utilities/_jitextension.py Fixes _JITConstant identity and makes constant-slot assignment deterministic for same-named constants.
tests/test_0103_jit_rampable_constants.py Adds regression tests for same-named constant placeholders and an end-to-end multi-material layered-viscosity solve.
src/underworld3/constitutive_models.py Adds an inline note explaining the historical symptom and pointing to the JIT fix/regression test.
Review details
  • Files reviewed: 3/3 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +430 to +433
sorted_constants = sorted(
constant_exprs,
key=lambda e: (e.name, getattr(e, "instance_number", -1), _stable_sort_key(e)),
)
Comment thread src/underworld3/constitutive_models.py Outdated
Comment on lines +4521 to +4529
# NB (2026-09-09): this class used to return the UNIFORM-viscosity
# answer (layered Couette L2 2.8e-1; identical to plain linear shear to
# 1.5e-10) while its composed flux and the constants manifest both
# looked correct. The cause was in the JIT, not here: every constituent
# names its viscosity \eta, and the constants[] placeholder was named
# for the expression alone, so sympy treated two slots as one symbol
# and the blend collapsed to a single viscosity. Fixed in
# utilities/_jitextension.py::_extract_constants; regression in
# tests/test_0103_jit_rampable_constants.py.
lmoresi and others added 2 commits September 10, 2026 11:35
The manifest sort key called instance_number directly. That is an int for
every UWexpression built today, because uw_object.__init__ assigns it — so
the TypeError Copilot describes is not reachable at present. But `_uw_id =
None` is a state UWexpression deliberately supports (its _hashable_content
branches on it), so a set mixing None with an int would sort-crash the moment
that state became reachable. Normalise instead of relying on the current
initialisation order.

The NB block in MultiMaterialConstitutiveModel narrated the whole incident —
debug numbers, root cause, file paths — in a hot runtime class. The style
charter asks for intent and constraints, not history, and the history already
lives in the commit message and the PR. Reduced to the constraint that
matters to a reader of that class: constituents that share a parameter name
depend on _JITConstant keeping its slots distinct, and here is the test.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSGAFeA7qYXgkuw9ud8F2G
The first fix put the constants[] slot index in the placeholder NAME. The
style review pushed toward the "structural" version -- Symbol.__xnew__ plus
the index in _hashable_content -- and I replaced one with the other. That was
wrong: they fix two different properties.

  identity  _hashable_content + __xnew__   without it, two same-named
                                           placeholders are ONE object and the
                                           blend silently collapses
  ordering  the index in the NAME          without it, Symbol.sort_key() ties,
                                           Add term order follows the hash
                                           seed, ranks emit different C, and
                                           getext aborts the run

sort_key() is derived from the name, so _hashable_content does nothing for
it. Identity without ordering is a parallel abort; ordering without identity
is a silently wrong answer. Both are now in, and the docstring says so.

Measured with only the identity half: three distinct compiled modules from one
model across four hash seeds (a6cb2e83, a6cb2e83, df7e63dd, 2f8aadf7), and
`mpirun -np 2` aborting on 4 of 6 launches. With both halves, one module
across five seeds and 0 of 6 launches aborting.

Three regression tests, each verified to FAIL on the specific bug it guards:

- distinct sort_key, and a sum that canonicalises identically written either
  way. Fails with only the name fix reverted, while the identity test still
  passes -- so the pre-existing tests could not have caught this.
- slot assignment unmoved by a value ramp. NB the first version of this test
  used 1000 -> 1e-6 and was VACUOUS, because "1.00000000000000e-6" sorts as a
  prefix of "1.00000000000000"; 1000 -> 0.5 does permute.
- a hermetic hash-seed sweep (UW_JIT_CACHE_DIR, cold cache per seed) asserting
  one compiled module. This replaces an mpirun-based test written first and
  DELETED: it caught the bug 0 times in 6 with the defect present and
  confirmed, because whether two ranks disagree depends on their seeds. A
  coin-flip assertion is not a regression test.

Also dropped the instance_number None guard added in the previous round: the
charter forbids guarding states that cannot occur, and the comment justifying
it was factually wrong (UWexpression.__init__ assigns _uw_id directly; it
never calls uw_object.__init__). Made the _JITConstant docstring raw -- it
contained \e, an invalid escape that is scheduled to become a SyntaxError.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSGAFeA7qYXgkuw9ud8F2G
@lmoresi

lmoresi commented Sep 10, 2026

Copy link
Copy Markdown
Member Author

Adversarial review found that the previous revision was not safe to merge, and the cause was a decision I made while responding to the first round of review.

The bug has two independent halves:

mechanism symptom without it
identity _hashable_content + Symbol.__xnew__ two same-named placeholders are one object → uniform-viscosity answer, silently
ordering the slot index in the name sort_key() ties → Add term order follows the hash seed → ranks emit different C → getext aborts

The original fix did ordering. I replaced it with the structural identity fix and removed the name change, since the design doc deprecates name-munging — but sort_key() is derived from the name, so _hashable_content does nothing for it. Measured with only the identity half: three distinct compiled modules from one model across four hash seeds (a6cb2e83, a6cb2e83, df7e63dd, 2f8aadf7), and mpirun -np 2 aborting on 4 of 6 launches. With both halves: one module across five seeds, 0 of 6 aborting.

Three regression tests, each verified to fail on the specific bug it guards — including one that isolates the ordering half while the identity test still passes, demonstrating the pre-existing tests could not have caught it.

Two things I removed rather than kept:

  • An mpirun-based determinism test I wrote first, deleted because it caught the bug 0 times in 6 with the defect present and confirmed. Whether two ranks disagree depends on their seeds; a coin-flip assertion is not a regression test. The hermetic seed sweep replaces it.
  • A first version of the slot-permutation test that was vacuous: it ramped 1000 → 1e-6, and "1.00000000000000e-6" sorts as a prefix of "1.00000000000000", so nothing moved. 1000 → 0.5 does permute.

Also dropped the instance_number None guard from the previous round (charter §5 forbids guarding unreachable states, and the comment justifying it misdescribed UWexpression.__init__), and made the docstring raw — it contained \e, an invalid escape scheduled to become a SyntaxError.

Full level_1 and tier_a: 1186 passed. One unrelated timing failure in test_0054_hang_report under heavy concurrent load; 8/8 three times in isolation.

Underworld development team with AI support from Claude Code

@lmoresi
lmoresi merged commit 343de17 into development Sep 11, 2026
2 checks passed
lmoresi added a commit that referenced this pull request Sep 11, 2026
…sal message

Three conflicts, all from this branch still carrying d53d003 -- the original
version of the JIT constant-collision fix -- after that work was split out to
PR #717 and diverged there. Resolved to development's version in every case:

- _jitextension.py and test_0103: development has both halves of the fix
  (the slot index in the NAME for ordering, and in _hashable_content for
  identity) plus the three regression tests. This branch had only the first
  attempt.
- constitutive_models.py: development carries the shortened note the charter
  review asked for, not the incident narrative.

Taking --theirs wholesale for _jitextension.py also discarded an unrelated
change this branch has in the same file: the integration-point derivative
refusal message, which points at proxy_location='cells' and says evaluate()
answers the same query. test_0071 caught it. Restored -- minus the benchmark
figure it quoted, which belongs in the subsystem docs rather than in a runtime
error (charter section 4).

Full level_1 and tier_a on the merged tree: 1245 passed, 3 skipped, 1 xfailed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSGAFeA7qYXgkuw9ud8F2G
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants