Skip to content

test/desauty: user-side alias break in Enzyme init closure (fixes SCC init @test_broken)#1454

Merged
ChrisRackauckas merged 2 commits into
SciML:masterfrom
ChrisRackauckas-Claude:enzyme-init-test-user-alias-break
May 25, 2026
Merged

test/desauty: user-side alias break in Enzyme init closure (fixes SCC init @test_broken)#1454
ChrisRackauckas merged 2 commits into
SciML:masterfrom
ChrisRackauckas-Claude:enzyme-init-test-user-alias-break

Conversation

@ChrisRackauckas-Claude
Copy link
Copy Markdown
Contributor

@ChrisRackauckas-Claude ChrisRackauckas-Claude commented May 25, 2026

Note

Draft — please ignore until reviewed by @ChrisRackauckas.

Summary

The SCC init Enzyme test was @test_broken because the previous closure-based pattern misused Enzyme's API: Enzyme.gradient(set_runtime_activity(Reverse), Const(closure), itunables) doesn't allocate shadows for the closure's captured mutable state. Since the closure captured iprob (whose iprob.p.caches is shared with the new MTKParameters produced by irepack's @set!-based repack), the derivative info from the inner solve!'s cache writes had nowhere to land — and the gradient silently came out zero.

The fix is to use the idiomatic Enzyme pattern: a plain function whose captured mutable state is passed as an explicit Duplicated argument:

function enzyme_init_loss(t, iprob_)
    _, irepack_, _ = SS.canonicalize(SS.Tunable(), parameter_values(iprob_))
    iprob2 = remake(iprob_, p = irepack_(t))
    sol = solve(iprob2, NewtonRaphson())
    return sum(sol.u)
end
diprob = Enzyme.make_zero(iprob)
dtunables = zero(itunables)
Enzyme.autodiff(
    set_runtime_activity(Reverse),
    Const(enzyme_init_loss), Active,
    Duplicated(itunables, dtunables),
    Duplicated(iprob, diprob),
)

irepack is reconstructed inside the loss from the duplicated iprob_, so its captured parameter template shares Enzyme's shadow. With this change the SCC init gradient matches FD to 8 significant figures.

Verified locally:

  • FD: [0, 0, 0.3663, 0.3101, 0, 0, 0.7325, 0, 0, 0]
  • Enzyme: [0, 0, 0.3663, 0.3101, 0, 0, 0.7325, 0, 0, 0]

@test_broken@test; both use_scc = false and use_scc = true branches share the same test body.

Companion PRs

These are independent SciML-side fixes uncovered during the investigation (legitimate bugs in their own right; needed for the test to pass cleanly):

Test plan

  • Local desauty_dae_mwe.jl Enzyme block matches FD to 8 sig figs.
  • CI green on full SciMLSensitivity suite.

The Enzyme block for the SCC init was previously @test_broken because
`Enzyme.gradient(set_runtime_activity(Reverse), Const(closure), itunables)`
doesn't allocate shadows for the closure's captured mutable state. The
closure captures `iprob` (an `SCCNonlinearProblem`) whose `iprob.p.caches`
is shared with the new `MTKParameters` built by `irepack`'s `@set!`-based
repack — and the inner `solve!`'s cache writes flow into that shared
buffer with no shadow to carry the derivative info, silently producing
zero gradients.

The idiomatic Enzyme pattern is to express the loss as a plain function
whose captured mutable state is passed as an explicit `Duplicated`
argument. `irepack` is also reconstructed inside the loss from the
duplicated `iprob_` so its captured parameter template shares Enzyme's
shadow:

    function enzyme_init_loss(t, iprob_)
        _, irepack_, _ = SS.canonicalize(SS.Tunable(), parameter_values(iprob_))
        iprob2 = remake(iprob_, p = irepack_(t))
        sol = solve(iprob2, NewtonRaphson())
        return sum(sol.u)
    end
    diprob = Enzyme.make_zero(iprob)
    dtunables = zero(itunables)
    Enzyme.autodiff(
        set_runtime_activity(Reverse),
        Const(enzyme_init_loss), Active,
        Duplicated(itunables, dtunables),
        Duplicated(iprob, diprob),
    )

Verified locally: matches FiniteDiff to 8 significant figures.

`@test_broken` → `@test`; both `use_scc = false` and `use_scc = true`
branches share the same test body.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
@ChrisRackauckas-Claude
Copy link
Copy Markdown
Contributor Author

CI is failing because the test requires SciML/NonlinearSolve.jl#936 (the _accum_tangent! caches walk in NonlinearSolveBase), which is merged on master at v2.26.1 but not yet registered — General registry still has v2.26.0 (pre-NS#936). Local runs match FD to 8 sig figs because they pick up NonlinearSolveBase from a dev'd path that includes NS#936; CI resolves to the registry version which doesn't have it.

Triggered the registrator on the NS#936 merge commit: SciML/NonlinearSolve.jl@a32ebd6#commitcomment-186443744

Once NonlinearSolveBase v2.26.1 lands in the General registry this PR's CI should pass without further changes (the compat in SciMLSensitivity's Project.toml is open enough already — it'll naturally pick up the new version).

The desauty SCC init Enzyme test relies on the `_accum_tangent!`
caches walk from NS#936, which is in NonlinearSolveBase v2.26.1+ but
only registered as v2.27.0 in the General registry. Add
NonlinearSolveBase as a direct dep with `compat = "2.27"` so CI
resolves to a version that includes NS#936.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
@ChrisRackauckas ChrisRackauckas marked this pull request as ready for review May 25, 2026 15:46
@ChrisRackauckas ChrisRackauckas merged commit 44e98dd into SciML:master May 25, 2026
43 of 50 checks passed
ChrisRackauckas-Claude pushed a commit to ChrisRackauckas-Claude/SciMLSensitivity.jl that referenced this pull request May 26, 2026
Apply the same pattern as test/desauty (SciML#1454) to the
`Adjoint through Prob (Enzyme)` testset: express the loss as a plain
function `enzyme_loss(t, prob_)`, pass `prob` as an explicit
`Duplicated` argument (with `make_zero(prob)` shadow), and reconstruct
`repack` *inside* the loss from `prob_` so its captured parameter
template shares Enzyme's shadow. This is the idiomatic Enzyme
formulation for a loss that captures mutable state; `Const(loss)` on
a closure capturing mutable `prob` doesn't allocate shadows for the
captures, while `autodiff(... Duplicated(prob, dprob))` does.

The `@test_broken` is retained because the rewrite now exposes a
deeper, well-localized blocker: `Enzyme.autodiff` enters the
`GaussAdjoint` `_concrete_solve_adjoint` rule, which calls
`_init_originator_gradient` to differentiate the parameter-init
`tunables -> new_u0` mapping. That helper invokes
`Enzyme.gradient(Enzyme.Reverse, Const(init_loss), tunables)` without
`set_runtime_activity`, so the inner Enzyme call still raises
`EnzymeRuntimeActivityError` at the MTK init `remake` — the outer
call's runtime-activity setting does not propagate into that nested
gradient. The test now clearly indicates what's left: teach
`_init_originator_gradient(::EnzymeOriginator, ...)` to use
`set_runtime_activity(Reverse)` (or `autodiff` + `Duplicated` itself).
Flipping `@test_broken` -> `@test` is the only change needed here
once that fix lands.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
ChrisRackauckas-Claude pushed a commit to ChrisRackauckas-Claude/SciMLSensitivity.jl that referenced this pull request May 26, 2026
Refactors the DAE-init/SCCNonlinearProblem `@test_broken` block in
`test/mtk.jl` to the same pattern as the desauty SCC init test in
`desauty_dae_mwe.jl` (SciML#1454): a plain top-level function (no captured
mutable closure) with the `ODEProblem` passed as `Duplicated` and the
tunables as `Duplicated(t, dt)`.

`sensealg` is now a module-level `const _MTK_SENSEALG` (rather than a
captured closure variable). Under the rewrite shape, passing it as a
function argument under `set_runtime_activity(Reverse)` causes Enzyme
to promote it to `Duplicated`, which the `solve_up` Enzyme rule in
`DiffEqBaseEnzymeExt` rejects (the rule signature requires
`sensealg::Union{Const{Nothing}, Const{<:AbstractSensitivityAlgorithm}}`).
Referencing it via a `const` makes Enzyme observe it as `Const`.

Status: the block remains `@test_broken`. The first configuration
(`prob_correctu0` + `CheckInit`) advances past the closure-capture
shadowing issue but still errors during reverse autodiff with the
same `MixedDuplicated`/MTK runtime-activity wrapping pattern tracked
in SciMLSensitivity.jl#1359 — the structural ODEProblem fields
carrying `MTKParameters` are seen as `Duplicated` and propagate into
sensealg/u0/p in the underlying `solve_up` dispatch. The refactor
matches the documented user-side pattern so that when SciML#1359 lifts,
only flipping `@test_broken` → `@test` is needed.

The companion Mooncake `@test_broken` block below is unchanged.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
ChrisRackauckas-Claude pushed a commit to ChrisRackauckas-Claude/SciMLSensitivity.jl that referenced this pull request May 26, 2026
Apply the same pattern as test/desauty (SciML#1454) to the
`Adjoint through Prob (Enzyme)` testset: express the loss as a plain
function `enzyme_loss(t, prob_)`, pass `prob` as an explicit
`Duplicated` argument (with `make_zero(prob)` shadow), and reconstruct
`repack` *inside* the loss from `prob_` so its captured parameter
template shares Enzyme's shadow. This is the idiomatic Enzyme
formulation for a loss that captures mutable state; `Const(loss)` on
a closure capturing mutable `prob` doesn't allocate shadows for the
captures, while `autodiff(... Duplicated(prob, dprob))` does.

The `@test_broken` is retained because the rewrite now exposes a
deeper, well-localized blocker: `Enzyme.autodiff` enters the
`GaussAdjoint` `_concrete_solve_adjoint` rule, which calls
`_init_originator_gradient` to differentiate the parameter-init
`tunables -> new_u0` mapping. That helper invokes
`Enzyme.gradient(Enzyme.Reverse, Const(init_loss), tunables)` without
`set_runtime_activity`, so the inner Enzyme call still raises
`EnzymeRuntimeActivityError` at the MTK init `remake` — the outer
call's runtime-activity setting does not propagate into that nested
gradient. The test now clearly indicates what's left: teach
`_init_originator_gradient(::EnzymeOriginator, ...)` to use
`set_runtime_activity(Reverse)` (or `autodiff` + `Duplicated` itself).
Flipping `@test_broken` -> `@test` is the only change needed here
once that fix lands.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
ChrisRackauckas-Claude pushed a commit to ChrisRackauckas-Claude/SciMLSensitivity.jl that referenced this pull request May 26, 2026
Refactors the DAE-init/SCCNonlinearProblem `@test_broken` block in
`test/mtk.jl` to the same pattern as the desauty SCC init test in
`desauty_dae_mwe.jl` (SciML#1454): a plain top-level function (no captured
mutable closure) with the `ODEProblem` passed as `Duplicated` and the
tunables as `Duplicated(t, dt)`.

`sensealg` is now a module-level `const _MTK_SENSEALG` (rather than a
captured closure variable). Under the rewrite shape, passing it as a
function argument under `set_runtime_activity(Reverse)` causes Enzyme
to promote it to `Duplicated`, which the `solve_up` Enzyme rule in
`DiffEqBaseEnzymeExt` rejects (the rule signature requires
`sensealg::Union{Const{Nothing}, Const{<:AbstractSensitivityAlgorithm}}`).
Referencing it via a `const` makes Enzyme observe it as `Const`.

Status: the block remains `@test_broken`. The first configuration
(`prob_correctu0` + `CheckInit`) advances past the closure-capture
shadowing issue but still errors during reverse autodiff with the
same `MixedDuplicated`/MTK runtime-activity wrapping pattern tracked
in SciMLSensitivity.jl#1359 — the structural ODEProblem fields
carrying `MTKParameters` are seen as `Duplicated` and propagate into
sensealg/u0/p in the underlying `solve_up` dispatch. The refactor
matches the documented user-side pattern so that when SciML#1359 lifts,
only flipping `@test_broken` → `@test` is needed.

The companion Mooncake `@test_broken` block below is unchanged.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants