LLVM GC invariant violation (Illegal inttoptr) with in-place NonlinearProblem on Julia 1.12

[ChrisRackauckas-Claude]

Description

Enzyme.gradient through a NonlinearProblem with an in-place function f!(residual, u, p) causes an LLVM abort due to Illegal inttoptr in the GC invariant verifier on Julia 1.12. On Julia 1.10, the same code works.

The crash occurs during Enzyme's code generation for the augmented_primal rule in NonlinearSolveBaseEnzymeExt. Simplified in-place functions without NonlinearSolve work fine, so the bug is triggered by the specific code patterns in NonlinearSolve's Enzyme extension interacting with Enzyme's LLVM passes.

MWE (NonlinearSolve + Enzyme, no MTK)

import Pkg
Pkg.activate(; temp = true)
Pkg.add(["Enzyme", "NonlinearSolve", "SciMLSensitivity"])

using NonlinearSolve, Enzyme, SciMLSensitivity

function f!(residual, u, p)
    residual .= u .^ 2 .- p
end

function loss_inplace(p)
    prob = NonlinearProblem(f!, [1.0], p)
    sol = solve(prob, NewtonRaphson())
    return sum(sol.u)
end

println("Forward: ", loss_inplace([2.0]))

# Crashes on Julia 1.12, works on Julia 1.10
Enzyme.gradient(Enzyme.set_runtime_activity(Enzyme.Reverse), loss_inplace, [2.0])

Error (Julia 1.12.4)

Illegal inttoptr
	  %13 = inttoptr i64 %.unpack3 to ptr addrspace(10), !dbg !87

signal 6 (-6): Aborted

Full crash trace shows the abort originates in llvm-gc-invariant-verifier.cpp:196 during LLVMRunJuliaPasses, called from Enzyme's post_optimize! → _thunk → cached_compilation.

Notes

• Works on Julia 1.10 with same package versions
• Simplified in-place code (hand-written Newton iteration without NonlinearSolve) works fine on Julia 1.12 — the crash is specific to Enzyme processing the NonlinearSolveBaseEnzymeExt augmented_primal/reverse rules
• Out-of-place f(u, p) with NonlinearSolve works on both Julia versions

Environment

• Julia 1.12.4 (linux x64)
• Enzyme v0.13.132
• NonlinearSolve v4.16.0
• SciMLSensitivity v7.99.0

Context

From investigation in SciML/SciMLSensitivity.jl#1358 (comment)

🤖 Generated with Claude Code

Co-Authored-By: Chris Rackauckas accounts@chrisrackauckas.com

[wsmoses]

using Enzyme
using ConcreteStructs

@concrete struct MyExtractedAlg
    linesearch
    trustregion
    descent
    forcing
    max_shrink_times::Int
    autodiff
    vjp_autodiff
    jvp_autodiff
    concrete_jac
    name::Symbol
end

@noinline function rule_func_inner(s::MyExtractedAlg, z::Float64)
    # Nested AD!
    f_dummy(x) = x[1] * 2.0 + (s.max_shrink_times === nothing ? 0.0 : Float64(s.max_shrink_times))
    grad = Enzyme.gradient(Enzyme.set_runtime_activity(Enzyme.Reverse), f_dummy, [z])
    return grad[1][1] + (s.name === :NewtonRaphson ? 1.0 : 0.0)
end

@noinline function rule_func_const(z::Float64, args...)
    return rule_func_inner(args[1], z)
end

# Define custom rule to trigger rule wrapper processing
function Enzyme.EnzymeRules.augmented_primal(config, func::Enzyme.EnzymeRules.Const{typeof(rule_func_const)}, ::Type{<:Enzyme.EnzymeRules.Annotation}, z::Enzyme.EnzymeRules.Active, args...)
    # Forward pass: compute primal only if needed
    primal = Enzyme.EnzymeRules.needs_primal(config) ? rule_func_const(z.val, map(x->x.val, args)...) : nothing
    return Enzyme.EnzymeRules.AugmentedReturn(primal, nothing, nothing)
end

function Enzyme.EnzymeRules.reverse(config, func::Enzyme.EnzymeRules.Const{typeof(rule_func_const)}, dret::Enzyme.EnzymeRules.Active, tape, z::Enzyme.EnzymeRules.Active, args...)
    # We don't care about correctness of reverse for this test, just if it crashes
    return (dret.val, ntuple(_->nothing, Val(length(args)))...)
end

function loss_inplace(p)
    alg = MyExtractedAlg(
        nothing, nothing, nothing, nothing, 10,
        nothing, nothing, nothing,
        nothing, :NewtonRaphson
    )
    # Direct pass (unboxed) to trigger ABI mismatch on main branch simulated setup
    return rule_func_const(p[1], alg)
end

println("Calling autodiff with extracted unboxed struct and nested AD...")
p = [2.0]
Enzyme.gradient(Enzyme.set_runtime_activity(Enzyme.Reverse), loss_inplace, p)
println("Done without crash!")

[wsmoses]

mwe should be fixed by #3060

@ChrisRackauckas-Claude can you retry (and reopen if not)?

[ChrisRackauckas-Claude]

Retried with Enzyme.jl main at bf6bccf (two commits past #3060) — original issue MWE still crashes on Julia 1.12 with the same Illegal inttoptr GC-invariant abort:

Enzyme version: 0.13.139
Forward: 1.4142135623730951
Illegal inttoptr
	  %13 = inttoptr i64 %.unpack3 to ptr addrspace(10), !dbg !87

[...] signal 6 (-6): Aborted
run at llvm-gc-invariant-verifier.cpp:196
LLVMRunJuliaPasses → post_optimize! (compiler/optimize.jl:454) → _thunk → cached_compilation
... → loss_inplace → Enzyme.gradient(NonlinearSolve solve ...)

Setup: Julia 1.12.4, Enzyme.jl main dev-installed, Enzyme_jll relaxed to allow 0.0.256 (since #3060 only modifies src/rules/customrules.jl — Julia-side only). Crash also reproduces with a locally-built libEnzyme-18.so from upstream main (147MB, freshly compiled), so it's not a stale-jll issue either.

The reduced MWE I posted on March 28 (custom rules + nested AD, no NonlinearSolve) does pass on main — so #3060 fixed that pattern, but the original NonlinearSolve in-place f! path is still broken. Reopening per your request.

Co-Authored-By: Chris Rackauckas accounts@chrisrackauckas.com

[wsmoses]

@ChrisRackauckas-Claude can you re minimize?

[ChrisRackauckas-Claude]

@wsmoses Tried hard to reduce. Couldn't get it pure-Enzyme — every NonlinearSolve-free reproduction (handcrafted FWW + custom rule + dynamic dispatch through Any-field, etc.) ran clean. The bug seems intrinsically tied to NonlinearSolve's solve dispatch chain.

But I can drop SciMLSensitivity — and that turns the GC-invariant abort into a clean assertion that's probably easier to debug. Here's the reduced MWE (Julia 1.12.4, Enzyme.jl main at bf6bccf):

import Pkg
Pkg.activate(; temp = true)
Pkg.add(["NonlinearSolve", "Enzyme"])

using NonlinearSolve, Enzyme

function f!(residual, u, p)
    residual .= u .^ 2 .- p
end

function loss(p)
    prob = NonlinearProblem(f!, [1.0], p)
    sol = solve(prob, NewtonRaphson())
    return sum(sol.u)
end

Enzyme.gradient(Enzyme.set_runtime_activity(Enzyme.Reverse), loss, [2.0])

Without SciMLSensitivity loaded, instead of the GC-invariant Illegal inttoptr you get:

AssertionError: Enzyme : mismatch between innerTy LLVM.PointerType(ptr addrspace(11))
                                       and tape type LLVM.PointerType(ptr addrspace(10))
mi  = MethodInstance for (::FunctionWrappersWrapper{Tuple{
        FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}},
        FunctionWrapper{Nothing, Tuple{Vector{Dual{...,Float64,1}}, Vector{Dual{...,Float64,1}}, Vector{Float64}}},
        FunctionWrapper{Nothing, Tuple{Vector{Dual{...,Float64,1}}, Vector{Dual{...,Float64,1}}, Vector{Dual{...,Float64,1}}}},
        FunctionWrapper{Nothing, Tuple{Vector{Dual{...,Float64,1}}, Vector{Float64}, Vector{Dual{...,Float64,1}}}},
      }, AllowNonIsBits, SingleCacheStorage})(::Vector{Float64}, ::Vector{Float64}, ::Vector{Float64})
ami = MethodInstance for Enzyme.Compiler.custom_rule_method_error(...,
        ::Duplicated{FunctionWrappersWrapper{...}}, ::Type{Const{Nothing}},
        ::Duplicated{Vector{Float64}}, ::Duplicated{Vector{Float64}}, ::Duplicated{Vector{Float64}})
[1] enzyme_custom_common_rev(...)
[2] enzyme_custom_rev(...)
[4] EnzymeCreatePrimalAndGradient(...)
[17] runtime_generic_augfwd(..., NonlinearSolveBase.solve_call, ::NonlinearProblem{..., AutoSpecialize, AutoSpecializeCallable{FunctionWrappersWrapper{...}}}, ..., GeneralizedFirstOrderAlgorithm{..., NewtonDescent{...}}, ::Nothing)

What seems to be happening: Enzyme falls back to runtime_generic_augfwd for solve_call(prob, alg). Inside, when the FunctionWrappersWrapper is called, runtime-activity propagation marks it as Duplicated{FWW}. But FunctionWrappersWrappersEnzymeExt only registers rules for Const{<:FWW}, so dispatch hits custom_rule_method_error, and somewhere upstream of that the addrspace(10)/(11) mismatch is left in the IR — that's what fails verification (or trips the GC invariant verifier when SciMLSensitivity is present).

I tried 11 reduction variants probing each piece individually:

What I tried	Result
Drop SciMLSensitivity	AssertionError above (cleaner than Illegal inttoptr)
init+step! instead of solve	Same AssertionError
SimpleNewtonRaphson (SimpleNonlinearSolve only)	LLVM error: function failed verification (4)
Hand-rolled solve_up with AugmentedReturn{RT,RT,Any} + closure-in-tape	No crash
FWW direct call with in-place f!	No crash
FWW with 4 specs (Float64 + Dual variants) called dynamically through Any field	No crash
Stub NonlinearProblem-like struct + custom rule mirroring NonlinearSolveBaseEnzymeExt	No crash

Hopefully the Duplicated{FunctionWrappersWrapper} annotation surfacing inside runtime_generic_augfwd is enough of a thread to pull on. Happy to test patches.

Co-Authored-By: Chris Rackauckas accounts@chrisrackauckas.com

[wsmoses]

@ChrisRackauckas-Claude can you re-assess?

[ChrisRackauckas-Claude]

Reassessment — still reproduces on current stack

Tested on Julia 1.12.6 + Enzyme 0.13.146 + NonlinearSolve 4.19.1 + SciMLSensitivity 7.109.0 (latest as of today). The original MWE still crashes deterministically with the same Illegal inttoptr LLVM GC invariant abort:

using NonlinearSolve, Enzyme, SciMLSensitivity

function f!(residual, u, p)
    residual .= u .^ 2 .- p
end

function loss_inplace(p)
    prob = NonlinearProblem(f!, [1.0], p)
    sol = solve(prob, NewtonRaphson())
    return sum(sol.u)
end

Enzyme.gradient(Enzyme.set_runtime_activity(Enzyme.Reverse), loss_inplace, [2.0])

Illegal inttoptr
	  %13 = inttoptr i64 %.unpack3 to ptr addrspace(10), !dbg !87

[3953088] signal 6 (-6): Aborted
run at /workspace/srcdir/julia/src/llvm-gc-invariant-verifier.cpp:196
LLVMRunJuliaPasses → post_optimize! (Enzyme/compiler/optimize.jl:478)
_thunk → cached_compilation → thunkbase → thunk_generator

Same crash site (post_optimize! → LLVM GC invariant verifier) as before — none of the in-flight Enzyme work has touched it.

Without SciMLSensitivity — cleaner surface for the underlying bug

Drop SciMLSensitivity and the Illegal inttoptr becomes an AssertionError, same as in my April 30 reduction:

AssertionError: Enzyme : mismatch between innerTy
                 LLVM.PointerType(ptr addrspace(11))
               and tape type
                 LLVM.PointerType(ptr addrspace(10))
mi  = MethodInstance for (::FunctionWrappersWrapper{...})(::Vector{Float64}, ...)
ami = MethodInstance for Enzyme.Compiler.custom_rule_method_error(...,
        ::Duplicated{FunctionWrappersWrapper{...}}, ::Type{Const{Nothing}},
        ::Duplicated{Vector{Float64}}, ::Duplicated{Vector{Float64}},
        ::Duplicated{Vector{Float64}})
[1] enzyme_custom_common_rev    (customrules.jl:1831)
[2] enzyme_custom_rev           (customrules.jl:2462)
[4] EnzymeCreatePrimalAndGradient
...
[17] runtime_generic_augfwd → NonlinearSolveBase.solve_call

runtime_generic_augfwd invokes FunctionWrappersWrapper with Duplicated{FWW} activity, hits custom_rule_method_error (FWW's rule in FunctionWrappersWrappersEnzymeExt only matches Const), and the addrspace(10)/(11) mismatch ends up in the IR. With SciMLSensitivity loaded, the same path produces the Illegal inttoptr instead (different verifier catches it first).

Note that the current NonlinearSolveBaseEnzymeExt.augmented_primal/reverse rules are already generic over RT <: Enzyme.Annotation and dispatch correctly — the failure isn't there. It's downstream, after Enzyme falls back to runtime_generic_augfwd for the solve() dispatch chain and tries to differentiate the FWW invocation inside.

Confirmed reproduction matrix (today)

Julia	SciMLSensitivity	Result
1.12.6	yes	Illegal inttoptr (this issue)
1.12.6	no	AssertionError (FWW Duplicated)
1.10.11	yes (SCC variant)	works 10/10 (#3026 original now fixed)
1.10.11	no	AssertionError (FWW Duplicated)

The Duplicated{FunctionWrappersWrapper} AssertionError is the common throughline. Happy to test any patch.

🤖 Generated with Claude Code

Co-Authored-By: Chris Rackauckas accounts@chrisrackauckas.com

[wsmoses]

@ChrisRackauckas-Claude youre on an old version of Enzyme can you retry

[ChrisRackauckas-Claude]

Retest on Enzyme.jl main (commit 40e5fab)

Dev-installed Enzyme.jl#main on top of the same project. The original Illegal inttoptr is gone — but the MWE still crashes 5/5 on Julia 1.12.6, just with different failure modes downstream.

Same MWE, different crashes

using NonlinearSolve, Enzyme, SciMLSensitivity
f!(residual, u, p) = (residual .= u .^ 2 .- p)
loss_inplace(p) = sum(solve(NonlinearProblem(f!, [1.0], p), NewtonRaphson()).u)
Enzyme.gradient(Enzyme.set_runtime_activity(Enzyme.Reverse), loss_inplace, [2.0])

5 runs on Julia 1.12.6 + Enzyme.jl#main (40e5fab) + NonlinearSolve 4.19.1 + SciMLSensitivity 7.109.0, 0 pass:

Run	Failure
1	signal 7 (Bus error) in ~PassBuilder
2	signal 11 (Segmentation fault) in ~PassBuilder
3	EnzymeInternalError from fixup_callconv!
4	signal 11 (Segmentation fault) in ~PassBuilder
5	EnzymeInternalError from fixup_callconv!

Bus error / segfault stack:

~PassBuilder at PassBuilder.h:104
runJuliaPasses at NewPM.cpp:541
LLVMRunJuliaPasses → run! at LLVM/newpm.jl:367
middle_optimize! at Enzyme/compiler/optimize.jl:83  ← (was post_optimize!:478 before)
optimize! at Enzyme/compiler/optimize.jl:173
compile_unhooked at Enzyme/compiler.jl:5550
_thunk → cached_compilation → thunkbase → thunk_generator
[...]
runtime_generic_augfwd → NonlinearSolveBase.solve_call

EnzymeInternalError stack (cleaner failure path of the same compile):

[1] julia_error                @ Enzyme/errors.jl:1226
[2] julia_error                @ Enzyme/errors.jl:997
[3] LLVMRunJuliaPasses
[8] fixup_callconv!            @ Enzyme/compiler/optimize.jl:407
[9] _thunk                     @ Enzyme/compiler.jl:6875
[...]
[14] runtime_generic_augfwd    @ Enzyme/rules/jitrules.jl:464

So the path that used to abort at post_optimize! → LLVM GC invariant verifier (Illegal inttoptr) on v0.13.147 now gets past that pass and dies one stage earlier in middle_optimize! / fixup_callconv! instead. Net: same MWE, same site of bad-IR generation in runtime_generic_augfwd, different verifier catches it.

Reproduction matrix on Enzyme.jl#main (today)

Julia	Project	Result
1.12.6	NonlinearSolve + SciMLSensitivity (this issue)	0/5 — Bus/Segfault/EnzymeInternalError
1.12.6	NonlinearSolve only (no SciMLSensitivity)	AssertionError: Duplicated{FunctionWrappersWrapper} ... addrspace(11)/(10) mismatch
1.12.6	SCC variant (#3026 original)	0/5 — same mix of Bus/Segfault/EnzymeInternalError
1.10.11	SCC variant (#3026 original)	5/5 pass ✅
1.10.11	NonlinearSolve only (no SciMLSensitivity)	same Duplicated{FunctionWrappersWrapper} AssertionError

The underlying Duplicated{FunctionWrappersWrapper} → custom_rule_method_error path that runtime_generic_augfwd takes when hitting the in-place NonlinearProblem function is unchanged. Happy to enable Enzyme.Compiler.VERBOSE_ERRORS[] = true and post the full dump if useful.

🤖 Generated with Claude Code

Co-Authored-By: Chris Rackauckas accounts@chrisrackauckas.com

[wsmoses]

Yes please enable verbose errors and now attempt to minimize as much as possible