lakefile: whole-archive sparkle C externs into precompiled .so#54
Merged
Conversation
PR #53's `precompileModules := true` on `lean_lib Sparkle` made Lake build `libsparkle_Sparkle.so` alongside the oleans. That .so was linked against the C extern_lib archives (`libsparkle_barrier.a`, `libsparkle_jit.a`) but the linker's default `--as-needed` pass discarded every symbol no Lean wrapper currently *calls* directly — including `sparkle_jit_load`, which the dlsym-loaded `JIT.load` boxing wrapper looks up at run time. Tutorial-notebooks CI consequently broke with: symbol lookup error: .lake/build/lib/libsparkle_Sparkle.so: undefined symbol: sparkle_jit_load while building `Notebooks.Gen.Ch03_Sequential` (the new chapter that exercises JIT.load). Fix: pass `-Wl,--whole-archive` around the two extern archives in `moreLinkArgs`, so the linker retains every symbol from them. The `-L ./.lake/build/c_src` path points at the directory the `extern_lib` blocks emit those archives into. `lake build Sparkle`, `lake build TutorialNotebooks`, and `lake exe svparser-test` (42/0) all succeed locally with this change.
junjihashimoto
added a commit
that referenced
this pull request
Jun 14, 2026
The downstream-smoke CI added in this branch turned red on all three OSes the first time it ran, surfacing a real bug in PR #54's \`-Wl,--whole-archive\` linker hack: - **Linux**: \`-L ./.lake/build/c_src\` resolves against the downstream consumer's cwd (\`my-blinky/.lake/build/c_src\`), not Sparkle's package dir, so the archives weren't found. - **macOS / Windows**: Apple ld64 and lld-link don't understand \`--whole-archive\` / \`--no-whole-archive\` at all. The actual fix (independently identified in PR #55) is to force default visibility on the FFI export functions in the .c sources themselves, via: #pragma GCC visibility push(default) ... #pragma GCC visibility pop That overrides leanc's mandatory \`-fvisibility=hidden\` per-TU and is portable across all three OSes (clang/gcc/MSVC all accept the GCC visibility pragma; on macOS it maps to the dylib's exported-symbol list, on Windows it's a no-op but the underlying linker behaviour does the right thing anyway because lld-link keeps externally-referenced \`__declspec(dllexport)\`-ish symbols by default). With this, drop the entire \`moreLinkArgs\` block from \`lean_lib Sparkle\`. Verified locally with a clean \`.lake\` + a downstream Lake project that pins Sparkle by path: blinky 16 cycles: [0x0#4, 0x1#4, ..., 0xf#4] Co-authored-by: @wvhulle (#55 author — same diagnosis, same fix)
junjihashimoto
added a commit
that referenced
this pull request
Jun 14, 2026
The downstream-smoke CI added in this branch turned red on all three OSes the first time it ran, surfacing a real bug in PR #54's \`-Wl,--whole-archive\` linker hack: - **Linux**: \`-L ./.lake/build/c_src\` resolves against the downstream consumer's cwd (\`my-blinky/.lake/build/c_src\`), not Sparkle's package dir, so the archives weren't found. - **macOS / Windows**: Apple ld64 and lld-link don't understand \`--whole-archive\` / \`--no-whole-archive\` at all. The actual fix (independently identified in PR #55) is to force default visibility on the FFI export functions in the .c sources themselves, via: #pragma GCC visibility push(default) ... #pragma GCC visibility pop That overrides leanc's mandatory \`-fvisibility=hidden\` per-TU and is portable across all three OSes (clang/gcc/MSVC all accept the GCC visibility pragma; on macOS it maps to the dylib's exported-symbol list, on Windows it's a no-op but the underlying linker behaviour does the right thing anyway because lld-link keeps externally-referenced \`__declspec(dllexport)\`-ish symbols by default). With this, drop the entire \`moreLinkArgs\` block from \`lean_lib Sparkle\`. Verified locally with a clean \`.lake\` + a downstream Lake project that pins Sparkle by path: blinky 16 cycles: [0x0#4, 0x1#4, ..., 0xf#4] Co-authored-by: @wvhulle (#55 author — same diagnosis, same fix)
junjihashimoto
added a commit
that referenced
this pull request
Jul 1, 2026
lakefile: whole-archive sparkle C externs into precompiled .so
junjihashimoto
added a commit
that referenced
this pull request
Jul 1, 2026
The downstream-smoke CI added in this branch turned red on all three OSes the first time it ran, surfacing a real bug in PR #54's \`-Wl,--whole-archive\` linker hack: - **Linux**: \`-L ./.lake/build/c_src\` resolves against the downstream consumer's cwd (\`my-blinky/.lake/build/c_src\`), not Sparkle's package dir, so the archives weren't found. - **macOS / Windows**: Apple ld64 and lld-link don't understand \`--whole-archive\` / \`--no-whole-archive\` at all. The actual fix (independently identified in PR #55) is to force default visibility on the FFI export functions in the .c sources themselves, via: #pragma GCC visibility push(default) ... #pragma GCC visibility pop That overrides leanc's mandatory \`-fvisibility=hidden\` per-TU and is portable across all three OSes (clang/gcc/MSVC all accept the GCC visibility pragma; on macOS it maps to the dylib's exported-symbol list, on Windows it's a no-op but the underlying linker behaviour does the right thing anyway because lld-link keeps externally-referenced \`__declspec(dllexport)\`-ish symbols by default). With this, drop the entire \`moreLinkArgs\` block from \`lean_lib Sparkle\`. Verified locally with a clean \`.lake\` + a downstream Lake project that pins Sparkle by path: blinky 16 cycles: [0x0#4, 0x1#4, ..., 0xf#4] Co-authored-by: @wvhulle (#55 author — same diagnosis, same fix)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the `tutorial-notebooks` CI step on main, which has been
breaking with:
```
symbol lookup error:
/home/runner/work/sparkle/sparkle/.lake/build/lib/libsparkle_Sparkle.so:
undefined symbol: sparkle_jit_load
error: Lean exited with code 127
```
What happened
PR #53 (`xlean-mcp + JIT-in-kernel + Real-Time Collaboration`)
added `precompileModules := true` to `lean_lib Sparkle` so the
xlean WASM kernel could dlsym the Lean-side boxing wrappers from a
shared library at notebook run time.
That worked for everything except symbols that no Lean wrapper
currently calls directly — like `sparkle_jit_load`, which is
only ever dlsym'd via `Sparkle.Core.JIT.JIT.load`'s boxing
wrapper. When Lake linked `libsparkle_Sparkle.so` against the C
extern archives (`libsparkle_barrier.a`, `libsparkle_jit.a`),
the linker's default `--as-needed` / `--gc-sections` pass
discarded every symbol nothing was statically referencing. PR #53
itself slipped through CI because none of its own files triggered
`JIT.load`; PR #53 + the existing Ch03 tutorial chapter does.
Fix
Wrap the two extern archives with `-Wl,--whole-archive ...
-Wl,--no-whole-archive` in `Sparkle`'s `moreLinkArgs`, so the
linker retains every symbol they contain. Pair with
`-L ./.lake/build/c_src` to point at the directory the
`extern_lib` blocks emit those archives into.
Verification
`*:dynlib` target with "libsparkle_barrier.a not found"
until `-L` was added).
`Notebooks.Gen.Ch03_Sequential` chapter that triggered the
original failure.
regression on the prior PRs' SVParser fixes).
CI for this branch should be green.