Cross-compile design review: close cache identity, dependency-kind, and fail-open gaps

[cpunion]

Context

This is a design review of the current Cross-compile Design wiki and its implementation in #166 (reviewed at d7738d32e944525842a242e19659dc49c36b2f83) against current main (b4164750da04f08c179965eb2f7b5087b1fc260c).

This is a follow-up to #110, not a replacement for it. The earlier proposal already identified some requirements, especially cache identity, that are no longer present in the current wiki.

Review conclusion

The overall direction is sound and worth keeping:

• a language-neutral build.Target boundary;
• C/LLVM/sysroot details outside build.Builder;
• a coherent SDK/sysroot rather than treating libc.so as an interchangeable library;
• non-invasive default cross compilation for existing Formulas;
• cross-built artifacts transferred to and exercised on target runners.

However, the current design should not be considered complete yet. Several gaps can cause stale, mislabeled, cross-contaminated, or incorrectly installed artifacts.

Blocking findings

1. Hidden sysroot and toolchain inputs are absent from artifact/cache identity

The default sysroot is deliberately excluded from Project.Deps, build results, and artifact dependencies. The current cache key is only module, version, and matrix:

llar/internal/build/cache/cache.go

Lines 10 to 13 in b416475

	type Key struct {
	Module module.Version
	Matrix string
	}

As a result, changing any of these inputs does not necessarily invalidate an existing consumer artifact:

• default glibc or macOS SDK version;
• sysroot artifact contents;
• LLVM version/digest;
• Darwin minimum deployment target;
• target ABI defaults.

This requirement existed explicitly in #110 under Cache Variant, but the current wiki has no corresponding contract.

Introduce a normalized BuildVariant containing at least the target ABI, sysroot provider identity/digest, and toolchain identity/digest. These facts also need to be recorded as build provenance even when they are not runtime dependencies.

2. Build dependencies and runtime/install dependencies are conflated

The default sysroot is completely hidden, while a Formula-owned sysroot is an ordinary dependency and therefore remains in artifact dependencies. The design also states that a sysroot is a build input rather than a runtime dependency.

Those rules conflict: an ordinary artifact dependency participates in the install dependency closure, so a custom glibc or SDK can be downloaded and installed like a normal consumer dependency.

The model needs typed dependency edges, for example:

build input: toolchain / sysroot / build tool
consumer dependency: headers / link library / runtime

Both default and Formula-selected sysroots should be visible, auditable build inputs, but excluded from the runtime/install closure.

3. Partial and unsupported targets fail open

A missing OS or architecture is currently represented as an empty string and compared directly with the host:

llar/cmd/llar/internal/make.go

Lines 134 to 143 in d7738d3

	var targetOS, targetArch string
	if values := matrix.Require["os"]; len(values) > 0 {
	targetOS = values[0]
	}
	if values := matrix.Require["arch"]; len(values) > 0 {
	targetArch = values[0]
	}
	crossCompile := targetOS != runtime.GOOS || targetArch != runtime.GOARCH
	if runTest && crossCompile {
	return fmt.Errorf("llar test cannot run %s/%s target on %s/%s host", targetOS, targetArch, runtime.GOOS, runtime.GOARCH)

Unsupported cross targets then return a nil built-in target:

llar/internal/crosscompile/crosscompile.go

Lines 117 to 136 in d7738d3

	// Load returns the target used to cross-compile root. A nil target means the
	// requested matrix is native or has no built-in C target policy.
	func Load(ctx context.Context, root module.Version, config Config) (build.Target, error) {
	var targetOS, targetArch string
	if values := config.Matrix.Require["os"]; len(values) > 0 {
	targetOS = values[0]
	}
	if values := config.Matrix.Require["arch"]; len(values) > 0 {
	targetArch = values[0]
	}
	if targetOS == runtime.GOOS && targetArch == runtime.GOARCH {
	return nil, nil
	}
	// TODO: Add other language target policies alongside this C case when
	// they provide build.Target implementations.
	cSysroot, ok := c.Sysroot(targetOS, targetArch)
	if !ok {
	return nil, nil
	}

This can run the host build path while retaining a target-looking matrix, producing a host artifact labeled as another platform.

Resolve a complete TargetSpec before loading/building. Either fill missing hosted dimensions from the build host or reject them. When cross compilation is requested and no provider owns the target, fail closed. A Formula-owned toolchain should require an explicit opt-out/provider declaration rather than relying on unsupported fallback.

4. Formula-owned sysroot injection is stateful and not concurrency-safe

The presence of Matrix.Require["libc"] disables the default sysroot before LLAR has verified that the Formula resolved and injected another one.

x/autotools.Sysroot then mutates process-global flags and injects both GNU and Apple spellings into every flags variable:

llar/x/autotools/autotools.go

Lines 32 to 40 in d7738d3

	// Sysroot sets the target system root for the compiler and linker. Both
	// compiler spellings are supplied so the same Formula works for generic and
	// Apple targets.
	func (a *AutoTools) Sysroot(root string) {
	for _, key := range []string{"CPPFLAGS", "CFLAGS", "CXXFLAGS", "LDFLAGS"} {
	appendFlag(key, "--sysroot="+root)
	appendFlag(key, "-isysroot"+root)
	}
	}

Disjoint builds are allowed to run concurrently, so two target builds can observe or restore each other's environment. The helper also cannot choose the correct target-specific spelling because it has only a path, not a target.

Resolve the sysroot provider before command execution, project one target-specific spelling, and keep environment changes on the helper/command instance. Target.Use should be error-returning, immutable or concurrency-safe, and must not use panic for recoverable preparation errors:

Use(Command) (Patch, error)

5. The pkg-config contract is internally inconsistent

The wiki specifies PKG_CONFIG_SYSROOT_DIR plus dependency and sysroot paths in PKG_CONFIG_LIBDIR. The implementation deliberately leaves PKG_CONFIG_SYSROOT_DIR unset and copies only dependency paths:

llar/internal/crosscompile/c/target.go

Lines 198 to 207 in d7738d3

	func (c *Target) pkgConfigPatch(commandEnv []string) build.Patch {
	if c.sysroot == "" {
	return build.Patch{}
	}
	env := append([]string(nil), commandEnv...)
	libDirs, _ := envValue(env, "PKG_CONFIG_PATH")
	// Use stores LLAR dependency .pc directories in PKG_CONFIG_PATH. Restrict
	// lookup to them without rewriting their absolute installation prefixes.
	env = setMissingEnv(env, "PKG_CONFIG_LIBDIR", libDirs)
	return build.Patch{Env: env}

Simply implementing the wiki text is not sufficient. A global sysroot prefix would also rewrite -I/-L paths emitted by LLAR dependency .pc files whose prefixes are absolute build-side output paths. PKG_CONFIG_LIBDIR also replaces the normal search path rather than extending it.

For MVP, explicitly scope support to LLAR dependency .pc files. Full sysroot plus dependency support needs a unified relocatable overlay, generated .pc view, or wrapper that can distinguish the origin of each .pc file.

6. Raw linker commands are not target-prepared

CC and CXX contain target, linker-driver, and sysroot arguments, but Toolchain.Linker() contains only the path to ld.lld or ld64.lld:

llar/internal/crosscompile/c/llvm/toolchain.go

Lines 100 to 130 in d7738d3

	linker, err := find(linkerName)
	if err != nil {
	return nil, err
	}
	archiver, err := find("llvm-ar")
	if err != nil {
	return nil, err
	}
	ranlib, err := find("llvm-ranlib")
	if err != nil {
	return nil, err
	}
	nm, err := find("llvm-nm")
	if err != nil {
	return nil, err
	}
	strip, err := find("llvm-strip")
	if err != nil {
	return nil, err
	}
	cc := []string{ccPath, "--target=" + triple, "-fuse-ld=lld"}
	cxx := []string{cxxPath, "--target=" + triple, "-fuse-ld=lld"}
	if config.Sysroot != "" {
	flag := "--sysroot=" + config.Sysroot
	if config.OS == "darwin" {
	flag = "-isysroot" + config.Sysroot
	}
	cc = append(cc, flag)
	cxx = append(cxx, flag)
	}
	toolchain := c.NewToolchain(cc, cxx, []string{linker}, archiver, ranlib, nm, strip)

Consequently, direct ld rewriting and Autotools LD receive no target/sysroot/platform arguments. The model should distinguish:

• compiler-driver link commands;
• fully prepared raw-linker commands;
• the linker executable consumed internally by CMake.

Add cross-link E2E coverage for shared libraries and executables, especially Darwin. The current Darwin zlib flow primarily produces a static archive and performs the final consumer link natively on the target runner.

Current documentation and implementation drift

• The wiki selects glibc 2.17; feat(build): support crosscompile  #166 code and E2E select Bootlin glibc 2.24.
• The proposed production glibc 2.17 Formula is not validated by the current experiments.
• The production llarhub does not currently contain the documented bminor/glibc or joseluisq/macosx-sdks Formulas; feat(build): support crosscompile  #166 creates temporary fixture repositories in CI.
• Automatic macOS SDK acquisition/distribution remains undefined. The selected SDK repository itself asks users to review the Xcode license terms first.
• The wiki module specification uses internal/build/c; feat(build): support crosscompile  #166 implements internal/crosscompile/c.
• C++ has command-level tests but no equivalent runtime compatibility E2E.
• feat(build): support crosscompile  #166 has not incorporated merged feat(execbroker): propagate middleware errors #167. Merging its head with current main is textually clean, but targeted Go tests fail to compile because targetMiddleware still returns the old middleware signature:

internal/build/target.go:31:9: cannot use func(req execbroker.Request) execbroker.Request
as execbroker.Middleware value in return statement

The corresponding review thread is still unresolved:
#166 (comment)

Suggested core model

TargetSpec
  build platform
  runtime/host platform
  OS, architecture, ABI/libc, minimum OS, CPU/features

BuildVariant
  normalized TargetSpec
  ToolchainID + digest
  SysrootID + digest

BuildInput
  kind: toolchain | sysroot | build-tool
  provider module/version
  artifact digest

BuildInput should participate in cache identity and provenance without automatically becoming a runtime/install dependency.

Acceptance criteria

• Cache identity changes when the default sysroot, SDK, toolchain, deployment target, or ABI changes.
• Artifact metadata records default and custom toolchain/sysroot build inputs.
• Build inputs are distinct from runtime/install dependencies.
• Partial and unsupported hosted targets cannot silently use the native build path.
• A libc matrix key cannot disable the default without resolving a valid replacement provider.
• Target injection uses command-local environment and propagates errors without panic.
• The documented pkg-config behavior matches a tested relocation model.
• Raw linker commands carry complete target/platform/sysroot information or are not exposed as supported commands.
• The documented default glibc/SDK versions match published Formula providers.
• Cross-link and target-run E2E covers C shared libraries/executables and at least one C++ consumer for each supported target family.

[MeteorsLiu]

1. About Cache Key

There are only two possibilities regarding the cache key:

1. If a custom libc exists, such changes will be reflected in the matrix key.
2. If there is no libc but cross-building is involved, LLAR will inject a default libc dependency. Under normal circumstances, this dependency does not change because there is no need to update the libc version.

In summary, the only remaining issue is the toolchain version number, which we will set aside for now since the toolchain design has not yet begun.

2. Dependency

It's expected to hide libc dependency, because we don't have a reason to show them to formula(and unnecessary)

We should inject those libc dependency to sysroot which is managed by llar.

Audited Problem

• GetEnv / SetEnv not isolated feat(execbroker): scope environment updates #172