Skip to content

Kotlin/Native + KMP in rules_kotlin: a synthesis of prior efforts + a working prototype #1682

Description

@cgruber

TL;DR

There have been several parallel runs at this: a tracking issue (#567), a JS re-enable (#1347), an initial native toolchain + kt_library (#1351), a roadmap outline (#1466), a Build Tools API plan (#1481), an external ruleset (kitterion/rules_kotlin_native), and the rules_jvm_external dependency work (bazel-contrib/rules_jvm_external#1357). None has landed, but together they've already established most of the design decisions. I've built a prototype that validates that roadmap and advances past the point where #1351 stalled. Specifically, it solves the "blocking non-hermetic downloads" problem #1466 calls out as painful, and reaches a runnable native binary (cross-compiled Linux ELF, no JVM, run in a bare container).

KMP/Native hasn't been on the core team's roadmap (@Bencodes, 2024: open to community contributions but no team plan). I'm a rules_kotlin maintainer, though to be fair, I've been largely absent for a couple of years. This proposal is part of my return to active maintenance, and it's work I intend to see through. With that on the table, I'm putting KMP/Native on the roadmap by making this bid: this issue (1) consolidates the prior art and the decisions it produced, (2) reports what the prototype demonstrates, and (3) proposes a resolution to the one open API-shape question that #1351 explicitly deferred (the last thing blocking a phased landing), for us to converge on.

This builds directly on @restingbull's roadmap (#1466) rather than competing with it.

1. Prior art and the decisions it already produced

Thread State What it established
#567 Cross-Platform Roadmap open (2021–) The long-standing tracking issue. @Bencodes (2024): no core-team plan to build KMP, but open to community contributions. This issue is me acting on exactly that: it takes over as the active umbrella, with #567 referencing it and superseded once the roadmap here is accepted.
#1347 Re-enable JavaScript closed/stale Decision (@restingbull): start with a Kotlin/Native toolchain and treat JS as an output of the shared IR/klib backend, not a separate JS path. (Extending the same backend to Wasm is my inference, not his.)
#1351 initial native toolchain + kt_library closed/stale The closest prior implementation (@smocherla-brex): a kotlin_native_compiler_repository, a per-(exec,target) kt_native_toolchain, konan.home exposed as a DirectoryInfo artifact, a kt_library rule producing klibs for commonMain-style shared code, provider KtKlibInfo{klibs, transitive_klibs}, integrated into the KotlinBuilder worker (a Platform.NATIVE task executor). Explicit open TODO: "decide if a separate rule is needed for klibs or fit it into existing platform-specific rules."
#1466 [kmp] Outline of a roadmap open @restingbull's plans/kmp.md. Phases: toolchain foundation → rules → dependency resolution. Calls out: klib is the intermediate format (he notes it is probably not fully platform-independent, he hedges); blocking the compiler's runtime downloads is painful; BTAPI is required for long-term maintenance; deps via the Gradle resolver in rules_jvm_external. Includes a hard-won list of compiler flags (see Section 3).
#1481 Build Tools API plan open @agluszak's plan to route compilation through the Kotlin Build Tools API. Names "easier KMP integration (#1466)" as a motivation. This is the intended long-term compiler-integration layer.
kitterion/rules_kotlin_native external A separate community ruleset for K/N by @kitterion; worth reviewing for overlap/ideas.
JS/klib saga: #1185 drop JS, #808 klib deps, #1227 Wasm closed Kotlin 2.0 moved JS/Native artifacts to klib (no more JARs), which is why JS was dropped and why a klib-producing core is the unlock for JS+Native+Wasm.
Toolchain machinery: #1206/#1213 closed define_kt_toolchain already grew exec_compatible_with/target_compatible_with: the per-target toolchain selection KMP needs is already supported.
Deps: bazel-contrib/rules_jvm_external#864, bazel-contrib/rules_jvm_external#989, bazel-contrib/rules_jvm_external#1357 mixed KMP Maven artifacts (Gradle Module Metadata) are an rules_jvm_external concern; @smocherla-brex's bazel-contrib/rules_jvm_external#1357 adds Gradle-resolver support (ArtifactView, defaults to JVM, can select platform variants). Cross-repo coordination needed.

Net of the decisions: native-first; klib as the shared IR artifact (JS/Wasm later);
konan.home as a directory artifact; BTAPI as the eventual compiler API; deps through the
rules_jvm_external Gradle resolver; and an unresolved choice between a dedicated klib
rule vs. folding klibs into the existing platform rules.

2. What the prototype demonstrates (and where it stands vs. the prior art)

Built on master at 84fe8d3d, Bazel 9.1.0, Kotlin/Native 2.3.21. Branch:
cgruber/rules_kotlin@kotlin_native.

  • kt_native_library: .kt → .klib for a konan target, host and cross-target,
    with transitive deps. Verified hermetic: builds under darwin-sandbox with
    --sandbox_default_allow_network=false; the compiler's dependency downloader is never
    invoked for klib compilation (a fresh KONAN_DATA_DIR stays empty).
  • kt_native_binary: links a real native executable (-produce program). The
    ~650–720 MB LLVM/sysroot/cross-toolchain bundle is pre-fetched by a repo rule and
    staged into a sandboxed KONAN_DATA_DIR (regenerating the compiler's .extracted
    marker), so the link runs sandboxed, no network. linux_x64 verified with the
    network disabled; macos_arm64 links in-sandbox using the host Xcode SDK. This is the
    item [kmp] Outline of a roadmap #1466 flags as "this will be painful," demonstrated working.
  • e2e: a Bazel sh_test builds+runs a host binary; a script cross-compiles a
    linux_x64 binary on macOS and runs it in a bare ubuntu container with no JRE: a true
    native ELF, zero JVM references.

This advances past #1351 (which reached klib-only and stalled before hermetic dep
provisioning and binary linking). It is a prototype, though: several things are
deliberate shortcuts that should be replaced before this is the final form. Calling them
out explicitly so the gap to a mergeable design is clear:

  1. konan.home exposure → use DirectoryInfo. I used filegroup(glob(["**"]));
    Add initial kotlin native toolchain and kt_library rule #1351 + [kmp] Outline of a roadmap #1466 expose it as a bazel-skylib DirectoryInfo (passed as a system
    property). The directory artifact is the right answer: it avoids enumerating tens of
    thousands of inputs and is far friendlier to RBE. The prototype's filegroup is a
    shortcut to drop.
  2. Compiler invocation → BTAPI seam. Add initial kotlin native toolchain and kt_library rule #1351 integrated native into the
    KotlinBuilder worker (Platform.NATIVE); I shell konanc via a small launcher. The
    launcher is a deliberate, thin seam that's easy to swap for the BTAPI (Build Tools API plan #1481),
    which [kmp] Outline of a roadmap #1466 says is the long-term target and "not worth undertaking [other
    compiler-integration] effort while pending." I've run this past Eugene
    Zhuravlev (@eugenezh) at JetBrains (their point person for the Bazel Kotlin rules, working
    on the Build Tools API integration), flagging both the wrapped-compiler question and the
    deliberate choice here to keep K/N compiler invocation isolated behind a thin seam while
    the Build Tools API is still stabilizing; he didn't raise concerns. That isolation is the point: it keeps the eventual BTAPI migration a
    localized swap rather than a rewrite, so the two efforts don't have to move in lockstep.
    Whether the interim seam should be the launcher or the worker is still an open call
    (tracked in the Compiler invocation sub-issue, Section 6).
  3. Reproducibility flags → not yet applied. I haven't yet wired [kmp] Outline of a roadmap #1466's
    -Xklib-relative-path-base / -Xdebug-prefix-map (needed for cross-machine remote
    cache hits) or -Xoverride-konan-properties=airplaneMode=true (belt-and-suspenders
    download disable on top of pre-staging). All should go in.
  4. Unpinned dependency hashes → must pin. The konan dist + LLVM/sysroot bundle are
    fetched without sha256 pins in the prototype (fast iteration). The final form needs
    pinned, integrity-checked downloads. Non-negotiable for a hermetic, reproducible repo
    rule.
  5. Single-host coverage → needs the full matrix. Only the macos-aarch64 host dep
    table is filled in; the rule shape is host-generic but the other host entries
    (linux-x86_64, etc.) and their hashes are stubs. Cross-target (--platforms) is
    verified; cross-host provisioning is not yet populated.

In other words: the prototype is empirical confirmation that #1466's plan works, plus a
solution to its hardest Phase-1 item. But it is a prototype, and the items above (plus
the API-shape decision in Section 4) are what stand between it and a form worth merging. It
should be re-based onto #1351/#1466's conventions, not kept as a parallel third thing.

3. The flags/lessons, reconciled

#1466 lists the hard-won compiler flags; the prototype exercised several. Consolidated,
the working recipe is:

  • Distribution: expose konan.home as a DirectoryInfo; pass via system property.
  • Block downloads: pre-stage LLVM/sysroot/cross-toolchain as a repo rule with the
    dependencies/.extracted marker and set
    -Xoverride-konan-properties=airplaneMode=true.
  • Cache isolation: -Xauto-cache-dir / -Xauto-cache-from to keep the compiler's
    internal cache out of the read-only external repo (the prototype hit exactly the
    klib/cache/<target>STATIC write-back [kmp] Outline of a roadmap #1466 predicts; redirecting fixes it).
  • Reproducible klibs (RBE): -Xklib-relative-path-base, -Xdebug-prefix-map;
    explicit stdlib as a declared input (-no-default-libs + explicit -library).
  • Cross-compile: ship a set of platform()s; konan target comes from --platforms

4. The open API-shape question (tracked as a sub-issue)

This is #1351's deferred TODO and the crux. The constraint that decides it concerns how
"common" code can be packaged. Two things observed/known:

  • Platform klibs are target-tagged, directly verified in the prototype: the same
    sources compiled for different targets produce klibs whose manifests read
    native_targets=macos_arm64 vs. native_targets=linux_x64. So a platform klib is not
    a single artifact consumable by every target. ([kmp] Outline of a roadmap #1466 flags this too, though
    @restingbull hedges whether it's still strictly true; the prototype manifests are the
    concrete evidence.)
  • Common/commonMain code is a separate question I have not yet built: KMP models
    it as a metadata klib plus per-target compilations, and expect/actual requires the
    actual to be compiled in the same module as its expect, so "common" is not a
    standalone linked artifact once expect/actual is used; it's source co-compiled per target
    (or a metadata klib). This is unverified in the prototype; it's the KMP-common work
    (phase P6 in Section 5 below).

Given that, the realistic options:

  • (A) Dedicated klib rule. kt_native_library/kt_library produces klibs;
    platform-specific rules consume them. This is what Add initial kotlin native toolchain and kt_library rule #1351 built and what @restingbull's
    native-first steer (Re-enable Javascript support #1347) points to. Clean; covers pure-common well; needs an
    expect/actual story (a metadata klib or a friend-association, and note rules_kotlin
    already has associates).
  • (B) Platform-as-configuration. One rule, backend chosen by --platforms. Most
    Bazel-native; expect/actual "just works" because each platform build co-compiles
    common+platform; but bazel build //:x produces different artifacts by config and it
    fights the rich existing kt_jvm_library surface.
  • (C) Fold into existing rules. The other half of Add initial kotlin native toolchain and kt_library rule #1351's TODO; least new surface,
    but muddies kt_jvm_library.

My current thinking: (A) as the core (it matches the prior decisions and the
klib-not-portable reality), with target selection by --platforms (the configuration
half of (B)), and a later kt_multiplatform_library (or metadata-klib support) for
expect/actual + hierarchical source sets. Naming needs a call too: #1351 used
kt_library for the neutral klib rule; the prototype uses kt_native_library. We should
converge on one. Decision tracked in the sub-issue "API shape + naming" (Section 6).

4a. JVM consumption of a common kt_library (a goal for (A))

A goal for (A): a klib-producing kt_library should be usable as an upstream dependency of
kt_jvm_*
, with the JVM rule getting real JVM artifacts out of it. A klib is never a JVM
classpath input, so this isn't automatic: it needs a JVM facet (the common sources compiled
to bytecode, surfaced as JavaInfo). Pure-common code can do this cleanly (a likely modest
change to kt_jvm_library, which already understands JavaInfo), and it needs no
co-compilation machinery, so it sequences early, right after the klib rule (P2a in Section 5).
It's a cross-rule commitment, so it's tracked as a spike, with mechanics, prototype scope, and
success criteria there: sub-issue "Spike: JVM facet for common kt_library" (Section 6).

A second, related-but-distinct case is defining JVM actuals for common expects (a JVM variant
of an expect-bearing library). Kotlin requires expect and actual in the same module, so
this can't be facet consumption at all: it needs the common expect sources and the JVM actual
sources co-compiled into one artifact. (Friend/associates visibility doesn't satisfy
expect/actual matching, so it's no shortcut.) That shares the co-compilation machinery of the
broader KMP-common work, so it lives in P6, but as P6's first slice, pulled to the front as
far as the machinery preconditions allow (JVM being the cheapest second target). This facet spike
scopes only the pure-common case.

4b. C interop and external native linking (the OpenSSL case)

Consuming a C library (the canonical example is linking OpenSSL into a Linux binary) is on
the roadmap but not built in the prototype, and it's the difference between toy native rules
and ones that build real software. Kotlin/Native does it in two steps: cinterop reads a .def
and emits a target-specific bindings klib (it runs libclang from the heavy konan deps,
so it inherits P3's provisioning, which is why it sequences after P3), then the final link must
reach the actual native library. The open question is where the headers and the compiled lib
come from: a .def+sysroot/prebuilt interim, vs. the CcInfo-bridged final form (a
Bazel cc_library feeding both cinterop and the link, hermetic and cross-target-correct).
I lean toward making the CcInfo-bridged form work, largely for coherence with the rest of
the build: a C dependency should surface the ordinary Bazel way, as a cc_library (wrapping a
prebuilt binary dep, vendored sources, or whatever), carry its headers and linkable artifacts as
CcInfo, and be consumed by a kt_native target like any other dep. One firm constraint on
that: it has to work by consuming cc_common/CcInfo as they stand. If wiring it into
cinterop and the konanc link turns out to require changes to rules_cc itself, that's a
strong signal it's the wrong approach, and we stay on the .def+sysroot/prebuilt interim until
there's a cleaner path. That tension (konanc's self-driven link vs. Bazel cc_common;
cross-target lib availability) is exactly the genuine research item, so it's tracked as a
spike, with full mechanics and success criteria there: sub-issue "Spike: C interop + external
linking via CcInfo"
(Section 6).

4c. Dependency resolution and native Maven coordinates

Pulling KMP/native dependencies in is its own design problem, and it deserves directed
attention rather than being treated as a downstream detail. Maven isn't inherently JVM, but it's
substantially JVM-oriented in practice, and native code surfaces in Maven coordinates in several
ways (KMP's Gradle Module Metadata variants, platform/classifier suffixes like -iosarm64, and
so on). The task is to research the community-standard representations first, then design K/N
dependency consumption to integrate with them, and with @smocherla-brex's Gradle-resolver
work in rules_jvm_external (#1357: ArtifactView, platform-variant selection), rather than
inventing a parallel scheme. Same guardrail as the rest of the interop work: keep it consistent
and "bazely", and push changes upstream (into rules_jvm_external or the resolver) only if we
hit a real design flaw or an unanticipated second-order effect with no clean alternative.
Tracked as a spike, with the #1357 coordination folded in: sub-issue "Spike: native
dependency resolution & Maven coordinates"
(Section 6).

5. Proposed phased plan (reconciled with #1466)

This mirrors #1466's phases; the prototype already has working code for P1–P3.

  • P1: Toolchain foundation. konan dist repo rule (DirectoryInfo), per-(exec,target)
    toolchains, blocking non-hermetic downloads (pre-staged deps + airplaneMode).
    Prototype: working + network-off verified; remaining-for-final: DirectoryInfo (vs.
    the prototype filegroup), pinned dep sha256s, full host matrix (see Section 2).
  • P2: klib rule. kt_native_library (source-only klib, transitive deps),
    reproducibility flags, cache isolation. Prototype: done; add -Xklib-relative-path-base/-Xdebug-prefix-map.
  • P2a: JVM facet for pure-common code. The common rule also emits a JVM facet (a JAR
    surfaced as JavaInfo), so a kt_jvm_* target can depend on a common kt_library and get
    real JVM artifacts out of it (Section 4a). No expect/actual and no co-compilation
    involved, so this needs none of the P6 machinery: it can land as soon as the common rule
    exists and the API shape is settled, which is why it sits here rather than with the
    KMP-common work. Prototype: not built; tracked as the JVM-facet spike (Section 6).
  • P3: Binary linking. kt_native_binary (executables; later static/dynamic/
    framework). Prototype: linux_x64 + macos_arm64 working; Apple frameworks/xcframework
    later (macOS-host-locked).
  • P4: C interop & external native linking. A kt_native_cinterop rule (.def →
    bindings klib via the bundled cinterop/libclang) plus wiring external native libraries
    into the kt_native_binary link: the OpenSSL case (Section 4b). Sequenced here, right after
    binary linking and ahead of the JS/Wasm and KMP expansions, because real native binaries
    need it sooner. Prototype: not built; design in Section 4b. Interim .def+sysroot/prebuilt;
    the CcInfo-bridged, cross-target form is the open research item.
  • P5: JS/Wasm as outputs of the same klib core (closes the Re-enable Javascript support #1347/Drop support for JavaScript #1185 loop).
  • P6: KMP common / expect-actual. First slice, pulled to the front of the phase as far
    as the machinery preconditions allow: the JVM variant of an expect-bearing common
    library
    (common expect + JVM actual co-compiled in one module), since JVM is the
    cheapest second target (no native provisioning, no metadata-klib work). Then the general
    case: metadata klibs, hierarchical source sets, native/JS actuals (see Section 4a). The
    hardest phase overall, gated on the K2 metadata-compilation flags.
  • P7: Dependency resolution via the rules_jvm_external Gradle resolver
    (Add support for gradle resolver rules_jvm_external#1357); add kotlin_native_library support there.
    Native-in-Maven representation and the community standards for it need their own research
    pass (Section 4c). Coordinate w/ @smocherla-brex.
  • P8: BTAPI migration (Build Tools API plan #1481): swap the konanc launcher for the Build Tools API
    once its native facade ships with published coordinates. A follow-up after the swap: drop
    the interim seam and integrate more deeply wherever the performance gains justify it.

Definition of done (the acceptance bar): a set of real examples build green in CI: an
iOS Compose-Multiplatform + Metro app, the same as a macOS desktop app, a Linux binary
linking a C library (e.g. OpenSSL) via cinterop, a Windows native artifact, and a shared
KMP library consumed across JVM + native.

6. Sub-issues (tracked work)

This issue is the umbrella (with #567 referencing it, and superseded by it if this roadmap
is accepted). The open decisions and spikes are broken out as child issues so each can be
discussed and closed independently; this issue tracks them. (Numbers filled in once the
children are filed.)

Decisions / discussion

Spikes / research

I'll drive this: split it into reviewable PRs and keep it aligned with #1466 and #1481. The
prototype branch is up at cgruber/rules_kotlin@kotlin_native if anyone wants to walk
through it. Let's land the approval and the API-shape call first; those unblock the rest.


Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions