feature - Preheat generated library dependencies into the build cache

[dannymeijer]

Area

• Tooling (CLI/formatter/test runner)

Problem statement

Generated library builds can spend an unacceptable amount of time compiling large Rust dependency graphs even when those dependencies have not changed. A concrete case is an InQL package that depends on heavy crates such as DataFusion and DataFusion Substrait: incan build --lib may sit silently for a long time while Cargo compiles the generated crate's release dependency graph.

Incan already has dependency preheat machinery during lock generation, but it does not currently warm the same cache domain that incan build --lib uses. The lock preheat path builds the generated lock workspace with cargo test --no-run into the project target/ directory, while generated library builds use cargo build --release with the parent-scoped generated Cargo target directory (target/.cargo-target). As a result, the existing preheat does not reliably remove the cold DataFusion/Substrait build cost from the library build path.

This is not blocking the 0.3 release, but it is a good candidate for the next Incan release because it materially improves package-build ergonomics for projects with stable heavy Rust dependencies.

Proposed solution

Add a generated-library dependency preheat path that compiles the lock workspace into the same Cargo target directory and profile used by incan build --lib.

The intended shape is:

• Reuse the generated lock workspace dependency graph from target/incan_lock/Cargo.toml.
• Preheat with the same Cargo policy flags and feature selection that the real build will use.
• Use the same generated Cargo target directory as ProjectGenerator::build() for library builds.
• Use the release profile when the subsequent library build uses cargo build --release.
• Fingerprint the lock workspace, cargo flags, selected profile, and target directory so unchanged dependency graphs are not rebuilt.
• Coordinate concurrent preheat attempts with the existing cooperative lock/stamp pattern.

After this, a project with heavy but unchanged Rust dependencies should be able to run lock/preheat once and have subsequent incan build --lib invocations reuse the warmed Cargo artifacts.

Alternatives considered

• Split heavy adapter dependencies out of downstream packages. That is still architecturally useful for package authors, but it does not solve the general Incan tooling problem: any generated library with stable heavy Rust dependencies can hit the same cold-build cost.
• Rely only on external CI cache configuration. CI caching helps, but Incan should still align its own preheat target/profile with the generated library build path so local and CI workflows can benefit consistently.
• Only stream Cargo output during incan build --lib. Better output would improve perceived hangs, but it does not reduce the actual repeated compile cost. Build-output streaming can be tracked separately.
• Do nothing. That keeps cold builds expensive and leaves existing preheat machinery ineffective for the generated library release path.

Scope / acceptance criteria

• In scope:

  • Add or extend dependency preheat so generated library builds can reuse warmed Cargo artifacts.
  • Align preheat target directory and profile with the generated library cargo build --release path.
  • Include cargo policy flags, feature selection, lock inputs, profile, and target directory in the preheat fingerprint.
  • Preserve or extend existing env-based controls for disabling preheat where needed.
  • Add tests for target/profile alignment and stamp reuse.

• Out of scope:

  • Changing package semantics or dependency resolution rules.
  • Moving downstream projects' heavy dependencies into separate packages.
  • Full build-output streaming for incan build --lib.

• Done when:

  • A project with stable heavy [rust-dependencies] can preheat once and then run incan build --lib without recompiling that full dependency graph.
  • Re-running the same build with unchanged lockfile/features/profile reuses the preheat stamp.
  • Changing relevant dependency inputs invalidates the stamp and recompiles as expected.
  • Existing test and script preheat behavior does not regress.