Proposal: Efficient Collection API for Build Pipelines with Matrices

[nrdxp]

TL;DR

Proposing a new Collection API for atom's module system to optimize build pipelines and separate concerns between Nix as an evaluator and as a builder.

Background

While Nix has been used for various purposes with extensive library code available, this can impede the construction of an efficient build pipeline. Nix, as a builder, is primarily concerned with building derivations, not with complex evaluation processes.

The atom module system aims to provide a clean separation of concerns, allowing user-facing tools (eka) to decouple previously tightly coupled components. In other words, this approach recognizes the potential conflict between Nix as a general-purpose evaluator and Nix as a builder, and tries to draw a clear boundary that enables a bounded collection of requested builds (a straight list).

Proposed Solution

We propose a new format for specifying builds, requesting CI runs, or any context where efficient build pipeline execution is crucial:

[ <drv>? <matrix>? ] # any number of derivations or matrices of derivation variants

Where a matrix would look like:

{
  # build:?host:?target:?<arbitrary-dimensions>
  x86_64-linux.aarch64-linux.static = drv;
  x86_64-linux.x86_64-linux.static = drv;
}

Benefits

1. Efficient Evaluation: By gathering derivations and matrices into a single list, we can reduce evaluation costs by sharing evaluation dependencies.
2. Batch Processing: We can evaluate whole matrices of derivation variants together and send requests to the daemon in batches.
3. Flexibility: Matrices can be made sparse to disable broken or unsupported entries while maintaining a well-ordered set.

Implementation Details

The module system will avoid the combinatorial explosion of nixpkgs and flakes by taking matrix variants as inputs rather than having duplicate outputs per system. This allows for easy evaluation of arbitrary matrices with minimal effort.

Implementation is straightforward with atoms, as we are guaranteed to have a hierarchical key space. We can implement such a collection as a simple map over the keys passed on the CLI into a list.

Comparison with Current Approach

Currently, when calling nix build .#foo .#bar .#baz, Nix initiates a separate evaluation from scratch for each output. Our proposed approach evaluates these as a single value, potentially reducing evaluation cost by sharing work between common dependencies.

Examples

A rough draft of what this might look like on the manifest side can be found here.

Potential Challenges

• Educating users on the new API and its benefits

Next Steps

1. Refine the matrix specification format
2. Develop a prototype implementation
3. Benchmark performance against current Nix evaluation methods

Request for Feedback

We welcome input on:

1. The proposed matrix format
2. Potential use cases we may have overlooked
3. Suggestions for improving the API's usability