Skip to content

feat(microvm): cortex-cuda as delivered guest of cortex (fleet participation) - #111

Open
sini wants to merge 1 commit into
mainfrom
feat/delivered-child-host
Open

feat(microvm): cortex-cuda as delivered guest of cortex (fleet participation)#111
sini wants to merge 1 commit into
mainfrom
feat/delivered-child-host

Conversation

@sini

@sini sini commented Jun 11, 2026

Copy link
Copy Markdown
Owner

Summary

Rewire cortex-cuda from an isolated-resolve guest to a microvm guest of cortex. cortex declares guests.cortex-cuda; the guest authors honest nixos and is routed into cortex.microvm.vms.cortex-cuda.config with no standalone nixosConfigurations.cortex-cuda output (intoAttr = []). The guest now participates in the fleet — agenix (host key retargeted to cortex's identity) and core.users fire inside it — while the GPU passthrough/vfio/arbitration overlay is layered host-side by virtualization/microvm-guests.nix (facter devices, vfio ExecCondition gates, gpu-claims producer, CUDA microvm.pkgs, root keys).

The guests policy (host.guests option, the isolated guest entity kind, the resolve + route + isolation delivery) lives here in nix-config (modules/den/aspects/virtualization/guests.nix), composed entirely from public den primitives. It is a consumer pattern with a single (microvm) use case — not den-core machinery — so it lives with its consumer rather than inflating every den user's surface.

Guest-config reconciliation with the now-participating fleet baseline:

  • neutralize the tailscale the fleet users drag in via roles.default (this headless GPU inference VM is reached via the host on the internal bridge, not the tailnet, and the user-scope tailscale age-secret isn't collected for a delivered guest);
  • mkForce system.stateVersion and openssh PermitRootLogin (guest pins its own vs the fleet default/baseline);
  • enable programs.zsh so the participating users' login shells resolve.

Dependency

Requires den @ e8876f3e (flake.lock bumped here), which carries only the genuine engine primitives:

  • route reinstantiate flag — the delivery target (microvm.vms.<n>.config) re-instantiates the collected nixos as its own NixOS system, so the route must deliver the keyed module wrappers verbatim rather than pre-evaluating them into resolved config (which stripped base-module defaults + the dedup keys). Covered by a standalone route test.
  • projected hasAspect keyed by entity id_hash — the per-scope path-set that backs in-context hasAspect is re-keyed by context-free entity identity instead of a scope-string. The set is produced by an entity's standalone resolve (no ancestors) but consumed in the fleet resolve (host nested under environment/fleet); keying by id_hash (kind+name, stable across both) removes the ancestor-stripping reconciliation entirely, so e.g. agenix's hasAspect core.impermanence resolves true on the axon/cortex hosts and identityPaths point at /persist/etc/ssh.

The delivered-child policy that previously lived in den core has been removed from den — it moved here as the guests policy.

Test plan

  • agenix age.secrets + core.users fire inside the guest (no eval throw)
  • GPU parity is byte-identical to the pre-refactor baselinenix-diff of the cortex toplevel shows the only delta is the flake-source store path that any repo edit rehashes; the vfio-gate derivation is identical
  • cortex toplevel builds; guest microvm.vms.cortex-cuda re-instantiates
  • no nixosConfigurations.cortex-cuda output
  • axon + cortex identityPaths = /persist/etc/ssh
  • den CI 915/915; nix fmt clean

@sini
sini force-pushed the feat/delivered-child-host branch 2 times, most recently from 3cc482c to c7cff18 Compare June 12, 2026 02:38
@sini sini changed the title feat(microvm): cortex-cuda as delivered child host (fleet participation) feat(microvm): cortex-cuda as delivered guest of cortex (fleet participation) Jun 12, 2026
…ipation)

Rewire cortex-cuda from an isolated-resolve guest to a microvm guest of
cortex: cortex declares `guests.cortex-cuda`, the guest authors honest `nixos`
and is routed into `cortex.microvm.vms.cortex-cuda.config` with no standalone
nixosConfigurations output (intoAttr = []). The guest now PARTICIPATES in the
fleet — agenix (host key retargeted to cortex's identity) and core.users fire
inside it — while the GPU passthrough/vfio/arbitration overlay is layered
host-side by virtualization/microvm-guests.nix (facter devices, vfio
ExecCondition gates, gpu-claims producer, CUDA microvm.pkgs, root keys).

The guests policy itself (host.guests option, the isolated `guest` entity
kind, the resolve+route+isolation delivery) lives here in nix-config
(modules/den/aspects/virtualization/guests.nix), composed entirely from public
den primitives — it is a consumer pattern with a single (microvm) use case,
not den-core machinery.

Depends on den @ e8876f3e, which carries only the genuine engine primitives:
  - route `reinstantiate` flag: the delivery target re-instantiates the
    collected nixos, so it must receive keyed module wrappers verbatim, not
    pre-evaluated resolved config;
  - projected hasAspect keyed by entity id_hash: the per-scope path-set is
    re-keyed by context-free entity identity instead of a scope-string, so the
    in-context hasAspect needs no ancestor reconciliation — e.g. agenix's
    `hasAspect core.impermanence` resolves true on the axon/cortex hosts and
    identityPaths point at /persist/etc/ssh again.

Guest-config reconciliation with the now-participating fleet baseline:
- neutralize the tailscale the fleet users drag in via roles.default — this
  headless GPU inference VM is reached via the host on the internal bridge, not
  the tailnet, and the user-scope tailscale age-secret isn't collected for a
  delivered guest;
- mkForce system.stateVersion and openssh PermitRootLogin (guest pins its own
  vs the fleet default/baseline);
- enable programs.zsh so the participating users' login shells resolve.

Verified: agenix age.secrets + core.users fire (no throw); GPU parity is
byte-identical to the pre-refactor baseline (nix-diff: the only delta is the
flake-source store path that any repo edit rehashes; the vfio-gate derivation
is identical); cortex toplevel builds; no cortex-cuda flake output;
axon-01/02/03 + cortex identityPaths = /persist/etc/ssh.
@sini
sini force-pushed the feat/delivered-child-host branch from c7cff18 to 9beec57 Compare June 12, 2026 15:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant