[WSLC] (1/4) Move the lifecycle daemon off the create-and-run image store

docs/backends/wslc/wsl-container-getting-started.md

@@ -131,6 +131,14 @@ Or pull a single image directly:
     --setup-wslc --image alpine:latest
 ```
 
+Repeat `--image` to pull several in one go, which opens each store once instead
+of once per image:
+
+```powershell
+.\src\target\x86_64-pc-windows-msvc\release\wxc-exec.exe `
+    --setup-wslc --image alpine:latest --image python:3.12-alpine
+```
+
 Pulled images persist in the cache until you remove them — pay the
 cost once per image, not once per run.
 
@@ -141,6 +149,15 @@ cost once per image, not once per run.
 > `wxc-exec.exe`), otherwise the runner will not find what you just
 > pulled and will pull it again under its own path.
 
+> **Container lifecycle operations use a different store.** The lifecycle
+> daemon leases its store from a pool under
+> `%LOCALAPPDATA%\mxc\wslc-store-slots` (or `%TEMP%\mxc\wslc-store-slots` when
+> `%LOCALAPPDATA%` is unresolvable), which `-StoragePath` does not reach.
+> Warm it with `setup-wslc.ps1 -DaemonSlots 1`, or `--wslc-daemon-slots 1` on
+> `wxc-exec.exe`; both default to 0, and each slot warmed costs another pull
+> and another multi-GiB store. See
+> [the lifecycle guide](wslc-state-aware.md#image-store).
+
 > **Bring-up reaches the network.** A cache miss makes the host fetch
 > from the image's registry before the container starts. That fetch is
 > outside the sandbox's own network policy, so a config declaring
@@ -213,6 +230,25 @@ prepare a machine that will run offline — warm the cache first:
 .\scripts\setup-wslc.ps1 -Image <image>
 ```
 
+### How warming works
+
+A store is a directory holding a `storage.vhdx`, and the SDK offers no way to
+write into one directly — a pull needs a live session. So warming opens a
+short-lived WSLc session against each target store, pulls every requested image
+into it, and releases it. The session is only a way to reach the file; what
+persists is the VHDX, which a later run or a lifecycle daemon reads.
+
+One session is opened per *store*, not per image, so warming three images into
+two stores costs two sessions rather than six. Passing every image to one
+invocation is therefore cheaper than invoking the tool once per image:
+
+```powershell
+.\scripts\setup-wslc.ps1 -Image alpine:latest, python:3.12-alpine -DaemonSlots 1
+```
+
+That warms two stores — the create-and-run store and lifecycle slot 0 — with
+both images, booting two utility VMs in total.
+
 ## Usage
 
 ### TypeScript SDK
@@ -312,7 +348,7 @@ WSLC-specific settings go under `wslc` in the JSON config:
 | `cpuCount` | number | Host default | Number of CPU cores for the container |
 | `memoryMb` | number | Host default | Memory limit in MB |
 | `gpu` | boolean | `false` | Enable GPU passthrough |
-| `storagePath` | string | System default | Host path for container storage (VHD) |
+| `storagePath` | string | System default | Host path for container storage (VHD). Create-and-run execution only |
 | `imageTarPath` | string | — | Path to a local tar file to import as the image |
 | `portMappings` | array | `[]` | Host→container TCP forwards; each entry takes `windowsPort` and `containerPort` |
 
@@ -350,8 +386,11 @@ The separate `provision` operation takes the same list under
 
 The lifecycle daemon creates one container per sandbox, so a forward belongs
 to the sandbox that declared it. The session-wide `cpuCount` / `memoryMb` /
-`gpu` / `storagePath` settings remain available only for create-and-run execution, because that daemon shares a
-single WSL session across every sandbox.
+`gpu` settings remain available only for create-and-run execution, because that daemon shares a
+single WSL session across every sandbox. `storagePath` is create-and-run-only
+for a different reason: the daemon leases its store from a slot pool rather
+than taking a caller-named path, so warming its cache is a separate step. See
+[the lifecycle guide](wslc-state-aware.md#image-store).
 
 ### Image sources
 
@@ -633,7 +672,8 @@ images — cannot be used.
 | `Failed to load wslcsdk.dll` | DLL not in same directory as `wxc-exec.exe` | Copy `wslcsdk.dll` next to the binary |
 | `WSLC runtime unavailable` | WSL runtime package is missing, older than 2.9.9, or the Virtual Machine Platform optional component is disabled | Update WSL with `wsl --update --pre-release`, verify the installed version with `wsl --version`, and enable the Virtual Machine Platform optional component if required. The WSLC SDK DLL is a separate dependency and does not replace the WSL runtime package. |
 | `WSLC runtime unavailable. Missing components: SdkNeedsUpdate` | The opposite direction: your installed WSL is **newer** than the WSLc SDK this MXC build ships (specified by `WSLC_SDK_VERSION` in `src/mxc-sdk/build/build_wslc_common.rs`) | Update MXC to a build that includes a newer SDK. Do **not** update WSL — it is already ahead, and updating it further will not clear this. |
-| `WSLC image '<name>' is not cached, and this sandbox declares no egress` | An isolated config named an image the store does not have | Warm the cache with `--setup-wslc`, set `imageTarPath`, or allow egress |
+| `WSLC image '<name>' is not cached, and this sandbox declares no egress` | An isolated config named an image the store does not have | Warm the cache with `--setup-wslc`, set `imageTarPath`, or allow egress. For a lifecycle operation the store is a daemon slot, so warm it with `setup-wslc.ps1 -DaemonSlots 1` |
+| `WSLc store unavailable: the image store at <path> is already held by another live session` | Two processes reached the same store, which admits one live session | Wait for the holder to release it. A running `wxc-wslc-daemon.exe` releases its slot after `MXC_WSLC_DAEMON_IDLE_TIMEOUT_SECS` (default 300s); a create-and-run execution releases its store when the run ends |
 | `WSLC image '<name>' cannot be pulled: '<host>' is not in the administrative registry allowlist` | Machine policy restricts which registries may be used | Use a permitted registry, set `imageTarPath`, or ask an administrator to widen `WslcAllowedImageRegistries` |
 | `WSLC image '<name>' did not finish pulling within <n>s and was stopped` | The pull exceeded its budget and was aborted | Retry, raise `MXC_WSLC_PULL_TIMEOUT_SECS`, or warm the cache from a faster network |
 | `WSLC image '<name>' did not finish pulling within <n>s. The transfer was abandoned` | The registry stopped responding, so the pull was given up on rather than ended | Retry, or warm the cache from a faster network. For a separate `provision` operation, raise `MXC_WSLC_DAEMON_CALL_TIMEOUT_SECS` alongside `MXC_WSLC_PULL_TIMEOUT_SECS`, since the pull must finish before the daemon-call deadline |

docs/backends/wslc/wslc-state-aware.md

@@ -55,6 +55,7 @@ waits for the run, because deleting the container would free a handle the run is
 | Lifecycle backend | `src/mxc-sdk/src/backends/wslc/common/state_aware.rs` (`WslcStateAwareRunner`) | Translates runtime `WslcProvisionConfig` and shared policy fields into daemon protocol frames; implements `StatefulSandboxBackend` (`ID_PREFIX`/`BACKEND_KEY` = `wslc`). |
 | Policy support checks | `src/mxc-sdk/src/backends/wslc/common/policy.rs` | Per-phase validation of which policy fields are enforced or rejected. |
 | Daemon client | `src/mxc-sdk/src/backends/wslc/common/daemon_client.rs` | Discovers / spawns the daemon, connects the control pipe, sends `DaemonRequest` frames, reads responses; typed `DaemonError`. |
+| Store slot pool | `src/mxc-sdk/src/backends/wslc/common/store_slots.rs` | Owns the daemon's image stores: slot paths and leasing. See [Image store](#image-store). |
 | Daemon | `src/mxc-sdk/src/bin/wslc_daemon/` (`wxc-wslc-daemon.exe`) | Long-lived host process holding `WslcSession` / `WslcContainer`; worker thread drives the SDK; idle-timeout watchdog tears the session down when unused. |
 | Engine routing | `src/mxc-sdk/src/core/mxc_engine/state_aware.rs` | Selects the WSLc lifecycle backend (Windows + `wslc` feature). |
 | Prefix registration | `src/mxc-sdk/src/tools/mxc_common/state_aware_dispatch.rs` (`backend_from_prefix`) | Maps the `wslc:` id prefix back to the WSLc backend for post-provision phases. |
@@ -99,8 +100,10 @@ The field requires development contract `1.1.0-alpha`. Requests declaring
 `0.9.0-alpha` cannot include it and must select `1.1.0-alpha` explicitly.
 The daemon shares one session (VM) but creates a separate container for each
 provision, so a mapping applies only to the container that declared it, unlike
-the session-wide `cpuCount` / `memoryMb` / `gpu` / `storagePath` knobs that
-remain available only for create-and-run execution.
+the session-wide `cpuCount` / `memoryMb` / `gpu` knobs that
+remain available only for create-and-run execution. `storagePath` is not among them either way:
+the daemon leases its store from the slot pool rather than taking a caller-named
+path (see [Image store](#image-store)).
 
 The create-and-run `wslc.portMappings` list and `wslc.provision.portMappings` share
 one duplicate check, so two entries claiming the same `windowsPort` are
@@ -130,6 +133,80 @@ as `--container-id`; the dispatcher derives the backend from the `wslc:` prefix
 | stop | `WslcStopContainer`. |
 | deprovision | `WslcDeleteContainer`; release the session + SDK when the last container is gone (daemon may then exit / idle-time out). |
 
+## Image store
+
+A WSLc store is a host directory handed to `WslcCreateSession`; the SDK creates
+`storage.vhdx` inside it and mounts that VHDX into the session's WSL2 utility
+VM, where it holds the session's pulled images. A warmed store runs to a couple
+of GiB.
+
+The daemon leases its store from a pool rather than taking a caller-named path:
+
+```text
+%LOCALAPPDATA%\mxc\wslc-store-slots\slot-0, slot-1, …
+```
+
+`%TEMP%\mxc\wslc-store-slots` is used instead when `%LOCALAPPDATA%` is
+unresolvable. A session leases the lowest free slot when it boots and returns it
+when it is released, so a daemon holding one session always runs on `slot-0`.
+Create-and-run execution keeps its own default of `%TEMP%\mxc-wslc-sessions` and
+never touches the pool.
+
+Both the daemon and the command that warms the pool resolve these variables from
+their own environment, so warm the pool as the user the daemon runs as. A
+refusal names the directory it needs filled, so a warm that reports success
+against a different directory has filled the wrong store.
+
+Separating the two is what lets both surfaces run at once. A store's VHDX admits
+one live session, so while the daemon held `%TEMP%\mxc-wslc-sessions` every
+create-and-run WSLc execution on the same host failed with
+`WslcListSessionImages failed: HRESULT 0x80070020` until the daemon idled out.
+
+The pool does not cap how many sessions a host may run, and a slot count would
+be the wrong control for that: a session carries its own cpu and memory, so the
+same count can mean very different host load. Admission belongs with the work
+that lets a caller request a second session.
+
+### Slot contents and disk
+
+A returned slot keeps its images — that is the point of leasing rather than
+minting. A slot that has never been leased holds nothing at all, and an uncached
+image under the isolated posture is refused before the VM is reached, so a cold
+slot fails a provision outright rather than pulling. Warm the pool with either:
+
+```powershell
+.\scripts\setup-wslc.ps1 -Image alpine:latest -DaemonSlots 1
+wxc-exec.exe --setup-wslc --image alpine:latest --wslc-daemon-slots 1
+```
+
+Both default to 0 and warm slots from 0 up, so `1` is what a daemon holding one
+session needs. Each store is opened once and filled with every image given, so
+pass them all to one invocation. See
+[the getting-started guide](wsl-container-getting-started.md#how-warming-works)
+for how warming works, and
+[Step 3](wsl-container-getting-started.md#step-3--container-images-optional-pre-pull)
+for the create-and-run store, which these do not replace.
+
+A slot directory is created when it is first leased, so an unused index costs
+nothing. Slots are identified by index and never repurposed, so nothing
+accumulates beyond the number of sessions the host has actually run at once.
+Reclaim the disk by deleting the pool directory while no daemon is running.
+
+### Store exclusivity
+
+The SDK does not report a store conflict at create time: `WslcCreateSession`
+against a store another live session holds returns a healthy-looking handle, and
+the violation appears only on first use. The daemon therefore probes the store
+itself immediately after create (`WslcListSessionImages`) and treats a failure
+as a *create* failure, so a conflict can never mint a sandbox id that dies at an
+arbitrary later phase.
+
+Lease tracking covers one process, so two sessions inside one daemon can never
+be handed the same slot. A holder outside that bookkeeping — typically a daemon
+from a previous build still inside its idle window — fails the provision with an
+error naming the slot, rather than relocating the session onto a cold slot and
+reporting a missing image instead.
+
 ### exec output semantics
 
 `provision` / `start` / `stop` / `deprovision` return a JSON object containing `result` or `error` on stdout.
@@ -303,7 +380,7 @@ it can observe idle-teardown within seconds.
 ## Testing
 
 `tests/scripts/run_wslc_state_aware_tests.ps1` is the multi-invocation E2E harness (requires a WSL2
-host that can reach a registry or already has the image cached, and `wxc-wslc-daemon.exe` staged next to `wxc-exec.exe`). It exercises
+host that can reach a registry or already has the image cached in daemon store slot 0, and `wxc-wslc-daemon.exe` staged next to `wxc-exec.exe`). It exercises
 core lifecycle, warm-reuse (a marker written by one `exec` is read back by a separate `exec`
 process — only possible if the container stayed warm), filesystem volumes, bridged networking +
 proxy, validation rejections, exec concurrency, and idle teardown. Fixtures live in

scripts/setup-wslc.ps1

@@ -9,16 +9,21 @@
     Warms the WSLC SDK image cache ahead of time. A run pulls on a cache miss
     by itself, so this is an optimisation rather than a prerequisite — it moves
     the download off the critical path of a later run, and it is how you
-    populate a cache for a host that cannot reach a registry. Each image is
-    pulled via `wxc-exec.exe --setup-wslc --image <name>`, which opens a minimal
-    WSLC session against the configured `storage_path` and invokes
-    `WslcPullSessionImage`. The pulled images persist in that storage path
-    and become visible to subsequent runtime executions.
+    populate a cache for a host that cannot reach a registry. All images are
+    pulled via a single `wxc-exec.exe --setup-wslc --image <name> ...`
+    invocation per store, which opens one WSLC session against that store and
+    invokes `WslcPullSessionImage` for each image. The pulled images persist in
+    that storage path and become visible to subsequent runtime executions.
 
     The storage path you pass here MUST match the value used at run time
     (the `wslc.storagePath` field of the config, or the
     runner's default of `%TEMP%\mxc-wslc-sessions` when omitted).
 
+    Container lifecycle operations do not read that store. The lifecycle
+    daemon leases its store from a pool under
+    %LOCALAPPDATA%\mxc\wslc-store-slots, so -DaemonSlots warms those slots
+    separately.
+
 .PARAMETER Image
     One or more image references to pre-pull. Defaults to a small
     starter set (alpine:latest, python:3.12-alpine) suitable for a quick
@@ -33,13 +38,25 @@
 .PARAMETER StoragePath
     WSLC storage path to populate. When omitted, the runner default
     (`%TEMP%\mxc-wslc-sessions`) is used. Set this if your runtime configs
-    override `wslc.storagePath`.
+    override `wslc.storagePath`. This reaches create-and-run execution only;
+    use -DaemonSlots for the container-lifecycle store.
+.PARAMETER DaemonSlots
+    How many container-lifecycle store slots to warm as well, from slot 0 up.
+    The lifecycle daemon leases its store from a separate pool under
+    %LOCALAPPDATA%\mxc\wslc-store-slots, which -StoragePath does not reach, and
+    a slot that has never been leased holds no images. Defaults to 0, matching
+    `wxc-exec.exe --wslc-daemon-slots`, so warming a create-and-run cache stays
+    one pull per image. Pass 1 before running container lifecycle operations
+    against an isolated posture; each slot costs another pull and another
+    multi-GiB store.
 
 .PARAMETER DebugLogs
     Enable verbose logging from wxc-exec (passes `--debug`).
 
 .PARAMETER Force
-    Continue pulling subsequent images even if one fails.
+    Retained for compatibility. Every image is now pulled in one `wxc-exec`
+    invocation per store, which always attempts them all and reports each
+    failure, so there is no longer a run to continue past.
 
 .EXAMPLE
     .\setup-wslc.ps1
@@ -52,13 +69,19 @@
 .EXAMPLE
     .\setup-wslc.ps1 -StoragePath C:\wslc-cache
     Pulls into a non-default cache directory.
+
+.EXAMPLE
+    .\setup-wslc.ps1 -DaemonSlots 1
+    Also warms the store slot the container-lifecycle daemon leases.
 #>
 
 [CmdletBinding()]
 param(
     [string[]]$Image = @("alpine:latest", "python:3.12-alpine"),
     [string]$WxcExecPath,
     [string]$StoragePath,
+    [ValidateRange(0, 64)]
+    [int]$DaemonSlots = 0,
     [switch]$DebugLogs,
     [switch]$Force
 )
@@ -99,37 +122,44 @@ if ($StoragePath) {
 } else {
     Write-Host "Storage path : (default) %TEMP%\mxc-wslc-sessions" -ForegroundColor Gray
 }
+if ($DaemonSlots -gt 0) {
+    Write-Host "Daemon slots : $DaemonSlots (%LOCALAPPDATA%\mxc\wslc-store-slots)" -ForegroundColor Gray
+} else {
+    Write-Host "Daemon slots : none" -ForegroundColor Gray
+}
 Write-Host "Images       : $($Image -join ', ')`n" -ForegroundColor Gray
 
 $failed = @()
+Write-Host "  pulling $($Image.Count) image(s) ... " -NoNewline
+
+$wxcArgs = @("--setup-wslc")
 foreach ($img in $Image) {
-    Write-Host "  $img ... " -NoNewline
-    $wxcArgs = @("--setup-wslc", "--image", $img)
-    if ($StoragePath) {
-        $wxcArgs += @("--storage-path", $StoragePath)
-    }
-    if ($DebugLogs) {
-        $wxcArgs += "--debug"
-    }
+    $wxcArgs += @("--image", $img)
+}
+if ($StoragePath) {
+    $wxcArgs += @("--storage-path", $StoragePath)
+}
+if ($DaemonSlots -gt 0) {
+    $wxcArgs += @("--wslc-daemon-slots", "$DaemonSlots")
+}
+if ($DebugLogs) {
+    $wxcArgs += "--debug"
+}
+
+$prevPref = $ErrorActionPreference
+$ErrorActionPreference = "Continue"
+$output = & $WxcExec @wxcArgs 2>&1 | Out-String
+$exitCode = $LASTEXITCODE
+$ErrorActionPreference = $prevPref
 
-    $prevPref = $ErrorActionPreference
-    $ErrorActionPreference = "Continue"
-    $output = & $WxcExec @wxcArgs 2>&1 | Out-String
-    $exitCode = $LASTEXITCODE
-    $ErrorActionPreference = $prevPref
-
-    if ($exitCode -eq 0) {
-        Write-Host "OK" -ForegroundColor Green
-    } else {
-        Write-Host "FAIL (exit $exitCode)" -ForegroundColor Red
-        $output -split "`n" | Where-Object { $_.Trim() -ne "" } | ForEach-Object {
-            Write-Host "    > $($_.TrimEnd())" -ForegroundColor Gray
-        }
-        $failed += $img
-        if (-not $Force) {
-            break
-        }
+if ($exitCode -eq 0) {
+    Write-Host "OK" -ForegroundColor Green
+} else {
+    Write-Host "FAIL (exit $exitCode)" -ForegroundColor Red
+    $output -split "`n" | Where-Object { $_.Trim() -ne "" } | ForEach-Object {
+        Write-Host "    > $($_.TrimEnd())" -ForegroundColor Gray
     }
+    $failed += $Image
 }
 
 Write-Host ""