CI Validation Infra: fixes for WSLC test prereqs, Windows log collection, LXC bridge NAT, and rate limits

.github/copilot-instructions.md

@@ -72,9 +72,10 @@ workflows) before calling the matrix job.
 
 - `scripts/ci/validation-test-matrix.json` is the catalog: `platforms` (each
   with per-architecture target/artifact/1ES pool and the backends that platform
-  supports) and `triggers` (which OS/backend pairs each plan runs). The
-  `triggers` keys *are* the plan list — the resolver reads them at run time, so
-  adding a plan needs no script change.
+  supports), `triggers` (which OS/backend pairs each plan runs), and the
+  optional `backendDelayedStart` (per-backend job-start stagger, in seconds).
+  The `triggers` keys *are* the plan list — the resolver reads them at run time,
+  so adding a plan needs no script change.
 - `scripts/ci/resolve-validation-test-matrix.mjs` validates that catalog and
   expands a plan (currently `pr`, `nightly`, `weekly`, `enabled`) into GitHub
   Actions matrices. It rejects an invalid catalog before any specialized test
@@ -95,7 +96,9 @@ the matrix `backend` id to the repository's existing backend suite. Ids that
 share a suite get their own case (`process-t1` and `process-t3` both run
 `WinProcessContainer-Tests.ps1`, which derives the tier it expects from the
 host's own `--probe`). A backend with no wired suite fails loudly rather than
-reporting a false success.
+reporting a false success. The Windows dispatcher points `TEMP` at
+`$RUNNER_TEMP` before running a suite, so anything a test writes to the temp
+directory is picked up by the job's log upload without per-file CI wiring.
 
 ### Individual components
 

.github/workflows/Validation.Tests.Matrix.Job.yml

@@ -38,7 +38,7 @@ jobs:
       fail-fast: false
       matrix: ${{ fromJSON(needs.resolve.outputs.windows) }}
     runs-on: [ self-hosted, "1ES.Pool=${{ matrix.pool }}", "JobId=mxc-e2e-${{ github.run_id }}-${{ github.run_number }}-${{ github.run_attempt }}" ]
-    timeout-minutes: 60
+    timeout-minutes: 180
     steps:
       - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
 
@@ -48,6 +48,17 @@ jobs:
           name: ${{ matrix.artifact }}
           path: artifacts/bin
 
+      - name: Delayed start
+        if: matrix.startup_delay_seconds > 0
+        shell: pwsh
+        run: |
+          # Spreads network-heavy setup across the pool's shared egress address
+          # instead of issuing every job's downloads at once. Configured by
+          # backendDelayedStart in scripts/ci/validation-test-matrix.json.
+          $seconds = ${{ matrix.startup_delay_seconds }}
+          Write-Host "Waiting $seconds second(s) before downloading prerequisites."
+          Start-Sleep -Seconds $seconds
+
       - name: Prepare backend prerequisites
         timeout-minutes: 15
         shell: pwsh
@@ -59,8 +70,6 @@ jobs:
           if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
 
       - name: Run backend tests
-        # A backend that hangs (rather than failing) would otherwise burn the
-        # full job timeout. Fail fast enough to keep the log useful.
         timeout-minutes: 45
         shell: pwsh
         run: |
@@ -71,16 +80,25 @@ jobs:
             Tee-Object -FilePath (Join-Path $env:RUNNER_TEMP 'mxc-ci.log') -Append
           if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
 
-      - name: Upload failure logs
-        if: failure() || cancelled()
+      - name: Upload logs
+        if: always()
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
         with:
           name: logs-${{ inputs.plan }}-${{ matrix.os }}-${{ matrix.architecture }}-${{ matrix.backend }}-${{ github.run_attempt }}
+          # The dispatcher points TEMP here, so suite logs and scratch trees are
+          # collected without enumerating each one. Exclude bulk binaries that
+          # tooling also drops in TEMP; they dwarf the logs and aid no triage.
           path: |
-            ${{ runner.temp }}/mxc-ci.log
-            ${{ runner.temp }}/mxc-wpc-tests/logs
-            ${{ runner.temp }}/WinProcessContainer-Tests.results.*
-            ${{ runner.temp }}/mxc_concurrent_oneshot
+            ${{ runner.temp }}
+            !${{ runner.temp }}/**/*.vhd
+            !${{ runner.temp }}/**/*.vhdx
+            !${{ runner.temp }}/**/*.tar
+            !${{ runner.temp }}/**/*.msi
+            !${{ runner.temp }}/**/*.msix
+            !${{ runner.temp }}/**/*.appx
+            !${{ runner.temp }}/**/*.zip
+            !${{ runner.temp }}/**/*.exe
+            !${{ runner.temp }}/**/*.dll
           if-no-files-found: ignore
           retention-days: 7
 
@@ -92,7 +110,7 @@ jobs:
       fail-fast: false
       matrix: ${{ fromJSON(needs.resolve.outputs.linux) }}
     runs-on: [ self-hosted, "1ES.Pool=${{ matrix.pool }}", "JobId=mxc-e2e-${{ github.run_id }}-${{ github.run_number }}-${{ github.run_attempt }}" ]
-    timeout-minutes: 60
+    timeout-minutes: 180
     steps:
       - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
 
@@ -102,6 +120,18 @@ jobs:
           name: ${{ matrix.artifact }}
           path: artifacts/bin
 
+      - name: Delayed start
+        if: matrix.startup_delay_seconds > 0
+        shell: bash
+        run: |
+          # Spreads network-heavy setup across the pool's shared egress address
+          # instead of issuing every job's downloads at once. Configured by
+          # backendDelayedStart in scripts/ci/validation-test-matrix.json.
+          set -euo pipefail
+          seconds=${{ matrix.startup_delay_seconds }}
+          echo "Waiting $seconds second(s) before downloading prerequisites."
+          sleep "$seconds"
+
       - name: Prepare backend prerequisites
         timeout-minutes: 15
         shell: bash
@@ -126,8 +156,8 @@ jobs:
               tee -a "$RUNNER_TEMP/mxc-ci.log"
           fi
 
-      - name: Upload failure logs
-        if: failure() || cancelled()
+      - name: Upload logs
+        if: always()
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
         with:
           name: logs-${{ inputs.plan }}-${{ matrix.os }}-${{ matrix.architecture }}-${{ matrix.backend }}-${{ github.run_attempt }}
@@ -143,7 +173,7 @@ jobs:
       fail-fast: false
       matrix: ${{ fromJSON(needs.resolve.outputs.macos) }}
     runs-on: ${{ matrix.runner }}
-    timeout-minutes: 60
+    timeout-minutes: 180
     steps:
       - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
 
@@ -153,6 +183,18 @@ jobs:
           name: ${{ matrix.artifact }}
           path: artifacts/bin
 
+      - name: Delayed start
+        if: matrix.startup_delay_seconds > 0
+        shell: bash
+        run: |
+          # Spreads network-heavy setup across the pool's shared egress address
+          # instead of issuing every job's downloads at once. Configured by
+          # backendDelayedStart in scripts/ci/validation-test-matrix.json.
+          set -euo pipefail
+          seconds=${{ matrix.startup_delay_seconds }}
+          echo "Waiting $seconds second(s) before starting tests."
+          sleep "$seconds"
+
       - name: Run backend tests
         timeout-minutes: 60
         shell: bash
@@ -163,8 +205,8 @@ jobs:
             '${{ matrix.backend }}' "$GITHUB_WORKSPACE/artifacts/bin" 2>&1 |
             tee "$RUNNER_TEMP/mxc-ci.log"
 
-      - name: Upload failure logs
-        if: failure() || cancelled()
+      - name: Upload logs
+        if: always()
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
         with:
           name: logs-${{ inputs.plan }}-${{ matrix.os }}-${{ matrix.architecture }}-${{ matrix.backend }}-${{ github.run_attempt }}

docs/ci-validation-infrastructure.md

@@ -27,11 +27,11 @@ the individual local test scripts are documented in
 |------|------|
 | `.github/workflows/Validation.Tests.Scheduled.yml` | Scheduled entry point. Builds artifacts, then calls the matrix job. |
 | `.github/workflows/Validation.Tests.Matrix.Job.yml` | `workflow_call`-only. Resolves the plan and runs the per-family test jobs. |
-| `scripts/ci/validation-test-matrix.json` | The matrix: OS versions, backends, triggers. |
+| `scripts/ci/validation-test-matrix.json` | The matrix: OS versions, backends, triggers, job staggering. |
 | `scripts/ci/resolve-validation-test-matrix.mjs` | Matrix validator + plan expander. Emits the GitHub Actions matrices. |
 | `scripts/ci/prepare-windows-host.ps1` | Per-backend Windows host preparation / prerequisite assertions. |
 | `scripts/ci/prepare-linux-host.sh` | Per-backend Linux package install and service startup (distro-aware). |
-| `tests/scripts/run_ci_backend_tests.ps1` | Windows dispatcher: backend id → existing backend suite. |
+| `tests/scripts/run_ci_backend_tests.ps1` | Windows dispatcher: backend id → existing backend suite. Also points `TEMP` at `$RUNNER_TEMP` so logs get collected. |
 | `tests/scripts/run_ci_backend_tests.sh` | Linux/macOS dispatcher: backend id → existing backend suite. |
 
 ### Flow
@@ -75,10 +75,9 @@ Build artifacts are kept for 1 day — they exist only to feed these jobs.
 | `macos` | GitHub-hosted `${{ matrix.runner }}` | Download artifact → `chmod +x` → `run_ci_backend_tests.sh <backend id>`. No host-prep step. |
 
 Per-job display name: `<platform id>, <architecture>, <backend>` (macOS omits
-the architecture). Job timeout 60 min; host prep 15 min; the test step 45 min
-(60 on macOS), so a hung backend fails while the log is still useful. On failure
-or cancellation the job uploads `mxc-ci.log` plus the Process Container log
-directories as `logs-<plan>-<os>-<arch>-<backend>-<attempt>`, kept 7 days.
+the architecture). Job timeout 180 min; host prep 15 min; the test step 45 min
+(60 on macOS), so a hung backend fails while the log is still useful. Logs are
+uploaded either way — see [Log collection](#log-collection).
 
 ## The catalog
 
@@ -157,6 +156,39 @@ isolation-session, wslc; 25H2/24H2/23H2 × process-t3 + wslc) and
 8 Linux (each of the four distros × bubblewrap + lxc). macOS resolves empty
 because Seatbelt has no wired suite.
 
+### `backendDelayedStart`
+
+Optional. Staggers the start of jobs for a named backend instead of letting
+them all begin at once:
+
+```json
+"backendDelayedStart": [
+  { "backend": "wslc", "seconds": 300 }
+]
+```
+
+Every runner in a pool shares one egress address, so a backend whose setup
+pulls down a large runtime or several container images concentrates all that
+traffic into a burst the moment its jobs start together. Public registries
+answer with rate limiting and stalled downloads.
+
+`seconds` is the gap between consecutive jobs of that backend, counted per
+backend and following the resolved job order. With the entry above, four WSLC
+jobs start at 0, 300, 600, and 900 seconds.
+
+The resolver puts the offset on every matrix entry as
+`startup_delay_seconds` — `0` where no stagger applies — and the job sleeps
+that long before its first network step. Job timeout is a flat 180 minutes,
+with plenty of room for any wait you'd reasonably configure.
+
+Leave the section out (or empty) and every job starts as soon as its runner is
+ready. A backend id that no plan schedules is accepted; it just never applies.
+
+Do keep in mind that the runner is held while it sleeps — Actions can't defer
+allocating a matrix job, so the wait has to happen inside it. Use no more than
+the contention calls for. This spreads simultaneous load and nothing else; a
+single download that stalls on its own is unaffected.
+
 ## Backend status
 
 Snapshot of what the matrix actually proves today. Update this table as backends
@@ -167,8 +199,8 @@ get fixed or wired.
 | Process T1 | ✅ Good | Prerelease Windows only. Remaining failures are genuine MXC bugs or harness limitations. |
 | Process T3 | ✅ Good | Non-prerelease Windows builds only, until the testing suite is updated. |
 | Bubblewrap | ✅ Good | |
-| LXC | ⚠️ Mostly good | Some networking tests fail on distros other than Ubuntu 24.04; host-vs-MXC cause not yet isolated. |
-| WSLC | ⚠️ Mostly good | Can hit a download rate limit while updating WSL / pulling container images. Planned fix: split into several jobs spaced ~15 min apart. |
+| LXC | ✅ Good | Some networking tests fail on distros other than Ubuntu 24.04; seems to be an issue with MXC. |
+| WSLC | ✅ Good | Might have to retry hung jobs - this is an issue with overzealous agent reclaiming. |
 | IsolationSession | ⚠️ Blocked | `Feature_AgentSessionsBaseSupport` is not enabled on the pool image yet. |
 | Windows Sandbox | ⛔ Not scheduled | Dispatcher case is wired; no trigger entry yet. |
 | MicroVM | ⛔ Not working | Windows cold and warm starts hang; no Linux suite. The artifact payload is currently commented out in the build jobs. |
@@ -204,13 +236,33 @@ message instead of surfacing later as an opaque backend error.
 - `bubblewrap` — installs `bwrap` (apt/dnf/yum/microdnf) and relaxes
   `kernel.apparmor_restrict_unprivileged_userns` (ephemeral CI hosts only).
 - `lxc` — installs the LXC stack, reloads the AppArmor profile, starts and waits
-  for `lxcbr0`, and prints network diagnostics. On RHEL-likes it first needs
-  EPEL, because Red Hat dropped LXC after RHEL 7 and ships no replacement.
+  for `lxcbr0`, enables bridge netfilter, and makes sure the bridge's NAT rule
+  is in place. On RHEL-likes it needs EPEL first, because Red Hat dropped LXC
+  after RHEL 7 and ships no replacement.
 - `microvm` — asserts the NanVix payload exists.
 - `hyperlight` — no-op.
 
 macOS has no preparation step.
 
+## Log collection
+
+Every job uploads its logs whether it passed or failed, as
+`logs-<plan>-<os>-<arch>-<backend>-<attempt>`, kept 7 days.
+
+The catch is that `$env:TEMP` is not `$RUNNER_TEMP`. The Windows suites write their
+scratch trees, transcripts, and results files under the user's temp directory
+(`C:\Users\<user>\AppData\Local\Temp`), but `upload-artifact` reads
+`${{ runner.temp }}` (`C:\a\_work\_temp`). Anything left in the former is simply
+never collected, which is why the artifact used to arrive nearly empty.
+
+So `run_ci_backend_tests.ps1` points `TEMP` and `TMP` at `$RUNNER_TEMP` before
+it dispatches. Parameter defaults, `[System.IO.Path]::GetTempPath()`, and child
+processes all read those variables, so everything temp-rooted lands in the
+upload directory without CI having to know a single filename.
+
+Linux and macOS need none of this — those suites log to stdout, and the run
+step tees that into `$RUNNER_TEMP/mxc-ci.log`.
+
 ## Runbook
 
 Always finish with a local resolve, which runs the full catalog validation:
@@ -282,6 +334,19 @@ Replace the explicit failure in the dispatcher with the suite invocation, add
 any host prerequisites, then add the OS/backend pair to a trigger. Always verify 
 by testing it ahead of time. 
 
+### Stagger a backend's job starts
+
+Add or edit its `backendDelayedStart` entry in the catalog, then resolve
+locally to confirm the offsets. Worth reaching for when a backend's setup is
+network-heavy enough that concurrent jobs run into rate limits or stalled
+downloads — and worth removing again once that pressure is gone.
+
+### Collect a new log file
+
+Have the suite write it under `$env:TEMP`. The dispatcher redirects that to the
+upload directory, so nothing in CI needs to change. See
+[Log collection](#log-collection).
+
 ### Change the schedule
 
 Everything schedule-related lives in `Validation.Tests.Scheduled.yml`: the two