CI Validation Infra: fixes for WSLC test prereqs, Windows log collection, LXC bridge NAT, and rate limits

[Elliot (theelliotm)]

📖 Description

A batch of validation-matrix (E2E) CI fixes. Four independent threads, all
confined to the matrix workflow, its resolver, and the CI-facing test entry
points.

1. LXC host preparation — container networking.

lxc-net skips its firewall setup when it believes another manager owns the
ruleset, leaving a bridge that hands out DHCP leases the container cannot
actually use. prepare-linux-host.sh now enables bridge netfilter
(a hard prerequisite — MXC refuses to report success
for a policy it cannot enforce without it) and ensures the bridge's MASQUERADE
rule exists.

2. Log collection — artifacts were arriving nearly empty.

$env:TEMP is not $RUNNER_TEMP. The Windows suites write scratch trees,
transcripts, and results files to the former; upload-artifact reads the
latter, so almost nothing was collected.

run_ci_backend_tests.ps1 now points TEMP/TMP at $RUNNER_TEMP before
dispatching.

This is a step towards the goal of having an internal dashboard where scheduled tests and logs are easily navigable.

3. backendDelayedStart — staggering network-heavy job starts.

Every runner in a 1ES pool shares one egress address, so a backend that pulls a
large runtime or several container images concentrates that traffic into a
burst when its jobs start together, drawing rate limiting and stalled
downloads. The matrix config .json file now accepts:

"backendDelayedStart": [
  { "backend": "wslc", "seconds": 45 }
]

The resolver validates it and emits startup_delay_seconds on every matrix
entry; each job sleeps that long before its first network step. In this example,
the first WSLC job starts immediately, the second WSLC job starts in 45 seconds, the third
in 90 seconds, and so on.

4. WSLC read-only mount test — missing fixture.

Removed Strict mode from the main dispatch script. Also added test.txt to the environment since a few tests required it, but I removed it during the merge conflict since it became obvious it was very recently fixed.

Limitations

• There is some evidence that idle runners are more likely to be reclaimed by
  the pool, so keep stagger times short until a better long-term solution is reached.
• This does not address the remaining LXC network failures ([LXC/Bubblewrap] Network policy: egress enforcement, hook scoping, and proxy pinning #896) or the
  process-container ACL/UI failures, which are product-side.

🔗 References

Related product-side work for remaining LXC failures: #896.

Documentation: docs/ci-validation-infrastructure.md is updated for all of the
above (new "Log collection" section, backendDelayedStart reference, and
matching runbook entries).

🔍 Validation

• node scripts/ci/resolve-validation-test-matrix.mjs --plan <pr|nightly|weekly|enabled> — all four exit 0.
• WSLC entries stagger 0/45/90/135 seconds; every other entry carries startup_delay_seconds: 0.
• Workflow YAML parses; step order verified in all three jobs.
• Confirmed that files written to $TEMP by a child process land in $RUNNER_TEMP.

✅ Checklist

• Signed the Contributor License Agreement
• Updated documentation (if applicable)
• Updated Copilot instructions (if build, architecture, or conventions changed)
• If this PR changes Cargo.lock, the dependency-feed-check check passes (see docs/pull-requests.md)

📋 Issue Type

• Bug fix
• Task

Microsoft Reviewers: Open in CodeFlow

[Copilot]

Pull request overview

Improves validation-matrix reliability, networking setup, artifact collection, and test reproducibility.

Changes:

• Adds configurable backend startup staggering.
• Fixes Windows log collection and WSLC fixtures.
• Configures LXC bridge NAT and netfilter prerequisites.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
.github/workflows/Validation.Tests.Matrix.Job.yml	Adds delays, longer timeouts, and unconditional log uploads.
.github/copilot-instructions.md	Documents new CI conventions.
docs/ci-validation-infrastructure.md	Updates CI infrastructure guidance.
scripts/ci/prepare-linux-host.sh	Configures LXC bridge NAT and netfilter.
scripts/ci/resolve-validation-test-matrix.mjs	Validates and assigns startup delays.
scripts/ci/validation-test-matrix.json	Configures WSLC staggering.
tests/scripts/run_ci_backend_tests.ps1	Redirects Windows temporary output for collection.
tests/scripts/run_ci_backend_tests.sh	Clarifies mandatory LXC execution.
tests/scripts/run_wslc_all_tests.ps1	Creates the filesystem test fixture.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Suppressed comments (2)

scripts/ci/prepare-linux-host.sh:151

• This idempotence check uses the bridge address (for example, 10.0.3.1/24) but iptables -S canonicalizes it to the network (10.0.3.0/24), so the grep misses an existing equivalent rule and appends another MASQUERADE rule on every preparation. The grep -q pipeline can also report false under pipefail if iptables receives SIGPIPE. Resolve the actual connected subnet and use iptables -C to test the exact rule instead.

    # Match on the source subnet rather than the rule text: lxc-net's own rule
    # and ours are equivalent however they are spelled.
    if sudo iptables -t nat -S POSTROUTING 2>/dev/null |
        grep -q -- "-s ${subnet%%/*}"; then

docs/ci-validation-infrastructure.md:182

• The workflow sleeps only after checkout and actions/download-artifact in every job, so this is not before the first network step and those artifact downloads remain unstaggered. Either move the delay ahead of those steps or describe the narrower behavior accurately.

The resolver puts the offset on every matrix entry as
`startup_delay_seconds` — `0` where no stagger applies — and the job sleeps
that long before its first network step. Job timeout is a flat 180 minutes,
with plenty of room for any wait you'd reasonably configure.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Suppressed comments (2)

scripts/ci/prepare-linux-host.sh:151

• This existence check is not idempotent. ip addr returns the bridge address (for example 10.0.3.1/24), while iptables -S prints the canonical network (10.0.3.0/24); stripping the prefix searches for -s 10.0.3.1, so an existing lxc-net rule is missed and another MASQUERADE rule is appended on every preparation. Query the exact rule through iptables -C, which performs the same CIDR normalization as insertion.

    if sudo iptables -t nat -S POSTROUTING 2>/dev/null |
        grep -q -- "-s ${subnet%%/*}"; then

.github/workflows/Validation.Tests.Matrix.Job.yml:60

• These offsets are measured from each runner's independently varying allocation time, not from a common epoch. If the zero-delay runner becomes ready late while a higher-offset runner is allocated promptly, prerequisite downloads can still overlap or even start in reverse order, so the advertised 0/45/90/135-second staggering is not guaranteed and may not prevent the rate-limit burst. Coordinate affected jobs against shared absolute start deadlines or serialize them; otherwise this needs to be documented as best-effort rather than fixed spacing.

          $seconds = ${{ matrix.startup_delay_seconds }}
          Write-Host "Waiting $seconds second(s) before downloading prerequisites."
          Start-Sleep -Seconds $seconds

[Branden Bonaby (bbonaby)]

thought: I guess you're trying to find the sweet spot here with timing? Question though, I forgot to ask. Do we get emails about failures anywhere?

[Elliot (theelliotm)]

No, not currently. Great idea though. I'll add it to the list :)

[Branden Bonaby (bbonaby)]

question: Why use the whole temp here again? perhaps I'm misunderstanding the comment above but I'm still not sure why we wouldn't want to specify a specific folder inside type or file.

[Elliot (theelliotm)]

Making it a catch-all for now since T1 tests create a ton of different logs in a couple folders. I'm willing to narrow the scope of it in the future.

[Branden Bonaby (bbonaby)]

issue: Soham Das (@SohamDas2021) are we good with this removal?

[Elliot (theelliotm)]

I added it during my prior PR. Strict mode was unnecessary and I've stripped it back out.