fix(worker): let sterile supervisors schedule on the sandbox pool

[haasonsaas]

Problem

The sterile supervisor Pod is an ordinary (non-sandboxed) Pod with no tolerations, so it can only be scheduled on the cluster's general-purpose node pools. On evalops-prod those pools (primary, primary-balanced) have sat at 93-99% CPU requests since ~2026-08-28, while the sandboxwich-gvisor pool that hosts the tenant Pod runs at 27-82%. Every sterile provision therefore waited on a node scale-up for the 50m-CPU supervisor:

0/27 nodes are available: 9 Insufficient cpu, 1 Insufficient memory, 18 node(s) had untolerated taint(s)
kubernetes_provider_transient: sterile candidate supervisor pod did not become ready: timed out waiting for the condition on pods/sandboxwich-supervisor-…

Dex's dex-synthetic-canary has failed every hour since 2026-08-30 (runtime_dispatched_unanswered, 180s budget expired) because the operating runtime never gets its sandbox in time.

Change

When a RuntimeClass is configured, the supervisor manifest now carries:

• a toleration for sandbox.gke.io/runtime=<runtime class>:NoSchedule, the taint managed sandbox pools (GKE Sandbox, and the same convention on kata pools) put on their nodes, and
• a preferred (not required) node affinity for the matching sandbox.gke.io/runtime node label, so the supervisor follows the tenant Pod's pool but can still fall back to any node.

Without a RuntimeClass nothing is added. The toleration is a no-op on clusters without the taint. No change to the tenant Pod.

Validation

• cargo test -p sandboxwich-worker: 291 passed; kubernetes_pod_marks_only_authoritative_sterile_pool_candidates now asserts the toleration and affinity on the rendered supervisor manifest.
• Production evidence read live on 2026-09-05: per-node Allocated resources, FailedScheduling/TriggeredScaleUp events on sandboxwich-supervisor-*, sandboxwich-worker rollback logs.
• Pairs with evalops/k8s#190 (primary pool ceiling 8 → 12). After promotion, the next hourly canary should pass and sandboxwich-worker should stop logging rolled back resources … after failed staged provision.

Change-Origin: agent
Agent-Contributor: role=implementation; harness=claude-code; provider=anthropic
Human-Edited: false
Human-Reviewed: false

🤖 Generated with Claude Code

https://claude.ai/code/session_01CEJ13RFW9vDqg1BeacstYs