ateapi: auto-inject the egress trust volume into every actor - #1252
ateapi: auto-inject the egress trust volume into every actor#1252Max Thompson (thompsonmax) wants to merge 1 commit into
Conversation
94b3367 to
a04a3ce
Compare
Max Thompson (@thompsonmax) , do you know the time overhead of auto-injecting the egress trust volume? IIUC, this will be on the critical path of every actor resume operation. AFAIK, we currently plan to auto-inject MITM trust bundle and actor identity. Each of these auto-injection will have a overhead. If the number of auto-injected info keep going up, the time overhead may be too high for our sub-100ms actor-resume goal.
Should we disallow users to mount things at Strictly speaking, the egress MITM trust volume is only needed for actors generating HTTPS traffic. I wonder whether we should allow users to indicate whether an actor generates HTTPS traffic. This can be a followup PR. Bowei Du (@bowei) , WDYT? |
There was a problem hiding this comment.
Max Thompson (@thompsonmax) , what is the impact of this change on e2e-test presubmit? How much longer does it make?
There was a problem hiding this comment.
Quick timing check:
Both runs are from today. Main is commit 9b333c6, run 33915867922 at 20:22 UTC. The PR is 6ae844e, run 33923715673 at 22:01 UTC.
| Step | main | PR |
|---|---|---|
| Deploy MITM egress (sdsmint) | 24s | 53s |
| Recreate counter demos under injection | none | 56s |
| E2E egress MITM trust, gVisor | 18s | 45s |
| E2E egress MITM trust, micro-VM | 21s | 41s |
| Total | 63s | 195s |
So, this adds about 2 minutes at the step level.
Good question, I did some microbenchmarking on a kind cluster and came up with these numbers on how this affects actor resume latency. Checked the operation latency for both gVisor and microVM. Local restores, p50 in ms, 20 samples per cell:
Golden restores, p50 in ms, 5 samples per cell:
So it seems to extend the E2E latency by about 3-5 ms for either runtime. Doesn't seem like an immediate concern, but perhaps something to keep an eye on as/if we increase auto-injected volumes. Here's the branch with the microbenchmarking code: https://github.com/thompsonmax/substrate/tree/egress-trust-injection-bench |
2fe9f00 to
6eca3eb
Compare
Deployments that MITM actor egress TLS need every actor to trust the minting CA, not just templates that project the bundle themselves. Under the new --inject-egress-trust-bundle flag, ateapi appends a well-known systemInfo volume projecting egress-mitm.ate.dev to /run/substrate/certs/egress-mitm.ate.dev.pem on every workload spec it sends to atelet, which resolves it node-side and fails the actor start closed while the bundle is absent, exactly as for a declared projection. The volume is wire-level policy, invisible to templates. Its name, trust.ate.dev, cannot collide with template volumes (template volume names must be DNS labels, which cannot contain '.'). A template that mounts its own volume at /run/substrate/certs, or anywhere below it, opts that container out — kubelet's serviceaccount-token override rule. Nested mounts opt out too because the injected parent bind would shadow them on the micro-VM runtime, which mounts binds grouped by volume kind rather than parent-first. If every container opts out, no volume is injected at all. The flag must be set before templates and workloads exist: a FULL snapshot only restores against the spec it was captured with, so gVisor refuses to restore a snapshot spanning a flip, and a micro-VM guest resumes its snapshotted mount table without the file. CI therefore recreates the counter demos after enabling injection so their goldens regenerate under the new policy. hack/install-ate.sh enables the flag whenever the sdsmint egress variant is installed (the only deployment shape that publishes the bundle). The egressmitm e2e proves the policy end to end — a probe template that declares nothing about trust reads the published anchors at the well-known path and completes TLS through the MITM gateway with only those anchors — and a functional test pins the injected volume in the wire spec of every atelet RPC across an actor's lifecycle.
6eca3eb to
6ae844e
Compare
2 participants
Part of #932 (PR 3 of 3). PR 1 (#941) is merged; PR 2 (#1231) is orthogonal and can land in either order. This closes #871's open item: on a deployment that intercepts actor egress TLS, every actor gets the CA anchors with nothing declared in its template.
What this does
A new ateapi flag,
--inject-egress-trust-bundle(default off), adds one systemInfo volume,trust.ate.dev, projecting theegress-mitm.ate.devbundle to/run/substrate/certs/egress-mitm.ate.dev.pemin every container. Injection lives in anActorWorkflow.workloadSpecwrapper on the seam every atelet RPC builds its spec through (the proto-template path, after #1232), so Run, Restore, both Checkpoint types, and Terminate agree on the volume set. The wire still carries only{name, path}; atelet resolves the bundle node-side as for a declared projection, so a missing or unusable bundle fails actor start with no new code. No atelet, proto, CRD, or RBAC changes.Injection rules
., sotrust.ate.devcannot collide. The injector also fails the spec build if a spec somehow carries the name; the proto-template path does not validate volume names yet, so this check is load-bearing there./run/substrate/certsor below it (the injected parent bind would shadow a nested mount on the micro-VM runtime), or an image volume above it (micro-VM binds image volumes after systemInfo, shadowing the injected mount). Every opt-out is logged with the actor, container, and colliding path. If all containers opt out, no volume is injected.Deployment wiring
ateapi has no runtime signal for "MITM is deployed" (
--egress-gateway-addressis set on passthrough installs too, and ateapi deliberately has no ClusterTrustBundle RBAC), so the gate is an install-time flag.hack/install-ate.shsplices it into the ate-api-server args whenever an invocation that deploys ate-api-server runs under--experimental-use-sdsmint, the only install shape that publishes the bundle. After every ate-api-server rollout it verifies live cluster state: a deployed sdsmint gateway with the flag missing fails the install, so a later plain redeploy cannot silently strip injection while MITM keeps intercepting. CI's MITM step redeploys ateapi with the flag and recreates the counter demos through a new demo-only--deploy-demo-counter-microvmhandler; the fullrun-microvm-demo-kind.shpath would redeploy the control plane without the sdsmint switch and revert both the gateway and the flag.The snapshot constraint
Flipping the flag changes the spec of every later activation, and a FULL snapshot only restores against the spec it was captured with: gVisor refuses the mismatch outright, and a micro-VM guest resumes its old mount table without the file until its next cold boot. The shipped rule (docs, flag help, and CI's own runbook): enable injection at install time, before templates and workloads exist; a later flip means regenerating goldens and discarding paused or suspended actors. We could put the injection state into snapshots, which would remove the constraint entirely, but left that out of this PR.
Tests
TestActorEgressTrustAutoInjection(egressmitm suite, both sandbox classes): a probe that declares nothing about trust reads the injected file, which must match the published ClusterTrustBundle as certificate sets (sanitization shuffles), then completes TLS through the MITM gateway using only those anchors.Open choices
egress-mitm.ate.dev.pemkeeps/run/substrate/certscollision-free if more bundles inject later; happy to switch to theca.pemfrom Publish a guide on how to make actors trust the egress MITM trust bundle #1005's draft guidance.SSL_CERT_FILEand friends) stay out of scope; Publish a guide on how to make actors trust the egress MITM trust bundle #1005's proposal has no signoff, and whatever lands there can point at this path.Not in this PR
Live refresh of running actors' projections (#1231, which an injected volume rides for free once merged), the configurable backend registry (#932), and trust-pickup guidance (#1005).