Skip to content

Domain-based allowOut enforcement: offering our TCP egress proxy implementation — design check first #157

Description

@sunkencity999

Context

src/sandbox/network/policy.rs already accepts and validates domain entries in allowOut, then refuses them at runtime:

domain entries in allowOut require the TCP egress proxy, which is not enabled yet: {:?}

So the API surface, the validation, and the error path exist — the enforcement does not. This issue is to ask whether you would welcome that implementation upstream, and to describe the approach before writing a PR against it.

We needed domain-level egress for a deployment where sandboxes must reach a small set of package registries and nothing else, so we built it against v0.1.1 and have been running it in production. Rather than carry the fork indefinitely, we would rather it live here — but only if the design matches where you want the networking layer to go.

Approach

A TCP egress proxy on the node, off by default behind config:

[network.egress.proxy]
enabled = true
http_port = 18080
tls_port = 18443
  • Each deny-by-default sandbox's outbound tcp/80 and tcp/443 are DNAT-redirected to the proxy, per network slot.
  • The proxy matches the HTTP Host header (:80) or the TLS SNI (:443) against that sandbox's allowOut domains, supporting exact and *.wildcard entries.
  • The name is resolved host-side, not by the sandbox. A forged SNI cannot select the upstream address, which closes the obvious DNS-rebinding path.
  • Matching connections are spliced; non-matching get a 403 on :80 and a closed connection on :443.
  • Slot teardown flushes rules, including warm slots, so policy does not leak between sandbox generations. This overlaps with the concerns in fix(proxy): isolate connections across runtime generations #56 and fix(sandbox): isolate envd bootstrap connections across generations #103.

Domain rules only arm under a Deny base policy, which is consistent with how allowOut already behaves for CIDRs.

Honest limitations

Worth stating up front, because they bound what this feature claims:

  • Non-web ports are not proxied. tcp/80 and tcp/443 only; everything else remains governed by CIDR rules and the base policy.
  • UDP/QUIC is not proxied. A client that negotiates QUIC bypasses the allowlist unless UDP is otherwise denied.
  • DNS remains reachable, so DNS tunnelling is an available exfiltration channel. This is the same residual that Unit 42 demonstrated against a comparable "isolated" sandbox mode earlier this year, so it should be documented rather than implied away.
  • The host firewall must admit the redirected traffic. A default-deny INPUT chain silently blackholes it, and the symptom is confusing: every proxied destination times out, allowed or not, which looks like a broken allowlist rather than a firewall problem. With ufw the rule is scoped to the internal ranges:
    ufw allow proto tcp from 10.11.0.0/16 to 10.12.0.0/16 port 18080,18443
    
    (matching host_interaction_cidr and veth_cidr defaults). We would document this next to the config rather than leave people to discover it.

Size

Roughly 880 lines against v0.1.1: a new src/sandbox/network/egress_proxy.rs (~540) plus wiring in policy.rs, slot.rs, cfg/network.rs, bin/server.rs, the sandbox impl, and docs. Unit tests cover SNI and Host parsing, wildcard matching, and the per-slot registry. There is a companion CLI change (aenv start --allow-out, plus an aenv network subcommand for live updates) which could land separately or not at all if you would rather shape the CLI yourself.

It is written against v0.1.1 and would need rebasing onto current main — the networking layer has moved since (#108, #110, #145), so some of the slot handling will want reworking rather than replaying.

Questions before we write anything

  1. Do you want domain egress in AgentENV at all, or is the intent that operators put a proxy in front and the placeholder error stays?
  2. If yes: proxy-in-process as described, or would you prefer it as a separate helper the node supervises?
  3. Any constraint we should respect from the RL-training workloads this is primarily built for — throughput on the proxy path, or per-slot memory?

Happy to split it into reviewable pieces (config plumbing, proxy core, slot wiring, docs) rather than one large PR. Also happy to drop it entirely if it cuts across something you have planned — no attachment to our implementation, only to not maintaining a fork of it.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions