Skip to content

[Bug] allowOut CIDR rules fail for cross-subnet internal IPs when CubeSandbox is on non-default NIC #591

Description

@woshihoujinxin

Environment

  • Host has dual NICs: eth0 (internal network), eth1 (external network)
  • eth0 gateway 10.20.11.1, reachable to 10.20.11.x (same subnet) and 10.69.85.x (other internal subnets)
  • eth1 gateway is an external gateway, does not route internal addresses
  • CubeSandbox is deployed on eth1
  • allow_internet_access = false, allow_out = ["10.20.11.74/32", "10.69.85.182/32"]

Problem

On the same host:

Target Host curl Sandbox curl Result
10.20.11.74:32312 (same L2 subnet) Reachable Reachable
10.69.85.182:80 (cross-subnet internal IP) Reachable (HTTP 403) Timeout, no SYN-ACK

Sandbox curl -v --connect-timeout 5 http://10.69.85.182:80 output:

* Trying 10.69.85.182:80...
* Connection timed out after 5003 milliseconds

Host routing confirms 10.69.85.182 is reachable via eth0:

$ ip route get 10.69.85.182
10.69.85.182 via 10.20.11.1 dev eth0

Root Cause

This shares the same root cause as #495: the from_cube BPF bpf_redirect bypasses the kernel routing table.

Host network topology:
 ┌─────────────────────────────────────┐
 │         Host 10.20.11.35             │
 │                                      │
 │  eth0 (internal)      eth1 (external) ← CubeSandbox bound here
 │  gateway 10.20.11.1    gateway ???    │
 │  reachable:            reachable:     │
 │    10.20.11.x             internet    │
 │    10.69.85.x                        │
 └─────────────────────────────────────┘

Sandbox → from_cube BPF → SNAT → bpf_redirect(eth1_ifindex, 0) → eth1
                                                                    │
                                       10.69.85.182 is NOT reachable via eth1's gateway
                                       → SYN goes out the wrong NIC, never reaches target

Why 10.20.11.74 works: It shares the same L2 broadcast domain as eth1 (both are 10.20.11.x). eth1 can ARP directly, no gateway needed.

Why 10.69.85.182 fails: Cross-subnet, requires gateway 10.20.11.1 which is only reachable via eth0. bpf_redirect forces all traffic to eth1, whose external gateway does not know how to route 10.69.85.x. The packet is dropped.

Expected Behavior

The from_cube BPF should support per-destination NIC selection (policy routing), or at minimum allow snat_iplist to carry per-NIC SNAT IP/ifindex pairs and select the correct egress interface based on the destination address.

Affected Components

  • from_cube BPF (TC ingress on TAP)
  • snat_iplist map
  • CubeNet / network-agent

Relationship to #495

#495 describes "allow internet traffic to egress via a non-default NIC (GRE tunnel)". This issue describes the reverse: "allow internal traffic to egress via the correct NIC when CubeSandbox is bound to an external-facing NIC". Both share the same root cause — bpf_redirect lacks multi-NIC policy routing capability.

Metadata

Metadata

Assignees

Labels

area/CubeNetImpacts the CubeNet module and network-agentbugSomething isn't workingneeds:compliance

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions