Distributed G1 locomotion RL across a WendyOS mesh fleet (ES + PPO + GPU/Warp)

[Joannis]

Distributed G1 locomotion RL across a WendyOS mesh fleet

Two example apps that train a Unitree G1 velocity-tracking policy with genuinely distributed compute — work split across three Spark devices (asset ids 284/283/211) that discover and talk to each other over the WendyOS mesh — plus a GPU path built on mujoco_warp.

Built spec → plan → TDD → real-fleet verification. Design/plan in specs/2026-07-11-g1-fleet-distributed-rl-*.md; usage in Examples/README-G1Fleet.md.

What's here

• Examples/g1fleet/ — shared core (G1 env, MLP policy, pluggable rollout backend, mesh peer wiring, gzip numpy wire codec, ES + PPO drivers). 18 unit/smoke tests, all green.
• Examples/G1FleetES/ — Evolution Strategies fleet. Coordinator broadcasts params; each device evaluates a mirrored-perturbation slice and returns scalars; coordinator applies the ES gradient + Adam. Mesh-light. Coordinator double-duties as a local worker.
• Examples/G1FleetPPO/ — actor–learner PPO. Learner serves versioned weights + ingests trajectories; actors roll out and POST experience; staleness-bounded updates.
• Examples/G1FleetESGpu/ + warp_backend.py — GPU backend: the ES population is the batch dimension, run as batched mujoco_warp worlds with a CUDA-graph-captured physics step; per-world MLP in CuPy.
• Examples/G1GpuProbe/ — GPU viability + throughput probe.

Verified on the real fleet

• Mesh works on hardware — agent logs show CNI IP allocation + mesh egress applied for serviceCIDR 10.99.0.0/16, LAN-direct across the three Sparks.
• ES (3 devices): mean_return −1.50 → +1.34 over gens 12–32, n_seeds=60/60 (all devices contributing).
• PPO (learner + actor): trained to version ~1489, mean_return ~400 → ~1142.
• GPU probe: CDI GPU passthrough + Warp sm_121 (Blackwell) kernel compile + batched mujoco_warp G1 all work in-container; CUDA-graph capture gives ~734k env-steps/s (~30× vs launch-bound plain stepping).

Real bugs fixed during E2E

• Model wasn't actually offline (runtime robot_descriptions git-fetch crashed under mesh's no-internet egress) → vendored to /opt/g1_model, verified with docker run --network none.
• ES coordinator did no rollouts (its slice always timed out) → loopback double-duty worker.
• Flaky smoke test (fixed ports) → ephemeral ports.

Known limitations / follow-ups

• GPU ES live training not yet confirmed end-to-end. The backend builds, deps resolve, and it runs to init on the GPU, but a first-evaluate_returns crash (hardened since: cross-stream sync + CUDA-graph fallback + stage logging) is pending clean re-verification. Re-verification is currently gated by environment friction: the 6.8 GB CUDA image push over the tunnel, intermittent sandbox DNS on the buildx base-image pull, and log-export rate-limiting on GPU-busy devices. Reproduce reads with wendy cloud device logs --app <name> --tail N (live-only default otherwise hides history).
• MJX/JAX path intentionally not pursued — mujoco_warp is the proven-on-Blackwell recipe (per mjlab).
• CPU remains the portable default backend; SIM_BACKEND=warp selects GPU.

🤖 Generated with Claude Code

[github-actions]

Note

Integration test review — automated suggestions from Claude. Apply, adapt, or dismiss as needed.

NO_CHANGES_NEEDED

The PR introduces Examples/G1FleetES/ (and companion GPU/PPO examples), which are application examples demonstrating distributed RL training on a WendyOS mesh fleet. Reviewing what WendyOS CLI features, device capabilities, entitlements, or deployment modes this PR actually introduces or changes:

• No new entitlements are declared (no new "type": ... values in any wendy.json).
• No new deployment modes are introduced (no new wendy.json fields, no new compose-style structure, no new serviceName/resources/multiservice patterns not already covered).
• No new CLI features are exercised (no new wendy subcommands or flags).
• Mesh networking (mesh egress, CNI IP allocation, serviceCIDR) is referenced in the PR description as already verified on real hardware, but it is a runtime/agent-side networking behavior — not a new CLI feature, entitlement type, or wendy.json field that the CI test suite would exercise via wendy deploy + app output. There is no "type": "mesh" entitlement or similar construct introduced in the diff.
• The GPU passthrough path (SIM_BACKEND=warp, CDI GPU) would map to the existing python-gpu entitlement ({"type": "gpu"}), which already has coverage.

The diff is purely example application code (RL training logic, MuJoCo env wrappers, HTTP-based parameter servers) with no changes to the WendyOS platform surface area that integration tests are meant to cover.

[github-actions]

AI Security Review

This PR introduces a distributed reinforcement learning system (ES + PPO) for robotics control, spanning multiple Docker images and Python modules deployed across a WendyOS mesh fleet. While the codebase contains no hardcoded secrets, PII, or payment data, it presents a cluster of meaningful security risks: unauthenticated HTTP control-plane endpoints expose model weights, training gradients, and rollout data to any mesh peer without authentication or authorization; unbounded deserialization of numpy/gzip payloads from untrusted peers creates denial-of-service and potential memory-exhaustion vectors; Python's http.server.ThreadingHTTPServer is not production-grade and lacks request-size limits, TLS, or rate-limiting; the debugpy package is baked into production images (remote code execution risk); and dependency pinning is absent, creating supply-chain risk. The system also suppresses all HTTP access logging, eliminating the audit trail required by SOC 2 CC7 and ISO 27001 A.12. These issues are collectively significant for an internal training workload but become critical if any mesh node is reachable from less-trusted networks.

Security & Compliance Review: PR #1423 — Distributed G1 Locomotion RL Fleet

Executive Summary

This PR introduces a distributed reinforcement learning system (ES + PPO) for robotics control, spanning multiple Docker images and Python modules deployed across a WendyOS mesh fleet. While the codebase contains no hardcoded secrets, PII, or payment data, it presents a cluster of meaningful security risks: unauthenticated HTTP control-plane endpoints expose model weights, training gradients, and rollout data to any mesh peer without authentication or authorization; unbounded deserialization of numpy/gzip payloads from untrusted peers creates denial-of-service and potential memory-exhaustion vectors; Python's http.server.ThreadingHTTPServer is not production-grade and lacks request-size limits, TLS, or rate-limiting; the debugpy package is baked into production images (remote code execution risk); and dependency pinning is absent, creating supply-chain risk. The system also suppresses all HTTP access logging, eliminating the audit trail required by SOC 2 CC7 and ISO 27001 A.12. These issues are collectively significant for an internal training workload but become critical if any mesh node is reachable from less-trusted networks.

---

Findings Table

Severity	Standards	File	Line(s)	Title
CRITICAL	SOC2-CC6, ISO27001-A.9, NIST-AC-3	es.py, ppo.py (all copies)	All HTTP handlers	No authentication or authorisation on any control-plane endpoint
HIGH	SOC2-A1, NIST-SI-10	es.py, ppo.py (all copies)	/returns, /rollout POST handlers	Unbounded Content-Length read → memory exhaustion / DoS
HIGH	SOC2-C1, ISO27001-A.8, NIST-SC-8	mesh.py (all copies)	http_get, http_post	All inter-node traffic is plaintext HTTP; model weights exfiltrated in transit
HIGH	SOC2-CC9, ISO27001-A.8	G1FleetES/requirements.txt	Line 4	debugpy in production image enables unauthenticated remote code execution
HIGH	SOC2-CC9, ISO27001-A.8	requirements.txt, Dockerfiles	All	No version pins on any dependency — supply-chain risk
MEDIUM	SOC2-CC7, ISO27001-A.12, NIST-AU-2	es.py, ppo.py (all copies)	log_message override	HTTP access logging deliberately suppressed; no audit trail
MEDIUM	SOC2-CC6, NIST-AC-3	es.py, ppo.py (all copies)	/status GET handlers	Training metrics and model version exposed publicly with no auth
MEDIUM	SOC2-A1, NIST-SI-10	netcodec.py (all copies)	decode_named	numpy zip-bomb / decompression-bomb via malicious gzip+npz payload
MEDIUM	SOC2-CC6, ISO27001-A.9	mesh.py (all copies)	parse_peers, MeshConfig.from_env	Peer hostnames constructed from environment variable content without validation
MEDIUM	SOC2-CC8, ISO27001-A.8	Dockerfiles (ES, PPO, ESGpu)	EXPOSE 5678	Debug port 5678 (debugpy) exposed in container manifest
MEDIUM	SOC2-CC6, NIST-AC-3	es.py (all copies)	run_coordinator	HTTP server binds to 0.0.0.0; relies entirely on mesh network policy for isolation
LOW	SOC2-CC7, NIST-AU-9	es.py, ppo.py (all copies)	_checkpoint	Checkpoint writes are non-atomic (data corruption on crash mid-write)
LOW	NIST-SI-10	rollout.py (all copies)	_infer_hidden	Float arithmetic used for security-sensitive topology inference
INFORMATIONAL	SOC2-CC8	All directories	—	Massive code duplication across G1FleetES, G1FleetESGpu, G1FleetPPO

---

Detailed Findings

---

1. CRITICAL — No Authentication or Authorisation on Any Control-Plane Endpoint

Standards: SOC2-CC6, ISO27001-A.9 (Access Control), NIST AC-3

Description:
Every HTTP endpoint — /params (model weights), /returns (ES gradient data), /weights (PPO policy), /rollout (trajectory data), /status — requires zero authentication. Any host that can reach port 8080 can: pull the current policy parameters, inject arbitrary gradient returns to poison the ES optimiser, or POST fabricated rollout trajectories to corrupt PPO training. Within the mesh, any compromised peer can take full control of the training process.

Relevant snippet (es.py, all copies):

def do_GET(self):
    if self.path == "/params":
        with lock:
            body = encode_named({
                "theta": state["theta"],
                ...
            })
        self.send_response(200)
        ...
        self.wfile.write(body)

def do_POST(self):
    if self.path == "/returns":
        length = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(length)
        try:
            d = decode_named(body)
            ...
            for s, p, m in zip(seeds, rp, rm):
                state["returns_plus"][s] = float(p)
                state["returns_minus"][s] = float(m)

Remediation:

• Add a shared secret (HMAC-SHA256 signature of request body) injected via an environment variable (e.g. MESH_SHARED_SECRET). All senders sign; all receivers verify before processing.
• Alternatively, use mutual TLS (mTLS) with per-device certificates issued at container start.
• At minimum, validate that the Host header matches an expected peer.
• Consider adding per-seed/per-generation nonces to prevent replay attacks on /returns.

---

2. HIGH — Unbounded Content-Length Read Enables Memory Exhaustion / DoS

Standards: SOC2-A1, NIST SI-10

Description:
Both /returns and /rollout handlers read Content-Length bytes directly from the socket with no upper bound check. A malicious or buggy peer can send Content-Length: 10000000000 and exhaust the process's memory. There is also no timeout on the read itself.

Relevant snippet (es.py, ppo.py, all copies):

def do_POST(self):
    if self.path == "/returns":
        length = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(length)

Remediation:

MAX_BODY = 50 * 1024 * 1024  # 50 MB — tune to actual max payload size
length = int(self.headers.get("Content-Length", 0))
if length > MAX_BODY:
    self.send_response(413); self.end_headers(); return
body = self.rfile.read(length)

Also set a socket.settimeout() on the server socket.

---

3. HIGH — All Inter-Node Traffic is Plaintext HTTP

Standards: SOC2-C1, ISO27001-A.8 (Data in Transit), NIST SC-8

Description:
All calls between coordinator and workers, and between learner and actors, use plain http://. Model weights (theta), rollout trajectories, and gradient returns are transmitted without encryption. On a shared LAN this exposes intellectual property (trained policy parameters) and allows passive eavesdropping or active MITM to inject poisoned data.

Relevant snippet (mesh.py, all copies):

coord_url = f"http://{coord}"
...
body = http_get(f"{coord_url}/params")

Remediation:

• Switch to HTTPS with self-signed certificates generated at image build time (one CA cert baked in, per-device leaf certs). Python's ssl.wrap_socket can be used with ThreadingHTTPServer.
• If the WendyOS mesh itself provides authenticated/encrypted transport, document that reliance explicitly and enforce that SIM_BACKEND != cpu (cleartext) paths are not used on shared networks.

---

4. HIGH — debugpy in Production Image Enables Unauthenticated Remote Code Execution

Standards: SOC2-CC9, ISO27001-A.8

Description:
debugpy is listed in the production requirements.txt and port 5678 is EXPOSEd in the Dockerfile. debugpy by default allows any client to attach and execute arbitrary Python. If the debug listener is started (e.g., by any imported library or future code), it is a full remote code execution endpoint on the container network.

Relevant snippet (G1FleetES/requirements.txt):

debugpy

Relevant snippet (Dockerfile):

EXPOSE 8080 5678

Remediation:

• Remove debugpy from requirements.txt entirely. Use a separate debug image/build stage if needed during development.
• Remove EXPOSE 5678 from the Dockerfile.
• If debugpy is needed for development, gate it behind a DEBUG=1 build arg and never include it in the production image.

---

5. HIGH — No Version Pins on Any Dependency (Supply-Chain Risk)

Standards: SOC2-CC9, ISO27001-A.8 (Vulnerability Management)

Description:
All entries in requirements.txt are unpinned (numpy, mujoco, robot_descriptions, debugpy). The GPU Dockerfile uses loose version ranges ("warp-lang>=1.14.0", "numpy<2.5"). A malicious or compromised upstream release could be transparently incorporated into a rebuilt image.

Relevant snippet (G1FleetES/requirements.txt):

numpy
mujoco
robot_descriptions
debugpy

Remediation:

• Pin all dependencies to exact versions: numpy==2.1.3, mujoco==3.10.0, etc.
• Generate a requirements.lock or use pip-compile / uv lock.
• Add hash verification: pip install --require-hashes -r requirements.txt.
• Consider using a private mirror or vendoring wheels into the image rather than fetching from PyPI at build time.

---

6. MEDIUM — HTTP Access Logging Deliberately Suppressed; No Audit Trail

Standards: SOC2-CC7, ISO27001-A.12, NIST AU-2

Description:
log_message is overridden to a no-op in both es.py and ppo.py handlers. This silences all HTTP access logs. There is no record of which peer fetched weights, posted returns, or triggered a training step. This violates SOC 2 CC7 (system operations monitoring) and ISO 27001 A.12 (logging and monitoring).

Relevant snippet (es.py, ppo.py, all copies):

class Handler(BaseHTTPRequestHandler):
    def log_message(self, fmt, *args):
        pass  # quiet; app-level logging happens in _advance_locked

Remediation:

• Emit at minimum: timestamp, remote IP, method, path, response code, and Content-Length to structured logs on every request.
• Use logging module rather than print so log levels and sinks can be configured externally.
• For the coordinator, log the peer identity (IP + self-reported self_id) alongside each /returns POST.

---

7. MEDIUM — Training Metrics and Model Version Exposed Without Authentication

Standards: SOC2-CC6, NIST AC-3

Description:
The /status endpoint returns generation number, mean/best return, KL divergence, and policy loss over unauthenticated HTTP. While not immediately exploitable, this reveals internal training state to any observer on the mesh and could facilitate timing attacks on checkpoint files.

Relevant snippet (es.py):

elif self.path == "/status":
    ...
    payload = json.dumps({
        "generation": state["generation"],
        "mean_return": state["mean_return"],
        "best_return": state["best_return"],
    }).encode()

Remediation:

• Apply the same HMAC/shared-secret check recommended in Finding 1 to the /status endpoint.
• Consider whether external status exposure is needed at all, or whether it should only be accessible from 127.0.0.1.

---

8. MEDIUM — Decompression Bomb / Memory Exhaustion via Malicious gzip+npz

Standards: SOC2-A1, NIST SI-10

Description:
decode_named decompresses gzip then loads numpy zip data from untrusted peers with no size limits at any stage. A peer (or MITM) can send a valid gzip stream that decompresses to hundreds of GB (a "zip bomb"), or a numpy zip with a crafted array shape (e.g., shape=(2**32, 2**32)) that triggers a massive allocation before any bounds check.

Relevant snippet (netcodec.py, all copies):

def decode_named(b: bytes) -> dict[str, np.ndarray]:
    with np.load(io.BytesIO(gzip.decompress(b))) as z:
        return {k: z[k] for k in z.files}

Remediation:

MAX_DECOMPRESSED = 200 * 1024 * 1024  # 200 MB
decompressed = gzip.decompress(b)
if len(decompressed) > MAX_DECOMPRESSED:
    raise ValueError(f"decompressed size {len(decompressed)} exceeds limit")
with np.load(io.BytesIO(decompressed), allow_pickle=False) as z:
    # Validate each array's dtype and shape before loading
    result = {}
    for k in z.files:
        arr = z[k]
        if arr.nbytes > MAX_DECOMPRESSED:
            raise ValueError(f"array {k} too large")
        result[k] = arr
    return result

Note: allow_pickle=False should already be the default in recent NumPy, but is worth being explicit.

---

9. MEDIUM — Peer Hostnames Constructed from Unvalidated Environment Variable Content

Standards: SOC2-CC6, ISO27001-A.9, NIST AC-3

Description:
parse_peers constructs target URLs directly from the MESH_PEERS environment variable. While environment variables are typically set by an operator, if this variable can be influenced by a less-privileged input source (e.g., a WendyOS configuration API), it could be used to redirect HTTP calls to internal network addresses (SSRF) or to cause unexpected DNS resolution.

Relevant snippet (mesh.py, all copies):

head, _, tail = item.partition(":")
if head.isdigit():
    ...
    target = f"device-{head}.cloud.wendy.dev:{port}"
elif ":" in item:
    target = item   # ← arbitrary host:port accepted
else:
    target = f"{item}:{default_port}"

Remediation:

• Validate that peer addresses match an allowlist of expected patterns (e.g., device-\d+\.cloud\.wendy\.dev:\d+ or private RFC-1918 ranges).
• Reject any peer that resolves to loopback, link-local, or metadata service addresses (169.254.x.x).
• Validate port numbers are in the expected range (e.g., 1024–65535).

---

10. MEDIUM — Debug Port 5678 Exposed in Container Manifest

Standards: SOC2-CC8, ISO27001-A.8

Description:
The EXPOSE 5678 directive in the ES and PPO Dockerfiles advertises the debugpy port in the container manifest. Even if the listener is not started by default, this increases the attack surface and signals to tooling that the port should be accessible.

Relevant snippet (Dockerfiles):

EXPOSE 8080 5678

Remediation:

• Remove EXPOSE 5678. Address this alongside Finding 4 (removing debugpy from requirements entirely).

---

11. MEDIUM — HTTP Server Binds to 0.0.0.0; Network Isolation Relies Entirely on Mesh Policy

Standards: SOC2-CC6, NIST AC-3

Description:
Both coordinator (ES) and learner (PPO) bind ThreadingHTTPServer to ("0.0.0.0", cfg.port), meaning they listen on all interfaces. Isolation from unauthorised callers depends entirely on the WendyOS mesh CNI policy (serviceCIDR 10.99.0.0/16). If that policy is misconfigured or if a future deployment context does not enforce it, all endpoints become openly accessible.

Relevant snippet (es.py, ppo.py):

httpd = ThreadingHTTPServer(("0.0.0.0", cfg.port), Handler)

Remediation:

• Where possible, bind to the specific mesh interface IP rather than 0.0.0.0.
• Add defence-in-depth: implement the HMAC authentication from Finding 1 so the application itself enforces access control independent of network policy.
• Document the network policy dependency explicitly and add a startup check that verifies the expected network configuration.

---

12. LOW — Non-Atomic Checkpoint Writes

Standards: SOC2-CC8, ISO27001-A.8

Description:
np.save writes directly to the final checkpoint path. A process crash or OOM between opening the file and completing the write leaves a corrupt checkpoint. On recovery, the coordinator would load a truncated theta.npy and produce undefined behaviour.

Relevant snippet (es.py, all copies):

def _checkpoint(theta: np.ndarray) -> None:
    try:
        os.makedirs(cfg.ckpt_dir, exist_ok=True)
        np.save(os.path.join(cfg.ckpt_dir, "theta.npy"), theta)

Remediation:

import tempfile
tmp = os.path.join(cfg.ckpt_dir, "theta.npy.tmp")
np.save(tmp, theta)
os.replace(tmp, os.path.join(cfg.ckpt_dir, "theta.npy"))  # atomic on POSIX

---

13. LOW — Float Arithmetic Used for Architecture Topology Inference

Standards: NIST SI-10

Description:
_infer_hidden uses floating-point square root and rounding to determine the neural network hidden layer size from a parameter vector length. Floating-point imprecision could cause the wrong architecture to be inferred for certain combinations of obs_dim, act_dim, and total_params, potentially silently setting incorrect policy weights without error.

Relevant snippet (rollout.py, all copies):

h = int(round((-b + disc ** 0.5) / 2))
if h > 0:
    n = obs_dim * h + h + h * h + h + h * act_dim + act_dim
    if n == total_params:
        return (h, h)
return fallback

Remediation:

• The integer check n == total_params provides a correctness guard, so this is low risk. However, consider passing the architecture explicitly (it's already available at construction time) rather than inferring it. This removes the ambiguity entirely.

---

14. INFORMATIONAL — Massive Code Duplication Across Subdirectories

Standards: SOC2-CC8

Description:
es.py, ppo.py, g1env.py, mesh.py, netcodec.py, policy.py, and rollout.py are byte-for-byte identical across G1FleetES/, G1FleetESGpu/, and G1FleetPPO/. Any security fix must be applied in three places, making it likely that patches will be missed.

Remediation:

• Refactor into a single shared g1fleet package installable via pip install -e or a local path dependency.
• Each app directory then has only its app.py, Dockerfile, and wendy.json.

---

Compliance Summary

Framework	Checked	Violations Found
SOC 2 (CC6, CC7, CC8, CC9, A1, C1)	✅	YES — CC6 (no auth), CC7 (no logging), CC8 (non-atomic writes, duplication), CC9 (unpinned deps, debugpy), A1 (DoS vectors), C1 (plaintext transport)
ISO/IEC 27001:2022 (A.8, A.9, A.12)	✅	YES — A.8 (secure coding, dependency management), A.9 (access control absent), A.12 (audit logging suppressed)
PCI DSS v4.0	✅	Not applicable — no payment or card data flows
GDPR / Privacy	✅	Not applicable — no personal data processed
HIPAA	✅	Not applicable — no health data
NIST SP 800-53 / CSF 2.0 (AC, AU, IA, SC, SI)	✅	YES — AC-3 (no authz), AU-2/AU-9 (logging suppressed, non-atomic audit records), SC-8 (no encryption in transit), SI-10 (no input validation on payload sizes)