| title | PatchCascade SOC | |
|---|---|---|
| emoji | 🛡️ | |
| colorFrom | blue | |
| colorTo | red | |
| sdk | docker | |
| app_port | 8000 | |
| pinned | false | |
| tags |
|
Train AI agents to manage vulnerability patches across enterprise networks—without crashing production.
Quick Start • The Challenge • Architecture • Grading Logic • API Reference • Contributors
🏆 Meta PyTorch OpenEnv Hackathon 2026 Submission
Modern enterprise networks face a critical unsolved problem: how to autonomously patch security vulnerabilities without causing service outages. This is a fundamentally sequential decision-making problem with:
- Dependency-aware constraints: Patching node A may crash nodes B, C, D
- Multi-objective optimization: Minimize both security risk AND downtime
- Dynamic threat landscapes: New zero-day vulnerabilities emerge unpredictably
- Cascading failure risks: One wrong action can take down entire infrastructure
PatchCascade SOC provides a research-grade RL environment that captures the full complexity of this problem. Unlike toy grid-worlds or game environments, PatchCascade models real SOC workflows with:
- Realistic network topologies with tiered criticality and service dependencies
- CVSS-based vulnerability scoring matching industry-standard severity ratings
- Dense reward shaping that provides continuous learning signal
- Dynamic events including exploit spreading and zero-day injection
- Multi-dimensional evaluation across completion, efficiency, safety, and strategy
This environment is designed to train agents that could eventually assist human SOC analysts in making high-stakes patching decisions.
Our reward design implements potential-based reward shaping (Ng, Harada & Russell, 1999), which provides dense learning signal while preserving optimal policy invariance. The key insight is:
R'(s, a, s') = R(s, a, s') + γΦ(s') - Φ(s)
Where Φ(s) is our potential function (total penalty from risk + downtime). This guarantees that any policy optimal under the shaped reward is also optimal under the original sparse reward.
Key References:
- Ng, A. Y., Harada, D., & Russell, S. (1999). Policy invariance under reward transformations. ICML.
- Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction. MIT Press.
- Mnih, V., et al. (2015). Human-level control through deep reinforcement learning. Nature.
"Every unpatched CVE is a ticking time bomb. But every patch is a potential outage."
Security teams face an impossible tradeoff:
| Patch Immediately | Delay Patching |
|---|---|
| ✅ Fixes vulnerability | ❌ Leaves exploit window open |
| ❌ Causes service downtime | ✅ Maintains uptime |
| ❌ Risk of cascade failures | ❌ Risk of breach |
| ❌ Angry customers | ❌ Angry regulators |
The real nightmare? Modern infrastructure has dependencies. Patch your database, and suddenly your web servers crash. Take down authentication, and your entire stack follows. One wrong move triggers a cascade failure that costs millions.
PatchCascade SOC trains AI agents to navigate this paradox—learning to patch vulnerabilities in the optimal order while minimizing downtime and avoiding catastrophic cascades.
| Feature | Traditional RL Envs | PatchCascade SOC |
|---|---|---|
| Observation Space | Numeric arrays | Rich JSON with semantic meaning |
| Action Space | Discrete indices | Named actions with parameters |
| Reward Signal | Sparse (win/lose) | Dense (continuous feedback) |
| Real-World Mapping | Abstract | Direct SOC workflow simulation |
| LLM Compatibility | Requires embedding | Native JSON schema descriptions |
| Dependency Modeling | None | Full cascade simulation |
| Dynamic Events | Static scenarios | Exploit spreading + zero-day injection |
| Evaluation | Single metric | Multi-dimensional (4 axes) |
| Task Curriculum | 1-2 levels | 5 progressive difficulty levels |
Not all servers are equal. Our environment models real-world criticality:
| Tier | Type | Examples | Downtime Penalty | Patch Rules |
|---|---|---|---|---|
| 1 | 🔴 CRITICAL | Databases, Auth Servers | 3x multiplier | Must SUSPEND before patching |
| 2 | 🟡 IMPORTANT | Web Servers, APIs | 2x multiplier | Can patch while ONLINE |
| 3 | 🟢 STANDARD | Dev Servers, Monitoring | 1x multiplier | Can patch while ONLINE |
Real-time cascade failure simulation with hard and soft dependencies:
graph TD
LB[🔵 lb-primary-01<br>Tier 2] -->|soft| W1[🟡 web-frontend-01<br>Tier 2]
LB -->|soft| W2[🟡 web-frontend-02<br>Tier 2]
W1 -->|hard| A1[🟡 app-server-01<br>Tier 2]
W2 -->|hard| A2[🟡 app-server-02<br>Tier 2]
A1 -->|hard| DB[🔴 db-primary-01<br>Tier 1 CRITICAL]
A2 -->|hard| DB
A1 -->|hard| AUTH[🔴 auth-server-01<br>Tier 1 CRITICAL]
A2 -->|hard| AUTH
DB -->|hard| REP[🔴 db-replica-01<br>Tier 1]
style DB fill:#dc3545,color:white
style AUTH fill:#dc3545,color:white
style REP fill:#dc3545,color:white
style LB fill:#0d6efd,color:white
style W1 fill:#ffc107,color:black
style W2 fill:#ffc107,color:black
style A1 fill:#ffc107,color:black
style A2 fill:#ffc107,color:black
⚠️ If DB-Primary goes OFFLINE → App servers CRASH → Web servers CASCADE CRASH
Vulnerabilities are modeled with real-world severity metrics:
- CVSS 9.0-10.0 (CRITICAL): Remote code execution, zero-click exploits
- CVSS 7.0-8.9 (HIGH): Privilege escalation, data exfiltration
- CVSS 4.0-6.9 (MEDIUM): DoS, information disclosure
- Exploit in Wild: 2x penalty multiplier for actively exploited CVEs
In incident_response and hard modes, exploited CVEs that remain unpatched on ONLINE nodes for 4+ turns spread to connected nodes via the dependency graph. This creates urgency and rewards proactive patching.
In zero_day mode, new CVEs are injected mid-episode:
- Turn 5: CRITICAL zero-day (CVSS 9.9, actively exploited)
- Turn 15: HIGH severity CVE (CVSS 8.4)
The agent must dynamically adapt its strategy when new threats emerge.
Unlike sparse-reward environments, PatchCascade provides continuous feedback:
Reward = (Previous Penalty) - (Current Penalty) - 0.1
Where Penalty = Risk_Penalty + Downtime_Penalty
The -0.1 time pressure ensures every step has non-zero reward.
| Event | Reward Impact |
|---|---|
| Patch a CRITICAL vuln | +9.0 to +19.6 (doubled if exploit active) |
| Cause a cascade crash | -6.0 to -12.0 per crashed node |
| Invalid action | -0.5 penalty |
| Victory (all patched) | +50.0 bonus |
| Catastrophic failure | -100.0 penalty |
The environment state at turn t is defined as:
S_t = (N, V_t, D, H_t)
N = {n_i} — Set of server nodes with (hostname, tier, state, services)
V_t = {v_j} — Set of active vulnerabilities at turn t
D = {d_k} — Dependency graph edges (immutable)
H_t — Aggregate health metrics at turn t
The reward at each step uses potential-based reward shaping:
R_t = Φ(S_{t-1}) - Φ(S_t) - 0.1 + R_terminal
Where:
Φ(S) = Risk_Penalty(S) + Downtime_Penalty(S)
Risk_Penalty = Σ_j [ cvss_j × |affected_online_j| × (2 if exploit_in_wild_j else 1) ]
Downtime_Penalty = Σ_i [ tier_mult(n_i) × (2 if crashed(n_i) else 1) ] ∀ n_i ∉ ONLINE
-0.1 = time pressure penalty (ensures dense non-zero reward every step)
R_terminal = { +50 if all vulns patched, -100 if all nodes crashed, 0 otherwise }
This formulation guarantees that every step produces non-zero reward, providing truly dense learning signal throughout the episode.
Final scores are normalized to [0, 1] for comparability:
Score = clamp((Σ R_t - R_min) / (R_max - R_min), 0.001, 0.999)
Where R_min = -300.0, R_max = 50.0
PatchCascade offers five progressive difficulty levels, each building on the skills learned in previous levels:
graph LR
E["🟢 Easy<br>Basic Patching"] --> M["🟡 Medium<br>Dependencies"]
M --> H["🔴 Hard<br>Complex Graph"]
H --> IR["🟣 Incident Response<br>Active Breach"]
IR --> ZD["⚫ Zero-Day<br>Dynamic Threats"]
style E fill:#198754,color:white
style M fill:#ffc107,color:black
style H fill:#dc3545,color:white
style IR fill:#6f42c1,color:white
style ZD fill:#212529,color:white
"Learn the basics"
| Parameter | Value |
|---|---|
| Nodes | 3-5 servers |
| Dependencies | None |
| Vulnerabilities | 1 (Medium/High) |
| Max Turns | 30 |
| Key Skill | Basic patch sequencing |
"Handle dependencies"
| Parameter | Value |
|---|---|
| Nodes | 5-8 servers |
| Dependencies | Linear chain (Web → App → DB) |
| Vulnerabilities | 2 (including 1 on Tier 1) |
| Max Turns | 50 |
| Key Skill | Suspend-patch-resume workflow |
"Survive the chaos"
| Parameter | Value |
|---|---|
| Nodes | 10-15 servers |
| Dependencies | Complex graph (LB → Web → App → DB + Auth) |
| Vulnerabilities | 5 (2 actively exploited CRITICAL) |
| Max Turns | 100 |
| Key Skill | Multi-objective optimization under pressure |
"Triage an active breach"
| Parameter | Value |
|---|---|
| Nodes | 8 servers (2 pre-CRASHED) |
| Dependencies | Complex with hard + soft edges |
| Vulnerabilities | 3 (2 actively exploited, spreading!) |
| Max Turns | 60 |
| Key Mechanic | Exploit spreading — unpatched exploited CVEs infect connected nodes every 4 turns |
| Key Skill | Damage assessment, recovery-under-pressure, threat containment |
"Adapt or die"
| Parameter | Value |
|---|---|
| Nodes | 10 servers |
| Dependencies | Multi-layer (Web → Gateway → App → DB/Auth) |
| Vulnerabilities | 2 initial + 2 dynamically injected |
| Max Turns | 80 |
| Key Mechanic | Zero-day injection — new CRITICAL CVE at turn 5, HIGH CVE at turn 15 |
| Key Skill | Adaptive planning, strategy revision, reprioritization |
Unlike simple pass/fail or single-metric grading, PatchCascade evaluates agents across four orthogonal dimensions:
graph TB
subgraph "Composite Score (0.0 - 1.0)"
C["📋 Completion (40%)<br>Were all vulns patched?"]
E["⚡ Efficiency (20%)<br>Steps vs. optimal?"]
S["🛡️ Safety (20%)<br>Cascades avoided?"]
ST["🧠 Strategy (20%)<br>Smart decisions?"]
end
C --> F["Final Score = Σ w_i × d_i"]
E --> F
S --> F
ST --> F
style C fill:#198754,color:white
style E fill:#0d6efd,color:white
style S fill:#dc3545,color:white
style ST fill:#6f42c1,color:white
style F fill:#ffc107,color:black
| Dimension | Weight | What It Measures | Perfect Score |
|---|---|---|---|
| Completion | 40% | Fraction of vulnerabilities patched | All CVEs resolved |
| Efficiency | 20% | Steps taken vs. theoretical minimum | Completed at or near optimal_steps |
| Safety | 20% | Cascade failures avoided | Zero cascade failures |
| Strategy | 20% | Decision quality (exploit priority, dependency ordering, action validity) | Exploited CVEs patched first, correct suspend order |
Note: Weights vary by task type. Incident Response uses safety-focused weights (35% safety), while Zero-Day uses efficiency-focused weights (30% efficiency).
| Agent Behavior | Completion | Efficiency | Safety | Strategy | Final |
|---|---|---|---|---|---|
| Perfect optimal agent | 1.00 | 1.00 | 1.00 | 1.00 | 1.000 |
| Patches everything, slowly | 1.00 | 0.40 | 1.00 | 0.80 | 0.84 |
| Fast but causes cascades | 0.80 | 0.90 | 0.30 | 0.50 | 0.60 |
| Random agent | 0.20 | 0.10 | 0.30 | 0.30 | 0.22 |
graph TB
subgraph "PatchCascade SOC Stack"
INF["inference.py<br>🤖 LLM Agent"] --> CLI["client.py<br>📡 HTTP Client"]
CLI --> SRV["server.py<br>🚀 FastAPI"]
SRV --> ENV["environment.py<br>⚙️ Core Logic"]
ENV --> MOD["models.py<br>📦 Pydantic Schemas"]
SRV --> GRD["grader.py<br>📊 Multi-Dim Grading"]
SRV --> TSK["tasks/<br>📋 5 Task Definitions"]
end
style INF fill:#0d6efd,color:white
style CLI fill:#198754,color:white
style SRV fill:#dc3545,color:white
style ENV fill:#ffc107,color:black
style MOD fill:#6f42c1,color:white
style GRD fill:#fd7e14,color:white
style TSK fill:#20c997,color:white
| Component | Purpose |
|---|---|
models.py |
Pydantic schemas with rich Field() descriptions for LLM comprehension |
environment.py |
Core state machine: reset, step, cascade logic, dynamic events, reward calculation |
server.py |
FastAPI wrapper exposing /reset, /step, /observation, /grade endpoints |
grader.py |
Multi-dimensional programmatic graders (completion, efficiency, safety, strategy) |
tasks/ |
5 task definitions with individual grader configurations and success criteria |
client.py |
Async HTTP client with type-safe request/response handling |
inference.py |
Baseline LLM agent using OpenAI-compatible API |
# Build the container
docker build -t patchcascade-soc .
# Run the server
docker run -p 8000:8000 patchcascade-soc
# Test the endpoint
curl -X POST http://localhost:8000/reset \
-H "Content-Type: application/json" \
-d '{"task_level": "medium"}'# Install dependencies
pip install -r requirements.txt
# Start the server
uvicorn server:app --host 0.0.0.0 --port 8000 --reload
# Run the baseline agent
export HF_TOKEN="your_huggingface_token"
export MODEL_NAME="Qwen/Qwen2.5-72B-Instruct"
python inference.pybash validate-submission.sh https://your-space.hf.spacefrom client import PatchCascadeLocalClient, PatchCascadeAction
# Initialize local client (no server needed)
client = PatchCascadeLocalClient()
# Try the new Incident Response mode!
obs = client.reset(task_level="incident_response")
print(f"Nodes: {[n.hostname for n in obs.nodes]}")
print(f"Crashed: {[n.hostname for n in obs.nodes if n.state == 'crashed']}")
print(f"Vulns: {[v.cve_id for v in obs.vulnerabilities]}")
print(f"Messages: {obs.messages}")
# Output: "⚠️ ACTIVE BREACH: Multiple nodes are already compromised..."
# Recover a crashed node first
from models import ActionType
action = PatchCascadeAction(
action_type=ActionType.RESUME_SERVICE,
target="db-primary-01",
reason="Recover crashed database to restore app layer"
)
result = client.step(action)
print(f"Reward: {result.reward:.2f}, Done: {result.done}")The environment provides a beautiful ASCII network diagram for debugging:
╔════════════════════════════════════════════════════════════════════╗
║ 🛡️ PatchCascade SOC — Turn 5/50 (Incident Response) ║
╠════════════════════════════════════════════════════════════════════╣
║ NETWORK TOPOLOGY ║
║ ║
║ 🟢 db-primary-0 [ ONLINE ] T1 ⚠️ 🔴 app-server-0 [CRASHED ] T2 🔥║
║ 🟢 web-frontend [ ONLINE ] T2 ⚠️ 🟡 cache-redis- [SUSPENDED] T3 ║
║ 🔵 auth-server- [PATCHING] T2 🟢 api-gateway [ ONLINE ] T2 ║
║ ║
║ DEPENDENCIES ║
║ web-fronte ━━► app-server ║
║ app-server ━━► db-primary ║
║ auth-serve ━━► db-primary ║
║ ║
╠════════════════════════════════════════════════════════════════════╣
║ VULNS: 3 active (1 CRIT, 2 HIGH) (1 exploited!) ║
║ HEALTH: 4/6 online | 1 crashed | Risk: 12.5 | Downtime: 8.0 ║
║ REWARD: +15.50 (last: +3.20) ║
╚════════════════════════════════════════════════════════════════════╝
Legend: 🟢 Online | 🔴 Crashed | 🟡 Suspended | 🔵 Patching |
Here's a step-by-step walkthrough of an optimal agent solving the medium task:
Turn 0: Observe — 6 nodes, 2 CVEs (db-primary-01 has CVE-2024-2001, web frontends have CVE-2024-2002)
Dependencies: web → app → db-primary-01
Turn 1: suspend_service(web-frontend-01) — Protect from cascade
Turn 2: suspend_service(web-frontend-02) — Protect from cascade
Turn 3: suspend_service(app-server-01) — Protect from cascade
Turn 4: suspend_service(app-server-02) — Protect from cascade
Turn 5: suspend_service(db-primary-01) — Required: Tier 1 must be SUSPENDED
Turn 6: apply_patch(db-primary-01, CVE-2024-2001) — Patch critical DB vuln
[Patch completes next turn → db-primary-01 returns to ONLINE]
Turn 7: resume_service(app-server-01) — DB is online, safe to resume
Turn 8: resume_service(app-server-02) — Resume second app server
Turn 9: resume_service(web-frontend-01) — Resume web (still has CVE-2024-2002)
Turn 10: apply_patch(web-frontend-01, CVE-2024-2002) — Patch web vuln
Turn 11: resume_service(web-frontend-02)
Turn 12: apply_patch(web-frontend-02, CVE-2024-2002) — Patch second web server
Result: All patched in 12 steps, 0 cascade failures, 0 invalid actions
Score: completion=1.0, efficiency=0.85, safety=1.0, strategy=1.0 → Final: 0.95
Initialize a new episode.
// Request
{ "task_level": "incident_response", "seed": 42 }
// Response
{ "observation": { "nodes": [...], "vulnerabilities": [...], ... } }Execute an action.
// Request
{
"action_type": "apply_patch",
"target": "web-frontend-01",
"cve_id": "CVE-2024-1234"
}
// Response
{
"observation": { ... },
"reward": 7.5,
"done": false,
"truncated": false,
"info": { "valid": true, "cascade_failures": 0, "total_cascade_failures": 0 }
}List all 5 tasks with grader information.
Grade an episode using multi-dimensional programmatic grading.
Get full environment metadata including all tasks, graders, and schemas.
| Action | Target Required | CVE Required | Effect |
|---|---|---|---|
scan_host |
✅ | ❌ | Inspect node details |
suspend_service |
✅ | ❌ | Gracefully offline node |
apply_patch |
✅ | ✅ | Fix vulnerability (1 turn) |
resume_service |
✅ | ❌ | Bring node back online |
noop |
❌ | ❌ | Skip turn |
- Suspend dependents first: Before patching a Tier 1 node, suspend all nodes that depend on it
- Prioritize exploited CVEs:
exploit_in_wild=truemeans 2x risk penalty per turn — and in advanced modes, they spread to connected nodes - Batch patches efficiently: While one node is PATCHING, work on independent branches
- Don't fear downtime: A controlled SUSPENDED state is better than an uncontrolled CRASH
- Watch for dynamic events: In zero-day mode, new CVEs appear at turns 5 and 15 — be ready to reprioritize
- Recover before patching: In incident response mode, crashed nodes must be resumed before they can be patched
| Metric | Description | Goal |
|---|---|---|
| Composite Score | Weighted multi-dimensional score | Maximize (0.0-1.0) |
| Completion | % of vulnerabilities patched | Maximize |
| Efficiency | Steps vs. theoretical optimum | Minimize steps |
| Safety | Cascade failures avoided | Zero cascades |
| Strategy | Decision quality | Maximize |
We evaluate four agent types across all five task levels using our multi-dimensional grading system. Scores are composite (Completion × Efficiency × Safety × Strategy), normalized to [0.0, 1.0].
Run
python benchmark.py --episodes 10to reproduce these results.
| Agent | Easy | Medium | Hard | IR | Zero-Day | Avg |
|---|---|---|---|---|---|---|
| Random | 0.80 | 0.42 | 0.32 | 0.37 | 0.43 | 0.47 |
| Heuristic | 0.95 | 0.89 | 0.79 | 0.74 | 0.95 | 0.86 |
| PPO (RL-trained) | TBD | TBD | TBD | TBD | TBD | TBD |
| LLM Agent | TBD | TBD | TBD | TBD | TBD | TBD |
Note: RL training uses PPO via Stable-Baselines3 with our Gymnasium wrapper. See
train_rl.pyfor training scripts and hyperparameters.
PatchCascade includes a Gymnasium-compatible wrapper for seamless integration with standard RL libraries:
# Quick training with Stable-Baselines3
from gym_wrapper import PatchCascadeGymEnv
from stable_baselines3 import PPO
env = PatchCascadeGymEnv(task_level="medium")
model = PPO("MlpPolicy", env, verbose=1)
model.learn(total_timesteps=50_000)# CLI training (all levels, curriculum learning, plotting)
python train_rl.py --task easy --steps 10000 # Quick test
python train_rl.py --all --steps 50000 # Train all levels
python train_rl.py --curriculum # Curriculum: easy→medium→hard
python train_rl.py --plot # Generate training curves
python benchmark.py --episodes 20 # Full benchmark suiteApache 2.0 — See LICENSE for details.
 Ayush Kumar 🚀 Team Lead | Core Builder |
 Ravi Prashant 🏗️ Architect and Developer |
| Technology | Purpose |
|---|---|
| Python 3.11+ | Core language |
| Pydantic v2 | Data validation & serialization |
| FastAPI | High-performance async API |
| Uvicorn | ASGI server |
| Gymnasium | Standard RL environment interface |
| Stable-Baselines3 | RL training (PPO, A2C) |
| Docker | Containerization |
| OpenAI SDK | LLM integration |
- Meta AI — For hosting the PyTorch OpenEnv Hackathon
- Hugging Face — For Spaces infrastructure
- OpenEnv Community — For the standardized RL environment protocol
Created by Ayush Kumar & Ravi Prashant
Train smarter. Patch faster. Crash never.
Made with ❤️ in India