Leaderboard  ·  Paper (PDF)  ·  HotInfra '26  ·  Scholar  ·  Contribute

InfraBench evaluates AI agents on realistic infrastructure incidents across the full stack (L1 hardware → L4 applications) and the operational lifecycle — durability, invariants, cleanup, and risk — not just binary pass/fail.

Paper: Beyond Pass/Fail: Evaluating Infrastructure Agents Across Layers, Lifecycle, and Risk · HotInfra '26 (co-located with ISCA '26)

---

01   What's in this repo

This is the public InfraBench surface: curated task packages (instruction, environment, verifier). The evaluation runtime (Syscraft) is not open-sourced here yet.

Task	Layer	Diff.	Backend
hello-world	—	Easy	Docker · start here
ipmi-node-power-recovery	L1	Easy	BM Cluster
cassandra-nic-split-brain	L2	Medium	BM Cluster
cassandra-dead-node-removal	L3	Medium	BM Cluster
vm-ceph-bootstrap	L3	Hard	VM cluster
db-wal-recovery	L4	Hard	Container

More of the 12-task HotInfra suite will land here over time · tasks/README.md

Task package shape

my-task/
├── task.toml          # env, resources, difficulty
├── instruction.md     # what the agent sees
├── environment/       # Dockerfile  —or—  setup.sh + bootstrap.sh
├── solution/          # optional reference solve.sh
└── tests/             # verifier → /logs/verifier/reward.{txt,json}

Authoring: CONTRIBUTING.md · AGENTS.md

---

02   Evaluation runtime

End-to-end runs use Syscraft (provisioning, fault scenarios, agent adapters).
That harness is not published in this repository yet.

• Treat these tasks as layout + verifier references today.
• hello-world is the on-ramp for a future lightweight Docker runner.
• Need eval access? Open an issue or contact the authors via the paper / Scholar.

InfraBench/
├── tasks/           curated packages
├── CONTRIBUTING.md  write a task
├── AGENTS.md        agent authoring notes
├── paper/           HotInfra '26 PDF
└── docs/            project website (GitHub Pages)

---

03   Citation

@inproceedings{gao2026infrabench,
  title     = {Beyond Pass/Fail: Evaluating Infrastructure Agents Across Layers, Lifecycle, and Risk},
  author    = {Gao, Yuan and Yang, Zeren and Li, Junnan and Zhong, Shawn (Wanxiang) and Dajani, Ahmed and Zheng, Mai and Arpaci-Dusseau, Andrea and Arpaci-Dusseau, Remzi},
  booktitle = {Workshop on Hot Topics in System Infrastructure (HotInfra)},
  year      = {2026},
  note      = {Co-located with ISCA '26},
  url       = {https://hotinfra.org/2026/papers/hotinfra26-final71.pdf}
}

Project site · PDF · HotInfra '26 · Google Scholar (Yuan Gao)

---

Acknowledgments

• Task interviews: CHTC · DoIT · ARA Wireless Living Lab
• Compute: CloudLab
• API credits: Google Gemini Academic Program

---

Site · Paper · HotInfra · Scholar · Contribute · Apache 2.0

_{University of Wisconsin–Madison · Iowa State University}