Agents propose manipulation tasks, a verification gate decides which are worth keeping — built on the Genesis SDK, running on Apple Silicon
A generative-simulation pipeline built on the Genesis SDK: agents propose robot manipulation tasks, the tasks are built and simulated in Genesis World, and a verification gate decides which are worth keeping. The number the pipeline optimises is keep-rate, not generation-rate.
Everything runs natively on Apple Silicon — Metal backend, no Linux box, no CUDA.
.venv/bin/python live_server.py # then open http://127.0.0.1:8420Pick an agent count, press Run. Each agent is its own process with its own Genesis instance, so the concurrency is real. The console streams over SSE and shows, live:
- Agent lanes — what each agent is doing right now: proposing, building the scene, verifying, running a rollout.
- Proposals — every generated scene rendered, streaming in, with its verdict and the named reasons it was rejected.
- Rollouts — looping animations of the arm attempting the tasks that survived the gate.
- Live metrics — proposed, survived, keep-rate, solve-rate, throughput, elapsed.
Measured on an M4 Pro with 4 agents: ~49 tasks/min, keep-rate 17–24%, solve-rate 43–67%.
Generation is cheap. Validity is not. An ungrounded proposer produces objects wider than the gripper, placements outside the arm's reach, objects spawned inside each other, and goals already satisfied at t=0. Roughly four in five proposals die, and the reasons are named, so a failure is repairable rather than merely discarded — the agent gets told the gripper is 8 cm, not that its task was bad.
The rollout stage then kills a further third to a half. That is the expensive class: a task can pass every geometric check and still be unsolvable, and without this stage you only discover it after training on it.
Checks run cheapest-first, so a broken spec never reaches the simulator:
| stage | checks |
|---|---|
| schema | typed spec, unique ids, plausible extents |
| static geometry | graspable width, workspace envelope, pairwise AABB overlap, goal reachable, goal not already met |
| physics | settles without drifting, IK converges above the object, no collision at the pre-grasp pose |
| solve | scripted Cartesian pick-and-place reaches the goal region |
The same pipeline is exposed as an MCP server, so any agent — Claude Code, Claude Desktop, or their own — can drive Genesis World directly.
claude mcp add genesis-taskforge -- \
/Users/bash/Desktop/physical-ai/genesis-lab/.venv/bin/python \
/Users/bash/Desktop/physical-ai/genesis-lab/server.py| tool | what it does |
|---|---|
list_capabilities |
the affordance manifest — robot, gripper width, workspace envelope, spec schema, every rejection reason |
verify_task |
run a proposed spec through the gate; returns a verdict and named reasons |
attempt_task |
scripted pick-and-place; answers whether the task is solvable at all |
render_task |
render the settled scene so a vision model can inspect it |
pipeline_stats |
keep-rate, solve-rate, rejection histogram |
The intended loop: the agent reads list_capabilities, submits a spec, and repairs against the
rejection reasons without a human in the loop.
Genesis needs the process main thread. It pulls in GLFW, and macOS kills any process that
touches Cocoa off the main thread. MCP tool handlers run on a worker thread, so the simulator gets
its own process and talks over a pipe (worker.py). Crash isolation came free.
Joint-space interpolation sweeps the object off the table. Commanding a joint target moves the end effector along an arc, not a line. Waypoints are Cartesian and each one waits for convergence before the next is issued.
Position control cannot hold a grasp — the object back-drives the fingers. Closing is force control at −1 N. At −8 N the fingers eject the cube across the table.
live_server.py SSE server + agent pool console.html the live UI
agent_worker.py one autonomous agent process server.py MCP server
worker.py simulator process for MCP taskforge/ spec, world, verify, solve
build_demo.py offline render of a full sweep dashboard.py static HTML report
demo_sweep.py headless keep-rate sweep FINDINGS.md Genesis/Quadrants benchmarks
Built on genesis-world 1.3.3, quadrants 1.3.0, Metal backend, Python 3.13.