rl-environments

Hand-built agent-evaluation environments: graded coding tasks with deliberately underspecified tickets, planted ground truth, and verifier-owned, shape-agnostic graders — hidden tests assert the behavioral contract, never the reference fix's mechanism, so a strong agent's valid alternative solution scores full marks.

Start with ARCHITECTURE.md for how everything fits together, and FINDINGS.md for the validated lessons — difficulty levers, grader failure modes and gates, anti-leak, and environment engineering. The design record lives in docs/ (numbered); docs/00-origin-thesis.md is the original project thesis.

The three lanes

Lane	What	Example tasks
1 — SDUI gym	UI-from-mock and runtime-bug tasks on an owned server-driven-UI framework; difficulty = withheld judgment	order-confirmation-fidelity, stale-screen-cache
2 — real-repo debugging	Real OSS bugs at the pre-fix commit, history scrubbed, symptoms-only ops ticket	celery-worker-stall-after-reconnect, redis-cache-invalidation-hang
3 — delegation lane	Multi-unit feature builds whose primary readout is agent architecture (lead/sidekick delegation strategy, brief fidelity, scope discipline)	vendor-statements, pandera-geodataframe-support, marketplace-q3-milestone, marketplace-quarter-close

Task anatomy

tasks/<name>/
  task.md        # the ticket (behavioral voice — what a user/ops/PM would say)
  task.toml      # metadata, provenance, design notes, weights
  world/         # the tree the agent works in (real-repo worlds are
                 #   gitignored and rebuilt: scripts/make_world.sh)
  hidden/        # verifier-owned contract tests (never shown to agents)
  grade.py       # weighted checkpoints (+ blockers on newer tasks);
                 #   self-contained: grade.py --solution <tree> --out grade.json
  qa/            # reference implementations + hack probes (qa/qa.py)
  CHANGELOG.md   # gates run, grader post-mortems, calibration notes
  runs/          # archived agent runs (transcript, diff, grade, meta)

Running a task

# Real-repo tasks: build the world + grading venv first
tasks/<name>/scripts/make_world.sh

# Point any agent at a COPY of world/ with task.md as the prompt, then:
python3 tasks/<name>/grade.py --solution /path/to/agent-tree

# Grader QA (before trusting any score):
python3 tasks/<name>/qa/qa.py

The maintained runner is sidekick-harness — it auto-discovers these tasks, runs lead/sidekick agent configurations against them, and logs per-run transcripts, diffs, costs, and delegation metrics. Its FINDINGS.md reports the experiments built on these tasks. Legacy per-task run_agent.sh scripts (headless Claude Code) also work.

House rules (the short version)

• Never grade against a world with git history — agents git show old commits to find the fix. rm -rf .git always; four guard points enforce it.
• Hidden tests assert the contract, never the oracle's mechanism. Every grader must pass a second valid fix/implementation with a different mechanism before any agent run.
• The first agent run is grader QA — read the agent's fix before believing a low score.
• Audit every transcript for oracle-hunting (upstream fetches of post-cutoff fixes) before trusting a score.

Full pipeline: .claude/skills/creating-real-repo-tasks/ — harvest → f2p gate → scrubbed world → behavioral ticket → shape-agnostic grader → second-valid-fix QA → calibrate.

Contamination note

Hidden graders and oracle provenance live in this repo. While it is private, tasks are valid for frontier evaluation; anything published should be treated as burned for evaluation purposes thereafter.