Feat/agentic rollout engine - #15
Merged
Merged
Conversation
Stage 1 §3.2 + §3.4 of docs/AGENTIC_ROLLOUT_SPEC.md.
CodeRunner — fork+execvp Python with hard resource caps and parent-side
wall-clock enforcement:
- RLIMIT_AS / RLIMIT_CPU / RLIMIT_FSIZE / RLIMIT_NOFILE / RLIMIT_NPROC
- prctl PR_SET_PDEATHSIG (child dies if parent crashes) + NO_NEW_PRIVS
- setpgid(0,0) so parent can SIGKILL the whole subtree on timeout
- poll()-drained bounded stdout/stderr with truncation flags
- best-effort unshare(CLONE_NEWUSER | CLONE_NEWNET) for net isolation;
soft-fails with a stderr warning when unprivileged userns are off
Bubblewrap / firejail / seccomp are documented follow-ups; the CodeRunner
interface is designed so a real sandbox swaps in as a subclass.
CodeContestsEnv — public/hidden test split implementing the Env interface
(feedback for the model, reward for GRPO). RewardShape::AllOrNothing and
::FractionPassed; whitespace-normalized stdout equality; per-test status
tags surfaced in the observation. extract_code() pulls the first fenced
Python block from model output (```python / ```py / untagged), skipping
non-Python languages.
test_cp_env — 16 tests:
extract_code (5 cases incl. ```cpp skip + untagged + no-fence)
runner 3+5=8, infinite loop killed under 1s, 256MB blocked under
64MB cap, socket(1.1.1.1:443) blocked by network isolation
env correct→1.0, wrong→0.0 in AON; wrong→<0.5 in FractionPassed;
timeout solution→0; feedback formats sample test outcomes.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
scripts/prepare_cp_data.py: downloads a HuggingFace CP dataset (default deepmind/code_contests), extracts public + private + generated tests, and writes a compact JSONL — one JSON object per problem with id, description, public_tests, hidden_tests. Hidden tests are (private ++ generated), capped by --max-hidden to keep evaluation tractable. Filters on min-public, difficulty range, and max-samples for bounded smoke runs. include/training/cp_dataset.h: header-only loader returning std::vector<CPProblem> (or CPDatasetEntry with description). Skips malformed lines with a bounded stderr warning so a long file with a handful of bad entries still loads. Caps for max_problems / max_public_tests / max_hidden_tests. test_cp_dataset: 8 unit tests covering malformed-line skip, empty public/hidden skip, all three caps, missing-file handling, and an end-to-end load → CodeContestsEnv → SubprocessPythonRunner reward check on the synthetic 'add' / 'mul' problems. Smoke-tested against real data: prepare_cp_data.py --split test --max-samples 20 produced a 59 KB JSONL with 20 CodeContests problems (first: '1575_A. Another Sorting Problem'); the C++ loader read all 20 with parse_errs=0. Also enables test_cp_env / test_cp_dataset in CMakeLists.txt and Makefile, and updates docs/AGENTIC_ROLLOUT_SPEC.md progress table. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Introduces the Stage 1 “agentic CP environment + dataset” building blocks: a competitive-programming environment with public/hidden test splits, a sandboxed Python subprocess runner, utilities for extracting code from model output, and dataset preparation/loading support for CodeContests-style JSONL.
Changes:
- Added a sandboxed
CodeRunnerabstraction withSubprocessPythonRunner(resource limits + best-effort network isolation). - Added
Env/CodeContestsEnv(public feedback formatting + hidden-test reward) plusextract_code. - Added CP dataset tooling:
prepare_cp_data.py, a JSONL loader (cp_dataset.h), and new C++ tests wired into Make/CMake.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/training/test_cp_env.cpp | New end-to-end tests for extract_code, runner limits/isolation, and CodeContestsEnv reward/feedback. |
| tests/training/test_cp_dataset.cpp | New tests for JSONL loading caps/skips and basic env integration. |
| scripts/prepare_cp_data.py | New dataset prep script: HF parquet shards → JSONL with public/hidden tests. |
| include/training/code_runner.h | New sandboxed Python subprocess runner + resource limit enforcement. |
| include/training/env.h | New CP environment interface/impl and fenced-code extraction helper. |
| include/training/cp_dataset.h | New JSONL loader for CP problems + convenience wrapper. |
| docs/AGENTIC_ROLLOUT_SPEC.md | Updated progress checklist to reference new env/runner work. |
| Makefile | Added the new C++ tests to the CPP_TESTS list. |
| CMakeLists.txt | Added build targets for the new CP tests. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+24
to
+28
| #include <cstdio> | ||
| #include <cstring> | ||
| #include <string> | ||
| #include <vector> | ||
|
|
Comment on lines
+30
to
+36
| #include <cerrno> | ||
| #include <chrono> | ||
| #include <cstdio> | ||
| #include <cstdlib> | ||
| #include <cstring> | ||
| #include <string> | ||
| #include <vector> |
Comment on lines
+102
to
+108
| int sin[2], sout[2], serr[2], errpipe[2]; | ||
| if (pipe2(sin, O_CLOEXEC) || pipe2(sout, O_CLOEXEC) || | ||
| pipe2(serr, O_CLOEXEC) || pipe2(errpipe, O_CLOEXEC)) { | ||
| res.status = RunResult::Status::SpawnFailed; | ||
| res.spawn_error = "pipe2 failed: " + std::string(strerror(errno)); | ||
| return res; | ||
| } |
Comment on lines
+275
to
+276
| if (open2 && (pfds[2].revents & (POLLIN | POLLHUP | POLLERR))) | ||
| open2 = drain(errpipe[0], errpipe_buf, 4096, /*ignored*/res.stdout_truncated); |
Comment on lines
+124
to
+128
| // auto-closes on successful exec; parent reads EOF → exec worked). | ||
|
|
||
| // Disable CLOEXEC on errpipe write end so we can write to it on errors. | ||
| // (Actually we WANT CLOEXEC: on successful exec, parent sees EOF; on | ||
| // pre-exec failure, we write before exec.) |
Comment on lines
+65
to
+69
| auto take_tests = [](const nlohmann::json& arr, int cap) { | ||
| std::vector<TestCase> v; | ||
| if (!arr.is_array()) return v; | ||
| int n = std::min((int)arr.size(), cap); | ||
| v.reserve(n); |
Comment on lines
+79
to
+82
| std::string tag = text.substr(after, nl - after); | ||
| while (!tag.empty() && (tag.back() == ' ' || tag.back() == '\r' || tag.back() == '\t')) | ||
| tag.pop_back(); | ||
| std::transform(tag.begin(), tag.end(), tag.begin(), ::tolower); |
Comment on lines
+178
to
+180
| std::ostringstream os; | ||
| int show = std::min(public_show_, (int)p.public_tests.size()); | ||
| int passed = 0; |
Comment on lines
+181
to
+205
| bool any_pass = false; | ||
| for (int t = 0; t < show; ++t) { | ||
| const TestCase& tc = p.public_tests[t]; | ||
| RunResult r = runner_->run(code, tc.input, limits_); | ||
| std::string got = cp_env_detail::normalize_output(r.stdout_str); | ||
| std::string exp = cp_env_detail::normalize_output(tc.expected); | ||
| bool ok = (r.status == RunResult::Status::Ok) && (got == exp); | ||
| if (ok) { ++passed; any_pass = true; } | ||
| os << "=== sample test " << (t + 1) << "/" << show | ||
| << " [" << (ok ? "PASS" : "FAIL") | ||
| << " | " << cp_env_detail::status_tag(r.status) | ||
| << " | " << (int)(r.wall_time_s * 1000) << " ms]\n" | ||
| << "input:\n" << cp_env_detail::clip(tc.input, 512) << "\n" | ||
| << "expected:\n" << cp_env_detail::clip(exp, 512) << "\n" | ||
| << "got stdout:\n"<< cp_env_detail::clip(got, 512) << "\n"; | ||
| if (!r.stderr_str.empty()) | ||
| os << "stderr:\n" << cp_env_detail::clip(r.stderr_str, 512) << "\n"; | ||
| } | ||
| os << "(public: " << passed << "/" << show << " passed)\n"; | ||
| fb.observation = os.str(); | ||
| // Early-stop: if the first public test crashed with SPAWN_FAILED, the | ||
| // env itself is broken; bail rather than waste turns. | ||
| fb.stop = !any_pass && (show > 0) && | ||
| /* heuristic: any spawn failure → abort episode */ | ||
| false; |
Comment on lines
+18
to
+20
| | CP env + sandboxed Python runner — `Env` / `CodeContestsEnv` / `extract_code` ([env.h](../include/training/env.h)) + `SubprocessPythonRunner` w/ setrlimit + unshare net ([code_runner.h](../include/training/code_runner.h)) | ✅ done — 16/16 tests pass: AllOrNothing + FractionPassed reward, sample-test feedback formatting, hard timeout, RLIMIT_AS, CLONE_NEWNET isolation verified on this box ([test_cp_env.cpp](../tests/training/test_cp_env.cpp)) | | ||
| | Trainer rewrite (`Episode`, rollout driver, obs masking) | ⬜ not started (§3.1, 3.3, 3.5–3.8) | | ||
| | CP dataset loader (CodeContests / TACO → `CPProblem` list) | ⬜ not started (gated on §6.2) | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stage 1 — CP environment + dataset (commits 07fc1ee, 115189f)
CodeRunnerinterface +SubprocessPythonRunner(fork+setrlimit+poll+unshare CLONE_NEWNET). Documented sandbox caveats; designed so
bubblewrap/firejail/seccomp swap in as a subclass.
CodeContestsEnvwith AllOrNothing / FractionPassed reward shapes;public/hidden test split; per-turn feedback formatter.
extract_codeutility (parses fenced Python blocks).scripts/prepare_cp_data.py— HF CodeContests → JSONL; works againstTACO / other CP datasets via
--dataset.include/training/cp_dataset.h— JSONL loader with caps + gracefulmalformed-line skip.
CodeContests test split (165 raw → 20 valid problems @ 59 KB).