Skip to content

fix(benchmark): preserve full checkpoint preparation within trial phase - #6618

Merged
huangruiteng merged 2 commits into
mainfrom
codex/checkpoint-capture-followthrough-20261016
Oct 16, 2026
Merged

huangruiteng merged 2 commits into
mainfrom
codex/checkpoint-capture-followthrough-20261016

Conversation

@loopx-agent

Copy link
Copy Markdown
Collaborator

Full automatic controller checkpoints can miss mature research boundaries because the same 30-second window includes complete native/research preparation, stability checks and live-source capture. The original work correctly continues after an optional miss, but that does not establish checkpoint coverage.

Give both full controller capture hooks one shared 90-second window, capped by the original trial phase and carried unchanged into the controller. Post-release sealing retains its separate 30-second allowance and the same phase cap. Other capture declarations retain 30 seconds; direct controller calls retain their explicit timeout. First-attempt deduplication, source/cursor validation, original opportunity/cooldown, and default-off behavior are unchanged.

Validation:

  • Focused capture, handoff, CLI and archive tests: 106 passed, 15 platform/explicit-integration skips.
  • Isolated Linux Docker/native HTTP checks: 95 passed; one external-controller native fixture remained explicitly skipped. Separate real native-session qualifications used a mature retained point: full visible history and research evidence adopted in a non-root offline receiver; full Docker/controller/native-state recapture and byte verification passed with conserved history, incumbent, opportunity and remaining phase. No inference or grading was invoked by those real-case qualifications.
  • Premerge: all direct checks and 19 selected catalog/risk/boundary checks passed. The existing maintainability ratchet remains advisory. Benchmark-sensitive manual review remains required; this is not merge readiness or merge authorization.
  • Ruff, compile, diff hygiene and added-line public/private scans passed. Real-case artifacts stay private.

Placement/refactor: reuse the existing benchmark provider transport owner and one shared window constant; no new module, protocol vocabulary or parallel decision owner. Core typed quota/artifact semantics and evaluator behavior are unchanged. The bilingual provider guide discloses the explicit-on timeout change. This bounded reliability qualification does not prove sustained capture success, strategy benefit or net efficiency.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
@songoow

songoow commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator

Folded into edgebenchtest (commit c1eb30315) as the experiment-preparation aggregate branch, so the next EdgeBench 24h run can pin one commit. Content is unchanged: 072178167 is an ancestor of that head, with the same 90-second capture window and phase cap. Leaving this PR open for its own review; no merge authority is claimed here.

@loopx-agent loopx-agent left a comment •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · gpt-6.1-sol · OpenAI · runtime_reported · reasoning_effort=max · observation_id=b03ff20e1b97e1566d2adbf1d42f9e0fcedd546825a3c8466e93a9093f157504

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

更正原评审:撤回此前“普通文件入口仍30秒”的 P2。 我把缺少 fresh_replan 的 controller 请求直接交给 stub,绕过真实 parser,构造了生产合法入口无法到达的反例。peer 指出后已用真实文件解析与 preflight 转换复核;这是我的评审错误,不要求作者修改合法入口来满足非法测试。原失败日志保留为无效反例,不作为产品缺陷或性能证据。本完整结论替换同一条原评审,未改 PR head,也未合并。

动机

维护者需要在原始研究决策边界保存完整 checkpoint,即可供接续的代码、研究与原生会话来源。完整准备超过30秒时,原可选捕获会错过该点;本提案把显式完整 controller 捕获的观察窗口增加到90秒,同时继续服从原研究 phase 的累计截止时间。合法完成 preflight 与中断 hook 均获得这一增量,普通非 controller 路径保持30秒;没有新增人工准备或确认轮次。本次只验证可独立回滚的观察预算修复,不授予执行、评分、合并权限,也不证明科学收益、安装态或持续规模资格。成熟真实工作量的全 controller 捕获、持续运行和研究效果仍归原研究 Task;90秒只是有界观察预算修复。

改动思路

精确 head 072178167e8e3389986502922495f2cc9a32e919,实际共同祖先 b543d076995423544b0eeb1990d59d7480f48339。完整声明由原 parser 验证,controller 必须同时指定 fresh_replan 和三个 state participants。preflight 在原 typed replan 边界生成有效请求,删除原 trigger,再经 supplied_request 调用完成捕获;中断 hook 直接使用已验证请求。共享90秒常量放在现有 checkpoint provider,准入、原始来源和 writer barrier 留在原 owner。当前 PR 只交付观察预算增量,不扩大原 phase、机会、冷却或执行权限。

路径 合法行为
controller 文件缺 fresh_replan parser 拒绝,不能成为普通捕获反例
合法 fresh 文件直接 after-hook pre_replan_hook_required,不晚捕原决策点
preflight 有效 supplied request / 中断 hook 90秒并受原 deadline 限制
普通非 controller / 功能关闭 原30秒 / 无额外IO

这是源代码及测试路径说明,不是成熟原生性能图。增加观察预算有真实完整历史消费者,不能以裁剪历史取得较短时间;它也不能放宽 frozen trial 或原 phase 授权上限。

具体改动

完整审阅5文件 +104/-2 和两个带 DCO 的提交:共享 CHECKPOINT_CAPTURE_SECONDS=90,两处合法完整 controller 捕获消费;封存 CONTROLLER_SEAL_SECONDS=30 未变;中英文 README 披露观察预算变化;六个虚拟时钟用例保留原 phase 上限和可选失败不重放工作。

关键代码:parse_checkpoint_request 的55–58行拒绝缺失 trigger/participants;capture_before_replan 的有效请求删掉 trigger,保留原始声明 bytes 与 pending guard,才调用 capture_after_wake。因此后者按 supplied_request 判断90秒与真实入口相符。CheckpointWake._capture 按已验证的自身 request 选择同一常量,并与原 monotonic wake deadline 取最小值。controller_completion_deadline 保持独立封存窗口及原 phase。错误反例漏掉了前两个必经步骤,本次四个真实文件/preflight用例可以直接抓住该错误。

规范来源:docs/architecture/rfcs/hierarchical-agent-stride-control-v0.md#77-research-state-branching-and-matched-continuation,不可变版本 b543d076995423544b0eeb1990d59d7480f48339;criterion 7.7 Research-state branching and matched continuation,以及同版本 benchmark README 的 Controller-only native capsule transport,均按原完整来源、原决策点、原预算和未知结果恢复义务核验。关联预算增量已 implemented;不把该增量当整个研究 checkpoint acceptance 完成。

对主干的风险

初次四模块独立执行 105 passed / 28 skipped。补充真实 parser/preflight 四例在实际 base/head 各 4 passed,证明非法请求不能越过 parser,合法文件必须经 pre-hook 转换。有效内部请求使用同一35秒准备矩阵:实际 base 4 failed / 2 passed,head 6 passed;20秒 phase 始终拒绝,40秒与无额外 phase 上限的90秒路径通过。原无效 env-entry 反例及私有“修复”不再作为红绿证据。

进一步使用既有 Codex 二进制、隔离 home 和本地静态响应运行实际 source suite:57 passed。这覆盖真实原生 writer/readback、CLI settlement、SQLite/typed artifacts、source/cursor/profile 变化及重复边界;没有调用付费模型或活动 Goal。虚拟时钟不能证明成熟规模性能,Linux writer/真实 guest 与全 controller 持续资格仍按各自来源保留。

Ruff、diff、semantic advisory 后的完整 semantic smoke 通过。风险 canary 4direct和18/19selected通过;1项DSH maintainability advisory 保留,实际base→6618→6622完整报告仅归一化checkout根目录后相同(93 statements/51 decisions)。整体仍为 quality_receipt_missing,没有借作者receipt、修改阈值或声称全绿;未查询或等待CI。

我的整体评价

APPROVE,绑定上述未变 head;无当前阻塞 finding。交付判断为 justified_increment;long_horizon/user_experience 为 preserved:没有复制 authority、延期原研究 phase、裁剪见证或引入新确认轮次。成熟真实工作量的全 controller 捕获、持续运行和研究效果仍归原研究 Task;90秒只是有界观察预算修复。 原研究 owner 继续既有完整 qualification,公开批准不等于长期可靠性或研究效果成立。窗口选择已复用原 provider,有界 future-facing pass 不需再加通用超时层;应保留这次“真实 parser→preflight→有效请求”的反证资格,防止再次用不可达 stub 输入制造结论。

English verdict: APPROVE — 6618@072178167e8e3389986502922495f2cc9a32e919. Correction: the earlier file-entry P2 is withdrawn; its mocked request bypassed mandatory fresh_replan parsing and was unreachable from a legal declaration. Real file/preflight checks pass4/4 on base and head; valid35s window cases improve from expected4fail/2pass to6pass while20s phase remains binding; 57 real native/CLI/artifact source cases pass. This is a bounded observation-budget increment, not sustained controller/scientific qualification or merge authority. The independent quality receipt hold remains.

@huangruiteng
huangruiteng merged commit 9aeb496 into main Oct 16, 2026
20 of 31 checks passed
@huangruiteng
huangruiteng deleted the codex/checkpoint-capture-followthrough-20261016 branch October 16, 2026 09:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants