Repository navigation
fix(benchmark): preserve full checkpoint preparation within trial phase - #6618
Conversation
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
|
Folded into |
There was a problem hiding this comment.
Reviewer: model_agent · gpt-6.1-sol · OpenAI · runtime_reported · reasoning_effort=max · observation_id=b03ff20e1b97e1566d2adbf1d42f9e0fcedd546825a3c8466e93a9093f157504
Approval conclusion (author-owned PR; GitHub blocks formal self-approval)
更正原评审:撤回此前“普通文件入口仍30秒”的 P2。 我把缺少 fresh_replan 的 controller 请求直接交给 stub,绕过真实 parser,构造了生产合法入口无法到达的反例。peer 指出后已用真实文件解析与 preflight 转换复核;这是我的评审错误,不要求作者修改合法入口来满足非法测试。原失败日志保留为无效反例,不作为产品缺陷或性能证据。本完整结论替换同一条原评审,未改 PR head,也未合并。
动机
维护者需要在原始研究决策边界保存完整 checkpoint,即可供接续的代码、研究与原生会话来源。完整准备超过30秒时,原可选捕获会错过该点;本提案把显式完整 controller 捕获的观察窗口增加到90秒,同时继续服从原研究 phase 的累计截止时间。合法完成 preflight 与中断 hook 均获得这一增量,普通非 controller 路径保持30秒;没有新增人工准备或确认轮次。本次只验证可独立回滚的观察预算修复,不授予执行、评分、合并权限,也不证明科学收益、安装态或持续规模资格。成熟真实工作量的全 controller 捕获、持续运行和研究效果仍归原研究 Task;90秒只是有界观察预算修复。
改动思路
精确 head 072178167e8e3389986502922495f2cc9a32e919,实际共同祖先 b543d076995423544b0eeb1990d59d7480f48339。完整声明由原 parser 验证,controller 必须同时指定 fresh_replan 和三个 state participants。preflight 在原 typed replan 边界生成有效请求,删除原 trigger,再经 supplied_request 调用完成捕获;中断 hook 直接使用已验证请求。共享90秒常量放在现有 checkpoint provider,准入、原始来源和 writer barrier 留在原 owner。当前 PR 只交付观察预算增量,不扩大原 phase、机会、冷却或执行权限。
| 路径 | 合法行为 |
|---|---|
| controller 文件缺 fresh_replan | parser 拒绝,不能成为普通捕获反例 |
| 合法 fresh 文件直接 after-hook | pre_replan_hook_required,不晚捕原决策点 |
| preflight 有效 supplied request / 中断 hook | 90秒并受原 deadline 限制 |
| 普通非 controller / 功能关闭 | 原30秒 / 无额外IO |
这是源代码及测试路径说明,不是成熟原生性能图。增加观察预算有真实完整历史消费者,不能以裁剪历史取得较短时间;它也不能放宽 frozen trial 或原 phase 授权上限。
具体改动
完整审阅5文件 +104/-2 和两个带 DCO 的提交:共享 CHECKPOINT_CAPTURE_SECONDS=90,两处合法完整 controller 捕获消费;封存 CONTROLLER_SEAL_SECONDS=30 未变;中英文 README 披露观察预算变化;六个虚拟时钟用例保留原 phase 上限和可选失败不重放工作。
关键代码:parse_checkpoint_request 的55–58行拒绝缺失 trigger/participants;capture_before_replan 的有效请求删掉 trigger,保留原始声明 bytes 与 pending guard,才调用 capture_after_wake。因此后者按 supplied_request 判断90秒与真实入口相符。CheckpointWake._capture 按已验证的自身 request 选择同一常量,并与原 monotonic wake deadline 取最小值。controller_completion_deadline 保持独立封存窗口及原 phase。错误反例漏掉了前两个必经步骤,本次四个真实文件/preflight用例可以直接抓住该错误。
规范来源:docs/architecture/rfcs/hierarchical-agent-stride-control-v0.md#77-research-state-branching-and-matched-continuation,不可变版本 b543d076995423544b0eeb1990d59d7480f48339;criterion 7.7 Research-state branching and matched continuation,以及同版本 benchmark README 的 Controller-only native capsule transport,均按原完整来源、原决策点、原预算和未知结果恢复义务核验。关联预算增量已 implemented;不把该增量当整个研究 checkpoint acceptance 完成。
对主干的风险
初次四模块独立执行 105 passed / 28 skipped。补充真实 parser/preflight 四例在实际 base/head 各 4 passed,证明非法请求不能越过 parser,合法文件必须经 pre-hook 转换。有效内部请求使用同一35秒准备矩阵:实际 base 4 failed / 2 passed,head 6 passed;20秒 phase 始终拒绝,40秒与无额外 phase 上限的90秒路径通过。原无效 env-entry 反例及私有“修复”不再作为红绿证据。
进一步使用既有 Codex 二进制、隔离 home 和本地静态响应运行实际 source suite:57 passed。这覆盖真实原生 writer/readback、CLI settlement、SQLite/typed artifacts、source/cursor/profile 变化及重复边界;没有调用付费模型或活动 Goal。虚拟时钟不能证明成熟规模性能,Linux writer/真实 guest 与全 controller 持续资格仍按各自来源保留。
Ruff、diff、semantic advisory 后的完整 semantic smoke 通过。风险 canary 4direct和18/19selected通过;1项DSH maintainability advisory 保留,实际base→6618→6622完整报告仅归一化checkout根目录后相同(93 statements/51 decisions)。整体仍为 quality_receipt_missing,没有借作者receipt、修改阈值或声称全绿;未查询或等待CI。
我的整体评价
APPROVE,绑定上述未变 head;无当前阻塞 finding。交付判断为 justified_increment;long_horizon/user_experience 为 preserved:没有复制 authority、延期原研究 phase、裁剪见证或引入新确认轮次。成熟真实工作量的全 controller 捕获、持续运行和研究效果仍归原研究 Task;90秒只是有界观察预算修复。 原研究 owner 继续既有完整 qualification,公开批准不等于长期可靠性或研究效果成立。窗口选择已复用原 provider,有界 future-facing pass 不需再加通用超时层;应保留这次“真实 parser→preflight→有效请求”的反证资格,防止再次用不可达 stub 输入制造结论。
English verdict: APPROVE — 6618@072178167e8e3389986502922495f2cc9a32e919. Correction: the earlier file-entry P2 is withdrawn; its mocked request bypassed mandatory fresh_replan parsing and was unreachable from a legal declaration. Real file/preflight checks pass4/4 on base and head; valid35s window cases improve from expected4fail/2pass to6pass while20s phase remains binding; 57 real native/CLI/artifact source cases pass. This is a bounded observation-budget increment, not sustained controller/scientific qualification or merge authority. The independent quality receipt hold remains.
Full automatic controller checkpoints can miss mature research boundaries because the same 30-second window includes complete native/research preparation, stability checks and live-source capture. The original work correctly continues after an optional miss, but that does not establish checkpoint coverage.
Give both full controller capture hooks one shared 90-second window, capped by the original trial phase and carried unchanged into the controller. Post-release sealing retains its separate 30-second allowance and the same phase cap. Other capture declarations retain 30 seconds; direct controller calls retain their explicit timeout. First-attempt deduplication, source/cursor validation, original opportunity/cooldown, and default-off behavior are unchanged.
Validation:
Placement/refactor: reuse the existing benchmark provider transport owner and one shared window constant; no new module, protocol vocabulary or parallel decision owner. Core typed quota/artifact semantics and evaluator behavior are unchanged. The bilingual provider guide discloses the explicit-on timeout change. This bounded reliability qualification does not prove sustained capture success, strategy benefit or net efficiency.