Skip to content

fix(benchmark): retire research_loop from #6946 - #6952

Merged
huangruiteng merged 2 commits into
codex/decouple-benchmark-researchfrom
codex/benchmark-observation-retirement
Oct 20, 2026
Merged

huangruiteng merged 2 commits into
codex/decouple-benchmark-researchfrom
codex/benchmark-observation-retirement

Conversation

@loopx-agent

Copy link
Copy Markdown
Collaborator

Problem and outcome

The requested review of #6946 calls for removing research_loop. This contribution targets its integration branch and removes the bundled candidate/research initializer, deferred successor, fixed Vision mutation, forced evidence_conditioned_v1 and preset-only prompt. Ordinary Goal/Todo/replan/Explore determine continuation; independent execution_observation="local_probe" remains available.

Both retired configuration fields fail before runner setup or wake creation, including false/zero values, so old configurations cannot silently select a different experiment. New trials choose observations and stride independently. Historical configurations, trajectories, receipts and outcomes retain their original revision and treatment.

The original-Goal initialization repair and the shared TypeScript evidence-reference compatibility fix from #6946 are retained. The ordinary planner receives the full first task instruction unless an explicit objective was supplied; later phases preserve that objective. Measurement guidance lives beside its existing producer, with no new capability, lifecycle or configuration owner. The Python code is the existing benchmark provider adapter; shared Explore semantics remain TypeScript owned.

Validation

  • Task-entry configuration, installation assembly, real fresh SQLite bootstrap and later-phase retention: 96 passed. Harbor 0.20.0 is installed; none of these cases skipped.
  • Real local process/CLI, File/SQLite execution→Explore→replan→changed execution and independent stride-off/wait/Stop: 14 passed. Ordinary scheduler/Host stage continuation, accepted requirements, owner withdrawal and once-only settlement: 4 passed. Actors are deterministic; no live model research benefit is claimed.
  • Five TypeScript evidence/context/history/stride files: 91 passed, zero skips.
  • Shared worker runtime and native-Goal regression: 94 passed. 39 opt-in Docker checkpoint-writer cases skipped; this contribution does not change that writer, and no Docker installation or live checkpoint restore qualification is claimed.
  • Python compilation, Ruff, diff hygiene, paired-RFC gate, diff-scoped vocabulary advisory and public/private scan completed. No new supported vocabulary carriers were detected; the advisory does not inspect all dynamic semantics.
  • Standard repository premerge: 19 executed checks passed (five direct checks also passed); the benchmark-sensitive maintainer-review hold remains. An initial attempt stopped because the clean checkout lacked the TypeScript npm dependency; it is preserved as a failed environment check, followed by npm ci --ignore-scripts and a rerun.

The first combined focused run had one test-environment error (a unit test used an unstaged Harbor CLI path); the corrected task-entry suite passed all 96 cases. Real integration results are kept separate from synthetic installation/host captures. No paid benchmark, hidden evaluation or model call is used as validation.

Scope and integration

Base: codex/decouple-benchmark-research at 90d51cf24a19ea9ee8852b79361c9a8af899ea3a. This PR supplies the retirement delta for #6946 without overwriting its author's branch. It preserves the contributor's independent observation and reference-validation work. Update #6946's description after adopting this patch so it no longer advertises the retired preset.

Applied bounded refactor: remove the obsolete preset module and place environment selection beside the existing observation producer. Removed tests solely for the retired initializer; retained the independent process, attribution, continuation, wait/Stop and acceptance coverage. No UI behavior or first-screen change. No scoring, task semantics, submission policy, budgets or authorization owner changes.

Merge remains subject to exact-head review, required validation and readiness. This retirement is a maintainer-directed simplification; engineering checks do not prove autonomous model equivalence or an efficiency gain. Experiments resume only after the resulting latest-main bundle is qualified.

中文:直接移除 research_loop 开关、固定两阶段、强制 stride 与预设提示;执行观测独立保留,后续推进走普通 Goal/Todo/replan/Explore。旧字段明确报迁移错误,历史实验固定原版本和归属。真实进程、File/SQLite、普通接续与 TypeScript 证据链回归通过;模型自主收益和最新版本 ckpt 恢复需另外验证。

Submitted head: 989e67bd0f6d66c4b03ead5f7c782610dfdd4801, tree 2f5227ed1df2e38da3498963f1643bda8c9e9de1.

Written by: model_agent — OpenAI GPT-6.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
@huangruiteng
huangruiteng merged commit 0985397 into codex/decouple-benchmark-research Oct 20, 2026
4 of 6 checks passed
@huangruiteng
huangruiteng deleted the codex/benchmark-observation-retirement branch October 20, 2026 16:22

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

Reviewer: model_agent; gpt-6.1-sol; OpenAI; runtime_reported; reasoning_effort=max

动机

benchmark 操作者需要在普通任务中测量执行,而不额外声明研究模式。旧研究开关会预建候选与研究两个阶段、改写接续范围并强制策略;本次按维护者要求删除这套捆绑,执行观测继续独立使用。新配置沿用普通 Goal/Todo/replan/Explore;旧开关在产生 runner 或 wake 副作用前明确报迁移错误。本次不证明模型自主研究效果、不改历史结果,也不修改评分、提交机会、原预算或 Stop 权限。不处理会让一次普通测量继续携带额外研究任务和不透明的策略组合,增加维护与实验归因成本。这里验收的是退役这一明确请求,不把工程回归等同于模型研究收益。

改动思路

观测工具只负责测量及证据来源,后续做什么仍由原 Goal、已接受要求和普通任务规划决定。删除的内容包括固定后继创建、初始化 Vision 写入、强制 stride 和预设提示;保留的内容包括独立 producer、真实进程测量、Explore 写回和已有引用规则。这个边界直接去掉额外编排,避免再维护两个任务决策入口;迁移错误只拦截已退役字段,不会阻塞普通配置。当前 PR 只贡献给 #6946 的集成分支,后续主干集成和实验资格分别验收。

具体改动

规范依据是维护者公开评审 https://github.com/loopx-project/loopx/pull/6946#pullrequestreview-5586393819,固定版本标识 pullrequestreview-5586393819;唯一直接请求为“research loop直接去掉吧”。准则 remove-research-loop 在本 head 已实现:活动代码不再导出预设或创建研究阶段,旧 kwarg 和环境变量明确拒绝;普通入口仍保留首次完整目标和后续 phase 的原目标。

关键代码讲解

  • BenchmarkCodex.__init__ 首先拒绝已退役的 kwarg,即使值为 False;正常配置独立解析观测与 stride,不再强制研究策略。
  • BenchmarkCodex._prepare_phase 只在普通 seeded-todo 入口预置任务,loopx-planned 保持由既有 planner 产出工作;没有隐藏研究后继或初始化 lease 操作。
  • execution_observation_from_env 统一处理 worker 与提示的测量选择,拒绝遗留环境变量;run_once 在创建 wake 之前执行该检查。测量指引仍要求复用已有执行、匹配候选和输入、缺测未知,不为回执重复实验。

12 组相同普通配置在固定基线和当前 head 的环境、完整规划提示及 replan 回执逐字相同。相同四组真实 SQLite 初始化两版本均通过;新退役断言在旧版本四组均失败,在当前版本通过。当前任务入口 96 项、真实 File/SQLite 执行与接续 18 项、TypeScript 五文件 91 项、共享 worker/native-Goal 94 项通过。标准 premerge 的 19 项检查及 5 项直接检查在准确 head 通过。

对主干的风险

这是有意的配置不兼容:旧 research_loop=False 和环境值 0 也须删除,不能静默改为另一种实验。文档明确建立新 trial,旧版本、配置和历史回执保留。普通任务的等待、Stop、目标、原生评分与权限 owner 没有被改写;独立观测关闭时完整规划提示和环境保持基线一致。39 项 Docker checkpoint-writer 检查属于未显式开启的可选测试,未计为通过;没有 Docker 安装、线上模型或付费实验资格声明。首轮缺 TypeScript 依赖和一项测试环境错误均保留,依赖恢复及最终相关检查已通过。benchmark 敏感变更仍需维护者合并规则与最终 readiness,本评审不会自动提供合并权限。

语义与 CI 对齐

复用既有 Goal/Todo/Explore 语义;producer 选择为 benchmark 本地 provider 配置,没有新增共享词汇或控制平面义务。观测说明是提示,旧字段拒绝是强制迁移错误,两者分别验证。按当前 Goal 的 wait_for_ci=false,不查询或等待 CI;本地必需检查已经执行,未将平台 merge 状态当作绿色 CI。

我的整体评价

准确版本 989e67bd0f6d66c4b03ead5f7c782610dfdd4801 满足维护者要求的退役范围。长程工作契约保持:已请求的后续任务仍通过普通调度、验收与一次结算继续;去掉预建研究阶段是已接受的处理变化,不声称模型行为等价。使用体验改善为普通配置和独立测量,保留可操作的旧配置报错。代码与文档删除是同一边界的简化,保留证据能力和正常回归,规模合理;历史恢复、实际模型采用与后续实验收益继续单独验证。当前没有发现阻塞此增量的缺陷。

English verdict: APPROVE

The exact head removes the requested research preset while preserving independently selected execution observations and ordinary continuation contracts. Local real-process/provider checks and baseline comparisons passed; live research benefit and Docker qualification remain outside this review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants