Repository navigation
harness: add bounded VATS/GVO runtime and benchmark adapters - #488
amy-why-3459 wants to merge 9 commits into
Conversation
Separate editable lineage from candidate-specific verification evidence. Retain scoped attempts and dependency epochs, select complete development candidates, and enumerate bounded completion plans with shared preparation and cost caps. Cover failed-parent repair, stale evidence, deterministic selection and planner feasibility with CPU controls; document the shared execution integration boundary. Agent-assisted-by: Codex (implementation, tests and documentation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent non-voting review: not approved; acceptance and execution evidence are incomplete.
Reviewed head 853c92f35f3b4d3df841958ba33568f690f86c5d against base 8aba657da79e8ed9f6e6fbb33437e9d9fdd8f1c7, on the other-author draft PR. I inspected both new executable modules, the synthetic fixture and both new test files, plus machine CI evidence. AGENTS and repository review guidance were checked. No other reviews, PR result narrative or mixed Issue discussion informed this assessment.
Code facts established by static inspection
- Search evidence is bound to the complete registered Attempt, including candidate/artifact, context, check, execution key, reservation and lease identity. Receipt registration rejects changed bindings/conflicting duplicate receipts; evidence selects the latest attempt and excludes invalidated dependencies and INCONCLUSIVE outcomes. Acceptance requires every required check to pass, including the objective, and preserves the qualified root on a score tie.
- Candidate edit ancestry is distinct from generation/evidence dependencies: a fully checked repaired child can be selected despite a failed or incomplete edit parent, while inherited dependency invalidation removes affected descendants from admissible selection. This is a substantive semantic distinction from treating every parent failure as automatic descendant failure. It should not be assumed to settle the older bounded-successor RFC's acceptance contract.
- Completion planning constructs prerequisite closures, counts shared jobs once, applies cost/job/attempt bounds and conflicts, rejects outstanding unsettled work, and returns search identity/state revision without spending the supplied budget. A plan is not an execution reservation or proof that its costs were incurred.
- The inspected tests cover wrong receipt bindings, late results after invalidation, same-byte distinct candidates, immutable identities, fresh retries after INCONCLUSIVE, bounded fallback, and a three-candidate exhaustive planning oracle. The engine/skill-labelled fixture uses retained files containing synthetic numbers; it is not two real adapter integrations, independent holdout qualification or a performance experiment.
I did not identify a confirmed actionable defect in this bounded static pass. One useful additional coverage case is cross-candidate prerequisite closure: selecting one completion seed can incidentally complete another candidate. The current oracle fixture shares setup but does not exercise that coupling. Keep the seed-enumeration bound distinct from a bound on all incidentally completed candidates; this is a coverage/contract clarification, not a demonstrated runtime defect.
Important limits: SearchState revision checks operate on the supplied in-memory state. These modules do not themselves implement durable compare-and-swap, crash recovery, actual reservation/lease authorization, cancellation, process cleanup, holdout access control or atomic budget accounting. Nonempty reservation/lease identifiers are identity bindings, not proof those resources were reserved. The PR's isolated scope is offline candidate trees and verification planning; absence of a runner here is not by itself a code defect, but these helpers must not be presented as completion of those wider operational guarantees.
Evidence actually inspected
- lint-and-smoke 134586515319: exact head appears in the raw log; Ruff passes and 8 passed in 0.05s. Those smoke tests do not establish the new search-tree/planning tests.
- installed-package 3.10 134586617451: raw successful sdist/wheel build at synthetic merge
b92eff795921472b952a4e8e0401cffce4a866e0of this head into the stated base. - At inspection, full-check 3.10/3.11 and Windows were still running, macOS queued; browser was skipped. Installed-package 3.11, references and declarations had success metadata. Completed full-check logs were not available to verify the new tests.
No repository tests or code were executed by me. The detailed current AC remains unavailable in a safely isolated form; the earlier #363 requirements are not automatically the full edited VATS/GVO contract. Changed documentation was not used as acceptance authority or as proof of implementation. This COMMENT records a limited independent code/evidence review, not whole-RFC acceptance, workflow efficacy, permission for pilots, code changes or merge.
david6666666
left a comment
There was a problem hiding this comment.
One numeric boundary to address in the offline search helpers.
| def verified_gain(state): | ||
| selected = accept(state) | ||
| check = state.contract.objective_check | ||
| gain = (evidence(state, selected.candidate_id)[check].objective |
There was a problem hiding this comment.
Both receipts can have finite objectives while their difference overflows. With a root of -1e308 and a selected candidate of 1e308, verified_gain() returns inf, which downstream strict JSON cannot serialize. Could we reject a non-finite gain explicitly (or bound the objective domain) and add a regression for both objective directions?
There was a problem hiding this comment.
Valid for the original implementation; addressed in abe72f6 and retained in the current PR head dc7292a. verified_gain() explicitly rejects a non-finite subtraction before the value can reach strict JSON. The regression covers finite -1e308/1e308 objectives in both maximize and minimize directions.
Rechecked on the pushed revision: test_speculative_search.py passes as part of the local affected-area suite (216 passed, 54 subtests passed); Ruff also passes. The current-head full CI is running; these local results are not a full-suite or campaign-acceptance claim.
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent non-voting CI follow-up to review 5951024572: completed regression evidence, but not whole-scope approval.
Head 853c92f35f3b4d3df841958ba33568f690f86c5d, base 8aba657da79e8ed9f6e6fbb33437e9d9fdd8f1c7. I independently retrieved the newly completed machine logs rather than treating success metadata or author statements as test evidence.
Both full-check logs in Ready run 44240402818 show checkout b92eff795921472b952a4e8e0401cffce4a866e0, explicitly merging this exact head into this base:
- Python 3.10 / 134586617402: Ruff passes; 5149 passed, 36 skipped, 671 subtests passed, 1 warning, 474.66s.
- Python 3.11 / 134586617133: Ruff passes; 5149 passed, 36 skipped, 671 subtests passed, 1 warning, 510.69s.
These invoke pytest -rs across the configured tests directory. I rechecked the current test_speculative_search.py and test_verification_plans.py: 19 and 10 test functions before parameter expansion, with no skip decorators found; neither appears in the complete skip summaries. This is aggregate execution evidence for the newly inspected offline tests, not an individually verbose PASS list or an independent local rerun. The 36 skips include optional dependencies, real Cannbot qualification and hardware/platform-specific cases; they are not 36 passes.
Raw platform logs at the same merge report Windows 8 passed + 25 passed, and macOS 8 passed. Their selected modules concern plugin/DSH and Windows portability/session behavior, not a full cross-platform execution of the new planner/tree tests. Dashboard-browser is skipped, not passed. Other Ready jobs are now successful by metadata. This resolves the pending CI evidence noted previously.
The isolated specification clarification confirms max_targets limits the union of completion seeds, not the number of candidates incidentally completed by prerequisite closure. Consequently I do not classify extra incidental completions as a cap violation; my prior cross-candidate prerequisite suggestion remains additional coverage only.
The offline code assessment is unchanged. Receipt identity/dependency bookkeeping and proposal generation do not establish durable CAS/recovery, authenticated reservations/leases, real cancellation, holdout isolation or actual Engine/Skill adapter efficacy. Detailed current RFC acceptance provenance remains incomplete; old bounded-successor requirements must not silently substitute for the edited VATS/GVO scope. No code/test execution, campaign restart, model/device/paid experiment, code change or merge was performed by me.
Source accounting: isolated metadata for inline4609449850@2026-11-18T05:09:44Z contains no direct owner mention or independent machine locator. Its inseparable technical narrative was not consumed or adjudicated. This follow-up is driven by the independently verified CI transition, not that source's opinion.
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent incremental review — NOT APPROVED. COMMENT rather than a Request changes vote.
Range: previous reviewed head 853c92f → abe72f6 → current 5ee4181; base remains 8aba657. Previous own review: #488 (review) . This is a substantial expansion from offline primitives into persisted VATS execution, CPU GVO roles, continuation/final control and Engine/Skill adapters, not merely the earlier passing offline tests.
P2 — Persisted not-started expiry strands Search recovery. In harness/vats.py:391–421, _execute atomically writes search.pending (and registers a verification Attempt) with its claim, then calls WorkControl.launch. In work_control.py:171–188, expiry between claim and launch conclusively records settled/not_started and releases the domain, but leaves search.pending untouched. Since the adapter never ran, vats.py:356–364 cannot find its receipt and permanently reports Unknown execution; finish at 635–640 also rejects pending work. A scheduling pause or approaching absolute deadline can therefore prevent a search with a valid qualified root from completing normally even though execution is known not to have started. Reconcile the durable not_started terminal record with the pending Attempt/work-block state, without relaunching uncertain work. Add a Search-level claim→expiry→recover/limit-finish test; the existing WorkControl-only expiry test does not exercise this composition.
P2 — Known overspend is ignored while cleanup is unknown. work_control.py:213–217 takes the unknown branch whenever cleanup_verified is false, even when actual_ms is known and exceeds the reservation. available() at 103–105 subtracts only the old cap, and another domain can still claim. Static example: campaign cap 1000, final reserve 100, claimed cap 200, then settle(actual_ms=950, cleanup_verified=False). spent remains zero, blocked remains false and available reports 700, allowing another 700 commitment despite known spend already consuming the final reserve. Retain the domain/unknown-work restrictions, but conservatively account for the known expenditure lower bound or block on known overspend; reconcile later without double charging. Add the combined known-overrun + unknown-cleanup + second-domain claim regression. This is source-path reasoning, not a locally executed reproducer.
Other inspected changes / prior findings: the latest Engine post-execution frozen-resource recheck and expanded Skill controller identity are present; their new mutation test is a source assertion, not independently executed evidence here. GVO separates forced continuation scorer targets from actor groups, verifies retained training labels, and uses retained checkpoints for Reset/Carry recovery. I did not establish an additional concrete training-formula defect in this bounded pass. CPU update timing is measured after update returns; block-on-overrun is not preemptive termination. Native host/process cleanup is not filesystem isolation or a GPU scheduler reservation. The previous completion-seed clarification remains unchanged and is not reopened.
Evidence: exact-head lint-and-smoke job 134828062677 raw log identifies 5ee4181, Ruff success and 8 passed in 0.04s for test_smoke.py/test_claim.py. That does not execute the new VATS/GVO tests. Current full-check 3.10/3.11 and Windows are in progress, macOS queued, dashboard-browser skipped; package/reference/declaration checks have success metadata. Cancelled intermediate-head jobs and historical 5149-test results are not current-head passes.
Before approval, address the two control-state/accounting paths and demonstrate their regression tests plus applicable new-head full integration checks. Current Issue #363 normative sections were isolated from opinions: its proposed recovery/budget/final/dual-pilot requirements are not evidence of maintainer acceptance or authorization for real experiments. Synthetic CPU fixtures, finite-edit softmax roles and a comparator bridge do not establish both real Engine/Skill VATS/GVO/final pilots or performance gains. Whole-RFC acceptance provenance remains unresolved.
Performed: actual incremental code/test inspection across execution/accounting, adapters/CLI, roles/GVO/continuation/final; current AGENTS, isolated normative requirements, own-review deduplication and machine CI evidence. Not performed: repository-code or test execution, model/device/provider trials, paid experiments, edits, CI rerun or merge. No other reviewers' opinions were read or used. Optional follow-up: keep explicit tests distinguishing known not-started work, unknown launched work and known overspend with unknown cleanup, rather than testing these states only in isolation.
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent incremental review: 5ee4181 → bd2bfc4; base remains 8aba657. Previous own review: #488 (review) . COMMENT, still not approved.
The four-file delta adds an initial_conditions digest to Search.start and requires equality across GVO examples. It includes candidate/provenance, dependency epochs, actual execution/reservation/lease identities and receipt evidence, excluding only receiving attempt_id/search_id. I cross-checked import_qualified_root: its receiving-scope rebinding changes those two excluded fields, so independent searches can still share the same qualified evidence without falsely differing only by their local IDs. The added assertion rejects a differing digest. The documentation clarifies use of Comparator.holdout_scope, not an executable final-gate change. No new confirmed source defect was identified in this narrow delta.
Neither prior P2 is resolved by this commit. work_control.py is the identical blob, and the relevant VATS claim/launch/recover paths are unchanged:
- A conclusively not-started expired launch still leaves search.pending (and, for verification, its Attempt) behind. Receipt-only recovery cannot obtain a receipt from an adapter that never ran; normal finish remains blocked. Coordinate the durable not_started outcome with search/work-block state and add the composed expiry/recovery regression.
- Known actual_ms above cap with unknown cleanup still takes the unknown branch without blocking or charging the known overrun; available() retains only the original reservation. Other domains can still claim against an overstated available budget. Preserve resource exclusion while conservatively accounting/blocking the known overspend, with an idempotent reconciliation regression.
These remain static code-path findings, not executed failures. The new initial-condition equality does not address either state/accounting path.
Current evidence: raw lint-and-smoke job 134831307114 identifies this revision, Ruff success and 8 passed in 0.07s for smoke/claim tests. At inspection the full-check, host and package jobs were still in progress/queued; dashboard-browser was skipped. No completed current-head new-runtime regression result was established. Prior-head aggregate passes are not relabeled as this head's results.
Before approval: resolve the two P2 paths, execute the targeted combined-state regressions and applicable full checks, and retain the previously stated distinction between CPU/synthetic controls, proposed Issue #363 acceptance and real Engine/Skill pilots. The earlier acceptance-provenance and real-pilot gaps are not cleared here.
Non-blocking coverage suggestions: construct same-root-byte groups with changed actual receipt evidence/epochs to test digest derivation, plus same evidence with different receiving IDs to test admissibility. For compatibility with previously persisted experimental searches, define an explicit version/migration policy: records predating initial_conditions cannot be consumed by direct dictionary indexing without that field; do not fabricate their historical starting evidence.
Performed: all four changed files, relevant root-import and unchanged failure paths, own-review/head checks and machine smoke evidence. Not performed: repository-code/test execution, real model/device/provider trials, edits, CI rerun or merge. No other reviewers' opinions were read or used.
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent incremental review: bd2bfc4 → 96e3c50; unchanged base 8aba657. Previous own checkpoint: #488 (review) . COMMENT, not approval.
I inspected all seven changed files. The delta removes direct Model imports from the new Harness modules and dispatches through the injected snapshot operations. For the supplied CPU Snapshot, choose/predict wrap the same choose/dot functions; train reconstructs Choice with the same tuple conversions previously performed in gvo, then passes the same reference/actors/values/pairs/settings to update; restore constructs the checkpoint's Snapshot type. The Harness finite-return validation remains equivalent for recorded numeric observations. I found no new concrete parameter/serialization defect in this bounded refactor. The documented dependency direction is improved; this is not an executed integration result.
The two previous P2 findings remain unresolved. I rechecked the current failure paths rather than relying on discussion state:
- work_control.py is still blob 3a0993c42845765146573e31d4bf1a1ce3e60628. Expired launch persists settled/not_started without reconciling VATS pending/Attempt state; recover still requires an adapter receipt that cannot exist for unstarted work, and finish still rejects pending work.
- The unknown-cleanup branch still ignores a known actual_ms overrun for blocking/available-budget purposes, while available subtracts only the old reservation. The previously described 1000-cap/100-final-reserve/200-claim/950-known-use sequence can still expose 700 available to another domain. Resource exclusion alone does not make this budget view conservative.
Before approval, fix those two composed-state/accounting paths and supply their targeted regression execution plus the applicable current-head complete checks. This dependency refactor does not resolve them. Earlier initial-evidence consistency and frozen-resource improvements remain present; no formerly resolved item is reopened.
Machine evidence inspected: exact-head lint-and-smoke job 134833551072 raw log identifies 96e3c50, Ruff success and 8 passed in 0.06s for tests/test_smoke.py and tests/test_claim.py. At inspection full checks, Windows/package/reference jobs were running, macOS queued and dashboard-browser skipped. This does not establish completion of the new search/GVO runtime tests. Historical-head results are not current-head passes.
Optional coverage: add direct injected-role boundary tests for the round-trip Choice document, checkpoint restore and same-input predict/train behavior, so future non-CPU role implementations cannot silently diverge at this seam.
Performed: seven-file static delta review, current original failure-path reread, own-review/head deduplication and raw CI evidence. Not performed: repository-code/test execution, real model/device/provider pilots, edits, CI reruns or merge. Prior proposed Issue #363 acceptance provenance and real-pilot boundaries remain unchanged. No other reviewers' opinions were read or used.
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
david6666666
left a comment
There was a problem hiding this comment.
Two integration failures on the current Ready merge with main.
| from inferrsi.harness.work_control import WorkControl | ||
| from inferrsi.model.search_roles import Snapshot | ||
|
|
||
| from .search_benchmarks import EngineEvaluator |
There was a problem hiding this comment.
Could this import move into the engine command path? Loading the ordinary project CLI now reaches Benchmark through search_benchmarks and improvement, even when search is never used. The existing lifecycle isolation test fails on the Ready merge.
There was a problem hiding this comment.
Correct for the reviewed revision. Fixed in bb6ecbd: EngineEvaluator is imported inside the explicit engine-command path, after the status/export branches. Ordinary project CLI loading no longer imports Benchmark through this module.
Rechecked at pushed head dc7292a: test_project_lifecycle_never_imports_benchmark, the project lifecycle suite and module-boundary checks pass within the local affected-area run (216 passed, 54 subtests passed). The current-head full CI is running.
| raise ValueError("Frozen benchmark resources changed") | ||
| output.mkdir(parents=True) | ||
| candidates.retain(candidate / "candidate.py", output / "candidate.py") | ||
| protocol = task_protocol.protocol(self.frozen) |
There was a problem hiding this comment.
The Ready merge with current main no longer has task_protocol.protocol(), so this adapter raises before the resource-mutation guard is exercised. Could this use main’s packaged child-task API and rerun that regression on the merge tree?
There was a problem hiding this comment.
Correct for the reviewed merge. Fixed in bb6ecbd after integrating main cf671a8: the adapter uses task_protocol.run() for arguments/assessment in isolated child processes, with the remaining campaign deadline.
Rechecked at pushed head dc7292a: both frozen-resource mutation regressions and the protocol-child tests pass. This revision also adds invalid-deadline rejection and an Engine arguments → execution → assessment test showing remaining time decreases under the original deadline. The affected-area run reports 216 passed, 54 subtests passed, with Ruff passing; full CI for this head is running. These are software-control checks, not an NPU performance qualification.
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent incremental review: 96e3c50 → bb6ecbd. Base advances from 8aba657 to cf671a8. Previous own review: #488 (review) . COMMENT, not approval.
I compared actual old/new contents of all 30 current PR paths and separately checked the base-to-base changes. The inherited R3 protocol-child and Apps/storage/dashboard changes are not being counted as entirely new search implementation. VATS, WorkControl, GVO/roles, adapters other than Engine, planning/tree and the existing VATS/work-control regression files are byte-identical to the previous reviewed head.
The substantive integration changes are appropriate in this bounded static pass: Engine now invokes arguments/assess through task_protocol.run rather than the removed in-process protocol API; both protocol calls and benchmark execution derive remaining time from the same supplied deadline. The optional keyword-only monotonic deadline in task_protocol.run is finite-validated, rejects already expired work, and is combined with the existing protocol timeout; _supervise also checks expiry before starting. The new tests cover a short deadline and already expired no-launch case. EngineEvaluator is imported only for the engine command, preserving the lightweight status/export path. I found no additional concrete source defect in these changes; source assertions are not test execution.
Both existing P2 blockers remain. WorkControl and the relevant VATS state transitions are unchanged:
- A conclusively not-started expired launch leaves search.pending/verification Attempt unreconciled; recover seeks a receipt from an adapter never executed, and finish refuses pending state. The new protocol-child timeout tests do not exercise this separate claim→launch→Search-recovery composition.
- Known actual_ms overspend with cleanup_verified=False still leaves only the old reservation deducted and does not block further different-domain claims. The shared budget can therefore remain overstated despite known expenditure. Deadline propagation does not correct that accounting branch.
Before approval, resolve these paths and demonstrate the combined-state regressions plus applicable rebased integration checks. The earlier qualified-root evidence and frozen-resource improvements remain present; no resolved item is reopened. Proposed Issue #363 acceptance provenance and real Engine/Skill pilot qualifications remain separate from this code integration.
Current machine evidence: raw lint-and-smoke job 134843558515 identifies bb6ecbd, Ruff success and 8 passed in 0.06s for smoke/claim tests. At inspection full-check 3.10/3.11 and Windows were running, macOS queued; installed-package 3.10/3.11, references and declarations had success metadata, dashboard-browser was skipped. I did not establish completed new-head full runtime verification or relabel historical-head test results.
Optional coverage: test invalid deadline inputs, omitted-deadline compatibility, and an Engine arguments→execution→assess scenario showing that the original total deadline is not reset.
Performed: old/new content and base separation, actual Engine/protocol/CLI/test inspection, unchanged P2 paths, AGENTS and own-review/current-head checks, raw smoke evidence. Not performed: repository-code/test execution, real model/device/provider trials, edits, CI reruns or merge. No other reviewers' opinions were read or used.
Recover conclusively unstarted work without adapter execution, keep unknown launches blocked, and reserve known usage until cleanup is reconciled. Reject legacy searches without retained initial evidence and cover the reviewed planner, role, identity and deadline boundaries. Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
zengchuang-hw
left a comment
There was a problem hiding this comment.
增量独立检视:bb6ecbd5a6888f43345ba5376f71f2c6e55f0c25 → dc7292a。
上轮:#488 (review)
结论:本轮两个已报 P2 的源码修复成立,新增回归定义覆盖原失败路径;目前仍为 COMMENT,不把尚未完成的验证或 proposed RFC 验收计为通过。
已解决(源码层面):
- 过期未启动导致 Search pending 无法恢复:recover 现在只对持久 settled + not_started/deadline 的明确证据清理 pending/active_action;verify 对原绑定 Attempt 写 control-only INCONCLUSIVE,无 objective/PASS,不调用 adapter 或再次 dispatch。未知 launch intent 仍须真实 adapter receipt,不能走此例外。新增 draft/verify 两类恢复测试包含重开 RunStore、重复恢复、零额外收费、根 incumbent 和 domain 释放,并有未知 intent 的负向用例。
- 已知超额但 cleanup 未知时预算虚增:known_ms 保留已知用量下界,available 按 max(cap_ms, known_ms) 保留占用;超过可执行预算立即 blocked,未知 cleanup 继续保留 domain。最终结算不得低于已知用量,最终收费及重复回执保持幂等。新增 200 cap / 950 known usage 用例覆盖跨 domain 拒绝、重开库、后续未知观测不抹除下界、拒绝倒退、一次收费及最终 domain 释放。
其他增量:
- legacy initial_conditions 缺失改为明确拒绝 Search 恢复及 GVO examples,仍允许 WorkControl 原始读取;不从后来的树猜测历史初始证据。初始证据测试区分 receiving IDs 与真实 execution/evidence/epoch。
- 阅读共享 Engine 总 deadline、无效 protocol deadline、注入角色序列化及 checkpoint、completion prerequisite 的新增测试定义。本轮静态阅读未发现新增确定缺陷。
实际检查:
- 独立阅读本轮全部 11 个变更文件的差异,并复核 WorkControl/Search/GVO/record 编解码控制路径和仓库 AGENTS;未读他人检视意见。
- 已读取当前 head 的 lint-and-smoke 原始日志:Ruff 通过;tests/test_smoke.py tests/test_claim.py 为 8 passed in 0.07s。证据:https://github.com/InferMatrixLab/InferRSI/actions/runs/44319975680/job/134856020871
- 最近检查快照:full-check 3.10/3.11、Windows 仍进行中,macOS 排队;installed-package 3.10/3.11、references、declarations 为成功状态,dashboard-browser skipped。状态成功不等于我已检查其全部日志。
批准前必做 / 未执行:
- 取得当前 head 对上述新增 WorkControl/VATS 等回归的实际执行结果及必要完整检查结果;smoke 的 8 项不覆盖这些新增用例。本轮没有在本地执行仓库代码或测试。
- 保留上轮对 #363 proposed 范围、owner 验收与真实 Engine/Skill pilot 证据的区分;本轮新增单元/替身测试不能替代真实集成或性能验收。
非阻塞建议:可另补仅 operation deadline 过期、campaign deadline 尚未到达时的恢复后继续搜索用例,区别于本轮将 campaign 时钟整体推进到期的回归。
未合并、未修改分支、未启用自动合并。
|
Review follow-up at pushed head dc7292a (base cf671a8). The two P2 findings in review 5957370567, reiterated in 5957447009, 5957503359 and 5957834456, were valid. Regression tests reproduced both before the fix. This reply consolidates the repeated findings without treating them as separate defects.
The three inline findings were also valid for their reviewed revisions and had already been fixed; they were rechecked and answered in their original threads: finite gain, CLI import isolation, main's isolated protocol API. I found no substantive finding that needed rejection as incorrect. The subsequent review of dc7292a confirms the two source fixes and identifies no new concrete defect. Its additional operation-only-expiry/continued-search suggestion is reasonable non-blocking follow-up coverage; this revision's expiry tests deliberately exercise campaign-limit fallback, so they are not presented as execution evidence for that additional scenario. Validation on this revision: the local affected-area suite reports 216 passed, 54 subtests passed; Ruff and diff checks pass. Current-head full regression: Python 3.10 and Python 3.11 each report 5,288 passed, 37 skipped, 672 subtests passed, 1 warning. Both raw logs identify merge checkout The acceptance-scope cautions remain correct: repository tests and finite-edit CPU training do not establish real Engine/Skill VATS/GVO/final pilots, production resource isolation, complete provider accounting or performance gains. The PR description and support matrix retain those gaps, and the PR remains Draft. This follow-up records fixes and test evidence, not independent approval, whole-issue completion or a merge decision. |
zengchuang-hw
left a comment
There was a problem hiding this comment.
同 head CI 证据增量检视,精确 head:dc7292a3cd3d624794ed280a0d677d20e03fff29。源码范围 dc7292a → dc7292a(无新提交);上轮源码检视:#488 (review)
评论 6981045390 提供 Ready run 定位。本轮不采纳评论中的修复/验收判断,直接核验 GitHub 当前对象及原始 job 日志:
- full-check Python 3.11:Ruff 通过,pytest -rs 为 5288 passed、37 skipped、672 subtests passed、1 warning,498.38s。日志检出的执行对象是合成 merge e371b3b(当前 dc7292a 合入 cf671a8),不是另一个旧 head。https://github.com/InferMatrixLab/InferRSI/actions/runs/44319989516/job/134856070233
- Windows host compatibility:8 passed + 25 passed;macOS:8 passed,均同一合成 merge。它们是选择性 host 测试,不能宣称覆盖全部 VATS/WorkControl 场景。
- 最近状态:full-check 3.10 仍进行中,installed-package 双版本成功,dashboard-browser skipped。37 个 pytest skips 包含可选依赖/平台/opt-in 环境缺口,不能写成全能力验证。
已解决项仍沿上轮:两项 P2 的源码与回归定义已核验。本轮补到完整 3.11 suite 成功的服务端运行证据,不是本地重跑,也不以 host 8/25 项替代具体新回归。没有源码变化,不重复声称重新完成全部代码审查。
批准前仍需:当前 head 必要检查的完整结果,以及此前记录的范围/owner 验收与真实 Engine/Skill 试点证据边界保持明确。真实集成、硬件性能、生产资源隔离和完整 provider 成本没有在本轮得到验证;CI 不能替代这些证明。当前仍 COMMENT,不宣称总体验收通过。
实际执行:读取作者/head/本人既有 review、隔离评论请求和证据定位、读取上述原始 Actions 日志。未读取他人检视意见,未执行仓库代码,未修改分支或合并。非阻塞补测建议仍见上轮,无新增。
david6666666
left a comment
There was a problem hiding this comment.
One recovery case after a verified late cleanup.
| body["spent_ms"] += actual_ms | ||
| if body["spent_ms"] > body["cap_ms"]: | ||
| body["blocked"] = True | ||
| if transition is not None and not body["blocked"]: |
There was a problem hiding this comment.
After a known overrun blocks the campaign, a later verified cleanup settles this operation but skips the transition, so search.pending still points at settled work. On reopen, Search.recover() only handles the already-settled deadline case and can ask the adapter for a receipt again. Could recovery archive and clear this settled pointer without adopting its output or unblocking dispatch? Please cover overrun → verified cleanup → reopen and retry.
Retain protocol timeout receipts with supervisor cleanup facts so known usage settles and clean domains are released without relaxing overrun blocking. Resume committed ACCEPT decisions without sampling again and preserve consumed steps when restoring paid continuations. Reject development Skill evaluators before final claims. Cover process timeouts, uncertain cleanup, restart windows and final scope validation, and document the differences from the SpecRSI reference implementation. Agent-assisted-by: Codex (implementation, tests and reference analysis) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
zengchuang-hw
left a comment
There was a problem hiding this comment.
增量独立检视:dc7292a3cd3d624794ed280a0d677d20e03fff29 → b849894(1 commit / 8 files);base 仍为 cf671a8。上次本人发表:#488 (review) 。本轮 COMMENT,尚不批准。
独立阅读全部八文件增量及相关调用链后,本轮未发现新增确定性源码缺陷;以下是源码层面的确认,不是运行结果:
- Engine 把 TaskProtocolError(timeout) 转为保留的 INCONCLUSIVE 评估,并合取 task/protocol cleanup,arguments 尚未启动 task 时允许 execution digest 为 null;ArtifactAdapter.verify 随后保留实测 elapsed_ms 与绑定回执。非 timeout 的身份/协议异常仍不被这个分支放宽。新增测试既有真实睡眠子进程 timeout 控制,也有 arguments/assess × cleanup True/False/None 的替身用例,二者证据性质不同。
- task_protocol 仅由 supervisor 的 domain_empty,或明确未启动,给出 timeout cleanup_verified。未知清理不能因捕获异常而变成已释放;实际超额的预算阻塞仍保留。此前已修复的 WorkControl 已知用量下界与未启动恢复问题未被本轮变更重新打开。
- Search.run 在重采样前识别已持久化 ACCEPT;continue_from 的 forced ACCEPT 重启也走终止路径。decision_count 将被省略的自然前缀 step 加回 trace 长度,同时用于步数上限与随机流索引。已交叉检查 continuation 拷贝如何保留 draft_attempts、spent_before 与 forced trace,避免将只训练 suffix 理解成获得额外执行步数。新增测试覆盖重开 store、重复恢复及晚期 continuation 的两步上限。
- Skill validate_final 在 claim/access 前拒绝 development、缺失或不匹配 holdout scope;已有 final_result 仍只返回旧结果。新增合法路径的二次调用确认不再次访问 holdout。
批准前尚需当前提交的执行证据。 当前关联 Ready run 44352212642:full-check 3.10/3.11 的 Ruff step 成功、测试进行中;Windows 进行中、macOS 排队;installed-package 双版本与 pr-references 成功,dashboard-browser skipped。这是连接器 job/step 状态,不是已读完整测试日志。PR 正文目前仍列 dc7292a 的 head/5288-pass 旧验证,不能作为本轮新增测试通过的依据;请在本轮检查结束后更新准确 head 与结果。
范围边界: #363 的 RFC specrsi-r2-2026-11-18 仍为 proposed,不能把本 PR 的局部软件控制当作 AC-01–AC-10 全部完成。新增 reference-comparison 文档明确有限编辑词表、线性特征、默认无 refinement 和缺少真实试点的限制;本轮没有独立核验外部论文/参考仓库的逐项一致性,因此不为“复现论文”背书。真实 Engine/Skill pilots、硬件资格、生产资源隔离、全生命周期/provider 成本仍未在本轮验证。
非阻塞补测建议:在 Engine timeout 回执之外,增加实际 task cleanup=False 与 protocol cleanup=True 的合取用例,确保不释放未知 task domain;此建议不将现有源码合取判断描述为已知缺陷。
实际执行:连接器读取作者/head、本人历史 review、八文件完整增量、AGENTS、关联 RFC 需求与 adapter/continuation/final/protocol 调用路径、当前 CI 元数据;未读取他人检视意见。未运行仓库代码/测试、真实模型或设备/provider 实验,未修改分支、合并或启用自动合并。
zengchuang-hw
left a comment
There was a problem hiding this comment.
当前 head b84989494dfa69628aa918ed85653afc936a5493 的 CI 证据增量检视。源码范围 b849894 → b849894(无新提交);上次独立源码检视:#488 (review) 。本轮仍为 COMMENT,尚不批准。
上轮等待的常规 CI 执行证据已补齐。 我读取了 Ready run 44352212642 的原始日志,而非仅依据绿色状态:
- Python 3.10 与 Python 3.11 均 Ruff 通过,5344 passed、37 skipped、672 subtests passed、1 warning,分别 530.86s / 575.75s。
- Windows 为 8 项兼容性与 25 项 primitives 测试通过;macOS 为 8 项兼容性测试通过。
- installed-package 双版本原始日志确认构建 wheel、在独立 venv / checkout 外以 isolated Python 检查,均列出 128 records、930 个资源文件及相同资源摘要。references、draft smoke、declarations 当前成功;dashboard-browser skipped,不能计作通过。
执行对象更正: 上述原始日志实际 checkout 是合成 merge e6f9d7b8d…,将当前 b849894 合入 965ab5cf83399a99530949b04bb934a892ee417c。PR API 的 base 字段仍返回 cf671a8,不能因此把此轮测试写成运行于该旧 base;PR 正文也仍保留 dc7292a / 5288 的旧说明。应按实际 H/B/T 更新验证段落。
本轮重新核对 AGENTS、协作契约及当前 speculative-search 支持/验收矩阵:上轮源码发现的修复仍适用,没有新源码缺陷结论。剩余批准前事项不是再次跑同一常规 suite: #363 当前仍为 proposed,并要求维护者冻结/接受交付契约;需要明确该实验性局部交付的接受范围及其未完成项处置,不能由全绿 CI 代替。完整 AC-01–AC-10 所需真实 Engine/Skill VATS/GVO/final pilots、资源/成本资格与对照实验没有在本轮得到新证据。若维护者明确接受有边界的软件切片,应据该接受范围判断 PR readiness,而不把整个 RFC 的完成当作本次源码检视已证实。
已执行:精确 head、本人历史发表、六个完整原始 job 日志、当前 checks/status、当前仓库契约及 #363 正文核对。未执行:本地测试/构建、仓库代码、模型或设备实验、原论文复现实验;未读取他人评审意见,未修改分支或合并。原非阻塞补测建议仍见上轮,不重复升级为缺陷。
zengchuang-hw
left a comment
There was a problem hiding this comment.
NO PASS
同 head 证据增量:b84989494dfa69628aa918ed85653afc936a5493 → b84989494dfa69628aa918ed85653afc936a5493,承接 上轮检视。新增单卡 910B1 诊断把证据推进到了 Engine 搜索与独立 final。作者报告搜索命中 1.35× 为 3/3,但独立 final 达标仅 1/3;110.08 秒是首次观察到搜索命中的平均耗时,不是可靠确认达标的耗时。速度比相对原始 denoiser,不是 PR 对 base 的收益。
批准前必做:明确由维护者接受的 #363 交付切片。若以这轮设备试验满足验收,请给出外置 TargetSearch 的源码 pin 和原始逐候选回执;当前 vats.py:635–693 按顺序执行 action,内置停止条件是预算或截止时间,不含试验的逐候选达标早停。若按完整 #363 验收,仍缺真实 Engine GVO、Agent/Skill 完整试点及 AC-10 的配对和消融对照。没有发现新增源码缺陷,也不把常规 CI 当成门槛。
本轮读了新增证据、实际控制路径和 #363 交付要求,未运行代码或 NPU,未复算尚未公开的原始数据。
Problem and resulting behavior
The serial lifecycle cannot edit incomplete/failed candidates, select bounded verification bundles, or train search roles from independent search returns. This PR adds a synchronous VATS/GVO runtime: retain real edits, verify selected candidates, atomically settle evidence and costs, update separate CPU Drafter/Policy/Scorer parameters, and search again with the new frozen checkpoint. A repaired child can win independently of its parent's quality outcome, with its own complete evidence.
Refs #363
This is an experimental local implementation toward the full proposal. The support and remaining-acceptance matrix identifies its boundaries. The full RFC is not claimed complete; the PR remains draft.
Implementation
inferrsi search engine/status/exportand Python composition. Benchmark is loaded only by the explicit Engine entry; ordinary project CLI/session behavior keeps its existing dependency boundary.Validation
The software regression results below were recorded at earlier revision
dc7292a3cd3d624794ed280a0d677d20e03fff29, integrating maincf671a880ac8037e01922e1be0b701771fb9922f. The current PR head isb84989494dfa69628aa918ed85653afc936a5493; this description update does not claim a fresh full regression run at that head.ruff check src tests toolsand staged diff checks passed.e371b3b7fd663b8e84bb3f8ce0ebe982a38dc857, merging that revision into the stated base, and show Ruff passing. Installed-package checks for both Python versions, Windows/macOS compatibility, lint/smoke and reference checks also succeeded; dashboard-browser is skipped for the draft.qwen21-denoise-npubenchmark was previously run once in CPUsmoke/check-onlymode with a mathematically equivalent actual edit. Baseline and candidate correctness and weight-update checks passed;speedupis null. This is correctness evidence, not NPU performance or full-task qualification. Subsequent NPU diagnostic measurements are reported separately below.The process-backed numeric grader and Skill report fixtures are software controls, not production pilot evidence. No live provider/model API experiment or production pilot is claimed; the NPU diagnostic runs below have a narrower scope. Earlier Harness-to-Model imports, eager Benchmark loading and the changed main-branch protocol API were fixed before this revision; their original failing runs are not counted as passing validation.
NPU diagnostic results (2026-11-19)
Scope under #363: one earlier serial reference and three additional VATS searches, with up to three candidates per VERIFY group. These are descriptive measurements; general search efficacy remains INCONCLUSIVE. The earlier single VATS pilot is excluded from the three-run statistics.
qwen21-denoise-npu, a scaled Qwen-Image 2.1 denoiser with random BF16 weights and full prefill plus eight denoising steps in the measured scenarios. All five cases run: compact/portrait/landscape contribute timing; padded-batch/smoke provide additional validation. This is not pretrained image-quality or production service evidence.2.10.0+cpuandtorch_npu 2.10.0.post7;pairs=2,samples=2,warmups=1. Speedup is the geometric mean of the three per-case baseline-median/candidate-median ratios, relative to the original denoiser, not a PR-versus-base speedup.cf671a880ac8037e01922e1be0b701771fb9922f; VATS uses PR headb84989494dfa69628aa918ed85653afc936a5493plus an experiment-localTargetSearchwrapper. The per-candidate target-stop wrapper is outside the PR source, so these results are not a measurement of the unmodified PR head alone.07_batched_conditioning08_gate_up_fusion07_batched_conditioning07_batched_conditioningThe serial first-hit time was not separately instrumented. VATS first-hit timing starts at controller initialization and includes root qualification and search; total timing additionally includes independent final and reporting. Candidate preparation, device queueing, container startup and environment preflight are excluded.
Search reached the target in 3/3 VATS repeats; independent final reproduced it in only 1/3. Mean first observed search-hit time was 110.08 s (median 99.83 s; range 63.62–166.79 s). Mean total time was 143.35 s. First observed search-hit time must not be described as time to a reliably confirmed target. All three outcomes, including both final target misses, are retained.
Actual VATS verification order and early stopping:
run-01:10_kv_buffers → 04_shared_modulation → 05_decode_broadcast → 08_gate_up_fusion; two remaining jobs in the active group skipped.run-02:04_shared_modulation → 07_batched_conditioning; one remaining job skipped.run-03:07_batched_conditioning; two remaining jobs skipped.The serial reference completed nine full evaluations / 486 compute calls; the three VATS repeats completed thirteen / 702. All passed the task's numerical correctness checks, including per-step and weight-update checks. Frozen-file integrity checks passed; skipped jobs created no Attempt/Receipt. Numerical correctness does not imply meeting the 1.35× performance target.
Comparison limits: the serial reference was measured earlier, without a contemporaneous paired serial control for these repeats. The three VATS runs used fresh containers/state on the same exclusively allocated NPU, while the CPU host was shared. UUID-dependent tie ordering can vary the verification order despite a fixed seed. The lower observed wall times accompany fewer candidate validations; this small, fixed-catalog experiment does not establish a stable causal gain from the PR, parallel execution, trained search roles, or reproduction of the paper's results.
Source summaries retained outside the repository:
pr488-qwen-cap10-ab-20261119/results/comparison.jsonandpr488-qwen-bundle3-repeat-20261119/summary.json. Raw experiment bundles are not publicly attached to this PR; the table is a summary of those retained results, not a publicly replayable acceptance package.Remaining full-issue requirements
The supported profile is a trusted local host, one worker and small menus. Domain exclusion covers cooperating controllers sharing the same RunStore; a name does not allocate a device or provide OS confinement. Runtime adapters package setup into whole checks; arbitrary standalone shared-setup jobs are represented only by the offline planner.
Production resource qualification, hard preemption of in-process callbacks, preflight/setup lifecycle accounting, campaign-wide provider token/currency reservations and multi-worker comparisons remain outstanding. Existing per-comparison provider bounds do not establish a campaign-wide token/currency budget. The supplied operator model provides neither an LLM training adapter nor supervised warmup data.
The NPU runs above exercise Engine search and independent final with a fixed candidate catalog. Full Engine GVO and live Agent/Skill pilots, and the complete AC-10 matched serial/VATS, bundle, learning, Reset/Carry and worker comparisons remain outstanding. A controlled live Agent/provider configuration and budget have not been supplied. Promoting a learned checkpoint as a default improver also requires separate independent qualification.
No parent-issue closure, production acceptance, efficacy gain or merge approval is claimed.