Repository navigation
fix(benchmark): preserve scoped run identities through study readback - #5758
Conversation
Signed-off-by: Jim-jimu <49069997+bmh201708@users.noreply.github.com>
…ntity Signed-off-by: Jim-jimu <49069997+bmh201708@users.noreply.github.com>
…ntity Signed-off-by: Jim-jimu <49069997+bmh201708@users.noreply.github.com>
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent; model=gpt-6.1-sol; provider=OpenAI; declaration_source=runtime_reported; reasoning_effort=xhigh
APPROVE。独立审查 exact head a8331167c39a2bcec06b21703d1984ba71fb495c,不可变基线 acc3ede6af7e3033ef01a93a4d0d1dbcbadf902c。以下为当前精确版本的正式批准审查。
动机
在研究看板中比较两个 case 或打开具体运行的人,需要看到属于该 case 的对照、来源和分析内容。
原先两个 case 使用同名运行时,后台会把其中一组对照关联到另一 case,来源或分析也可能串线;点击第二个 case 还会打开第一个并同时高亮两行。现在后台按完整身份关联,界面链接保留 case;歧义或过期链接提示重新选择,合法选择和刷新能读回正确记录。
真实 CLI 和临时 JSONL 存储保留两组正确对照、各自来源及分析;打包界面点击、刷新、拒绝歧义和键盘恢复均显示正确 case,移动端无水平溢出。
本次只关闭同名运行的身份关联缺陷,不关闭完整研究资格任务,不修改评分或 ratio 方向,也不认证实验原始事实、隐私或真实模型采用效果。
改动思路
既有 ledger 已用 benchmark、study、case、run 四个字段确定一个运行。缺陷发生在后来做对照、来源、分析和界面选择时:部分消费者丢掉了前三个字段,只按 run_id 查找。修复重用这份持久身份,没有新增需要手工同步的“已绑定”状态。
后台先沿用既有 normalization、过滤与排序,再按完整身份索引。anchor 只替换 run_id,继承当前记录的 benchmark/study/case;来源和 insight 也使用相同键。真正重复的完整身份明确拒绝,别的 case 中同名运行仍合法,不能借来填补当前 case 缺失的 anchor。
现有 dashboard packet 已限定一个 benchmark/study,因此界面只需保留 caseId 和 runId。Cases 点击直接把已知 case 写入 URL,无需新表单;统一的 TS selector 只接受唯一匹配,刷新后仍能选中同一个对象。唯一的旧 runId 链接继续工作,无查询参数时保持原先首行默认;歧义或过期的明确链接不再退回不相关运行,原表格就是恢复入口。
我检查了既有 row-key、upload envelope/replay/supersession、所有被改的 join 和 router/页面调用者。专门 benchmark provider 保留 Python,Zod 类型化展示留在既有 TS owner;不是另加通用 Python 决策。邻接重构已体现在共用四字段 helper、统一 selector 和单个 selected-object 高亮,旧 bare-id 规则被删除。
最强风险是“收紧身份”误拒绝合法同名记录,或只修一个 join 却让另一消费者继续串线。基线失败、完整真实读回与反方向的拒绝/恢复证明这个修复覆盖当前有用路径。
具体改动
全量十四路径 +313/-36,包括三个公开 synthetic PNG。生产改动集中在已有 experiment_identity、experiment_board、study_projection 以及现有 dashboard data/router/page;其余为现有 Python/TS/browser 回归、README 和展示证据。没有新 schema、provider、CLI 安装入口、评分或 metric 定义变更。
关键代码讲解
- experiment_run_key:复用已归一化记录的四字段身份;对 anchor 只替换 run_id,其它 scope 继承当前运行。
- build_benchmark_experiment_board:按完整键建索引并拒绝重复完整身份;同名但不同 scope 保持合法,对照只能引用自己的 scope。
- _validate_case_insight_attachment:现有 append 前校验只接受精确终态运行,不能用另一个 case 的同名记录填补。
- build_benchmark_study_dashboard:来源和 insight 读回使用同一完整键,修订一个 case 不改变其它 case。
- selectBenchmarkStudyRun:已有类型化 packet 内按 case/run 唯一匹配;歧义或过期明确链接返回 undefined,无查询的首行默认保留。
规范:docs/architecture/rfcs/benchmark-study-upload-dashboard-v0.md,spec_revision acc3ede6af7e3033ef01a93a4d0d1dbcbadf902c。原规范和相关 owner 在固定基线已核对;没有修改规范迎合实现。
3.2:implemented;envelope 与 payload 身份对齐,真实 append/readback/replay 继续工作。3.3:implemented;脱敏 insight 属于精确终态运行,错 scope 不写入。4:implemented;只读 case/run 展示保留实际对象及来源,选择后刷新一致。7:implemented;非法身份及重复完整身份拒绝,合法修订和幂等重放保留。
这是这些条款下同名身份缺陷的关闭,不是对整份 RFC、#5203 或旧 #3335 全部问题的验收。
对主干的风险
独立复核 current source 的 281 项 benchmark 测试全部通过,包含三个 scope 各 24 种输入排列。将同一组八个新增回归仅改为指向不可变基线的生产代码,八项均因预期的身份碰撞失败;当前版本全部通过。独立实际 upload-local / study-dashboard 经临时 JSONL 存储:两组 case 内同名运行保留正确对照,非法 attachment 返回 exit 2 且存储内容摘要不变,纠正后合法 append;case-1 的明确修订成为 9/10,case-2 仍为自身的 8/10,来源及 insight 各自正确。七条传输记录反向重放完整输出一致,重复提交没有 append。无碰撞唯一 ID 的相同输入在基线和当前版本得到完全相等的完整 packet,无归一化掩盖差异。
TS contract smoke、dashboard typecheck/build 与 Chat bundle build/verify、Ruff、diff check、advisory 后的 canonical semantic-vocabulary smoke 均通过。初次 build 的 LinkifyIt 导出失败来自已安装 linkify-it 5.0.2 与 lock 中 6.1.0 不一致;按原 lock 做 npm ci 后恢复,未改源码、断言或 lock。PR 描述中更早的 broad pytest 和 SQLite aggregate 失败保留;我没有认证其历史根因或声称完整 suite 全绿,当前改动边界由上述独立测试和真实入口覆盖。未查询、轮询或等待 CI。
打包界面使用本次真实 CLI/JSONL 生成的 packet,不用浏览器拦截构造“正确结果”:第二 case 的 Cases 点击打开正确 8/10,单行高亮;刷新保持 URL caseId 与详情;同名旧链接、过期 caseId 均无 run detail 或误高亮,现有表格 Enter 可恢复正确选择。在 390×844 且 reduced-motion 的视口,正确选择仍成立,scrollWidth 与 viewport 都为 390。完整首屏保留已有 read-only/freshness/分母/目标层次,没有增添表单或确认;本次是选择正确性修复,未改 hero/nav 布局。已检查三张公开 synthetic 图片的前后与拒绝状态。
这些证据是当前 source build 和本地真实 provider,不是 installed desktop 或远端 provider;原 PR 浏览器 smoke 的源代码已读,本次用独立 Ego 交互覆盖相关风险,未称它执行过。无外部上传、付费 benchmark、live Goal 改写或原始实验认证。
语义与 CI 对齐
Correctallrunjoins; rejectduplicatefullidentity; retainuniquelegacy/noquerydefault; ambiguous/staleURLs showselectionprompt insteadwrongfirstdetail. No scoring/ratio/defaultcapability activationchange. Exact4tuple backend; exactcase/runUI matches.length===1; noqueryfirstdefault;复用现有 typed owner,未添加 substring denylist 或仅靠 prose 的状态分类。ToolkitREADME scopedRunidentitysection,PRdescription/currentreview,Python/TS/browserregressions 已公开新旧默认/拒绝行为。校验是机器执行的条件,研究认证是另一个被明确保留的验收;不能把前者叫作可忽略 guidance,也不能让只读界面或配置可发现性产生启动、账户、发布、lease 或 quota 权限。canonical smoke 的扫描限制保留,不把绿色或零候选当作整个系统语义等价证明。
我的整体评价
APPROVE。这个同名运行身份修复任务 goal_achieved;long_horizon=improved,重复读回与明确修订保留每个 case 的承诺和来源;user_experience=improved,同样最短点击路径能得到正确对象,拒绝后用原表格恢复。现有完整身份被复用,邻接的 ratio 方向与完整研究资格未关闭。只需要当前 exact-head 的维护者合并,没有扩大账户或发布权限。
兼容保留依据:Persisted4identity unchanged; independent legacybookmarks validonlyunique. Noquery firstrowdefault retained. Ambiguous/stalepreviouswrongfallback deliberatelyremoved/documented; persisted records andlegalreplay/supersession retained. Applied bounded future-facing pass: consolidate tuple semantics inexisting experiment_identity helper, retire bare-idjoins; one TS selector andselectedobject highlight. No newprovider/catalog/requestversion/framework; broadermetric/ratio issue explicitlydeferredoutsidecurrentdefect. 范围与已复现问题相称,现有 caller 和公开入口完成 useful slice;没有为了多写代码或多几个测试而加入框架。剩余证据边界:Broader #5203 integrated-study qualification and historical #3335 ratio-direction finding remain unchanged; no paid benchmark, remote upload, installed desktop runtime or live user-state qualification. 所有批准结论只绑定上面的 head;改变版本后需要重新判断。
English verdict: APPROVE — exact head a833116. Canonical benchmark/study/case/run identity is preserved across anchors, provenance, insights and packaged selection.281tests passed; all8new regressions fail on immutable baseline. Real CLI/JSONL rejection, replay/supersession, complete unique-id parity and packaged desktop/mobile reload/recovery passed. Historical ratio/full-study qualification remain outside scope. Local lint, semantic and build checks where applicable passed; CI was not consulted. Merge authority remains with the maintainer for this PR.
Goal And Delivered Outcome
(benchmark_id, study_id, case_id, run_id), but comparison anchors, upload provenance, insights and dashboard selection used bare run ids. Reusing a run id across cases could exclude a valid pair, misattribute provenance, reject a valid insight or open the wrong run.main. This fixes the identity defect, not the entire integrated-study qualification task.Implemented against
benchmark_experiment_board_row_keyand the accepted study upload/dashboard contract, inspected at42e55a809eb94f13443d303d76118735d4112182.experiment_identity.experiment_run_key; board anchor lookuptest_board_resolves_reused_run_ids_within_the_persisted_scope; missing-anchor and duplicate-identity regressionsstudy_projection._validate_case_insight_attachmenttest_case_local_run_ids_round_trip_with_insights_and_supersessionScope And Continuation
caseId; unique legacyrunIdbookmarks still work. Ambiguous or stale links show a selection prompt instead of silently displaying another run. Initial Runs view without a selection retains its first-run default.Validation
a8331167c39a2bcec06b21703d1984ba71fb495c, including upstreamacc3ede6af7e3033ef01a93a4d0d1dbcbadf902c. Earlier broad-suite results are explicitly marked below.acc3ede6before publication. The contribution diff is byte-for-byte identical to revision896fb299; relevant tests, types, lint, build, browser interaction and premerge checks were rerun after synchronization.finishedsynthetic,public_fixtureregression_paritypassed42e55a8and pass with the fix. Anchor coverage includes all 24 input permutations for each of benchmark, study and case scope. The browser regression also fails on the original page by opening the wrong case.unitpassedpython -m pytest -q tests/capabilities/test_benchmark_*.py: 281 passed on the tested revision. Existing protocol, fidelity, diagnostic and factorial behavior remains covered.real_entrypointpassedbenchmark upload-localandstudy-dashboardagainst disposable JSONL files, including repeated case-local ids, insight ordering, supersession, replay and provenance readback. The provider is the existing local simulation, not a remote service.integrationpassednpm --prefix apps/presentation/dashboard run smoke:benchmark-studyandnode examples/dashboard-benchmark-study-browser-smoke.mjs; browser smoke also passed against the built dashboard throughLOOPX_BENCHMARK_STUDY_URL. Covers case-to-run navigation, reload, direct row selection, ambiguous/stale links and unique legacy links. Collision packets use the public fixture with rewritten ids through browser request interception.staticpassednpm run build(including Chat bundle build/verification), control-plane typecheck, repository Ruff checks, configured mypy checks, semantic advisory andgit diff --check.integrationpassedloopx canary premerge --from-git-diff: 19 executed checks, 0 failures, 0 warnings, plus 5 direct checks. The expectedbenchmark_sensitivemaintainer-review hold remains. Isolatedloopx checkreports a clean public-boundary scan.integrationpassed42e55a8; output-budget code is unchanged by this fix.unitfailedbenchmark/tests/test_resume_sessions.py::test_timeout_keeps_the_observed_session_for_next_heartbeatfails identically on the unmodified baseline and the repair working tree. The rest of the full suite was not completed.unitfailedFrontend / Visual Evidence
changedpublic_fixture, with synthetic case-local run ids.Type of Change
LoopX Area
Technical Direction
Shared-authority RFC fixture impact
Boundary Checklist
Signed-off-bytrailer (git commit -s).