Skip to content

Gate RSI search on measured bottleneck diagnosis - #519

Open
Bounty-hunter wants to merge 3 commits into
InferMatrixLab:mainfrom
Bounty-hunter:codex/rsi-diagnosis-first
Open

Bounty-hunter wants to merge 3 commits into
InferMatrixLab:mainfrom
Bounty-hunter:codex/rsi-diagnosis-first

Conversation

@Bounty-hunter

@Bounty-hunter Bounty-hunter commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Refs #337

问题

#501 的 on/off 对照已经冻结任务、评测协议和预算。通用 RSI loop 目前允许直接查知识或提交候选,瓶颈分析和 profiling 的顺序由 Agent 自行决定。这个 PR 把先诊断再搜索的规则接入共享 Harness。

改动

  • campaign diagnose 和 benchmarks session diagnose 复用同一控制器,记录当前瓶颈与 workload 对应的 UTF-8 测量摘要。摘要可来自当前 profiling 或已有的同条件测量。
  • Harness 默认给所有新会话启用诊断。受管 knowledge 和候选 round 都要求先登记诊断,并记录关联的诊断 ID。service-reuse 建立新的参考条件后,需要登记当前 epoch 的诊断。
  • benchmarks compare 的 on/off 两组、独立 benchmark 和用户直接发起的普通项目优化都走同一规则。on 组在有具体实现疑问时按需检索,off 组完成诊断后直接提交候选。回收时核对测量摘要、原始轮次引用和诊断时间。
  • start 沿用现有参数。既有会话仍按各自的冻结计划执行。

流程

  1. 对冻结基线进行 profiling,或读取同条件的既有测量。
  2. 用 diagnose 保存瓶颈及对应观测。
  3. 有实现疑问时按知识开关检索,再提交候选进行评测。
  4. 根据反馈更新诊断或继续使用仍相关的记录,最后独立复测。

Harness 核查诊断记录和顺序。测量口径及瓶颈推断由执行者根据当前任务确认。

验证

  • Ruff 和 diff 检查通过。
  • 本地核心回归 129 项通过、1 项跳过、43 项子检查通过。覆盖 benchmark on/off、普通项目、service-reuse 诊断入口和证据回收。

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NO PASS

首次独立检视:9c7b7b05042f54ca71d13b2897a7505d818d3429 → f473060feb5ab51ea4a1d3ae3cab75840e9449b4,范围 #337 的诊断顺序和对照回收。

批准前必做:报告需确认引用的诊断仍含测量凭据。 optimization._load 会核对保留的评估/交付证据,但不检查新增的诊断记录;benchmark_compare 回收时只检查 diagnosis_id 是否存在。若归档中诊断 ID 保留、测量摘要缺失或为空,报告仍不会在此处增加缺证据项,可能把无法审计的对照算入有效结果。这与本 PR 要求保留测量诊断不一致。建议在回收时校验引用诊断的必填字段与摘要;增加摘要缺失/空白时报告降为不可纳入的回归。不是要求 Harness 判断测量本身是否科学有效。

启动、诊断登记、knowledge/round 门槛及 service-reuse epoch 分流已静态核对。未运行代码或测试,未查 CI。

@Dong1017 Dong1017 added area:harness Harness module: profile, A/B loop, quality gates, evidence reports kind:task Bounded delivery task with a delegated slice labels Nov 19, 2026

@david6666666 david6666666 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comparison collector should verify the recorded activity-to-diagnosis binding, not only the existence of a diagnosis.

Comment thread src/inferrsi/apps/benchmark_compare.py Outdated
if plan.get("diagnosis_required"):
diagnoses = {record["id"] for record in control.get("diagnoses", [])}
if any(record.get("diagnosis_id") not in diagnoses
for record in (*control["rounds"], *control["contexts"])):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also accepts a valid diagnosis from the wrong point in the run. If a collected round's state.json reference is changed to a second, later diagnosis, the original observation.json hash still verifies and the membership check passes, even though the retained observation names the first diagnosis. Please check the round's reference against its retained observation (and the activity order), with a regression that swaps between two complete diagnoses.

@Bounty-hunter
Bounty-hunter marked this pull request as ready for review November 19, 2026 10:19
zengchuang-hw
zengchuang-hw previously approved these changes Nov 19, 2026

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PASS

独立增量检视:f473060feb5ab51ea4a1d3ae3cab75840e9449b4 → 680827c3fae2c5c780542179e6a53e7e0b7f54a5,上轮意见。

上轮缺项已修复:回收读取会检查诊断 ID 唯一、瓶颈与测量摘要非空、诊断时间有效,与对应轮次的绑定及 epoch 一致。不再只检验 ID 存在,轮次的诊断引用与已固定的 observation 对照;service-reuse 恢复也复用该校验。回归覆盖摘要缺失/空白、重新绑定较晚诊断和记录时间失真时不计入有效对照。既有会话仍按冻结计划读取。未发现新增阻塞。

静态核对了代码和回归;这是对诊断记录完整性及顺序约束的检视,不验证摘要中的测量结论。未运行代码、测试或查 CI。

@david6666666 david6666666 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round binding is now checked against retained evidence. Knowledge capture still has a separate gap.

Comment thread src/inferrsi/harness/optimization.py Outdated
if not isinstance(reference, str) or reference not in diagnoses:
raise ValueError("Campaign activity lacks a retained bottleneck diagnosis")
diagnosis = diagnoses[reference]
if round_record and (

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both knowledge-capture read paths call this with round_record=False, and the diagnosis ID is stored only in the mutable context reference. As a result, rebinding a retained knowledge capture to a later or different-epoch diagnosis still passes as long as that ID exists. Please bind the diagnosis ID and the capture epoch/order to retained capture evidence and verify it on collection, with a regression for a knowledge capture rebound across epochs.

@david6666666

Copy link
Copy Markdown
Collaborator

#516 has been merged into main. Please rebase this branch onto the latest main and adjust the changes as needed. Please verify that the diagnosis gate still allows a managed knowledge lookup once the required diagnosis has been recorded.

Require measured bottleneck evidence before managed lookup and candidate search. Verify each capture against the diagnosis and reference conditions in effect at lookup, and keep independent final validation separate.

Agent-assisted-by: Codex (implementation, review fixes and regression tests)
Signed-off-by: dengyunyang <584797741@qq.com>
Agent-assisted-by: Codex (regression test)
Signed-off-by: dengyunyang <584797741@qq.com>
Report invalid baseline and parent evidence before asking for a new diagnosis. Candidate execution still requires a diagnosis for the current conditions.

Agent-assisted-by: Codex (implementation)
Signed-off-by: dengyunyang <584797741@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:harness Harness module: profile, A/B loop, quality gates, evidence reports kind:task Bounded delivery task with a delegated slice

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants