Repository navigation
Conversation
|
我对照当前 main 先说已经对上我在 #454 里提的几条: 还需要处理的点:
我没有重新跑 我不会在这个 head 上 Approve,直到 #454 被接受,并且上面的 rebase / 文档与测试缺口处理完毕。 |
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
Reviewed base cd8fcb452bd2c7969a7a6e2544aa1cb5f8ad5332 and head 1b2b4593cb72ef7b956f82f000313a9910cffbd7 (draft).
Acting reviewer: cursor[bot] via standing InferRSI steward delegation. Binding formal review to current head.
Blockers before Approve / ready-for-review:
- Governing issue not accepted. #454 is still Proposed with goal-acceptance owner TBD. Steward decision will be posted on #454; do not treat this draft as authorized delivery until that record exists.
- Rebase onto current main. Head diverges: ahead 5 / behind 1 vs
cd8fcb45(#453), which also touchedsrc/inferrsi/arguments.py. Rebase and re-run the focused harness tests on the new head; refresh declarationhead_shaif needed. - Dual-switch priority and point/interval disagreement are under-specified in docs/tests. In
assess()(project.py),--decisionruns first;--require-explanationonly rewrites aPASS. So with both switches on,within_noiseFAIL stays FAIL even if explanation isunverifiable. Also, paired mode keys only ongate_relation(above_gate→ PASS), so a point estimate below the gate can still PASS when the interval clears the gate. Document both behaviors in #454 anddocs/optimization-workflow-unification.md, and add regression tests (both switches on;agrees_with_point_gate is Falsewithabove_gate).
Checked / helpful but not enough to Approve: mechanical explanation.status definitions look consistent with the RFC; defaults-off report-only path is covered; mechanism binding / tamper / recover handling is present; tools/measurement_simulation.py exists with stated seeds. I did not re-run the 200×10000 simulation or GPU/NPU hardware checks. Hosted CI on this draft is limited (declarations green); full ready-lane evidence is still missing.
Hardware A/A + known-gain confirmation is not a merge prerequisite while both switches default off (steward decision on #454). Re-request review after acceptance + rebase + the doc/test gaps above.
|
回看了 Dong1017 在 #454 的 steward 决定:三个 owner 决策已经给出(可选能力、默认关闭、按一个 PR 评审),goal-acceptance owner 记为 Dong1017,硬件 A/A 也不再作为合入前提。这一条外部阻塞已经解除。 当前 head 仍是
因此我不会在这个 head 上 Approve。cursor[bot] 在同一 head 上也给出了 CHANGES_REQUESTED / DO NOT PASS,结论一致。请在处理完上述两点后(通常会形成新的 head)再提请复核。 |
|
Notes on The module docstring already says the interval's coverage is not calibrated. Near the 5-pair floor it is off by enough to change what a paired PASS means. At 5, 7 and 13 pairs, Three smaller points:
Both switches are off by default, so this is a good time to fix these. |
…ithin noise The percentile bootstrap lower bound for a median of n pairs is an order statistic whose one-sided error exceeds 2.5% at small n (3.1% at 5 pairs, 6.25% at 7). Use the exact distribution-free order-statistic interval (the inverted sign test) instead: the rank keeps each tail at or below 2.5%, so the minimum is six pairs and A/A at the gate passes at most 2.5% of the time. Declared expectations are judged as a family: region and counter intervals use the Bonferroni confidence 1 - 5%/k, and too few pairs for it is unverifiable. A declared-unchanged item is reported as not contradicted, never confirmed. A paired interval that straddles the gate shows only that passing was not demonstrated, which is not evidence that the gate was missed, so the round is INCONCLUSIVE (within_noise) and still does not block the search. Records say report_only: false when the interval decides the verdict. Reported by linyueqian on InferMatrixLab#457. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
1b2b459 to
ac0510b
Compare
|
已在新 head amy-why-3459 / cursor[bot] / steward 条件
linyueqian(谢谢,复算全部成立)
另外 Codex CLI(gpt-6.1-sol,xhigh)对新 head 做了一轮只读对抗评审,结论和处理见 PR 正文。请在新 head 上复核。 |
amy-why-3459
left a comment
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 016c7c7df023944f0f46d8e777d6d31f1df1764c and head ac0510b21b887e34b7357b13dc3d2eb7c9e86c5f(draft)。
上次在 1b2b4593cb72ef7b956f82f000313a9910cffbd7 上的阻塞已处理:
- #454 已附条件接受(steward 评论 6897439035)。
- 已 rebase 到当前 main
016c7c7d(ahead 8 / behind 0)。 - 两开关优先级与「点估计未过、区间过门槛」已写入
docs/optimization-workflow-unification.md和 #454,并有test_both_switches_apply_the_decision_before_the_explanation、test_paired_decision_passes_when_only_the_point_estimate_misses_the_gate覆盖。
本轮还核对了 linyueqian 在旧 head 上的意见及后续修订:精确次序统计量区间、MIN_PAIRS=6、Bonferroni 家族校正、跨门槛记 INCONCLUSIVE(within_noise)、--decision paired 时 report_only: false,以及 profile 报告失败不改变默认判定。我用 n=1…40 的二项尾计算复核了 interval_rank,与文档中的对数要求(k=1 为 6,k=2–3 为 7,k=4–6 为 8,k=7–12 为 9)一致。declarations 已通过。我没有重跑 tools/measurement_simulation.py 的仿真,也没有跑 hosted 全量 pytest。
david6666666
left a comment
There was a problem hiding this comment.
One issue with how normalized evidence affects the explanation gate.
| [v / k for v, k in zip(c, clock[1])], | ||
| confidence=confidence) | ||
| entry.update(normalized=normalized, observed=normalized["observed"], | ||
| basis="normalized_to_reference_regions") |
There was a problem hiding this comment.
Could a sign reversal between raw and normalized evidence stay inconclusive unless the mechanism explicitly predicts a relative effect? This overwrites the raw direction. In a 10-pair check, the raw region interval shows a 20% increase, the normalized interval shows a decrease, and the reference interval merely crosses 1. The result is explained for an absolute decrease. Please preserve that disagreement in the explanation gate and add a regression test.
There was a problem hiding this comment.
Thanks, agreed and fixed in 6a3c1236. When a region's raw and reference-normalized intervals point in opposite definite directions, the evidence now records observed: "raw_normalized_conflict" and keeps both raw and normalized. That check's outcome is unverifiable, so explanation.status is unverifiable (INCONCLUSIVE under --require-explanation). The normalized sign no longer wins. The mechanism format has no way to declare a relative-only effect, so a reversal is never read as either direction.
A raw no-clear-change with a definite normalized direction is still resolved by normalization; that's the drift it is meant to cancel. A definite raw direction with no clear normalized change stays not_shown.
Regression: test_raw_and_normalized_sign_reversal_is_not_read_as_either_direction reproduces your 10-pair case. The raw target is +20%; two references each cross 1 in different pairs, so neither is contradicted, yet their sum rises in every pair. The normalized result is a decrease. Before the fix this was explained; now it is unverifiable. test_normalization_still_resolves_a_raw_change_hidden_by_drift keeps the intended normalization path. Codex also built a single-reference variant and checked 8,000 direction combinations without finding another reversal.
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 016c7c7 and head ac0510b (draft).
Re-review after author push from prior CHANGES_REQUESTED on 1b2b459. Prior blockers cleared on this head:
- Governing #454 acceptance already recorded.
- Rebased onto 016c7c7 with focused harness tests reported in the PR body.
- Dual-switch priority and point/interval disagreement documented in docs/optimization-workflow-unification.md and covered by test_both_switches_apply_the_decision_before_the_explanation plus test_paired_decision_passes_when_only_the_point_estimate_misses_the_gate (agrees_with_point_gate is False). Defaults-off report-only path retained.
Merge process (not content blockers on this draft head): still draft; head is behind current main again after later merges — rebase before marking ready; run ready-lane CI on the rebased tip. Hardware A/A remains non-blocking while switches default off.
Acting steward re-review while hosted Dong-bot is on usage_limit; formal Approve as cursor[bot].
…ithin noise The percentile bootstrap lower bound for a median of n pairs is an order statistic whose one-sided error exceeds 2.5% at small n (3.1% at 5 pairs, 6.25% at 7). Use the exact distribution-free order-statistic interval (the inverted sign test) instead: the rank keeps each tail at or below 2.5%, so the minimum is six pairs and A/A at the gate passes at most 2.5% of the time. Declared expectations are judged as a family: region and counter intervals use the Bonferroni confidence 1 - 5%/k, and too few pairs for it is unverifiable. A declared-unchanged item is reported as not contradicted, never confirmed. A paired interval that straddles the gate shows only that passing was not demonstrated, which is not evidence that the gate was missed, so the round is INCONCLUSIVE (within_noise) and still does not block the search. Records say report_only: false when the interval decides the verdict. Reported by linyueqian on InferMatrixLab#457. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
When a region's raw and reference-normalized intervals point in opposite definite directions, an absolute increase can read as a relative decrease (two references can each cross 1 while their sum rises in every pair). No declaration predicts a relative effect, so record raw_normalized_conflict and leave that check unverifiable instead of letting the normalized sign win. Under --require-explanation, only contradicted evidence fails. An unsupported prediction is missing evidence, like a paired interval within noise, so it is INCONCLUSIVE; the candidate is still not adopted and the search continues. Reported by david6666666 on InferMatrixLab#454 and InferMatrixLab#457. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
6a3c123
ac0510b to
6a3c123
Compare
|
New head It also addresses david6666666's two points:
The 34 related test files and ruff pass on this head. A Codex review of the change found no code issues; its only item was syncing the PR body, which is now done. Please re-review on |
amy-why-3459
left a comment
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 6c7e237935a22cc5e90e954d4f397376e753e7d8 and head 6a3c123691b0ba6937084837c62a888b9735f59b(draft)。
对照上次已通过的 ac0510b21b887e34b7357b13dc3d2eb7c9e86c5f,本轮增量主要是 6a3c123691b0ba6937084837c62a888b9735f59b,并已 rebase 到 main 6c7e2379(含 #461,--decision / --require-explanation / --mechanism-file 现位于 src/inferrsi/apps/cli/arguments.py)。
已核对 david6666666 的两点:
--require-explanation下只有contradicted记 FAIL,unsupported/unverifiable/not_declared记 INCONCLUSIVE(src/inferrsi/harness/project.py与tests/test_project_explanation.py)。- raw/normalized 方向冲突记为
raw_normalized_conflict,_check_outcome将其判为unverifiable;回归测试test_raw_and_normalized_sign_reversal_is_not_read_as_either_direction覆盖了所述 10 对情形,test_normalization_still_resolves_a_raw_change_hidden_by_drift保留了归一化抵消漂移的路径。文档与 #454 已同步。
合并流程(非本 head 内容阻塞):仍为 draft;相对当前 main 2cbfab66b00d75e69be583ebfeabee2b8f83c4ea(#470/#472)落后 2 个提交,但无文件重叠,GitHub 显示可合并无冲突。标记 ready 前请再 rebase。我没有重跑作者声称通过的相关测试和 ruff,也没有跑 hosted 全量 pytest。
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 6c7e237935a22cc5e90e954d4f397376e753e7d8 and head 6a3c123691b0ba6937084837c62a888b9735f59b (draft).
Re-review after author push from prior PASS on ac0510b21b88 (Approves dismissed by rebase). Incremental head addresses david6666666: raw/normalized definite sign reversal records raw_normalized_conflict → check unverifiable; under --require-explanation only contradicted is FAIL and unsupported/unverifiable/not_declared stay INCONCLUSIVE. Prior blockers remain cleared (#454 accepted; dual-switch priority + point/interval disagreement documented/tested). Still draft; behind current main by 2 non-overlapping commits — rebase before ready. Formal Approve as cursor[bot].
|
Re-checked on
The four changed test files pass on this head (160 passed). One new gap. |
…ithin noise The percentile bootstrap lower bound for a median of n pairs is an order statistic whose one-sided error exceeds 2.5% at small n (3.1% at 5 pairs, 6.25% at 7). Use the exact distribution-free order-statistic interval (the inverted sign test) instead: the rank keeps each tail at or below 2.5%, so the minimum is six pairs and A/A at the gate passes at most 2.5% of the time. Declared expectations are judged as a family: region and counter intervals use the Bonferroni confidence 1 - 5%/k, and too few pairs for it is unverifiable. A declared-unchanged item is reported as not contradicted, never confirmed. A paired interval that straddles the gate shows only that passing was not demonstrated, which is not evidence that the gate was missed, so the round is INCONCLUSIVE (within_noise) and still does not block the search. Records say report_only: false when the interval decides the verdict. Reported by linyueqian on InferMatrixLab#457. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
When a region's raw and reference-normalized intervals point in opposite definite directions, an absolute increase can read as a relative decrease (two references can each cross 1 while their sum rises in every pair). No declaration predicts a relative effect, so record raw_normalized_conflict and leave that check unverifiable instead of letting the normalized sign win. Under --require-explanation, only contradicted evidence fails. An unsupported prediction is missing evidence, like a paired interval within noise, so it is INCONCLUSIVE; the candidate is still not adopted and the search continues. Reported by david6666666 on InferMatrixLab#454 and InferMatrixLab#457. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
With --require-explanation, fewer than six repetitions give no interval for any expectation, so every round would be unverifiable and nothing could be adopted; campaign start now rejects that. A round whose declaration needs more pairs than the frozen repetitions (Bonferroni over its expectations) is rejected before execution, so it spends no budget. Without the switch a declaration remains a report and is not restricted. Reported by linyueqian on InferMatrixLab#457. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
60ba2d0
6a3c123 to
60ba2d0
Compare
507ce15 to
0d6ab1f
Compare
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent incremental review (COMMENT), bound to head 0d6ab1f; previous reviewed head 507ce15. Current base and observed main both equal 8aba657.
Method: compared the two complete Git trees directly, not the three-dot compare (the histories have diverged). Read source and my own prior reviews only; PR/issue narratives were kept out of this review context, with original #454 requirements extracted separately without review opinions.
Findings: no new blocking defect found in the inspected #457 feature path. measurement.py, test_harness_measurement.py and test_project_explanation.py are byte-identical to the previously reviewed head. The relevant rebased changes in project.py / optimization.py / optimization_session.py move hash, lock and atomic-write imports to Harness-owned primitives and extend runtime identity. I inspected the new primitives against the Infra implementations; the corresponding bodies preserve their hash/serialization/locking behavior. Other mainline delivery/channel changes are not independently certified by this focused feature review.
Requirement cross-check: the source retains frozen switches and repetition checks; per-repetition exact-order-statistic intervals; Bonferroni profile checks and non-equivalence semantics for unchanged; pre-execution mechanism binding and digest verification; performance-first/explanation-only-downgrade precedence; paired-median selection rather than independent-median veto; and selected-candidate/selected-mechanism final retesting. I inspected regression source covering default report-only behavior, both-switch precedence, negative point-estimate paired PASS followed by selection/final, uncheckable declarations before budget consumption, and integrity/recovery behavior. The Chinese documentation and synthetic ablation script are present and were inspected. Real-device A/A and hardware qualification are not claimed and are outside the extracted merge requirements.
Verification at this head: hosted run 44254600097 has full-check (3.11), installed-package (3.10/3.11), host-compatibility (Windows/macOS), and pr-references successful; both declarations checks and knowledge-model-policy are successful. full-check (3.10) is still in progress (Ruff successful, pytest running). dashboard-browser is skipped because this PR is draft; this is not a browser-test pass. I did not execute repository code, pytest, lint, simulation, or hardware work locally in this review.
Conclusion: COMMENT, not APPROVE yet. The previously missing original-requirement provenance and much of the hosted execution evidence are now available, but the required hosted-CI completion condition is not yet established while full-check (3.10) remains running. No merge, deployment, code change, or experiment restart performed.
…ithin noise The percentile bootstrap lower bound for a median of n pairs is an order statistic whose one-sided error exceeds 2.5% at small n (3.1% at 5 pairs, 6.25% at 7). Use the exact distribution-free order-statistic interval (the inverted sign test) instead: the rank keeps each tail at or below 2.5%, so the minimum is six pairs and A/A at the gate passes at most 2.5% of the time. Declared expectations are judged as a family: region and counter intervals use the Bonferroni confidence 1 - 5%/k, and too few pairs for it is unverifiable. A declared-unchanged item is reported as not contradicted, never confirmed. A paired interval that straddles the gate shows only that passing was not demonstrated, which is not evidence that the gate was missed, so the round is INCONCLUSIVE (within_noise) and still does not block the search. Records say report_only: false when the interval decides the verdict. Reported by linyueqian on InferMatrixLab#457. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
0d6ab1f to
3464bce
Compare
When a region's raw and reference-normalized intervals point in opposite definite directions, an absolute increase can read as a relative decrease (two references can each cross 1 while their sum rises in every pair). No declaration predicts a relative effect, so record raw_normalized_conflict and leave that check unverifiable instead of letting the normalized sign win. Under --require-explanation, only contradicted evidence fails. An unsupported prediction is missing evidence, like a paired interval within noise, so it is INCONCLUSIVE; the candidate is still not adopted and the search continues. Reported by david6666666 on InferMatrixLab#454 and InferMatrixLab#457. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
With --require-explanation, fewer than six repetitions give no interval for any expectation, so every round would be unverifiable and nothing could be adopted; campaign start now rejects that. A round whose declaration needs more pairs than the frozen repetitions (Bonferroni over its expectations) is rejected before execution, so it spends no budget. Without the switch a declaration remains a report and is not restricted. Reported by linyueqian on InferMatrixLab#457. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
|
Rebased onto current main
|
david6666666
left a comment
There was a problem hiding this comment.
Static rebase review at 3464bce2; the AB/BA and service-reuse paths remain separate.
|
|
||
| 两个开关同时打开时按固定顺序判定:先由 `--decision` 得到性能结论,`--require-explanation` 只作用于性能结论为 PASS 的轮次,把它降为 FAIL 或 INCONCLUSIVE,不会提升或改写非 PASS 的结论。因此性能结论已是 FAIL 或 INCONCLUSIVE(包括 `within_noise`)时,无论 `explanation.status` 是什么,最终 verdict 和 `reason` 都保持性能结论;`explanation` 仍完整写入报告。 | ||
|
|
||
| 两个开关和 `--mechanism-file` 只适用于默认的 AB/BA 评估:`--evaluation-mode service_reuse` 不记录逐对样本和 profile,因此该模式下 `campaign start` 拒绝这两个开关,`campaign round` 拒绝 `--mechanism-file`,不会悄悄忽略。 |
There was a problem hiding this comment.
Could this say service reuse does not produce the paired AB/BA samples or per-measurement breakdown/counters these switches consume? It still collects and retains its own profiling stage under #480. Saying that it does not record profiles misstates the current service-reuse contract.
There was a problem hiding this comment.
Agreed, thanks. That wording misstated #480. Fixed in 89f6c47f: the doc now says that service reuse does not produce the AB/BA pair samples or the per-measurement breakdown/counters these switches read. It also notes that the mode keeps its own profiling stage, which is not an input to them. The code comment at the start-time refusal says the same.
zengchuang-hw
left a comment
There was a problem hiding this comment.
NO PASS
Independent incremental code review of 3464bce258da5ac0add08727571dabe6c51ef315, targeting main at 9b418d7fe16f3979b67f74f290949629ef103e8f, against my previous review. Compared complete trees and the current feature diff against the governing scope in #454.
Required: keep scoreless settlement specific to paired project evaluation. optimization.py:304–310 now treats any successful execution with an assessor result of status="completed", eligible=False and no score as an ordinary completion. The base controller rejected that result as an infrastructure failure and blocked the session. The new packaged benchmark test codifies the changed behavior: rounds remain active and final exits successfully without a score. This changes a packaged benchmark contract outside #454, which scopes the feature to ordinary project evaluation and preserves default behavior. Please gate the scoreless exception on a project plan with the frozen paired decision enabled, retain rejection for invalid benchmark assessments, and cover both cases in regression tests.
The paired statistics and project assessor are byte-identical to my last reviewed head. I checked the new guards that reject paired/explanation options and mechanism declarations in service reuse, plus the packaged CLI rejection of mechanism declarations. No repository code, tests, lint, simulation, or hardware benchmarks were run for this static code review. This is a GitHub comment, not an approval or an issue acceptance.
Pair the k-th baseline and candidate samples that steps() already alternates, and report the median paired speedup with a seeded percentile bootstrap interval and its relation to the frozen gate. The field is report-only: verdict, eligibility and selection are unchanged, and any value that cannot be computed is reported as unavailable so records stay strict JSON. Fewer than five pairs report no interval because simulated coverage at three pairs is about 75%. tools/measurement_simulation.py reproduces the synthetic ablation. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Agent-assisted-by: Codex CLI / gpt-6.1-sol [effort=high] (review) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
…ches Measurement scripts may add per-step region breakdowns and work counters. A round may declare its mechanism and expected region/counter directions. Regions declared unchanged act as an in-step clock, so ratios to them and their share of the step total cancel device-wide clock or thermal drift; counters such as launches or DRAM bytes show whether a small gain is reproducible. Directions come only from paired intervals, with no repository-wide tolerance. Two switches are frozen at campaign start and default to the existing behaviour: --require-explanation adopts only candidates whose mechanism is confirmed by the data, and --decision paired requires the paired interval to clear the gate, failing within-noise rounds without blocking the search. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
… evidence Bind the round's mechanism into control state before the candidate runs and assess against that copy, so a candidate cannot rewrite its own declaration. Require five pairs before any profile direction, treat zero-baseline ratios as undefined, report unavailable shares instead of raising, keep raw evidence for every region, and never normalize declared references against each other. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Agent-assisted-by: Codex CLI / gpt-6.1-sol [effort=xhigh] (review) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
The candidate runs with the caller's privileges and can write state.json or the attempt copy. Pass the declaration held since submission to the assessor, record its digest, and treat a copy that no longer matches as an integrity failure. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Agent-assisted-by: Codex CLI / gpt-6.1-sol [effort=xhigh] (review) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
An interrupted attempt is settled by a new process that can only read files the candidate could write. Settle it without the declaration, mark the record, and leave require_explanation to keep the round unadopted. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Agent-assisted-by: Codex CLI / gpt-6.1-sol [effort=xhigh] (review) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
With both switches on, --decision sets the performance verdict first and --require-explanation only downgrades a PASS. The paired decision reads the interval alone, so a candidate whose point estimate misses the gate but whose paired interval clears it passes, with agrees_with_point_gate recorded false. Regression tests pin both behaviors. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
…ithin noise The percentile bootstrap lower bound for a median of n pairs is an order statistic whose one-sided error exceeds 2.5% at small n (3.1% at 5 pairs, 6.25% at 7). Use the exact distribution-free order-statistic interval (the inverted sign test) instead: the rank keeps each tail at or below 2.5%, so the minimum is six pairs and A/A at the gate passes at most 2.5% of the time. Declared expectations are judged as a family: region and counter intervals use the Bonferroni confidence 1 - 5%/k, and too few pairs for it is unverifiable. A declared-unchanged item is reported as not contradicted, never confirmed. A paired interval that straddles the gate shows only that passing was not demonstrated, which is not evidence that the gate was missed, so the round is INCONCLUSIVE (within_noise) and still does not block the search. Records say report_only: false when the interval decides the verdict. Reported by linyueqian on InferMatrixLab#457. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
A float 2 ** pairs overflows near a thousand pairs; compare binomial tails as exact rationals instead. A failure while building profile evidence is reported as unavailable rather than raised, so it cannot change the default verdict. Found by Codex review of the exact-interval change. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Agent-assisted-by: Codex CLI / gpt-6.1-sol [effort=xhigh] (review) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
When a region's raw and reference-normalized intervals point in opposite definite directions, an absolute increase can read as a relative decrease (two references can each cross 1 while their sum rises in every pair). No declaration predicts a relative effect, so record raw_normalized_conflict and leave that check unverifiable instead of letting the normalized sign win. Under --require-explanation, only contradicted evidence fails. An unsupported prediction is missing evidence, like a paired interval within noise, so it is INCONCLUSIVE; the candidate is still not adopted and the search continues. Reported by david6666666 on InferMatrixLab#454 and InferMatrixLab#457. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
With --require-explanation, fewer than six repetitions give no interval for any expectation, so every round would be unverifiable and nothing could be adopted; campaign start now rejects that. A round whose declaration needs more pairs than the frozen repetitions (Bonferroni over its expectations) is rejected before execution, so it spends no budget. Without the switch a declaration remains a report and is not restricted. Reported by linyueqian on InferMatrixLab#457. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
Under --decision paired the verdict read the paired interval, but adoption still compared independent candidate medians, so a paired PASS could be outranked and never reach final. The frozen selection is now the median paired speedup against the baseline (1.0), and that same speedup is the round's score, speedup and improvement. Point values stay in point_estimate. Each round records why it was or was not selected. The default point mode and benchmark sessions keep their selection. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
…aigns Service-reuse campaigns (InferMatrixLab#480) record neither AB/BA pairs nor per-step profiles, so --decision paired, --require-explanation and --mechanism-file would be silently ignored there. start and round now reject them. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
A completed, ineligible round could settle without a score for any assessor, which changed the packaged-task contract outside InferMatrixLab#454. Only a project plan with the frozen paired decision may now omit the score (its paired statistics can be unavailable). Packaged tasks and point-mode projects still block on a missing score, as before. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
Service reuse keeps its own profiling stage (InferMatrixLab#480); what it does not produce are the AB/BA pairs and per-measurement breakdown/counters that the paired decision and explanation checks read. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
The delivery projection (InferMatrixLab#485) reported 1.0 for a retained baseline whenever the final completed. Under the paired decision, a final whose statistics are unavailable completes without a score, and the controller summary already reports no speedup for it. The delivery bundle now agrees. Also covers InferMatrixLab#501's --strategy/--knowledge-ref alongside a bound mechanism on one round and its final. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
3464bce to
b9325aa
Compare
|
Rebased onto current main @zengchuang-hw, the scoreless-settlement scope (review). Agreed, and fixed in
Other changes on this head:
Local checks: ruff and With the reviewers' points addressed, I'm marking this ready for review. |
zengchuang-hw
left a comment
There was a problem hiding this comment.
PASS
应本轮请求增量核对:3464bce → b9325aabca4593b8c2eae460b8d574dbf109d83d,上轮意见。
上轮阻塞解除:只有普通 project、冻结的 paired 决策且本轮明确不能入选时,才允许无分数结算;Benchmark 和 point 仍要求有限正分。新增回归覆盖这两条拒绝路径和 paired 可继续搜索,独立 final 缺分时交付也不再显示伪造的 baseline 1.0。核对 #501 合入后的接口,strategy、knowledge_refs 与 mechanism 同时传递,final 仍复核机制。配对统计与评估脚本本轮无实质变更,未发现新阻塞。
已静态核对实际增量及添加的测试;未执行仓库代码、测试或查看 CI。
|
#516 has been merged into |
Governing Issue and delivery slice
Refs #454
campaign start/round/final)新增三项能力,默认都不改变现有判定:steps()已有的 AB/BA 交替,把同一次重复里的基线和候选样本配对,报告逐对加速比中位数的精确次序统计量区间(符号检验反演,至少 6 对),以及区间与门槛的关系。breakdown(分区域耗时)和counters(工作量计数);每轮可以用--mechanism-file声明机制和预期方向;Harness 按声明条数做 Bonferroni 校正后机械核对,给出每条核对的outcome和explanation.status。--require-explanation只采纳explained的候选;--decision paired用配对区间判定,并按逐对加速比中位数选优(未配对中位数不能否决配对 PASS);区间跨门槛判 INCONCLUSIVE(within_noise),不被采纳也不阻塞搜索。两者同时打开时,先由--decision得出性能结论,--require-explanation只降级 PASS。benchmarks session拒绝--mechanism-file);Dashboard 展示;与具体 profiler 绑定的采集脚本;GPU/NPU 实测。Knowledge authorship declaration
本 PR 不触及 knowledge-model policy 的受保护路径,预期为 Not applicable。
Changed artifacts and evidence
src/inferrsi/harness/measurement.pyinterval_rank、median_interval、min_pairs,二项尾用有理数精确比较)、配对统计、profile_evidence(Bonferroni)(原始与归一化方向相反时记为raw_normalized_conflict)、validate_mechanism、explain;无法计算的值一律返回unavailable,记录保持严格 JSONsrc/inferrsi/harness/project.pyassess()接入两个开关和机制核对;--decision paired时score/speedup/improvement_pct取逐对加速比中位数(基线 1.0),点估计保留在point_estimate,不可表示的值记为空而不阻断;profile 报告出错时记为unavailable,不影响判定;新增设置项的校验(--require-explanation至少 6 次重复)src/inferrsi/harness/optimization.pyselection(adopted/not_eligible/not_better_than_best/baseline_identity);只有冻结了paired判定的 project 计划中、明确eligible: false的完成轮次可以没有 score(题包与 point 模式仍按缺 score 阻断)src/inferrsi/harness/optimization_session.py--evaluation-mode service_reuse(#480)下拒绝两个开关和--mechanism-file,不悄悄忽略;paired时把选优冻结为paired_median_speedup(maximize,初值 1.0);读取并校验--mechanism-file,开关打开时在执行前拒绝所需对数超过冻结重复次数的声明;_runtime_identity纳入measurement.pysrc/inferrsi/apps/views/dashboard_data.pymetrics.name/direction)标注中位数,不用选优指标src/inferrsi/apps/delivery/projection.pysrc/inferrsi/apps/cli/arguments.py--decision、--require-explanation、--mechanism-filedocs/optimization-workflow-unification.mdpaired下的选优与selection记录、局限tests/test_harness_measurement.pyunchanged只记"未推翻"、原始与归一化方向相反不被读成任一方向、机制校验与分类tests/test_project_explanation.py--strategy/--knowledge-ref与机制声明同轮保留;final 无 score 时 delivery 不报 1.0;point 模式缺 score 仍阻断;配对 PASS 按配对加速比被采纳并由 final 复测(评审者的 6 vs 9 情形)、更快的早期候选使后来的 PASS 记为not_better_than_best;配对统计不可用或点估计溢出时记录轮次而不阻断;确认的机制在 final 复核;不可采纳情形都让搜索继续;开关关闭时只报告;within_noise为 INCONCLUSIVE;两开关组合 6 例;点估计未过而区间过门槛判 PASS;profile 报告出错不改判;重复次数不足时 start 拒绝、声明无法核对时在执行前拒绝且不耗预算、开关关闭时仍只报告;篡改state.json或副本;中断后伪造声明的恢复tests/test_project_optimization.py、tests/test_optimization_session.pytests/test_project_service_reuse.pyservice_reuse下--decision paired、--require-explanation与--mechanism-file被拒绝,不写计划、不记轮次tools/measurement_simulation.py--gate-pct 0给出门槛处 A/A 比例与精确上界判定逻辑
explanation.status--require-explanation下explainedconfirmed,且没有任何一条contradictedunsupportednot_shown(未检出;区间仍可能包含预期的变化)contradictedunchanged的项检出变化unverifiablenot_declared--decision paired区间与门槛above_gatepaired_interval_above_gate(即使点估计未过门槛;此时agrees_with_point_gate: false)straddles_gatewithin_noise(没能证明过门槛,不是门槛未达到的证据)below_gate、gate_unreachableperformance_gate_not_metpaired_statistics_unavailable,不退回未配对规则在
--require-explanation下只有contradicted(数据与预期相反)记 FAIL,证据不足的情形都记 INCONCLUSIVE,与配对性能判定的原则一致。若某区域原始比较和"除以参照区域"后的比较给出明确相反的方向(例如原始增加 20%、相对参照却下降),记为raw_normalized_conflict,该条核对为unverifiable:机制声明无法表达"只预期相对变化",两个方向都不采信。声明为
unchanged的项只会被记为not_contradicted(未检出变化)或contradicted,不会被"确认不变":没有等价性容差时,区间包含 1 不是"没变"的证据。声明为unchanged的区域同时是本轮的假设:其他区域会另外给出"除以同一步骤参照区域"后的比较,share_based_speedup用参照占比推算整体加速。只有让一个步骤内所有区域按同一比例缩放的变化,才会被这样抵消。参照区域本身只做原始比较,不会互相归一化。合成消融(
python tools/measurement_simulation.py,CPU 仿真,非设备实测)每格 200 次 campaign,数据 seed 20261115;不配对区间这一对照规则用 bootstrap 10000 次。200 次 campaign 的 Monte Carlo 标准误约 0.01–0.03,门槛处的误报以下方精确上界和大样本复核为准。
Checks and dependencies
ruff check src tests tools:通过。bebd4914)上运行 36 个 harness / campaign / project / dashboard / optimization 相关测试文件(含tests/test_harness_measurement.py、tests/test_project_explanation.py、tests/test_project_optimization.py、tests/test_optimization_session.py、tests/test_module_boundaries.py、tests/test_target_module_import_gate.py、tests/test_package_root_inventory.py和 Dashboard 测试):全部通过,无失败。Git 相关测试使用临时 Git 2.49.1(本机 2.43 不支持--no-lazy-fetch)。gpt-6.1-sol,xhigh)本地只读评审,每一轮都用对抗输入去复现问题。Infinity;零基线计数在 1 对时就给出确定方向;breakdown 全为零时抛异常;多个参照区域互相归一化;候选在执行期间改写mechanism.json或state.json中的声明;中断后经recover接受伪造的声明(现记录mechanism_unverified_after_recovery,按未声明处理)。2 ** n转浮点溢出,且 profile 报告异常会改变默认判定。已修复(有理数精确比较;报告出错记为unavailable)并补回归测试。第八轮复核结论:Ready(1024 对复现通过;注入报告异常后默认判定不变;31,255 组精确秩检查无偏差)。unverifiable;8,000 组方向组合检查无反向误判;原始样本不足或零基线不会经归一化变成明确方向;assess 与独立 final 都是unsupported → INCONCLUSIVE、contradicted → FAIL,搜索继续;rebase 后无旧 CLI 路径残留。结论 Ready with fixes,唯一一项是同步本 PR 说明(已同步)。--require-explanation下 start 拒绝 3/5 次重复、round 在执行前拒绝 6 次重复下的 4 条预期,返回用法错误,campaign 不变、不耗轮次;k = 1…12 的所需对数与pairs_needed完全一致;开关关闭时仍可运行。结论 Ready,无 P0/P1/P2。plan.json已有plan_digest保护。measurement.py只用标准库。steps()两侧等长且交替执行;区间的精确性依赖各次重复独立、比值来自连续分布,离散计数出现并列时偏保守。harness/*.py会改变 runtime identity,旧 runtime 启动的会话需要用原来冻结的 runtime 恢复(既有规则)。评审后的修订
a3e33851(含 #461 的 CLI 迁移、#474、#323、#477、#466、#483、#475、#478、#485、#484、#456、#486、#493、#495、#499、#496、#503、#480、#508、#514、#501、#507、#510);#478 把sha256移到harness.artifact_primitives,project.py的导入随之更新;#480 把 AB/BA 参数改为非必填并新增service_reuse模式,两个开关保留在 AB/BA 路径,_session_assess合并 #480 的run_protocol与执行前绑定的声明;#501 的strategy/knowledge_refs与mechanism在evaluate_round、pending 和 round CLI 中并存。optimization.py的_settle与 main891aadda冲突:保留 main 新增的"执行必须正常结束"检查,其后用执行前绑定的声明调用判定器;新 head 上重跑测试test_both_switches_apply_the_decision_before_the_explanation(6 例)、test_paired_decision_passes_when_only_the_point_estimate_misses_the_gateorder_statistic_rank、achieved_confidenceunchanged不是证据,且没有多重比较校正family_size、confidence、pairs_needed);unchanged只记not_contradicted/contradicteddocs/contributing.md的 FAIL 定义冲突within_noise);低于门槛仍为 FAILreport_only: true--decision paired时为falseraw_normalized_conflict,该条核对unverifiable;test_raw_and_normalized_sign_reversal_is_not_read_as_either_direction复现了他描述的 10 对情形(两个参照各自跨过 1,但总和每对都上升),修复前判explained,修复后为unverifiableunsupported是证据不足,应与contradicted区分--require-explanation下unsupported改为 INCONCLUSIVE,只有contradicted记 FAIL;候选都不被采纳--require-explanation在少于 6 次重复时仍可启动,任何预期都拿不到区间,没有候选能被采纳;单轮声明所需对数超过冻结重复次数时也注定unverifiablecampaign start拒绝少于 6 次重复;round --mechanism-file在执行前拒绝所需对数(与profile_evidence的pairs_needed同一公式)超过冻结重复次数的声明,不记录轮次、不耗预算;开关关闭时声明仍只报告--decision paired下判定看配对区间,选优仍比较未配对中位数:9 对、1% 门槛时区间 [1.111, 1.111] 判 PASS,但中位数 6 vs 9 使基线仍被选中,final 没有复测该候选selection = paired_median_speedup(maximize,基线 1.0),score/speedup/improvement_pct取逐对加速比中位数,未配对值留在point_estimate;每轮记录selection.outcome说明是否被选中及原因;默认point与题包会话的选优不变。test_paired_decision_also_selects_the_candidate_that_final_retests端到端复现他的 9 对情形(采纳、final 复测、optimization_speedup1.111)3464bce2)paired判定时放行;题包与 point 模式仍按缺 score 阻断,test_packaged_task_assessment_without_a_score_still_blocks、test_point_decision_project_rounds_must_still_be_scored覆盖(修复前均失败)breakdown/counters;代码注释同步Remaining work and Issue closure
Remaining Issue requirements / follow-up:
hypothesis、verdict 和采纳结果;mechanism、explanation(含每条outcome)、profile_evidence、paired_statistics已在同一份控制记录里,后续可在 Apps 视图层只读投影,不另写记录;另有互补的知识 PR(task: add GPU measurement-validity and profiler-attribution guidance to inference-benchmark-design #455),两者互不依赖。
Goal-acceptance owner or accepted closure delegation: Dong1017(rfc: 可解释的改进——用 profile 证据和配对统计判定 PASS #454 记录);未授予关闭委托。
Goal ready for acceptance? Not claimed