Skip to content

feat(benchmark): show source-bound retained outcome improvement - #6829

Merged
huangruiteng merged 2 commits into
mainfrom
codex/research-retained-value-20261018
Oct 18, 2026
Merged

huangruiteng merged 2 commits into
mainfrom
codex/research-retained-value-20261018

Conversation

@loopx-agent

@loopx-agent loopx-agent commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

The six process judgments do not show whether the retained native outcome improved. Add a separate Retained improvement / 已采用成果增量 metric, computed by the existing benchmark-toolkit TypeScript owner from an explicit baseline, native optimization direction and source-bound retention evidence.

Native error 7 → retained 3 displays +4 points. Rejecting a candidate with error 9 while retaining 7 displays 0, with candidate change −2 inside the disclosure. Missing baseline or adoption evidence stays unknown; actual adopted regressions remain negative. This is a provider-declared native comparison, not a composite score, objective completion or LoopX causal attribution.

  • Optional comparison extends the existing native-capture contract; omitting it preserves the prior projection shape and model analysis key. An archived submission time may explicitly remain null instead of inventing a clock. Known times retain calendar/window validation.
  • Goal overview shows one compact outcome value before the process judgments. Hover/focus/tap explains it; expansion opens baseline, retained measurement and actual retention evidence. Detailed native records move into the existing Sources and usage disclosure. Changed sources remain visibly historical.
  • Selecting an analysis scope now creates a browser history entry. Back/Forward restores the real selected scope instead of dropping the previous selection.
  • Owning capability: builtin benchmark-toolkit, provider loopx-core. Python remains the existing file/HTTP transport; it only admits the additional exact bound citation ranges. No solver, evaluator, scoring, authority-store, model dispatch default or installed App changes.

Validation

  • 17 focused TypeScript assessment/native-outcome tests; 16 Python source-store and production HTTP API tests passed.
  • Dashboard + packaged Chat build, control-plane typecheck, documented Ruff and no-argument mypy profiles passed.
  • Ego Lite against the packaged bundle and actual local HTTP/native-pack owner: positive/zero/unknown, original retention citation, changed-source historical warning, reload, real selector Back/Forward, hover/keyboard/tap explanation and 390 px overflow checks passed. Synthetic screenshots below; the browser acceptance runner is extended for the same semantics, with its full automated run left to CI.
  • Additional file-target mypy trajectory_assessment_store.py probe failed outside the documented profile, reporting 4,920 diagnostics across 582 imported files. It is retained as a failed probe, not treated as a passing check.
  • Premerge automatic checks pass; benchmark-sensitive manual review remains required. No merge/install/release is claimed. Private benchmark trajectories and local operating state are excluded.

Remaining integration and CI boundary

This PR supplies the analysis metric and per-Goal renderer. Fixed case-pack registration does not create a real experiment child Goal. The remaining experiment journey must use actual parent/child registration and run/source identities, with comparison at the parent and analysis in the corresponding child.

GitHub kernel-static-checks currently fails the pinned top-level module budget: 147 modules against a limit of 145. Base 3203b55843d2dcca5b77822ed857442adf944587 and this head have the same 147 top-level Python modules; this PR adds none. The hard gate remains held; its budget is not increased to mask the failure. Other CI and exact-head review remain pending.

Visual evidence

Tested source revision: c01d3863cec14cfc21e05724d14212a72a922f2a. Actual packaged browser captures with synthetic input and a disposable deterministic analysis child; these images are not benchmark findings or installed-App evidence. The before renderer uses the same synthetic capture. Desktop is the initial 1716×980 viewport; mobile is the affected research section at 390×844 after scrolling. Native details and optional explanations are collapsed by default.

Before After
Before: process judgments and expanded native readback After: compact retained improvement before process judgments

Narrow-screen retained improvement, explanations collapsed

Synthetic local preview (author machine): open packaged research overview.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
@huangruiteng
huangruiteng merged commit ca96996 into main Oct 18, 2026
8 of 10 checks passed
@huangruiteng
huangruiteng deleted the codex/research-retained-value-20261018 branch October 18, 2026 16:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants