feat(metrics): add imp@baseline to catch slow multi-week regressions#1
Open
nanookclaw wants to merge 1 commit intochoutos:mainfrom
Open
feat(metrics): add imp@baseline to catch slow multi-week regressions#1nanookclaw wants to merge 1 commit intochoutos:mainfrom
nanookclaw wants to merge 1 commit intochoutos:mainfrom
Conversation
imp@week compares each week to the immediately prior week. This misses slow monotonic regressions: if an agent degrades 0.05/week, each consecutive delta reads 'stable' while cumulative drift grows silently. Add --baseline <WEEK> flag to compute-impk.sh that also prints an imp@baseline section anchored to a fixed reference week: imp@baseline = avg_score(current) - avg_score(anchor) Usage: ./compute-impk.sh --baseline 2026-W12 ./compute-impk.sh 2026-W16 --baseline 2026-W12 The baseline report runs alongside the existing imp@week report. Omitting --baseline preserves full backward compatibility. Also document imp@baseline in: - evals/METHODOLOGY.md (new subsection with example output) - metrics/README.md (table + updated compute-impk.sh docs) - README.md (Core Metrics table + Trend Classification note)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
imp@weekcompares each week to the immediately prior week. This misses slow, monotonic regressions: if an agent degrades 0.05/week, each consecutive delta reads "stable" while cumulative drift grows silently.Example: Ada drops from 4.5 → 4.2 → 3.9 → 3.6 over four weeks. Each
imp@weekis −0.3 (visible), but if the drop were 0.05/week instead, all four weeks would report "stable" despite a cumulative −0.2 loss.Solution
Add
--baseline <WEEK>flag tocompute-impk.shthat prints an additionalimp@baselinesection anchored to a fixed reference week:The baseline report runs alongside the existing
imp@weekreport. Omitting--baselinepreserves full backward compatibility.Usage
# Both imp@week (W16 vs W15) and imp@baseline (W16 vs W12) in one run ./metrics/scripts/compute-impk.sh 2026-W16 --baseline 2026-W12Example output:
imp@weekshows a small regression this week;imp@baselinereveals −1.23 cumulative drift since W12 — the real story.Changes
metrics/scripts/compute-impk.sh—--baselineflag, newimp@baselineoutput section,declare -A current_avgsto avoid double-computing agent scoresevals/METHODOLOGY.md— Newimp@baselinesubsection with rationale + example outputmetrics/README.md— Table updated,compute-impk.shdocs extended with--baselineusageREADME.md— Core Metrics table includesimp@baseline, Trend Classification section notes the blind spotBackground
This follows naturally from the METHODOLOGY.md principle: "Always look at the delta, not the absolute." Two deltas — consecutive and anchored — give you the full picture. Weekly volatility + cumulative drift.