Closed
Copy link
Labels
status: confirmedA maintainer confirmed the item is real and in scope; ready to be worked.A maintainer confirmed the item is real and in scope; ready to be worked.
Description
Refs #7582
Outcome
The session-flow description trim pilot runs and is measured. The skill-visibility measurement that gates Q183-Q187 is posted, so #7582's description standard and #7588's listing budget can proceed.
Decisions
- Q185: trim pilot on session-flow, non-inferiority with a 5-point margin.
- Q137, Q182: the skill-visibility measurement gates Q183-Q187.
- Q187:
skillListingBudgetFractionstays 0.07 until the pilot ships.
Done when
- The session-flow trim pilot's result is posted as a
Probe result:comment that shows non-inferiority within a 5-point margin. The skill-visibility measurement that gates Q183-Q187 is posted with it. -
skillListingBudgetFractionis still 0.07 when the pilot's result is posted; this slice does not change it (Q187).
Out of scope
- Setting the fleet listing budget (marketplace: pstack recheck and listing budget #7588, feat(context-budget): keep plugin skill descriptions visible when the skill listing overflows #6458).
Metadata
Metadata
Assignees
Labels
status: confirmedA maintainer confirmed the item is real and in scope; ready to be worked.A maintainer confirmed the item is real and in scope; ready to be worked.
Type
Fields
No fields configured for Task.
Activity
kyle-sexton commentedon Nov 1, 2026
Claimed. Backlog HITL host-run worker, 2026-11-01. Running the session-flow description trim pilot: live invocation probes (claude plugin eval, Skill-fired graders) on current versus trimmed descriptions, plus the skill-visibility listing measurement. Branch perf/11954-session-flow-trim-pilot. The coordinator posts the result.
kyle-sexton commentedon Nov 1, 2026
Probe result: Host run. Ran the session-flow description trim pilot as live invocation probes (
claude plugin eval) on the current and the trimmed descriptions, and took the skill-visibility listing measurement on both fleets.Verdict: the trim clears non-inferiority at the 5-point margin (Q185).
What ran
claude-opus-5-5was pinned in both arms, and every trace confirms it.Use whenwith at most three quoted triggers, aiming for 250 characters, none over 419.check(174 characters),setupandtidy-workare unchanged.setupandtidy-workare user-invoked and have no probes.check-skill.sh plugins/session-flow/skillspassed with 0 errors. Check 3 gave advisory "dropped trigger" warnings, which are expected with this standard.plugins/session-flow/probes/(17 files, 18 queries each, 306 in all).measure-invocation.sh emit-plugin-eval --runs 1.claude plugin eval <arm>/session-flow --ablation none --keep-temp -j 2, at most two suites at once, each in a capped scope.plugins/session-flowcopied from origin/main8eca2165df. The trim arm is the same copy with only the 16SKILL.mddescription lines changed.--disallow-tools Read Glob Grep.--disallow-tools Read Glob Grep Agent.--disallow-tools Read Glob Grep Agent.run-validity.pygave VALID for all six suites: every trace resolves, there were no run errors, no permission denials, and one model throughout.Agent/Taskstayed callable. Some kept traces in both arms do call them.Agent, and pass the probe-tools check.Scores (1 run per case)
listing-overlapfloor is not a model rate. It counts words shared with the listing, so a shorter description scores lower by construction:Limits on this result
validategives 35 WARNs where a positive probe copies listing words. That favours the baseline wording, so it biases against the trim.Skill-visibility measurement (gates Q183-Q187)
The run was
audit_skill_visibility.py --plugins-rootover the whole 93-plugin checkout fleet (386 skills), base versus trim. Only session-flow differs between the two. The fraction is 0.07, read from.claude/settings.json.skillListingBudgetFraction: unchanged at 0.07. It is at.claude/settings.json:61on origin/main8eca2165dfand on the trim head (Q187). This slice does not touch it.Branch:
perf/11954-session-flow-trim-pilot, headba4f5ab39b933a875bc4d1f5c21a01af6345f67d, pushed.SKILL.mddescription lines plus.changes/session-flow/perf-11954-description-trim-pilot-7c3e91a4.md.scripts/check-affected.sh --testson the committed head: 14 passed, 2 skipped.Target labels: add none; remove host-run.