You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Have implementation:implement read the consumer's own idioms for the language it detects #15835
When /implementation:implement settles a type choice, it uses the consumer's own idioms for the language it is working in, found in the consumer's repository, instead of only the language-neutral rules.
Demo: in a repository that declares an idioms file for its language, implement a change that needs a cast or a parse; the run cites that file. In a repository that declares none, the run says so and uses the neutral rules.
Source: Lauren Tan's pstack (cursor/plugins, MIT), pstack/skills/typescript-best-practices/SKILL.md: a per-language rule table read while building. Her TypeScript rules are not taken.
Decisions
Owner, 2026-11-12: a skip must be validated. The earlier reason ("it conflicts with shipping no language specifics") hid a real gap: the type-discipline rules say "Each skill or agent that reads this file says where its own language idioms live, if it has any", and implement names none.
The marketplace ships no idioms file for this. The consumer declares one per language, resolved through the ecosystem-resolution procedure; with none declared, implement reports that and continues on the neutral rules.
No fallback to a shipped per-language page: the approved ecosystem-specifics plan removes the one that exists.
Key interfaces
The ecosystem-resolution contract (the consumer's records, then inference) is how the language and its idioms file are found.
Done when
The implement skill states where its language idioms come from: a consumer-declared file for the detected language.
With no declaration, the run names the missing idioms source and proceeds; it reads no shipped language page.
One new eval case covers each branch; the skill's existing implement cases are non-inferior on a two-version compare (opus and sonnet aliases, three runs per side, --case filter, a --max-cost-usd ceiling, no new batch after a usage-limit error), run in the one local eval-runner session; the pull request carries do-not-merge until the record lands.
The pull request body names this repository's dogfooding step.
I understand this as: /implementation:implement should read a consumer-declared idioms file for the detected language. With no declaration, it should say so and use the neutral rules. Pull request #16086 merged with Refs:, so the issue stayed open. The implement skill and its evals already mention idioms, so part of this may be done. Nobody has yet checked the "Done when" items against what #16086 left: the skill wording, the two eval cases, the two-version compare record, and the dogfooding note. Please confirm which items remain, or close the issue if all are met.
Not admitted to the issue lane: merged pull request #16086 names it with Refs: after its newest decision or refinement comment, so its work may already be done: close the issue if it is, or post a comment whose first line is **Decision recorded.** saying what is left, which sends it back through intake. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).
melodic-automation-lanes commented on Nov 14, 2026
Pull request #16086 merged with Refs:, so this issue stays open and nothing starts the lane on it again. work-unblock removed status: ready and agent-ready, so the hourly triage sweep sends it back to intake triage. Admission then parks it at status: confirmed for a check of what that pull request left, until a **Decision recorded.** comment says what is left or someone closes it.
Session note (agent, not an owner decision). Eval compare result: not measured. The implementation implement cases cannot show whether #16086 kept behavior or whether cases 45 and 46 pass, so this issue still owes a measurable compare.
What ran (claude plugin eval, model opus, judge opus, 3 runs per case):
Base 6fa15a3 and head e4055f4 (feat(implementation): read the consumer's own language idioms in implement's type decisions #16086's merge commit), each converted from skills/implement/evals/evals.json with the evals plugin's convert-evals-json.py and trimmed to the same 7 cases: implement-bugfix-mode-incremental-cadence, implement-contract-change-runs-propagation-sweep, implement-feature-fifth-branch-names-domain-structure, implement-no-suppression-to-pass-lint, implement-refactor-subtraction-first-commit, and the two new cases implement-type-decision-cites-consumer-idioms-file (45) and implement-type-decision-no-idioms-declared-proceeds-on-neutral-rules (46). Cases 45 and 46 also ran on base as the detection control.
Both batches finished (partial: false, no run errors), and both are verdict: INVALID under run-validity.py: permission denials of Glob/Grep calls aimed outside the run's working directory (case 46 on base; cases 45 and 46 on head).
The underlying fault: every run in both versions and both arms stopped before the task ("the working directory is empty"). These are narration: true cases. The converted suite gives them no fixtures and only Read, Glob, Grep and Skill, so a run starts in an empty sandbox and cannot make the edits the expectations grade. No case is comparable, so this record quotes no score, and the non-inferiority margin (0.05) and the detection control are undecided.
Sonnet was not run: it would have repeated the same empty-workspace stop.
Caveats that would apply even to a valid run: three runs per side is a small sample, and the judges are uncalibrated (the converted samples are unlabeled expected_output text).
What is left:
Make cases 45 and 46, and the regression cases chosen for the compare, runnable under claude plugin eval. Give each the repository its prompt describes (context.add_dirs or a scaffold_script holding tsconfig.json, src/webhooks/carrier.ts, docs/standards/README.md with the typescript row and typescript.md for 45; pyproject.toml, worker.py and an empty standards setup for 46), and grant Write and Edit, or rewrite the code-change expectations to grade the narrated plan.
Rerun the base versus head compare on opus and sonnet, 3 runs per side, margin 0.05, with 45 and 46 on base as the detection control.
Session note (agent, not an owner decision). Eval compare result for #16086, replacing the "not measured" record above.
Result
Model
Cases compared
Head minus base
95% interval
Verdict at margin 0.05
opus
7
+0.01
-0.03 to +0.05
non-inferior
sonnet
6 (bugfix case excluded, see below)
+0.04
-0.10 to +0.17
not shown non-inferior
So the done-when's compare is met on opus and not met as written on sonnet. Nothing here shows a regression: the sonnet point estimate favours head, and its interval is wide at three runs per side.
What ran (claude plugin eval, Claude Code 2.1.326, judge opus, 3 runs per case per version, with-arm only, --scaffold, --allow-tools Write Edit Bash)
Base 6fa15a3 and head e4055f4, the same 7 cases as the earlier attempt. Each case now has a scaffold script that writes the repository its prompt describes (files, a passing test suite, a git repository on the named branch), and sets max_turns: 60 and timeout_seconds: 900. The earlier attempt's fault (every run stopped at an empty working directory) is gone: every counted run edited files, ran tests and committed.
A one-case pilot (case 45, head) passed the validity gate before any batch.
opus: base verdict: VALID. Head had one run end with an authentication error ("OAuth token revoked"), so that case (implement-feature-fifth-branch-names-domain-structure) was rerun alone, 3 runs, and replaces the original case in the head result; the merged result is verdict: VALID.
sonnet: both versions are verdict: INVALID for the full suite, for one reason: in implement-bugfix-mode-incremental-cadence, the only case not started with /implementation:implement, sonnet did not invoke the skill in 6 of 6 runs (3 per version). That case is not comparable on sonnet in either version. With it removed, both versions are verdict: VALID, and the table's sonnet row is those 6 cases.
The one sonnet case below base is the refactor case, and the gap is one run (head 0.5, 0.5, 0; base 0.5, 0.5, 0.75).
Detection control: not shown by judged score, shown by the traces
By judged score, base (which lacks the idioms rule) scores about the same as head on cases 45 and 46, so the judged compare could not have detected the loss of the rule. Two causes, read in the traces: an llm judge on focus: trace sees only the first 12 and last 12 messages, and these runs are 70 to 365 messages long; and both prompts state where the idioms file is, or that there is none.
Read directly from the traces (opus, the compare's 6 runs of cases 45 and 46 plus 6 more runs of those two cases per version): head read reference/ecosystem-resolution.md in 12 of 12 runs; base in 0 of 12. On case 45 a deterministic tool_used grader for that read passed 3 of 3 on head and 0 of 3 on base, with the validity gate's other checks passing for those runs. The 6 extra runs of case 46 are verdict: INVALID in both versions (the agent's Grep or Glob aimed at the plugin directory was denied), so they give no score; the counts above are trace reads only.
Case 46's report line is unreliable on head: the run printed a none declared line in 3 of 6 opus runs (base: 1 of 6; the first three runs per version were searched for none declared, the other three for none declared for Python). The second done-when item ("the run names the missing idioms source and proceeds") holds in about half the runs measured.
Caveats
Three runs per side is a small sample. The judges are uncalibrated: the converted samples are unlabeled, so no judge grader passed calibration.
Low scores on the contract, fifth-branch and refactor cases are the same in both versions; part of that is the judge's trace window, not the skill.
The scaffolds and the changed case files exist only in a scratch copy; nothing was committed.
What is left
Owner decision: accept the sonnet half as "not shown at three runs" or rerun sonnet with more runs per side.
The none declared report appears in about half the opus runs of case 46. Either the skill text needs to make the report unmissable, or the item is accepted as is.
If these cases are to be rerun later, cases 45 and 46 need deterministic graders (the read of reference/ecosystem-resolution.md through any tool, the report line) and prompts that do not state the answer; the judge-on-trace graders cannot see the middle of a long run.
List-price estimate for everything run in this session: about 352 USD, of which about 99 USD was two opus batches discarded because the default 10-turn limit stopped runs.
Activity
github-actions commentedon Nov 13, 2026
Size: Small
Complexity: Low
Uncertainty: Knowable
Flags: none
Blocked by: none
I understand this as:
/implementation:implementshould read a consumer-declared idioms file for the detected language. With no declaration, it should say so and use the neutral rules. Pull request #16086 merged withRefs:, so the issue stayed open. The implement skill and its evals already mention idioms, so part of this may be done. Nobody has yet checked the "Done when" items against what #16086 left: the skill wording, the two eval cases, the two-version compare record, and the dogfooding note. Please confirm which items remain, or close the issue if all are met.Automated triage. A maintainer reviews it.
melodic-automation-lanes commentedon Nov 13, 2026
Not admitted to the issue lane: merged pull request #16086 names it with
Refs:after its newest decision or refinement comment, so its work may already be done: close the issue if it is, or post a comment whose first line is**Decision recorded.**saying what is left, which sends it back through intake. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).melodic-automation-lanes commentedon Nov 14, 2026
{"v":1,"actor_id":337595936,"title_sha256":"5866b93483c856b5a3b10dcbd50ffd9f0e09f54ced5a6b67166f4ec477e257b4","body_sha256":"88e8af13fb5cac1f7ebb3f9da6997688bdaa7acdbaec61a4c232f9909a4fbb85"}
github-actions commentedon Nov 14, 2026
melodic-automation-lanes commentedon Nov 14, 2026
Pull request #16086 merged with
Refs:, so this issue stays open and nothing starts the lane on it again. work-unblock removedstatus: readyandagent-ready, so the hourly triage sweep sends it back to intake triage. Admission then parks it atstatus: confirmedfor a check of what that pull request left, until a**Decision recorded.**comment says what is left or someone closes it.github-actions commentedon Nov 14, 2026
The triage sweep dispatched intake triage for this issue (dispatch 1; 2 runs that reach intake, or 6 dispatches, hand it to a maintainer).
kyle-sexton commentedon Nov 16, 2026
Session note (agent, not an owner decision). Eval compare result: not measured. The
implementationimplement cases cannot show whether #16086 kept behavior or whether cases 45 and 46 pass, so this issue still owes a measurable compare.What ran (
claude plugin eval, model opus, judge opus, 3 runs per case):skills/implement/evals/evals.jsonwith the evals plugin'sconvert-evals-json.pyand trimmed to the same 7 cases: implement-bugfix-mode-incremental-cadence, implement-contract-change-runs-propagation-sweep, implement-feature-fifth-branch-names-domain-structure, implement-no-suppression-to-pass-lint, implement-refactor-subtraction-first-commit, and the two new cases implement-type-decision-cites-consumer-idioms-file (45) and implement-type-decision-no-idioms-declared-proceeds-on-neutral-rules (46). Cases 45 and 46 also ran on base as the detection control.partial: false, no run errors), and both areverdict: INVALIDunderrun-validity.py: permission denials of Glob/Grep calls aimed outside the run's working directory (case 46 on base; cases 45 and 46 on head).narration: truecases. The converted suite gives them no fixtures and only Read, Glob, Grep and Skill, so a run starts in an empty sandbox and cannot make the edits the expectations grade. No case is comparable, so this record quotes no score, and the non-inferiority margin (0.05) and the detection control are undecided.Caveats that would apply even to a valid run: three runs per side is a small sample, and the judges are uncalibrated (the converted samples are unlabeled
expected_outputtext).What is left:
claude plugin eval. Give each the repository its prompt describes (context.add_dirsor ascaffold_scriptholding tsconfig.json,src/webhooks/carrier.ts,docs/standards/README.mdwith the typescript row andtypescript.mdfor 45;pyproject.toml,worker.pyand an empty standards setup for 46), and grant Write and Edit, or rewrite the code-change expectations to grade the narrated plan.kyle-sexton commentedon Nov 17, 2026
Session note (agent, not an owner decision). Eval compare result for #16086, replacing the "not measured" record above.
Result
So the done-when's compare is met on opus and not met as written on sonnet. Nothing here shows a regression: the sonnet point estimate favours head, and its interval is wide at three runs per side.
What ran (
claude plugin eval, Claude Code 2.1.326, judge opus, 3 runs per case per version, with-arm only,--scaffold,--allow-tools Write Edit Bash)max_turns: 60andtimeout_seconds: 900. The earlier attempt's fault (every run stopped at an empty working directory) is gone: every counted run edited files, ran tests and committed.verdict: VALID. Head had one run end with an authentication error ("OAuth token revoked"), so that case (implement-feature-fifth-branch-names-domain-structure) was rerun alone, 3 runs, and replaces the original case in the head result; the merged result isverdict: VALID.verdict: INVALIDfor the full suite, for one reason: in implement-bugfix-mode-incremental-cadence, the only case not started with/implementation:implement, sonnet did not invoke the skill in 6 of 6 runs (3 per version). That case is not comparable on sonnet in either version. With it removed, both versions areverdict: VALID, and the table's sonnet row is those 6 cases.Per case, mean of 3 runs (head / base)
The one sonnet case below base is the refactor case, and the gap is one run (head 0.5, 0.5, 0; base 0.5, 0.5, 0.75).
Detection control: not shown by judged score, shown by the traces
llmjudge onfocus: tracesees only the first 12 and last 12 messages, and these runs are 70 to 365 messages long; and both prompts state where the idioms file is, or that there is none.reference/ecosystem-resolution.mdin 12 of 12 runs; base in 0 of 12. On case 45 a deterministictool_usedgrader for that read passed 3 of 3 on head and 0 of 3 on base, with the validity gate's other checks passing for those runs. The 6 extra runs of case 46 areverdict: INVALIDin both versions (the agent's Grep or Glob aimed at the plugin directory was denied), so they give no score; the counts above are trace reads only.none declaredline in 3 of 6 opus runs (base: 1 of 6; the first three runs per version were searched fornone declared, the other three fornone declared for Python). The second done-when item ("the run names the missing idioms source and proceeds") holds in about half the runs measured.Caveats
What is left
none declaredreport appears in about half the opus runs of case 46. Either the skill text needs to make the report unmissable, or the item is accepted as is.reference/ecosystem-resolution.mdthrough any tool, the report line) and prompts that do not state the answer; the judge-on-trace graders cannot see the middle of a long run.List-price estimate for everything run in this session: about 352 USD, of which about 99 USD was two opus batches discarded because the default 10-turn limit stopped runs.
kyle-sexton commentedon Nov 17, 2026
Decision recorded. Owner, 2026-11-17, in session ("Go with recommendations."), posted by an agent session:
none declaredreport appearing in about half the measured runs is a follow-up: Make implementation:implement always report when no language idioms file is declared #16703.With the compare record posted and these two decided, nothing is left on this issue.