Sync the trigger surface, split the SDK reference, and guard both - #12
Merged
Merged
Conversation
main is what npx skills add installs, and it has been serving a description that does not trigger. Measured on held-out prompts, one fresh session each, counting only a real Skill invocation: where the data need is implied rather than naming a platform, the old description fired on 1 of 5 and the new one on 4 of 5, with 0 of 4 false positives on should-not-trigger prompts. Prompts that name a platform fire 5/5 on both, which is why this was invisible to a spot check. The body gate compared bodies against the live site and stayed green for seven weeks. An agent loads only name and description (plus when_to_use) before deciding whether to read the body, so the field that decides whether the skill runs at all was the one field nothing compared. check_skills.py now compares description and when_to_use against the live skill and checks references/sdks.md, so neither the trigger text nor the new reference can rot behind a green body check. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kev1n
force-pushed
the
sync-latency-preference
branch
from
August 31, 2026 08:19
3866eb8 to
57c768f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
mainis whatnpx skills add getanyapi-com/skillsinstalls, and it has been serving adescription that does not trigger. Measured on held-out prompts, one fresh session each,
counting only a real Skill invocation: where the data need is implied rather than naming a
platform, the old description fired on 1 of 5, the new one on 4 of 5, with 0/4
false positives on should-not-trigger prompts (a refactor, a general-knowledge question, an
OAuth explanation, summarizing a local file).
Explicit prompts that name a platform fire 5/5 on both, which is why this was invisible: any
spot check with "get the last 20 instagram posts from @nasa" says the skill is fine.
Why it drifted
The body gate compared bodies against the live site and stayed green for seven weeks. An agent
loads only
nameanddescription(pluswhen_to_use) before deciding whether to read thebody, so the one field that decides whether the skill ever runs was the one field nothing
compared.
scripts/sync-skills.tsupstream generated this frontmatter from a second hardcodedliteral; both surfaces now derive from the same constants.
Changes
descriptionandwhen_to_use, generated upstream, pluslicense,argument-hint, andallowed-tools.allowed-toolscovers only free commands -anyapi runis excluded so thefirst call that charges the wallet still prompts.
references/sdks.md, loaded only once an agent decides tointegrate. Activation-time body drops from roughly 6,300 to 5,400 tokens against the Agent
Skills spec's <5,000 recommendation.
check_skills.pynow comparesdescriptionandwhen_to_useagainst the live skill andchecks
references/sdks.md, so neither the trigger text nor the new reference can rot behinda green body check.
Upstream shipped first (getanyapi-com/anyapi#918, verdict SHIPPED), so this gate now passes
against the live site:
skills checks passed, andclaude plugin validate .passes.