Skip to content

Sync the trigger surface, split the SDK reference, and guard both - #12

Merged
kev1n merged 1 commit into
mainfrom
sync-latency-preference
Aug 31, 2026
Merged

kev1n merged 1 commit into
mainfrom
sync-latency-preference

Conversation

@kev1n

@kev1n kev1n commented Aug 31, 2026

Copy link
Copy Markdown
Member

main is what npx skills add getanyapi-com/skills installs, and it has been serving a
description that does not trigger. Measured on held-out prompts, one fresh session each,
counting only a real Skill invocation: where the data need is implied rather than naming a
platform, the old description fired on 1 of 5, the new one on 4 of 5, with 0/4
false positives on should-not-trigger prompts (a refactor, a general-knowledge question, an
OAuth explanation, summarizing a local file).

Explicit prompts that name a platform fire 5/5 on both, which is why this was invisible: any
spot check with "get the last 20 instagram posts from @nasa" says the skill is fine.

Why it drifted

The body gate compared bodies against the live site and stayed green for seven weeks. An agent
loads only name and description (plus when_to_use) before deciding whether to read the
body, so the one field that decides whether the skill ever runs was the one field nothing
compared. scripts/sync-skills.ts upstream generated this frontmatter from a second hardcoded
literal; both surfaces now derive from the same constants.

Changes

  • New description and when_to_use, generated upstream, plus license, argument-hint, and
    allowed-tools. allowed-tools covers only free commands - anyapi run is excluded so the
    first call that charges the wallet still prompts.
  • SDK and raw-REST sections move to references/sdks.md, loaded only once an agent decides to
    integrate. Activation-time body drops from roughly 6,300 to 5,400 tokens against the Agent
    Skills spec's <5,000 recommendation.
  • check_skills.py now compares description and when_to_use against the live skill and
    checks references/sdks.md, so neither the trigger text nor the new reference can rot behind
    a green body check.
  • Plugin and marketplace versions bumped to 0.3.0 together, which the manifest guard requires.

Upstream shipped first (getanyapi-com/anyapi#918, verdict SHIPPED), so this gate now passes
against the live site: skills checks passed, and claude plugin validate . passes.

main is what npx skills add installs, and it has been serving a description that
does not trigger. Measured on held-out prompts, one fresh session each, counting
only a real Skill invocation: where the data need is implied rather than naming a
platform, the old description fired on 1 of 5 and the new one on 4 of 5, with 0
of 4 false positives on should-not-trigger prompts. Prompts that name a platform
fire 5/5 on both, which is why this was invisible to a spot check.

The body gate compared bodies against the live site and stayed green for seven
weeks. An agent loads only name and description (plus when_to_use) before
deciding whether to read the body, so the field that decides whether the skill
runs at all was the one field nothing compared.

check_skills.py now compares description and when_to_use against the live skill
and checks references/sdks.md, so neither the trigger text nor the new reference
can rot behind a green body check.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@kev1n
kev1n force-pushed the sync-latency-preference branch from 3866eb8 to 57c768f Compare August 31, 2026 08:19
@kev1n
kev1n merged commit 8d39beb into main Aug 31, 2026
1 check passed
@kev1n
kev1n deleted the sync-latency-preference branch August 31, 2026 08:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant