Skip to content

feat(tools): per-tool descriptor and skill-use tracking for every tool (#1221) - #1245

Merged
jeff-r2026 merged 2 commits into
Tencent:mainfrom
SaulMoro:feat/1221-tool-descriptor
Oct 21, 2026
Merged

jeff-r2026 merged 2 commits into
Tencent:mainfrom
SaulMoro:feat/1221-tool-descriptor

Conversation

@SaulMoro

@SaulMoro SaulMoro commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Step A1 of #1221: one typed descriptor per tool, and skill usage recorded the same way for every tool.

flowchart LR
    subgraph callers ["callers"]
        hooks["hook-dispatch<br/>post-tool-use '*', prompt-submit"]:::added
        init["init<br/>home detection"]:::added
        pull["pull, uninstall, doctor,<br/>models switch, local agent"]
        test["matrix test"]:::todo
    end
    subgraph core ["src/tools.ts"]
        desc["TOOLS<br/>home, skillLoad"]:::added
        rest["channels, support"]:::todo
        locate["locate(tool, resource, install, host)"]:::todo
    end
    use["skill-use<br/>skill tool, SKILL.md read, mention<br/>installed only, once per turn"]:::added
    host{{"Host"}}:::todo
    page["docs/guide/tool-support.md"]:::todo
    old["per-tool Skill hook,<br/>SKILL_TOOL_MATCHER"]:::removed
    res["per-tool resolvers"]
    hooks --> use --> desc
    init --> desc
    pull & test -.-> locate
    locate -.-> desc
    locate -.-> rest & host
    rest -.-> page
    pull --> res
    hooks -.-> old
    classDef added fill:#2f8f4f,stroke:#1f6b39,color:#fff;
    classDef todo fill:#5c6670,stroke:#3d454c,color:#fff,stroke-dasharray:4 3;
    classDef removed fill:#a33a3a,stroke:#7a2a2a,color:#fff,stroke-dasharray:4 3;
Loading

Green: this PR. Gray: later steps of #1221 (A2 Host and locate, then B1-B4, C, E, D). Red: removed.

  • A skill counts when the agent calls its skill tool, reads an installed SKILL.md (shell reads included), or the prompt names it: /name, Codex $name, Pi /skill:name. Once per turn, installed skills only.
  • Fixes Copilot, Pi, OMP, dsh, OpenCode 2.x and Codex, which record no skill usage today.
  • OpenCode's plugin sends raw tool names; Skill is still accepted for one release.
  • A team that disabled Hook dispatch post-tool-use Skill keeps tracking off.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature causing existing behavior to change)
  • Documentation only
  • Refactor / internal cleanup

Evidence

Before: on main, the Codex, Pi, Copilot, OMP, dsh and OpenCode 2.x payloads captured for #1221 record no skill use.
After: the built CLI at 90c1b356, isolated HOME, user scope, teamai hook-dispatch ... --stdin:

codex   "$hello-skill ..." + cat SKILL.md x2, same turn   -> 1 (codex)
codex   cat ~/.codex/skills/hello-skill/SKILL.md, new turn -> 1 (codex)
claude  Skill call                                       -> 1 (claude)
claude  Skill call, skill not installed                  -> 0

Earlier payloads from real runs (Pi /skill:x, stale --matcher Skill hook, invalid team hook or builtin block, retired key, hooks.yaml syntax error) are covered by skill-use.test.ts and hooks-skill-retire.test.ts.

Test Plan

  • npx tsc --noEmit passes
  • npm run lint passes
  • npx vitest run passes (11459 passed, 21 skipped)
  • Added/updated tests for the change: skill-use.test.ts, hooks-skill-retire.test.ts, tools.test.ts
  • npm run build, then the full npm run test:e2e: 1394 passed, 27 skipped, 2 failed. The 2 are the typed-terminal Ctrl-C cases in workflow-signals.test.ts; they fail the same way on main (c6bd1264) on this machine, and pass in main's CI.
  • Not run here: real Copilot, CodeBuddy, Muse, dsh and Pi hosts, Windows, the gitlab and github providers.

Merge Danger

Door: two-way. Reverting restores the Skill hook on the next pull.

Blast Radius: usage counts.

  • Counts change on purpose: once per turn, installed skills only, /name only at the start.
  • Codex hooks.json trust keys are positional, so removing the Skill entry re-trusts the later entries on pull.
  • Known gaps: hosts with no turn id (Pi, OMP, OpenCode) can lose a credit when a detached hook lands after the next prompt; Cursor may count once per conversation.

Related Issues

Part of #1221.

@github-actions

Copy link
Copy Markdown
  • [P1 blocking] Prompt deduplication drops skill-use webhooks — src/hook-handlers.ts:374: A /tdd prompt credits the installed skill without sending a webhook. The subsequent Skill call in the same turn returns no new credit, so this loop sends nothing either. Existing webhook users therefore lose notifications for slash-invoked skills. Send the webhook when the prompt wins the credit, or deduplicate webhook delivery separately.

  • [P2 non-blocking] Quoted HOME paths are not tracked — src/utils/shell-command.ts:131: Expansion runs only outside quotes. A successful cat "$HOME/.codex/skills/hello-skill/SKILL.md" leaves $HOME unresolved and records no skill use, unlike the equivalent unquoted command. Handle $HOME and ${HOME} inside double quotes while preserving single-quoted literals.

  • [P2 non-blocking] Name-only mentions accept another tool’s installations — src/skill-use.ts:131: The existence gate searches every tool’s roots without receiving the calling tool. Consequently, Codex credits $pi-only when that skill exists solely under ~/.pi/agent/skills, contradicting the documented “installed for the tool” restriction. Restrict name-only candidates to the caller’s discoverable roots.

The PR description supplies sufficient built-CLI/e2e evidence. No testing-record finding; nothing from the PR was run, built, or installed.

@SaulMoro
SaulMoro force-pushed the feat/1221-tool-descriptor branch from 7d9c987 to 5832f54 Compare October 20, 2026 21:18
@SaulMoro
SaulMoro marked this pull request as ready for review October 20, 2026 22:14
@SaulMoro
SaulMoro marked this pull request as draft October 20, 2026 22:14
@github-actions

Copy link
Copy Markdown

No new findings in the scoped diff.

Earlier findings are resolved:

  • Prompt deduplication now claims webhook delivery separately from usage credit.
  • $HOME and ${HOME} expand inside double quotes while single-quoted literals remain unchanged.
  • Name-only candidates now use the calling tool’s skill roots.

The PR description documents sufficient built-CLI/e2e verification. No testing-record finding; deferred provider/agent coverage is not blocking.

Nothing from the PR was run, built, or installed.

…nce (Tencent#1221)

TOOLS records each built-in tool's home directory and how its agent loads
a skill (dedicated skill tool, prompt mention, or 'unknown'), with an
evidence class per value. ToolId is derived from DEFAULT_TOOL_PATHS, so a
new built-in tool fails tsc until it has a descriptor.

KNOWN_AGENTS user skills paths for pi, omp and opencode now match their
user-scope defaults; home detection and the init picker read home.dir so
Pi detection does not move. Tests pin the tool lists to ToolId, every
user-scope path under its home, and skillLoad to the tools that receive
the wildcard post-tool-use hook.
Tencent#1221)

One background skill-use handler on the wildcard post-tool-use pass credits
a skill when the call is the tool's skill tool (TOOLS[tool].skillLoad) or a
read of an installed skill's SKILL.md (classifyToolCall, shell reads with
~/ and $HOME/ included). Prompt-submit credits /name at the start, Codex
$name, and the block Pi expands /skill:name into. Each skill counts once per
session turn and only when installed under a root skill delivery uses.

Removes the per-tool Skill hook, SKILL_TOOL_MATCHER, and the OpenCode
matcher passes; OpenCode sends raw tool names, with a one-release Skill
alias. A team that disabled the retired built-in key keeps tool-call
tracking off.
@SaulMoro
SaulMoro force-pushed the feat/1221-tool-descriptor branch from 5832f54 to 90c1b35 Compare October 21, 2026 03:13
@SaulMoro SaulMoro changed the title [After #1204] feat(tools): per-tool descriptor and skill-use tracking for every tool (#1221) feat(tools): per-tool descriptor and skill-use tracking for every tool (#1221) Oct 21, 2026
@SaulMoro
SaulMoro marked this pull request as ready for review October 21, 2026 03:15
@github-actions

Copy link
Copy Markdown

No new findings in the scoped diff.

Earlier findings are resolved:

  • Webhook delivery is claimed separately from usage credit.
  • $HOME and ${HOME} expand inside double quotes; single-quoted literals remain unchanged.
  • Name-only candidates use the calling tool’s skill roots.

The PR description documents sufficient built-CLI/e2e verification. Deferred provider/agent coverage is not blocking.

Nothing from the PR was run, built, or installed.

@jeff-r2026
jeff-r2026 merged commit 22dfe27 into Tencent:main Oct 21, 2026
16 checks passed
@SaulMoro SaulMoro changed the title feat(tools): per-tool descriptor and skill-use tracking for every tool (#1221) feat(tools): tool descriptor, skill-use tracking and locate for skills (#1221) Oct 21, 2026
@SaulMoro SaulMoro changed the title feat(tools): tool descriptor, skill-use tracking and locate for skills (#1221) feat(tools): per-tool descriptor and skill-use tracking for every tool (#1221) Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants