Add an agent-skill eval scenario for the skill builder

[Karim Mehalebi (karimad)]

Summary

#47 added the agent-skill (generic) architecture but only covered it with structural/manifest tests (architecture-registry.test.ts, catalogue-registry.test.ts) — nothing exercises the real builder LLM against the new catalogue the way the existing skillbuilder eval suite does for scout/cowork.

Adds two scenarios to evals/skillbuilder/scenarios.ts:

• github-issue-triage-agent-skill — a clone of the existing github-issue-triage-skill scenario, retargeted to architecture: "agent-skill". Requires the gh CLI (a universal primitive) while forbidding any vendor-specific tool name (workiq_*, m365_*, named browser-tool suites).
• teams-digest-agent-skill — a clone of cowork-teams-digest, retargeted to agent-skill. A harder case than the gh-CLI scenario, since there's no CLI at all for Teams. Confirms the builder doesn't invent a plausible-sounding native tool name when none exists in the catalogue — it correctly falls back to documented Microsoft Graph HTTP calls via curl instead.

Test plan

• npm run typecheck — clean
• npm run eval:skill -- --only=github-issue-triage-agent-skill — 100% pass against the real Copilot CLI builder; generalizes to gh CLI shell steps with zero product-specific tool references
• npm run eval:skill -- --only=teams-digest-agent-skill — 100% pass; generalizes to Microsoft Graph HTTP calls (curl) rather than inventing a native Teams tool
• Full npm run eval:skill suite — no regressions (pre-existing scout/cowork scenarios flaked independent of this change and passed on rerun, consistent with known LLM run-to-run variance)

Related: #19, #47

[adilei]

Thanks for adding this. I’m closing the PR because these scenarios evaluate only the builder’s proposed plan inside Copilot CLI, rather than evaluating the generated skill within the target architecture itself.

For an agent-skill eval to be meaningful, it should generate and export the skill, load it into the target architecture, execute the task there, and score the resulting behavior. Checking that the builder mentions gh, curl, or another tool does not establish that the target can load the skill, provide the required capabilities and authentication, or complete the task successfully.

We’re open to other ideas for how to evaluate this well, including proposals for representative target architectures or an interoperability/conformance approach.

[Karim Mehalebi (karimad)]

Thanks for adding this. I’m closing the PR because these scenarios evaluate only the builder’s proposed plan inside Copilot CLI, rather than evaluating the generated skill within the target architecture itself.

For an agent-skill eval to be meaningful, it should generate and export the skill, load it into the target architecture, execute the task there, and score the resulting behavior. Checking that the builder mentions gh, curl, or another tool does not establish that the target can load the skill, provide the required capabilities and authentication, or complete the task successfully.

We’re open to other ideas for how to evaluate this well, including proposals for representative target architectures or an interoperability/conformance approach.

Thanks for the feedback. I understand the distinction you’re making between evaluating the SkillBuilder output and evaluating the generated skill at runtime.

I noticed that the existing scenarios in evals/skillbuilder/scenarios.ts also focus on validating the builder’s proposed plan (tool selection, step structure, constraints, etc.) rather than executing the generated skill in a target architecture.

Would it make sense to separate these into two categories:

• builder-level evals: validating the generated skill specification/plan
• runtime/conformance evals: exporting the skill, loading it into a target architecture, executing the task, and scoring the result

I’d be interested in contributing toward the runtime/conformance direction if that aligns with where the project is heading.

issue created : #55

[Karim Mehalebi (karimad)]

Follow-up on this feedback: #70 adds the runtime-conformance layer you described — actually generates → exports → loads → executes a SKILL.md in a fresh Copilot session and scores the real resulting gh calls against a mocked issue set, rather than scoring plan text. Uses only what's already required for this eval suite (signed-in Copilot CLI, @github/copilot-sdk) — no new dependency or credential.