Evaluation test set authoring path is broken end-to-end: YAML kinds silently dropped by sync, sub-agent emits wrong CSV format, MS Learn docs not consulted

[Dewain27]

Summary

Asking the copilot-studio:test sub-agent to "create evaluation test sets" for an agent produces output that does not work through any documented path:

1. The five *.eval.mcs.yml files the sub-agent generates (with kind: EvaluationSet / EvaluationData) validate cleanly but are silently dropped by push — getLocalChanges returns [] and the files never reach the cloud. No warning, no error.
2. The bulk-import test-set-import.csv the sub-agent also generates is in Kit/Dataverse format, not the Copilot Studio portal Evaluate-tab format. Upload to the portal's Evaluate → New evaluation page fails with "Something went wrong while uploading your file" (Single response) and "This file uses the wrong template" (Conversation preview).
3. The actual portal CSV template is documented on MS Learn (Question, Expected response, Testing method) but the sub-agent did not consult those docs before generating output. It guessed column headers (Input, Expected response, Expected keywords, Expected topic), which the portal rejects. A quick web search of MS Learn would have produced the correct schema in the first attempt.

End-to-end, this means a user who asks for evaluation test sets gets ~25 minutes of work product that fails on every supported import surface, with no actionable error from the tooling itself — the only useful error came from the Copilot Studio portal UI.

Reproduction

1. Clone any agent via the copilot-studio:manage sub-agent.

2. Invoke the copilot-studio:test sub-agent and ask for evaluation test sets: "Create evaluation test sets for this agent".

   The sub-agent generates:

   • evaluations/graders.eval.mcs.yml (kind: EvaluationSet)
   • evaluations/knowledge-grounding.eval.mcs.yml (kind: EvaluationData)
   • evaluations/topic-trigger-joinemaillist.eval.mcs.yml (kind: EvaluationData)
   • evaluations/out-of-scope.eval.mcs.yml (kind: EvaluationData)
   • evaluations/negative-triggers.eval.mcs.yml (kind: EvaluationData)
   • evaluations/test-set-import.csv (Kit/Dataverse columns)

3. Validate the YAML — all pass (e.g., "7 passed, 0 warnings, 0 failures").

4. Check for local changes via manage-agent changes — localChanges is []. Files silently ignored.

5. Push via manage-agent push — HTTP 200, no errors, but nothing about evals was pushed.

6. Upload test-set-import.csv to the Copilot Studio portal:

   • Evaluation → New evaluation → Single response → upload
     → "Something went wrong while uploading your file. Please try again."
   • Evaluation → New evaluation → Conversation (preview) → upload
     → "This file uses the wrong template. Make sure you're using the right CSV template for the data type you selected."

Expected vs. actual

Issue 1: EvaluationSet / EvaluationData sync gap

Step	Expected	Actual
validate	Pass	Pass
manage-agent changes	Flag the files as either syncable or explicitly-unsupported	localChanges: [] — silently ignored
manage-agent push	Either push them OR refuse with a clear message	Reports success, pushes nothing eval-related
Portal Evaluate tab	Show the new test set	Nothing appears

The bot.schema.yaml-authoring.json schema accepts these kinds (they are real definitions with SimpleEvaluationCase, MultiTurnEvaluationCase, ExpectedTopicTriggeredStep, grader configs, etc.). However, the LSP binary's getLocalChanges classifier only recognizes DialogComponent, GptComponent, and KnowledgeSourceComponent as component-bearing kinds — EvaluationSet / EvaluationData files are not in that classifier. Net result: schema-valid + LSP-invisible = silent drop.

Issue 2: Sub-agent generates Kit-format CSV when user wants portal Evaluate

The copilot-studio:test sub-agent assumed the Power CAT Copilot Studio Kit Dataverse import was the canonical path and generated a CSV with Kit-specific columns. The much more discoverable Copilot Studio portal Evaluation tab is a different surface with a different CSV schema, and the sub-agent did not generate output for it.

The sub-agent's README even claimed "Built-in Copilot Studio Evaluations consumes the *.eval.mcs.yml files" — there is no such pipeline today; the Evaluate tab only ingests its own CSV format (or in-portal manual entry).

Issue 3: Sub-agent should consult MS Learn docs before guessing schemas

When asked to regenerate the CSV in the correct portal format, the sub-agent's first attempt guessed column headers based on YAML property names (Input, Expected response, Expected keywords, Expected topic). The portal rejected both files.

The actual format is fully documented at:

• https://learn.microsoft.com/en-us/microsoft-copilot-studio/analytics-agent-evaluation-create
• https://learn.microsoft.com/en-us/microsoft-copilot-studio/analytics-agent-evaluation-multi-turn
• https://learn.microsoft.com/en-us/microsoft-copilot-studio/analytics-agent-evaluation-overview

Correct schema (Single response):

Question,Expected response,Testing method

Where Testing method is one of: General quality, Compare meaning, Similarity, Exact match, Keyword match.

Correct schema (Conversation preview): none — there is no CSV upload path for multi-turn. The portal only offers Quick conversation set, Full conversation set, and Use your test chat. The sub-agent generated evaluate-conversation.csv with invented columns; the portal correctly rejected it because no such template exists.

After explicit instruction to use firecrawl-search / web search on MS Learn, the sub-agent found the documented schema on the first try. So the docs are there — the sub-agent just did not reach for them by default.

Suggested fixes

In order of impact:

1. copilot-studio:test sub-agent (highest impact):

   • Default to generating Copilot Studio portal Evaluate-tab CSV (Question, Expected response, Testing method) for single-response cases, since that is the most-used surface.
   • Do not generate *.eval.mcs.yml files until LSP sync supports them (or at least add a banner in the README that they are not yet syncable).
   • For multi-turn cases, generate instructions for the user to seed conversations via Test pane → Use your test chat, not invented CSV files.
   • Add a guardrail: before emitting any new schema as a primary deliverable, consult MS Learn or the firecrawl-search skill. Schema-validity and runtime-import-support are independent.

2. copilot-studio:manage / LSP getLocalChanges (medium impact):

   • Surface a warning when scanning encounters a kind: that is schema-valid but not in the syncable-components list. Silent drops are worse than explicit "not yet supported" warnings.
   • Publish the canonical list of LSP-synced component classifications in the plugin README so authors do not have to discover by trial.

3. copilot-studio:validate / lookup-schema (lower impact, but good UX):

   • When a user authors a kind: that is schema-valid but not LSP-supported, the validator could include an informational note: "Note: this kind is defined in the schema but is not currently picked up by push. To get this content into your environment, see [link to docs]."

4. README / docs:

   • Add a "What gets pushed by the sync surface" section listing the supported component classifications.
   • Add a separate "Evaluations" section pointing to the MS Learn docs for the portal Evaluate tab and the Power CAT Kit for batch runs — and clarifying that local YAML evaluations are not synced today.

Environment

• Plugin: skills-for-copilot-studio v1.0.4 (token cache at ~/.claude/plugins/cache/skills-for-copilot-studio/copilot-studio/1.0.4/)
• Affected sub-agents: copilot-studio:test (generates wrong-format files), copilot-studio:manage (silent drop), copilot-studio:author (lookup-schema reports the eval kinds as valid)
• Tested on: Contoso (default) environment, agent cref7_PluginDemoAgent
• Repro confirmed via portal upload error messages

Workaround (for users who hit this in the meantime)

1. Ignore the *.eval.mcs.yml files the sub-agent generates — they will not push to the cloud.
2. Hand-author a CSV with these exact columns: Question,Expected response,Testing method.
3. Upload via Copilot Studio portal → Evaluation → New evaluation → Single response (NOT Conversation preview, which has no upload path).
4. For multi-turn: use the Test pane in the portal to run conversations manually, then New evaluation → Conversation (preview) → Use your test chat.

Impact

Medium. Doesn't block agent authoring itself, but the evaluation surface is exactly the thing teams reach for when productionizing an agent — wasted authoring effort + silent failures + repeated portal rejections is a rough first impression of the evaluation tooling.

[GiorgioUghini]

It seems to use the right skill and produce the correct CSV...

[GiorgioUghini]

Actually, I'm noticing you're not using the latest version (v1.0.11) but you are instead of v.1.0.4.
Can you please try to update to latest version and retry?

[deldos]

Adding an independent CI/CD use case and one distinction that may help scope this issue.

There appear to be two separate concerns:

1. Generating the correct evaluation artifacts, which recent plugin updates may address.

2. Headless creation/import of the actual Copilot Studio Maker Evaluation test set, which still appears to be a platform/API gap.

We independently validated that an existing Maker Evaluation test set can be handled headlessly through the documented Power Platform API: discover the test set, start an evaluation run, poll it, and retrieve the results. This makes API-driven regression execution practical.

The remaining blocker is the beginning of that lifecycle. I cannot find a documented API operation for creating/importing or updating a Maker Evaluation test set and its test cases. A new test corpus therefore still has to be seeded through the Copilot Studio authoring experience, even when subsequent execution is automated.

For a repository-driven CI/CD workflow, the desired lifecycle would be:

version-controlled test corpus -> create/update test set -> run -> poll -> retrieve results -> map results back to source cases

with no portal dependency.

Useful acceptance criteria would be:

• create/import a test set through a supported API or CLI;
• update or replace its test cases deterministically;
• support the same core fields available through the supported CSV/import experience;
• expose stable test-case identity or enough case content to map API results back to the source corpus;
• support idempotent execution from CI;
• provide an explicit unsupported/error response rather than silently ignoring evaluation artifacts.

This is independent of the older plugin-version/CSV-generation problem described above: even with a perfectly generated CSV, the test-set provisioning step remains non-headless.

Closing that final gap would allow Copilot Studio evaluation to operate as a normal regression gate rather than a hybrid portal/API workflow.