Add Modern Agent support with skill filtering

.github/workflows/validate-skill-types.yml

@@ -0,0 +1,36 @@
+name: Validate skill agent-types
+
+on:
+  pull_request:
+    paths:
+      - 'skills/*/SKILL.md'
+
+jobs:
+  validate:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Check all skills have agent-types frontmatter
+        run: |
+          ERRORS=0
+          for dir in skills/*/; do
+            skill_name=$(basename "$dir")
+            skill_file="$dir/SKILL.md"
+            [ -f "$skill_file" ] || continue
+            case "$skill_name" in int-*) continue ;; esac
+
+            agent_types=$(head -20 "$skill_file" | grep "^agent-types:" | sed 's/agent-types: *//')
+            if [ -z "$agent_types" ]; then
+              echo "::error::Skill '$skill_name' is missing agent-types in SKILL.md frontmatter. Add 'agent-types: classic', 'agent-types: modern', or 'agent-types: both'."
+              ERRORS=1
+            elif [ "$agent_types" != "classic" ] && [ "$agent_types" != "modern" ] && [ "$agent_types" != "both" ]; then
+              echo "::error::Skill '$skill_name' has invalid agent-types: '$agent_types'. Must be 'classic', 'modern', or 'both'."
+              ERRORS=1
+            fi
+          done
+
+          if [ "$ERRORS" -eq 1 ]; then
+            exit 1
+          fi
+          echo "All skills have valid agent-types frontmatter."

agents/copilot-studio-author.md

@@ -1,8 +1,8 @@
 ---
 name: Copilot Studio Author
 description: >
-  [THIS IS A SUB-AGENT] Copilot Studio YAML authoring specialist. This sub-agent creates and edits topics, actions, knowledge sources, child agents, and global variables. Use when building or modifying Copilot Studio agent YAML files. Always use this in case there's overlap with a skill.
-  USE FOR: build Copilot Studio agent, create new agent, scaffold agent project, create topic, add knowledge source, add action, edit topic, create child agent, add global variable, new Copilot Studio bot, GPT agent, AI agent in Copilot Studio.
+  [THIS IS A SUB-AGENT] Copilot Studio YAML authoring specialist. This sub-agent creates and edits topics, actions, knowledge sources, skills, tools, child agents, and global variables. Supports both classic and modern agents. Use when building or modifying Copilot Studio agent YAML files. Always use this in case there's overlap with a skill.
+  USE FOR: build Copilot Studio agent, create new agent, scaffold agent project, create topic, add knowledge source, add action, edit topic, create child agent, add global variable, new Copilot Studio bot, GPT agent, AI agent in Copilot Studio, create skill, add tool, edit skill.
   DO NOT USE FOR: deploying agents (use manage), testing agents (use test), debugging YAML errors (use advisor).
   Always use this agent when the user wants to build or modify Copilot Studio agent YAML files, even if there's overlap with a skill.
 skills:
@@ -39,12 +39,25 @@ Then close with:
 
 Do **not** proceed with any authoring task until an `agent.mcs.yml` file exists.
 
+## CRITICAL: Detect agent type — classic vs modern
+
+After finding `agent.mcs.yml`, read `settings.mcs.yml` in the same directory to determine the agent type:
+
+- **Modern agent** if `template` contains `cliagent` or the recognizer is `CLICopilotRecognizer` or `CLIAgentRecognizer`
+- **Classic agent** otherwise
+
+This determines which skills you can use. **Using the wrong skill type will fail** — classic skills don't work on modern agents and vice versa. Use the correct dispatch table below.
+
 ## CRITICAL: Always use skills — never do things manually
 
 You MUST use the appropriate skill for every task. **NEVER** write or edit YAML files yourself when a skill exists for that task. Skills contain the correct templates, schema validation, and patterns — doing it manually risks hallucinated kinds, missing required fields, and broken YAML.
 
 **Before acting on any request**, check this list and invoke the matching skill:
 
+### Classic agent skills
+
+Use these when the agent type is **classic** (standard Generative Orchestration agent):
+
 | Task | Skill to invoke |
 |------|----------------|
 | Create a new topic | `/copilot-studio:new-topic` |
@@ -58,11 +71,37 @@ You MUST use the appropriate skill for every task. **NEVER** write or edit YAML
 | Edit agent settings or instructions | `/copilot-studio:edit-agent` |
 | Modify trigger phrases or model description | `/copilot-studio:edit-triggers` |
 | Add an adaptive card | `/copilot-studio:add-adaptive-card` |
+| List all topics in the agent | `/copilot-studio:list-topics` |
+
+### Modern agent skills
+
+Use these when the agent type is **modern** (agent with `cliagent-1.0.0` template):
+
+| Task | Skill to invoke |
+|------|----------------|
+| Create a new skill | `/copilot-studio:new-skill` |
+| Edit a skill | `/copilot-studio:edit-skill` |
+| Add a tool (connector, MCP, workflow) | `/copilot-studio:add-tool` |
+| Add a knowledge source | `/copilot-studio:add-knowledge-modern` |
+| List skills and tools | `/copilot-studio:list-skills` |
+| Edit agent settings, instructions, output | `/copilot-studio:edit-agent-modern` |
+
+Modern agents do NOT have topics, action nodes, triggers, adaptive cards, global variables, or generative answer nodes. If the user asks for any of these, explain that modern agents use a different model:
+- **Topics** → Skills (markdown instructions the orchestrator invokes)
+- **Connector actions** → Tools — use `/copilot-studio:add-tool` (guides through UI connection setup, then YAML editing)
+- **Trigger phrases** → Skill descriptions (the orchestrator routes based on description)
+- **Adaptive Cards** → Not available (output is text or structured JSON)
+- **Global variables** → Not supported at runtime yet
+- **Knowledge sources** → Use `/copilot-studio:add-knowledge-modern` (websites are YAML-authorable, others need UI)
+
+### Shared skills (both agent types)
+
+| Task | Skill to invoke |
+|------|----------------|
 | Reference a pattern's YAML structure | Read the pattern file from `int-patterns` |
 | Validate a YAML file | `/copilot-studio:validate` |
 | Look up a schema definition | `/copilot-studio:lookup-schema` |
 | List valid kind values | `/copilot-studio:list-kinds` |
-| List all topics in the agent | `/copilot-studio:list-topics` |
 
 Only if NO skill matches the task may you work manually — and even then, you MUST validate with `/copilot-studio:validate` afterward.
 

evals/evaluate.py

@@ -120,9 +120,12 @@ def run_cli(cli: str, prompt: str, cwd: Path, timeout: int = 600, plugin_dir: st
         cmd.extend(["--allowedTools", "Bash(node *) Read Write Glob Edit"])
         if plugin_dir:
             cmd.extend(["--plugin-dir", plugin_dir])
-        # Inject PreToolUse hook to trace skill invocations inside sub-agents
+        # Inject PreToolUse hooks:
+        # 1. Trace skill invocations inside sub-agents
+        # 2. Filter skills by agent type (blocks incompatible skills)
         # Use forward slashes for cross-platform compatibility in node command
         hook_path = str(HOOK_SCRIPT).replace("\\", "/")
+        filter_path = str(REPO_ROOT / "hooks" / "filter-skills.js").replace("\\", "/")
         hook_settings = json.dumps({
             "hooks": {
                 "PreToolUse": [{

evals/fixtures/modern-agent/agent.mcs.yml

@@ -0,0 +1 @@
+kind: GptComponentMetadata

evals/fixtures/modern-agent/knowledge/httpswwwcontosocom_abc1.mcs.yml

@@ -0,0 +1,6 @@
+mcs.metadata:
+  componentName: https://www.contoso.com
+kind: KnowledgeSourceConfiguration
+source:
+  kind: WebsiteKnowledgeSource
+  siteUrl: https://www.contoso.com

evals/fixtures/modern-agent/settings.mcs.yml

@@ -0,0 +1,24 @@
+displayName: Eval Modern Agent
+schemaName: eval_modernAgent
+accessControlPolicy: GroupMembership
+authenticationMode: None
+authenticationTrigger: AsNeeded
+configuration:
+  recognizer:
+    $kind: CLICopilotRecognizer
+
+  agentSettings:
+    $kind: AgentSettings
+    instructions:
+      $kind: Instructions
+      segments:
+        - $kind: StaticSegment
+          value: You are a helpful customer support agent for Contoso Electronics.
+
+    conversationStarters:
+      - $kind: ConversationStarter
+        title: Get Started
+        text: How can you help me?
+
+template: cliagent-1.0.0
+language: 1033

evals/fixtures/modern-agent/topics/Default_orderLookup.mcs.yml

@@ -0,0 +1,15 @@
+mcs.metadata:
+  componentName: OrderLookup
+  description: Looks up order status when the customer asks about shipping or delivery.
+kind: InlineAgentSkill
+content: |-
+  ---
+  name: OrderLookup
+  description: Looks up order status when the customer asks about shipping or delivery.
+  ---
+  # Order Lookup
+
+  When the user asks about their order status:
+  1. Ask for their order number
+  2. Tell them you're looking it up
+  3. Respond with a placeholder status

evals/fixtures/modern-agent/translations/Default_draft_eval.action.MSNWeather-Getcurrentweather.mcs.yml

@@ -0,0 +1,7 @@
+mcs.metadata:
+  componentName: MSN Weather — Get current weather
+  description: Get the current weather for a location.
+kind: ConnectorTool
+connectorId: /providers/Microsoft.PowerApps/apis/shared_msnweather
+connectionReference: eval_modernAgent.shared_msnweather.abc123
+operationId: CurrentWeather

evals/scenarios/skill-filtering.json

@@ -0,0 +1,91 @@
+{
+  "scenario_name": "skill-filtering",
+  "evals": [
+    {
+      "id": 1,
+      "name": "Create a new skill on modern agent",
+      "prompt": "Create a skill called ReturnPolicy that explains our 30-day return policy when customers ask about returns or refunds.",
+      "fixture": "modern-agent",
+      "mock_scripts": [],
+      "checks": {
+        "skill_invoked": "copilot-studio:new-skill",
+        "files_created": [
+          {
+            "pattern": "topics/*.mcs.yml",
+            "min_count": 1
+          }
+        ],
+        "content_contains": [
+          "InlineAgentSkill",
+          "return",
+          "30"
+        ],
+        "no_placeholders": true
+      }
+    },
+    {
+      "id": 2,
+      "name": "Edit existing skill description on modern agent",
+      "prompt": "Update the OrderLookup skill description to also mention 'order cancellation' and 'refund status' so the orchestrator routes those questions to it too.",
+      "fixture": "modern-agent",
+      "mock_scripts": [],
+      "checks": {
+        "skill_invoked": "copilot-studio:edit-skill",
+        "content_contains": [
+          "cancellation",
+          "refund"
+        ]
+      }
+    },
+    {
+      "id": 3,
+      "name": "Edit modern agent instructions",
+      "prompt": "Change the agent's instructions to: 'You are a friendly tech support agent for Contoso Electronics. Always greet the customer by name if available. Escalate complex hardware issues to a human agent.'",
+      "fixture": "modern-agent",
+      "mock_scripts": [],
+      "checks": {
+        "skill_invoked": "copilot-studio:edit-agent-modern",
+        "content_contains": [
+          "Contoso Electronics",
+          "escalate",
+          "hardware"
+        ]
+      }
+    },
+    {
+      "id": 4,
+      "name": "Add website knowledge source on modern agent",
+      "prompt": "Add https://learn.microsoft.com/en-us/copilot-studio/ as a knowledge source so the agent can answer questions about Copilot Studio.",
+      "fixture": "modern-agent",
+      "mock_scripts": [],
+      "checks": {
+        "skill_invoked": "copilot-studio:add-knowledge-modern",
+        "files_created": [
+          {
+            "pattern": "knowledge/*.mcs.yml",
+            "min_count": 1
+          }
+        ],
+        "content_contains": [
+          "WebsiteKnowledgeSource",
+          "learn.microsoft.com"
+        ]
+      }
+    },
+    {
+      "id": 5,
+      "name": "Add conversation starters on modern agent",
+      "prompt": "Add three conversation starters: 'Check order status', 'Return a product', and 'Contact support'.",
+      "fixture": "modern-agent",
+      "mock_scripts": [],
+      "checks": {
+        "skill_invoked": "copilot-studio:edit-agent-modern",
+        "content_contains": [
+          "ConversationStarter",
+          "order status",
+          "Return"
+        ]
+      }
+    }
+  ]
+}

hooks/filter-skills.js

@@ -0,0 +1,74 @@
+const fs = require('fs');
+const path = require('path');
+
+const pluginRoot = process.env.CLAUDE_PLUGIN_ROOT || path.join(__dirname, '..');
+
+// Detect agent type from settings.mcs.yml
+function detectAgentType(cwd) {
+  function findSettings(dir, depth) {
+    if (depth > 4) return null;
+    try {
+      const entries = fs.readdirSync(dir, { withFileTypes: true });
+      for (const e of entries) {
+        if (e.name === 'settings.mcs.yml' && e.isFile()) return path.join(dir, e.name);
+      }
+      for (const e of entries) {
+        if (e.isDirectory() && !e.name.startsWith('.') && e.name !== 'node_modules') {
+          const found = findSettings(path.join(dir, e.name), depth + 1);
+          if (found) return found;
+        }
+      }
+    } catch {}
+    return null;
+  }
+
+  const settingsPath = findSettings(cwd, 0);
+  if (!settingsPath) return '';
+
+  const content = fs.readFileSync(settingsPath, 'utf8');
+  if (/template:\s*cliagent-/.test(content) ||
+      /\$kind:\s*CLICopilotRecognizer/.test(content) ||
+      /kind:\s*CLICopilotRecognizer/.test(content) ||
+      /kind:\s*CLIAgentRecognizer/.test(content)) {
+    return 'modern';
+  }
+  return 'classic';
+}
+
+// Read agent-types from a skill's SKILL.md frontmatter
+function getSkillAgentType(skillName) {
+  const skillFile = path.join(pluginRoot, 'skills', skillName, 'SKILL.md');
+  try {
+    // Read just the frontmatter (first 20 lines is plenty)
+    const content = fs.readFileSync(skillFile, 'utf8');
+    const match = content.match(/^agent-types:\s*(.+)$/m);
+    return match ? match[1].trim() : 'both';
+  } catch {
+    return 'both'; // Unknown skill — allow
+  }
+}
+
+let input = '';
+process.stdin.setEncoding('utf8');
+process.stdin.on('data', (chunk) => { input += chunk; });
+process.stdin.on('end', () => {
+  try {
+    const data = JSON.parse(input);
+    const skillField = (data.tool_input && data.tool_input.skill) || '';
+    const command = skillField.includes(':') ? skillField.split(':').pop() : skillField;
+    if (!command) process.exit(0);
+
+    const skillType = getSkillAgentType(command);
+    if (skillType === 'both') process.exit(0);
+
+    const cwd = data.cwd || process.cwd();
+    const agentType = detectAgentType(cwd);
+    if (!agentType || agentType === skillType) process.exit(0);
+
+    const labels = { modern: 'Modern Agents', classic: 'Generative Orchestration agents' };
+    process.stdout.write(JSON.stringify({
+      decision: 'block',
+      reason: 'The "' + command + '" skill is for ' + labels[skillType] + ' only. This workspace contains a ' + labels[agentType] + ' workspace.'
+    }));
+  } catch {}
+});

hooks/hooks.json

@@ -17,6 +17,19 @@
           }
         ]
       }
+    ],
+    "PreToolUse": [
+      {
+        "matcher": "Skill",
+        "hooks": [
+          {
+            "type": "command",
+            "command": "node -e \"var r=process.env.CLAUDE_PLUGIN_ROOT;if(r){var p=require('path');require(p.join(r,'hooks','filter-skills.js'))}\"",
+            "bash": "node \"${CLAUDE_PLUGIN_ROOT}/hooks/filter-skills.js\"",
+            "powershell": "node \"${CLAUDE_PLUGIN_ROOT}/hooks/filter-skills.js\""
+          }
+        ]
+      }
     ]
   }
 }

hooks/system-prompt.md

@@ -38,6 +38,15 @@ This check prevents users from ending up in a dead end where YAML files are crea
 - As you have now understood, for Copilot Studio projects and requests, you're the 'manager' of those sub-agents that can work for you. In all cases, regardless of the task, you are still allowed to ask the user for clarifications if you don't understand the request, or if you need more details to be able to provide a better answer or to be able to choose better which sub-agent to call and how. Sub-agents might also ask you for clarifications, and in those cases, you should relay the questions to the user and then provide the answers back to the sub-agent.
 - Sub-agents should be given the broad context on the task they need to perform, with all the details you can gather from the user's request, but without providing yourself the way to achieve the task, because that's the job of the sub-agent. For example, if the user is asking to add a new feature to their agent, you should provide the sub-agent with all the details about the feature, about the agent, and about anything else that can be useful for the sub-agent to perform the task, but you should not provide instructions to the sub-agent on how to implement that feature like 'build a topic with this YAML code [...]', because that's what the sub-agent is for. You should let the sub-agent figure out how to implement that feature by itself, based on the context and details you provided. Sub-agents are designed to be autonomous and to figure out by themselves how to perform the tasks they're assigned, so you should trust them and give them the freedom to do their job. The only exception for this is if the user explicitly gives you non-functional requirements on how the task should be performed, for example if the user says 'I want you to build a topic that [...]', in such case you can tell the sub-agent that a topic should be built. But for general requests like 'The agent should be able to [...]' then you shouldn't provide instructions to the sub-agent on how to implement that, but just give them the context and let them figure out the best way to do it.
 
+## Agent types: classic vs modern
+
+Copilot Studio has two agent types. The plugin auto-detects which type is in the workspace and blocks incompatible skills:
+
+- **Classic (Generative Orchestration)** agents use topics, action nodes, triggers, Power Fx, and adaptive cards. Most existing skills target this type.
+- **Modern** agents use instructions, inline skills, declarative tools, and markdown. They have `template: cliagent-1.0.0` in `settings.mcs.yml`. A different set of skills targets this type.
+
+The Author agent detects the type automatically and uses the correct skills. You don't need to worry about this — just delegate to the Author and it will handle it.
+
 ## Sub-agents available for Copilot Studio requests
 The agents you have at your disposal to handle Copilot Studio requests include, but are not limited to:
 - Advisor Agent: this is the advisory agent for design guidance, agent review, and troubleshooting. It recommends proven design patterns before authoring begins, reviews existing agent YAML against patterns and known pitfalls, and troubleshoots validation errors and unexpected behavior. Use this agent when the user asks for design recommendations ("how should I build…"), wants their agent reviewed or audited, or reports something not working ("my topic isn't triggering", "the agent is hallucinating", "wrong topic fires", validation errors, unexpected behavior). The Advisor presents patterns as suggestions — the user decides what to adopt. You can also call the Advisor for troubleshooting when needed (e.g., the Author hits a validation error you can't resolve).