From 19f97f2100b067b69d369cc1068f93df6e30c54a Mon Sep 17 00:00:00 2001 From: Robert Standefer Date: Wed, 19 Aug 2026 11:18:03 -0700 Subject: [PATCH] feat: refine Copilot Studio harness guidance Add conditional runtime ownership guidance, harness-specific recommendations, updated scoring documentation, and end-to-end coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 691d09c1-d5c6-440b-b1e4-3351646bbf8f --- .github/copilot-instructions.md | 8 +- README.md | 12 +- apa.yaml | 135 +++++++++++++++---- assets/apa.css | 6 + assets/apa.js | 198 +++++++++++++++++++++++++--- docs/CHANGELOG.md | 7 +- docs/FLOWCHART.md | 19 ++- docs/SCORING.md | 89 ++++++++----- tests/e2e/harness-guidance.spec.js | 50 +++++++ tests/e2e/share-buttons.spec.js | 8 +- tests/e2e/shared-link.spec.js | 31 ++++- tests/e2e/wizard-completion.spec.js | 64 ++++++++- 12 files changed, 536 insertions(+), 91 deletions(-) create mode 100644 tests/e2e/harness-guidance.spec.js diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 8993e61..2d078eb 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -49,7 +49,8 @@ Documented in `docs/SCORING.md` and `docs/FLOWCHART.md`: 1. **Hard rules** zero out platforms for disqualifying answer combinations 2. **Raw scores** sum across 5 questions (max 15 per platform) 3. **Tiebreakers** in `apa.yaml` resolve equal scores using persona context -4. **Thresholds** map scores to fit labels: Strong (12–15), Good (8–11), Partial (4–7), Not recommended (0–3) +4. **Conditional runtime distinction** asks who should operate the runtime only when Copilot Studio and Foundry are the top two platforms within 2 points +5. **Thresholds** map scores to fit labels: Strong (12–15), Good (8–11), Partial (4–7), Not recommended (0–3) `meta.platforms` lists four platforms, but `m365_copilot` is always zeroed in the scored wizard (`if (!fastTrack) zeroed['m365_copilot'] = true`), so only three can actually win. M365 Copilot is reached through the entry-point wizard, or via the legacy `?ft=1` / `?dt=copilot_chat` share links. @@ -60,6 +61,7 @@ Result links are shared externally, so **old parameter shapes must keep resolvin | Param | Meaning | |---|---| | `q1`, `q8`, `q2`, `q4`, `q3` | Scored-wizard answers (option IDs) | +| `q9=q9a|q9d` | Conditional runtime ownership tie-breaker | | `dt=m365_copilot\|cowork\|scout\|both` | Entry-point destination | | `st=chat\|agents` | Which M365 Copilot surface to feature | | `r=` + `d=YYYYMMDD` | Original recommendation + date; drive the temporal-change banner | @@ -81,8 +83,8 @@ Key constraints: ## Conventions -- Always update `docs/CHANGELOG.md` after making changes. Sections are dated by commit date (`## 2026-07-24`); work in progress sits under `## Unreleased` until it's committed. +- Always update `docs/CHANGELOG.md` after making changes. Sections are dated by commit date (`## 2026-07-24`). - Always update `docs/FLOWCHART.md` and `docs/SCORING.md` after changes that affect user flow or scoring logic. -- Question IDs in `apa.yaml` are not sequential (e.g., `q1, q8, q2, q4, q3`) — they preserve identity across schema changes. Display order is the array order in `apa.yaml`, not the numeric ID. +- Question IDs in `apa.yaml` are not sequential (e.g., `q1, q8, q2, q4, q3`) — they preserve identity across schema changes. Display order is the array order in `apa.yaml`, not the numeric ID. `q9` is a conditional, non-scored runtime distinction. - Entry-point cards (`ENTRY_POINT_PLATFORMS` in `apa.js`) render their accordions expanded; scored comparison cards stay collapsed. - In tests, don't identify a recommendation card by a headline substring alone — several destinations share "Microsoft 365 Copilot" wording. Assert exact `.rec-platform-name` text plus something distinguishing. diff --git a/README.md b/README.md index 7ee5b7f..db130b2 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,7 @@ The start page presents ways to use or build agents: From **Get Started**, users choose one of three paths: 1. **Entry-point wizard** ("Help me find the right place to get work done") — asks how hands-on you want to be, then routes to Microsoft 365 Copilot (featuring either Copilot Chat or its built-in agents), Cowork, Scout, or a Cowork+Scout pair. Non-scored. -2. **Custom agent assessment** — runs the scored 5-question wizard for Agent Builder, Copilot Studio, and Foundry. +2. **Custom agent assessment** — runs the scored 5-question wizard for Agent Builder, Copilot Studio, and Foundry, with a conditional runtime tie-breaker when the two enterprise platforms remain close. 3. **Explore what's possible** — compares ways to use or build agents before deciding whether to take the assessment. The custom agent assessment asks about: @@ -40,9 +40,12 @@ The custom agent assessment asks about: - What the agent should do: Q&A, multi-turn conversation, content/data analysis, multi-step action workflows, or complex orchestration - What information it needs: Microsoft 365 content, connector-backed systems, Dataverse/custom APIs, public web/uploaded files, or custom retrieval architecture +When Copilot Studio and Foundry are the top two platforms within 2 points, the wizard asks one final non-scored distinction: whether Microsoft or the user's engineering team should operate the agent runtime. + After completing a path, users get: - A primary recommendation with fit badge, key factors, and platform-specific guidance +- A "Start with this harness" callout when Copilot Studio wins - A "Start Here" callout on entry-point results naming the surface to open first - A secondary "Also Consider" card when another option is close or complementary - A score comparison panel for the scored wizard @@ -88,6 +91,7 @@ There are two recommendation modes: - Hard rules zero out platforms for disqualifying combinations before scoring. - Raw scores sum across 5 questions, with a maximum of 15 points per platform. - Persona preferences and tiebreakers adjust ranking when scores are tied or misleading for the selected builder persona. + - A conditional runtime distinction resolves close Copilot Studio/Foundry cases without changing their scores. - Thresholds map scores to fit labels: Strong fit (12-15), Good fit (8-11), Partial fit (4-7), Not recommended (0-3). Microsoft 365 Copilot, Cowork, and Scout are not part of the 0-15 scored wizard. They are reached only through the entry-point wizard. @@ -97,7 +101,7 @@ Microsoft 365 Copilot, Cowork, and Scout are not part of the 0-15 scored wizard. The advisor reflects the current split between Microsoft agent options: - **Agent Builder** has expanded beyond SharePoint/OneDrive-only scenarios. It now covers no-code agents grounded in Microsoft 365 content, scoped web, uploaded files, and admin-enabled Microsoft 365 Copilot connectors, including lightweight content and data-analysis helpers. -- **Copilot Studio** is the default governed low-code path for enterprise agents that need actions, workflows, triggers, connectors, MCP tools, computer use, connected agents, evaluation, monitoring, and multi-channel deployment. +- **Copilot Studio** is the managed enterprise agent platform spanning GitHub Copilot harness agents for adaptive work, standard harness agents for predictable conversations, Copilot chat harness agents for Microsoft 365 knowledge extensions, and deterministic workflows. - **Microsoft Foundry** is the developer-controlled production runtime for prompt agents, hosted code agents, custom retrieval, managed endpoints, toolboxes, MCP, identity, private networking, tracing, evaluation, monitoring, and custom app/service integration. - **Microsoft 365 Copilot** is treated as the built-in productivity layer, and as one product rather than several: Copilot Chat, Copilot Search, app-native Copilot, Pages, Notebooks, and Microsoft-built agents are all surfaces within it. - **Copilot Cowork** and **Microsoft Scout** are personal agents you delegate work to, not platforms in the scored build assessment. @@ -106,7 +110,7 @@ The advisor reflects the current split between Microsoft agent options: Share links encode the recommendation path: -- Wizard results include selected answers, the recommended platform, and the recommendation date. +- Wizard results include selected scored answers (`q1`, `q8`, `q2`, `q4`, and `q3`), the optional runtime distinction (`q9`) when asked, the recommended platform, and the recommendation date. - Entry-point results include `dt=m365_copilot`, `dt=cowork`, `dt=scout`, or `dt=both`, plus `st=chat` or `st=agents` for the Microsoft 365 Copilot starting surface. - Older links keep working: `ft=1` and `dt=copilot_chat` both resolve to the Microsoft 365 Copilot card. - Recipients can view the recommendation directly or retake the assessment with answers pre-filled. @@ -123,7 +127,7 @@ npm test # headless npm run test:headed # with browser visible ``` -There are 39 tests across 6 spec files covering wizard completion, shared link loading, temporal change detection, legacy fast-track links, entry-point wizard routing, and share button behavior. To run a single file or test: `npx playwright test tests/e2e/delegate-path.spec.js` or `npx playwright test -g "completes full wizard"`. CI runs automatically on push and pull request via GitHub Actions. +There are 48 tests across 7 spec files covering wizard completion, the conditional runtime distinction, Copilot Studio harness guidance, shared link loading, temporal change detection, legacy fast-track links, entry-point wizard routing, and share button behavior. To run a single file or test: `npx playwright test tests/e2e/delegate-path.spec.js` or `npx playwright test -g "completes full wizard"`. CI runs automatically on push and pull request via GitHub Actions. ## Contributing diff --git a/apa.yaml b/apa.yaml index 49ff1de..c2486d9 100644 --- a/apa.yaml +++ b/apa.yaml @@ -34,7 +34,7 @@ questions: label: Business user or subject matter expert — no coding experience scores: agent_builder: 3 - copilot_studio: 1 + copilot_studio: 2 foundry: 0 - id: q1b @@ -55,7 +55,7 @@ questions: label: Data scientist or AI/ML engineer — works with AI models and data infrastructure scores: agent_builder: 0 - copilot_studio: 1 + copilot_studio: 2 foundry: 3 - id: q8 @@ -159,7 +159,7 @@ questions: label: Create or analyze content inside Copilot — documents, charts, images, or lightweight data analysis scores: agent_builder: 3 - copilot_studio: 2 + copilot_studio: 3 foundry: 2 - id: q4c @@ -173,7 +173,7 @@ questions: label: Manage complex workflows — coordinate multiple agents or handle long-running processes scores: agent_builder: 0 - copilot_studio: 2 + copilot_studio: 3 foundry: 3 hard_rule_trigger: true @@ -224,6 +224,27 @@ questions: copilot_studio: 1 foundry: 3 +################################################################################ +# CONDITIONAL RUNTIME TIE-BREAKER +################################################################################ + +runtime_tiebreaker: + id: q9 + label: Who should operate the agent runtime? + prompt: Copilot Studio and Microsoft Foundry both fit your functional requirements. Choose who should own the execution environment. + threshold_points: 2 + compare: [copilot_studio, foundry] + legacy_managed_option_ids: [q9b, q9c] + options: + - id: q9a + label: Microsoft should manage the agent runtime, sandbox, tools, and Power Platform governance + prefer: copilot_studio + over: foundry + + - id: q9d + label: Our engineering team should own the code runtime, framework, endpoints, networking, identity, memory, or retrieval + zero: [agent_builder, copilot_studio] + ################################################################################ # SCORING ENGINE ################################################################################ @@ -256,6 +277,9 @@ scoring: q3f: zero: [agent_builder] label: "Custom retrieval architecture — Agent Builder cannot directly use custom RAG, Azure AI Search, private indexes, Foundry IQ, or engineering-managed retrieval systems" + q9d: + zero: [agent_builder, copilot_studio] + label: "Developer-owned runtime — Agent Builder and Copilot Studio do not provide an engineering-owned code runtime, framework, endpoint, network, identity, memory, or retrieval architecture" # Step 2 — Sum raw scores across all answered questions. raw_score_max: 15 # 5 questions × max score of 3 @@ -270,8 +294,17 @@ scoring: rationale: > Copilot Studio is recommended over Agent Builder for data scientists and AI/ML engineers because it supports curated model selection, agent evaluations, Foundry IQ integration, - code-first development via the VS Code extension, and more flexible orchestration — - capabilities that Agent Builder does not offer. + managed reasoning, portable skills, and more flexible orchestration — capabilities + that Agent Builder does not offer. Use Foundry when the scenario requires a custom + model or developer-owned runtime. + - when: + q9: q9a + prefer: copilot_studio + over: foundry + rationale: > + Copilot Studio is recommended because you want Microsoft to manage the reasoning + harness, sandbox, tools, and Power Platform governance. Choose Foundry when your + engineering team needs to own the runtime and infrastructure. # Step 3 — Map raw scores to recommendation strength. recommendation_thresholds: @@ -305,8 +338,9 @@ scoring: prefer: copilot_studio rationale: > When the builder is a professional developer and scores are equal, - Copilot Studio is the better fit — it supports code-first authoring - via the VS Code extension while Agent Builder is no-code only. + Copilot Studio is the better fit — it supports managed enterprise + orchestration, evaluation, governance, and developer authoring workflows + while Agent Builder is designed for lightweight no-code agents. - when: q1: q1d platforms: [copilot_studio, foundry] @@ -317,7 +351,7 @@ scoring: agents while Foundry is reserved for deeper custom orchestration. valid_pairs: - platforms: [copilot_studio, foundry] - rationale: Build in Copilot Studio, extend with custom code in Foundry. + rationale: Choose Copilot Studio when Microsoft should manage the reasoning harness, sandbox, tools, and Power Platform governance. Choose Foundry when your team must own the code runtime, framework, endpoint, network, identity, memory, or retrieval architecture. - platforms: [m365_copilot, copilot_studio] rationale: Microsoft 365 Copilot for end users, Copilot Studio for customization and extension. - platforms: [agent_builder, m365_copilot] @@ -344,9 +378,10 @@ scoring: q1: q1a q4: q4d note: > - Complex multi-agent orchestration typically requires professional development - skills. Consider partnering with your development team, or selecting a simpler - task type that matches your experience level. + Copilot Studio can now author reasoning-heavy, multi-step agents through natural + language, but production use still requires careful tool permissions, evaluation, + monitoring, human review, and Copilot Credit governance. Consider involving your + IT or platform team before deployment. - when: q1: q1a q3: q3c @@ -550,21 +585,27 @@ recommendations: copilot_studio: headline: Copilot Studio - description: Build governed enterprise agents that combine knowledge, tools, workflows, event triggers, human review, connected agents, evaluation, monitoring, and multi-channel deployment + description: Build governed enterprise agents with managed reasoning harnesses, natural-language authoring, knowledge, tools, workflows, triggers, human review, evaluation, monitoring, and multi-channel deployment resources_url: https://microsoft.github.io/agent-resources/copilot-studio/ - scoring_summary: The governed low-code platform for enterprise agents that need actions, workflows, connectors, evaluation, monitoring, and broad deployment. - exploration_best_for: Build governed enterprise agents and workflows + scoring_summary: The managed enterprise agent platform for adaptive agents, predictable conversations, deterministic workflows, Microsoft 365 Copilot extensions, and broad deployment. + exploration_best_for: Build managed, governed enterprise agents and workflows exploration_summary: > Choose Copilot Studio when you need agents that go beyond quick knowledge lookup — connecting to CRMs, databases, APIs, MCP servers, workflows, Microsoft 365 context, - and other agents. Great for makers and IT teams building governed agents that can - reason, take actions, evaluate quality, and deploy across channels. + files, and other agents. Choose among the GitHub Copilot, standard, and Copilot chat + harnesses based on whether the work should adapt, follow explicit rules, or extend + Microsoft 365 Copilot. Great for makers, IT teams, and developers building governed + agents that can reason, take actions, evaluate quality, and deploy across channels. summary: > - The best low-code platform for governed enterprise agents — connect to outside - systems, automate business processes, use generative orchestration, evaluate - quality, monitor behavior, and deploy to websites, Teams, Microsoft 365, and more. + The managed platform for governed enterprise agents — use natural-language-first + adaptive orchestration, predictable topic-driven agents, or deterministic workflows; + connect to outside systems, evaluate quality, monitor behavior, and deploy broadly. best_for: - - Low-code makers and IT pros + - Business users, low-code makers, IT pros, and developers who want a Microsoft-managed agent runtime + - Reasoning-heavy, multi-step business processes that plan, adapt, recover from failures, and work across several tools + - Native creation, editing, and reasoning over Word, Excel, PowerPoint, and PDF files + - Predictable topic-driven conversations and deterministic triggered or scheduled workflows + - Internal Microsoft 365 Copilot knowledge extensions through the Copilot chat harness - Agents connecting to external APIs, CRM, ERP, Dataverse, databases, MCP servers, or custom connectors - Event-triggered, scheduled, or user-triggered workflows with actions, branching, human review, and approvals - Multi-agent orchestration (now generally available) — child agents, connected Copilot Studio agents, and A2A integrations @@ -573,16 +614,62 @@ recommendations: - Internal and external-facing deployment - Built-in evaluation test sets, activity maps, analytics, monitoring, and tenant-wide agent inventory watch_out_for: + - Choose the harness when you create the agent — GitHub Copilot, standard, and Copilot chat harness agents can't be transferred between harnesses + - GitHub Copilot harness agents and workflows consume Copilot Credits while building, previewing, testing, evaluating, and running in production - Complex ML pipelines or custom model training require Foundry - - Highly custom, high-scale, or code-first multi-agent orchestration is better served by Foundry + - Use Foundry when your team must own the code runtime, custom framework, stable endpoint, private network, dedicated identity, custom memory, or retrieval architecture - Supports curated GA, preview, and experimental models, including deep reasoning models; use Foundry for arbitrary bring-your-own model hosting, custom model training, model routing, and full runtime control - - Memory is available in the new experience preview with per-user/per-agent storage and lifecycle limits; use Foundry when you need custom memory architecture, longer retention, or application-controlled memory + - Memory is preview, stored separately per user, deleted after 28 days without interaction, unavailable in group chats and Teams channels, and not visible to makers - Computer use requires careful credential, supervision, access-control, observability, and cost governance - Copilot Credits can scale quickly with generative answers, agent actions, tenant graph grounding, workflows, reasoning models, voice, and computer use + harnesses: + github_copilot: + label: GitHub Copilot harness + url: https://learn.microsoft.com/microsoft-copilot-studio/agents-experience/overview + tagline: Managed adaptive execution for reasoning-heavy, multi-step work + description: > + Start here when the agent should take a goal, plan the steps, use several tools + or agents, create or edit Office and PDF files, adapt when conditions change, + and recover when a step fails. Tasks run in a secure Copilot Studio sandbox. + considerations: + - Harness choice is fixed for the agent; it can't later transfer to the standard or Copilot chat harness + - Copilot Credits apply during building, previewing, testing, evaluating, and production use + standard: + label: Standard harness + url: https://learn.microsoft.com/microsoft-copilot-studio/harnesses-overview + tagline: Predictable topic-driven conversations and explicit rules + description: > + Start here when the agent should follow defined topics, branches, prompts, and + repeatable conversation paths rather than decide its own plan dynamically. + considerations: + - Prefer the workflow experience instead when the primary requirement is scheduled or event-triggered deterministic automation + - Harness choice is fixed for the agent and should be validated before implementation + copilot_chat: + label: Copilot chat harness + url: https://learn.microsoft.com/microsoft-copilot-studio/microsoft-365-copilot-extend-with-agents + tagline: Internal knowledge extensions inside Microsoft 365 Copilot Chat + description: > + Start here when employees need Microsoft 365 Copilot Chat grounded in your + organization's knowledge without leaving their everyday Microsoft 365 experience. + considerations: + - Publishes to internal teams rather than external customers + - Agent Builder remains the faster option for the simplest small-team no-code scenario + workflow: + label: Copilot Studio workflow + url: https://learn.microsoft.com/microsoft-copilot-studio/workflows-experience/flows-overview + tagline: Deterministic automation with triggers, branching, AI actions, and agent handoffs + description: > + Start with a workflow when a schedule, event, user, or agent should trigger a + predictable sequence of connector actions, control logic, human input, and AI steps. + considerations: + - Each production action consumes Copilot Studio capacity, so monitor environment allocation and enforcement + - Use an adaptive GitHub Copilot harness agent when the process must decide and re-plan rather than follow a defined path persona_tips: q1c: > - 🛠️ Developer tip: You can build Copilot Studio agents in YAML using the + 🛠️ Developer tip: You can author Copilot Studio components in YAML using the Copilot Studio extension for VS Code + for source-controlled collaboration. This improves the development workflow but does + not provide a developer-owned code runtime; use Foundry when you need that control. foundry: headline: Microsoft Foundry diff --git a/assets/apa.css b/assets/apa.css index 6ed5241..bcca66f 100644 --- a/assets/apa.css +++ b/assets/apa.css @@ -1717,6 +1717,12 @@ body { line-height: 1.5; margin: 0; } +.rec-harness-considerations { + margin: 12px 0 0; +} +.rec-harness-considerations li { + color: var(--muted-foreground); +} /* Recommendation section navigation */ .rec-nav { diff --git a/assets/apa.js b/assets/apa.js index c0ad1e8..c3f37fd 100644 --- a/assets/apa.js +++ b/assets/apa.js @@ -6,6 +6,7 @@ let delegateResult = null; // 'm365_copilot' | 'cowork' | 'scout' | 'both' — s let delegateStart = null; // 'chat' | 'agents' — which surface inside Microsoft 365 Copilot to start with let delegateAnswers = {}; // { involvement: 'interactive'|'delegate', taskType: 'general'|'specialized', cadence: 'ondemand'|'continuous'|'unsure', reach: 'm365'|'cross'|'unsure' } let currentQuestionIndex = 0; +let runtimeTieBreakerActive = false; let listenersReady = false; let recommendedPlatformId = null; let isURLLoaded = false; // true when loaded from shared URL params @@ -23,8 +24,8 @@ function showSection(id) { } // === HISTORY NAVIGATION === -function pushState(section, questionIndex) { - const state = { section, questionIndex: questionIndex ?? null }; +function pushState(section, questionIndex, runtimeTieBreaker = false) { + const state = { section, questionIndex: questionIndex ?? null, runtimeTieBreaker }; history.pushState(state, '', ''); } @@ -35,8 +36,12 @@ window.addEventListener('popstate', (e) => { return; } if (state.section === 'assessment-section' && state.questionIndex != null) { - currentQuestionIndex = state.questionIndex; - renderQuestion(); + if (state.runtimeTieBreaker) { + renderRuntimeTieBreaker(); + } else { + currentQuestionIndex = state.questionIndex; + renderQuestion(); + } } else if (state.section === 'recommendation-section') { renderRecommendation(); } else if (state.section === 'exploration-section') { @@ -262,6 +267,43 @@ function badgeClass(label) { return 'badge-not'; } +function normalizeRuntimeOptionId(optionId) { + const runtime = apa.runtime_tiebreaker; + if (!runtime || !optionId) return null; + if ((runtime.legacy_managed_option_ids || []).includes(optionId)) return 'q9a'; + return runtime.options.some(option => option.id === optionId) ? optionId : null; +} + +function shouldAskRuntimeTieBreaker(answersMap) { + const runtime = apa.runtime_tiebreaker; + if (!runtime) return false; + + const baseAnswers = { ...answersMap }; + delete baseAnswers[runtime.id]; + const zeroed = getZeroedPlatforms(baseAnswers); + const ranked = rankPlatforms(baseAnswers); + const [firstId, secondId] = runtime.compare; + const first = ranked.find(platform => platform.id === firstId); + const second = ranked.find(platform => platform.id === secondId); + const topTwoIds = new Set(ranked.slice(0, 2).map(platform => platform.id)); + + return !!( + first && second && + topTwoIds.has(firstId) && topTwoIds.has(secondId) && + !zeroed[firstId] && !zeroed[secondId] && + first.score > 0 && second.score > 0 && + Math.abs(first.score - second.score) <= runtime.threshold_points + ); +} + +function resolveCopilotStudioHarness(answersMap) { + if (answersMap.q9 === 'q9d') return null; + if (answersMap.q4 === 'q4d' || answersMap.q4 === 'q4e') return 'github_copilot'; + if (answersMap.q2 === 'q2c' || answersMap.q4 === 'q4c') return 'workflow'; + if (answersMap.q2 === 'q2a' && answersMap.q4 === 'q4a') return 'copilot_chat'; + return 'standard'; +} + function buildPlatformCard(platformId, ranked, answersMap, isPrimary, showBadge, startKey) { const rec = apa.recommendations[platformId]; @@ -308,6 +350,23 @@ function buildPlatformCard(platformId, ranked, answersMap, isPrimary, showBadge, `; })() : ''; + const harnessKey = isPrimary && platformId === 'copilot_studio' + ? resolveCopilotStudioHarness(answersMap) + : null; + const harness = harnessKey && rec.harnesses ? rec.harnesses[harnessKey] : null; + const harnessHtml = harness ? ` +
+
Start with this harness
+
+ ${harness.label} +
+
${harness.tagline}
+

${harness.description}

+ ${(harness.considerations || []).length > 0 + ? `
    ${harness.considerations.map(item => `
  • ${item}
    • `).join('')}
` + : ''} +
` : ''; + const firstPartyHtml = (rec.first_party_agents || []).length > 0 ? `
@@ -352,6 +411,7 @@ function buildPlatformCard(platformId, ranked, answersMap, isPrimary, showBadge, ${descriptionHtml}

${rec.summary}

+ ${harnessHtml} ${spotlightHtml} ${rec.persona_tips && rec.persona_tips[answersMap.q1] ? `
${rec.persona_tips[answersMap.q1]}
` @@ -400,14 +460,22 @@ function restoreAnswersFromStorage() { const stored = sessionStorage.getItem('apa-answers'); if (!stored) return null; const parsed = JSON.parse(stored); + const normalized = {}; // Schema drift check: validate every key/value against current YAML const validQuestionIds = new Set(apa.questions.map(q => q.id)); for (const [qId, optId] of Object.entries(parsed)) { + if (qId === apa.runtime_tiebreaker?.id) { + const normalizedRuntime = normalizeRuntimeOptionId(optId); + if (!normalizedRuntime) { clearAnswersFromStorage(); return null; } + normalized[qId] = normalizedRuntime; + continue; + } if (!validQuestionIds.has(qId)) { clearAnswersFromStorage(); return null; } const question = apa.questions.find(q => q.id === qId); if (!question.options.some(o => o.id === optId)) { clearAnswersFromStorage(); return null; } + normalized[qId] = optId; } - return parsed; + return normalized; } catch (e) { return null; } } @@ -455,10 +523,22 @@ async function boot() { const restored = restoreAnswersFromStorage(); if (restored && Object.keys(restored).length > 0) { answers = restored; - currentQuestionIndex = Math.min(Object.keys(restored).length, apa.questions.length - 1); - renderQuestion(); + const answeredCore = apa.questions.filter(question => answers[question.id]).length; + currentQuestionIndex = Math.min(answeredCore, apa.questions.length - 1); + const resumeRuntimeTieBreaker = + answeredCore === apa.questions.length && + shouldAskRuntimeTieBreaker(answers); + if (resumeRuntimeTieBreaker) { + renderRuntimeTieBreaker(); + } else { + renderQuestion(); + } showSection('assessment-section'); - history.replaceState({ section: 'assessment-section', questionIndex: currentQuestionIndex }, '', ''); + history.replaceState({ + section: 'assessment-section', + questionIndex: currentQuestionIndex, + runtimeTieBreaker: resumeRuntimeTieBreaker, + }, '', ''); } else { showSection('welcome-section'); history.replaceState({ section: 'welcome-section' }, '', ''); @@ -490,6 +570,7 @@ function setupListeners() { function handlePrescreenNo() { fastTrack = false; delegateResult = null; + runtimeTieBreakerActive = false; if (Object.keys(answers).length === 0) { currentQuestionIndex = 0; } @@ -656,6 +737,7 @@ function renderExploration() { } function renderQuestion() { + runtimeTieBreakerActive = false; const question = apa.questions[currentQuestionIndex]; const total = apa.questions.length; @@ -682,6 +764,7 @@ function renderQuestion() { `; const select = () => { answers[question.id] = opt.id; + delete answers[apa.runtime_tiebreaker?.id]; saveAnswersToStorage(); renderQuestion(); }; @@ -700,21 +783,85 @@ function renderQuestion() { document.getElementById('prev-btn').disabled = false; } +function renderRuntimeTieBreaker() { + const runtime = apa.runtime_tiebreaker; + if (!runtime) return; + runtimeTieBreakerActive = true; + + document.getElementById('question-counter').textContent = 'One final distinction'; + document.getElementById('question-title').textContent = runtime.label; + document.getElementById('question-subtitle').textContent = runtime.prompt; + + const optionsList = document.getElementById('options-list'); + optionsList.innerHTML = ''; + runtime.options.forEach(option => { + const isSelected = answers[runtime.id] === option.id; + const div = document.createElement('div'); + div.className = 'option-card' + (isSelected ? ' selected' : ''); + div.setAttribute('role', 'button'); + div.setAttribute('tabindex', '0'); + div.setAttribute('aria-pressed', String(isSelected)); + div.innerHTML = ` + +
+
${option.label}
+
`; + const select = () => { + answers[runtime.id] = option.id; + saveAnswersToStorage(); + renderRuntimeTieBreaker(); + }; + div.addEventListener('click', select); + div.addEventListener('keydown', event => { + if (event.key === 'Enter' || event.key === ' ') { + event.preventDefault(); + select(); + } + }); + optionsList.appendChild(div); + }); + + const nextBtn = document.getElementById('next-btn'); + nextBtn.disabled = !answers[runtime.id]; + nextBtn.textContent = 'Get Recommendation ▶'; + document.getElementById('prev-btn').disabled = false; +} + function handleNext() { - const question = apa.questions[currentQuestionIndex]; + if (runtimeTieBreakerActive) { + renderRecommendation(); + showSection('recommendation-section'); + pushState('recommendation-section'); + return; + } if (currentQuestionIndex < apa.questions.length - 1) { currentQuestionIndex++; renderQuestion(); pushState('assessment-section', currentQuestionIndex); } else { - renderRecommendation(); - showSection('recommendation-section'); - pushState('recommendation-section'); + if (!answers[apa.runtime_tiebreaker?.id] && shouldAskRuntimeTieBreaker(answers)) { + renderRuntimeTieBreaker(); + pushState('assessment-section', currentQuestionIndex, true); + } else { + renderRecommendation(); + showSection('recommendation-section'); + pushState('recommendation-section'); + } } } function handlePrev() { + if (runtimeTieBreakerActive) { + delete answers[apa.runtime_tiebreaker?.id]; + saveAnswersToStorage(); + currentQuestionIndex = apa.questions.length - 1; + renderQuestion(); + pushState('assessment-section', currentQuestionIndex); + return; + } if (currentQuestionIndex > 0) { currentQuestionIndex--; renderQuestion(); @@ -780,12 +927,17 @@ function getScoreReason(platformId, ranked, answersMap) { const isWinner = winner && winner.id === platformId; const contribs = getContributions(platformId, answersMap); const perQ = getPerQuestionScores(platformId, answersMap); + const answeredCount = perQ.filter(q => answersMap[q.qId]).length; const perfectCount = perQ.filter(q => q.score === 3).length; const zeroCount = perQ.filter(q => q.score === 0).length; if (isWinner) { - if (perfectCount === 5) return 'Perfect fit — scored highest on every dimension.'; - if (perfectCount >= 4) return 'Strong match across nearly all dimensions.'; + if (answeredCount > 0 && perfectCount === answeredCount) { + return 'Perfect fit — scored highest on every dimension.'; + } + if (answeredCount > 1 && perfectCount >= answeredCount - 1) { + return 'Strong match across nearly all dimensions.'; + } const tops = contribs.slice(0, 2).map(c => `${c.questionLabel.replace(/\?$/, '')}`); return `Strongest on ${tops.join(' and ')}.`; } @@ -1117,6 +1269,7 @@ function restart() { delegateStart = null; delegateAnswers = {}; currentQuestionIndex = 0; + runtimeTieBreakerActive = false; recommendedPlatformId = null; isURLLoaded = false; originalPlatformId = null; @@ -1137,6 +1290,7 @@ function startFullAssessment() { delegateStart = null; answers = {}; currentQuestionIndex = 0; + runtimeTieBreakerActive = false; renderQuestion(); showSection('assessment-section'); pushState('assessment-section', 0); @@ -1192,7 +1346,19 @@ function parseURLParams() { } }); - // Check for questions in YAML not present in URL + const runtimeId = apa.runtime_tiebreaker?.id; + const runtimeValue = runtimeId ? params.get(runtimeId) : null; + if (runtimeValue) { + const normalizedRuntime = normalizeRuntimeOptionId(runtimeValue); + if (normalizedRuntime) { + answers[runtimeId] = normalizedRuntime; + } else { + hasDrift = true; + } + } + + // Check for scored questions in YAML not present in URL. The runtime + // distinction is conditional, so its absence is not schema drift. apa.questions.forEach(q => { if (!answers[q.id]) hasDrift = true; }); @@ -1220,6 +1386,8 @@ function buildShareableURL() { apa.questions.forEach(q => { if (answers[q.id]) params.set(q.id, answers[q.id]); }); + const runtimeId = apa.runtime_tiebreaker?.id; + if (runtimeId && answers[runtimeId]) params.set(runtimeId, answers[runtimeId]); } params.set('r', recommendedPlatformId || ''); diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index c039dd8..a252d38 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -4,10 +4,15 @@ All notable changes to Agent Platform Advisor are documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), organized by repository commit date. -## Unreleased +## 2026-08-19 ### Changed +- **Reframed Copilot Studio guidance around its three harnesses and workflow experience.** The assessment remains five scored questions, but now asks a conditional final distinction (`q9`) only when Copilot Studio and Foundry are the top two platforms within 2 points: should Microsoft manage the runtime, or should the user's engineering team own it? Copilot Studio results include a **Start with this harness** callout for the GitHub Copilot harness, standard harness, Copilot chat harness, or workflow experience. +- **Updated the Copilot Studio/Foundry boundary from task complexity to runtime ownership.** Copilot Studio now scores 3 for file/content work and complex orchestration because the GitHub Copilot harness supports adaptive multi-step execution, failure recovery, native Office/PDF file work, skills, memory, tools, connected agents, and secure sandboxing. Foundry remains the required path when the team must own the code runtime, framework, endpoint, network, identity, memory, or retrieval architecture. +- **Added harness-specific cautions.** Copilot Studio guidance now calls out fixed harness selection, Copilot Credit consumption during building/testing/evaluation as well as production, and memory's preview lifecycle and channel limitations. The core scoring model remains five questions, 1,920 combinations, and a 15-point maximum. +- **Preserved runtime-question share links across the schema refinement.** Links without `q9` remain valid; the previous `q9b` and `q9c` managed-platform values normalize to `q9a`; harness guidance is derived from task and deployment answers. +- **Expanded Playwright coverage to 48 tests across 7 spec files.** New tests cover showing and skipping the conditional distinction, both runtime-ownership outcomes, all four Copilot Studio starting points, `q9` sharing, and previous `q9` URL aliases. - **Refreshed platform guidance in `apa.yaml` for mid-2026 product updates** (content only — the scoring matrix is unchanged, so `docs/SCORING.md` distribution figures still hold). Verified against Microsoft Learn and June 2026 release notes that two constraints are still accurate and left them in place: the no-code **Agent Builder is still reactive Q&A/retrieval only** (no action execution or event triggers — MCP Apps add UI, not actions), and **Cowork is still Microsoft 365-only** (no desktop/shell/browser/local runtime), so its reach split with Scout is unchanged. - **Copilot Cowork now runs scheduled prompts.** Added a *Best For* line for scheduling recurring Microsoft 365 tasks (daily briefings, status roundups, inbox triage), and rewrote the "on demand and user-initiated" caution to distinguish Cowork's *scheduled/recurring* runs from Scout's *always-on, condition-monitoring* autonomy. Sourced from the Cowork overview (Learn, updated 2026-07-27). - **Microsoft Foundry:** called out the **Agent Optimizer** evaluate-and-optimize loop in the lifecycle bullet and added a *Best For* line for **Foundry Local on Azure Local** (air-gapped, disconnected, or on-premises deployment). diff --git a/docs/FLOWCHART.md b/docs/FLOWCHART.md index fd6fb24..42a1969 100644 --- a/docs/FLOWCHART.md +++ b/docs/FLOWCHART.md @@ -16,10 +16,10 @@ flowchart TD REACH -->|"Undecided signals"| BOTH["**Both** — complementary pair\nScout monitors · Cowork delivers"] Q1["**Q1: Who is building this agent?**"] - Q1 -->|Business user / no code| Q1A["AB:3 · CS:1 · Foundry:0"] + Q1 -->|Business user / no code| Q1A["AB:3 · CS:2 · Foundry:0"] Q1 -->|Low-code maker / IT pro| Q1B["AB:1 · CS:3 · Foundry:0"] Q1 -->|"🔀 Professional developer"| Q1C["AB:0 · CS:2 · Foundry:3\n→ TIEBREAKER: AB tie → prefer CS"] - Q1 -->|"🔀 Data scientist / AI-ML"| Q1D["AB:0 · CS:1 · Foundry:3\n→ PERSONA PREF: CS always over AB\n→ TIEBREAKER: CS/Foundry tie → prefer CS"] + Q1 -->|"🔀 Data scientist / AI-ML"| Q1D["AB:0 · CS:2 · Foundry:3\n→ PERSONA PREF: CS always over AB\n→ TIEBREAKER: CS/Foundry tie → prefer CS"] Q1A & Q1B & Q1C & Q1D --> Q8 @@ -42,9 +42,9 @@ flowchart TD Q4["**Q4: What should this agent do?**"] Q4 -->|Q&A, lookups, summaries| Q4A["AB:3 · CS:3 · Foundry:1"] Q4 -->|Multi-turn conversation| Q4B["AB:2 · CS:3 · Foundry:2"] - Q4 -->|Create/analyze content in Copilot| Q4E["AB:3 · CS:2 · Foundry:2"] + Q4 -->|Create/analyze content in Copilot| Q4E["AB:3 · CS:3 · Foundry:2"] Q4 -->|"⚠️ Multi-step action workflows"| Q4C["AB:0 · CS:3 · Foundry:3\n→ HARD RULE: AB=0"] - Q4 -->|"⚠️ Complex workflows / multi-agent"| Q4D["AB:0 · CS:2 · Foundry:3\n→ HARD RULE: AB=0"] + Q4 -->|"⚠️ Complex workflows / multi-agent"| Q4D["AB:0 · CS:3 · Foundry:3\n→ HARD RULE: AB=0"] Q4A & Q4B & Q4E & Q4C & Q4D --> Q3 @@ -62,7 +62,14 @@ flowchart TD SCORE --> PREF["**Persona Preferences**\nSoft overrides: force ranking order\nwithout changing scores\n(e.g. q1d → CS always over AB)"] - PREF --> RESULT["**Recommendation Thresholds**\n12–15: Strong fit\n8–11: Good fit\n4–7: Partial fit\n0–3: Not recommended"] + PREF --> RUNTIME{"**Are Copilot Studio and Foundry**\nthe top two within 2 points?"} + RUNTIME -->|No| RESULT + RUNTIME -->|Yes| Q9["**One final distinction**\nWho should operate the agent runtime?"] + Q9 -->|Microsoft-managed platform| Q9A["Prefer Copilot Studio over Foundry\nScores stay unchanged"] + Q9 -->|"⚠️ Engineering-owned runtime"| Q9D["HARD RULE: AB=0, CS=0\nRecommend Foundry"] + Q9A & Q9D --> RESULT + + RESULT["**Recommendation Thresholds**\n12–15: Strong fit\n8–11: Good fit\n4–7: Partial fit\n0–3: Not recommended"] RESULT --> NOTES["**Post-processing**\nCross-question contradiction notes\nWinner-persona mismatch warnings\nTie handling → complementary pairs"] @@ -81,10 +88,12 @@ flowchart TD style Q2C fill:#fff3cd,stroke:#ffc107 style Q4C fill:#fff3cd,stroke:#ffc107 style Q4D fill:#fff3cd,stroke:#ffc107 + style Q9D fill:#fff3cd,stroke:#ffc107 style Q3C fill:#fff3cd,stroke:#ffc107 style Q3F fill:#fff3cd,stroke:#ffc107 style SCORE fill:#e8f4fd,stroke:#0078D4 style PREF fill:#e8f0fe,stroke:#4a86e8 + style RUNTIME fill:#e8f0fe,stroke:#4a86e8 style RESULT fill:#d4edda,stroke:#28a745 style NOTES fill:#f8f0fb,stroke:#6f42c1 ``` diff --git a/docs/SCORING.md b/docs/SCORING.md index 66ee307..f8f0c23 100644 --- a/docs/SCORING.md +++ b/docs/SCORING.md @@ -8,14 +8,14 @@ Reference document for how the APA scoring engine works. All data is driven from |---|---|---| | `agent_builder` | Agent Builder | No-code declarative agents inside Microsoft 365 Copilot | | `m365_copilot` | Microsoft 365 Copilot | Built-in Copilot experiences — Copilot Chat, Search, app-native assistance, and Microsoft-built agents (entry-point wizard only) | -| `copilot_studio` | Copilot Studio | Governed low-code enterprise agents with tools, workflows, triggers, computer use, evaluation, monitoring, and broad publishing | +| `copilot_studio` | Copilot Studio | Managed enterprise agents across the GitHub Copilot, standard, and Copilot chat harnesses, plus deterministic workflows | | `foundry` | Microsoft Foundry | Managed production agent runtime for prompt agents, hosted code agents, custom retrieval, tools, identity, observability, and Azure-scale controls | M365 Copilot is excluded from the scored assessment. It is only recommended via the entry-point wizard (or the legacy `?ft=1` / `?dt=copilot_chat` share links). In the full wizard, `m365_copilot` is always zeroed. ## Non-scored destinations: entry-point wizard (Microsoft 365 Copilot, Cowork & Scout) -Microsoft 365 Copilot, Cowork, and Scout are **not** build platforms — they are ready-made places to *get work done*, not platforms you build on. They are **not** part of the scored wizard, are **not** in `meta.platforms`, and never enter the 0–15 sum. They are reached via the prescreen path **"Help me find the right place to get work done,"** which opens a short **entry-point wizard** ("Where should you get this work done?"). This wizard exists because Microsoft asks end users to choose between too many entry points (Microsoft 365 Copilot vs. Cowork vs. Scout); the wizard resolves that choice from work patterns instead of product names. There is no longer a separate "built-in Microsoft 365 Copilot experience" prescreen tile — that destination now lives inside this wizard. +Microsoft 365 Copilot, Cowork, and Scout are **not** build platforms — they are ready-made places to *get work done*, not platforms you build on. They are **not** part of the scored wizard and never enter the 0–15 sum. They are reached via the prescreen path **"Help me find the right place to get work done,"** which opens a short **entry-point wizard** ("Where should you get this work done?"). This wizard exists because Microsoft asks end users to choose between too many entry points (Microsoft 365 Copilot vs. Cowork vs. Scout); the wizard resolves that choice from work patterns instead of product names. There is no longer a separate "built-in Microsoft 365 Copilot experience" prescreen tile — that destination now lives inside this wizard. **Copilot Chat is not a destination.** Copilot Chat and the built-in agents (Researcher, Analyst, Facilitator, Interpreter, …) are *surfaces of* Microsoft 365 Copilot, not products that compete with it. Staying hands-on therefore always resolves to the single `m365_copilot` card; the task-type answer only selects which surface the card tells you to **Start Here** with, via `recommendations.m365_copilot.start_here` in `apa.yaml` (`chat` or `agents`). @@ -56,18 +56,18 @@ The first question forks the flow: ## Questions and Scoring Matrix -Five questions, each scored 0–3 per platform. Max raw score: **15** (5 × 3). +Five questions, each scored 0–3 per platform. Max raw score: **15** (5 × 3). A conditional, non-scored runtime distinction appears only when Copilot Studio and Foundry are the top two platforms within 2 points. ### Q1 — Who is building this agent? | Option | ID | Agent Builder | CS | Foundry | |---|---|---|---|---| -| Business user / SME — no coding | q1a | **3** | 1 | 0 | +| Business user / SME — no coding | q1a | **3** | 2 | 0 | | Low-code maker / IT pro | q1b | 1 | **3** | 0 | | Professional developer | q1c | 0 | 2 | **3** | -| Data scientist / ML engineer | q1d | 0 | 1 | **3** | +| Data scientist / ML engineer | q1d | 0 | 2 | **3** | -CS gets 2 for q1c because it supports pro developers via YAML authoring and the VS Code extension. +Copilot Studio gets 2 for business users because the GitHub Copilot harness uses natural-language-first authoring, while Agent Builder remains the simpler no-code option. CS also supports developer authoring workflows through YAML and the VS Code extension, but that does not make it a developer-owned code runtime. ### Q8 — Who will use this agent? @@ -95,17 +95,37 @@ Foundry now scores higher for deployment flexibility because Foundry agents can ### Q4 — What should this agent do? -Task complexity is the strongest discriminator between Agent Builder, Copilot Studio, and Foundry. Agent Builder now scores well for lightweight content/data-analysis capabilities enabled in declarative agents, but is still zeroed for action workflows. +Task complexity separates Agent Builder from the enterprise platforms, but no longer cleanly separates Copilot Studio from Foundry. The GitHub Copilot harness gives Copilot Studio adaptive multi-step reasoning, failure recovery, native Office/PDF file work, skills, memory, and multi-tool orchestration. Runtime ownership is therefore resolved through a conditional final distinction when CS and Foundry remain close. | Option | ID | Agent Builder | CS | Foundry | Hard Rule | |---|---|---|---|---|---| | Simple Q&A / lookups | q4a | **3** | **3** | 1 | — | | Conversational (multi-turn) | q4b | 2 | **3** | 2 | — | -| Create/analyze content in Copilot | q4e | **3** | 2 | 2 | — | +| Create/analyze content in Copilot | q4e | **3** | **3** | 2 | — | | Multi-step tasks with actions | q4c | 0 | **3** | **3** | Zeros AB | -| Complex orchestration | q4d | 0 | 2 | **3** | Zeros AB, M365 | +| Complex orchestration | q4d | 0 | **3** | **3** | Zeros AB, M365 | -Foundry gets 1 for q4a because it can do simple Q&A, but is usually overkill for simple knowledge scenarios. It gets 2 for q4e because code interpreter, file search, and hosted agents can support richer content/data-analysis workloads when the team needs developer control. +Foundry gets 1 for q4a because it can do simple Q&A, but is usually overkill for simple knowledge scenarios. Both enterprise platforms can now handle complex orchestration; Foundry becomes the stronger choice when the conditional runtime distinction requires engineering ownership. + +### Conditional runtime distinction — Who should operate the agent runtime? + +This is not part of the score and does not appear for every user. After the five scored questions, APA asks it only when Copilot Studio and Foundry are the top two viable platforms and their scores are within 2 points. + +| Option | ID | Effect | +|---|---|---| +| Microsoft should manage the runtime, sandbox, tools, and Power Platform governance | q9a | Prefer Copilot Studio over Foundry without changing either score | +| Engineering should own the code runtime and infrastructure | q9d | Zero Agent Builder and Copilot Studio; recommend Foundry | + +The previous `q9b` and `q9c` URL values remain accepted and normalize to `q9a`, preserving links created during the earlier six-question schema. + +When Copilot Studio wins, the **Start with this harness** callout is derived from the existing task and deployment answers rather than from the runtime distinction: + +| Scenario signal | Copilot Studio starting point | +|---|---| +| Complex orchestration or native content/file creation | GitHub Copilot harness | +| Event-triggered or multi-step action workflow | Copilot Studio workflow | +| Microsoft 365 Copilot chat + simple Q&A | Copilot chat harness | +| Other conversational scenarios | Standard harness | ### Q3 — What information does this agent need to access? @@ -135,6 +155,7 @@ Hard rules zero out platforms before scores are summed. They represent real plat | q2c (background) | AB | No event-driven or autonomous background runtime | | q3c (direct business system integration) | AB | Cannot directly connect to Dataverse, custom connectors, or business APIs | | q3f (custom retrieval architecture) | AB | Cannot directly use custom RAG, Azure AI Search, private indexes, Foundry IQ, or engineering-managed retrieval systems | +| q9d (developer-owned runtime) | AB, CS | Requires an engineering-owned code runtime, framework, endpoint, network, identity, memory, or retrieval architecture | Additionally, M365 Copilot is always zeroed in the full assessment (hard-coded in JS). @@ -142,13 +163,14 @@ Additionally, M365 Copilot is always zeroed in the full assessment (hard-coded i For each platform not zeroed: sum the scores from all answered questions. Range: 0–15. -### Step 2.5 — Persona preferences (soft overrides) +### Step 2.5 — Ranking preferences (soft overrides) -Persona preferences force one platform above another in ranking regardless of scores. Unlike hard rules, all scores are preserved — the override only affects sort order. A rationale message is displayed as a key factor on the recommendation card. +Ranking preferences force one platform above another without changing scores. A rationale message is displayed as a key factor on the recommendation card. | Trigger | Prefer | Over | Rationale | |---|---|---|---| -| q1d (data scientist / AI-ML) | Copilot Studio | Agent Builder | CS supports curated model selection, evaluations, Foundry IQ integration, code-first development, and flexible orchestration that AB lacks | +| q1d (data scientist / AI-ML) | Copilot Studio | Agent Builder | CS supports curated model selection, evaluations, Foundry IQ integration, managed reasoning, portable skills, and flexible orchestration that AB lacks | +| q9a (Microsoft-managed runtime) | Copilot Studio | Foundry | CS provides the requested managed harness, sandbox, tools, and Power Platform governance | ### Step 3 — Threshold labels @@ -163,13 +185,17 @@ Persona preferences force one platform above another in ranking regardless of sc Platforms are sorted by score descending. The highest-scoring platform is the primary recommendation. The second-highest is shown as "Also consider" when it is viable. -### Step 5 — Tie handling +### Step 5 — Conditional runtime distinction + +If Copilot Studio and Foundry are the top two viable platforms within 2 points and no runtime answer is already present in the URL, ask who should operate the runtime. The answer applies the q9a ranking preference or q9d hard rule. + +### Step 6 — Tie handling When the top two platforms score within **2 points**, they're presented as a complementary pair when that pair is listed in `valid_pairs`. | Pair | Rationale | |---|---| -| Copilot Studio + Foundry | Build in CS, extend with custom code in Foundry | +| Copilot Studio + Foundry | CS for a Microsoft-managed harness, sandbox, tools, and Power Platform governance; Foundry for an engineering-owned runtime and infrastructure | | M365 Copilot + Copilot Studio | M365 Copilot for end users, CS for customization | | Agent Builder + M365 Copilot | AB for Microsoft 365-native agents, M365 for extensibility | @@ -177,10 +203,10 @@ When the top two platforms score within **2 points**, they're presented as a com | Trigger | Platforms | Prefer | Rationale | |---|---|---|---| -| q1c (professional developer) | AB, CS | CS | CS supports code-first authoring via VS Code extension | +| q1c (professional developer) | AB, CS | CS | CS supports managed enterprise orchestration, evaluation, governance, and developer authoring workflows | | q1d (data scientist / AI-ML) | CS, Foundry | CS | CS provides a faster path to production agents | -### Step 6 — Cross-question notes +### Step 7 — Cross-question notes Contextual warning banners when answer combinations are logically contradictory: @@ -188,49 +214,49 @@ Contextual warning banners when answer combinations are logically contradictory: |---|---| | q2c + q4a | Background agent doing simple Q&A — contradictory | | q8b + q2a | External users in Microsoft 365 Copilot chat — external users can't access your tenant | -| q1a + q4d | Business user wants complex orchestration — requires dev skills | +| q1a + q4d | Natural-language authoring lowers the build barrier, but production governance, permissions, evaluation, monitoring, and cost controls still need IT involvement | | q1a + q3c | Business user needs direct business system integration — requires technical expertise | | q1a + q3f | Business user needs custom retrieval architecture — requires engineering expertise | -### Step 7 — Winner-persona mismatch +### Step 8 — Winner-persona mismatch When Foundry wins but the builder is a business user (q1a), a banner advises partnering with a development team. ## Distribution Analysis -Across all 1,920 possible answer combinations: +Across all 1,920 possible scored-answer combinations, before the conditional runtime choice: | Platform | Wins | % | |---|---:|---:| -| Copilot Studio | 1,590 | 82.8% | -| Foundry | 270 | 14.1% | -| Agent Builder | 60 | 3.1% | +| Copilot Studio | 1,758 | 91.6% | +| Foundry | 122 | 6.4% | +| Agent Builder | 40 | 2.1% | -**Exact top-score ties:** 283 combos (14.7%) — 253 are CS/Foundry, 30 are AB/CS. **Close-score cases within 2 points:** 1,120 combos (58.3%) — most are CS/Foundry, reflecting the intentional overlap between Copilot Studio's governed low-code runtime and Foundry's developer-controlled runtime. +**Exact top-score ties:** 126 combos (6.6%) — 98 are CS/Foundry and 28 are AB/CS. **Close-score cases within 2 points:** 963 combos (50.2%) — 847 are CS/Foundry and 116 are AB/CS. The conditional runtime distinction appears for those 847 CS/Foundry cases (44.1% of scored scenarios), resolving the most meaningful ambiguity without lengthening every assessment. ### When Agent Builder wins -AB now wins beyond the old SharePoint/OneDrive-only path. Its sweet spot is: **business user or low-code maker, small team or undecided internal audience, Microsoft 365 Copilot surface, Q&A/conversation/content-analysis, and Microsoft 365, web/uploaded, or connector-backed knowledge**. +AB's sweet spot is: **business user or low-code maker, small team or undecided internal audience, Microsoft 365 Copilot surface, lightweight Q&A/content work, and Microsoft 365/web/uploaded knowledge**. Agent Builder still loses whenever the user needs external publishing, custom app deployment, background execution, direct business system integration, custom retrieval architecture, or action workflows that update external systems. ### When Foundry wins -Foundry wins when answers include strong technical or production-runtime signals: pro dev or ML persona (q1c/q1d), custom app or multi-surface deployment (q2b/q2d), complex or long-running orchestration (q4d), custom retrieval architecture (q3f), external-facing scenarios, or a need for managed endpoints, hosted code agents, private networking, tracing, evaluation, and full Azure control. Copilot Studio still ties or beats Foundry for event-triggered workflows and business APIs unless the scenario clearly needs full-code control. +Foundry wins directly when several technical signals reinforce the need for developer control. In close cases, selecting engineering ownership in the conditional distinction makes Foundry the required platform. Complex orchestration alone no longer makes Foundry the default because the GitHub Copilot harness now covers managed adaptive execution. ### Copilot Studio dominance -CS remains the default recommendation for most combinations because it bridges Agent Builder's no-code Microsoft 365-native scenarios and Foundry's full-code scenarios. It wins when the user needs broader internal or external deployment, actions, branching workflows, event triggers, enterprise governance, Dataverse/custom connectors, MCP tools, computer use, evaluation, monitoring, or a safer path when scope is undecided. +CS remains the default recommendation for most combinations because it spans four managed execution patterns: GitHub Copilot harness agents for adaptive work, standard harness agents for predictable conversations, Copilot chat harness agents for Microsoft 365 knowledge extensions, and workflows for deterministic automation. It wins when Microsoft should manage execution while the solution still needs enterprise governance, tools, actions, evaluation, monitoring, and broad deployment. ### Score ranges when winning | Platform | Min | Max | Avg | |---|---:|---:|---:| -| Agent Builder | 11 | 15 | 12.8 | -| Copilot Studio | 9 | 15 | 12.4 | -| Foundry | 10 | 15 | 12.7 | +| Agent Builder | 12 | 15 | 13.3 | +| Copilot Studio | 11 | 15 | 13.2 | +| Foundry | 12 | 15 | 13.2 | -No combination produces a "best platform" below 8, so every user gets at least a "Good fit" recommendation. +The conditional runtime choice changes ranking or eligibility without changing the displayed raw scores. ## Cross-question note frequency @@ -241,6 +267,5 @@ No combination produces a "best platform" below 8, so every user gets at least a | BizUser + Orchestrate | 96 | 5.0% | | BizUser + Business APIs | 80 | 4.2% | | BizUser + Custom retrieval | 80 | 4.2% | -| Foundry + BizUser (persona mismatch) | 19 | 1.0% | Notes are not mutually exclusive — a single combo can trigger multiple notes. diff --git a/tests/e2e/harness-guidance.spec.js b/tests/e2e/harness-guidance.spec.js new file mode 100644 index 0000000..6b89a3e --- /dev/null +++ b/tests/e2e/harness-guidance.spec.js @@ -0,0 +1,50 @@ +// @ts-check +const { test, expect } = require('@playwright/test'); + +function resultUrl(overrides = {}) { + const answers = { + q1: 'q1b', + q8: 'q8c', + q2: 'q2a', + q4: 'q4b', + q9: 'q9a', + q3: 'q3b', + ...overrides, + }; + const params = new URLSearchParams({ + ...answers, + r: 'copilot_studio', + d: '20260819', + mode: 'card', + }); + return `/?${params.toString()}`; +} + +async function expectHarness(page, key, label) { + await expect(page.locator('#recommendation-section')).toBeVisible(); + const guidance = page.locator(`#rec-primary-card .rec-harness-guidance[data-harness="${key}"]`); + await expect(guidance).toBeVisible(); + await expect(guidance).toContainText(label); +} + +test.describe('Copilot Studio harness guidance', () => { + test('recommends GitHub Copilot harness for managed adaptive work', async ({ page }) => { + await page.goto(resultUrl({ q9: 'q9a', q4: 'q4d' })); + await expectHarness(page, 'github_copilot', 'GitHub Copilot harness'); + }); + + test('recommends standard harness for predictable conversations', async ({ page }) => { + await page.goto(resultUrl({ q9: 'q9a', q4: 'q4b' })); + await expectHarness(page, 'standard', 'Standard harness'); + }); + + test('recommends workflow for deterministic action automation', async ({ page }) => { + await page.goto(resultUrl({ q9: 'q9a', q2: 'q2c', q4: 'q4c' })); + await expectHarness(page, 'workflow', 'Copilot Studio workflow'); + }); + + test('recommends Copilot chat harness for internal knowledge extensions', async ({ page }) => { + await page.goto(resultUrl({ q9: 'q9a', q4: 'q4a', q3: 'q3a' })); + await expectHarness(page, 'copilot_chat', 'Copilot chat harness'); + }); +}); diff --git a/tests/e2e/share-buttons.spec.js b/tests/e2e/share-buttons.spec.js index fd17650..8502acf 100644 --- a/tests/e2e/share-buttons.spec.js +++ b/tests/e2e/share-buttons.spec.js @@ -22,6 +22,10 @@ async function completeWizard(page) { await page.locator('#next-btn').click(); } + await expect(page.locator('#question-counter')).toHaveText('One final distinction'); + await page.locator('#options-list .option-card').first().click(); + await page.locator('#next-btn').click(); + await expect(page.locator('#recommendation-section')).toBeVisible(); } @@ -45,6 +49,7 @@ test.describe('Share Button', () => { const clipboardText = await page.evaluate(() => navigator.clipboard.readText()); expect(clipboardText).toContain('q1=q1c'); expect(clipboardText).toContain('q8=q8a'); + expect(clipboardText).toContain('q9=q9a'); expect(clipboardText).toContain('mode=card'); expect(clipboardText).toContain('r='); expect(clipboardText).toContain('d='); @@ -65,7 +70,7 @@ test.describe('Share Button', () => { }); test('share link from URL-loaded results contains all answers', async ({ page }) => { - const params = 'q1=q1c&q8=q8a&q2=q2d&q4=q4b&q3=q3b&r=copilot_studio&d=20260401&mode=card'; + const params = 'q1=q1c&q8=q8a&q2=q2d&q4=q4b&q9=q9a&q3=q3b&r=copilot_studio&d=20260401&mode=card'; await page.goto(`/?${params}`); await expect(page.locator('#recommendation-section')).toBeVisible(); @@ -79,6 +84,7 @@ test.describe('Share Button', () => { expect(shareUrl).toContain('q8=q8a'); expect(shareUrl).toContain('q2=q2d'); expect(shareUrl).toContain('q4=q4b'); + expect(shareUrl).toContain('q9=q9a'); expect(shareUrl).toContain('q3=q3b'); expect(shareUrl).toContain('mode=card'); }); diff --git a/tests/e2e/shared-link.spec.js b/tests/e2e/shared-link.spec.js index 59cdf3a..52ff991 100644 --- a/tests/e2e/shared-link.spec.js +++ b/tests/e2e/shared-link.spec.js @@ -2,8 +2,9 @@ const { test, expect } = require('@playwright/test'); // Pre-built URL params that produce a Copilot Studio recommendation: -// q1=q1c (pro dev), q8=q8a (internal), q2=q2d (multiple), q4=q4b (convo), q3=q3b (other systems) -const SHARED_PARAMS = 'q1=q1c&q8=q8a&q2=q2d&q4=q4b&q3=q3b&r=copilot_studio&d=20260401&mode=card'; +// q1=q1c (pro dev), q8=q8a (internal), q2=q2d (multiple), q4=q4b (convo), +// q9=q9a (managed adaptive execution), q3=q3b (other systems) +const SHARED_PARAMS = 'q1=q1c&q8=q8a&q2=q2d&q4=q4b&q9=q9a&q3=q3b&r=copilot_studio&d=20260401&mode=card'; test.describe('Shared Link Loading', () => { test('loads results directly from URL parameters', async ({ page }) => { @@ -31,6 +32,7 @@ test.describe('Shared Link Loading', () => { // Primary recommendation card should exist await expect(page.locator('#rec-primary-card .rec-card')).toBeVisible(); + await expect(page.locator('#rec-primary-card .rec-harness-guidance')).toContainText('Standard harness'); }); test('handles invalid URL params gracefully', async ({ page }) => { @@ -42,7 +44,7 @@ test.describe('Shared Link Loading', () => { }); test('handles partial URL params with schema drift', async ({ page }) => { - // Only 3 of 5 answers — should still load results but flag drift + // Only 3 of 6 answers — should still load results but flag drift const partialParams = 'q1=q1c&q8=q8a&q2=q2d&r=copilot_studio&d=20260401&mode=card'; await page.goto(`/?${partialParams}`); @@ -52,4 +54,27 @@ test.describe('Shared Link Loading', () => { // Drift note should be visible await expect(page.locator('#decision-card-drift')).toBeVisible(); }); + + test('preserves pre-q9 links without treating the conditional question as schema drift', async ({ page }) => { + const legacyParams = 'q1=q1c&q8=q8a&q2=q2d&q4=q4b&q3=q3b&r=copilot_studio&d=20260401&mode=card'; + await page.goto(`/?${legacyParams}`); + + await expect(page.locator('#recommendation-section')).toBeVisible(); + await expect(page.locator('#decision-card-drift')).toBeHidden(); + await expect(page.locator('#rec-primary-card .rec-harness-guidance')).toContainText('Standard harness'); + }); + + test('maps previous managed q9 options to the new runtime distinction', async ({ page }) => { + const previousParams = 'q1=q1c&q8=q8a&q2=q2d&q4=q4b&q9=q9c&q3=q3b&r=copilot_studio&d=20260819&mode=card'; + await page.goto(`/?${previousParams}`); + + await expect(page.locator('#recommendation-section')).toBeVisible(); + await expect(page.locator('#decision-card-drift')).toBeHidden(); + + const shareUrl = await page.evaluate(() => { + // @ts-ignore — global function + return buildShareableURL(); + }); + expect(shareUrl).toContain('q9=q9a'); + }); }); diff --git a/tests/e2e/wizard-completion.spec.js b/tests/e2e/wizard-completion.spec.js index 99e707c..9e38cc0 100644 --- a/tests/e2e/wizard-completion.spec.js +++ b/tests/e2e/wizard-completion.spec.js @@ -2,7 +2,9 @@ const { test, expect } = require('@playwright/test'); // Answers that produce a clear Copilot Studio winner: -// q1c (pro dev), q8a (internal), q2d (multiple places), q4b (conversation), q3b (other systems) +// q1c (pro dev), q8a (internal), q2d (multiple places), q4b (conversation), +// q3b (other systems). Copilot Studio and Foundry finish within two points, +// so the conditional runtime distinction appears. const WIZARD_ANSWERS = { q1: 'q1c', // Professional developer q8: 'q8a', // Internal employees @@ -11,7 +13,7 @@ const WIZARD_ANSWERS = { q3: 'q3b', // Other business systems }; -// The YAML question order is: q1, q8, q2, q4, q3 +// The scored YAML question order is: q1, q8, q2, q4, q3 const QUESTION_ORDER = ['q1', 'q8', 'q2', 'q4', 'q3']; // Map question IDs to option indices (0-based) for clicking @@ -38,7 +40,7 @@ test.describe('Wizard Completion', () => { await page.locator('#prescreen-no').click(); await expect(page.locator('#assessment-section')).toBeVisible(); - // Answer all 5 questions + // Answer all questions for (let i = 0; i < QUESTION_ORDER.length; i++) { const qId = QUESTION_ORDER[i]; const optIdx = OPTION_INDEX[qId]; @@ -59,11 +61,17 @@ test.describe('Wizard Completion', () => { await page.locator('#next-btn').click(); } + await expect(page.locator('#question-counter')).toHaveText('One final distinction'); + await expect(page.locator('#question-title')).toHaveText('Who should operate the agent runtime?'); + await page.locator('#options-list .option-card').first().click(); + await page.locator('#next-btn').click(); + // Verify recommendation section is visible await expect(page.locator('#recommendation-section')).toBeVisible(); // Verify a platform card is rendered await expect(page.locator('#rec-primary-card .rec-card')).toBeVisible(); + await expect(page.locator('#rec-primary-card .rec-harness-guidance')).toContainText('Standard harness'); // Verify score breakdown exists await expect(page.locator('#rec-score-comparison')).toBeVisible(); @@ -115,4 +123,54 @@ test.describe('Wizard Completion', () => { // Next button should now be enabled await expect(page.locator('#next-btn')).toBeEnabled(); }); + + test('developer-owned runtime requirement disqualifies Copilot Studio', async ({ page }) => { + const params = [ + 'q1=q1c', + 'q8=q8a', + 'q2=q2d', + 'q4=q4d', + 'q9=q9d', + 'q3=q3f', + 'r=foundry', + 'd=20260819', + 'mode=card', + ].join('&'); + + await page.goto(`/?${params}`); + + await expect(page.locator('#rec-primary-card .rec-platform-name')).toContainText('Microsoft Foundry'); + await expect(page.locator('#rec-score-comparison')).toContainText('Developer-owned runtime'); + }); + + test('conditional engineering ownership recommends Foundry', async ({ page }) => { + await page.goto('/'); + await page.locator('#start-btn').click(); + await page.locator('#prescreen-no').click(); + + for (const questionId of QUESTION_ORDER) { + await page.locator('#options-list .option-card').nth(OPTION_INDEX[questionId]).click(); + await page.locator('#next-btn').click(); + } + + await expect(page.locator('#question-counter')).toHaveText('One final distinction'); + await page.locator('#options-list .option-card').nth(1).click(); + await page.locator('#next-btn').click(); + + await expect(page.locator('#rec-primary-card .rec-platform-name')).toContainText('Microsoft Foundry'); + }); + + test('skips the runtime distinction when Copilot Studio and Foundry are not close', async ({ page }) => { + await page.goto('/'); + await page.locator('#start-btn').click(); + await page.locator('#prescreen-no').click(); + + for (const optionIndex of [0, 0, 0, 0, 0]) { + await page.locator('#options-list .option-card').nth(optionIndex).click(); + await page.locator('#next-btn').click(); + } + + await expect(page.locator('#recommendation-section')).toBeVisible(); + await expect(page.locator('#question-title')).not.toHaveText('Who should operate the agent runtime?'); + }); });