From 6544d9f5456a5454b557b7a8ef1868eff0033504 Mon Sep 17 00:00:00 2001 From: Denise <43950360+denise-msft@users.noreply.github.com> Date: Fri, 17 Apr 2026 16:17:19 -0700 Subject: [PATCH 1/2] docs: overhaul feature documentation with accurate dependencies Updated 15 documentation files to accurately reflect the March 2026 release: - TECHDETAILS.md: Expanded from 3 to 9 connection references, 5 to 20 environment variables - Added Technical Details sections to all feature docs with: - Connection references and connector IDs - Environment variables with schema names and defaults - Cloud flows with trigger types and descriptions - Dataverse tables with logical names - DLP connector requirements - Fixed missing connectors in Compliance Hub prerequisites (Office 365 Groups) - Added Feature-to-Component Matrix in TECHDETAILS.md All changes verified against solution.xml and customizations.xml from the actual solution package. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ADAPTIVE_CARDS_GALLERY.md | 105 ++- AGENT_INSIGHTS_HUB.md | 1144 ++++++++++++++++-------------- AGENT_INVENTORY.md | 183 +++-- AGENT_REVIEW_TOOL.md | 95 ++- AGENT_VALUE_SUMMARY_DASHBOARD.md | 231 +++--- AUTOMATED_TESTING.md | 83 ++- COMPLIANCE_HUB.md | 871 +++++++++++++---------- CONVERSATION_ANALYZER.md | 101 +-- CONVERSATION_KPIS.md | 84 ++- FILE_SYNCHRONIZATION.md | 82 ++- PROMPT_ADVISOR.md | 214 +++--- SETUP_WIZARD.md | 87 ++- TECHDETAILS.md | 203 ++++-- TESTING_CAPABILITIES.md | 96 ++- WEBCHAT_PLAYGROUND.md | 174 +++-- 15 files changed, 2201 insertions(+), 1552 deletions(-) diff --git a/ADAPTIVE_CARDS_GALLERY.md b/ADAPTIVE_CARDS_GALLERY.md index 2aff817b..1de06760 100644 --- a/ADAPTIVE_CARDS_GALLERY.md +++ b/ADAPTIVE_CARDS_GALLERY.md @@ -1,35 +1,70 @@ -# Adaptive cards gallery -## Setup Note - -To get the Adaptive cards gallery working after you install the Copilot Studio Kit solution, you will need to go to the ````Adaptive Card Gallery```` Agent that was installed in the solution file. - -* Open Copilot Studio in your current environment -* Open the ````Adaptive Card Gallery```` Agent that was deployed in the solution file -* Publish the ````Adaptive Card Gallery```` Agent -* Open the mobile app channel to view the published token endpoint for this Agent -* Copy the token endpoint and then update the ````Agent Token Endpoint```` environment variable - * You can find this environment variable by opening up the Default Solution and then filtering to the Environment Variables. - * Paste the token endpoint value into the ````Agent Token Endpoint```` -* Reload the Adaptive cards gallery page and you should be up and running now! - -## About adaptive cards - -Adaptive Cards are a lightweight UI framework that enables developers to create dynamic and interactive content that can be rendered consistently across different platforms and applications. They are defined using a JSON-based schema, allowing for flexibility in layout and interactivity while adapting to the look and feel of the host application. - -Learn more about adaptive cards from [here](https://adaptivecards.io/). - -## Adaptive cards gallery overview - -Adaptive cards gallery is a collection of adaptive cards demonstrating different scenarios with the focus being on Copilot Studio custom agent interactions. - -![adaptivecards1](https://github.com/user-attachments/assets/e35288e3-77e5-4f76-8b57-4406332d4544) - -Each sample adaptive card comes with a template and sample data. Adaptive cards are all implemented in a Copilot Studio custom agent as well, which can be activated for live preview, by entering its token endpoint in an environment variable **Agent Token Endpoint**. **Open in Copilot Studio** opens a topic associated with the selected adaptive card from the custom agent into a new tab so that backend implementation details can be studied and copied. - -![adaptivecards2](https://github.com/user-attachments/assets/d4e7ca66-2e7c-497c-94e8-0695e12253dd) - -> [!NOTE] -> By default, emulator is used to preview the adaptive cards. To preview the adaptive cards using a real Copilot Studio webchat, publish the "**Adaptive Card Gallery**"-agent that ships with Copilot Studio Kit, and set the environment variable **cat_AgentTokenEndpoint** to its Token Endpoint. The token endpoint of Copilot Studio custom agent can be copied from Channels -> Mobile app. - - -Back to the [landing page](./README.md#power-cat-copilot-studio-kit) +# Adaptive cards gallery +## Setup Note + +To get the Adaptive cards gallery working after you install the Copilot Studio Kit solution, you will need to go to the ````Adaptive Card Gallery```` Agent that was installed in the solution file. + +* Open Copilot Studio in your current environment +* Open the ````Adaptive Card Gallery```` Agent that was deployed in the solution file +* Publish the ````Adaptive Card Gallery```` Agent +* Open the mobile app channel to view the published token endpoint for this Agent +* Copy the token endpoint and then update the ````Agent Token Endpoint```` environment variable + * You can find this environment variable by opening up the Default Solution and then filtering to the Environment Variables. + * Paste the token endpoint value into the ````Agent Token Endpoint```` +* Reload the Adaptive cards gallery page and you should be up and running now! + +## About adaptive cards + +Adaptive Cards are a lightweight UI framework that enables developers to create dynamic and interactive content that can be rendered consistently across different platforms and applications. They are defined using a JSON-based schema, allowing for flexibility in layout and interactivity while adapting to the look and feel of the host application. + +Learn more about adaptive cards from [here](https://adaptivecards.io/). + +## Adaptive cards gallery overview + +Adaptive cards gallery is a collection of adaptive cards demonstrating different scenarios with the focus being on Copilot Studio custom agent interactions. + +![adaptivecards1](https://github.com/user-attachments/assets/e35288e3-77e5-4f76-8b57-4406332d4544) + +Each sample adaptive card comes with a template and sample data. Adaptive cards are all implemented in a Copilot Studio custom agent as well, which can be activated for live preview, by entering its token endpoint in an environment variable **Agent Token Endpoint**. **Open in Copilot Studio** opens a topic associated with the selected adaptive card from the custom agent into a new tab so that backend implementation details can be studied and copied. + +![adaptivecards2](https://github.com/user-attachments/assets/d4e7ca66-2e7c-497c-94e8-0695e12253dd) + +> [!NOTE] +> By default, emulator is used to preview the adaptive cards. To preview the adaptive cards using a real Copilot Studio webchat, publish the "**Adaptive Card Gallery**"-agent that ships with Copilot Studio Kit, and set the environment variable **cat_AgentTokenEndpoint** to its Token Endpoint. The token endpoint of Copilot Studio custom agent can be copied from Channels -> Mobile app. + +## Technical Details + +### Connection References + +| Connection Reference | Connector | +|:---------------------|:----------| +| Copilot Studio Kit - Dataverse | Microsoft Dataverse | + +### Environment Variables + +| Display Name | Schema Name | Required | Description | +|:-------------|:------------|:---------|:------------| +| Agent Token Endpoint | cat_AgentTokenEndpoint | Optional | Token endpoint for live webchat preview with the Adaptive Card Gallery agent | + +### Cloud Flows + +None — this component has no associated cloud flows. + +### Dataverse Tables + +| Display Name | Schema Name | Purpose | +|:-------------|:------------|:--------| +| Agent Card | cat_CopilotCard | Stores adaptive card templates and sample data | + +### DLP Configuration + +This component requires the **Microsoft Dataverse** connector. Ensure your DLP policies allow this connector in the target environment. + +### Canvas App + +| Display Name | Schema Name | +|:-------------|:------------| +| Adaptive Cards Gallery | cat_adaptivecards | + +> **Note:** This component includes a Copilot Studio agent named **Adaptive Card Gallery** that must be published for the live webchat preview to function. + +Back to the [landing page](./README.md#power-cat-copilot-studio-kit) diff --git a/AGENT_INSIGHTS_HUB.md b/AGENT_INSIGHTS_HUB.md index 08f6cb6a..e8979fb8 100644 --- a/AGENT_INSIGHTS_HUB.md +++ b/AGENT_INSIGHTS_HUB.md @@ -1,540 +1,604 @@ -# Agent Insights Hub (Preview) - -## Table of Contents - -- [1. Overview](#1-overview) -- [2. Prerequisites](#2-prerequisites) -- [3. Setup Instructions](#3-setup-instructions) - - [3.1 Integrate Copilot Studio with Application Insights](#31-integrate-copilot-studio-with-application-insights) - - [3.2 Register Azure App for Application Insights Access](#32-register-azure-app-for-application-insights-access) - - [3.3 Run Agent Inventory Sync](#33-run-agent-inventory-sync) - - [3.4 Add an Agent Configuration](#34-add-an-agent-configuration) - - [3.5 Run Your First Sync](#35-run-your-first-sync) -- [4. How It Works — End-to-End Flow](#4-how-it-works--end-to-end-flow) -- [5. Action Buttons](#5-action-buttons) -- [6. Filters](#6-filters) -- [7. Tabs](#7-tabs) - - [7.1 Overview](#71-overview) - - [7.2 Daily Metrics](#72-daily-metrics) - - [7.3 Topics](#73-topics) - - [7.4 Tools](#74-tools) - - [7.5 Actions](#75-actions) - - [7.6 Errors](#76-errors) - - [7.7 Transcripts](#77-transcripts) - - [7.8 Usage History](#78-usage-history) - - [7.9 Sync Logs (Troubleshooting)](#79-sync-logs-troubleshooting) - - [7.10 Transcripts Staging (Troubleshooting)](#710-transcripts-staging-troubleshooting) -- [8. Data Sources & Differences](#8-data-sources--differences) -- [9. Troubleshooting](#9-troubleshooting) - ---- - -## 1. Overview - -**Agent Insights Hub** is a comprehensive analytics and monitoring dashboard for Microsoft Copilot Studio agents. It provides real-time visibility into agent performance, conversation metrics, topic analytics, tool execution, and error tracking. - -Built as a Power Apps Code App using React and Fluent UI v9, it connects directly to Dataverse to display aggregated telemetry data imported from three sources: - -| Data Source | Tabs Powered | What It Captures | -|---|---|---| -| **Azure Application Insights** | Overview, Daily Metrics, Topics, Tools, Actions, Errors | Telemetry logs from agent conversations — response times, topic triggers, tool calls, errors | -| **Conversation Transcripts** | Transcripts | Session-level data from Copilot Studio — engagement, resolution, escalation, CSAT, feedback | -| **Power Platform Admin Center** | Usage History | Copilot credit consumption — billed and non-billed credits by agent, environment, and feature | - - ---- - -## 2. Prerequisites - -Before using Agent Insights Hub, ensure the following are in place: - -1. **Agent Inventory Sync must be run** — The Agent Inventory sync job must have completed at least once so that agent details are available for selection. - -2. **Application Insights integration** — Your Copilot Studio agents must be connected to an Azure Application Insights resource. See [Section 3.1](#31-integrate-copilot-studio-with-application-insights). - -3. **Azure App Registration** — An Azure AD app must be registered with access to the Application Insights telemetry data. Follow the setup guide: [Enable Application Insights Support](https://github.com/microsoft/Power-CAT-Copilot-Studio-Kit/blob/main/ENABLE-APPINSIGHTS.md#enable-application-insights-support). - -4. **App sharing** — The owner of the Copilot Studio Kit code app (or the System Administrator who installed kit) must share the app with other users on their team. Follow these steps: - 1. Go to [make.powerapps.com](https://make.powerapps.com/) - 2. Click **Apps** in the left navigation - 3. Search for **Copilot Studio Kit** and locate the one with type **Code** (not the Model-driven app) - 4. Click the **three dots (...)** menu next to the app - 5. Select **Share** - 6. Search for and add the users or security groups who need access - 7. Click **Share** to confirm - - For detailed instructions, see: [Share a canvas app with your organization](https://learn.microsoft.com/en-us/power-apps/maker/canvas-apps/share-app). This step will be automated in a future release. - ---- - -## 3. Setup Instructions - -### 3.1 Integrate Copilot Studio with Application Insights - -To collect telemetry from your Copilot Studio agents, you must connect them to Azure Application Insights: - -1. Open [Copilot Studio](https://copilotstudio.microsoft.com) and select your agent -2. Go to **Settings** > **Advanced** > **Application Insights** -3. Enter your Application Insights **Connection String** (found in Azure Portal > Application Insights > Overview > Connection String) -4. Click **Save** - -For detailed instructions, refer to the official Microsoft documentation: -[Connect your copilot to Application Insights](https://learn.microsoft.com/en-us/microsoft-copilot-studio/advanced-bot-framework-composer-capture-telemetry) - -### 3.2 Register Azure App for Application Insights Access - -The sync flow needs programmatic access to query Application Insights telemetry. This requires an Azure AD app registration with appropriate permissions. - -Follow the complete setup guide: **[Enable Application Insights Support](https://github.com/microsoft/Power-CAT-Copilot-Studio-Kit/blob/main/ENABLE-APPINSIGHTS.md#enable-application-insights-support)** - -This guide covers: -- Creating an Azure AD app registration -- Granting the app access to your Application Insights resource -- Obtaining the Application ID, Tenant ID, Client ID, and Client Secret -- Configuring secret storage (Dataverse only currently; Key Vault support planned for a future release) - -### 3.3 Run Agent Inventory Sync - -Before adding agents to the Insights Hub, ensure the **Agent Inventory** sync has been run at least once. This populates the agent list that appears in the "Add Agent" dialog. For detailed instructions, see: [Agent Inventory Documentation](https://github.com/microsoft/Power-CAT-Copilot-Studio-Kit/blob/main/AGENT_INVENTORY.md). - -### 3.4 Add an Agent Configuration - -1. Open Agent Insights Hub -2. Click the **Add agent** button -3. Fill in the configuration form: - -| Field | Description | -|---|---| -| **Agent** | Select an agent from the inventory. This auto-fills Name, Agent Name, Agent ID, and Dataverse URL. | -| **Name** | A friendly display name for this configuration. Must be unique. | -| **Agent Name** | The name of the Copilot Studio agent. Auto-filled when an agent is selected. Not visible when adding a new agent. | -| **Agent ID** | The unique identifier (GUID) of the Copilot Studio agent. Found in Copilot Studio under Settings > Session Info. | -| **Dataverse URL** | The base URL of the Dataverse environment (e.g., `https://org.crm.dynamics.com`). | -| **Application Insights App ID** | The Application ID from your Azure Application Insights resource (Azure Portal > Application Insights > API Access). | -| **Tenant ID** | The Azure AD Tenant ID where the Application Insights resource is registered. | -| **Client ID** | The Client ID of the Azure AD app registration used to authenticate with Application Insights. | -| **Secret Location** | Currently supports Dataverse storage only. Key Vault storage will be supported in a future release. | -| **Secret** | The client secret for the Azure AD app registration. | -| **KPI Source** | Select which data sources to use: Application Insights, Conversation Transcripts, Usage History. Select one, two or all three options based on requirement. | -| **Capture User Details** | When enabled, user identity information (User name interacted with agent) will be captured from telemetry logs. | -| **Capture User Feedback** | When enabled, end-user satisfaction feedback will be collected. | - -4. Click **Save** - -### 3.5 Run Your First Sync - -After adding an agent configuration: - -1. Click the **Sync** button -2. In the Generate Metrics dialog: - - Select a configuration (or "All Agents" to sync all at once) - - Set the **Start Date** and **End Date** for historical data extraction (up to 365 days back) -3. Click **Run** -4. The sync triggers a Cloud Flow that: - - Queries Application Insights for telemetry data - - Fetches Conversation Transcripts from Copilot Studio - - Processes and aggregates the data into KPI records - - Writes the results to Dataverse tables -5. Monitor progress in the **Sync Logs** tab - ---- - -## 4. How It Works — End-to-End Flow - -```mermaid -flowchart TD - A[Add Agent Configuration] --> B{Two Sync Modes} - - B --> S[Scheduled Sync - Runs Daily] - B --> O[On-Demand Sync - Via Sync Button] - - S --> S1[Pull Last 24h App Insights Telemetry] - S --> S2[Pull Last 24h Conversation Transcripts] - - O --> O1[Pull App Insights Telemetry for Selected Date Range] - O --> O2[Pull Conversation Transcripts for Selected Date Range] - - S1 --> P1[Process & Aggregate Data] - S2 --> P1 - O1 --> P2[Process & Aggregate Data] - O2 --> P2 - - P1 --> J[Write to Dataverse Tables] - P2 --> J - - J --> K[Daily Metrics Table] - J --> L[Topic Metrics Table] - J --> M[Tool Metrics Table] - J --> N[Action Metrics Table] - J --> Q[Error Details Table] - J --> T[Transcript Metrics Table] - - K --> R[Agent Insights Hub] - L --> R - M --> R - N --> R - Q --> R - T --> R - - R --> Tab1[Overview Tab] - R --> Tab2[Daily Metrics Tab] - R --> Tab3[Topics Tab] - R --> Tab4[Tools Tab] - R --> Tab5[Actions Tab] - R --> Tab6[Errors Tab] - R --> Tab7[Transcripts Tab] - R --> Tab8[Usage History Tab] -``` - -### Sync Modes - -| Mode | Trigger | Data Range | Use Case | -|---|---|---|---| -| **Scheduled** | Runs automatically every day | Last 24 hours | Keeps metrics up to date with daily incremental syncs | -| **On-Demand** | User clicks **Sync** button | Custom start and end date (up to 365 days) | Backfill historical data or re-sync a specific date range | - ---- - -## 5. Action Buttons - -These buttons appear on the Overview section header and in empty state screens: - -| Button | Description | -|---|---| -| **Sync** | Opens the Generate Metrics dialog to trigger a historical data sync for one or all agents. Select a date range (up to 365 days) and click Run. | -| **Add agent** | Opens the Add Agent dialog to create a new agent configuration with Application Insights credentials and KPI source selection. | -| **Show agents** | Opens a dialog listing all configured agents with their details, KPI sources, and status. Allows editing, cloning, and deleting configurations. | - ---- - -## 6. Filters - -A global filter bar appears on all tabs except Sync Logs, Transcripts Staging, and Usage History (which has its own filters). - -| Filter | Options | Default | Description | -|---|---|---|---| -| **Agent** | All agents, or a specific configured agent | All agents | Filters data to a specific agent configuration. | -| **Date Range** | Last 7 / 30 / 90 / 180 / 365 days, Custom range | Last 90 days | Controls the time period for all metrics. Custom range shows start/end date pickers. | -| **Channel** | All channels, Teams, Web Chat, Direct Line, Test Panel, Autonomous, Published Engine, Mobile, WhatsApp, Unknown | All channels | Filters by the communication channel used for conversations. | -| **Data Mode** | Production, Test data, All data | Production | Filters by data source type. Production shows real user data; Test data shows test panel interactions. | - -**Usage History** has its own independent filters: - -| Filter | Description | -|---|---| -| **Environment** | Filter by Power Platform environment | -| **Agent** | Filter by agent (derived from usage data, not configurations) | -| **Date** | Month to date, previous months, or last 6 months | -| **Feature** | Filter by Copilot feature name | - ---- - -## 7. Tabs - -### 7.1 Overview - -**Data Source**: Application Insights telemetry (Daily Metrics, Topic Metrics, Tool Metrics, Error Details tables) - -The Overview tab provides a high-level summary of agent performance. It displays 7 KPI cards with trend indicators comparing the current period against the previous period of equal length. - -#### KPI Cards - -| KPI | Calculation | Display | Trend | -|---|---|---|---| -| **Conversations** | Sum of Daily Conversation Count across all days | Number | Higher is better | -| **Average DAU** | Sum of Daily Unique Users / Number of distinct days in the period | Number (rounded) | Higher is better | -| **Responses** | Sum of Total Responses (bot messages sent) | Number | Higher is better | -| **Avg Response** | Sum of Total Response Time (ms) / Sum of Total Responses | Seconds (if >= 1s) or ms | Lower is better (inverted trend) | -| **Duration** | Sum of Total Duration (ms) / Sum of Conversations With Duration Data / 60,000 | Minutes (1 decimal) | Lower is better (inverted trend) | -| **Tool Success** | (Sum of Tool Success Count / Sum of Tool Call Count) x 100 | Percentage | Higher is better | -| **Errors** | Sum of Error Count | Number | Lower is better (inverted trend) | - -**Trend Calculation**: Compares the current period value against the previous period of equal length. For example, if viewing the last 30 days, the trend compares against the 30 days before that. When the previous period has no data but the current period does, it shows "New" instead of a percentage. - -#### Charts - -| Chart | Type | Description | -|---|---|---| -| **Top Topics** | Donut | Top 10 topics ranked by trigger count | -| **Slowest Topics** | Horizontal Bar | Top 5 topics by weighted average duration (Total Duration / Duration Count per topic) | -| **Conversations by Channel** | Donut | Distribution of conversations across channels (Teams, Web Chat, etc.) | -| **Single vs Multi-turn** | Donut | Single-turn conversations (1 user message) vs multi-turn (2+ messages) | -| **Session Duration Distribution** | Donut | Short (<1 min), Medium (1-10 min), Long (>10 min) | -| **Error Distribution** | Donut | Top 10 errors by error code | -| **Top Tools** | Donut | Top 10 tools by call volume | -| **Slowest Tools** | Horizontal Bar | Top 5 tools by weighted average response time (Avg Response Time x Call Count / Total Calls per tool) | -| **Response Time** | Grouped Bar | Daily average and P90 response times | -| **Unique Users** | Interactive tags | Displays unique users from Application Insights telemetry logs. Anonymous users are filtered out. | - ---- - -### 7.2 Daily Metrics - -**Data Source**: Application Insights telemetry (Daily Metrics table) - -Shows daily aggregated conversation metrics with time-series charts and a detailed data table. - -#### Charts - -| Chart | Type | Description | -|---|---|---| -| **Conversation Volume** | Area (full width) | Daily distinct conversation count — shows engagement trends and peak usage | -| **Daily Active Users** | Area (half width) | Daily unique users — requires user capture enabled in agent configuration | -| **Daily Messages** | Area (half width) | User messages and bot responses sent per day | - -#### Data Table - -Contains all daily metric fields including: conversation counts, unique users/tools/topics, message counts, response time percentiles (P10/P50/P90/P95/P99), duration metrics, conversation type breakdown (single/multi-turn, short/medium/long), tool stats, and error counts. - ---- - -### 7.3 Topics - -**Data Source**: Application Insights telemetry (Topic Metrics table) - -Analyzes topic performance — which topics users trigger most, completion rates, and where errors occur. - -#### KPI Cards - -| KPI | Calculation | Description | -|---|---|---| -| **Total Triggers** | Sum of Trigger Count | Total times topics were triggered (TopicStart events) | -| **Completed** | Sum of Completion Count | Topics that ran to completion (TopicEnd events) | -| **Abandoned** | Sum of Abandonment Count | Topics where triggers exceeded completions and redirects | -| **Avg Duration** | Sum of Total Duration (ms) / Sum of Duration Count | Average time from TopicStart to TopicEnd, displayed as human-readable duration | -| **Errors** | Sum of Error Count | Errors attributed to topics based on the most recent active topic | -| **Messages** | Sum of Message Count | Total user messages (text only) attributed to topics | - -#### Charts - -| Chart | Type | Description | -|---|---|---| -| **Topic Triggers Over Time** | Area | Daily trend of topic triggers | -| **Topics with Most Errors** | Horizontal Bar | Top 5 topics by error count (only shown if errors exist) | -| **Top Topics by Triggers** | Donut | Top 10 most frequently triggered topics | -| **Messages In vs Out** | Donut | User messages vs bot responses attributed to topics | - ---- - -### 7.4 Tools - -**Data Source**: Application Insights telemetry (Tool Metrics table) - -Monitors the reliability and speed of external dependencies (HTTP requests, connectors, flows, GenAI calls) that agents invoke during conversations. - -#### KPI Cards - -| KPI | Calculation | Description | -|---|---|---| -| **Total Calls** | Sum of Call Count | Total external dependency calls | -| **Unique Tools** | Count of distinct Tool Name values | Number of different tools called | -| **Success Rate** | (Sum of Success Count / Sum of Call Count) x 100 | Percentage of successful tool calls | -| **Total Failures** | Sum of Failure Count | Number of failed tool calls | -| **Avg Response Time** | Weighted average of Avg Response Time (ms), weighted by Call Count per record | Average tool response time, displayed in ms or seconds | -| **P90 Response Time** | Weighted average of P90 Response Time (ms), weighted by Call Count per record | 90th percentile — 90% of calls were faster than this value | - -#### Charts - -| Chart | Type | Description | -|---|---|---| -| **Tool Calls Over Time** | Area | Daily trend of tool invocations | -| **Tool Type Distribution** | Donut | Distribution by tool type (Action, Connector, etc.) | -| **Tool Failures Over Time** | Area | Daily trend of failures — spikes indicate reliability issues | -| **Tool Success Rate (%)** | Horizontal Bar | Top 10 tools ranked by success percentage (Success Count / Call Count x 100 per tool) | - ---- - -### 7.5 Actions - -**Data Source**: Application Insights telemetry (Action Metrics table) - -Tracks how action nodes perform — execution volume, speed, and the types of actions being used (HTTP requests, Flow invocations, Connector actions, etc.). - -#### KPI Cards - -| KPI | Calculation | Description | -|---|---|---| -| **Total Executions** | Sum of Execution Count | Total action node executions | -| **Unique Actions** | Count of distinct Node Kind values | Number of different action types (HTTP, Flow, Connector, etc.) | -| **Avg Elapsed Time** | Sum of Total Elapsed (ms) / Sum of Execution Count | Average time per action execution, displayed in ms or seconds | - -#### Charts - -| Chart | Type | Description | -|---|---|---| -| **Execution Trend** | Area | Daily count of action executions over time | -| **Node Kind Distribution** | Donut | Breakdown by action type — shows most used types | - ---- - -### 7.6 Errors - -**Data Source**: Application Insights telemetry (Error Details table) - -Identifies what's going wrong — which errors are most frequent, how many users are impacted, and how errors trend over time. - -#### KPI Cards - -| KPI | Calculation | Description | -|---|---|---| -| **Total Errors** | Sum of Error Count | Total OnErrorLog events recorded | -| **Unique Error Codes** | Count of distinct Error Code values | Number of different error types encountered | -| **Affected Conversations** | Sum of Daily Affected Conversations | Conversations that experienced errors (may include duplicates across days) | -| **Impacted Users** | Sum of Daily Affected Users | Users who encountered errors (may include duplicates across days) | - -#### Charts - -| Chart | Type | Description | -|---|---|---| -| **Error Trend** | Area | Daily error counts — spikes indicate system issues | -| **Top Error Codes** | Horizontal Bar | Top 5 most frequent error codes | -| **Errors by Channel** | Donut | Error distribution across channels | -| **Impacted Users** | Area | Daily trend of users impacted by errors | - ---- - -### 7.7 Transcripts - -**Data Source**: Conversation Transcripts from Copilot Studio (Transcript Metrics table) - -Analyzes conversation quality — how users engage, whether sessions get resolved or escalated, and overall satisfaction. - -#### KPI Cards (with period-over-period trends) - -| KPI | Calculation | Description | -|---|---|---| -| **Conversation Sessions** | Sum of Session Count | Total conversation sessions in the period | -| **Engaged** | Sum of Engaged Count. **Engagement Rate** = (Engaged Count / Session Count) x 100 | Sessions where the user sent at least one message after the greeting | -| **Avg. Turns** | Sum of Total Turns / Sum of Session Count | Average message exchanges per session | -| **Feedback** | Sum of Like Count + Dislike Count | User feedback reactions (thumbs up/down) | -| **Satisfaction Score** | Sum of CSAT Score / Sum of CSAT Count | Average satisfaction score on a 1–5 scale, displayed as X.X/5.0. Only shown when CSAT data exists. | -| **Resolved** | Sum of Resolved Count. **Resolution Rate** = (Resolved Count / Session Count) x 100 | Sessions marked as resolved | -| **Escalated** | Sum of Escalated Count. **Escalation Rate** = (Escalated Count / Session Count) x 100 | Sessions escalated to a human agent | -| **Abandoned** | Sum of Abandoned Count. **Abandonment Rate** = (Abandoned Count / Session Count) x 100 | Sessions where users left without resolution | - -#### Charts - -| Chart | Type | Description | -|---|---|---| -| **Conversation Outcomes** | Stacked Area + KPIs | Session outcomes over time (Resolved, Escalated, Abandoned, Unengaged) with side KPIs | -| **Engagement Breakdown** | Donut | Engaged vs unengaged sessions | -| **Session Outcomes** | Donut | Distribution of Resolved, Escalated, Abandoned outcomes | -| **Survey Satisfaction Score** | Stacked Bar + Area | Average CSAT score (1-5) with satisfaction trend over time. Satisfied: 4-5, Neutral: 3, Dissatisfied: 1-2 | -| **Satisfaction Score Trend** | Area | Daily average CSAT score over the last 45 days. Each day's score is computed as: **Daily CSAT Score Sum / Daily CSAT Response Count**, rounded to 1 decimal place. Days with no survey responses are excluded. | - -#### Feedback Dialog - -Clicking the Feedback KPI card opens a dialog that loads detailed feedback from file attachments (cat_feedbackdetailsfile). Each feedback record shows: reaction (thumbs up/down), agent name, conversation ID, conversation date, agent response, and user feedback text. Supports filtering by reaction type and Excel export. - -> **Note**: Feedback data is only available when **Capture User Feedback** is enabled in the agent configuration (Add/Edit Agent dialog). - ---- - -### 7.8 Usage History - -**Data Source**: Power Platform Admin Center (Agent Usage History table) - -Tracks Copilot credit consumption to understand cost and usage patterns across environments and features. - -> **Note**: This tab only appears when usage history data exists in the system. - -#### KPI Cards - -| KPI | Calculation | Description | -|---|---|---| -| **Billed Credits** | Sum of Billed Copilot Credits | Total billed credits consumed | -| **Non-billed Credits** | Sum of Non-billed Copilot Credits | Total non-billed credits consumed | - -#### Charts - -| Chart | Type | Description | -|---|---|---| -| **Credit Consumption Over Time** | Area | Billed and non-billed credit trends over time | -| **Credits by Environment** | Horizontal Bar | Top 5 environments by billed credit consumption | -| **Credits by Feature** | Donut | Credit distribution across Copilot features | - -This tab uses its own independent filters (Environment, Agent, Date, Feature) rather than the global filter bar. - ---- - -### 7.9 Sync Logs (Troubleshooting) - -Shows the execution history of data synchronization flows. Use this tab to verify whether metric generation jobs completed successfully. - -#### Data Table Columns - -| Column | Description | -|---|---| -| **Name** | Sync job name | -| **Status** | Execution status badge: Not Started, Running, Complete, Error, Cancelled | -| **Cloud Flow** | Link to the Cloud Flow run instance (opens in new tab) | -| **Start Date** | When the sync started | -| **End Date** | When the sync ended | -| **Duration** | Total flow execution time | -| **Error Message** | Error details if the sync failed | -| **Created** | When the log record was created | - ---- - -### 7.10 Transcripts Staging (Troubleshooting) - -Monitors the ingestion pipeline for transcript data. Use this tab to check processing status and identify stuck or failed records. - -#### Data Table Columns - -| Column | Description | -|---|---| -| **Name** | Record name | -| **Agent Name** | Agent that generated the transcript | -| **Conversation Date** | When the conversation occurred | -| **Conversation ID** | Unique conversation identifier | -| **Channel ID** | Communication channel used | -| **Data Source** | Badge: Production or Test Data | -| **Workflow Status** | Badge: Pending, Completed, or Failed | -| **Transcript ID** | Unique transcript GUID | - -Supports filtering by workflow status (All, Pending, Completed, Failed) with a refresh button. - ---- - -## 8. Data Sources & Differences - -### Important Note on Data Discrepancies - -There can be differences in KPI values generated from **Application Insights** versus **Conversation Transcripts** due to the nature of their data sources: - -| Factor | Application Insights | Conversation Transcripts | -|---|---|---| -| **Data availability** | Telemetry logs available as long as App Insights retention is configured (default 90 days, configurable up to 730 days) | Conversation transcripts **older than 30 days are automatically deleted** by a bulk-delete system job. To change or disable this, see [Change the default retention period](https://learn.microsoft.com/en-us/microsoft-copilot-studio/analytics-transcripts-powerapps#change-the-default-retention-period). | -| **Data completeness** | Only captures data if telemetry logging is enabled and the agent is connected to App Insights | Captures all conversations regardless of App Insights integration | -| **Metrics focus** | Response times, tool performance, error details, topic execution | Session outcomes, engagement, CSAT, user feedback | -| **Granularity** | Event-level telemetry aggregated daily | Session-level aggregated daily | - -**Recommendation**: Use Application Insights metrics (Overview, Daily Metrics, Topics, Tools, Actions, Errors tabs) for performance monitoring and debugging. Use Conversation Transcripts metrics (Transcripts tab) for conversation quality and customer satisfaction analysis. - ---- - -## 9. Troubleshooting - -### After Every Sync - -1. **Check Sync Logs tab** — Verify the sync job status is **Complete** (green badge). If it shows **Error** (red), check the Error Message column for details and the Cloud Flow link for the full run history. - -2. **Check Transcripts Staging tab** — For transcript data, verify that staging records have **Completed** status (green badge). If records are stuck in **Pending** or show **Failed**: - - Filter by "Failed" status to identify problematic records - - Check if the conversation transcript data is still available (transcripts older than 30 days may have been deleted) - - Re-run the sync for the affected date range - -### Common Issues - -| Issue | Possible Cause | Resolution | -|---|---|---| -| No data after sync | App Insights not connected to agent | Verify Application Insights integration in Copilot Studio Settings > Advanced | -| Sync job shows Error | Invalid App Insights credentials | Check Application ID, Tenant ID, Client ID, and Secret in the agent configuration | -| "No insights yet" on all tabs | No sync has been run | Click "Sync" and run a sync for your desired date range | -| Partial data on some tabs | KPI Source not selected | Edit the agent configuration and ensure the correct KPI sources are checked (Application Insights, Conversation Transcripts, Usage History) | -| Transcript data missing for older dates | Transcripts deleted after 30 days | A bulk-delete system job removes transcripts older than 30 days. Sync regularly to capture data before it expires, or disable/modify the job — see [Change the default retention period](https://learn.microsoft.com/en-us/microsoft-copilot-studio/analytics-transcripts-powerapps#change-the-default-retention-period). | -| Usage History tab not visible | No usage data available | The Usage History feature requires the separate **AgentInventoryUsage** solution to be imported. This solution connects to the Power Platform licensing API to pull credit consumption data. See [Using Usage Metrics in Agent Inventory](https://github.com/microsoft/Power-CAT-Copilot-Studio-Kit/blob/main/AGENT_INVENTORY.md#using-usage-metrics-in-agent-inventory) for installation steps. During import, set up the "HTTP with Microsoft Entra ID (preauthorized)" connection pointing to `https://licensing.powerplatform.microsoft.com/`. Requires Power Platform admin role and system admin permissions. | -| Filters return no results | Restrictive filter combination | Try resetting to default filters (All agents, Last 90 days, All channels, Production) | -| Run button greyed out in Sync dialog | No configuration selected | Select an agent configuration or "All Agents" from the dropdown before clicking Run | - +# Agent Insights Hub (Preview) + +## Table of Contents + +- [1. Overview](#1-overview) +- [2. Prerequisites](#2-prerequisites) +- [3. Setup Instructions](#3-setup-instructions) + - [3.1 Integrate Copilot Studio with Application Insights](#31-integrate-copilot-studio-with-application-insights) + - [3.2 Register Azure App for Application Insights Access](#32-register-azure-app-for-application-insights-access) + - [3.3 Run Agent Inventory Sync](#33-run-agent-inventory-sync) + - [3.4 Add an Agent Configuration](#34-add-an-agent-configuration) + - [3.5 Run Your First Sync](#35-run-your-first-sync) +- [4. How It Works — End-to-End Flow](#4-how-it-works--end-to-end-flow) +- [5. Action Buttons](#5-action-buttons) +- [6. Filters](#6-filters) +- [7. Tabs](#7-tabs) + - [7.1 Overview](#71-overview) + - [7.2 Daily Metrics](#72-daily-metrics) + - [7.3 Topics](#73-topics) + - [7.4 Tools](#74-tools) + - [7.5 Actions](#75-actions) + - [7.6 Errors](#76-errors) + - [7.7 Transcripts](#77-transcripts) + - [7.8 Usage History](#78-usage-history) + - [7.9 Sync Logs (Troubleshooting)](#79-sync-logs-troubleshooting) + - [7.10 Transcripts Staging (Troubleshooting)](#710-transcripts-staging-troubleshooting) +- [8. Data Sources & Differences](#8-data-sources--differences) +- [9. Troubleshooting](#9-troubleshooting) + +--- + +## 1. Overview + +**Agent Insights Hub** is a comprehensive analytics and monitoring dashboard for Microsoft Copilot Studio agents. It provides real-time visibility into agent performance, conversation metrics, topic analytics, tool execution, and error tracking. + +Built as a Power Apps Code App using React and Fluent UI v9, it connects directly to Dataverse to display aggregated telemetry data imported from three sources: + +| Data Source | Tabs Powered | What It Captures | +|---|---|---| +| **Azure Application Insights** | Overview, Daily Metrics, Topics, Tools, Actions, Errors | Telemetry logs from agent conversations — response times, topic triggers, tool calls, errors | +| **Conversation Transcripts** | Transcripts | Session-level data from Copilot Studio — engagement, resolution, escalation, CSAT, feedback | +| **Power Platform Admin Center** | Usage History | Copilot credit consumption — billed and non-billed credits by agent, environment, and feature | + + +--- + +## 2. Prerequisites + +Before using Agent Insights Hub, ensure the following are in place: + +1. **Agent Inventory Sync must be run** — The Agent Inventory sync job must have completed at least once so that agent details are available for selection. + +2. **Application Insights integration** — Your Copilot Studio agents must be connected to an Azure Application Insights resource. See [Section 3.1](#31-integrate-copilot-studio-with-application-insights). + +3. **Azure App Registration** — An Azure AD app must be registered with access to the Application Insights telemetry data. Follow the setup guide: [Enable Application Insights Support](https://github.com/microsoft/Power-CAT-Copilot-Studio-Kit/blob/main/ENABLE-APPINSIGHTS.md#enable-application-insights-support). + +4. **App sharing** — The owner of the Copilot Studio Kit code app (or the System Administrator who installed kit) must share the app with other users on their team. Follow these steps: + 1. Go to [make.powerapps.com](https://make.powerapps.com/) + 2. Click **Apps** in the left navigation + 3. Search for **Copilot Studio Kit** and locate the one with type **Code** (not the Model-driven app) + 4. Click the **three dots (...)** menu next to the app + 5. Select **Share** + 6. Search for and add the users or security groups who need access + 7. Click **Share** to confirm + + For detailed instructions, see: [Share a canvas app with your organization](https://learn.microsoft.com/en-us/power-apps/maker/canvas-apps/share-app). This step will be automated in a future release. + +--- + +## 3. Setup Instructions + +### 3.1 Integrate Copilot Studio with Application Insights + +To collect telemetry from your Copilot Studio agents, you must connect them to Azure Application Insights: + +1. Open [Copilot Studio](https://copilotstudio.microsoft.com) and select your agent +2. Go to **Settings** > **Advanced** > **Application Insights** +3. Enter your Application Insights **Connection String** (found in Azure Portal > Application Insights > Overview > Connection String) +4. Click **Save** + +For detailed instructions, refer to the official Microsoft documentation: +[Connect your copilot to Application Insights](https://learn.microsoft.com/en-us/microsoft-copilot-studio/advanced-bot-framework-composer-capture-telemetry) + +### 3.2 Register Azure App for Application Insights Access + +The sync flow needs programmatic access to query Application Insights telemetry. This requires an Azure AD app registration with appropriate permissions. + +Follow the complete setup guide: **[Enable Application Insights Support](https://github.com/microsoft/Power-CAT-Copilot-Studio-Kit/blob/main/ENABLE-APPINSIGHTS.md#enable-application-insights-support)** + +This guide covers: +- Creating an Azure AD app registration +- Granting the app access to your Application Insights resource +- Obtaining the Application ID, Tenant ID, Client ID, and Client Secret +- Configuring secret storage (Dataverse only currently; Key Vault support planned for a future release) + +### 3.3 Run Agent Inventory Sync + +Before adding agents to the Insights Hub, ensure the **Agent Inventory** sync has been run at least once. This populates the agent list that appears in the "Add Agent" dialog. For detailed instructions, see: [Agent Inventory Documentation](https://github.com/microsoft/Power-CAT-Copilot-Studio-Kit/blob/main/AGENT_INVENTORY.md). + +### 3.4 Add an Agent Configuration + +1. Open Agent Insights Hub +2. Click the **Add agent** button +3. Fill in the configuration form: + +| Field | Description | +|---|---| +| **Agent** | Select an agent from the inventory. This auto-fills Name, Agent Name, Agent ID, and Dataverse URL. | +| **Name** | A friendly display name for this configuration. Must be unique. | +| **Agent Name** | The name of the Copilot Studio agent. Auto-filled when an agent is selected. Not visible when adding a new agent. | +| **Agent ID** | The unique identifier (GUID) of the Copilot Studio agent. Found in Copilot Studio under Settings > Session Info. | +| **Dataverse URL** | The base URL of the Dataverse environment (e.g., `https://org.crm.dynamics.com`). | +| **Application Insights App ID** | The Application ID from your Azure Application Insights resource (Azure Portal > Application Insights > API Access). | +| **Tenant ID** | The Azure AD Tenant ID where the Application Insights resource is registered. | +| **Client ID** | The Client ID of the Azure AD app registration used to authenticate with Application Insights. | +| **Secret Location** | Currently supports Dataverse storage only. Key Vault storage will be supported in a future release. | +| **Secret** | The client secret for the Azure AD app registration. | +| **KPI Source** | Select which data sources to use: Application Insights, Conversation Transcripts, Usage History. Select one, two or all three options based on requirement. | +| **Capture User Details** | When enabled, user identity information (User name interacted with agent) will be captured from telemetry logs. | +| **Capture User Feedback** | When enabled, end-user satisfaction feedback will be collected. | + +4. Click **Save** + +### 3.5 Run Your First Sync + +After adding an agent configuration: + +1. Click the **Sync** button +2. In the Generate Metrics dialog: + - Select a configuration (or "All Agents" to sync all at once) + - Set the **Start Date** and **End Date** for historical data extraction (up to 365 days back) +3. Click **Run** +4. The sync triggers a Cloud Flow that: + - Queries Application Insights for telemetry data + - Fetches Conversation Transcripts from Copilot Studio + - Processes and aggregates the data into KPI records + - Writes the results to Dataverse tables +5. Monitor progress in the **Sync Logs** tab + +--- + +## 4. How It Works — End-to-End Flow + +```mermaid +flowchart TD + A[Add Agent Configuration] --> B{Two Sync Modes} + + B --> S[Scheduled Sync - Runs Daily] + B --> O[On-Demand Sync - Via Sync Button] + + S --> S1[Pull Last 24h App Insights Telemetry] + S --> S2[Pull Last 24h Conversation Transcripts] + + O --> O1[Pull App Insights Telemetry for Selected Date Range] + O --> O2[Pull Conversation Transcripts for Selected Date Range] + + S1 --> P1[Process & Aggregate Data] + S2 --> P1 + O1 --> P2[Process & Aggregate Data] + O2 --> P2 + + P1 --> J[Write to Dataverse Tables] + P2 --> J + + J --> K[Daily Metrics Table] + J --> L[Topic Metrics Table] + J --> M[Tool Metrics Table] + J --> N[Action Metrics Table] + J --> Q[Error Details Table] + J --> T[Transcript Metrics Table] + + K --> R[Agent Insights Hub] + L --> R + M --> R + N --> R + Q --> R + T --> R + + R --> Tab1[Overview Tab] + R --> Tab2[Daily Metrics Tab] + R --> Tab3[Topics Tab] + R --> Tab4[Tools Tab] + R --> Tab5[Actions Tab] + R --> Tab6[Errors Tab] + R --> Tab7[Transcripts Tab] + R --> Tab8[Usage History Tab] +``` + +### Sync Modes + +| Mode | Trigger | Data Range | Use Case | +|---|---|---|---| +| **Scheduled** | Runs automatically every day | Last 24 hours | Keeps metrics up to date with daily incremental syncs | +| **On-Demand** | User clicks **Sync** button | Custom start and end date (up to 365 days) | Backfill historical data or re-sync a specific date range | + +--- + +## 5. Action Buttons + +These buttons appear on the Overview section header and in empty state screens: + +| Button | Description | +|---|---| +| **Sync** | Opens the Generate Metrics dialog to trigger a historical data sync for one or all agents. Select a date range (up to 365 days) and click Run. | +| **Add agent** | Opens the Add Agent dialog to create a new agent configuration with Application Insights credentials and KPI source selection. | +| **Show agents** | Opens a dialog listing all configured agents with their details, KPI sources, and status. Allows editing, cloning, and deleting configurations. | + +--- + +## 6. Filters + +A global filter bar appears on all tabs except Sync Logs, Transcripts Staging, and Usage History (which has its own filters). + +| Filter | Options | Default | Description | +|---|---|---|---| +| **Agent** | All agents, or a specific configured agent | All agents | Filters data to a specific agent configuration. | +| **Date Range** | Last 7 / 30 / 90 / 180 / 365 days, Custom range | Last 90 days | Controls the time period for all metrics. Custom range shows start/end date pickers. | +| **Channel** | All channels, Teams, Web Chat, Direct Line, Test Panel, Autonomous, Published Engine, Mobile, WhatsApp, Unknown | All channels | Filters by the communication channel used for conversations. | +| **Data Mode** | Production, Test data, All data | Production | Filters by data source type. Production shows real user data; Test data shows test panel interactions. | + +**Usage History** has its own independent filters: + +| Filter | Description | +|---|---| +| **Environment** | Filter by Power Platform environment | +| **Agent** | Filter by agent (derived from usage data, not configurations) | +| **Date** | Month to date, previous months, or last 6 months | +| **Feature** | Filter by Copilot feature name | + +--- + +## 7. Tabs + +### 7.1 Overview + +**Data Source**: Application Insights telemetry (Daily Metrics, Topic Metrics, Tool Metrics, Error Details tables) + +The Overview tab provides a high-level summary of agent performance. It displays 7 KPI cards with trend indicators comparing the current period against the previous period of equal length. + +#### KPI Cards + +| KPI | Calculation | Display | Trend | +|---|---|---|---| +| **Conversations** | Sum of Daily Conversation Count across all days | Number | Higher is better | +| **Average DAU** | Sum of Daily Unique Users / Number of distinct days in the period | Number (rounded) | Higher is better | +| **Responses** | Sum of Total Responses (bot messages sent) | Number | Higher is better | +| **Avg Response** | Sum of Total Response Time (ms) / Sum of Total Responses | Seconds (if >= 1s) or ms | Lower is better (inverted trend) | +| **Duration** | Sum of Total Duration (ms) / Sum of Conversations With Duration Data / 60,000 | Minutes (1 decimal) | Lower is better (inverted trend) | +| **Tool Success** | (Sum of Tool Success Count / Sum of Tool Call Count) x 100 | Percentage | Higher is better | +| **Errors** | Sum of Error Count | Number | Lower is better (inverted trend) | + +**Trend Calculation**: Compares the current period value against the previous period of equal length. For example, if viewing the last 30 days, the trend compares against the 30 days before that. When the previous period has no data but the current period does, it shows "New" instead of a percentage. + +#### Charts + +| Chart | Type | Description | +|---|---|---| +| **Top Topics** | Donut | Top 10 topics ranked by trigger count | +| **Slowest Topics** | Horizontal Bar | Top 5 topics by weighted average duration (Total Duration / Duration Count per topic) | +| **Conversations by Channel** | Donut | Distribution of conversations across channels (Teams, Web Chat, etc.) | +| **Single vs Multi-turn** | Donut | Single-turn conversations (1 user message) vs multi-turn (2+ messages) | +| **Session Duration Distribution** | Donut | Short (<1 min), Medium (1-10 min), Long (>10 min) | +| **Error Distribution** | Donut | Top 10 errors by error code | +| **Top Tools** | Donut | Top 10 tools by call volume | +| **Slowest Tools** | Horizontal Bar | Top 5 tools by weighted average response time (Avg Response Time x Call Count / Total Calls per tool) | +| **Response Time** | Grouped Bar | Daily average and P90 response times | +| **Unique Users** | Interactive tags | Displays unique users from Application Insights telemetry logs. Anonymous users are filtered out. | + +--- + +### 7.2 Daily Metrics + +**Data Source**: Application Insights telemetry (Daily Metrics table) + +Shows daily aggregated conversation metrics with time-series charts and a detailed data table. + +#### Charts + +| Chart | Type | Description | +|---|---|---| +| **Conversation Volume** | Area (full width) | Daily distinct conversation count — shows engagement trends and peak usage | +| **Daily Active Users** | Area (half width) | Daily unique users — requires user capture enabled in agent configuration | +| **Daily Messages** | Area (half width) | User messages and bot responses sent per day | + +#### Data Table + +Contains all daily metric fields including: conversation counts, unique users/tools/topics, message counts, response time percentiles (P10/P50/P90/P95/P99), duration metrics, conversation type breakdown (single/multi-turn, short/medium/long), tool stats, and error counts. + +--- + +### 7.3 Topics + +**Data Source**: Application Insights telemetry (Topic Metrics table) + +Analyzes topic performance — which topics users trigger most, completion rates, and where errors occur. + +#### KPI Cards + +| KPI | Calculation | Description | +|---|---|---| +| **Total Triggers** | Sum of Trigger Count | Total times topics were triggered (TopicStart events) | +| **Completed** | Sum of Completion Count | Topics that ran to completion (TopicEnd events) | +| **Abandoned** | Sum of Abandonment Count | Topics where triggers exceeded completions and redirects | +| **Avg Duration** | Sum of Total Duration (ms) / Sum of Duration Count | Average time from TopicStart to TopicEnd, displayed as human-readable duration | +| **Errors** | Sum of Error Count | Errors attributed to topics based on the most recent active topic | +| **Messages** | Sum of Message Count | Total user messages (text only) attributed to topics | + +#### Charts + +| Chart | Type | Description | +|---|---|---| +| **Topic Triggers Over Time** | Area | Daily trend of topic triggers | +| **Topics with Most Errors** | Horizontal Bar | Top 5 topics by error count (only shown if errors exist) | +| **Top Topics by Triggers** | Donut | Top 10 most frequently triggered topics | +| **Messages In vs Out** | Donut | User messages vs bot responses attributed to topics | + +--- + +### 7.4 Tools + +**Data Source**: Application Insights telemetry (Tool Metrics table) + +Monitors the reliability and speed of external dependencies (HTTP requests, connectors, flows, GenAI calls) that agents invoke during conversations. + +#### KPI Cards + +| KPI | Calculation | Description | +|---|---|---| +| **Total Calls** | Sum of Call Count | Total external dependency calls | +| **Unique Tools** | Count of distinct Tool Name values | Number of different tools called | +| **Success Rate** | (Sum of Success Count / Sum of Call Count) x 100 | Percentage of successful tool calls | +| **Total Failures** | Sum of Failure Count | Number of failed tool calls | +| **Avg Response Time** | Weighted average of Avg Response Time (ms), weighted by Call Count per record | Average tool response time, displayed in ms or seconds | +| **P90 Response Time** | Weighted average of P90 Response Time (ms), weighted by Call Count per record | 90th percentile — 90% of calls were faster than this value | + +#### Charts + +| Chart | Type | Description | +|---|---|---| +| **Tool Calls Over Time** | Area | Daily trend of tool invocations | +| **Tool Type Distribution** | Donut | Distribution by tool type (Action, Connector, etc.) | +| **Tool Failures Over Time** | Area | Daily trend of failures — spikes indicate reliability issues | +| **Tool Success Rate (%)** | Horizontal Bar | Top 10 tools ranked by success percentage (Success Count / Call Count x 100 per tool) | + +--- + +### 7.5 Actions + +**Data Source**: Application Insights telemetry (Action Metrics table) + +Tracks how action nodes perform — execution volume, speed, and the types of actions being used (HTTP requests, Flow invocations, Connector actions, etc.). + +#### KPI Cards + +| KPI | Calculation | Description | +|---|---|---| +| **Total Executions** | Sum of Execution Count | Total action node executions | +| **Unique Actions** | Count of distinct Node Kind values | Number of different action types (HTTP, Flow, Connector, etc.) | +| **Avg Elapsed Time** | Sum of Total Elapsed (ms) / Sum of Execution Count | Average time per action execution, displayed in ms or seconds | + +#### Charts + +| Chart | Type | Description | +|---|---|---| +| **Execution Trend** | Area | Daily count of action executions over time | +| **Node Kind Distribution** | Donut | Breakdown by action type — shows most used types | + +--- + +### 7.6 Errors + +**Data Source**: Application Insights telemetry (Error Details table) + +Identifies what's going wrong — which errors are most frequent, how many users are impacted, and how errors trend over time. + +#### KPI Cards + +| KPI | Calculation | Description | +|---|---|---| +| **Total Errors** | Sum of Error Count | Total OnErrorLog events recorded | +| **Unique Error Codes** | Count of distinct Error Code values | Number of different error types encountered | +| **Affected Conversations** | Sum of Daily Affected Conversations | Conversations that experienced errors (may include duplicates across days) | +| **Impacted Users** | Sum of Daily Affected Users | Users who encountered errors (may include duplicates across days) | + +#### Charts + +| Chart | Type | Description | +|---|---|---| +| **Error Trend** | Area | Daily error counts — spikes indicate system issues | +| **Top Error Codes** | Horizontal Bar | Top 5 most frequent error codes | +| **Errors by Channel** | Donut | Error distribution across channels | +| **Impacted Users** | Area | Daily trend of users impacted by errors | + +--- + +### 7.7 Transcripts + +**Data Source**: Conversation Transcripts from Copilot Studio (Transcript Metrics table) + +Analyzes conversation quality — how users engage, whether sessions get resolved or escalated, and overall satisfaction. + +#### KPI Cards (with period-over-period trends) + +| KPI | Calculation | Description | +|---|---|---| +| **Conversation Sessions** | Sum of Session Count | Total conversation sessions in the period | +| **Engaged** | Sum of Engaged Count. **Engagement Rate** = (Engaged Count / Session Count) x 100 | Sessions where the user sent at least one message after the greeting | +| **Avg. Turns** | Sum of Total Turns / Sum of Session Count | Average message exchanges per session | +| **Feedback** | Sum of Like Count + Dislike Count | User feedback reactions (thumbs up/down) | +| **Satisfaction Score** | Sum of CSAT Score / Sum of CSAT Count | Average satisfaction score on a 1–5 scale, displayed as X.X/5.0. Only shown when CSAT data exists. | +| **Resolved** | Sum of Resolved Count. **Resolution Rate** = (Resolved Count / Session Count) x 100 | Sessions marked as resolved | +| **Escalated** | Sum of Escalated Count. **Escalation Rate** = (Escalated Count / Session Count) x 100 | Sessions escalated to a human agent | +| **Abandoned** | Sum of Abandoned Count. **Abandonment Rate** = (Abandoned Count / Session Count) x 100 | Sessions where users left without resolution | + +#### Charts + +| Chart | Type | Description | +|---|---|---| +| **Conversation Outcomes** | Stacked Area + KPIs | Session outcomes over time (Resolved, Escalated, Abandoned, Unengaged) with side KPIs | +| **Engagement Breakdown** | Donut | Engaged vs unengaged sessions | +| **Session Outcomes** | Donut | Distribution of Resolved, Escalated, Abandoned outcomes | +| **Survey Satisfaction Score** | Stacked Bar + Area | Average CSAT score (1-5) with satisfaction trend over time. Satisfied: 4-5, Neutral: 3, Dissatisfied: 1-2 | +| **Satisfaction Score Trend** | Area | Daily average CSAT score over the last 45 days. Each day's score is computed as: **Daily CSAT Score Sum / Daily CSAT Response Count**, rounded to 1 decimal place. Days with no survey responses are excluded. | + +#### Feedback Dialog + +Clicking the Feedback KPI card opens a dialog that loads detailed feedback from file attachments (cat_feedbackdetailsfile). Each feedback record shows: reaction (thumbs up/down), agent name, conversation ID, conversation date, agent response, and user feedback text. Supports filtering by reaction type and Excel export. + +> **Note**: Feedback data is only available when **Capture User Feedback** is enabled in the agent configuration (Add/Edit Agent dialog). + +--- + +### 7.8 Usage History + +**Data Source**: Power Platform Admin Center (Agent Usage History table) + +Tracks Copilot credit consumption to understand cost and usage patterns across environments and features. + +> **Note**: This tab only appears when usage history data exists in the system. + +#### KPI Cards + +| KPI | Calculation | Description | +|---|---|---| +| **Billed Credits** | Sum of Billed Copilot Credits | Total billed credits consumed | +| **Non-billed Credits** | Sum of Non-billed Copilot Credits | Total non-billed credits consumed | + +#### Charts + +| Chart | Type | Description | +|---|---|---| +| **Credit Consumption Over Time** | Area | Billed and non-billed credit trends over time | +| **Credits by Environment** | Horizontal Bar | Top 5 environments by billed credit consumption | +| **Credits by Feature** | Donut | Credit distribution across Copilot features | + +This tab uses its own independent filters (Environment, Agent, Date, Feature) rather than the global filter bar. + +--- + +### 7.9 Sync Logs (Troubleshooting) + +Shows the execution history of data synchronization flows. Use this tab to verify whether metric generation jobs completed successfully. + +#### Data Table Columns + +| Column | Description | +|---|---| +| **Name** | Sync job name | +| **Status** | Execution status badge: Not Started, Running, Complete, Error, Cancelled | +| **Cloud Flow** | Link to the Cloud Flow run instance (opens in new tab) | +| **Start Date** | When the sync started | +| **End Date** | When the sync ended | +| **Duration** | Total flow execution time | +| **Error Message** | Error details if the sync failed | +| **Created** | When the log record was created | + +--- + +### 7.10 Transcripts Staging (Troubleshooting) + +Monitors the ingestion pipeline for transcript data. Use this tab to check processing status and identify stuck or failed records. + +#### Data Table Columns + +| Column | Description | +|---|---| +| **Name** | Record name | +| **Agent Name** | Agent that generated the transcript | +| **Conversation Date** | When the conversation occurred | +| **Conversation ID** | Unique conversation identifier | +| **Channel ID** | Communication channel used | +| **Data Source** | Badge: Production or Test Data | +| **Workflow Status** | Badge: Pending, Completed, or Failed | +| **Transcript ID** | Unique transcript GUID | + +Supports filtering by workflow status (All, Pending, Completed, Failed) with a refresh button. + +--- + +## 8. Data Sources & Differences + +### Important Note on Data Discrepancies + +There can be differences in KPI values generated from **Application Insights** versus **Conversation Transcripts** due to the nature of their data sources: + +| Factor | Application Insights | Conversation Transcripts | +|---|---|---| +| **Data availability** | Telemetry logs available as long as App Insights retention is configured (default 90 days, configurable up to 730 days) | Conversation transcripts **older than 30 days are automatically deleted** by a bulk-delete system job. To change or disable this, see [Change the default retention period](https://learn.microsoft.com/en-us/microsoft-copilot-studio/analytics-transcripts-powerapps#change-the-default-retention-period). | +| **Data completeness** | Only captures data if telemetry logging is enabled and the agent is connected to App Insights | Captures all conversations regardless of App Insights integration | +| **Metrics focus** | Response times, tool performance, error details, topic execution | Session outcomes, engagement, CSAT, user feedback | +| **Granularity** | Event-level telemetry aggregated daily | Session-level aggregated daily | + +**Recommendation**: Use Application Insights metrics (Overview, Daily Metrics, Topics, Tools, Actions, Errors tabs) for performance monitoring and debugging. Use Conversation Transcripts metrics (Transcripts tab) for conversation quality and customer satisfaction analysis. + +--- + +## 9. Troubleshooting + +### After Every Sync + +1. **Check Sync Logs tab** — Verify the sync job status is **Complete** (green badge). If it shows **Error** (red), check the Error Message column for details and the Cloud Flow link for the full run history. + +2. **Check Transcripts Staging tab** — For transcript data, verify that staging records have **Completed** status (green badge). If records are stuck in **Pending** or show **Failed**: + - Filter by "Failed" status to identify problematic records + - Check if the conversation transcript data is still available (transcripts older than 30 days may have been deleted) + - Re-run the sync for the affected date range + +### Common Issues + +| Issue | Possible Cause | Resolution | +|---|---|---| +| No data after sync | App Insights not connected to agent | Verify Application Insights integration in Copilot Studio Settings > Advanced | +| Sync job shows Error | Invalid App Insights credentials | Check Application ID, Tenant ID, Client ID, and Secret in the agent configuration | +| "No insights yet" on all tabs | No sync has been run | Click "Sync" and run a sync for your desired date range | +| Partial data on some tabs | KPI Source not selected | Edit the agent configuration and ensure the correct KPI sources are checked (Application Insights, Conversation Transcripts, Usage History) | +| Transcript data missing for older dates | Transcripts deleted after 30 days | A bulk-delete system job removes transcripts older than 30 days. Sync regularly to capture data before it expires, or disable/modify the job — see [Change the default retention period](https://learn.microsoft.com/en-us/microsoft-copilot-studio/analytics-transcripts-powerapps#change-the-default-retention-period). | +| Usage History tab not visible | No usage data available | The Usage History feature requires the separate **AgentInventoryUsage** solution to be imported. This solution connects to the Power Platform licensing API to pull credit consumption data. See [Using Usage Metrics in Agent Inventory](https://github.com/microsoft/Power-CAT-Copilot-Studio-Kit/blob/main/AGENT_INVENTORY.md#using-usage-metrics-in-agent-inventory) for installation steps. During import, set up the "HTTP with Microsoft Entra ID (preauthorized)" connection pointing to `https://licensing.powerplatform.microsoft.com/`. Requires Power Platform admin role and system admin permissions. | +| Filters return no results | Restrictive filter combination | Try resetting to default filters (All agents, Last 90 days, All channels, Production) | +| Run button greyed out in Sync dialog | No configuration selected | Select an agent configuration or "All Agents" from the dropdown before clicking Run | + + +--- + +## Technical Details + +### Solution Components + +| Component | Type | Details | +|---|---|---| +| **Copilot Studio Kit** | Code App | cr7f9_copilotstudiokit — Built with React and Fluent UI v9 (not a traditional canvas app). Must be shared with users via make.powerapps.com. | + +### Cloud Flows + +| Flow Name | Trigger | Role | +|---|---|---| +| **Agent Insights \| On Demand** | Manual (Sync button) | Parent flow — orchestrates on-demand metric generation for a selected date range | +| **Agent Insights \| Pull Agent Metrics** | Child flow | Pulls Application Insights telemetry for a single agent configuration | +| **Agent Insights \| Transcript Metrics Date Range** | Grandchild flow | Coordinates transcript metric extraction across a date range | +| **Agent Insights \| Pull Transcript Metrics Per Date** | Great-grandchild flow | Processes transcript metrics for a single date | +| **Agent Insights \| Scheduler** | Scheduled (daily) | Runs automatically every day to pull the last 24 hours of telemetry and transcript data | + +### Connection References + +| Connection Reference | Connector | Required | +|---|---|---| +| **Copilot Studio Kit - Dataverse** | Microsoft Dataverse | Yes | + +### Dataverse Tables + +| Table | Logical Name | Purpose | +|---|---|---| +| **Action Metrics** | cat_ActionMetrics | Action node execution counts, types, and elapsed times | +| **Daily Metrics** | cat_DailyMetrics | Daily aggregated conversation, user, response time, and error metrics | +| **Topic Metrics** | cat_TopicMetrics | Per-topic trigger, completion, abandonment, duration, and error counts | +| **Tool Metrics** | cat_ToolMetrics | External tool/dependency call counts, success rates, and response times | +| **Transcript Metrics** | cat_TranscriptMetrics | Session-level engagement, resolution, escalation, CSAT, and feedback data | +| **Agent Insights Transcript Staging** | cat_AgentInsightsTranscriptStaging | Ingestion pipeline staging table for transcript processing | +| **Agent Fact Row Counts** | cat_AgentFactRowCounts | Row counts per agent per fact table (used for data management) | +| **Agent Configuration** | cat_CopilotConfiguration | Stores agent setup — App Insights credentials, KPI sources, capture settings | +| **Agent Details** | cat_AgentDetails | Agent metadata (dependency from Agent Inventory sync) | +| **Copilot Studio Kit Logs** | cat_CopilotStudioKitLogs | Sync execution logs — status, duration, error messages, flow run links | + +### Environment Variables + +No feature-specific environment variables. Application Insights credentials (App ID, Tenant ID, Client ID, Secret) are stored per-agent in the **Agent Configuration** table (cat_CopilotConfiguration). + +### DLP Connectors + +| Connector | Classification | +|---|---| +| **Microsoft Dataverse** | Business (required) | + +### Prerequisites Summary + +| Prerequisite | Why | +|---|---| +| **Agent Inventory sync** | Must run first to populate agent list for configuration selection | +| **Azure App Registration** | Required for programmatic access to Application Insights telemetry API | +| **Application Insights integration** | Agents must be connected to App Insights in Copilot Studio settings | +| **Code App sharing** | App owner must share the Code App with users via make.powerapps.com | + +--- + +*[← Back to Copilot Studio Kit Documentation](https://github.com/microsoft/Power-CAT-Copilot-Studio-Kit)* \ No newline at end of file diff --git a/AGENT_INVENTORY.md b/AGENT_INVENTORY.md index 16979672..a851526d 100644 --- a/AGENT_INVENTORY.md +++ b/AGENT_INVENTORY.md @@ -1,68 +1,115 @@ -# Agent Inventory (Preview) - -## Agent Inventory overview - -Copilot Studio Kit Agent Inventory feature can be used to easily get a tenant-wide visibility to all the Copilot Studio custom agents in the organization, across environments. Agent inventory data includes basic metadata like creation times, publish status and authentication mode, as well as information on the feature usage like knowledge sources used, usage of prompts, orchestration type and more. - -## Dashboard - -Agent Inventory feature ships with a dashboard that provides an overview on agents, growth and AI adoption. Detailed data is available for each agent and can be exported for use in other applications. - -![agent inventory dashboard](https://github.com/user-attachments/assets/f5c556f9-3dd0-41fb-acda-00c64c781b3e) - -## Detailed view - -From the Agent Inventory dashboard, users can see the total amount of agents in the tenant, usage % of generative AI features, actions and AI builder prompts -and how many agents are leveraging knowledge sources. Also visible are authentication mechanism used by the custom agents, agent creation timeline -visualizing the growth, and a list of top 5 environments by agent count. - -Selecting an agent and pressing *View details* brings up a detailed view of the selected agent, including basic metadata on the custom agent, -the environment, creation time and creator, and detailed information the usage of different features such as actions, generative AI, skills, prompts, -knowledge sources and more. - -![Copilot Studio Kit - Agent Details](https://github.com/user-attachments/assets/5e4e4344-2b0b-4ee6-91d9-7bc87f047fbe) - - -## List view - -And finally, pressing *Show more* from the dashboard view, brings up a list view where users can find the information they are looking for by filtering, sorting and adding additional columns. - -![agent inventory list view](https://github.com/user-attachments/assets/2e10abe5-e13e-4aae-a18b-ca6eb6c14469) - -## Using Usage Metrics in Agent Inventory -You can view usage details for your agent over the past 30 days in **Agent Inventory**. Agent Usage Metrics is distributed as an optional separate solution due to its connector requirements. - -### Prerequisites - -Before using the usage metrics feature: - -1. **Install** the **Copilot Studio Kit main solution**. -2. **Ensure** that the connector **HTTP with Microsoft Entra ID (preauthorized)** is allowed in your environment. - -### Installation Instructions - -To enable usage metrics on top of the Copilot Studio Kit main solution, you must **import the `AgentInventoryUsage` solution**, available in the **September release Assets directory**. - -During the import process, create a connection using the licensing host URL: https://licensing.powerplatform.microsoft.com/ - -agent inventory usage - -### How Usage Metrics Are Updated - -Usage data in the **Agent Details** table is refreshed in two ways: - -1. **Automatically** when the Agent inventory runs on a daily schedule. -2. **Manually** when you perform an **Agent Sync** operation. - -### Where to View Usage Metrics - -In the **Agent Inventory Dashboard**, review the **Agents** grid. -If the **Total Usage/Month** field contains a value, the **Usage Metrics** section will be displayed on the **Agent Details** page. - -![Copilot Studio Kit - Agent Details With Usage](https://github.com/user-attachments/assets/197f0539-016c-4c26-8439-e2382fab9349) - - -> [!NOTE] -> It is important to understand that the visibility to the agents is *limited* and *controlled* by the connection references in the solution. **Copilot Studio Kit - Power Platform for Admins** is used to fetch the list of environments in the tenant and **Copilot Studio Kit - Dataverse** is used to gather the agent information from the environments. For full visibility, the connection references have to be configured with account having Power Platform admin role and system admin level permission to all the environments. Other accounts can be used as well, but the visibility of the agent inventory is limited to the environments the user has system admin access to. - -Back to the [landing page](./README.md#power-cat-copilot-studio-kit) +# Agent Inventory (Preview) + +## Agent Inventory overview + +Copilot Studio Kit Agent Inventory feature can be used to easily get a tenant-wide visibility to all the Copilot Studio custom agents in the organization, across environments. Agent inventory data includes basic metadata like creation times, publish status and authentication mode, as well as information on the feature usage like knowledge sources used, usage of prompts, orchestration type and more. + +## Dashboard + +Agent Inventory feature ships with a dashboard that provides an overview on agents, growth and AI adoption. Detailed data is available for each agent and can be exported for use in other applications. + +![agent inventory dashboard](https://github.com/user-attachments/assets/f5c556f9-3dd0-41fb-acda-00c64c781b3e) + +## Detailed view + +From the Agent Inventory dashboard, users can see the total amount of agents in the tenant, usage % of generative AI features, actions and AI builder prompts +and how many agents are leveraging knowledge sources. Also visible are authentication mechanism used by the custom agents, agent creation timeline +visualizing the growth, and a list of top 5 environments by agent count. + +Selecting an agent and pressing *View details* brings up a detailed view of the selected agent, including basic metadata on the custom agent, +the environment, creation time and creator, and detailed information the usage of different features such as actions, generative AI, skills, prompts, +knowledge sources and more. + +![Copilot Studio Kit - Agent Details](https://github.com/user-attachments/assets/5e4e4344-2b0b-4ee6-91d9-7bc87f047fbe) + + +## List view + +And finally, pressing *Show more* from the dashboard view, brings up a list view where users can find the information they are looking for by filtering, sorting and adding additional columns. + +![agent inventory list view](https://github.com/user-attachments/assets/2e10abe5-e13e-4aae-a18b-ca6eb6c14469) + +## Using Usage Metrics in Agent Inventory +You can view usage details for your agent over the past 30 days in **Agent Inventory**. Agent Usage Metrics is distributed as an optional separate solution due to its connector requirements. + +### Prerequisites + +Before using the usage metrics feature: + +1. **Install** the **Copilot Studio Kit main solution**. +2. **Ensure** that the connector **HTTP with Microsoft Entra ID (preauthorized)** is allowed in your environment. + +### Installation Instructions + +To enable usage metrics on top of the Copilot Studio Kit main solution, you must **import the `AgentInventoryUsage` solution**, available in the **September release Assets directory**. + +During the import process, create a connection using the licensing host URL: https://licensing.powerplatform.microsoft.com/ + +### How Usage Metrics Are Updated + +Usage data in the **Agent Details** table is refreshed in two ways: + +1. **Automatically** when the Agent inventory runs on a daily schedule. +2. **Manually** when you perform an **Agent Sync** operation. + +### Where to View Usage Metrics + +In the **Agent Inventory Dashboard**, review the **Agents** grid. +If the **Total Usage/Month** field contains a value, the **Usage Metrics** section will be displayed on the **Agent Details** page. + +## Technical Details + +### Canvas Apps + +| Display Name | Logical Name | +|---|---| +| Agent Inventory | `cat_agentinventory` | +| Agent Inventory Component Library | `cat_mcsagentsinventorycomponentlibrary` | + +### Connection References + +| Connection Reference | Connector | Required | Purpose | +|---|---|---|---| +| Copilot Studio Kit - Dataverse | `shared_commondataserviceforapps` | ✅ Yes | Reads/writes agent data across environments | +| Copilot Studio Kit - Power Platform for Admins | `shared_powerplatformforadmins` | ✅ Yes | Fetches the list of environments in the tenant | +| Copilot Studio Kit - Power Platform for Admins V2 | `shared_powerplatformadminv2` | ✅ Yes | Enhanced admin APIs for environment and agent queries | + +> [!NOTE] +> Visibility to agents is controlled by these connection references. For full tenant-wide visibility, configure connection references with an account that has the **Power Platform admin role** and **system admin** permissions across all environments. Other accounts can be used, but agent inventory visibility will be limited to environments where the user has system admin access. + +### Environment Variables + +| Display Name | Logical Name | Description | +|---|---|---| +| Instance Url | `cat_InstanceUrl` | Dataverse environment URL for the current instance | + +### Cloud Flows + +| Flow Name | Trigger Type | Purpose | +|---|---|---| +| Agent Inventory \| Agents Data Load | Automated | Main orchestrator — initiates the full inventory sync across all environments | +| Agent Inventory \| Agents Data Load Child | Automated (child) | Per-environment processing — queries agents within a single environment | +| Agent Inventory \| Agents Data Load GrandChild | Automated (child) | Per-agent detail fetch — retrieves detailed metadata for an individual agent | +| Agent Inventory \| Agents Data Load On Demand | Instant (manual) | Manual trigger — lets users kick off an inventory sync on demand | +| Agent Inventory \| Agents Data Load Scheduler | Scheduled (daily) | Daily scheduled run — automatically refreshes inventory data every day | +| Agent Inventory \| Compliance Scan | Automated | Triggers a compliance check after inventory sync completes | + +### Dataverse Tables + +| Display Name | Logical Name | Description | +|---|---|---| +| Agent Details | `cat_AgentDetails` | Stores agent metadata collected from all environments | +| Agent Usage History | `cat_AgentUsageHistory` | Stores monthly usage metrics (requires optional `AgentInventoryUsage` solution) | + +### DLP Configuration + +The following connectors must be **allowed** in your environment's DLP policies for Agent Inventory to function: + +| Connector | Required For | +|---|---| +| Microsoft Dataverse | Core data storage and retrieval | +| Power Platform for Admins | Environment discovery | +| Power Platform for Admins V2 | Enhanced admin API access | +| HTTP with Microsoft Entra ID (preauthorized) | Optional — only needed for the Usage Metrics solution | + +Back to the [landing page](./README.md#power-cat-copilot-studio-kit) diff --git a/AGENT_REVIEW_TOOL.md b/AGENT_REVIEW_TOOL.md index 644873f6..d20cc418 100644 --- a/AGENT_REVIEW_TOOL.md +++ b/AGENT_REVIEW_TOOL.md @@ -1,38 +1,57 @@ -# Agent Review Tool -## Agent Review Tool overview - -Agent Review Tool is a solution analysis tool that can be used to review your agents for any potential issues or anti-patterns that might have negative impact on the performance or the security of your agent. After analysing the solution, Agent Review Tool presents the findings in easy to interpret format, with severity and details on how to address the issue. - -Agent Review Tool allows uploading a new solution package for analysis, or existing solution from the environment can be selected. The process starts by selecting "+ New Review" from the front page. - -![agent review tool](https://github.com/user-attachments/assets/85525ff7-95d0-499d-a371-5ea8507d12af) - -After selecting "+ New Review", the user can enter name for this review, description and pick the solution file. New solution can be uploaded as well. After selecting the solution file, the analysis will start by pressing "Save". Processing the solution can several minutes depending on the size of the solution, after which user is presented with the results. - -![agent review tool upload](https://github.com/user-attachments/assets/b31af6af-b16e-402e-8720-f310bb697587) - -## Results - -Agent Review Tool performs configuration analysis also for Power Apps and Power Automate components found in the solution, in addition to Copilot Studio custom agents. Results are presented in a format of a grouped list, and tabs can be used to filter findings to specific type. Each component is scored based on findings, lower score indicates higher amount of issues, anti-patterns and/or bad practices found in the component. - -Individual findings can be reviewed by clicking "**Review**" next to the component. - -![agent review tool results](https://github.com/user-attachments/assets/f4113a83-514b-492a-a0a4-8398e0f60816) - -## Details for a component - -In the detailed results view for a component, supported patterns and the analysis result are shown for each. Pie chart indicates the overall health for this specific component, with findings of different severity presented in a chart. - -User can further drill down on the findings to learn more on the issue and how to resolve it. - -![agent review tool findings](https://github.com/user-attachments/assets/704d9542-fa52-456b-850e-d6e5e924f791) - -## Details for a pattern - -Each pattern has a detailed description about the issue / bad practice / anti-pattern, information on how to address the issue, and often links to more documentation. - -![agent review tool pattern](https://github.com/user-attachments/assets/6aa68579-1623-49f1-85e5-49b603bca6a7) - -Performing periodical reviews helps the maker identify any changes that have been performed without following recommended and best practices , and could potentially have negative impact on the performance of the agent. - -Back to the [landing page](./README.md#power-cat-copilot-studio-kit) +# Agent Review Tool +## Agent Review Tool overview + +Agent Review Tool is a solution analysis tool that can be used to review your agents for any potential issues or anti-patterns that might have negative impact on the performance or the security of your agent. After analysing the solution, Agent Review Tool presents the findings in easy to interpret format, with severity and details on how to address the issue. + +Agent Review Tool allows uploading a new solution package for analysis, or existing solution from the environment can be selected. The process starts by selecting "+ New Review" from the front page. + +![agent review tool](https://github.com/user-attachments/assets/85525ff7-95d0-499d-a371-5ea8507d12af) + +After selecting "+ New Review", the user can enter name for this review, description and pick the solution file. New solution can be uploaded as well. After selecting the solution file, the analysis will start by pressing "Save". Processing the solution can several minutes depending on the size of the solution, after which user is presented with the results. + +![agent review tool upload](https://github.com/user-attachments/assets/b31af6af-b16e-402e-8720-f310bb697587) + +## Results + +Agent Review Tool performs configuration analysis also for Power Apps and Power Automate components found in the solution, in addition to Copilot Studio custom agents. Results are presented in a format of a grouped list, and tabs can be used to filter findings to specific type. Each component is scored based on findings, lower score indicates higher amount of issues, anti-patterns and/or bad practices found in the component. + +Individual findings can be reviewed by clicking "**Review**" next to the component. + +![agent review tool results](https://github.com/user-attachments/assets/f4113a83-514b-492a-a0a4-8398e0f60816) + +## Details for a component + +In the detailed results view for a component, supported patterns and the analysis result are shown for each. Pie chart indicates the overall health for this specific component, with findings of different severity presented in a chart. + +User can further drill down on the findings to learn more on the issue and how to resolve it. + +![agent review tool findings](https://github.com/user-attachments/assets/704d9542-fa52-456b-850e-d6e5e924f791) + +## Details for a pattern + +Each pattern has a detailed description about the issue / bad practice / anti-pattern, information on how to address the issue, and often links to more documentation. + +![agent review tool pattern](https://github.com/user-attachments/assets/6aa68579-1623-49f1-85e5-49b603bca6a7) + +Performing periodical reviews helps the maker identify any changes that have been performed without following recommended and best practices , and could potentially have negative impact on the performance of the agent. + +## Technical Details + +| Category | Details | +|---|---| +| **Canvas App** | None (uses model-driven app forms) | +| **Cloud Flows** | None specific | +| **Connection References** | Copilot Studio Kit - Dataverse | +| **Environment Variables** | None | +| **DLP Connectors** | Microsoft Dataverse | +| **Plugin Assembly** | POWERCAT.Plugins (used for solution analysis) | +| **Dataverse Tables** | Agent Review Tool FRE (`cat_agentreviewfre`) | +| | Agent Review (`cat_agentreviews`) | +| | Solution Component (`cat_SolutionComponent`) | +| | Solution Review (`cat_solutionreview`) | +| | Pattern Details (`cat_PatternDetails`) | +| | Pattern Identifier (`cat_PatternIdentifier`) | +| | Pattern Instance (`cat_PatternInstance`) | +| | Pattern Result (`cat_PatternResult`) | + +Back to the [landing page](./README.md#power-cat-copilot-studio-kit) diff --git a/AGENT_VALUE_SUMMARY_DASHBOARD.md b/AGENT_VALUE_SUMMARY_DASHBOARD.md index afe50003..796025f7 100644 --- a/AGENT_VALUE_SUMMARY_DASHBOARD.md +++ b/AGENT_VALUE_SUMMARY_DASHBOARD.md @@ -1,94 +1,137 @@ -# Agent value summary dashboard - -The **Agent Value Component** is a modular solution designed to classify conversational and autonomous agents in Microsoft Copilot Studio according to their **type**, **behavior**, and **value benefit**. This classification enables organizations to: - -* Understand the role and purpose of agents. -* Align agents with strategic goals. -* Quantify the value each agent delivers. - -The component is composed of: - -* A classification **Agent Value** table. -* A custom page that visualizes the spread and distribution of classified agents. -* Power Automate flows for automated classification using AI tools. -* An environment variable to enable the component - -This is version (v1) of the component. It classifies agents based on static metadata and prompt-driven logic. V2 may extend this with session-level telemetry for real-time value estimation. - -## Setup - -### Environment variable -The environment variable "**Enable Value Component**" must be set to 'Yes' in order for the automated flows to operate - -### Power Automate Flows -Must be 'Turned On' in order to operate - -### Populating the Agent Value table -The cloud flow "**Value Summary | Initialize agent value inventory**" must be manually run. - - -## Dataverse table - -### Agent Value -The Agent value table is the core metadata table in V1 of the agent value component. It stores the classification output for each agent, including: - -* Agent type -* Primary behavior -* Value benefit -* Date of classification - -This table is automatically populated by Power Automate flows using AI tools prompt responses. - -### Table schema - -| Column name | Type | Description | -|:-----------|:------|:-------------| -| AgentId | Guid | Id of classified agent| -| Name | Text | Agent display name | -| Agent Type | Text | One of: Collaborator, Performer, Retriever, Orchestrator, Assistant, Advisor | -| Agent Behavior | Text | One of Sense, Decide, Act, Collaborate, Reflect, Respond | -| Agent Value Benefit | Text | One of ten defined value benefits | -| Classification date | Date/Time | Date of agent classification | -| EnvironmentDisplayName | Text | Display name of agents environment | -| EnvironmentId | Text | Id of environment | - -## Power Automate Flows - -### Value summary | Initialize agent value inventory - -> **Important:** This flow must be enabled and run manually in order to populate the Agent Value table. Only agents with completed metadata (description and instructions) will be classified. - -**Triggered:** manually -**Purpose:** Iterates through all agents in the Agent Details table and classifies them if both the description and instructions are present. -**Actions:** -* Filter agents with missing metadata -* Call AI Builder 'Run a prompt' action with the embedded classification prompt -* Parse and update: -1. Agent Type -1. Agent Behavior -1. Agent Value Benefit -1. Environment Id -1. Environment Display Name -1. Classification date - -### Value summary | New agent classification - -**Triggered:** When a new row is added to Agent Details -**Purpose:** Runs classification immediately for new agents -**Actions:** -* Same logic as above, scoped to a single agent row - -## AI Tools prompt: Classify agent - -The "Run a prompt" action uses a structured system prompt to classify the agent. The embedded prompt includes: - -### Prompt objective - -Given an agent's instructions and description, assign it: - -- A primary Agent Type -- A dominant Behavior -- A Value benefit it most likely delivers -- A simple formula or signal to measure this value - -Back to the [landing page](./README.md#power-cat-copilot-studio-kit) +# Agent value summary dashboard + +The **Agent Value Component** is a modular solution designed to classify conversational and autonomous agents in Microsoft Copilot Studio according to their **type**, **behavior**, and **value benefit**. This classification enables organizations to: + +* Understand the role and purpose of agents. +* Align agents with strategic goals. +* Quantify the value each agent delivers. + +The component is composed of: + +* A classification **Agent Value** table. +* A custom page that visualizes the spread and distribution of classified agents. +* Power Automate flows for automated classification using AI tools. +* An environment variable to enable the component + +This is version (v1) of the component. It classifies agents based on static metadata and prompt-driven logic. V2 may extend this with session-level telemetry for real-time value estimation. + +## Setup + +### Environment variable +The environment variable "**Enable Value Component**" must be set to 'Yes' in order for the automated flows to operate + +### Power Automate Flows +Must be 'Turned On' in order to operate + +### Populating the Agent Value table +The cloud flow "**Value Summary | Initialize agent value inventory**" must be manually run. + + +## Dataverse table + +### Agent Value +The Agent value table is the core metadata table in V1 of the agent value component. It stores the classification output for each agent, including: + +* Agent type +* Primary behavior +* Value benefit +* Date of classification + +This table is automatically populated by Power Automate flows using AI tools prompt responses. + +### Table schema + +| Column name | Type | Description | +|:-----------|:------|:-------------| +| AgentId | Guid | Id of classified agent| +| Name | Text | Agent display name | +| Agent Type | Text | One of: Collaborator, Performer, Retriever, Orchestrator, Assistant, Advisor | +| Agent Behavior | Text | One of Sense, Decide, Act, Collaborate, Reflect, Respond | +| Agent Value Benefit | Text | One of ten defined value benefits | +| Classification date | Date/Time | Date of agent classification | +| EnvironmentDisplayName | Text | Display name of agents environment | +| EnvironmentId | Text | Id of environment | + +## Power Automate Flows + +### Value summary | Initialize agent value inventory + +> **Important:** This flow must be enabled and run manually in order to populate the Agent Value table. Only agents with completed metadata (description and instructions) will be classified. + +**Triggered:** manually +**Purpose:** Iterates through all agents in the Agent Details table and classifies them if both the description and instructions are present. +**Actions:** +* Filter agents with missing metadata +* Call AI Builder 'Run a prompt' action with the embedded classification prompt +* Parse and update: +1. Agent Type +1. Agent Behavior +1. Agent Value Benefit +1. Environment Id +1. Environment Display Name +1. Classification date + +### Value summary | New agent classification + +**Triggered:** When a new row is added to Agent Details +**Purpose:** Runs classification immediately for new agents +**Actions:** +* Same logic as above, scoped to a single agent row + +## AI Tools prompt: Classify agent + +The "Run a prompt" action uses a structured system prompt to classify the agent. The embedded prompt includes: + +### Prompt objective + +Given an agent's instructions and description, assign it: + +- A primary Agent Type +- A dominant Behavior +- A Value benefit it most likely delivers +- A simple formula or signal to measure this value + +## Technical Details + +### Connection References + +| Connection Reference | Connector | +|:---------------------|:----------| +| Copilot Studio Kit - Dataverse | Microsoft Dataverse | + +### Environment Variables + +| Display Name | Schema Name | Required | Description | +|:-------------|:------------|:---------|:------------| +| Enable Value Component | cat_Enablevaluecomponent | Yes | Must be set to "Yes" to activate the automated classification flows | + +### Cloud Flows + +| Display Name | Trigger | Purpose | +|:-------------|:--------|:--------| +| Value Summary \| Initialize agent value inventory | Manual | Iterates all agents and classifies them based on metadata | +| Value Summary \| New agent classification | Automated (new Agent Details row) | Classifies newly added agents automatically | + +### Dataverse Tables + +| Display Name | Schema Name | Purpose | +|:-------------|:------------|:--------| +| Agent Value | cat_Agentvalue | Stores classification output (type, behavior, value benefit) per agent | +| Customer | cat_Customer | Stores customer context associated with agent value tracking | + +### DLP Configuration + +This component requires the **Microsoft Dataverse** connector. Ensure your DLP policies allow this connector in the target environment. + +### Canvas App + +| Display Name | Schema Name | +|:-------------|:------------| +| Agent Value Summary | cat_agentvaluesummary | + +### Prerequisites + +- **Agent Inventory** must be populated before classification flows can operate. +- **AI Builder credits** must be assigned to the environment for the classification prompt to execute. + +Back to the [landing page](./README.md#power-cat-copilot-studio-kit) diff --git a/AUTOMATED_TESTING.md b/AUTOMATED_TESTING.md index 2616fa33..55107275 100644 --- a/AUTOMATED_TESTING.md +++ b/AUTOMATED_TESTING.md @@ -1,37 +1,46 @@ -# Automated testing and deployment of Copilot Studio agents using Power Platform Pipelines - -## Overview - -This feature allows users to automate the testing and deployment of Copilot Studio custom agents using Power Platform Pipelines. The goal is to ensure that agents are automatically validated through test runs before they are deployed to target environments (production). By integrating Power Automate flows with Dataverse and the Copilot Studio Kit, this approach introduces a quality gate into the deployment process. Only agents that pass the required amount of test cases are allowed to proceed, ensuring higher reliability and reducing manual intervention. This method supports continuous delivery practices and enhances the overall governance of the deployment lifecycle . - -## What it allows users to do - -*Automated Validation*: Automatically validate agent functionality through test runs before deployment. - -*Quality Gate*: Ensure only agents that pass the required amount of test cases proceed to deployment, enhancing reliability. - -*Continuous Delivery*: Support continuous delivery practices by integrating automated testing into the deployment lifecycle. - -*Governed Deployment*: Enforce governance and control over the deployment process, reducing manual steps and errors. - -## Requirements - -*Pipeline Host Environment*: The central control point for the deployment process. - -*Development (Source) Environment*: Where the Copilot Studio agent is developed and tested. - -*Target (Prod) Environment*: The final destination for deployment, where the agent is published after passing all validation checks. - -## High-Level Steps - -- Open the Deployment Pipeline Configuration App: Launch the app from the Power Apps portal to create and manage deployment pipelines. -- Configure the Pipeline: Provide a pipeline name and link development environments. -- Configure Deployment Stages: Add deployment stages to define target environments and enable pre-export and pre-deployment steps. -- Trigger on Deployment Request: The flow is triggered when a deployment request is initiated for a solution that includes a Copilot Studio Agent. -- Run Automated Tests: The flow pauses the deployment, runs automated tests, evaluates test results, and decides whether to proceed with deployment based on the test outcomes. - -## Detailed documentation - -[Automated Testing and Deployment of Copilot Studio Agents Overview](https://github.com/user-attachments/files/21301068/Automated.Testing.and.Deployment.of.Copilot.Studio.Agents.Overview.pdf) - - +# Automated testing and deployment of Copilot Studio agents using Power Platform Pipelines + +## Overview + +This feature allows users to automate the testing and deployment of Copilot Studio custom agents using Power Platform Pipelines. The goal is to ensure that agents are automatically validated through test runs before they are deployed to target environments (production). By integrating Power Automate flows with Dataverse and the Copilot Studio Kit, this approach introduces a quality gate into the deployment process. Only agents that pass the required amount of test cases are allowed to proceed, ensuring higher reliability and reducing manual intervention. This method supports continuous delivery practices and enhances the overall governance of the deployment lifecycle . + +## What it allows users to do + +*Automated Validation*: Automatically validate agent functionality through test runs before deployment. + +*Quality Gate*: Ensure only agents that pass the required amount of test cases proceed to deployment, enhancing reliability. + +*Continuous Delivery*: Support continuous delivery practices by integrating automated testing into the deployment lifecycle. + +*Governed Deployment*: Enforce governance and control over the deployment process, reducing manual steps and errors. + +## Requirements + +*Pipeline Host Environment*: The central control point for the deployment process. + +*Development (Source) Environment*: Where the Copilot Studio agent is developed and tested. + +*Target (Prod) Environment*: The final destination for deployment, where the agent is published after passing all validation checks. + +## High-Level Steps + +- Open the Deployment Pipeline Configuration App: Launch the app from the Power Apps portal to create and manage deployment pipelines. +- Configure the Pipeline: Provide a pipeline name and link development environments. +- Configure Deployment Stages: Add deployment stages to define target environments and enable pre-export and pre-deployment steps. +- Trigger on Deployment Request: The flow is triggered when a deployment request is initiated for a solution that includes a Copilot Studio Agent. +- Run Automated Tests: The flow pauses the deployment, runs automated tests, evaluates test results, and decides whether to proceed with deployment based on the test outcomes. + +## Detailed documentation + +[Automated Testing and Deployment of Copilot Studio Agents Overview](https://github.com/user-attachments/files/21301068/Automated.Testing.and.Deployment.of.Copilot.Studio.Agents.Overview.pdf) + +## Technical Details + +| Category | Details | +|---|---| +| **Cloud Flows** | Pipeline \| Validate Agent Using Test Cases | +| **Connection References** | Copilot Studio Kit - Dataverse | +| **Environment Variables** | None specific | +| **DLP Connectors** | Microsoft Dataverse | +| **Dataverse Tables** | Deployment Pipeline Configuration (`cat_DeploymentPipelineConfiguration`) | +| **Prerequisites** | Power Platform Pipelines must be configured; Test Automation feature must be set up | diff --git a/COMPLIANCE_HUB.md b/COMPLIANCE_HUB.md index e4677f66..683ca39e 100644 --- a/COMPLIANCE_HUB.md +++ b/COMPLIANCE_HUB.md @@ -1,390 +1,481 @@ -# Define and enforce agent compliance - -The Compliance Hub enables organizations using Copilot Studio to balance innovation with compliance. It provides visibility, automated compliance enforcement, and clear workflows for both administrators and makers. The system implements continuous compliance monitoring — agents are created freely within product guardrails configured in Power Platform admin center, but compliance is enforced after creation based on configurable risk thresholds. If compliance is not achieved within a defined SLA, automated enforcement actions (such as quarantine or delete) are applied. - -![Compliance Hub Dashboard Overview](./media/ch_0.png) -*Figure 1: Compliance Hub dashboard showing agent compliance status, open cases, and key performance indicators* - -## Key concepts -- Continuous compliance monitoring: Agents are created without friction within allowed limits configured by admin center controls; compliance is enforced post-creation if risk thresholds are breached. -- Compliance case: Automatically created when an agent breaches a risk threshold. Tracks remediation, SLA, and enforcement. -- SLA (Service Level Agreement): The time window for makers to remediate compliance issues before enforcement actions are triggered. -- Enforcement actions: Automated actions (Manual review, Quarantine, Delete) applied if compliance is not achieved within SLA. - - **Manual review**: Admin receives notification to review and take action. No automated enforcement. - - **Quarantine**: Agent is automatically disabled and cannot be invoked by end users. Agent owner retains access to edit and fix compliance issues. - - **Delete**: Agent is permanently deleted from the environment. This action cannot be undone. - -## Prerequisites - -Before setting up the Compliance Hub, ensure you have: - -- **Copilot Studio Kit prerequisites**: All [prerequisites for the Copilot Studio Kit](PREREQUISITES.md) -- **Permissions (for admin)**: - - System Administrator role in the target environment - - Ability to create and configure cloud flows - - Microsoft Entra ID (Azure AD) permissions to retrieve group IDs -- **Dependencies**: - - [Agent Inventory](./AGENT_INVENTORY.md) must be installed and configured first - - Microsoft Teams for notification functionality - - Outlook for additional notification channel -- **Data Loss Prevention policy** must allow the following connectors: - - Microsoft Dataverse - - Microsoft Teams - - Microsoft Standard Approvals - - Office 365 Outlook - - Office 365 Users - - Power Platform for Admins - - Power Platform for Admins (V2) - -## Roles and responsibilities - -There are two main personas involved in the main agent compliance workstream: agent administrator and agent maker. - -### Administrator - -The scope of the agent admin depends on the organization, however, the admin responsibilities for managing compliance commonly falls under the responsibility of either Microsoft tenant admin, Power Platform admin or the Agent/AI Center Of Excellence team. - -- Configure compliance thresholds, risk levels, enforcement actions, and SLA timers. -- Monitor agent inventory and compliance posture. -- Review, approve, or reject compliance cases. -- Maintain audit trails and documentation. - -### Maker - -Any user who has created a Copilot Studio agent in the organization's tenant. This agent creator is identified as the owning maker of the agent. - -- Receives notifications when compliance thresholds are breached. -- Provides business justification and confirms ownership for non-compliant agents. -- Remediates compliance issues within the SLA. -- Views compliance status and receives reminders. - -> **Note**: -> If the user who created the agent has left the company, the agent ownership is automatically reassigned to the environment system administrator. - -## Get started - -Setup and configure the governance components using the Copilot Studio Kit setup wizard. - -**Estimated time to complete**: 20 minutes (excluding prerequisite installations) - -1. Install and launch the Copilot Studio Kit. Follow the [installation instructions](./INSTALLATION_INSTRUCTIONS.md) and [prerequisites](./PREREQUISITES.md) for the essential install steps to install the base solution and [agent inventory](./AGENT_INVENTORY.md) before setting up the compliance components. - -1. Run the **setup wizard**. - -1. Update the **compliance hub environment variables**: - - Below is a list of the environment variables used in the solution. Variables are categorized by priority to help you configure essential settings first. - - ### Critical variables (Required) - - These variables must be configured before enabling the cloud flows for the Compliance Hub to function properly: - - | Display Name| Description| Example | - |--------------------------------------------|-----------------------------------------------------------------------------|------------------------| - | App ID | Unique identifier for the Copilot Studio Kit application | `12345678-1234-1234-1234-123456789abc` | - | Compliance Admin Group ID | Microsoft Entra ID (Azure AD) security group ID for compliance administrators | `87654321-4321-4321-4321-cba987654321` | - | Instance Url | URL of the environment instance | `https://orgname.crm.dynamics.com` | - | Maker Team ID | Group ID for makers in the organization | `12345678-1234-1234-1234-123456789abc` | - - > **Note**: Instance URL and App ID can be found in the player URL - > - > ![Locating Instance URL and App ID](media/ch_4.png) - > *Finding Instance URL and App ID in the Copilot Studio player URL* - - ### Recommended variables (optional but recommended) - - Configure these for a better user experience: - - | Display Name| Description| Example | - |--------------------------------------------|-----------------------------------------------------------------------------|------------------------| - | Compliance Documentation Link | URL to your organization's compliance documentation | `https://docs.contoso.com/compliance` | - | Compliance Support Contact Alias | Email alias for compliance support | `compliance-support@contoso.com` | - - ### Default Configuration Variables - - The following environment variables do not require additional configuration. The default values are recommended for first time setup. You may update values at your discretion during solution import or in the compliance hub settings page. - - - | Display Name| Description| Default value | - |--------------------------------------------|-----------------------------------------------------------------------------|------------------------| - | Admin Approval Before Maker Notification | Determines if admin approval is needed before notifying makers | Yes | - | Case Intake SLA | Service Level Agreement for case intake timing | 3 days | - | Case Review SLA | SLA for admins to review cases in the compliance process | 5 days | - | Case Summary Email Frequency | How often case summary emails are sent |WEEKLY| - | Require Case For No Risk | Indicates if a case is required even when risk is low | Yes | - | Send Case Alerts To Maker Via Email | Enables email alerts for makers when cases are triggered | Yes | - | Send Case Alerts To Maker Via Teams | Enables Teams alerts for makers when cases are triggered | Yes | - - > **Important:**
- > - **Admin Approval Before Maker Notification** - Recommend keeping this value enabled during initial configuration to avoid excessive notifications and prevent unwanted governance action automation on non-compliant cases (quarantine or delete). When enabled, the system will not immediately send notifications or start any SLA countdowns unless an admin decides to send the individual notification on the case details screen in compliance hub. This allows admins to: - > - Configure acceptable compliance thresholds - > - Run the compliance scan to review the resulting caseload - > - Adjust configurations based on team capacity and resources - > - Verify the policies accurately reflect organizational requirements - > - > Once thresholds are configured and validated, disable this setting so notices and policy enforcement resume automatically. Note: Bulk approval of cases is not currently available. - > - > - **Require Case For No Risk**: Disabled by default. When disabled, cases are only created for agents that violate a compliance threshold. If enabled, a case will be created for all agents, even if there is no violation. - -1. **Enable the agent inventory and compliance cloud flows** - - Navigate to Solutions > Copilot Studio Accelerator > Cloud flows in your Power Platform environment. - - Enable flows in the following order: - - 1. First, enable all flows marked as **(Grandchild)** - 2. Then, enable all flows marked as **(Child)** - 3. Finally, enable parent flows - - Flows to enable: - - All flows starting with "Agent Compliance |" - - All flows starting with "Agent Inventory |" - - > [!TIP] - > After enabling flows, verify they are running successfully by checking the run history for each flow. - - - -## Configure governance policies -Governance policies in compliance hub are configured by: -- Defining compliance 'violations' (by associating an agent configuration with a risk severity). -- Specifying actions taken when violations occur (for each severity: high, medium and low). - -The following components can be configured to define and enforce the compliance policies that best define your organization's requirements. - -### Compliance thresholds (`Threshold Config` table) -This is how 'violations' are defined in compliance hub. Each row in the `Threshold Config` table must be associated with a specific agent attribute (aka property/column) and a specific value. It is designed to be fully customizable within the scope of the agent components represented in Copilot Studio Kit's `Agent Details` custom table properties. - -![Threshold Configuration Table](./media/ch_1.png) -*Figure 2: Default configurations for Threshold Config table showing filter columns, operators, and risk levels* - -Each row contains the following fields used to construct a filter on the `Agent Detail` table. If the filter returns true for that agent, it will be flagged as a violation. - -- **Filter Column**: (Text) The logical name of the column that is evaluated (e.g., `cr6d0_authenticationmode`). -- **Filter Operator**: (Choice) The operator of the filter (e.g., Equals, Not Equals, Contains, Greater Than, Less Than). -- **Default Value**: (Text) The value to compare against. This represents the threshold value that triggers a violation (e.g., `No Authentication`, `true`, `100`). - -**Example threshold configuration:** -- Filter Column: `cat_enduserauthenticationtype` -- Filter Operator: `Equals` -- Default Value: `OAuth2` -- Risk Level: `Medium` - -This configuration will create a Medium risk case for agents using OAuth2 authentication. - -![Example Threshold Configuration](media/ch_3.png) -*Figure 4: Threshold configuration example showing OAuth2 authentication as a medium risk violation* - - -### Enforcement actions (`Action Policies` table) -Customize actions (Manual, Quarantine, Delete) and SLA timers for each risk level (Low, Medium, High). - -#### Example: Enforcement Actions by Risk Level - -| Risk Level | SLA Days | Enforcement Action |Description | -|-|-|-|-| -|Low |10| Manual| Admin reviews/reminders| -|Medium |5| Quarantine| Agent disabled if overdue| -|High| 3| Delete| Agent deleted if overdue| - -### Manage compliance cases and configure settings - -- Use the Dashboard to view KPIs (Total Agents, Open Cases, SLA Breaches, etc.). -- Access Inventory for a holistic list of agents. -- Adjust settings for thresholds, action policies, email templates, and environment variables as needed on the Settings page. - -#### Impact of threshold changes on existing cases - -When you modify compliance thresholds: -- **In-flight cases** will be evaluated against the updated standards during the next daily compliance scan -- **SLA deadlines remain unchanged** - the original SLA countdown continues from when the case was created -- **Action required**: Makers must still remediate within the original timeline to avoid enforcement actions -- **Best practice**: Communicate threshold changes to makers with active cases to avoid confusion - -## Compliance review lifecycle - -The compliance case lifecycle is a specialized approval process designed to help identify and manage agents that violate the compliance policies defined in this tool. - -The case stages define the current position in the lifecycle and next actions available, and shown below. - -![Compliance Case Lifecycle](./media/ch_2.png) -*Figure 3: Compliance case lifecycle showing stages from case creation through resolution or enforcement* - -### Compliance enforcement workflow - -Below is a high-level process for agents violating compliance thresholds: - -1. Agent is created or updated. -2. Daily automated detection checks for compliance. -3. If thresholds are breached: - - Compliance case is opened. - - Maker is notified and must complete intake. - - SLA countdown begins. -4. Admin reviews case and intake. -5. If compliant, case is approved. -6. If not compliant within SLA, enforcement action is applied (delete). - -![Flowchart of the high level overview of the compliance process flow](media/ch_6.png) - -### Run compliance scans - -The compliance scan is an asynchronous process that finds violations based on data in the `Agent Details` table, using the governance policies defined. - -Compliance scans are automatically run after the Agent Inventory scan is completed (typically daily). The compliance scan can also be manually triggered from the Compliance Hub dashboard by clicking the **Run Compliance Scan** button. - -The compliance scan will create new or update existing `Compliance Case` rows for violations. - -#### What the scan evaluates: - -- All agent configurations against defined threshold rules -- Risk level assignment based on matched thresholds -- SLA calculation and countdown initiation -- Notification triggers for makers and admins - -### Monitor and manage compliance cases - -1. **Navigate to compliance cases** - - Open the Compliance Hub app from your Power Apps environment - - Navigate to **Compliance Cases** in the left navigation menu - - Filter cases by status (Open, SLA Breach, Approved, Expired) and risk level (High, Medium, Low) - -2. **Review case details** - - Open a compliance case record to view: - - **Triggered thresholds**: Which policies were violated - - **Maker's intake**: Business justification, owner confirmation, data classification - - **Automated scan results**: DLP policies, connectors used, knowledge sources - - **SLA countdown**: Time remaining before enforcement action - - **Event timeline**: Historical record of all case activities - - ![Compliance Case Details View](media/ch_5.png) - *Figure 5: Compliance case details showing triggered thresholds, maker intake, and SLA countdown* - -3. **Approve or reject cases** - - Use the **Approve/Reject** buttons in the command bar to make decisions - - Add audit notes for documentation and future reference - - If intake is incomplete or violations persist past SLA, enforcement actions are triggered automatically - -4. **Audit and reporting** - - Use dashboards and charts for trend analysis (e.g., distribution of important case statuses, risk level distribution). - - Export case data for external reporting and compliance documentation. - - Maintain thorough documentation for all compliance cases. - - Enable auditing in the Power Platform Admin Center (PPAC) environment settings to ensure case level auditing is available. - - Timeline activities are also posted on cases for historical record - -## Agent maker responsibilities - -When a compliance case is opened, you'll receive a Microsoft Teams notification linked to your case. - -### Step-by-step guide for makers - -1. **Complete intake form** - - The Microsoft Teams notification is an adaptive card that accepts three input fields: - - - **Business justification**: Explain the purpose and business value of this agent - - **Data classification**: Specify what type of data the agent handles (Public, Internal, Confidential, Highly Confidential) - - **Expected number of users**: Estimate how many people will use this agent - - Submit the form directly from Teams to update your case. - -2. **Remediate compliance issues** - - Fix any compliance violations identified in the case that are medium or high risk: - - - Review the specific thresholds that were violated - - Make necessary changes in Copilot Studio (e.g., enable authentication, restrict sharing, add DLP policies) - - Publish your changes - - Re-evaluation happens daily (typically overnight), and you will be notified of the results - -3. **Monitor SLA countdown** - - - Track the SLA countdown via notifications that are sent daily - - You'll receive reminders as the SLA deadline approaches (3 days before, 1 day before, day of) - - Failure to comply within the SLA may result in agent quarantine or deletion - - Contact your compliance admin if you need an extension - -### Example scenarios - -**Scenario 1: No authentication configured** -- Violation: Agent has no authentication enabled (High risk) -- Remediation: Enable Microsoft Entra ID authentication in Copilot Studio security settings -- Timeline: 3-day SLA for high-risk violations - -**Scenario 2: Published to production without review** -- Violation: Agent published to production environment without admin review (low risk) -- Remediation: Provide detailed business justification and general intake. -- Timeline: Admin review required within 5 days - -## Reference: Data model and policies - -### Key Tables - -- **Compliance Case**: Tracks workflow, SLA, and enforcement. -- **Threshold Config**: Stores threshold settings. -- **Action Policy**: Defines enforcement actions per risk level. -- **Email Template**: Manages notification templates. - -> **Note**: -> If a threshold is defined for all possibilities in a certain category (for example, for all auth modes), then a case will be created for every agent. - -## FAQs and troubleshooting - -### General questions - -**Q: What happens if I don't complete intake within the SLA?** - -A: The system will automatically apply the configured enforcement action (e.g., quarantine or delete the agent). You'll receive daily reminder notifications as the SLA deadline approaches. - -**Q: How do I know which compliance issues to fix as a maker?** - -A: Review the checklist and findings in your compliance case. Automated checks highlight any "Fail" items that need to be addressed. Focus on high and medium risk violations first. - -**Q: Who do I contact for help as a maker?** - -A: Use the contact information provided in your notification email or reach out to your CoE Admin team. The compliance support contact alias is configured in the environment variables. - -**Q: Can I request an SLA extension as a maker?** - -A: Contact your compliance administrator to discuss SLA extensions. Extensions must be documented in the case audit notes. - -**Q: What happens to a quarantined agent?** - -A: A quarantined agent is disabled and cannot be accessed by end users. However, you as the owner can still edit the agent to fix compliance issues. Once compliant, request admin review to restore the agent. - -### Troubleshooting - -**Issue: Compliance scan is not detecting my agent** - -- Ensure Agent Inventory scan has completed successfully -- Check that your agent is published and visible in Copilot Studio -- Verify the agent is in an environment being monitored by the solution -- Review cloud flow run history for errors - -**Issue: Case status is not updating after remediation** - -- Compliance re-evaluation occurs daily (typically runs overnight) -- Manually trigger a compliance scan from the Compliance Hub dashboard -- Verify remediation changes are saved and published in Copilot Studio Kit Inventory -- Check the case timeline for evaluation history - -**Issue: Filter operators not working as expected** - -- Available filter operators: Equals -- Ensure the "Value" field matches the exact case and formatting of the column being evaluated -- Test your threshold configuration on a small sample size agent configuration before rolling out broadly - -### Best practices - -- **Start with Manual review enforcement**: Configure low-risk enforcement initially while testing thresholds. -- **Test with non-production agents first**: Validate your compliance policies don't have unintended consequences -- **Document your policies**: Maintain clear documentation of why each threshold exists and how to remediate -- **Regular policy reviews**: Schedule quarterly reviews of compliance thresholds and enforcement actions -- **Monitor SLA metrics**: Track average time to remediation and adjust SLAs if consistently missed - -### Additional resources - -- [Agent Inventory Documentation](./AGENT_INVENTORY.md) -- [Installation Instructions](./INSTALLATION_INSTRUCTIONS.md) -- [Prerequisites](./PREREQUISITES.md) -- [Troubleshooting Guide](./TROUBLESHOOT.md) +# Define and enforce agent compliance + +The Compliance Hub enables organizations using Copilot Studio to balance innovation with compliance. It provides visibility, automated compliance enforcement, and clear workflows for both administrators and makers. The system implements continuous compliance monitoring — agents are created freely within product guardrails configured in Power Platform admin center, but compliance is enforced after creation based on configurable risk thresholds. If compliance is not achieved within a defined SLA, automated enforcement actions (such as quarantine or delete) are applied. + +![Compliance Hub Dashboard Overview](./media/ch_0.png) +*Figure 1: Compliance Hub dashboard showing agent compliance status, open cases, and key performance indicators* + +## Key concepts +- Continuous compliance monitoring: Agents are created without friction within allowed limits configured by admin center controls; compliance is enforced post-creation if risk thresholds are breached. +- Compliance case: Automatically created when an agent breaches a risk threshold. Tracks remediation, SLA, and enforcement. +- SLA (Service Level Agreement): The time window for makers to remediate compliance issues before enforcement actions are triggered. +- Enforcement actions: Automated actions (Manual review, Quarantine, Delete) applied if compliance is not achieved within SLA. + - **Manual review**: Admin receives notification to review and take action. No automated enforcement. + - **Quarantine**: Agent is automatically disabled and cannot be invoked by end users. Agent owner retains access to edit and fix compliance issues. + - **Delete**: Agent is permanently deleted from the environment. This action cannot be undone. + +## Prerequisites + +Before setting up the Compliance Hub, ensure you have: + +- **Copilot Studio Kit prerequisites**: All [prerequisites for the Copilot Studio Kit](PREREQUISITES.md) +- **Permissions (for admin)**: + - System Administrator role in the target environment + - Ability to create and configure cloud flows + - Microsoft Entra ID (Azure AD) permissions to retrieve group IDs +- **Dependencies**: + - [Agent Inventory](./AGENT_INVENTORY.md) must be installed and configured first + - Microsoft Teams for notification functionality + - Outlook for additional notification channel +- **Data Loss Prevention policy** must allow the following connectors: + - Microsoft Dataverse + - Microsoft Teams + - Microsoft Standard Approvals + - Office 365 Outlook + - Office 365 Users + - Office 365 Groups + - Power Platform for Admins + - Power Platform for Admins (V2) + +## Roles and responsibilities + +There are two main personas involved in the main agent compliance workstream: agent administrator and agent maker. + +### Administrator + +The scope of the agent admin depends on the organization, however, the admin responsibilities for managing compliance commonly falls under the responsibility of either Microsoft tenant admin, Power Platform admin or the Agent/AI Center Of Excellence team. + +- Configure compliance thresholds, risk levels, enforcement actions, and SLA timers. +- Monitor agent inventory and compliance posture. +- Review, approve, or reject compliance cases. +- Maintain audit trails and documentation. + +### Maker + +Any user who has created a Copilot Studio agent in the organization's tenant. This agent creator is identified as the owning maker of the agent. + +- Receives notifications when compliance thresholds are breached. +- Provides business justification and confirms ownership for non-compliant agents. +- Remediates compliance issues within the SLA. +- Views compliance status and receives reminders. + +> **Note**: +> If the user who created the agent has left the company, the agent ownership is automatically reassigned to the environment system administrator. + +## Get started + +Setup and configure the governance components using the Copilot Studio Kit setup wizard. + +**Estimated time to complete**: 20 minutes (excluding prerequisite installations) + +1. Install and launch the Copilot Studio Kit. Follow the [installation instructions](./INSTALLATION_INSTRUCTIONS.md) and [prerequisites](./PREREQUISITES.md) for the essential install steps to install the base solution and [agent inventory](./AGENT_INVENTORY.md) before setting up the compliance components. + +1. Run the **setup wizard**. + +1. Update the **compliance hub environment variables**: + + Below is a list of the environment variables used in the solution. Variables are categorized by priority to help you configure essential settings first. + + ### Critical variables (Required) + + These variables must be configured before enabling the cloud flows for the Compliance Hub to function properly: + + | Display Name| Description| Example | + |--------------------------------------------|-----------------------------------------------------------------------------|------------------------| + | App ID | Unique identifier for the Copilot Studio Kit application | `12345678-1234-1234-1234-123456789abc` | + | Compliance Admin Group ID | Microsoft Entra ID (Azure AD) security group ID for compliance administrators | `87654321-4321-4321-4321-cba987654321` | + | Instance Url | URL of the environment instance | `https://orgname.crm.dynamics.com` | + | Maker Team ID | Group ID for makers in the organization | `12345678-1234-1234-1234-123456789abc` | + + > **Note**: Instance URL and App ID can be found in the player URL + > + > ![Locating Instance URL and App ID](media/ch_4.png) + > *Finding Instance URL and App ID in the Copilot Studio player URL* + + ### Recommended variables (optional but recommended) + + Configure these for a better user experience: + + | Display Name| Description| Example | + |--------------------------------------------|-----------------------------------------------------------------------------|------------------------| + | Compliance Documentation Link | URL to your organization's compliance documentation | `https://docs.contoso.com/compliance` | + | Compliance Support Contact Alias | Email alias for compliance support | `compliance-support@contoso.com` | + + ### Default Configuration Variables + + The following environment variables do not require additional configuration. The default values are recommended for first time setup. You may update values at your discretion during solution import or in the compliance hub settings page. + + + | Display Name| Description| Default value | + |--------------------------------------------|-----------------------------------------------------------------------------|------------------------| + | Admin Approval Before Maker Notification | Determines if admin approval is needed before notifying makers | Yes | + | Case Intake SLA | Service Level Agreement for case intake timing | 3 days | + | Case Review SLA | SLA for admins to review cases in the compliance process | 5 days | + | Case Summary Email Frequency | How often case summary emails are sent |WEEKLY| + | Require Case For No Risk | Indicates if a case is required even when risk is low | Yes | + | Send Case Alerts To Maker Via Email | Enables email alerts for makers when cases are triggered | Yes | + | Send Case Alerts To Maker Via Teams | Enables Teams alerts for makers when cases are triggered | Yes | + + > **Important:**
+ > - **Admin Approval Before Maker Notification** - Recommend keeping this value enabled during initial configuration to avoid excessive notifications and prevent unwanted governance action automation on non-compliant cases (quarantine or delete). When enabled, the system will not immediately send notifications or start any SLA countdowns unless an admin decides to send the individual notification on the case details screen in compliance hub. This allows admins to: + > - Configure acceptable compliance thresholds + > - Run the compliance scan to review the resulting caseload + > - Adjust configurations based on team capacity and resources + > - Verify the policies accurately reflect organizational requirements + > + > Once thresholds are configured and validated, disable this setting so notices and policy enforcement resume automatically. Note: Bulk approval of cases is not currently available. + > + > - **Require Case For No Risk**: Disabled by default. When disabled, cases are only created for agents that violate a compliance threshold. If enabled, a case will be created for all agents, even if there is no violation. + +1. **Enable the agent inventory and compliance cloud flows** + + Navigate to Solutions > Copilot Studio Accelerator > Cloud flows in your Power Platform environment. + + Enable flows in the following order: + + 1. First, enable all flows marked as **(Grandchild)** + 2. Then, enable all flows marked as **(Child)** + 3. Finally, enable parent flows + + Flows to enable: + - All flows starting with "Agent Compliance |" + - All flows starting with "Agent Inventory |" + + > [!TIP] + > After enabling flows, verify they are running successfully by checking the run history for each flow. + + + +## Configure governance policies +Governance policies in compliance hub are configured by: +- Defining compliance 'violations' (by associating an agent configuration with a risk severity). +- Specifying actions taken when violations occur (for each severity: high, medium and low). + +The following components can be configured to define and enforce the compliance policies that best define your organization's requirements. + +### Compliance thresholds (`Threshold Config` table) +This is how 'violations' are defined in compliance hub. Each row in the `Threshold Config` table must be associated with a specific agent attribute (aka property/column) and a specific value. It is designed to be fully customizable within the scope of the agent components represented in Copilot Studio Kit's `Agent Details` custom table properties. + +![Threshold Configuration Table](./media/ch_1.png) +*Figure 2: Default configurations for Threshold Config table showing filter columns, operators, and risk levels* + +Each row contains the following fields used to construct a filter on the `Agent Detail` table. If the filter returns true for that agent, it will be flagged as a violation. + +- **Filter Column**: (Text) The logical name of the column that is evaluated (e.g., `cr6d0_authenticationmode`). +- **Filter Operator**: (Choice) The operator of the filter (e.g., Equals, Not Equals, Contains, Greater Than, Less Than). +- **Default Value**: (Text) The value to compare against. This represents the threshold value that triggers a violation (e.g., `No Authentication`, `true`, `100`). + +**Example threshold configuration:** +- Filter Column: `cat_enduserauthenticationtype` +- Filter Operator: `Equals` +- Default Value: `OAuth2` +- Risk Level: `Medium` + +This configuration will create a Medium risk case for agents using OAuth2 authentication. + +![Example Threshold Configuration](media/ch_3.png) +*Figure 4: Threshold configuration example showing OAuth2 authentication as a medium risk violation* + + +### Enforcement actions (`Action Policies` table) +Customize actions (Manual, Quarantine, Delete) and SLA timers for each risk level (Low, Medium, High). + +#### Example: Enforcement Actions by Risk Level + +| Risk Level | SLA Days | Enforcement Action |Description | +|-|-|-|-| +|Low |10| Manual| Admin reviews/reminders| +|Medium |5| Quarantine| Agent disabled if overdue| +|High| 3| Delete| Agent deleted if overdue| + +### Manage compliance cases and configure settings + +- Use the Dashboard to view KPIs (Total Agents, Open Cases, SLA Breaches, etc.). +- Access Inventory for a holistic list of agents. +- Adjust settings for thresholds, action policies, email templates, and environment variables as needed on the Settings page. + +#### Impact of threshold changes on existing cases + +When you modify compliance thresholds: +- **In-flight cases** will be evaluated against the updated standards during the next daily compliance scan +- **SLA deadlines remain unchanged** - the original SLA countdown continues from when the case was created +- **Action required**: Makers must still remediate within the original timeline to avoid enforcement actions +- **Best practice**: Communicate threshold changes to makers with active cases to avoid confusion + +## Compliance review lifecycle + +The compliance case lifecycle is a specialized approval process designed to help identify and manage agents that violate the compliance policies defined in this tool. + +The case stages define the current position in the lifecycle and next actions available, and shown below. + +![Compliance Case Lifecycle](./media/ch_2.png) +*Figure 3: Compliance case lifecycle showing stages from case creation through resolution or enforcement* + +### Compliance enforcement workflow + +Below is a high-level process for agents violating compliance thresholds: + +1. Agent is created or updated. +2. Daily automated detection checks for compliance. +3. If thresholds are breached: + - Compliance case is opened. + - Maker is notified and must complete intake. + - SLA countdown begins. +4. Admin reviews case and intake. +5. If compliant, case is approved. +6. If not compliant within SLA, enforcement action is applied (delete). + +![Flowchart of the high level overview of the compliance process flow](media/ch_6.png) + +### Run compliance scans + +The compliance scan is an asynchronous process that finds violations based on data in the `Agent Details` table, using the governance policies defined. + +Compliance scans are automatically run after the Agent Inventory scan is completed (typically daily). The compliance scan can also be manually triggered from the Compliance Hub dashboard by clicking the **Run Compliance Scan** button. + +The compliance scan will create new or update existing `Compliance Case` rows for violations. + +#### What the scan evaluates: + +- All agent configurations against defined threshold rules +- Risk level assignment based on matched thresholds +- SLA calculation and countdown initiation +- Notification triggers for makers and admins + +### Monitor and manage compliance cases + +1. **Navigate to compliance cases** + - Open the Compliance Hub app from your Power Apps environment + - Navigate to **Compliance Cases** in the left navigation menu + - Filter cases by status (Open, SLA Breach, Approved, Expired) and risk level (High, Medium, Low) + +2. **Review case details** + + Open a compliance case record to view: + - **Triggered thresholds**: Which policies were violated + - **Maker's intake**: Business justification, owner confirmation, data classification + - **Automated scan results**: DLP policies, connectors used, knowledge sources + - **SLA countdown**: Time remaining before enforcement action + - **Event timeline**: Historical record of all case activities + + ![Compliance Case Details View](media/ch_5.png) + *Figure 5: Compliance case details showing triggered thresholds, maker intake, and SLA countdown* + +3. **Approve or reject cases** + - Use the **Approve/Reject** buttons in the command bar to make decisions + - Add audit notes for documentation and future reference + - If intake is incomplete or violations persist past SLA, enforcement actions are triggered automatically + +4. **Audit and reporting** + - Use dashboards and charts for trend analysis (e.g., distribution of important case statuses, risk level distribution). + - Export case data for external reporting and compliance documentation. + - Maintain thorough documentation for all compliance cases. + - Enable auditing in the Power Platform Admin Center (PPAC) environment settings to ensure case level auditing is available. + - Timeline activities are also posted on cases for historical record + +## Agent maker responsibilities + +When a compliance case is opened, you'll receive a Microsoft Teams notification linked to your case. + +### Step-by-step guide for makers + +1. **Complete intake form** + + The Microsoft Teams notification is an adaptive card that accepts three input fields: + + - **Business justification**: Explain the purpose and business value of this agent + - **Data classification**: Specify what type of data the agent handles (Public, Internal, Confidential, Highly Confidential) + - **Expected number of users**: Estimate how many people will use this agent + + Submit the form directly from Teams to update your case. + +2. **Remediate compliance issues** + + Fix any compliance violations identified in the case that are medium or high risk: + + - Review the specific thresholds that were violated + - Make necessary changes in Copilot Studio (e.g., enable authentication, restrict sharing, add DLP policies) + - Publish your changes + - Re-evaluation happens daily (typically overnight), and you will be notified of the results + +3. **Monitor SLA countdown** + + - Track the SLA countdown via notifications that are sent daily + - You'll receive reminders as the SLA deadline approaches (3 days before, 1 day before, day of) + - Failure to comply within the SLA may result in agent quarantine or deletion + - Contact your compliance admin if you need an extension + +### Example scenarios + +**Scenario 1: No authentication configured** +- Violation: Agent has no authentication enabled (High risk) +- Remediation: Enable Microsoft Entra ID authentication in Copilot Studio security settings +- Timeline: 3-day SLA for high-risk violations + +**Scenario 2: Published to production without review** +- Violation: Agent published to production environment without admin review (low risk) +- Remediation: Provide detailed business justification and general intake. +- Timeline: Admin review required within 5 days + +## Reference: Data model and policies + +### Key Tables + +- **Compliance Case**: Tracks workflow, SLA, and enforcement. +- **Threshold Config**: Stores threshold settings. +- **Action Policy**: Defines enforcement actions per risk level. +- **Email Template**: Manages notification templates. + +> **Note**: +> If a threshold is defined for all possibilities in a certain category (for example, for all auth modes), then a case will be created for every agent. + +## FAQs and troubleshooting + +### General questions + +**Q: What happens if I don't complete intake within the SLA?** + +A: The system will automatically apply the configured enforcement action (e.g., quarantine or delete the agent). You'll receive daily reminder notifications as the SLA deadline approaches. + +**Q: How do I know which compliance issues to fix as a maker?** + +A: Review the checklist and findings in your compliance case. Automated checks highlight any "Fail" items that need to be addressed. Focus on high and medium risk violations first. + +**Q: Who do I contact for help as a maker?** + +A: Use the contact information provided in your notification email or reach out to your CoE Admin team. The compliance support contact alias is configured in the environment variables. + +**Q: Can I request an SLA extension as a maker?** + +A: Contact your compliance administrator to discuss SLA extensions. Extensions must be documented in the case audit notes. + +**Q: What happens to a quarantined agent?** + +A: A quarantined agent is disabled and cannot be accessed by end users. However, you as the owner can still edit the agent to fix compliance issues. Once compliant, request admin review to restore the agent. + +### Troubleshooting + +**Issue: Compliance scan is not detecting my agent** + +- Ensure Agent Inventory scan has completed successfully +- Check that your agent is published and visible in Copilot Studio +- Verify the agent is in an environment being monitored by the solution +- Review cloud flow run history for errors + +**Issue: Case status is not updating after remediation** + +- Compliance re-evaluation occurs daily (typically runs overnight) +- Manually trigger a compliance scan from the Compliance Hub dashboard +- Verify remediation changes are saved and published in Copilot Studio Kit Inventory +- Check the case timeline for evaluation history + +**Issue: Filter operators not working as expected** + +- Available filter operators: Equals +- Ensure the "Value" field matches the exact case and formatting of the column being evaluated +- Test your threshold configuration on a small sample size agent configuration before rolling out broadly + +### Best practices + +- **Start with Manual review enforcement**: Configure low-risk enforcement initially while testing thresholds. +- **Test with non-production agents first**: Validate your compliance policies don't have unintended consequences +- **Document your policies**: Maintain clear documentation of why each threshold exists and how to remediate +- **Regular policy reviews**: Schedule quarterly reviews of compliance thresholds and enforcement actions +- **Monitor SLA metrics**: Track average time to remediation and adjust SLAs if consistently missed + +### Additional resources + +- [Agent Inventory Documentation](./AGENT_INVENTORY.md) +- [Installation Instructions](./INSTALLATION_INSTRUCTIONS.md) +- [Prerequisites](./PREREQUISITES.md) +- [Troubleshooting Guide](./TROUBLESHOOT.md) + +## Technical Details + +This section provides a consolidated technical reference of all solution components, dependencies, and configuration artifacts for the Compliance Hub. + +**Canvas App:** Compliance Hub (`cat_compliancehub`) + +### Connection References + +All connection references use the `Copilot Studio Kit` prefix and are shared across the kit's solutions. + +| Connection Reference | Connector | Required For | +|---|---|---| +| Copilot Studio Kit - Dataverse | Microsoft Dataverse | Core data operations across all flows | +| Copilot Studio Kit - Standard approvals (`shared_approvals`) | Standard Approvals | Case approval workflows | +| Copilot Studio Kit - Outlook (`shared_office365`) | Office 365 Outlook | Email notifications to makers and admins | +| Copilot Studio Kit - Microsoft Teams (`shared_teams`) | Microsoft Teams | Teams adaptive card notifications | +| Copilot Studio Kit - Office 365 Users (`shared_office365users`) | Office 365 Users | User profile lookup for case assignment | +| Copilot Studio Kit - Office 365 Groups (`shared_office365groups`) | Office 365 Groups | Group membership checks for admin authorization | +| Copilot Studio Kit - Power Platform for Admins V2 (`shared_powerplatformadminv2`) | Power Platform for Admins V2 | Quarantine and delete enforcement actions | + +### Environment Variables + +| Display Name | Schema Name | Category | Default | +|---|---|---|---| +| App ID | `cat_AppID` | Critical | — | +| Compliance Admin Group ID | `cat_GovernanceAdminAlias` | Critical | — | +| Instance Url | `cat_InstanceUrl` | Critical | — | +| Maker Team ID | `cat_AgentMakerTeamID` | Critical | — | +| Compliance Documentation Link | `cat_ComplianceDocumentationLink` | Recommended | — | +| Compliance Support Contact Alias | `cat_ComplianceSupportContactAlias` | Recommended | — | +| Admin Approval Before Maker Notification | `cat_AdminApprovalBeforeMakerNotification` | Default | Yes | +| Case Intake SLA | `cat_CaseIntakeSLA` | Default | 5 days | +| Case Review SLA | `cat_CaseReviewSLA` | Default | 5 days | +| Case Summary Email Frequency | `cat_CaseSummaryEmailFrequency` | Default | WEEKLY | +| Require Case For No Risk | `cat_RequireCaseForNoRisk` | Default | No | +| Send Case Alerts via Email | `cat_SendCaseAlertsToMakerViaEmail` | Default | Yes | +| Send Case Alerts via Teams | `cat_SendCaseAlertsToMakerViaTeams` | Default | Yes | + +### Cloud Flows + +The Compliance Hub includes 16 cloud flows. Enable them in dependency order: Grandchild → Child → Parent. + +| Flow Name | Type | Purpose | +|---|---|---| +| AgentCompliance \| Compliance Scan | Parent | Scans all agents against threshold rules; creates or updates compliance cases | +| AgentCompliance \| Initialize Data | Parent | Seeds default threshold configs and action policies on first run | +| AgentCompliance \| Request Intake | Parent | Sends intake adaptive card to maker via Teams when a case is created | +| AgentCompliance \| Start Admin Approval | Parent | Initiates admin approval workflow for compliance cases | +| AgentCompliance \| Execute Policy Action | Parent | Applies enforcement action (quarantine/delete) when SLA expires | +| AgentCompliance \| Quarantine | Parent | Quarantines (disables) a non-compliant agent via Admin connector | +| AgentCompliance \| Unquarantine | Parent | Restores a previously quarantined agent | +| AgentCompliance \| Get Quarantine Status | Parent | Checks current quarantine status of an agent | +| AgentCompliance \| Delete Case | Parent | Permanently deletes an agent as an enforcement action | +| AgentCompliance \| Create Case (Child) | Child | Creates a new compliance case record in Dataverse | +| AgentCompliance \| Update Case (Child) | Child | Updates fields on an existing compliance case | +| AgentCompliance \| Approve Case (Child) | Child | Processes case approval and closes the case | +| AgentCompliance \| Reject Case (Child) | Child | Processes case rejection | +| AgentCompliance \| Send Notification (Child) | Child | Sends notifications via email and/or Teams | +| AgentCompliance \| Check Compliance (Grandchild) | Grandchild | Evaluates a single agent against all active threshold rules | +| Agent Inventory \| Compliance Scan | Shared | Shared with Agent Inventory — triggers compliance scan after inventory sync | + +### Dataverse Tables + +| Display Name | Schema Name | Purpose | +|---|---|---| +| Agent Compliance | `cat_AgentCompliance` | Stores per-agent compliance status and risk assessment results | +| Compliance Case | `cat_ComplianceCase` | Tracks case lifecycle, SLA countdown, intake, and enforcement status | +| Action Policy | `cat_ActionPolicy` | Defines enforcement action and SLA duration per risk level | +| Threshold Config | `cat_ThresholdConfig` | Stores compliance threshold definitions (filter column, operator, value, risk) | + +### DLP Configuration + +The following connectors must be in the same DLP policy group (typically the **Business** data group) for the Compliance Hub cloud flows to execute successfully: + +| Connector | Used By | +|---|---| +| Microsoft Dataverse | All flows — core data operations | +| Standard Approvals | Admin approval workflows | +| Office 365 Outlook | Email notifications | +| Microsoft Teams | Teams adaptive card notifications | +| Office 365 Users | User profile lookup | +| Office 365 Groups | Admin group membership verification | +| Power Platform for Admins V2 | Quarantine and delete enforcement actions | + +> **Note**: Refer to [Prerequisites](./PREREQUISITES.md) for full DLP configuration guidance across all Copilot Studio Kit components. + +--- + +[Back to the landing page](./README.md) diff --git a/CONVERSATION_ANALYZER.md b/CONVERSATION_ANALYZER.md index b8524a01..bf2e83c3 100644 --- a/CONVERSATION_ANALYZER.md +++ b/CONVERSATION_ANALYZER.md @@ -1,49 +1,52 @@ -# Conversation Analyzer (Preview) - -Conversation analyzer allows makers to analyze the conversation transcripts of their Copilot Studio custom agents using custom prompts. The feature comes with two pre-canned prompts, sentiment analysis and PII analysis. Custom prompts can be created, used, saved and reused later. Using custom prompts to analyze the conversations can provide insights not available through traditional analytics. - -## Conversation Analyzer overview - -After navigating to the conversation analyzer feature, a list of agents is shown. Conversation Analyzer uses the agent data in the Agent Inventory, so it must be populated first before using conversation analyzer. - -The list of agents has "Action" column that has either "Analyze" or "View Results" hyperlink in it. "View Results" indicates that the recent transcripts have already been analyzed and can be viewed. - -"Analyze" indicates that no analysis exists and recent transcripts for the agent can be selected for analysis. - -overview - -## Creating new analysis - -After selecting an agent and clicking "Analyze" from the action, prompt selection screen is shown. The feature ships with two precanned prompts, "Sentiment Analysis" and "PII Analysis". - -![Copilot Studio Kit - New Analysis](https://github.com/user-attachments/assets/50aed206-b3d5-4a30-a464-4a6696d437e4) - - -The primary purpose of the feature is to allow makers to use custom prompts to analyze the conversations with. To create a new custom prompt or updating an existing custom prompt, go to the **Manage Prompts** page, enter a descriptive name in the Prompt Name field, and add your prompt text in the **Prompt** textbox. - -![Copilot Studio Kit - Manage Prompts](https://github.com/user-attachments/assets/86707fda-3291-4512-9d8e-5594667f3536) - -![Copilot Studio Kit - Manage Prompts - Edit](https://github.com/user-attachments/assets/27718f22-4ae1-4498-93d8-66c75cfeb0fe) - - -Once the custom prompt is saved, it becomes available for future use in the prompt selection dropdown list. - -When the analysis is started, it will take a moment to process. After the processing is finished, results will be presented. - -## Viewing existing analysis - -If "View Results" is selected from the agent selections screen, analysis results are shown. Same view is presented after successful (new) analysis has completed. - -analysis_ready - -Conversation Id, summary of the conversation, duration of the conversation, prompt used and analysis creation date are shown. Depending on the prompt, the summary may already provide valuable insights. Conversation and analysis details can be shown by pressing "View Results" from the "Action" column. - -## Conversation analysis details - -Conversation analysis details view varies slightly depending on the prompt used. The conversation can be seen on the right side of the screen as it happened, and analysis details are shown in the main view. - -results_details - -This feature is experimental and the team is looking forward to hearing feedback from users. - -Back to the [landing page](./README.md#power-cat-copilot-studio-kit) +# Conversation Analyzer (Preview) + +Conversation analyzer allows makers to analyze the conversation transcripts of their Copilot Studio custom agents using custom prompts. The feature comes with two pre-canned prompts, sentiment analysis and PII analysis. Custom prompts can be created, used, saved and reused later. Using custom prompts to analyze the conversations can provide insights not available through traditional analytics. + +## Conversation Analyzer overview +After navigating to the conversation analyzer feature, a list of agents is shown. Conversation Analyzer uses the agent data in the Agent Inventory, so it must be populated first before using conversation analyzer. + +## Technical Details + +### Canvas App + +| Display Name | Logical Name | +|---|---| +| Conversation Analyser | `cat_conversationanalyser` | + +### Cloud Flows + +| Flow Name | Purpose | +|---|---| +| Conversation Analyzer \| Analyze Conversation Based on Prompt | Runs the selected prompt against a conversation transcript using AI Builder | +| Conversation Analyzer \| Get the Conversation Transcript | Retrieves the conversation transcript from Copilot Studio for analysis | + +### Connection References + +| Connection Reference | Required | +|---|---| +| Copilot Studio Kit - Dataverse | Yes | + +### Environment Variables + +None specific to this feature. + +### Dataverse Tables + +| Display Name | Logical Name | Notes | +|---|---|---| +| Conversation Analyzer | `cat_ConversationAnalyzer` | Stores analysis results | +| Conversation Analyzer Prompt | `cat_ConversationAnalyzerPrompt` | Stores built-in and custom prompts | +| Agent Details | `cat_AgentDetails` | **Dependency** — requires Agent Inventory to be populated before use | + +### Prerequisites + +- **Agent Inventory** must be populated first. Conversation Analyzer reads agent data from the Agent Inventory tables; analysis cannot run without it. +- **AI Builder credits** are required for prompt-based analysis. + +### DLP Connectors + +| Connector | +|---| +| Microsoft Dataverse | + +Back to the [landing page](./README.md#power-cat-copilot-studio-kit) diff --git a/CONVERSATION_KPIS.md b/CONVERSATION_KPIS.md index 41b6270f..e4417de1 100644 --- a/CONVERSATION_KPIS.md +++ b/CONVERSATION_KPIS.md @@ -1,22 +1,62 @@ -## Conversation KPIs - -Conversation KPIs are designed to help makers track and analyze the performance of their custom agents. This feature complements the existing analytics built-in the Copilot Studio and simplifies the process of understanding conversation outcomes by providing aggregated data in Dataverse rather than requiring you to analyze the complex conversation transcripts. - -* Aggregated Data: The feature surfaces simplified conversation outcome results, making it easier to understand the performance of your custom agents. This includes metrics such as the number of sessions, turns, and the global outcome of conversations (e.g., resolved, partially resolved, escalated, or abandoned). - -* Sample Power BI Reports: The Copilot Studio Kit offers a sample Power BI reports based on aggregated KPIs. These reports provide a visual representation of the data, helping you quickly identify trends and areas for improvement, and provide a solid starting point for your own custom reports. - -* Tracked Variables: You can define specific variables to track as part of the aggregated conversation KPIs, such as custom Net Promoter Score (NPS) or other relevant metrics. This allows you to tailor the KPIs to your specific needs. - -* Optional Full Transcript Storage: Users have the option to include the full conversation transcript with the KPI records (as file). This allows for a more detailed analysis if needed, using the transcript visualizer built into the Copilot Studio Kit. - -* Long-term tracking: Conversation KPIs can be stored in the system for as long as required which allows tracking the impact of improvements and performance of custom copilots over long time-period. - -Conversation KPIs are generated automatically twice per day and on-demand generation is supported as well. To generate KPIs on-demand, navigate to the agent configuration, press "**Generate KPIs**" and select a date range. Up to 75,000 transcripts are processed per automatic run and up to 50,000 per on-demand run. Please see [agent configuration](./CONFIGURE_COPILOTS.md) for more details on how to configure the Conversation KPIs. - -![kpi report overview](https://github.com/user-attachments/assets/bca1bc9e-2d6f-42bc-a6b6-798003999f21) - -![kpi details1](https://github.com/user-attachments/assets/96b48373-a7a0-4062-adb8-68bd97d22e12) - -Read more on how to configure the agent for [KPI generation](./CONFIGURE_COPILOTS.md#configure-a-new-agent-for-conversation-kpis) -Back to the [landing page](./README.md#power-cat-copilot-studio-kit) +## Conversation KPIs + +Conversation KPIs are designed to help makers track and analyze the performance of their custom agents. This feature complements the existing analytics built-in the Copilot Studio and simplifies the process of understanding conversation outcomes by providing aggregated data in Dataverse rather than requiring you to analyze the complex conversation transcripts. + +* Aggregated Data: The feature surfaces simplified conversation outcome results, making it easier to understand the performance of your custom agents. This includes metrics such as the number of sessions, turns, and the global outcome of conversations (e.g., resolved, partially resolved, escalated, or abandoned). + +* Sample Power BI Reports: The Copilot Studio Kit offers a sample Power BI reports based on aggregated KPIs. These reports provide a visual representation of the data, helping you quickly identify trends and areas for improvement, and provide a solid starting point for your own custom reports. + +* Tracked Variables: You can define specific variables to track as part of the aggregated conversation KPIs, such as custom Net Promoter Score (NPS) or other relevant metrics. This allows you to tailor the KPIs to your specific needs. + +* Optional Full Transcript Storage: Users have the option to include the full conversation transcript with the KPI records (as file). This allows for a more detailed analysis if needed, using the transcript visualizer built into the Copilot Studio Kit. + +* Long-term tracking: Conversation KPIs can be stored in the system for as long as required which allows tracking the impact of improvements and performance of custom copilots over long time-period. + +Conversation KPIs are generated automatically twice per day and on-demand generation is supported as well. To generate KPIs on-demand, navigate to the agent configuration, press "**Generate KPIs**" and select a date range. Up to 75,000 transcripts are processed per automatic run and up to 50,000 per on-demand run. Please see [agent configuration](./CONFIGURE_COPILOTS.md) for more details on how to configure the Conversation KPIs. + +Read more on how to configure the agent for [KPI generation](./CONFIGURE_COPILOTS.md#configure-a-new-agent-for-conversation-kpis) + +## Technical Details + +### Canvas App + +| App | Logical Name | +|---|---| +| Conversation KPI | `cat_conversationkpi` | + +### Cloud Flows + +| Flow | Type | +|---|---| +| Conversation KPI \| Copy Agent Transcripts Scheduler | Scheduled (runs twice daily) | +| Conversation KPI \| Copy Transcripts (Child) | Child flow | +| Conversation KPI \| Copy Transcripts (Grandchild) | Child flow | +| Conversation KPI \| Generate Conversation KPIs Scheduler | Scheduled | +| Conversation KPI \| Get Conversation Transcripts Count | Utility | +| Conversation KPI \| On Demand | Manual trigger (from agent configuration) | + +### Dataverse Tables + +| Table | Logical Name | Description | +|---|---|---| +| Conversation KPI | `cat_CopilotKPI` | Stores aggregated KPI data | +| Agent Transcripts | `cat_AgentTranscripts` | Stores conversation transcript data | +| Agent Configuration | `cat_CopilotConfiguration` | Shared configuration table | + +### Connection References + +| Connection Reference | Required | +|---|---| +| Copilot Studio Kit - Dataverse | Yes | + +### Environment Variables + +| Variable | Logical Name | Description | +|---|---|---| +| Conversation KPIs Report | `cat_ConversationKPIReport` | JSON config with Power BI workspace/report details. Required for the embedded dashboard. | + +### DLP Connectors + +- Microsoft Dataverse + +Back to the [landing page](./README.md#power-cat-copilot-studio-kit) diff --git a/FILE_SYNCHRONIZATION.md b/FILE_SYNCHRONIZATION.md index d65e5666..790e196c 100644 --- a/FILE_SYNCHRONIZATION.md +++ b/FILE_SYNCHRONIZATION.md @@ -1,21 +1,61 @@ -## SharePoint synchronization - -SharePoint synchronization allows makers to configure periodical selective content synchronization from SharePoint locations to custom agent knowledge base as files. - -Benefits of using SharePoint synchronization (compared to using SharePoint site as knowledge source): -- Support for additional file types -- Support for larger files (up to 512MB) -- Lower latency in responses -- Indexing of non-text elements in PDFs - -![file sync](https://github.com/user-attachments/assets/90bc990a-646c-4311-863f-6c5cba43a48d) - -Please see [agent configuration](./CONFIGURE_COPILOTS.md) for more details and how to configure SharePoint synchronization. SharePoint synchronization is automatically performed daily and it can be run on-demand by selecting "**Sync Files**" from the agent configuration. - -You can do a dry run validation of your configuration by pressing "Validate Connection". - -You can read more on the approach from [this article in Linkedin](https://www.linkedin.com/pulse/sync-your-sharepoint-library-dataverse-how-hybrid-approach-r%C3%A9mi-dyon-8lyle) - -Read more on how to configure agent for [SharePoint synchronization](./CONFIGURE_COPILOTS.md#configure-a-new-agent-for-file-synchronization) - -Back to the [landing page](./README.md#power-cat-copilot-studio-kit) +## SharePoint synchronization + +SharePoint synchronization allows makers to configure periodical selective content synchronization from SharePoint locations to custom agent knowledge base as files. + +Benefits of using SharePoint synchronization (compared to using SharePoint site as knowledge source): +- Support for additional file types +- Support for larger files (up to 512MB) +- Lower latency in responses +- Indexing of non-text elements in PDFs + +Please see [agent configuration](./CONFIGURE_COPILOTS.md) for more details and how to configure SharePoint synchronization. SharePoint synchronization is automatically performed daily and it can be run on-demand by selecting "**Sync Files**" from the agent configuration. + +You can do a dry run validation of your configuration by pressing "Validate Connection". + +Read more on how to configure agent for [SharePoint synchronization](./CONFIGURE_COPILOTS.md#configure-a-new-agent-for-file-synchronization) + +## Technical Details + +### Canvas App + +| App | Description | +|---|---| +| Validate SharePoint Connection (`cat_validatesharepointconnection`) | Validates the SharePoint connection and site configuration before running a full synchronization | + +### Cloud Flows + +| Flow | Trigger | Description | +|---|---|---| +| SharePoint Synchronization \| On Demand | Manual | Triggers an on-demand file synchronization for a selected agent | +| SharePoint Synchronization \| Scheduler | Scheduled (daily) | Runs file synchronization automatically on a daily schedule | +| SharePoint Synchronization \| Synchronize files to Agent (Child) | Child flow | Performs the actual file synchronization logic for a given agent configuration | +| SharePoint Synchronization \| Validate Connection On Demand | Manual | Validates the SharePoint connection and configuration without synchronizing files | + +### Connection References + +| Connection Reference | Connector | Required | +|---|---|---| +| Copilot Studio Kit - Dataverse | Microsoft Dataverse | Yes | +| Copilot Studio Kit - SharePoint | SharePoint (`shared_sharepointonline`) | Yes | + +### Environment Variables + +| Display Name | Schema Name | Description | +|---|---|---| +| Instance Url | `cat_InstanceUrl` | Dataverse environment URL, required for file sync API calls | + +### Dataverse Tables + +| Display Name | Schema Name | Notes | +|---|---|---| +| File Indexer Configuration | `cat_CopilotFileIndexerConfiguration` | Stores SharePoint synchronization configuration per agent | +| Agent Configuration | `cat_CopilotConfiguration` | Shared table used across kit features | + +### DLP Considerations + +The following connectors must be in the same DLP group for file synchronization to function: + +- **Microsoft Dataverse** +- **SharePoint** + +Back to the [landing page](./README.md#power-cat-copilot-studio-kit) diff --git a/PROMPT_ADVISOR.md b/PROMPT_ADVISOR.md index fa919539..2a4503ec 100644 --- a/PROMPT_ADVISOR.md +++ b/PROMPT_ADVISOR.md @@ -1,87 +1,127 @@ -# Prompt Advisor - -## Refine your AI Builder custom prompts - -> ### Important -> - Features of Prompt Advisor have been partially incorporated in-product -> - Please see [here](https://learn.microsoft.com/en-us/ai-builder/draft-with-copilot) how to use the features in-product. - -Develop effective prompts while learning useful prompt engineering skills with the **Prompt advisor tool**. -Users can enter a prompt and receive a confidence evaluation with detailed reasoning for the score. Additionally, the tool provides a list of suggested refined prompts implementing various prompt techniques. Users can select from these optimized prompts to iteratively refine and improve their input. - -![image (1)](https://github.com/user-attachments/assets/5c0efa74-f97f-402a-8e1c-cb8242a810e2) - -### Prerequisites - -1. A **Microsoft Dataverse Environment** with [AI Builder prompts enabled in the admin center](https://learn.microsoft.com/ai-builder/administer#enable-or-disable-ai-builder-preview-features) and Copilot Studio Kit -1. **Permission requirements**: System customizer security role. -1. **AI Builder credits**: Ensure [AI Builder credits](https://learn.microsoft.com/ai-builder/credit-management) are assigned to your environment. -2. **Enable the Power Apps Component Framework Feature** : [Power Apps Component Framework Feature](https://learn.microsoft.com/power-apps/developer/component-framework/component-framework-for-canvas-apps#enable-the-power-apps-component-framework-feature) - ---- - -## Fine tune your prompt - -Using the **prompt advisor** involves an iterative process where you enter a prompt, analyze it to receive a -confidence score and feedback, and then refine the prompt based on the provided suggestions. This -cycle is repeated until the prompt achieves a high confidence score. - -1. **Enter a prompt**: Type the prompt you want to analyze. -![4 - Copy](https://github.com/user-attachments/assets/d0fb1e2f-4923-41ca-8967-7d7108e1b2d6) -1. **Click Analyze**: Evaluate your prompt. -1. **View confidence score**: Observe the score and feedback on the evaluation pane. -![5 - Copy](https://github.com/user-attachments/assets/ef14f920-d1ab-4f0f-a73c-90ba08202b19) -1. **Review refined prompts**: - - Click the expand button to view suggestions in full screen. -![6 - Copy](https://github.com/user-attachments/assets/edeb0fd6-8e9e-4b79-bc85-4e428fbb2fcc) - - Filter prompts using the technique dropdown menu. -![7 - Copy](https://github.com/user-attachments/assets/909ea9a7-e41a-42f1-b7a1-bb697ef3a7da) -1. **Select a refined prompt**: Choose a suggested prompt to improve your input. -![8 - Copy](https://github.com/user-attachments/assets/6d219336-5686-4658-9eff-86a053f78fb4) -1. **Click Apply and refine**: Iterate based on feedback to improve the confidence score. -![9 - Copy](https://github.com/user-attachments/assets/05939dfe-0719-44b3-877c-327cfaa83bda) -Repeat steps 1-6 until you've reached an acceptable confidence score, or the prompt satisfies the -needs for your use case. As you learn more about prompt engineering, you may observe it as an -iterative process. -1. **Copy the prompt**: Paste the refined prompt into the [**AI Builder custom prompt studio**](https://learn.microsoft.com/ai-builder/create-a-custom-prompt). - ---- - -## Interpreting the confidence score - -The **confidence score** is a numerical value that indicates the likelihood that your prompt will yield -accurate and relevant results. Here’s how to interpret it: - -- **High (80-100):** This indicates that your prompt is well-structured and likely to produce accurate -results. The reasoning provided will highlight the strengths of your prompt. -- **Medium (50-79):** This suggests that your prompt is good but may need some improvements. The -reasoning will point out areas that could be enhanced for better performance. -- **Low (0-49):** This indicates that your prompt may not be effective and requires significant changes. -The reasoning will provide feedback on what aspects need to be improved. - -> **Tip** -> If there are too few words entered, the evaluation might not be able to provide a response. - ---- - -## Troubleshooting and FAQ - -### What should I do if I don't see the Analyze button or get an AI Builder credits error? - -Ensure you have the necessary permissions and sufficient **AI Builder Credits** in your environment. - -### How can I get more AI Builder Credits? - -Purchase additional credits through your Microsoft account or contact your administrator. - -### Why is my confidence score low? - -The confidence score depends on clarity and relevance. Review the **evaluation pane feedback** to improve the prompt. - -### What do the different techniques mean? - -- Zero-shot prompting means giving the AI a task with just instructions and no examples, relying on its general knowledge to figure it out. -- Few-shot prompting means showing the AI a few examples of the task first so it can learn the pattern or style before giving the answer. -- Chain-of-thought prompting means asking the AI to explain its reasoning step by step, which helps it solve complex problems more accurately. - -Back to the [landing page](./README.md#power-cat-copilot-studio-kit) +# Prompt Advisor +![Deprecated](https://img.shields.io/badge/status-deprecated-red.svg) + +## Refine your AI Builder custom prompts (being deprecated) + +> ### Important +> - Features of Prompt Advisor have been incorporated in-product and Prompt Advisor in Copilot Studio Kit is on deprecation path and will be removed soon. +> - Please see [here](https://learn.microsoft.com/en-us/ai-builder/draft-with-copilot) how to use the features in-product. + +Develop effective prompts while learning useful prompt engineering skills with the **Prompt advisor tool**. +Users can enter a prompt and receive a confidence evaluation with detailed reasoning for the score. Additionally, the tool provides a list of suggested refined prompts implementing various prompt techniques. Users can select from these optimized prompts to iteratively refine and improve their input. + +> ### Important +> - This is a **preview feature**. +> - Preview features aren't meant for production use and might have restricted functionality. These features are available before an official release so that customers can get early access and provide feedback. + +![image (1)](https://github.com/user-attachments/assets/5c0efa74-f97f-402a-8e1c-cb8242a810e2) + +### Prerequisites + +1. A **Microsoft Dataverse Environment** with [AI Builder prompts enabled in the admin center](https://learn.microsoft.com/ai-builder/administer#enable-or-disable-ai-builder-preview-features) and Copilot Studio Kit +1. **Permission requirements**: System customizer security role. +1. **AI Builder credits**: Ensure [AI Builder credits](https://learn.microsoft.com/ai-builder/credit-management) are assigned to your environment. +2. **Enable the Power Apps Component Framework Feature** : [Power Apps Component Framework Feature](https://learn.microsoft.com/power-apps/developer/component-framework/component-framework-for-canvas-apps#enable-the-power-apps-component-framework-feature) + +--- + +## Fine tune your prompt + +Using the **prompt advisor** involves an iterative process where you enter a prompt, analyze it to receive a +confidence score and feedback, and then refine the prompt based on the provided suggestions. This +cycle is repeated until the prompt achieves a high confidence score. + +1. **Enter a prompt**: Type the prompt you want to analyze. +![4 - Copy](https://github.com/user-attachments/assets/d0fb1e2f-4923-41ca-8967-7d7108e1b2d6) +1. **Click Analyze**: Evaluate your prompt. +1. **View confidence score**: Observe the score and feedback on the evaluation pane. +![5 - Copy](https://github.com/user-attachments/assets/ef14f920-d1ab-4f0f-a73c-90ba08202b19) +1. **Review refined prompts**: + - Click the expand button to view suggestions in full screen. +![6 - Copy](https://github.com/user-attachments/assets/edeb0fd6-8e9e-4b79-bc85-4e428fbb2fcc) + - Filter prompts using the technique dropdown menu. +![7 - Copy](https://github.com/user-attachments/assets/909ea9a7-e41a-42f1-b7a1-bb697ef3a7da) +1. **Select a refined prompt**: Choose a suggested prompt to improve your input. +![8 - Copy](https://github.com/user-attachments/assets/6d219336-5686-4658-9eff-86a053f78fb4) +1. **Click Apply and refine**: Iterate based on feedback to improve the confidence score. +![9 - Copy](https://github.com/user-attachments/assets/05939dfe-0719-44b3-877c-327cfaa83bda) +Repeat steps 1-6 until you've reached an acceptable confidence score, or the prompt satisfies the +needs for your use case. As you learn more about prompt engineering, you may observe it as an +iterative process. +1. **Copy the prompt**: Paste the refined prompt into the [**AI Builder custom prompt studio**](https://learn.microsoft.com/ai-builder/create-a-custom-prompt). + +--- + +## Interpreting the confidence score + +The **confidence score** is a numerical value that indicates the likelihood that your prompt will yield +accurate and relevant results. Here's how to interpret it: + +- **High (80-100):** This indicates that your prompt is well-structured and likely to produce accurate +results. The reasoning provided will highlight the strengths of your prompt. +- **Medium (50-79):** This suggests that your prompt is good but may need some improvements. The +reasoning will point out areas that could be enhanced for better performance. +- **Low (0-49):** This indicates that your prompt may not be effective and requires significant changes. +The reasoning will provide feedback on what aspects need to be improved. + +> **Tip** +> If there are too few words entered, the evaluation might not be able to provide a response. + +--- + +## Troubleshooting and FAQ + +### What should I do if I don't see the Analyze button or get an AI Builder credits error? + +Ensure you have the necessary permissions and sufficient **AI Builder Credits** in your environment. + +### How can I get more AI Builder Credits? + +Purchase additional credits through your Microsoft account or contact your administrator. + +### Why is my confidence score low? + +The confidence score depends on clarity and relevance. Review the **evaluation pane feedback** to improve the prompt. + +### What do the different techniques mean? + +- Zero-shot prompting means giving the AI a task with just instructions and no examples, relying on its general knowledge to figure it out. +- Few-shot prompting means showing the AI a few examples of the task first so it can learn the pattern or style before giving the answer. +- Chain-of-thought prompting means asking the AI to explain its reasoning step by step, which helps it solve complex problems more accurately. + +## Technical Details + +### Connection References + +| Connection Reference | Connector | +|:---------------------|:----------| +| Copilot Studio Kit - Dataverse | Microsoft Dataverse | + +### Environment Variables + +None — this component has no specific environment variables. + +### Cloud Flows + +None — this component has no associated cloud flows. + +### Dataverse Tables + +None specific — this component uses **AI Builder prompts** for evaluation and refinement rather than custom Dataverse tables. + +### DLP Configuration + +This component requires the **Microsoft Dataverse** connector. Ensure your DLP policies allow this connector in the target environment. + +### Canvas App + +| Display Name | Schema Name | +|:-------------|:------------| +| Prompt Advisor Studio | cat_promptadvisorstudio | + +### Prerequisites + +- **AI Builder credits** must be assigned to the environment. +- **PCF component framework** must be enabled in the environment. See [Enable the Power Apps Component Framework Feature](https://learn.microsoft.com/power-apps/developer/component-framework/component-framework-for-canvas-apps#enable-the-power-apps-component-framework-feature). + +Back to the [landing page](./README.md#power-cat-copilot-studio-kit) diff --git a/SETUP_WIZARD.md b/SETUP_WIZARD.md index 857c4703..10481900 100644 --- a/SETUP_WIZARD.md +++ b/SETUP_WIZARD.md @@ -1,37 +1,50 @@ -# Setup wizard -## Overview - -Setup Wizard can be accessed from the home page of the Copilot Studio Kit. Setup Wizard provides a guided flow through most of the post deployment steps where pre-requisites are checked, connection references and environment variables can be set, and cloud flows can be turned on. You can move forward and backward between steps by clicking Back and Next buttons. - -![image](https://github.com/user-attachments/assets/db4095ff-9f35-4458-b2e5-b2d41e3ca7b9) - -## Pre-requisites - -On the pre-requisites screen the Setup Wizard ensures that the environment has the required dependencies installed and the correct settings. - -![image](https://github.com/user-attachments/assets/bcce912a-2ed5-40a6-9871-cb0a2374a652) - -## Connection References - -The connection references step lists all the connection references used by the Kit and allows the user to easily set and change the connection reference. - -![image](https://github.com/user-attachments/assets/608f61c5-25df-43a5-b995-a4ec93698189) - -## Environment Variables - -The environment variables step lists all the environment variables that are used in the Copilot Studio Kit, with descriptions how they are used, how they should be set and when. Setting the environment variables from this view is also possible. - -![image](https://github.com/user-attachments/assets/e0f6031b-17de-48af-843a-a773e7e66375) - -## Flow activation - -Flow activation view allows user to view the current status of Copilot Studio Kit flows and they can be easily turned on from this view as well. Some of the flows cannot be turned on before setting related connection references. Flows are named so that the feature they are used in can easily be seen as the prefix. Users only need to enable the flows for the features they are planning to use. - -![image](https://github.com/user-attachments/assets/fdbc6ab7-6512-4ab1-ae8f-965cfa2cb5ca) - -## What post deployment steps are not included in the Setup Wizard? - -* Publishing the Conversation KPIs sample dashboard and scheduling the data refresh -* Creating the required app registrations for test automation (authentication, app insights) - -Back to the [landing page](./README.md#power-cat-copilot-studio-kit) +# Setup wizard +## Overview + +Setup Wizard can be accessed from the home page of the Copilot Studio Kit. Setup Wizard provides a guided flow through most of the post deployment steps where pre-requisites are checked, connection references and environment variables can be set, and cloud flows can be turned on. You can move forward and backward between steps by clicking Back and Next buttons. + +![image](https://github.com/user-attachments/assets/db4095ff-9f35-4458-b2e5-b2d41e3ca7b9) + +## Pre-requisites + +On the pre-requisites screen the Setup Wizard ensures that the environment has the required dependencies installed and the correct settings. + +![image](https://github.com/user-attachments/assets/bcce912a-2ed5-40a6-9871-cb0a2374a652) + +## Connection References + +The connection references step lists all the connection references used by the Kit and allows the user to easily set and change the connection reference. + +![image](https://github.com/user-attachments/assets/608f61c5-25df-43a5-b995-a4ec93698189) + +## Environment Variables + +The environment variables step lists all the environment variables that are used in the Copilot Studio Kit, with descriptions how they are used, how they should be set and when. Setting the environment variables from this view is also possible. + +![image](https://github.com/user-attachments/assets/e0f6031b-17de-48af-843a-a773e7e66375) + +## Flow activation + +Flow activation view allows user to view the current status of Copilot Studio Kit flows and they can be easily turned on from this view as well. Some of the flows cannot be turned on before setting related connection references. Flows are named so that the feature they are used in can easily be seen as the prefix. Users only need to enable the flows for the features they are planning to use. + +![image](https://github.com/user-attachments/assets/fdbc6ab7-6512-4ab1-ae8f-965cfa2cb5ca) + +## What post deployment steps are not included in the Setup Wizard? + +* Publishing the Conversation KPIs sample dashboard and scheduling the data refresh +* Creating the required app registrations for test automation (authentication, app insights) + +## Technical Details + +| Category | Details | +|---|---| +| **Canvas App** | Setup Wizard (`cat_setupwizard`) | +| **Cloud Flows** | Setup Wizard \| Get Solution Connection References Details | +| | Setup Wizard \| Get Solution Environment Variable Definitions | +| | Setup Wizard \| Get Solution Flows Details | +| **Connection References** | Copilot Studio Kit - Dataverse | +| **Environment Variables** | None specific (reads/writes all environment variables defined in the Kit) | +| **DLP Connectors** | Microsoft Dataverse, Power Apps for Makers | +| **Dataverse Tables** | Solution Component (`cat_SolutionComponent`) | + +Back to the [landing page](./README.md#power-cat-copilot-studio-kit) diff --git a/TECHDETAILS.md b/TECHDETAILS.md index 8e52b1f3..1ca4fa95 100644 --- a/TECHDETAILS.md +++ b/TECHDETAILS.md @@ -1,42 +1,161 @@ -# Technical information about Copilot Studio Kit -## Environment variables - -| Name | Description | Notes | -| :-- | :-- | :-- | -| Agent Token Endpoint | Token endpoint for Copilot Studio custom agent which will be used for webchat preview in WebChat Playground and Adaptive Cards Gallery features | Optional. Emulator is used if not set. Please see [here](https://github.com/microsoft/Power-CAT-Copilot-Studio-Kit/blob/psimolin-may-release/ADAPTIVE_CARDS_GALLERY.md#setup-note) for setup information. | -| Conversation KPIs Report | Holds Conversation KPIs report and workspace details. | Required for embedded Conversation KPIs dashboard. [More details](https://github.com/microsoft/Power-CAT-Copilot-Studio-Kit/blob/main/INSTALLATION_INSTRUCTIONS.md#configure-the-embedded-conversation-kpi-dashboard) | -| Dataverse URL | URL of the Copilot Studio custom agent Dataverse instance | Required for SharePoint synchronization. | -| Delay for Azure Application Insights Enrichment (Minutes) | Delay between running the test case and trying to fetch additional information from App Insights | Default value 5 minutes | -| Delay for Conversation Transcripts Enrichment (Minutes) | Delay between running the test case and trying to fetch additional information from conversation transcript | Default value 60 minutes | - -## Connection references - -| Name | Description | Notes | -| :-- | :-- | :-- | -| Copilot Studio Kit - Dataverse | Microsoft Dataverse connection reference for Copilot Studio Kit | Required | -| Copilot Studio Kit - Power Platform for Admins | This connection reference is used to get a list of environments in Agent Inventory feature | Required for Agent Inventory | -| Copilot Studio Kit - SharePoint | This connection reference is used with SharePoint synchronization | Required for SharePoint synchronization | - -## Connectors used (for DLP configuration purposes) - -| Name | Description | Notes | -| :-- | :-- | :-- | -| Microsoft Dataverse | Provides access to Microsoft Dataverse actions and triggers for Power Platform environments. | Used everywhere in Kit | -| Sharepoint | | Used in SharePoint synchronization | -| Office 365 Outlook | | Used in Agent Review Tool | -| Power Platform for Admins | | Used in Agent Inventory | -| Microsoft Entra ID | Used to obtain a user access token for connecting to the agent during test automation when the agent is configured with manual authentication. | Used in Test Automation | -| [Legacy] Microsoft Dataverse | | | -| Direct Line channels in Copilot Studio | | Used in Test Automation | -| Power Apps for Makers | Used to get or create the required connections in the Power Platform environment. | Used in the Setup Wizard | - -## Agent Inventory data sources -Please refer to the [Agent Inventory -Data Source](AGENT_INVENTORY_DATA_SOURCE.md) file for more information. - - - - - - - - +# Technical Information about Copilot Studio Kit + +This document provides a comprehensive technical reference for the [Power CAT Copilot Studio Kit](https://github.com/microsoft/Power-CAT-Copilot-Studio-Kit), including environment variables, connection references, DLP connector requirements, and a feature-to-component matrix. Use this as a guide during installation, environment configuration, and DLP policy setup. + +--- + +## Environment Variables + +### General + +| Name | Schema Name | Type | Default | Description | Used By | +| :-- | :-- | :-- | :-- | :-- | :-- | +| Instance Url | `cat_InstanceUrl` | String | — | Dataverse environment URL | File Synchronization, Compliance Hub | +| Power Automate Region | `cat_PowerAutomateEndpoint` | String | `Commercial` | Power Automate environment region (e.g., Commercial, GCC, GCC High) | Global | +| App ID | `cat_AppID` | String | — | Application ID for the Copilot Studio Kit canvas app | Compliance Hub | +| Code App Shared Flag | `cat_CodeAppSharedFlag` | Yes/No | `No` | Flag indicating whether the Code App has been shared | Component Library | + +### Compliance Hub + +| Name | Schema Name | Type | Default | Description | Used By | +| :-- | :-- | :-- | :-- | :-- | :-- | +| Admin Approval Before Maker Notification | `cat_AdminApprovalBeforeMakerNotification` | Yes/No | `No` | If enabled, an approval is sent to the admin before the maker receives a notification | Compliance Hub | +| Maker Team ID | `cat_AgentMakerTeamID` | String | `00000000-0000-0000-0000-000000000000` | Azure AD group Object ID for agent makers | Compliance Hub | +| Case Intake SLA | `cat_CaseIntakeSLA` | Integer | `5` | Number of days for the maker to complete the intake form | Compliance Hub | +| Case Review SLA | `cat_CaseReviewSLA` | Integer | `5` | SLA in days for the admin to complete compliance case review | Compliance Hub | +| Case Summary Email Frequency | `cat_CaseSummaryEmailFrequency` | String | `WEEKLY` | How often admins receive compliance case summary emails (e.g., DAILY, WEEKLY) | Compliance Hub | +| Compliance Admin Group ID | `cat_GovernanceAdminAlias` | String | — | Azure AD Group Object ID for the compliance admin team | Compliance Hub | +| Compliance Documentation Link | `cat_ComplianceDocumentationLink` | String | `https://aka.ms/CopilotStudioKit` | Link to your organization's compliance documentation | Compliance Hub | +| Compliance Support Contact Alias | `cat_ComplianceSupportContactAlias` | String | — | Email alias for compliance support inquiries | Compliance Hub | +| Require Case For No Risk | `cat_RequireCaseForNoRisk` | Yes/No | `No` | If enabled, a compliance case is created even for agents classified as no risk | Compliance Hub | +| Send Case Alerts via Email | `cat_SendCaseAlertsToMakerViaEmail` | Yes/No | `Yes` | Send email notifications to makers for compliance case updates | Compliance Hub | +| Send Case Alerts via Teams | `cat_SendCaseAlertsToMakerViaTeams` | Yes/No | `Yes` | Send Teams notifications to makers for compliance case updates | Compliance Hub | + +### Test Automation + +| Name | Schema Name | Type | Default | Description | Used By | +| :-- | :-- | :-- | :-- | :-- | :-- | +| Delay for Azure App Insights Enrichment (Minutes) | `cat_DelayforAzureApplicationInsightsEnrichment` | Integer | `5` | Minutes to wait after running a test before fetching App Insights telemetry | Test Automation | +| Delay for Conversation Transcripts Enrichment (Minutes) | `cat_DelayforConversationTranscriptsEnrichment` | Integer | `35` | Minutes to wait after running a test before fetching conversation transcript data | Test Automation | + +### Adaptive Cards Gallery & WebChat Playground + +| Name | Schema Name | Type | Default | Description | Used By | +| :-- | :-- | :-- | :-- | :-- | :-- | +| Agent Token Endpoint | `cat_AgentTokenEndpoint` | String | — | Token endpoint for the Copilot Studio custom agent used for webchat preview. Optional — the built-in emulator is used if not set. See [setup instructions](https://github.com/microsoft/Power-CAT-Copilot-Studio-Kit/blob/main/ADAPTIVE_CARDS_GALLERY.md#setup-note). | Adaptive Cards Gallery, WebChat Playground | + +### Conversation KPIs + +| Name | Schema Name | Type | Default | Description | Used By | +| :-- | :-- | :-- | :-- | :-- | :-- | +| Conversation KPIs Report | `cat_ConversationKPIReport` | JSON | — | Power BI workspace and report details for the embedded KPI dashboard. See [configuration steps](https://github.com/microsoft/Power-CAT-Copilot-Studio-Kit/blob/main/INSTALLATION_INSTRUCTIONS.md#configure-the-embedded-conversation-kpi-dashboard). | Conversation KPIs | + +### Agent Value Summary + +| Name | Schema Name | Type | Default | Description | Used By | +| :-- | :-- | :-- | :-- | :-- | :-- | +| Enable Value Component | `cat_Enablevaluecomponent` | Yes/No | `No` | Enables the value classification component on the agent detail page | Agent Value Summary | + +--- + +## Connection References + +All connection references below are defined in the solution. Set these up during installation per the [Installation Instructions](https://github.com/microsoft/Power-CAT-Copilot-Studio-Kit/blob/main/INSTALLATION_INSTRUCTIONS.md). + +| # | Name | Connector | Required For | Notes | +| :-- | :-- | :-- | :-- | :-- | +| 1 | Copilot Studio Kit - Dataverse | `shared_commondataserviceforapps` | **All features** | Core connection — required for every feature in the Kit | +| 2 | Copilot Studio Kit - Power Platform for Admins | `shared_powerplatformforadmins` | Agent Inventory | Used to retrieve the list of Power Platform environments | +| 3 | Copilot Studio Kit - Outlook | `shared_office365` | Compliance Hub | Used for sending email notifications for compliance cases | +| 4 | Copilot Studio Kit - SharePoint | `shared_sharepointonline` | File Synchronization | Used for synchronizing files from SharePoint document libraries | +| 5 | Copilot Studio Kit - Standard approvals | `shared_approvals` | Compliance Hub | Used for compliance case approval workflows | +| 6 | Copilot Studio Kit - Office 365 Groups | `shared_office365groups` | Compliance Hub | Used for Azure AD / Entra group membership lookups | +| 7 | Copilot Studio Kit - Office 365 Users | `shared_office365users` | Compliance Hub | Used for user profile lookups in compliance workflows | +| 8 | Copilot Studio Kit - Power Platform for Admins V2 | `shared_powerplatformadminv2` | Agent Inventory, Compliance Hub | Used for advanced admin operations across environments | +| 9 | Copilot Studio Kit - Microsoft Teams | `shared_teams` | Compliance Hub | Used for sending Teams notifications for compliance case alerts | + +--- + +## Connectors Used (for DLP Configuration) + +The following connectors **must** be classified as **Business** (or allowed) in your DLP policies for the Copilot Studio Kit to function correctly. They must all be in the **same DLP group**. + +| # | Connector | API Name | Used By | Notes | +| :-- | :-- | :-- | :-- | :-- | +| 1 | Microsoft Dataverse | `shared_commondataserviceforapps` | All features | Core data connector used by every flow and app | +| 2 | SharePoint | `shared_sharepointonline` | File Synchronization | Reading/writing files from SharePoint libraries | +| 3 | Office 365 Outlook | `shared_office365` | Compliance Hub | Sending email notifications and case summary digests | +| 4 | Power Platform for Admins | `shared_powerplatformforadmins` | Agent Inventory | Listing environments and agent metadata | +| 5 | Power Platform for Admins V2 | `shared_powerplatformadminv2` | Agent Inventory, Compliance Hub | Advanced admin operations (capacity, usage) | +| 6 | Standard Approvals | `shared_approvals` | Compliance Hub | Approval workflows for compliance cases | +| 7 | Office 365 Groups | `shared_office365groups` | Compliance Hub | Checking group membership for admin/maker teams | +| 8 | Office 365 Users | `shared_office365users` | Compliance Hub | Looking up user profiles and emails | +| 9 | Microsoft Teams | `shared_teams` | Compliance Hub | Sending adaptive card notifications via Teams | +| 10 | Microsoft Entra ID | — | Test Automation | Obtaining user access tokens for agents with manual authentication | +| 11 | [Legacy] Microsoft Dataverse | — | Legacy support | Retained for backward compatibility | +| 12 | Direct Line channels in Copilot Studio | — | Test Automation | Communicating with agents during automated test runs | +| 13 | Power Apps for Makers | — | Setup Wizard | Creating and validating connections during initial setup | +| 14 | HTTP with Microsoft Entra ID (preauthorized) | — | Agent Inventory (Usage Metrics) | Used by the separate Usage Metrics solution for API calls | + +> **Tip:** If you are using the [Agent Inventory Usage Metrics](https://github.com/microsoft/Power-CAT-Copilot-Studio-Kit/blob/main/AGENT_INVENTORY_DATA_SOURCE.md) add-on solution, ensure the **HTTP with Microsoft Entra ID (preauthorized)** connector is also allowed. + +--- + +## Feature → Component Matrix + +The table below maps each Kit feature to its primary canvas apps, model-driven apps, cloud flows, and Dataverse tables. + +### Canvas & Model-Driven Apps + +| Feature | App Name | App Type | +| :-- | :-- | :-- | +| All Features | Copilot Studio Kit | Canvas App (main hub) | +| Setup | Setup Wizard | Canvas App | +| Compliance Hub | Compliance Hub (embedded views) | Model-Driven App components | +| Component Library | Copilot Studio Kit - Component Library | Component Library | + +### Cloud Flows (Key Flows by Feature) + +| Feature | Key Cloud Flows | Trigger | +| :-- | :-- | :-- | +| Test Automation | Run Test Set, Enrich Results from App Insights, Enrich Results from Conversation Transcripts | Manual / Scheduled | +| File Synchronization | Sync Files from SharePoint | Automated (SharePoint trigger) | +| Agent Inventory | Fetch Environments, Sync Agent Metadata | Scheduled | +| Compliance Hub | Create Compliance Case, Send Case Notification (Email), Send Case Notification (Teams), Process Approval, Send Case Summary Digest | Automated / Scheduled | +| Conversation KPIs | Fetch Conversation Transcripts | Scheduled | +| Adaptive Cards Gallery | (Operates within the canvas app — no dedicated flows) | — | + +### Core Dataverse Tables + +| Table Display Name | Schema Name | Used By | +| :-- | :-- | :-- | +| Agent | `cat_copilot` | Agent Inventory, Test Automation, Compliance Hub | +| Agent Test | `cat_copilottest` | Test Automation | +| Agent Test Run | `cat_copilottestrun` | Test Automation | +| Agent Test Result | `cat_copilottestresult` | Test Automation | +| Compliance Case | `cat_compliancecase` | Compliance Hub | +| Compliance Case Activity | `cat_compliancecaseactivity` | Compliance Hub | +| Conversation KPI | `cat_conversationkpi` | Conversation KPIs | +| Agent Configuration | `cat_copilotconfiguration` | Global configuration | +| Synchronized File | `cat_synchronizedfile` | File Synchronization | +| Environment | `cat_environment` | Agent Inventory | +| Adaptive Card Template | `cat_adaptivecardtemplate` | Adaptive Cards Gallery | + +> **Note:** This is not an exhaustive list of all tables. See the solution file for the complete entity manifest. + +--- + +## Agent Inventory Data Sources + +Please refer to the [Agent Inventory — Data Source](AGENT_INVENTORY_DATA_SOURCE.md) file for detailed information about data sources, including Microsoft Graph, Power Platform Admin connectors, and the optional Usage Metrics solution. + +--- + +## Additional Resources + +- [Installation Instructions](INSTALLATION_INSTRUCTIONS.md) +- [Adaptive Cards Gallery](ADAPTIVE_CARDS_GALLERY.md) +- [Compliance Hub](COMPLIANCE_HUB.md) +- [Test Automation](TEST_AUTOMATION.md) +- [Agent Inventory Data Source](AGENT_INVENTORY_DATA_SOURCE.md) +- [Back to main page](README.md) diff --git a/TESTING_CAPABILITIES.md b/TESTING_CAPABILITIES.md index 37bffd2c..a85ff5f7 100644 --- a/TESTING_CAPABILITIES.md +++ b/TESTING_CAPABILITIES.md @@ -1,22 +1,74 @@ -## Testing capabilities -The Power CAT Copilot Studio Kit is a user-friendly application that empowers makers to configure agents and test sets. It has native capabilities such as Excel export or import for bulk creation and updates. - -By running individual tests against the Copilot Studio APIs (Direct Line), the agent responses are evaluated against expected results. -To further enrich results, additional data points can be retrieved from Azure Application Insights and from Dataverse, by analyzing Conversation Transcript records (to get the exact triggered topic name, intent recognition scores, etc.). -For AI-generated answers, that are by nature non-deterministic, AI Builder prompts are used to compare the generated answer with a sample answer or with validation instructions. - -Today, the tool supports these types of tests: -- Response match -- Attachments match -- Topic match (requires Dataverse enrichment) -- Generative answers (requires AI Builder for response analysis, and Azure Application Insights for details on why an answer was or was not generated) -- Multi-turn test type is a special test type that consists of a set of test cases of *regular* types that are executed in specified order in same conversation context. Multi-turn tests can be used to test scenarios end-to-end, and for testing custom agents with generative orchestration. -- Plan validation allows maker to validate the tools included in the dynamic plan of Copilot Studio custom agent with generative orchestration - -![image](https://github.com/microsoft/Power-CAT-Copilot-Studio-Kit/assets/37898885/33496e94-0c7a-4e63-9291-9e461aa9b9e7) - -![image](https://github.com/microsoft/Power-CAT-Copilot-Studio-Kit/assets/37898885/25f071ff-6b4f-4f5f-b6a3-20193a2d1feb) - -Read more on how to configure agent for [test automation](./CONFIGURE_COPILOTS.md#configure-a-new-agent-for-test-automation) - -Back to the [landing page](./README.md#power-cat-copilot-studio-kit) +## Testing capabilities +The Power CAT Copilot Studio Kit is a user-friendly application that empowers makers to configure agents and test sets. It has native capabilities such as Excel export or import for bulk creation and updates. + +By running individual tests against the Copilot Studio APIs (Direct Line), the agent responses are evaluated against expected results. +To further enrich results, additional data points can be retrieved from Azure Application Insights and from Dataverse, by analyzing Conversation Transcript records (to get the exact triggered topic name, intent recognition scores, etc.). +For AI-generated answers, that are by nature non-deterministic, AI Builder prompts are used to compare the generated answer with a sample answer or with validation instructions. + +Today, the tool supports these types of tests: +- Response match +- Attachments match +- Topic match (requires Dataverse enrichment) +- Generative answers (requires AI Builder for response analysis, and Azure Application Insights for details on why an answer was or was not generated) +- Multi-turn test type is a special test type that consists of a set of test cases of *regular* types that are executed in specified order in same conversation context. Multi-turn tests can be used to test scenarios end-to-end, and for testing custom agents with generative orchestration. +- Plan validation allows maker to validate the tools included in the dynamic plan of Copilot Studio custom agent with generative orchestration + +Read more on how to configure agent for [test automation](./CONFIGURE_COPILOTS.md#configure-a-new-agent-for-test-automation) + +## Technical Details + +### Connection References + +| Connection Reference | Connector | Required | +|---|---|---| +| Copilot Studio Kit - Dataverse | Microsoft Dataverse | Yes — core data operations for all test features | + +### Environment Variables + +| Display Name | Schema Name | Default | Description | +|---|---|---|---| +| Delay for Azure Application Insights Enrichment (Minutes) | `cat_DelayforAzureApplicationInsightsEnrichment` | 5 | Wait time before querying Application Insights after a test run, allowing telemetry ingestion to complete | +| Delay for Conversation Transcripts Enrichment (Minutes) | `cat_DelayforConversationTranscriptsEnrichment` | 35 | Wait time before querying Dataverse Conversation Transcripts after a test run, allowing transcript records to be created | + +### Cloud Flows + +| Flow Name | Trigger Type | Purpose | +|---|---|---| +| Test Automation \| Run Agent Tests | Automated | Main orchestrator — executes test cases against agents via Direct Line and coordinates enrichment steps | +| Test Automation \| Run Multiturn Tests (Child) | Child flow | Executes multi-turn test sequences within a single conversation context | +| Test Automation \| Analyze Generated Answers with AI Builder | Child flow | Uses AI Builder prompts to compare AI-generated responses against expected answers or validation instructions | +| Test Automation \| Enrich with Azure Application Insights | Child flow | Retrieves additional telemetry data (generative answer details) from Azure Application Insights | +| Test Automation \| Enrich with Dataverse Conversation Transcripts | Child flow | Analyzes Conversation Transcript records to extract triggered topic names, intent recognition scores, etc. | +| Test Automation \| Multi-Topic Transcripts Enrichment (Child) | Child flow | Handles transcript enrichment for conversations that span multiple topics | +| Test Automation \| Update Agent Test Run Rollup Columns | Automated | Recalculates and updates aggregate result columns on test run records | +| Test Automation \| Assign Copilot Studio Kit Tester Role to Team | Automated | Assigns the required security role to a team for test execution permissions | +| Test Automation \| AI Builder Credit Consumption | Automated | Tracks AI Builder credit usage from AI-powered test analysis | +| Test Automation \| Validate Agent Plans (Child) | Child flow | Validates tool selections in the dynamic plan of custom agents with generative orchestration | +| Test Automation \| Rubrics AI Grading Flow | Automated | Performs AI-based grading of agent responses using defined rubrics | +| Test Automation \| Rubrics Refinement | Automated | Iterates on rubric definitions to improve grading accuracy | +| Pipeline \| Validate Agent Using Test Cases | Automated | Pipeline-triggered flow that runs test cases as part of a deployment validation gate | + +### Dataverse Tables + +| Display Name | Schema Name | Shared | Description | +|---|---|---|---| +| Agent Test | `cat_CopilotTest` | No | Individual test case definitions (utterance, expected response, test type) | +| Agent Test Result | `cat_CopilotTestResult` | No | Stores the outcome of each executed test case | +| Agent Test Run | `cat_CopilotTestRun` | No | Represents a single execution run across a test set | +| Agent Test Set | `cat_CopilotTestSet` | No | Groups related test cases together for batch execution | +| Agent Configuration | `cat_CopilotConfiguration` | Yes — shared with other kit features | Stores agent connection details (token endpoint, Application Insights keys, etc.) | +| Error Details | `cat_ErrorDetails` | No | Captures detailed error information from failed test executions | +| Rubric | `cat_Rubric` | Yes — shared with Rubric Refinement | Defines grading criteria used by AI-based response evaluation | +| Rubric Example | `cat_RubricExample` | Yes — shared with Rubric Refinement | Sample graded responses used as reference examples for rubric-based evaluation | + +### DLP Configuration + +The following connectors must be allowed to communicate within the same DLP policy group for the testing features to function: + +- **Microsoft Dataverse** — Required for all test data operations +- **Microsoft Entra ID** — Required only when testing agents configured with manual authentication (SSO scenarios) +- **Direct Line channels in Copilot Studio** — Required for communicating with agents during test execution + +> **Note:** If your environment DLP policies block any of these connectors, the corresponding cloud flows will fail at runtime. Ensure all three connectors are in the same business data group when testing agents that use SSO authentication. + +Back to the [landing page](./README.md#power-cat-copilot-studio-kit) diff --git a/WEBCHAT_PLAYGROUND.md b/WEBCHAT_PLAYGROUND.md index c98020df..fcbc6c03 100644 --- a/WEBCHAT_PLAYGROUND.md +++ b/WEBCHAT_PLAYGROUND.md @@ -1,70 +1,104 @@ -# Webchat Playground -## Overview -Webchat Playground simplifies customizing the appearance and behavior of the copilot agent webchat, including colors, fonts, thumbnails and much more. Easy to use UI allows makers to define the look and feel of their webchat, and styles can be exported as JSON snippet or HTML sample. - -![webchat overview](https://github.com/user-attachments/assets/6e070ba5-e47d-4f95-8b24-6ca3ff9f62cc) - -Webchat Playground comes with predefined themes which demonstrate different customization options of the webchat, and can be used a starting point when designing a new theme. Makers can also create new themes and save them for future use. - -![theme2](https://github.com/user-attachments/assets/6b5f07aa-2462-4121-b813-7abddf31c8dc) - -## UI explained - -![ui](https://github.com/user-attachments/assets/150f4314-5909-4ff8-b788-d064a9a5a82c) - -1. Left pane shows the available saved themes along with their primary secondary and accent colors. Theme can be selected by clicking on the color box, and edited by clicking the pencil icon. -2. Middle pane provides a preview of the selected theme or the theme being edited. -3. Right pane can be toggled between **JSON** and **Code Snippet**. **JSON** (pictured) provides the current styles in a JSON array which can be pasted to existing WebChat. **Code Snippet** has sample HTML of a webchat with the styles embedded. - -![ui2](https://github.com/user-attachments/assets/19c75edf-20f7-48b1-b43a-b4f26e91a3d4) - -1. The left pane turns into a style editor when a theme is being edited or a new theme is being created. It has sections for **General**, **Send Box**, **Suggestion Box**, **Avatar** and **Bubble**. The editor also includes an accessibility checker for all applicable sections. Save and delete actions are available from the top. -2. Right pane showing **Code Snippet** - -## Sample HTML - -The Sample HTML snippet features a floating widget webchat control with the selected styles applied. - -![webchat floating widget](https://github.com/user-attachments/assets/5c6d9a9d-3aa7-4abb-a1a7-1a9659a4afab) - -## Functionality - -### Theme preview - -* Theme can be previewed on the **WebChat Preview** by selecting it from the left pane (the area with the colors visible). - -### Creating a new theme - -1. New theme can be created by clicking **+ Add a theme** from the left pane. -1. Left pane will turn into the theme editor-view where the theme style can be modified. -1. After entering a theme name, it can be saved by clicking the **Save** icon. - -### Editing a theme - -1. Existing theme can be edited by clicking the small pencil icon next to its name on the left pane. -1. Left pane will change to the theme editor-view, where the styles of the theme can be edited. -1. Changes can be saved by clicking **Save**. - -### Deleting a theme - -1. Existing theme can be deleted by first opening it in edit mode, -1. Select the trashcan symbol from the top - -### Exporting the styles - -1. Select the desired theme from the theme selector -1. Select the **JSON** tab from the right pane -1. Select all the text in the view and copy-paste into your webchat html as **styleoptions** variable - -Read more from [here](https://learn.microsoft.com/microsoft-copilot-studio/customize-default-canvas?tabs=web#customize-the-agent-icon-background-color-and-name). - -### Exporting the HTML snippet - -1. Select the desired theme from the theme selector -1. Select the **Code Snippet** tab from the right pane -1. Select all the HTML markup in the view and save it as file with html-extension. -1. The resulting html file can be uploaded to the web server. - -Read more from [here](https://learn.microsoft.com/microsoft-copilot-studio/customize-default-canvas?tabs=web#customize-the-default-canvas-simple). - -Back to the [landing page](./README.md#power-cat-copilot-studio-kit) +# Webchat Playground +## Overview +Webchat Playground simplifies customizing the appearance and behavior of the copilot agent webchat, including colors, fonts, thumbnails and much more. Easy to use UI allows makers to define the look and feel of their webchat, and styles can be exported as JSON snippet or HTML sample. + +![webchat overview](https://github.com/user-attachments/assets/6e070ba5-e47d-4f95-8b24-6ca3ff9f62cc) + +Webchat Playground comes with predefined themes which demonstrate different customization options of the webchat, and can be used a starting point when designing a new theme. Makers can also create new themes and save them for future use. + +![theme2](https://github.com/user-attachments/assets/6b5f07aa-2462-4121-b813-7abddf31c8dc) + +## UI explained + +![ui](https://github.com/user-attachments/assets/150f4314-5909-4ff8-b788-d064a9a5a82c) + +1. Left pane shows the available saved themes along with their primary secondary and accent colors. Theme can be selected by clicking on the color box, and edited by clicking the pencil icon. +2. Middle pane provides a preview of the selected theme or the theme being edited. +3. Right pane can be toggled between **JSON** and **Code Snippet**. **JSON** (pictured) provides the current styles in a JSON array which can be pasted to existing WebChat. **Code Snippet** has sample HTML of a webchat with the styles embedded. + +![ui2](https://github.com/user-attachments/assets/19c75edf-20f7-48b1-b43a-b4f26e91a3d4) + +1. The left pane turns into a style editor when a theme is being edited or a new theme is being created. It has sections for **General**, **Send Box**, **Suggestion Box**, **Avatar** and **Bubble**. The editor also includes an accessibility checker for all applicable sections. Save and delete actions are available from the top. +2. Right pane showing **Code Snippet** + +## Sample HTML + +The Sample HTML snippet features a floating widget webchat control with the selected styles applied. + +![webchat floating widget](https://github.com/user-attachments/assets/5c6d9a9d-3aa7-4abb-a1a7-1a9659a4afab) + +## Functionality + +### Theme preview + +* Theme can be previewed on the **WebChat Preview** by selecting it from the left pane (the area with the colors visible). + +### Creating a new theme + +1. New theme can be created by clicking **+ Add a theme** from the left pane. +1. Left pane will turn into the theme editor-view where the theme style can be modified. +1. After entering a theme name, it can be saved by clicking the **Save** icon. + +### Editing a theme + +1. Existing theme can be edited by clicking the small pencil icon next to its name on the left pane. +1. Left pane will change to the theme editor-view, where the styles of the theme can be edited. +1. Changes can be saved by clicking **Save**. + +### Deleting a theme + +1. Existing theme can be deleted by first opening it in edit mode, +1. Select the trashcan symbol from the top + +### Exporting the styles + +1. Select the desired theme from the theme selector +1. Select the **JSON** tab from the right pane +1. Select all the text in the view and copy-paste into your webchat html as **styleoptions** variable + +Read more from [here](https://learn.microsoft.com/microsoft-copilot-studio/customize-default-canvas?tabs=web#customize-the-agent-icon-background-color-and-name). + +### Exporting the HTML snippet + +1. Select the desired theme from the theme selector +1. Select the **Code Snippet** tab from the right pane +1. Select all the HTML markup in the view and save it as file with html-extension. +1. The resulting html file can be uploaded to the web server. + +Read more from [here](https://learn.microsoft.com/microsoft-copilot-studio/customize-default-canvas?tabs=web#customize-the-default-canvas-simple). + +## Technical Details + +### Connection References + +| Connection Reference | Connector | +|:---------------------|:----------| +| Copilot Studio Kit - Dataverse | Microsoft Dataverse | + +### Environment Variables + +| Display Name | Schema Name | Required | Description | +|:-------------|:------------|:---------|:------------| +| Agent Token Endpoint | cat_AgentTokenEndpoint | Optional | Token endpoint for live preview with a real Copilot Studio agent | + +### Cloud Flows + +None — this component has no associated cloud flows. + +### Dataverse Tables + +| Display Name | Schema Name | Purpose | +|:-------------|:------------|:--------| +| Chatbot Style | cat_ChatbotStyle | Stores saved webchat theme definitions | + +### DLP Configuration + +This component requires the **Microsoft Dataverse** connector. Ensure your DLP policies allow this connector in the target environment. + +### Canvas App + +| Display Name | Schema Name | +|:-------------|:------------| +| WebChat Customizer | cat_webchatcustomizer | + +Back to the [landing page](./README.md#power-cat-copilot-studio-kit) From f0a5de7c5043f715c7ca6dceaeb3316c6a7698a1 Mon Sep 17 00:00:00 2001 From: Denise <43950360+denise-msft@users.noreply.github.com> Date: Mon, 20 Apr 2026 17:48:18 -0700 Subject: [PATCH 2/2] docs: restore missing images in 5 feature docs Fixed image references that were accidentally dropped during the doc overhaul: - TESTING_CAPABILITIES.md: restored 2 screenshots - CONVERSATION_KPIS.md: restored 2 dashboard screenshots - CONVERSATION_ANALYZER.md: rebuilt with all 6 images (overview, analysis, prompts, results) - FILE_SYNCHRONIZATION.md: restored file sync screenshot - AGENT_INVENTORY.md: restored usage metrics + import screenshots Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- AGENT_INVENTORY.md | 8 ++- CONVERSATION_ANALYZER.md | 147 +++++++++++++++++++++++++-------------- CONVERSATION_KPIS.md | 6 +- FILE_SYNCHRONIZATION.md | 4 +- TESTING_CAPABILITIES.md | 6 +- 5 files changed, 114 insertions(+), 57 deletions(-) diff --git a/AGENT_INVENTORY.md b/AGENT_INVENTORY.md index a851526d..6b3f7315 100644 --- a/AGENT_INVENTORY.md +++ b/AGENT_INVENTORY.md @@ -43,7 +43,9 @@ Before using the usage metrics feature: To enable usage metrics on top of the Copilot Studio Kit main solution, you must **import the `AgentInventoryUsage` solution**, available in the **September release Assets directory**. -During the import process, create a connection using the licensing host URL: https://licensing.powerplatform.microsoft.com/ +During the import process, create a connection using the licensing host URL: https://licensing.powerplatform.microsoft.com/ + +agent inventory usage ### How Usage Metrics Are Updated @@ -55,7 +57,9 @@ Usage data in the **Agent Details** table is refreshed in two ways: ### Where to View Usage Metrics In the **Agent Inventory Dashboard**, review the **Agents** grid. -If the **Total Usage/Month** field contains a value, the **Usage Metrics** section will be displayed on the **Agent Details** page. +If the **Total Usage/Month** field contains a value, the **Usage Metrics** section will be displayed on the **Agent Details** page. + +![Copilot Studio Kit - Agent Details With Usage](https://github.com/user-attachments/assets/197f0539-016c-4c26-8439-e2382fab9349) ## Technical Details diff --git a/CONVERSATION_ANALYZER.md b/CONVERSATION_ANALYZER.md index bf2e83c3..00086bc2 100644 --- a/CONVERSATION_ANALYZER.md +++ b/CONVERSATION_ANALYZER.md @@ -1,52 +1,95 @@ -# Conversation Analyzer (Preview) - -Conversation analyzer allows makers to analyze the conversation transcripts of their Copilot Studio custom agents using custom prompts. The feature comes with two pre-canned prompts, sentiment analysis and PII analysis. Custom prompts can be created, used, saved and reused later. Using custom prompts to analyze the conversations can provide insights not available through traditional analytics. - -## Conversation Analyzer overview -After navigating to the conversation analyzer feature, a list of agents is shown. Conversation Analyzer uses the agent data in the Agent Inventory, so it must be populated first before using conversation analyzer. - -## Technical Details - -### Canvas App - -| Display Name | Logical Name | -|---|---| -| Conversation Analyser | `cat_conversationanalyser` | - -### Cloud Flows - -| Flow Name | Purpose | -|---|---| -| Conversation Analyzer \| Analyze Conversation Based on Prompt | Runs the selected prompt against a conversation transcript using AI Builder | -| Conversation Analyzer \| Get the Conversation Transcript | Retrieves the conversation transcript from Copilot Studio for analysis | - -### Connection References - -| Connection Reference | Required | -|---|---| -| Copilot Studio Kit - Dataverse | Yes | - -### Environment Variables - -None specific to this feature. - -### Dataverse Tables - -| Display Name | Logical Name | Notes | -|---|---|---| -| Conversation Analyzer | `cat_ConversationAnalyzer` | Stores analysis results | -| Conversation Analyzer Prompt | `cat_ConversationAnalyzerPrompt` | Stores built-in and custom prompts | -| Agent Details | `cat_AgentDetails` | **Dependency** — requires Agent Inventory to be populated before use | - -### Prerequisites - -- **Agent Inventory** must be populated first. Conversation Analyzer reads agent data from the Agent Inventory tables; analysis cannot run without it. -- **AI Builder credits** are required for prompt-based analysis. - -### DLP Connectors - -| Connector | -|---| -| Microsoft Dataverse | - -Back to the [landing page](./README.md#power-cat-copilot-studio-kit) +# Conversation Analyzer (Preview) + +Conversation analyzer allows makers to analyze the conversation transcripts of their Copilot Studio custom agents using custom prompts. The feature comes with two pre-canned prompts, sentiment analysis and PII analysis. Custom prompts can be created, used, saved and reused later. Using custom prompts to analyze the conversations can provide insights not available through traditional analytics. + +## Conversation Analyzer overview + +After navigating to the conversation analyzer feature, a list of agents is shown. Conversation Analyzer uses the agent data in the Agent Inventory, so it must be populated first before using conversation analyzer. + +The list of agents has "Action" column that has either "Analyze" or "View Results" hyperlink in it. "View Results" indicates that the recent transcripts have already been analyzed and can be viewed. + +"Analyze" indicates that no analysis exists and recent transcripts for the agent can be selected for analysis. + +overview + +## Creating new analysis + +After selecting an agent and clicking "Analyze" from the action, prompt selection screen is shown. The feature ships with two precanned prompts, "Sentiment Analysis" and "PII Analysis". + +![Copilot Studio Kit - New Analysis](https://github.com/user-attachments/assets/50aed206-b3d5-4a30-a464-4a6696d437e4) + + +The primary purpose of the feature is to allow makers to use custom prompts to analyze the conversations with. To create a new custom prompt or updating an existing custom prompt, go to the **Manage Prompts** page, enter a descriptive name in the Prompt Name field, and add your prompt text in the **Prompt** textbox. + +![Copilot Studio Kit - Manage Prompts](https://github.com/user-attachments/assets/86707fda-3291-4512-9d8e-5594667f3536) + +![Copilot Studio Kit - Manage Prompts - Edit](https://github.com/user-attachments/assets/27718f22-4ae1-4498-93d8-66c75cfeb0fe) + + +Once the custom prompt is saved, it becomes available for future use in the prompt selection dropdown list. + +When the analysis is started, it will take a moment to process. After the processing is finished, results will be presented. + +## Viewing existing analysis + +If "View Results" is selected from the agent selections screen, analysis results are shown. Same view is presented after successful (new) analysis has completed. + +analysis_ready + +Conversation Id, summary of the conversation, duration of the conversation, prompt used and analysis creation date are shown. Depending on the prompt, the summary may already provide valuable insights. Conversation and analysis details can be shown by pressing "View Results" from the "Action" column. + +## Conversation analysis details + +Conversation analysis details view varies slightly depending on the prompt used. The conversation can be seen on the right side of the screen as it happened, and analysis details are shown in the main view. + +results_details + +This feature is experimental and the team is looking forward to hearing feedback from users. + +## Technical Details + +### Canvas App + +| Display Name | Logical Name | +|:--|:--| +| Conversation Analyser | `cat_conversationanalyser` | + +### Connection References + +| Connection Reference | Connector | Required | +|:--|:--|:--| +| Copilot Studio Kit - Dataverse | `shared_commondataserviceforapps` | Yes | + +### Environment Variables + +None specific to this feature. + +### Cloud Flows + +| Flow Name | Purpose | +|:--|:--| +| Conversation Analyzer \| Analyze Conversation Based on Prompt | Runs the selected prompt against a conversation transcript using AI Builder | +| Conversation Analyzer \| Get the Conversation Transcript | Retrieves the conversation transcript from Copilot Studio for analysis | + +### Dataverse Tables + +| Display Name | Logical Name | Notes | +|:--|:--|:--| +| Conversation Analyzer | `cat_ConversationAnalyzer` | Stores analysis results | +| Conversation Analyzer Prompt | `cat_ConversationAnalyzerPrompt` | Stores built-in and custom prompts | +| Agent Details | `cat_AgentDetails` | **Dependency** — requires Agent Inventory to be populated before use | + +### Prerequisites + +- **Agent Inventory** must be populated first. Conversation Analyzer reads agent data from the Agent Inventory tables; analysis cannot run without it. +- **AI Builder credits** are required for prompt-based analysis. + +### DLP Configuration + +The following connectors must be allowed in your environment's DLP policy: + +| Connector | Reason | +|:--|:--| +| Microsoft Dataverse | Core data operations | + +Back to the [landing page](./README.md#power-cat-copilot-studio-kit) \ No newline at end of file diff --git a/CONVERSATION_KPIS.md b/CONVERSATION_KPIS.md index e4417de1..2806b4b1 100644 --- a/CONVERSATION_KPIS.md +++ b/CONVERSATION_KPIS.md @@ -12,7 +12,11 @@ Conversation KPIs are designed to help makers track and analyze the performance * Long-term tracking: Conversation KPIs can be stored in the system for as long as required which allows tracking the impact of improvements and performance of custom copilots over long time-period. -Conversation KPIs are generated automatically twice per day and on-demand generation is supported as well. To generate KPIs on-demand, navigate to the agent configuration, press "**Generate KPIs**" and select a date range. Up to 75,000 transcripts are processed per automatic run and up to 50,000 per on-demand run. Please see [agent configuration](./CONFIGURE_COPILOTS.md) for more details on how to configure the Conversation KPIs. +Conversation KPIs are generated automatically twice per day and on-demand generation is supported as well. To generate KPIs on-demand, navigate to the agent configuration, press "**Generate KPIs**" and select a date range. Up to 75,000 transcripts are processed per automatic run and up to 50,000 per on-demand run. Please see [agent configuration](./CONFIGURE_COPILOTS.md) for more details on how to configure the Conversation KPIs. + +![kpi report overview](https://github.com/user-attachments/assets/bca1bc9e-2d6f-42bc-a6b6-798003999f21) + +![kpi details1](https://github.com/user-attachments/assets/96b48373-a7a0-4062-adb8-68bd97d22e12) Read more on how to configure the agent for [KPI generation](./CONFIGURE_COPILOTS.md#configure-a-new-agent-for-conversation-kpis) diff --git a/FILE_SYNCHRONIZATION.md b/FILE_SYNCHRONIZATION.md index 790e196c..95eade8d 100644 --- a/FILE_SYNCHRONIZATION.md +++ b/FILE_SYNCHRONIZATION.md @@ -6,7 +6,9 @@ Benefits of using SharePoint synchronization (compared to using SharePoint site - Support for additional file types - Support for larger files (up to 512MB) - Lower latency in responses -- Indexing of non-text elements in PDFs +- Indexing of non-text elements in PDFs + +![file sync](https://github.com/user-attachments/assets/90bc990a-646c-4311-863f-6c5cba43a48d) Please see [agent configuration](./CONFIGURE_COPILOTS.md) for more details and how to configure SharePoint synchronization. SharePoint synchronization is automatically performed daily and it can be run on-demand by selecting "**Sync Files**" from the agent configuration. diff --git a/TESTING_CAPABILITIES.md b/TESTING_CAPABILITIES.md index a85ff5f7..28176bc9 100644 --- a/TESTING_CAPABILITIES.md +++ b/TESTING_CAPABILITIES.md @@ -11,7 +11,11 @@ Today, the tool supports these types of tests: - Topic match (requires Dataverse enrichment) - Generative answers (requires AI Builder for response analysis, and Azure Application Insights for details on why an answer was or was not generated) - Multi-turn test type is a special test type that consists of a set of test cases of *regular* types that are executed in specified order in same conversation context. Multi-turn tests can be used to test scenarios end-to-end, and for testing custom agents with generative orchestration. -- Plan validation allows maker to validate the tools included in the dynamic plan of Copilot Studio custom agent with generative orchestration +- Plan validation allows maker to validate the tools included in the dynamic plan of Copilot Studio custom agent with generative orchestration + +![image](https://github.com/microsoft/Power-CAT-Copilot-Studio-Kit/assets/37898885/33496e94-0c7a-4e63-9291-9e461aa9b9e7) + +![image](https://github.com/microsoft/Power-CAT-Copilot-Studio-Kit/assets/37898885/25f071ff-6b4f-4f5f-b6a3-20193a2d1feb) Read more on how to configure agent for [test automation](./CONFIGURE_COPILOTS.md#configure-a-new-agent-for-test-automation)