diff --git a/public/bundles/copilot-studio-agent-test.zip b/public/bundles/copilot-studio-agent-test.zip new file mode 100644 index 00000000..4805ac1a Binary files /dev/null and b/public/bundles/copilot-studio-agent-test.zip differ diff --git a/src/content/guides/booth-loop-video.md b/src/content/guides/booth-loop-video.md new file mode 100644 index 00000000..3e642e2d --- /dev/null +++ b/src/content/guides/booth-loop-video.md @@ -0,0 +1,83 @@ +# Booth Loop Video + +You need something on the screen behind your stand. Not a slideshow with a +transition every eight seconds, and not a video that needs an editor, a brief, and +three weeks. This skill renders a **silent, looping MP4** — 1920x1080, 30 fps, +one to two minutes — from a description of what you want it to say. + +Describe the message, hand over your brand colours, and the agent writes a Python +render script, shows you a still from every scene, and only then renders the video. + +## What it does + +The agent generates a single self-contained script that paints **every frame** with +Pillow and pipes them to ffmpeg. That means real motion — eased fades, animated +counters, progress bars, typed-text reveals, orbiting nodes — not a set of static +slides with a crossfade. + +Because the whole thing is a script, iteration is conversational: "slower orbit", +"make the headline bigger", "swap magenta for green" edits a few constants and +re-renders. + +## Why frame-by-frame instead of a slide export + +Exporting a deck to video gives you a deck playing at 0.1 fps. This gives you a +motion graphic. The trade-off is that you're writing code rather than dragging +boxes — which is exactly the trade-off an agent is good at absorbing. + +## What makes it different + +Most of this skill is the accumulated list of things that go wrong, written down so +they don't go wrong again: + +- **The z-order rule.** Pillow paints in call order. Draw a hub, then its spokes, + and the spoke lines run straight across the hub and through your labels. The skill + forces a two-pass draw — all connectors, then all nodes — for any diagram layout. + This is the single most common defect and it's invisible until you look at a frame. +- **Preview before render.** A full render takes minutes; a preview PNG takes + seconds. The agent renders one still per scene and shows them to you inline + *before* committing to the render. +- **`-pix_fmt yuv420p`.** Without it the MP4 plays fine in VLC and in nothing else — + not QuickTime, not PowerPoint, not the media player on the screen at the venue. +- **Font probing across platforms.** A hardcoded font path silently falls back to a + tiny bitmap font and produces a video that looks broken. The skill probes a + candidate list per weight and reports what actually resolved. +- **Contrast for a show floor.** Dark card on dark background is a design-tool + favourite and unreadable on a badly calibrated panel under bright lights. Chat and + quote UI goes on a white card, and contrast gets checked on a rendered frame. +- **Frames go to the system temp folder**, never the working directory — rendering + thousands of PNGs into a OneDrive- or Dropbox-synced folder stalls the render and + thrashes the sync client. + +## How to use it + +1. Tell the agent what the video is for and roughly what it should say — the pitch, + the three things you want a passer-by to take away, the call to action. +2. Give it your brand colours, or let it use the neutral navy/blue default. It won't + invent a brand for you. +3. Review the per-scene preview stills it sends back and correct anything that looks + wrong. This is the cheap moment to fix layout. +4. Approve the render. Output lands at `output/booth_loop.mp4`. +5. Iterate in plain language until it's right. + +## Requirements + +Python with `pillow`, `imageio`, `imageio-ffmpeg`, and `numpy`. The agent installs +them if they're missing. `imageio-ffmpeg` bundles its own ffmpeg binary, so there's +no system ffmpeg install and nothing to put on `PATH`. + +## Tips + +- **Write for eight seconds of attention at three metres.** If the opening scene + doesn't land on its own, nothing after it will. +- Keep it under ~25 words on screen at any moment. +- Make the first and last frames identical so the loop point disappears. +- It's silent by design. Booths are loud and most venue screens are muted anyway. +- The output works as a LinkedIn post or an embedded slide, not just a booth loop. + +## Known limitations + +- No audio track. Add one afterwards with ffmpeg if you need it. +- No live footage or video-file editing — this generates from scratch. +- Complex scenes at 30 fps take a few minutes to render. Preview stills are the + answer, not patience. diff --git a/src/content/guides/browser-uat-analyst.md b/src/content/guides/browser-uat-analyst.md new file mode 100644 index 00000000..073fc6b0 --- /dev/null +++ b/src/content/guides/browser-uat-analyst.md @@ -0,0 +1,89 @@ +# Browser UAT Analyst + +Testing done by an agent has a failure mode: it reads the docs, drives two clicks, +and tells you the feature works. This skill exists to stop that. It turns the agent +into a sceptical test analyst that will not claim a thing works until it has a +screenshot proving it, and that tells you *which kind* of failure it found. + +Point it at anything you can drive in a browser — a Copilot Studio agent, a +model-driven or canvas app, a Power Automate flow's run history, a Dataverse form, +a Dynamics 365 module, the Teams web client, a Microsoft 365 admin centre, or your +own web app. + +## What it does + +1. Writes a **test charter** first — objective, environment, test data, hypotheses, + success criteria, and what's out of scope. +2. Verifies environment readiness before touching anything. +3. Decomposes the request into user-visible scenarios, then adds an adversarial pass + for the ways it actually breaks in the field. +4. Executes with Playwright, screenshotting **before and after** every key action. +5. Keeps an evidence ledger as it goes. +6. Synthesises findings into a summary, an outcome matrix, and a recommendation. + +## What makes it different + +**It classifies failures instead of just reporting them.** Every failure gets tagged +as a product limitation, a product defect, a test-harness limitation, a tenant or +configuration issue, an authentication issue, or model drift. This one habit is the +difference between a report someone acts on and a report that gets dismissed — +because filing a tenant misconfiguration as a product bug destroys the credibility +of everything else in the document. + +**"Blocked" is a real result.** If the agent couldn't reach the state, it says so. +It never upgrades an unobserved behaviour to a pass. + +**The evidence ledger has a "what it proves" column.** If you can't fill that column +in, the screenshot is decoration and the scenario is untested. This is the discipline +that stops evidence theatre. + +**Failed approaches get their own slides.** The "we tried this and it doesn't work" +content is usually the most valuable output and the first thing that gets cut. +The skill requires it. + +**It knows how rich chat UI actually fails.** Testing a conversational agent's +formatting, Adaptive Cards, hosted images, dynamically generated images, and +interactive actions are five separate tests with five separate failure modes — and +all of them behave differently in the authoring test pane than in Teams. Model-emitted +Adaptive Card JSON rendering as a code block instead of a card is the classic example +this skill is built to catch. + +## How to use it + +Ask for what you need in plain language: + +- "UAT the new agent's escalation flow before we ship it" +- "Check whether the chart actually renders in Teams or just in the test pane" +- "Compare how this setting appears in the Teams admin centre vs the M365 admin centre" +- "Work out whether that failure is our config or a product bug" + +The agent will propose a charter before it starts executing. Correct the scope there +— it's much cheaper than correcting the report. + +## Requirements + +Playwright or equivalent browser automation, plus a signed-in browser session for +whatever you're testing. Screenshots are written to `output//`. + +For the reporting step, pair it with a PowerPoint skill if you want a deck; the +skill will export the slides to images and inspect them for overflow, tiny text, +and clipping before handing it over. + +## Tips + +- **Let it be slow with admin centres.** Premature assertions against heavy SPAs + generate false failures that then take longer to disprove than the wait would have + taken. +- Give it deterministic test data. Letting a model invent facts and then testing the + invention proves nothing. +- Ask it explicitly for the negative cases — empty data, wrong permissions, expired + session. That's where the real findings are. +- If a result surprises you, ask it what evidence would disambiguate the cause. It's + built to answer that rather than defend its first hypothesis. + +## Known limitations + +- Browser-observable behaviour only. Not for unit tests, API contract tests, or + load testing. +- Non-deterministic AI responses can't be asserted the way a deterministic UI can; + the skill flags model drift as its own category rather than pretending otherwise. diff --git a/src/content/guides/copilot-studio-agent-test.md b/src/content/guides/copilot-studio-agent-test.md new file mode 100644 index 00000000..915b97df --- /dev/null +++ b/src/content/guides/copilot-studio-agent-test.md @@ -0,0 +1,112 @@ +# Copilot Studio Agent Test + +Test a Microsoft Copilot Studio agent against your own question-and-answer set, and get back a +graded pass/fail report — without driving a browser. + +Point it at a Word document or CSV of questions with their expected answers. It converts them +into a test set, asks the agent every question in a **fresh conversation**, then reads each +answer against the expected one and writes a report with a pass rate, a per-question table, and +the reason behind every failure. + +Built for the case where a customer hands you a pile of Q&A documents and asks *"is our agent +actually answering these correctly?"* + +## Why not just use the browser? + +Driving the Copilot Studio test pane with browser automation works, but it is slow and brittle: +you are waiting on UI animations and guessing when an answer has finished streaming. This skill +talks to the agent through the **Microsoft 365 Agents SDK** instead (`CopilotStudio.Copilots.Invoke`), +which is roughly ten times faster and does not break when the UI changes. + +## What it does — and deliberately does not do + +**It does:** convert Word Q&A tables into test sets, run every question in an isolated +conversation, capture answers and citations, detect escalation and empty turns, check optional +required substrings, and assemble the final report. + +**It does not** decide by itself whether an answer is *correct*. That grading pass is done by +the agent running the skill (or by you), reading each answer against the expected one. + +This is a deliberate design decision, not a missing feature. A similarity score would +manufacture false failures: in practice a correct answer routinely differs in wording from the +expected text, and sometimes improves on it. So the tool captures evidence, a human-or-model +judgement produces verdicts, and the report carries those real verdicts instead of a number +nobody trusts. + +## Setup + +**Prerequisite:** Node.js 22 or later. + +Ask your assistant: **"set up the Copilot Studio agent test skill"**. It will ask you for four +values and configure everything: + +| Value | Where to find it | +|---|---| +| Application (client) ID | your Entra app registration → Overview | +| Directory (tenant) ID | same page | +| Environment ID | the GUID in the Copilot Studio URL, after `/environments/` | +| Schema name | see below — it differs between the two Copilot Studio experiences | + +**Schema name** is not the display name; it looks like `cr1a3_myAgentName`. + +- **New authoring experience** (Build / Preview / Evaluate / Monitor tabs): + agent → **⋯ (More options)** → **Settings** → **Agent details** → **Identity** → **Schema name** +- **Classic agents**: agent → **Settings** → **Advanced** → **Metadata** → **Schema name** + +You also need an Entra app registration with **Allow public client flows** enabled and the +delegated **`CopilotStudio.Copilots.Invoke`** permission (note the plural) granted admin +consent. The skill walks you through creating one if you do not have it. + +There is **no client secret** — sign-in is device code, so nothing confidential is stored. Your +settings go to `~/.cs-agent-test/`, never into the skill folder. + +## Using it + +Ask: **"test my Copilot Studio agent with these questions"** and give it your file. + +Or drive it yourself: + +``` +node scripts/cs-agent-test.cjs convert my-questions.docx testset.csv +node scripts/cs-agent-test.cjs test testset.csv --out .\results +node scripts/cs-agent-test.cjs report .\results\results-.json --template verdicts.csv +REM ... fill in PASS / FAIL / EXCLUDED and a reason for each row ... +node scripts/cs-agent-test.cjs report .\results\results-.json report.md --verdicts verdicts.csv +``` + +On macOS or Linux use `sh scripts/cs-agent-test.sh`, or `node scripts/cs-agent-test.cjs` +anywhere. + +Expect **30–60 seconds per question** — a 60-question set takes about 45 minutes. + +## Two things that will otherwise waste your afternoon + +**It tests the PUBLISHED agent, not the draft.** The Copilot Studio test pane tests the draft. +Anything you just changed in the authoring canvas is invisible here until you publish. + +**Agents built with the new authoring experience run on a different endpoint.** Reached on the +classic endpoint, they reply with *"Enhanced task completion preview has ended. Go to +copilotstudio.microsoft.com and republish the agent"* — which looks like a real answer and is +not fixed by republishing. The skill detects that notice and transparently retries on the +agentic runtime endpoint, so you should never have to think about it. + +## No executable + +This skill ships **no binary** — `scripts/cs-agent-test.cjs` is a single self-contained +JavaScript file (~1.5 MB, no `npm install` needed), with thin `.cmd` and `.sh` launchers. + +An earlier version shipped a 57 MB packaged `.exe`. That was replaced on purpose: an unsigned +executable is a reasonable thing for endpoint policy to block, code signing would not have +removed SmartScreen warnings anyway (reputation comes from download history, not from a +certificate), and 1.5 MB of readable JavaScript can actually be reviewed by a security team. +As a bonus, the JavaScript runs on macOS and Linux too. + +`references/THIRD-PARTY-NOTICES.md` lists every open-source package in the bundle. + +## Limitations + +- **Delegated auth only.** The Agents SDK does not accept service principal tokens, so fully + unattended CI is not possible. One device-code sign-in is cached and covers a whole run. +- **Windows, macOS and Linux**, wherever Node.js 22+ runs. +- Occasional consent prompts or transient stalls surface as errors on individual rows; re-run + those with `--ids`. diff --git a/src/content/guides/travel-cost-estimator.md b/src/content/guides/travel-cost-estimator.md new file mode 100644 index 00000000..309184f9 --- /dev/null +++ b/src/content/guides/travel-cost-estimator.md @@ -0,0 +1,102 @@ +# Travel Cost Estimator + +"Roughly what will that trip cost?" is a question that gets answered with a guess, +and the guess is wrong often enough that people stop trusting the number. This skill +answers it with **live fares** pulled from your corporate booking tool, and fills the +lines the booking tool doesn't cover by benchmarking against **your own approved +expense reports** — so the number holds up when your manager pushes back on it. + +It produces a forwardable estimate: headline figure, cost table, all the options it +considered, and the assumptions stated as assumptions. + +## The important part: it never books anything + +Your corporate booking tool is a live booking system. Real money, real tickets, real +change fees. Letting an agent loose in one is a genuinely bad idea unless the +boundary is explicit, so the boundary is the first thing in the skill and it's +absolute: + +> **Search and price only.** Never click `Continue` past fare selection, `Select +> hotel` into a booking funnel, `Book`, `Reserve`, `Hold`, `Confirm`, or anything on +> a traveller-details or payment page. Reading fares off a results page is safe. +> The results page is where you stop. + +If a flow ever lands on a payment or traveller-details page, the skill stops and +tells you where it is rather than clicking its way back out. Every estimate it +produces states **"Nothing booked."** + +## What makes it different + +**It benchmarks the unbookable lines against your actual reimbursements.** Ground +transport, meals, and parking aren't in the booking tool, and this is exactly where +estimates get invented. The skill pulls totals from your own approved expense +reports for comparable trips and derives the figures from those — then labels them +"assumption" in the table rather than dressing them up as priced. + +**It always surfaces a materially cheaper alternative**, even when that breaks your +stated airline or hotel preference. You get the trade-off — cost vs. proximity vs. +loyalty — and you choose. It won't silently optimise for price, and it won't +silently optimise for your preferences. Presenting one option as if it were the only +option is the failure mode it's built to avoid. + +**It knows the DOM tricks that make these portals workable.** Enterprise booking +tools are old, slow, and hostile to automation. The skill carries the specific +patterns: + +- **Date fields are hidden inputs behind a calendar widget.** Assigning `.value` + directly is ignored by the framework — you have to go through + `HTMLInputElement.prototype`'s native setter and dispatch `input`/`change`/`blur` + yourself. Driving the visible calendar picker instead is flaky and slow. +- **Location controls are select2-style widgets** whose real `