Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added public/bundles/copilot-studio-agent-test.zip
Binary file not shown.
83 changes: 83 additions & 0 deletions src/content/guides/booth-loop-video.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# Booth Loop Video

You need something on the screen behind your stand. Not a slideshow with a
transition every eight seconds, and not a video that needs an editor, a brief, and
three weeks. This skill renders a **silent, looping MP4** — 1920x1080, 30 fps,
one to two minutes — from a description of what you want it to say.

Describe the message, hand over your brand colours, and the agent writes a Python
render script, shows you a still from every scene, and only then renders the video.

## What it does

The agent generates a single self-contained script that paints **every frame** with
Pillow and pipes them to ffmpeg. That means real motion — eased fades, animated
counters, progress bars, typed-text reveals, orbiting nodes — not a set of static
slides with a crossfade.

Because the whole thing is a script, iteration is conversational: "slower orbit",
"make the headline bigger", "swap magenta for green" edits a few constants and
re-renders.

## Why frame-by-frame instead of a slide export

Exporting a deck to video gives you a deck playing at 0.1 fps. This gives you a
motion graphic. The trade-off is that you're writing code rather than dragging
boxes — which is exactly the trade-off an agent is good at absorbing.

## What makes it different

Most of this skill is the accumulated list of things that go wrong, written down so
they don't go wrong again:

- **The z-order rule.** Pillow paints in call order. Draw a hub, then its spokes,
and the spoke lines run straight across the hub and through your labels. The skill
forces a two-pass draw — all connectors, then all nodes — for any diagram layout.
This is the single most common defect and it's invisible until you look at a frame.
- **Preview before render.** A full render takes minutes; a preview PNG takes
seconds. The agent renders one still per scene and shows them to you inline
*before* committing to the render.
- **`-pix_fmt yuv420p`.** Without it the MP4 plays fine in VLC and in nothing else —
not QuickTime, not PowerPoint, not the media player on the screen at the venue.
- **Font probing across platforms.** A hardcoded font path silently falls back to a
tiny bitmap font and produces a video that looks broken. The skill probes a
candidate list per weight and reports what actually resolved.
- **Contrast for a show floor.** Dark card on dark background is a design-tool
favourite and unreadable on a badly calibrated panel under bright lights. Chat and
quote UI goes on a white card, and contrast gets checked on a rendered frame.
- **Frames go to the system temp folder**, never the working directory — rendering
thousands of PNGs into a OneDrive- or Dropbox-synced folder stalls the render and
thrashes the sync client.

## How to use it

1. Tell the agent what the video is for and roughly what it should say — the pitch,
the three things you want a passer-by to take away, the call to action.
2. Give it your brand colours, or let it use the neutral navy/blue default. It won't
invent a brand for you.
3. Review the per-scene preview stills it sends back and correct anything that looks
wrong. This is the cheap moment to fix layout.
4. Approve the render. Output lands at `output/booth_loop.mp4`.
5. Iterate in plain language until it's right.

## Requirements

Python with `pillow`, `imageio`, `imageio-ffmpeg`, and `numpy`. The agent installs
them if they're missing. `imageio-ffmpeg` bundles its own ffmpeg binary, so there's
no system ffmpeg install and nothing to put on `PATH`.

## Tips

- **Write for eight seconds of attention at three metres.** If the opening scene
doesn't land on its own, nothing after it will.
- Keep it under ~25 words on screen at any moment.
- Make the first and last frames identical so the loop point disappears.
- It's silent by design. Booths are loud and most venue screens are muted anyway.
- The output works as a LinkedIn post or an embedded slide, not just a booth loop.

## Known limitations

- No audio track. Add one afterwards with ffmpeg if you need it.
- No live footage or video-file editing — this generates from scratch.
- Complex scenes at 30 fps take a few minutes to render. Preview stills are the
answer, not patience.
89 changes: 89 additions & 0 deletions src/content/guides/browser-uat-analyst.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
# Browser UAT Analyst

Testing done by an agent has a failure mode: it reads the docs, drives two clicks,
and tells you the feature works. This skill exists to stop that. It turns the agent
into a sceptical test analyst that will not claim a thing works until it has a
screenshot proving it, and that tells you *which kind* of failure it found.

Point it at anything you can drive in a browser — a Copilot Studio agent, a
model-driven or canvas app, a Power Automate flow's run history, a Dataverse form,
a Dynamics 365 module, the Teams web client, a Microsoft 365 admin centre, or your
own web app.

## What it does

1. Writes a **test charter** first — objective, environment, test data, hypotheses,
success criteria, and what's out of scope.
2. Verifies environment readiness before touching anything.
3. Decomposes the request into user-visible scenarios, then adds an adversarial pass
for the ways it actually breaks in the field.
4. Executes with Playwright, screenshotting **before and after** every key action.
5. Keeps an evidence ledger as it goes.
6. Synthesises findings into a summary, an outcome matrix, and a recommendation.

## What makes it different

**It classifies failures instead of just reporting them.** Every failure gets tagged
as a product limitation, a product defect, a test-harness limitation, a tenant or
configuration issue, an authentication issue, or model drift. This one habit is the
difference between a report someone acts on and a report that gets dismissed —
because filing a tenant misconfiguration as a product bug destroys the credibility
of everything else in the document.

**"Blocked" is a real result.** If the agent couldn't reach the state, it says so.
It never upgrades an unobserved behaviour to a pass.

**The evidence ledger has a "what it proves" column.** If you can't fill that column
in, the screenshot is decoration and the scenario is untested. This is the discipline
that stops evidence theatre.

**Failed approaches get their own slides.** The "we tried this and it doesn't work"
content is usually the most valuable output and the first thing that gets cut.
The skill requires it.

**It knows how rich chat UI actually fails.** Testing a conversational agent's
formatting, Adaptive Cards, hosted images, dynamically generated images, and
interactive actions are five separate tests with five separate failure modes — and
all of them behave differently in the authoring test pane than in Teams. Model-emitted
Adaptive Card JSON rendering as a code block instead of a card is the classic example
this skill is built to catch.

## How to use it

Ask for what you need in plain language:

- "UAT the new agent's escalation flow before we ship it"
- "Check whether the chart actually renders in Teams or just in the test pane"
- "Compare how this setting appears in the Teams admin centre vs the M365 admin centre"
- "Work out whether that failure is our config or a product bug"

The agent will propose a charter before it starts executing. Correct the scope there
— it's much cheaper than correcting the report.

## Requirements

Playwright or equivalent browser automation, plus a signed-in browser session for
whatever you're testing. Screenshots are written to `output/<test-name>/`.

For the reporting step, pair it with a PowerPoint skill if you want a deck; the
skill will export the slides to images and inspect them for overflow, tiny text,
and clipping before handing it over.

## Tips

- **Let it be slow with admin centres.** Premature assertions against heavy SPAs
generate false failures that then take longer to disprove than the wait would have
taken.
- Give it deterministic test data. Letting a model invent facts and then testing the
invention proves nothing.
- Ask it explicitly for the negative cases — empty data, wrong permissions, expired
session. That's where the real findings are.
- If a result surprises you, ask it what evidence would disambiguate the cause. It's
built to answer that rather than defend its first hypothesis.

## Known limitations

- Browser-observable behaviour only. Not for unit tests, API contract tests, or
load testing.
- Non-deterministic AI responses can't be asserted the way a deterministic UI can;
the skill flags model drift as its own category rather than pretending otherwise.
112 changes: 112 additions & 0 deletions src/content/guides/copilot-studio-agent-test.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
# Copilot Studio Agent Test

Test a Microsoft Copilot Studio agent against your own question-and-answer set, and get back a
graded pass/fail report — without driving a browser.

Point it at a Word document or CSV of questions with their expected answers. It converts them
into a test set, asks the agent every question in a **fresh conversation**, then reads each
answer against the expected one and writes a report with a pass rate, a per-question table, and
the reason behind every failure.

Built for the case where a customer hands you a pile of Q&A documents and asks *"is our agent
actually answering these correctly?"*

## Why not just use the browser?

Driving the Copilot Studio test pane with browser automation works, but it is slow and brittle:
you are waiting on UI animations and guessing when an answer has finished streaming. This skill
talks to the agent through the **Microsoft 365 Agents SDK** instead (`CopilotStudio.Copilots.Invoke`),
which is roughly ten times faster and does not break when the UI changes.

## What it does — and deliberately does not do

**It does:** convert Word Q&A tables into test sets, run every question in an isolated
conversation, capture answers and citations, detect escalation and empty turns, check optional
required substrings, and assemble the final report.

**It does not** decide by itself whether an answer is *correct*. That grading pass is done by
the agent running the skill (or by you), reading each answer against the expected one.

This is a deliberate design decision, not a missing feature. A similarity score would
manufacture false failures: in practice a correct answer routinely differs in wording from the
expected text, and sometimes improves on it. So the tool captures evidence, a human-or-model
judgement produces verdicts, and the report carries those real verdicts instead of a number
nobody trusts.

## Setup

**Prerequisite:** Node.js 22 or later.

Ask your assistant: **"set up the Copilot Studio agent test skill"**. It will ask you for four
values and configure everything:

| Value | Where to find it |
|---|---|
| Application (client) ID | your Entra app registration → Overview |
| Directory (tenant) ID | same page |
| Environment ID | the GUID in the Copilot Studio URL, after `/environments/` |
| Schema name | see below — it differs between the two Copilot Studio experiences |

**Schema name** is not the display name; it looks like `cr1a3_myAgentName`.

- **New authoring experience** (Build / Preview / Evaluate / Monitor tabs):
agent → **⋯ (More options)** → **Settings** → **Agent details** → **Identity** → **Schema name**
- **Classic agents**: agent → **Settings** → **Advanced** → **Metadata** → **Schema name**

You also need an Entra app registration with **Allow public client flows** enabled and the
delegated **`CopilotStudio.Copilots.Invoke`** permission (note the plural) granted admin
consent. The skill walks you through creating one if you do not have it.

There is **no client secret** — sign-in is device code, so nothing confidential is stored. Your
settings go to `~/.cs-agent-test/`, never into the skill folder.

## Using it

Ask: **"test my Copilot Studio agent with these questions"** and give it your file.

Or drive it yourself:

```
node scripts/cs-agent-test.cjs convert my-questions.docx testset.csv
node scripts/cs-agent-test.cjs test testset.csv --out .\results
node scripts/cs-agent-test.cjs report .\results\results-<ts>.json --template verdicts.csv
REM ... fill in PASS / FAIL / EXCLUDED and a reason for each row ...
node scripts/cs-agent-test.cjs report .\results\results-<ts>.json report.md --verdicts verdicts.csv
```

On macOS or Linux use `sh scripts/cs-agent-test.sh`, or `node scripts/cs-agent-test.cjs`
anywhere.

Expect **30–60 seconds per question** — a 60-question set takes about 45 minutes.

## Two things that will otherwise waste your afternoon

**It tests the PUBLISHED agent, not the draft.** The Copilot Studio test pane tests the draft.
Anything you just changed in the authoring canvas is invisible here until you publish.

**Agents built with the new authoring experience run on a different endpoint.** Reached on the
classic endpoint, they reply with *"Enhanced task completion preview has ended. Go to
copilotstudio.microsoft.com and republish the agent"* — which looks like a real answer and is
not fixed by republishing. The skill detects that notice and transparently retries on the
agentic runtime endpoint, so you should never have to think about it.

## No executable

This skill ships **no binary** — `scripts/cs-agent-test.cjs` is a single self-contained
JavaScript file (~1.5 MB, no `npm install` needed), with thin `.cmd` and `.sh` launchers.

An earlier version shipped a 57 MB packaged `.exe`. That was replaced on purpose: an unsigned
executable is a reasonable thing for endpoint policy to block, code signing would not have
removed SmartScreen warnings anyway (reputation comes from download history, not from a
certificate), and 1.5 MB of readable JavaScript can actually be reviewed by a security team.
As a bonus, the JavaScript runs on macOS and Linux too.

`references/THIRD-PARTY-NOTICES.md` lists every open-source package in the bundle.

## Limitations

- **Delegated auth only.** The Agents SDK does not accept service principal tokens, so fully
unattended CI is not possible. One device-code sign-in is cached and covers a whole run.
- **Windows, macOS and Linux**, wherever Node.js 22+ runs.
- Occasional consent prompts or transient stalls surface as errors on individual rows; re-run
those with `--ids`.
102 changes: 102 additions & 0 deletions src/content/guides/travel-cost-estimator.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
# Travel Cost Estimator

"Roughly what will that trip cost?" is a question that gets answered with a guess,
and the guess is wrong often enough that people stop trusting the number. This skill
answers it with **live fares** pulled from your corporate booking tool, and fills the
lines the booking tool doesn't cover by benchmarking against **your own approved
expense reports** — so the number holds up when your manager pushes back on it.

It produces a forwardable estimate: headline figure, cost table, all the options it
considered, and the assumptions stated as assumptions.

## The important part: it never books anything

Your corporate booking tool is a live booking system. Real money, real tickets, real
change fees. Letting an agent loose in one is a genuinely bad idea unless the
boundary is explicit, so the boundary is the first thing in the skill and it's
absolute:

> **Search and price only.** Never click `Continue` past fare selection, `Select
> hotel` into a booking funnel, `Book`, `Reserve`, `Hold`, `Confirm`, or anything on
> a traveller-details or payment page. Reading fares off a results page is safe.
> The results page is where you stop.

If a flow ever lands on a payment or traveller-details page, the skill stops and
tells you where it is rather than clicking its way back out. Every estimate it
produces states **"Nothing booked."**

## What makes it different

**It benchmarks the unbookable lines against your actual reimbursements.** Ground
transport, meals, and parking aren't in the booking tool, and this is exactly where
estimates get invented. The skill pulls totals from your own approved expense
reports for comparable trips and derives the figures from those — then labels them
"assumption" in the table rather than dressing them up as priced.

**It always surfaces a materially cheaper alternative**, even when that breaks your
stated airline or hotel preference. You get the trade-off — cost vs. proximity vs.
loyalty — and you choose. It won't silently optimise for price, and it won't
silently optimise for your preferences. Presenting one option as if it were the only
option is the failure mode it's built to avoid.

**It knows the DOM tricks that make these portals workable.** Enterprise booking
tools are old, slow, and hostile to automation. The skill carries the specific
patterns:

- **Date fields are hidden inputs behind a calendar widget.** Assigning `.value`
directly is ignored by the framework — you have to go through
`HTMLInputElement.prototype`'s native setter and dispatch `input`/`change`/`blur`
yourself. Driving the visible calendar picker instead is flaky and slow.
- **Location controls are select2-style widgets** whose real `<select>` is
`aria-hidden` and unclickable. You drive the visible proxy span.
- **Pick the airport option, not the metro/city option** — the city entry resolves to
a different and much less useful inventory set.
- **Results take ~20 seconds.** Read the page text rather than snapshotting; the
results table is huge and a snapshot burns context for nothing.
- **A search click that times out waiting for navigation usually submitted fine.**
Retrying double-submits.
- **Don't reuse a deep link with a session token in it** — it expires and the symptom
is a misleading "Cookies are disabled" page, not a session error.

**One gotcha worth the price of admission on its own:** when forwarding the estimate
over Teams, resolve the recipient by **UPN**, not by vanity SMTP alias.
`First.Last@company.com` works for mail and fails Teams chat creation with
`user_not_found`. That error looks like "this person doesn't exist" and isn't.

## How to use it

1. Ask for a trip estimate — "price up a two-day visit to the Orlando customer the
week of the 26th".
2. Confirm your travel preferences the first time: home airport, preferred airline
and hotel chain, cabin class your policy allows. It reuses these afterwards but
re-confirms dates every run.
3. Watch it price flights and hotels, or don't — it writes the full working to a file.
4. Read the headline number and the one or two judgement calls it flags.
5. Optionally have it draft the message to your approver. It shows you the exact text
and recipient and waits for confirmation before sending.

## Requirements

- Browser automation (Playwright) with an existing SSO session to your corporate
booking tool. The skill never asks for credentials — if SSO isn't working it stops.
- Read access to your own mailbox, for the expense-report benchmarking step.
- The DOM selectors are written against Cytric/Amadeus-family portals. Concur Travel,
Egencia, and similar tools use different field names — inspect once and substitute.
The **techniques** (native setter, select2 proxy, long waits) transfer unchanged.

## Tips

- **Don't skip the home-airport confirmation.** Guessing it from your city silently
produces a plausible, useless number when your metro has several airports.
- Ask it explicitly whether a rental car is likely. It's the most common reason an
estimate comes in low.
- Prices drift daily. An estimate is a snapshot, and the skill says so in every
output — re-run it if the trip is more than a couple of weeks out.
- The estimate contains your dates, destination, and customer. Treat it as private
until you've decided to share it.

## Known limitations

- Estimates only. By design it cannot and will not book, hold, or confirm anything.
- Fares are not held. What it quotes today is not what you'll pay next week.
- Selectors are portal-specific and will need adjusting for a different booking tool.
Loading
Loading