Copilot CLI AgentOps for Azure

Find the failed or slow tool call in a Copilot CLI run. AgentOps turns local Copilot CLI sessions into a trace waterfall with failure details, tool latency and token use. Azure Monitor is optional, and prompts, code, tool arguments and tool results are not recorded by default.

Who it's for: developers building and debugging coding agents who want to see what a Copilot CLI run actually did.

Independent open-source project. Not an official Microsoft, GitHub, OpenAI, Azure or Grafana product. Developer preview.

Quickstart

Try the local demo. Node.js 20+ only; no account, Azure or Docker.

git clone https://github.com/c-mongan/copilot-cli-agentops-azure && cd copilot-cli-agentops-azure
node scripts/start-ui-demo.js

Open synthetic-demo-permission-denied, inspect the denied fetch call, then compare it with the 12-second bash step. This uses the real read-only UI with generated sample data, not production evidence. It reads no personal sessions and makes no model or cloud calls. Press Ctrl+C to stop and remove its temporary data.

Inspect your own Copilot CLI sessions. You need Node.js 20+ and the GitHub Copilot CLI, signed in.

# 1. Get the CLI (one command, no global install; npx fetches the release tarball)
alias agentops='npx --yes -p https://github.com/c-mongan/copilot-cli-agentops-azure/releases/download/v0.3.5-preview/copilot-agentops-cli-0.3.5-preview.tgz agentops'

# 2. Run an observed Copilot session (local only, no upload). In -p mode Copilot
#    cannot ask for permission, so allow tools explicitly; use a repo you trust.
agentops copilot-session launch --repo /path/to/repo --json \
  -- -p "Run the tests and explain any failure" --allow-all-tools > run.json

Choose one way to view the run:

# 3a. Interactive local web UI (foreground server; press Ctrl+C to stop)
agentops ui latest

Or write a static HTML page and open run.html in your browser:

# 3b. Static output (returns to the shell; reads step 2's run.json with Node.js)
agentops copilot-session view "$(node -p 'require("./run.json").sessionId')" \
  --run-id "$(node -p 'require("./run.json").runId')" --output run.html

With --json, stdout is only the run JSON; Copilot's transcript goes to stderr. ui latest opens the run from your most recent launch. See the local web UI.

Prefer a source checkout? git clone the repository and use alias agentops="node $PWD/agentops-cli/src/index.js". Each release lists its tarball's SHA256 in SHA256SUMS. The package is not on the npm registry yet.

Short video overview

21 seconds. Recorded local UI; developer preview. Azure is optional.

Watch on GitHub · Download MP4 · Transcript · Captions · Credits and source notes

There is also a 34-second v0.3.5 release demo; see limitations for what it does and does not show.

What it does

Question	Where you see it
Which tool call failed or was denied, and what came before it?	Local run view, failure detail
Where did the time go?	End-to-end timeline, span durations
How many tokens did each model call use?	Run summary, AgentOpsSpans_CL
What did a run cost?	Estimated cost from public list prices (local UI and digest); EstimatedCostUsd in Azure (nullable)
Did Azure store exactly what was sent?	KQL readback

Each of these was checked against real Copilot CLI runs on 2026-10-07. Costs are estimates, not billed cost. Results, caveats and the full list of known limits are in the docs, and the CLI E2E walkthrough shows the evidence.

Architecture

Copilot CLI emits native OpenTelemetry. A loopback-only, strict-privacy Collector receives it and AgentOps writes a local run ledger. Azure is optional: publishing is off by default (a 0-bytes-per-day cap) and goes through a Data Collection Rule into Log Analytics. You then investigate with KQL, an Azure Workbook or Grafana. Details: architecture and the full component map. Diagram source: docs/diagrams/agentops-cli-architecture.html.

Is this the right tool? Azure Monitor and Aspire already cover parts of this; see when to use AgentOps and limitations.

Optional: Azure upload

To publish to your own Azure workspace, click Deploy to Azure (metadata-only Log Analytics, ingestion endpoint and rule, Workbook and a monthly cost alert at a budget of 5 units of your subscription's billing currency by default, set with monthlyBudgetAmount; it alerts but does not stop spending; see Deploy to Azure) or follow the diagnostic pilot quickstart. Explicitly opt into upload RBAC and public network access, or configure a separately authorized sender and private connectivity. Then add --upload --yes with an explicit daily byte cap (AGENTOPS_MAX_PUBLISH_BYTES_PER_DAY), and check storage with the KQL query library.

Privacy by default

• Metadata only. Span names, timings, token counts, model names and outcomes. Repository identifiers are hashed in the ledger and in anything uploaded to Azure. The local UI, which never leaves your machine, shows your own repository folder names.
• Not recorded by default. Prompts, responses, source code, tool arguments and tool results. Content capture needs an explicit, separate opt-in and is meant for synthetic evaluation data only.
• Fail-closed cloud writes. Uploads need an explicit target, an approved subscription allowlist and a non-zero daily byte cap. The repository contains no subscription IDs or secrets.

See privacy modes, secure by default and the threat model.

Known limits

AgentOps is not a hosted service, a governance platform or a security boundary, and hooks, benchmarks and advanced dashboards are experimental. Cost is an estimate, and token totals must come from chat spans. See limitations for the full list.

Learn more

• Local web UI: runs list, span waterfall, tool latency and estimated cost
• agentops doctor: a green/red checklist of your whole setup, with a fix for each step
• Operator guide: all commands, Collector modes, Azure setup, plugin and removal
• E2E validation: CLI and VS Code walkthroughs with screenshots
• Weekly digest: agentops digest failure clusters, slow tools, tokens and trends
• VS Code native capture extension
• Deploy to Azure: one-click, metadata-only workspace with a budget alert
• Portable Grafana dashboard: import the one file grafana/agentops-copilot-cli.json into the Azure portal, Azure Managed Grafana or self-hosted Grafana. The other dashboard JSONs are legacy or advanced; see grafana/README.md
• Azure Workbook, Grafana tour and KQL query library
• OpenTelemetry GenAI export: send sessions to the App Insights Agents (preview) view
• Telemetry schema and the documentation index
• Changelog

Licensed under the MIT License.