Stagecraft: Node.js skill library and agent runtime

[stevez]

Summary

Split the project into two focused products:

• Dramaturg (Chrome extension) — human-driven: recorder, locator picker, console, code gen, browser runtime
• Stagecraft (Node.js) — agent-driven: skills library, web dashboard, CLI, agent connections, MCP server

Stagecraft connects to Dramaturg via the existing WebSocket bridge when it needs browser access. Dramaturg stays focused as the browser runtime; Stagecraft is the intelligence layer on top.

Architecture

Stagecraft (Node.js)              Dramaturg (Chrome extension)
┌─────────────────────┐           ┌──────────────────────┐
│  Skills library     │           │  Recorder            │
│  CLI tool           │    WS     │  Locator picker      │
│  Web dashboard      │◄────────►│  Console             │
│  Agent connections  │  bridge   │  Code gen            │
│  MCP server         │           │  Browser runtime     │
└─────────────────────┘           └──────────────────────┘

Key Differentiator: JavaScript Execution

Most browser automation MCP tools only expose keyword commands — the agent is limited to one tool call per action (click, fill, snapshot), round-tripping through MCP for every step.

Our platform lets the agent write and execute arbitrary Playwright JS directly in the browser via run_script. This is fundamentally more powerful:

Keywords/tools approach:              JS approach:
run_command("goto ...")               run_script(`
run_command("snapshot")                 await page.goto(...);
run_command("click e5")                 await page.getByText('Save PDF').click();
run_command("snapshot")                 const dl = await page.waitForEvent('download');
run_command("click e12")                console.log(dl.suggestedFilename());
  → 5 round-trips, agent blind        `)  → 1 round-trip, full control
  between calls                       

What this enables:

• Loops — check 5 checkboxes in a for loop, not 5 tool calls
• Downloads — await page.waitForEvent('download') returns the filename
• Error handling — try/catch inline, not fail-and-retry via MCP
• Complex logic — conditionals, DOM queries, data extraction in one script
• Speed — one round-trip instead of many

.pw is the authoring format (record → produce .pw for prototyping). .js is the runtime format (what agents actually execute). The agent or tooling converts .pw → .js when the skill needs logic beyond simple replay.

Use Case: T2 Tax Return

Automate repeated tax prep work: download bills from various websites, extract into Excel, compile for filing. Skills handle browser navigation; agent handles everything else.

```
.pw skills (browser navigation): Agent responsibilities (everything else):
├── navigate-to-rogers-bill.pw ├── Handle file downloads
├── navigate-to-bell-bill.pw ├── Read downloaded PDFs
├── navigate-to-hydro-bill.pw ├── Extract fields (date, amount, vendor, category)
└── ... ├── Compile into Excel
└── Categorize for T2 filing
```

Skills Structure

Each skill = `SKILL.md` (metadata) + `.pw` recipe + optional `.js` for complex logic.

```
packages/stagecraft/skills/
└── rogers/
├── SKILL.md # metadata, parameters, preconditions
├── download-bill.pw # simple template with {{variables}}
└── download-bill.js # full Playwright JS for complex logic
```

SKILL.md Format

```markdown
name: download-rogers-bill
description: Navigate to MyRogers and download bill PDFs
category: tax/bills/telecom
preconditions: Must be logged into rogers.com (use bridge mode)
parameters:

• billing_periods: list of dates to check
  output: PDF files downloaded to browser's default download folder
  ```

.pw vs .js skills

• `.pw` — simple, replayable, parameterized with `{{variables}}`. No loops or conditionals.
• `.js` — full Playwright API, loops, conditionals, complex logic. Runs in extension via serviceWorker.evaluate().
• Both live in the skills directory, SKILL.md describes which to use.

Template Variables

`.pw` files support `{{variable}}` placeholders, substituted at replay time:

```pw
goto https://www.rogers.com/consumer/self-serve/overview
click "View your bill" --exact
click "Save PDF"
check "{{billing_period}}"
click "Download bills"
```

```bash
pw-cli --replay skills/rogers/download-bill.pw --variable billing_period="February 24, 2026"
```

• Simple string replacement — no loops, no conditionals (intentional)
• Multiple variables: `--variable a="x" --variable b="y"`
• Missing variables throw clear error
• For complex logic, use `.js` skills

Design Decisions

Authentication

Bridge mode (primary) — connect to user's real Chrome session. User is already logged in. Agent runs skills against existing cookies/sessions.

Consumption by Agents

No new MCP tools needed. Agent uses existing `run_command` for everything. Agent knowledge lives in skill files / agent.md — instructions, not code.

Recording from AI Side

Agent navigates via MCP commands. Recorder converts ref IDs to stable text locators using snapshot data at record time. Both CLI and MCP share recording infrastructure via `packages/core`.

Ref-to-Locator Conversion

Snapshot maps every ref to accessible name + role. Recorder intercepts each command, looks up ref in snapshot, writes stable locator. Must happen at record time — mapping only exists during session.

Implementation Phases

Phase 1: Recording Commands (not started)

• Add `start-recording` / `stop-recording` as .pw commands in `packages/core`
• Move `SessionRecorder` from `packages/cli` to `packages/core`
• Add ref-to-locator resolution using snapshot data

Phase 2: Prove the Loop (not started)

• Write skill file (agent.md) with recording instructions
• Have Cowork record 2-3 bill navigation flows via MCP
• Verify output .pw files use stable text locators
• Replay and confirm they work across sessions

Phase 3: Skill Library (not started)

• Define SKILL.md schema
• CLI `stagecraft list` / `stagecraft run `

Phase 4: Dashboard (future)

• Web dashboard: execution log, skills catalog
• Error handling: session expired, element not found, retry logic

Completed (2026-04-26)

• `packages/stagecraft` package created with skills directory (feat: stagecraft skills package and pw-cli --replay with template variables #842)
• Rogers bill download skill (`.pw` + `.js` + `SKILL.md`)
• `pw-cli --replay` with `--variable key=value` template substitution (feat: stagecraft skills package and pw-cli --replay with template variables #842)
• `.replay --variable` support in REPL (feat: stagecraft skills package and pw-cli --replay with template variables #842)
• Pick locator fix for elements with structural children (fix: pick locator uses correct role and full name for elements with children (#843) #844)
• Strict mode visibility filter for hidden elements (fix: filter hidden elements to prevent strict mode violations #841)
• pw-cli screenshot/pdf saves file and prints path (fix: pw-cli screenshot/pdf saves file and prints path #840)

Known Issues

1. CLI --replay conflicts with running bridge

`playwright-repl --replay --bridge` starts a new bridge that crashes the existing MCP server. Should support `--http` mode to reuse the running server (like `pw-cli --replay` does).

2. Download file path not visible to agent

`.pw` scripts can click "Download bills" but the agent doesn't know where the PDF landed. Need a way to track browser downloads and return the file path.

3. Recording from AI/MCP side not implemented

`start-recording` / `stop-recording` commands not implemented yet. The recorder should convert ref IDs to stable text locators using snapshot data at record time.

4. Responsive layout breaks scripts

Rogers shows "Save PDF" at desktop width but "Save or download bill" when the side panel narrows the viewport. Scripts need resilience to responsive layout changes.

5. Custom checkbox accessibility

Rogers' design system uses `cdk-visually-hidden` native checkboxes with styled div overlays. `getByRole('checkbox')` returns 0. The visibility filter (#841) works around the label/input duplicate, but the root cause is inaccessible design system components.

Open Questions

• Where do skills live on disk? Per-project? Global `~/.stagecraft/skills/`?
• How does output pass between skills? (e.g., downloaded PDF path → extract step)
• How to handle sites that require 2FA during automation?
• How much intelligence goes in the skill vs. the agent?

6. Control download filename and location

When downloading bills, the browser saves with its default filename to the default folder. The agent needs to:

• Rename the downloaded file (e.g. rogers-bill-2026-03.pdf)
• Save to a specific directory (e.g. ~/tax/2026/bills/)

Playwright supports this via page.on('download') + download.saveAs(path). Need to investigate if this works through the extension in bridge mode, or if it requires a new .pw command like download-to <path>.