feat(user-testing): agent tools address environments and scenarios

[chelojimenez]

Why

ui_publish_chatbox still took a client and provisioned its host chatbox — the exact kind of row #3765 stopped showing. An agent asked to "create a scenario" would mint one, and it wouldn't appear in the list. The tools were the last place the pre-environments model survived.

Re-pointed, not duplicated

Chrome's WebMCP guidance is explicit that overlapping tools confuse the agent, and the catalog re-advertises on every chat POST — so the new semantics are picked up next turn without a host-addressed twin lingering beside them.

ui_publish_chatbox → publishes an environment

{ environment, access?, name? }, resolved by the name the Environments screen shows or by id.

• access and name ride in the same mutation as the publish (Update OAuth debugger documentation for PR #867 refactor #887), so a scenario is never briefly live in a mode nobody asked for.
• Re-publishing reports created: false with the existing name and access, rather than claiming it made something. The model repeats whatever we tell it.
• With Environments off it refuses as unsupported_in_mode instead of falling back to minting a client.

ui_delete_chatbox → deletes a scenario

{ scenario }, resolved against the same filtered rows the snapshot advertises — an agent can only delete something the user can also see. A row the list filters out is not addressable.

Its result states that the environment behind the scenario is untouched, so the model doesn't report more damage than was done.

Following the WebMCP guidance

• Semantic values, not internal identifiers — both tools take the name a human reads on screen; ids are accepted for disambiguation.
• Validate strictly in code, loosely in schema — validation lives in execute with descriptive errors, so an unknown access names the accepted values rather than just failing, and the model can self-correct.
• Refuse rather than guess — resolution is exact; unknown and ambiguous both refuse and say how to disambiguate, because publishing or deleting the wrong thing is silent when resolution guesses.
• Annotations unchanged and still honest: publish is idempotent/non-destructive, delete is destructive (approval pill).

Snapshot

Now advertises the project's environments with a published flag. Exact resolution refuses a name it can't match, so without this the publish tool's own input would be discoverable nowhere. Still redacted: no share token, no transcript text, no visitor PII — asserted.

Also

chatboxes:ensureChatboxForHost now has no caller on this surface.

Tests

UserTestingTab.agent.test.tsx rewritten around the new addressing (11 cases): environment publish with access in one call; the created: false honesty case; the Environments-off refusal; exact + ambiguous resolution; scenario-addressed delete that leaves the environment alone; the "can only reach rows the list advertises" case; and the new snapshot assertion.

groups/__tests__/chatboxes.test.ts covers the tool contract itself, including that absent optionals are omitted rather than sent as empty strings (an empty name would become the scenario's name).

npx vitest run client/src/components/__tests__ client/src/lib/webmcp client/src/components/chatboxes
# 94 files, 1046 tests passed

npx vitest run client/src/components/chatboxes client/src/components/__tests__ client/src/lib \
  client/src/__tests__/ChatboxesRoute.billing.test.tsx client/src/components/project-environments/__tests__ client/src/hooks
# 332 files, 3617 tests passed

vite build --config client/vite.config.ts    # ✓ built

Note on formatting: UserTestingTab.agent.test.tsx doesn't satisfy prettier --trailing-comma all — it already didn't on main, so I left its pre-existing lines alone rather than reformatting a file this PR only partly touches. Every other file here was conformant and stayed conformant.

Last of the User Testing environment-first program (#3761 identity, #3765 create + list, this one the agent surface).

🤖 Generated with Claude Code

---

Note

^{Cursor Bugbot is generating a summary for commit 070b295. Configure here.}

---

Summary by cubic

Repointed User Testing agent tools to environments and scenarios. Publish now targets an environment and delete targets a scenario, fixing invisible rows and aligning with the environment-first model.

• New Features

  • ui_publish_chatbox publishes an environment: { environment, access?, name? }. Resolves by on-screen name or id, applies access/name in the same mutation, returns created, and opens the scenario. Refuses with unsupported_in_mode when Environments is off.
  • ui_delete_chatbox deletes a scenario: { scenario }. Resolves against the same filtered list the snapshot shows, deletes only the scenario, and leaves its environment untouched.
  • Snapshot now lists environments with published to make publish inputs discoverable. Redactions unchanged (no share token, transcript text, or visitor PII).
  • Exact resolution with descriptive errors; no fuzzy guesses. Removed host back-minting; no callers of chatboxes:ensureChatboxForHost. Tests updated.

• Migration

  • Update tool payloads: publish uses environment (plus optional access, name); delete uses scenario. Replace any host-based calls.
  • Publishing via tool needs Environments enabled; otherwise, create scenarios from the New scenario screen.

^{Written for commit 070b295. Summary will update on new commits.}

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_215edbf5-1c8c-472e-b06e-6e2168804a17)

[chelojimenez]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[github-actions]

Internal preview

Preview URL will appear in Railway after the deploy finishes.
Deployed commit: 3efde70
PR head commit: 070b295
Backend target: staging fallback.
Access is employee-only in non-production environments.

[coderabbitai]

Walkthrough

User Testing replaces host-based scenario provisioning with environment-based publishing. Agents can resolve environments and scenarios by ID or case-insensitive name, publish scenarios with optional access and name settings, and delete visible scenarios without changing their environments. Agent snapshots now include environment metadata and publication status. WebMCP commands, shared payloads, and tests cover validation, feature gating, idempotent publishing, deletion, authentication, and environment discovery.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

mcpjam-inspector/client/src/lib/webmcp/groups/chatboxes.ts (1)

35-36: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Consider deriving ACCESS_VALUES from the shared preset type.

The same three access values now live in three places: ACCESS_VALUES here, the access union in mcpjam-inspector/shared/inspector-command.ts (Line 593), and the ChatboxAccessPreset cases in settingsFromChatboxAccessPreset. All three agree today. A future fourth preset must be added in all three, and only the runtime check here fails loudly if one is missed.

If chatbox-access-presets.ts exports the preset union, type ACCESS_VALUES against it so the compiler enforces the set.

♻️ Illustrative shape

-const ACCESS_VALUES = ["invited_only", "link_guests", "project"] as const;
-type AccessValue = (typeof ACCESS_VALUES)[number];
+import type { ChatboxAccessPreset } from "`@/lib/chatbox-access-presets`";
+
+const ACCESS_VALUES = [
+  "invited_only",
+  "link_guests",
+  "project",
+] as const satisfies readonly ChatboxAccessPreset[];
+type AccessValue = (typeof ACCESS_VALUES)[number];

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@mcpjam-inspector/client/src/lib/webmcp/groups/chatboxes.ts` around lines 35 -
36, Update ACCESS_VALUES and its AccessValue alias to derive from the shared
ChatboxAccessPreset union exported by chatbox-access-presets.ts, rather than
maintaining a separate access union. Ensure the shared preset type remains the
single source of truth while preserving the existing runtime values and
validation behavior.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@mcpjam-inspector/client/src/lib/webmcp/groups/chatboxes.ts`:
- Around line 35-36: Update ACCESS_VALUES and its AccessValue alias to derive
from the shared ChatboxAccessPreset union exported by chatbox-access-presets.ts,
rather than maintaining a separate access union. Ensure the shared preset type
remains the single source of truth while preserving the existing runtime values
and validation behavior.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: e9ed9a68-c937-4f8a-85d0-613e9953f4ce

📥 Commits

Reviewing files that changed from the base of the PR and between b61d9ac and 070b295.

📒 Files selected for processing (7)

• mcpjam-inspector/client/src/components/UserTestingTab.tsx
• mcpjam-inspector/client/src/components/__tests__/UserTestingTab.agent.test.tsx
• mcpjam-inspector/client/src/components/__tests__/UserTestingTab.journeys.test.tsx
• mcpjam-inspector/client/src/components/__tests__/UserTestingTab.scenario-list.test.tsx
• mcpjam-inspector/client/src/lib/webmcp/groups/__tests__/chatboxes.test.ts
• mcpjam-inspector/client/src/lib/webmcp/groups/chatboxes.ts
• mcpjam-inspector/shared/inspector-command.ts