feat(user-testing): create a scenario by publishing an environment - #3765
Conversation
A scenario is an environment shared with real people. The create flow asked for a server and a client template instead, and quietly called `hosts.createHost` — which mints a chatbox per host. Combined with the two seeds that run before anyone opens the tab (the Playground seeds three clients into an empty project, the host bar seeds "MCPJam"), a brand-new project listed four or five "scenarios" nobody had created. The empty-state copy even advertised it: "Every client you set up in Connect is published as one." Two changes, both flag-gated on `project-environments-enabled`: **Create publishes an environment.** Pick one, name it, choose who can open it — the environment already carries the client, servers, skills and sandbox image a tester will meet, so the flow asks only for what it does NOT carry. One write: `publishEnvironmentChatbox` applies the name and the access mode in the same mutation (mcpjam-backend #887), so a scenario is never briefly live in a mode nobody asked for. Re-publishing an already-published environment reports that and opens the existing scenario rather than failing. It never CREATES an environment — scenario surfaces select, only Swarms materializes. "New environment" hands off to the Environments editor with the typed name seeded, via the same sessionStorage handoff Connect uses. **The list shows scenarios, not clients.** `isDeliberateScenario` keeps rows that could only exist on purpose: environment-backed ones, rows whose access mode was changed off the auto-mint default, and rows with real tester history. Absent counters are read as no evidence rather than as zero, so a deployment predating them doesn't resurrect every seeded client. Filtered client-side deliberately: the same query feeds public API v1 and the Environments page's consumer counts, and neither should inherit this surface's editorial rule. The resolution ladder still searches the unfiltered rows — hiding a row says "not worth advertising", never "gone", so a direct link to one still opens. Flag-off projects keep the legacy flow and the unfiltered list wholesale: without environments there is no other kind of scenario, so filtering would leave them a surface they cannot use. Both legacy branches get deleted when the flag retires. Also restores `useEnvironmentChatbox` / `useEnvironmentChatboxMutations` (removed in #3658 with the old Environments publish panel) and adds a one-line pointer from an environment to the scenario it's published as. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
GitHub Actions
Error
dosubot
Bot
added
size:XL
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_3a1b8383-de5a-4c59-8108-8763736294e8) |
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
WalkthroughAdds environment-backed user testing scenarios. The change introduces shared access presets, deliberate-scenario detection, environment chatbox lookup, and publish mutations. It adds a scenario creation flow with environment selection, naming, access control, validation, and environment creation handoff. Environment-enabled tabs filter scenario rows while preserving direct-link resolution. Environment detail views link to published scenarios. Tests cover creation, filtering, publishing behavior, and route integration. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (4)
mcpjam-inspector/client/src/lib/chatbox-access-presets.ts (1)
22-49: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueReconcile the stated order with the actual exposure ranking.
The comment promises "least- to most-exposed", yet
link_guests(anyone, including account-less guests) precedesproject(signed-in project members only). By any reasonable reading, project members are the narrower audience. Either moveprojectahead oflink_guests, or reword the comment to describe the intended presentation order.♻️ Proposed reordering
{ value: "invited_only", label: "Invited users only", description: "Only people you invite by email can open this scenario.", }, { + value: "project", + label: "Project members", + description: + "Signed-in members of this project can open it with the link. Guests cannot.", + }, + { value: "link_guests", label: "Anyone with the link", description: "Anyone with the link can open it, including guests without an account. Guest usage runs on your organization's credits.", }, - { - value: "project", - label: "Project members", - description: - "Signed-in members of this project can open it with the link. Guests cannot.", - }, ];🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@mcpjam-inspector/client/src/lib/chatbox-access-presets.ts` around lines 22 - 49, Update CHATBOX_ACCESS_OPTIONS so its entries match the documented least-to-most-exposed ordering by placing the project preset before link_guests, while preserving each option’s existing labels, descriptions, and values.mcpjam-inspector/client/src/hooks/useChatboxes.ts (1)
211-214: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value
undefinedcarries two meanings here, and only one is documented.The doc states that
undefinedmeans "the list is still loading". Two other paths produce it:environmentId === null, and a skipped query (signed out, or no project id), whereuseChatboxListreportsisLoading: falseandchatboxes: undefined. A consumer that renders a spinner onchatbox === undefinedwould then spin forever.ProjectEnvironmentsRoute.tsxonly tests for truthiness, so nothing breaks today, but the contract invites the mistake.Consider returning
nullwhenenvironmentIdis absent, and letisLoadingalone gate the loading state.♻️ Proposed clarification
const chatbox = useMemo(() => { - if (!environmentId || chatboxes === undefined) return undefined; + if (!environmentId) return null; + if (chatboxes === undefined) return undefined; return chatboxes.find((row) => row.environmentId === environmentId) ?? null; }, [chatboxes, environmentId]);🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@mcpjam-inspector/client/src/hooks/useChatboxes.ts` around lines 211 - 214, Update the chatbox state derivation around useMemo so an absent environmentId returns null rather than undefined, reserving undefined for the chatboxes === undefined loading state. Keep the existing lookup and null fallback for loaded lists, and ensure consumers use the hook’s isLoading value to gate loading behavior.mcpjam-inspector/client/src/components/chatboxes/UserTestingScenarioCreateFlow.tsx (1)
236-251: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winThe access control announces its value but not its purpose.
<Label>Who can open it</Label>at line 237 has nohtmlFor, and the trigger at line 240 is a bare<button>. Assistive technology therefore reads only "Invited users only" with no indication of what that setting governs. The environment picker above it already carriestriggerAriaLabel; the same courtesy here costs one attribute.♿ Proposed fix
<button type="button" data-testid="user-testing-create-access" + aria-label="Who can open this scenario" disabled={isSaving}🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@mcpjam-inspector/client/src/components/chatboxes/UserTestingScenarioCreateFlow.tsx` around lines 236 - 251, Associate the “Who can open it” label with the access dropdown trigger in UserTestingScenarioCreateFlow by giving the button a matching identifier and adding htmlFor to the Label, or by providing an equivalent accessible name that includes the setting’s purpose. Preserve the existing accessLabel value and trigger behavior.mcpjam-inspector/client/src/components/chatboxes/__tests__/UserTestingScenarioCreateFlow.test.tsx (1)
104-123: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueTwo behaviours the suite describes but does not yet pin.
The header promises coverage of the access mode carried in the publish call, and the component invests a
savingRefin preventing a double publish. Both go unverified:
- Only the
invited_onlydefault is asserted. A regression that ignoredaccessPresetentirely would pass.- Two rapid clicks on Save should still yield one
onCreateScenariocall.The second is straightforward with the existing harness; the first needs the Radix dropdown driven or stubbed. Would you like me to draft both cases?
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@mcpjam-inspector/client/src/components/chatboxes/__tests__/UserTestingScenarioCreateFlow.test.tsx` around lines 104 - 123, Expand the tests around renderFlow and the Save interaction to cover both access modes: drive or stub the Radix accessPreset dropdown and assert the selected mode is included in the single onCreateScenario payload, not only the invited_only default. Add a rapid double-click Save case and verify savingRef-backed behavior still results in exactly one onCreateScenario call.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@mcpjam-inspector/client/src/components/chatboxes/UserTestingOverviewPanel.tsx`:
- Around line 230-232: Update the UserTestingOverviewPanel description so it is
supplied through the existing parent-to-child configuration alongside
createLabel, and select wording that matches whether
project-environments-enabled is active or the legacy host-based flow is used.
Ensure flag-off projects describe their server and client-template setup instead
of referring to environments.
In
`@mcpjam-inspector/client/src/components/chatboxes/UserTestingScenarioCreateFlow.tsx`:
- Around line 225-228: Update the name onChange handler in
UserTestingScenarioCreateFlow so userEditedNameRef.current is set based on the
trimmed input length, matching the sibling UserTestingCreateFlow behavior; treat
an empty or whitespace-only value as not user-edited while preserving the
entered value in setName.
---
Nitpick comments:
In
`@mcpjam-inspector/client/src/components/chatboxes/__tests__/UserTestingScenarioCreateFlow.test.tsx`:
- Around line 104-123: Expand the tests around renderFlow and the Save
interaction to cover both access modes: drive or stub the Radix accessPreset
dropdown and assert the selected mode is included in the single onCreateScenario
payload, not only the invited_only default. Add a rapid double-click Save case
and verify savingRef-backed behavior still results in exactly one
onCreateScenario call.
In
`@mcpjam-inspector/client/src/components/chatboxes/UserTestingScenarioCreateFlow.tsx`:
- Around line 236-251: Associate the “Who can open it” label with the access
dropdown trigger in UserTestingScenarioCreateFlow by giving the button a
matching identifier and adding htmlFor to the Label, or by providing an
equivalent accessible name that includes the setting’s purpose. Preserve the
existing accessLabel value and trigger behavior.
In `@mcpjam-inspector/client/src/hooks/useChatboxes.ts`:
- Around line 211-214: Update the chatbox state derivation around useMemo so an
absent environmentId returns null rather than undefined, reserving undefined for
the chatboxes === undefined loading state. Keep the existing lookup and null
fallback for loaded lists, and ensure consumers use the hook’s isLoading value
to gate loading behavior.
In `@mcpjam-inspector/client/src/lib/chatbox-access-presets.ts`:
- Around line 22-49: Update CHATBOX_ACCESS_OPTIONS so its entries match the
documented least-to-most-exposed ordering by placing the project preset before
link_guests, while preserving each option’s existing labels, descriptions, and
values.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: c80963ce-ed07-4092-a264-84d09eb5986c
📒 Files selected for processing (17)
mcpjam-inspector/client/src/components/UserTestingTab.tsxmcpjam-inspector/client/src/components/__tests__/UserTestingTab.agent.test.tsxmcpjam-inspector/client/src/components/__tests__/UserTestingTab.journeys.test.tsxmcpjam-inspector/client/src/components/__tests__/UserTestingTab.scenario-list.test.tsxmcpjam-inspector/client/src/components/chatboxes/UserTestingCreateFlow.tsxmcpjam-inspector/client/src/components/chatboxes/UserTestingOverviewPanel.tsxmcpjam-inspector/client/src/components/chatboxes/UserTestingScenarioCreateFlow.tsxmcpjam-inspector/client/src/components/chatboxes/__tests__/UserTestingScenarioCreateFlow.test.tsxmcpjam-inspector/client/src/components/project-environments/ProjectEnvironmentsRoute.tsxmcpjam-inspector/client/src/components/project-environments/__tests__/archive-consumer-counts.test.tsxmcpjam-inspector/client/src/components/project-environments/__tests__/project-environments-route.flag-gate.test.tsxmcpjam-inspector/client/src/components/project-environments/__tests__/project-environments-route.seed.test.tsxmcpjam-inspector/client/src/components/project-environments/__tests__/project-environments-route.tentative-drafts.test.tsxmcpjam-inspector/client/src/hooks/useChatboxes.tsmcpjam-inspector/client/src/lib/__tests__/user-testing-scenarios.test.tsmcpjam-inspector/client/src/lib/chatbox-access-presets.tsmcpjam-inspector/client/src/lib/user-testing-scenarios.ts
| A scenario is one of your environments — its client, servers and skills | ||
| — behind a link you can hand to a real person, so you can read what | ||
| happened in their sessions. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
This copy speaks of environments to projects that have none.
UserTestingTab chooses the creation flow by the project-environments-enabled flag, and the legacy host-based flow asks for a server and a client template — no environments in sight. This empty state, however, is unconditional. A flag-off project therefore reads "one of your environments" and then meets a form that never mentions one. The createLabel prop is already injected by the parent; the description could travel the same route.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@mcpjam-inspector/client/src/components/chatboxes/UserTestingOverviewPanel.tsx`
around lines 230 - 232, Update the UserTestingOverviewPanel description so it is
supplied through the existing parent-to-child configuration alongside
createLabel, and select wording that matches whether
project-environments-enabled is active or the legacy host-based flow is used.
Ensure flag-off projects describe their server and client-template setup instead
of referring to environments.
| onChange={(e) => { | ||
| userEditedNameRef.current = true; | ||
| setName(e.target.value); | ||
| }} |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Clearing the field permanently severs the name from the environment.
userEditedNameRef.current becomes true on any change, including a change to the empty string. A user who selects all and deletes has then "edited" the name; every subsequent environment pick leaves the field blank and Save disabled until they type something. The sibling flow guards against exactly this at UserTestingCreateFlow.tsx line 206, using the trimmed length.
🐛 Proposed fix
onChange={(e) => {
- userEditedNameRef.current = true;
+ userEditedNameRef.current = e.target.value.trim().length > 0;
setName(e.target.value);
}}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| onChange={(e) => { | |
| userEditedNameRef.current = true; | |
| setName(e.target.value); | |
| }} | |
| onChange={(e) => { | |
| userEditedNameRef.current = e.target.value.trim().length > 0; | |
| setName(e.target.value); | |
| }} |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@mcpjam-inspector/client/src/components/chatboxes/UserTestingScenarioCreateFlow.tsx`
around lines 225 - 228, Update the name onChange handler in
UserTestingScenarioCreateFlow so userEditedNameRef.current is set based on the
trimmed input length, matching the sibling UserTestingCreateFlow behavior; treat
an empty or whitespace-only value as not user-edited while preserving the
entered value in setName.
1 participant
Why
This is the PR that fixes the reported bug. A fresh project listed four or five "scenarios" nobody created —
Claude Code,MCPJam,ChatGPT,Cursor (CLI),Copilot, all with 0 testers.They were there because a chatbox row is minted 1:1 with every host, two seeds run before anyone opens the tab (the Playground seeds three clients into an empty project; the host bar seeds "MCPJam"), and the list rendered every row. The empty-state copy even advertised it: "Every client you set up in Connect is published as one."
The deeper cause: the create flow asked for a server and a client template and called
hosts.createHost. But a scenario is an environment shared with real people.What changed
Both changes are gated on
project-environments-enabled.Create publishes an environment
Pick an environment, name it, choose who can open it. The environment already carries the client, servers, skills and sandbox image a tester meets, so the flow asks only for what it does not carry.
One write.
publishEnvironmentChatboxapplies the name and access mode in the same mutation (mcpjam-backend #887), so a scenario is never briefly live in a mode nobody asked for. Re-publishing an already-published environment reports that and opens the existing scenario rather than erroring — and, per #887, cannot silently re-mode it.It never creates an environment. Scenario surfaces select; only Swarms materializes (the ad-hoc environments rule). "New environment" hands off to the Environments editor with the typed name seeded, through the same sessionStorage handoff Connect already uses.
The list shows scenarios, not clients
isDeliberateScenario(client/src/lib/user-testing-scenarios.ts) keeps rows that could only exist on purpose:project_members);Absent counters are read as no evidence, not as zero — otherwise a deployment predating them resurrects every seeded client.
Two deliberate scoping calls:
GET /v1/chatboxes) and the Environments page's consumer counts; neither should inherit this surface's editorial rule.Flag-off projects are untouched — legacy create flow, unfiltered list. Without environments there is no other kind of scenario, so filtering would leave them a surface they can't use. Both legacy branches get deleted when the flag retires.
Also
useEnvironmentChatbox/useEnvironmentChatboxMutations, removed in feat(environments): remove publish-as-chatbox panel from environment … #3658 along with the old Environments publish panel.ACCESS_OPTIONSmoved tolib/chatbox-access-presets.tsso both flows read the same wording.Known limit
A legacy host-backed scenario left at the default access mode whose sessions all predate the counters won't be listed. It stays reachable by its link and through v1 — the alternative is showing every client forever. Stated rather than hidden.
Tests
New:
user-testing-scenarios.test.ts(the filter matrix, including the absent-vs-zero distinction) andUserTestingScenarioCreateFlow.test.tsx(nothing written until Save; one call carrying name + mode; least-exposed default; name tracks the environment until you type; already-published reported as such; admin refusal surfaced verbatim; the handoff seeds the name).UserTestingTab.scenario-list.test.tsxreproduces the exact reported lineup and asserts the phantom rows are gone, the empty state appears instead, a used legacy row survives, a filtered-out row still opens by link, and flag-off doesn't filter.Two bugs in my own work were caught by these tests while writing them: the list rendered the unfiltered query, and (in #3761) a deleted mutation binding.
🤖 Generated with Claude Code
Note
Medium Risk
Changes scenario discovery and creation behind a feature flag, including access-mode defaults and admin-gated publish mutations; list filtering could hide edge-case legacy scenarios still reachable by link/API.
Overview
Fixes fresh projects showing phantom “scenarios” from every auto-minted client chatbox by client-side filtering with
isDeliberateScenariowhenproject-environments-enabledis on—environment publishes, non-default access, or real tester history stay visible; resolution by URL still searches the unfiltered list. Flag-off projects keep the legacy create flow and unfiltered list.Create path (flag on):
/user-testing/newusesUserTestingScenarioCreateFlow—pick an environment, name, and access—then a singlepublishEnvironmentChatboxcall (name + mode together; idempotent re-publish opens the existing scenario). “New environment” seeds the Environments editor via sessionStorage; scenarios never create environments here.Also restores
useEnvironmentChatbox/useEnvironmentChatboxMutations, centralizesCHATBOX_ACCESS_OPTIONS, updates empty-state copy, and adds an environment detail link to the published User Testing scenario when one exists.Reviewed by Cursor Bugbot for commit 2eb10e3. Bugbot is set up for automated code reviews on this repo. Configure here.
Summary by cubic
Create User Testing scenarios by publishing an environment, and hide auto-minted client rows so new projects no longer show phantom scenarios.
New Features
useEnvironmentChatboxanduseEnvironmentChatboxMutations; environment detail now links to its published scenario.lib/chatbox-access-presets.ts.project-environments-enabled.Bug Fixes
isDeliberateScenario: environment-backed, non-default access, or real tester activity. Direct links still work, and projects without environments keep the legacy, unfiltered list.Written for commit 2eb10e3. Summary will update on new commits.