test(mock-llm): add LLM profile lifecycle + folder→workspace E2E tests

[malhotra5]

• A human has tested these changes.

---

Why

Issue #511 identifies multiple fully-implemented features with zero test coverage. This PR adds mock-LLM E2E tests that exercise two coverage areas against the real agent-server.

Summary

1. LLM Profile lifecycle (mock-llm-profiles.spec.ts) — 4 serial tests

Step	What it tests	Key assertions
Create	Two profiles via Settings UI	Both appear in list + GET /api/profiles confirms server-side
Activate	Activate profile-beta	Badge moves to beta in UI + API active_profile === "profile-beta"
Rename active	Rename active profile	Badge follows new name, old name gone, API confirms new active_profile
Delete active	Delete the active profile	No badge on remaining profile, API returns active_profile: null

2. Folder browser → workspace → conversation (mock-llm-folder-workspace.spec.ts) — 1 test

Covers two "I can" statements:

• "I can browse local files and folders to choose where to begin"
• "I can start a conversation against a local Git repo without typing the path"

Flow: Open folder browser → navigate /tmp/e2e-folder-workspace-test/my-test-project via real filesystem → "Use this folder" → select in dropdown → confirm → type message → submit → verify:

• POST /api/conversations payload has workspace.working_dir === "/tmp/e2e-folder-workspace-test/my-test-project"
• selected_workspace persisted in localStorage under the conversation metadata key

Issue Number

#511

How to Test

npm run build:app  # if build/ is missing
npm run test:e2e:mock-llm

Type

• Bug fix
• Feature
• Refactor
• Breaking change
• Docs / chore

This PR was created by an AI agent (OpenHands) on behalf of the user.

---

🐳 Docker images for this PR

• GHCR package: https://github.com/OpenHands/agent-canvas/pkgs/container/agent-canvas

Component	Value
Image	ghcr.io/openhands/agent-canvas
Architectures	amd64, arm64
Agent Server	ghcr.io/openhands/agent-server:1.24.0-python
Automation	openhands-automation==1.0.0a5
Commit	1f5784c11ece56e23873f6bb85a5a9a8fd065faa

Pull (multi-arch manifest)

# Multi-arch manifest — Docker automatically pulls the correct architecture
docker pull ghcr.io/openhands/agent-canvas:sha-1f5784c

Run

docker run -it --rm \
  -p 8000:8000 \
  ghcr.io/openhands/agent-canvas:sha-1f5784c

All tags pushed for this build

ghcr.io/openhands/agent-canvas:sha-1f5784c-amd64
ghcr.io/openhands/agent-canvas:test-llm-profiles-snapshots-511-amd64
ghcr.io/openhands/agent-canvas:pr-946-amd64
ghcr.io/openhands/agent-canvas:sha-1f5784c-arm64
ghcr.io/openhands/agent-canvas:test-llm-profiles-snapshots-511-arm64
ghcr.io/openhands/agent-canvas:pr-946-arm64
ghcr.io/openhands/agent-canvas:sha-1f5784c
ghcr.io/openhands/agent-canvas:test-llm-profiles-snapshots-511
ghcr.io/openhands/agent-canvas:pr-946

About Multi-Architecture Support

• Each tag (e.g., sha-1f5784c) is a multi-arch manifest supporting both amd64 and arm64
• Docker automatically pulls the correct architecture for your platform
• Individual architecture tags (e.g., sha-1f5784c-amd64) are also available if needed

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
agent-canvas	Ready	Preview, Comment	Jun 1, 2026 3:16pm

[github-actions]

❌ Mock-LLM E2E Tests

7/11 passed · 1 failed · 3 skipped

Commit: 25d012c1 · Workflow run · Test artifacts

Status	Test	Duration
✅	mock-llm-automation.spec.ts › mock-LLM automation lifecycle › step 1: setup LLM profile and register automation trajectory	231ms
✅	mock-llm-automation.spec.ts › mock-LLM automation lifecycle › step 2: create automation and dispatch run via the UI	25.5s
✅	mock-llm-automation.spec.ts › mock-LLM automation lifecycle › step 3: verify automation and run on the automations page	4.2s
✅	mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 1: create an LLM profile pointing at the mock LLM server	4.1s
✅	mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 2: activate the mock-llm profile and verify settings API	5.0s
✅	mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 3: run a conversation with the mock LLM	4.4s
✅	mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 4: resume conversation from sidebar after navigating away	3.6s
❌	mock-llm-profiles.spec.ts › mock-LLM profile lifecycle › step 1: create two LLM profiles and verify they appear in the list	8.8s
⏭️	mock-llm-profiles.spec.ts › mock-LLM profile lifecycle › step 2: activate profile-beta and verify the active badge	0ms
⏭️	mock-llm-profiles.spec.ts › mock-LLM profile lifecycle › step 3: rename the active profile and verify badge follows the new name	0ms
⏭️	mock-llm-profiles.spec.ts › mock-LLM profile lifecycle › step 4: delete the active profile and verify no active badge remains	0ms

🔍 Failure details (1)

❌ mock-llm-profiles.spec.ts › mock-LLM profile lifecycle › step 1: create two LLM profiles and verify they appear in the list

Error: expect(locator).toHaveCount(expected) failed

Locator:  getByTestId('profile-row')
Expected: 0
Received: 1
Timeout:  5000ms

Call log:
  - Expect "toHaveCount" with timeout 5000ms
  - waiting for getByTestId('profile-row')
    9 × locator resolved to 1 element
      - unexpected value "1"

_{Posted by the Mock-LLM E2E workflow · results are deterministic (scripted LLM responses)}

[github-actions]

❌ Mock-LLM E2E Tests

7/11 passed · 1 failed · 3 skipped

Commit: 199e1239 · Workflow run · Test artifacts

Status	Test	Duration
✅	mock-llm-automation.spec.ts › mock-LLM automation lifecycle › step 1: setup LLM profile and register automation trajectory	256ms
✅	mock-llm-automation.spec.ts › mock-LLM automation lifecycle › step 2: create automation and dispatch run via the UI	24.5s
✅	mock-llm-automation.spec.ts › mock-LLM automation lifecycle › step 3: verify automation and run on the automations page	4.2s
✅	mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 1: create an LLM profile pointing at the mock LLM server	4.1s
✅	mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 2: activate the mock-llm profile and verify settings API	5.1s
✅	mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 3: run a conversation with the mock LLM	4.4s
✅	mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 4: resume conversation from sidebar after navigating away	3.7s
❌	mock-llm-profiles.spec.ts › mock-LLM profile lifecycle › step 1: create two LLM profiles and verify they appear in the list	8.9s
⏭️	mock-llm-profiles.spec.ts › mock-LLM profile lifecycle › step 2: activate profile-beta and verify the active badge	0ms
⏭️	mock-llm-profiles.spec.ts › mock-LLM profile lifecycle › step 3: rename the active profile and verify badge follows the new name	0ms
⏭️	mock-llm-profiles.spec.ts › mock-LLM profile lifecycle › step 4: delete the active profile and verify no active badge remains	0ms

🔍 Failure details (1)

❌ mock-llm-profiles.spec.ts › mock-LLM profile lifecycle › step 1: create two LLM profiles and verify they appear in the list

Error: expect(locator).toHaveCount(expected) failed

Locator:  getByTestId('profile-row')
Expected: 0
Received: 1
Timeout:  5000ms

Call log:
  - Expect "toHaveCount" with timeout 5000ms
  - waiting for getByTestId('profile-row')
    9 × locator resolved to 1 element
      - unexpected value "1"

_{Posted by the Mock-LLM E2E workflow · results are deterministic (scripted LLM responses)}

[github-actions]

❌ Mock-LLM E2E Tests

7/11 passed · 1 failed · 3 skipped

Commit: b235ecfc · Workflow run · Test artifacts

Status	Test	Duration
✅	mock-llm-automation.spec.ts › mock-LLM automation lifecycle › step 1: setup LLM profile and register automation trajectory	281ms
✅	mock-llm-automation.spec.ts › mock-LLM automation lifecycle › step 2: create automation and dispatch run via the UI	22.4s
✅	mock-llm-automation.spec.ts › mock-LLM automation lifecycle › step 3: verify automation and run on the automations page	4.1s
✅	mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 1: create an LLM profile pointing at the mock LLM server	4.2s
✅	mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 2: activate the mock-llm profile and verify settings API	5.1s
✅	mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 3: run a conversation with the mock LLM	4.4s
✅	mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 4: resume conversation from sidebar after navigating away	3.6s
❌	mock-llm-profiles.spec.ts › mock-LLM profile lifecycle › step 1: create two LLM profiles and verify they appear in the list	8.8s
⏭️	mock-llm-profiles.spec.ts › mock-LLM profile lifecycle › step 2: activate profile-beta and verify the active badge	0ms
⏭️	mock-llm-profiles.spec.ts › mock-LLM profile lifecycle › step 3: rename the active profile and verify badge follows the new name	0ms
⏭️	mock-llm-profiles.spec.ts › mock-LLM profile lifecycle › step 4: delete the active profile and verify no active badge remains	0ms

🔍 Failure details (1)

❌ mock-llm-profiles.spec.ts › mock-LLM profile lifecycle › step 1: create two LLM profiles and verify they appear in the list

Error: expect(locator).toHaveCount(expected) failed

Locator:  getByTestId('profile-row')
Expected: 0
Received: 1
Timeout:  5000ms

Call log:
  - Expect "toHaveCount" with timeout 5000ms
  - waiting for getByTestId('profile-row')
    9 × locator resolved to 1 element
      - unexpected value "1"

_{Posted by the Mock-LLM E2E workflow · results are deterministic (scripted LLM responses)}

[malhotra5]

Mock-LLM E2E Fix — Profile Cleanup

The failing mock-llm-profiles test (step 1: create two LLM profiles and verify they appear in the list) was hitting this assertion:

expect(profileRows).toHaveCount(0)  // Expected: 0, Received: 1

Root Cause

The cleanupProfiles() function only deleted named profiles via DELETE /api/profiles/{name}. However, the automation test's ensureMockLLMProfile() PATCHes /api/settings with agent_settings_diff.llm.*, which persists LLM config at the settings layer. After deleting all named profiles, the agent-server may still surface a "default" profile derived from those lingering settings — hence the 1 unexpected profile row.

The old cleanup also silently swallowed all errors (.catch(() => {})), making it impossible to diagnose whether GET /api/profiles returned a non-OK status (e.g. 401) or DELETE failed.

Fix (dae3e75)

1. Also PATCH /api/settings to clear llm.model, llm.api_key, llm.base_url after deleting named profiles — ensures no implicit "default" profile lingers from the settings layer.
2. Diagnostic logging for non-OK API responses instead of silently swallowing errors — if the GET or DELETE or PATCH returns a non-2xx status, the test now logs a console.warn so the failure reason is visible in CI output.
3. Verification step that re-fetches profiles after cleanup and warns if any remain.
4. Increased timeout for the empty-state assertion to 10s (from the default 5s) for slower CI environments.

Open question

If the settings-level reset via PATCH /api/settings turns out to not fully clear the implicit profile on agent-server 1.24.0, the diagnostic logging will reveal the exact failure mode in the next CI run (e.g. whether GET /api/profiles returns 401, or whether profiles still exist after the cleanup PATCH). That will tell us whether it's an auth issue, an API behavior change, or something else entirely.

---

This comment was generated by an AI agent (OpenHands).

[malhotra5]

Confirmed: GET /api/profiles auto-creates from settings

Dug into the agent-server source (profiles_router.py). The list_profiles endpoint has auto-creation logic:

# Auto-create profile from existing LLM settings if no profiles exist
# but an API key is configured.
if not summaries and settings.llm_api_key_is_set:
    llm = settings.agent_settings.llm
    profile_name = _model_to_profile_name(llm.model or "default")
    store.save(profile_name, llm, ...)
    settings_store.update(set_active)
    active_profile = profile_name

So settings do persist after deleting all profiles, and GET /api/profiles will auto-create a new profile from them when:

1. No named profiles exist, AND
2. settings.llm_api_key_is_set is true (i.e. a non-empty API key exists in agent_settings.llm)

This is exactly what happens:

1. Automation test → PATCH /api/settings with api_key: "mock-api-key-for-testing" (settings now have an API key)
2. Conversation test → creates + activates profile "mock-llm-e2e"
3. Profiles test cleanup → deletes "mock-llm-e2e" — but settings still have the API key
4. Profiles test → page loads → GET /api/profiles → no profiles exist, but llm_api_key_is_set is true → auto-creates "openai-mock-test-model" → test sees 1 profile instead of 0

The fix (PATCH settings to clear api_key) directly disables the auto-creation trigger.

---

This comment was generated by an AI agent (OpenHands).

[github-actions]

❌ Mock-LLM E2E Tests

7/11 passed · 1 failed · 3 skipped

Commit: dae3e750 · Workflow run · Test artifacts

Status	Test	Duration
✅	mock-llm-automation.spec.ts › mock-LLM automation lifecycle › step 1: setup LLM profile and register automation trajectory	227ms
✅	mock-llm-automation.spec.ts › mock-LLM automation lifecycle › step 2: create automation and dispatch run via the UI	31.5s
✅	mock-llm-automation.spec.ts › mock-LLM automation lifecycle › step 3: verify automation and run on the automations page	4.1s
✅	mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 1: create an LLM profile pointing at the mock LLM server	4.2s
✅	mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 2: activate the mock-llm profile and verify settings API	5.1s
✅	mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 3: run a conversation with the mock LLM	4.5s
✅	mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 4: resume conversation from sidebar after navigating away	3.7s
❌	mock-llm-profiles.spec.ts › mock-LLM profile lifecycle › step 1: create two LLM profiles and verify they appear in the list	14.1s
⏭️	mock-llm-profiles.spec.ts › mock-LLM profile lifecycle › step 2: activate profile-beta and verify the active badge	0ms
⏭️	mock-llm-profiles.spec.ts › mock-LLM profile lifecycle › step 3: rename the active profile and verify badge follows the new name	0ms
⏭️	mock-llm-profiles.spec.ts › mock-LLM profile lifecycle › step 4: delete the active profile and verify no active badge remains	0ms

🔍 Failure details (1)

❌ mock-llm-profiles.spec.ts › mock-LLM profile lifecycle › step 1: create two LLM profiles and verify they appear in the list

Error: expect(locator).toHaveCount(expected) failed

Locator:  getByTestId('profile-row')
Expected: 0
Received: 1
Timeout:  10000ms

Call log:
  - Expect "toHaveCount" with timeout 10000ms
  - waiting for getByTestId('profile-row')
    14 × locator resolved to 1 element
       - unexpected value "1"

_{Posted by the Mock-LLM E2E workflow · results are deterministic (scripted LLM responses)}

[github-actions]

❌ Mock-LLM E2E Tests

7/12 passed · 2 failed · 3 skipped

Commit: 1f5784c1 · Workflow run · Test artifacts

Status	Test	Duration
✅	mock-llm-automation.spec.ts › mock-LLM automation lifecycle › step 1: setup LLM profile and register automation trajectory	286ms
✅	mock-llm-automation.spec.ts › mock-LLM automation lifecycle › step 2: create automation and dispatch run via the UI	25.6s
✅	mock-llm-automation.spec.ts › mock-LLM automation lifecycle › step 3: verify automation and run on the automations page	4.1s
✅	mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 1: create an LLM profile pointing at the mock LLM server	4.2s
✅	mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 2: activate the mock-llm profile and verify settings API	5.5s
✅	mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 3: run a conversation with the mock LLM	4.5s
✅	mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 4: resume conversation from sidebar after navigating away	3.9s
⏱️	mock-llm-folder-workspace.spec.ts › mock-LLM folder browser → workspace → conversation › step 1: browse to a folder, add it as a workspace, and launch a conversation with the correct working_dir	62.4s
❌	mock-llm-profiles.spec.ts › mock-LLM profile lifecycle › step 1: create two LLM profiles and verify they appear in the list	14.1s
⏭️	mock-llm-profiles.spec.ts › mock-LLM profile lifecycle › step 2: activate profile-beta and verify the active badge	0ms
⏭️	mock-llm-profiles.spec.ts › mock-LLM profile lifecycle › step 3: rename the active profile and verify badge follows the new name	0ms
⏭️	mock-llm-profiles.spec.ts › mock-LLM profile lifecycle › step 4: delete the active profile and verify no active badge remains	0ms

🔍 Failure details (2)

⏱️ mock-llm-folder-workspace.spec.ts › mock-LLM folder browser → workspace → conversation › step 1: browse to a folder, add it as a workspace, and launch a conversation with the correct working_dir

Test timeout of 60000ms exceeded.

❌ mock-llm-profiles.spec.ts › mock-LLM profile lifecycle › step 1: create two LLM profiles and verify they appear in the list

Error: expect(locator).toHaveCount(expected) failed

Locator:  getByTestId('profile-row')
Expected: 0
Received: 1
Timeout:  10000ms

Call log:
  - Expect "toHaveCount" with timeout 10000ms
  - waiting for getByTestId('profile-row')
    14 × locator resolved to 1 element
       - unexpected value "1"

_{Posted by the Mock-LLM E2E workflow · results are deterministic (scripted LLM responses)}

[github-actions]

📸 Snapshot Test Report

✅ All snapshots match the main branch baselines.

Category	Count
🔴 Changed	0
🆕 New	0
✅ Unchanged	73
Total	73

✅ Unchanged snapshots (73)

archived-conversation

• conversation-panel-with-archived-badges
• conversation-view-archived
• conversation-view-sandbox-error

automations

• automations-delete-modal
• automations-list-active-inactive
• automations-no-automations
• automations-search-no-results

backends-extended

• backend-add-blank-disabled
• backend-add-cloud-advanced-open
• backend-add-cloud-no-key-disabled
• backend-add-cloud-with-key-enabled
• backend-add-form-partially-filled
• backend-add-invalid-url-disabled
• backend-add-local-ready
• backend-add-name-only-disabled
• backend-add-two-column-layout
• backend-add-whitespace-host-disabled
• backend-after-switch
• backend-cancel-nothing-saved
• backend-dropdown-two-backends
• backend-edit-prefilled
• backend-manage-after-removal
• backend-manage-two-listed
• backend-remove-cancelled
• backend-remove-confirmation
• backend-switch-overlay

backends

• backend-add-modal
• backend-manage-modal
• backend-selector-open

changes-tab

• changes-deleted-file
• changes-diff-viewer
• changes-empty

collapsible-thinking

• reasoning-content-collapsed
• reasoning-content-expanded
• think-action-collapsed
• think-action-expanded

mcp-page

• mcp-custom-server-1-editor-open
• mcp-custom-server-2-url-filled
• mcp-custom-server-3-all-filled
• mcp-custom-server-4-installed
• mcp-custom-server-editor
• mcp-empty-installed
• mcp-search-filtered
• mcp-slack-install-1-marketplace
• mcp-slack-install-2-modal
• mcp-slack-install-3-filled
• mcp-slack-install-4-installed

onboarding

• onboarding-step-0-choose-agent
• onboarding-step-1-check-backend
• onboarding-step-2-setup-llm
• onboarding-step-3-say-hello

projects-workspace-browser

• projects-workspace-browser

settings-page

• add-backend-modal
• analytics-consent-modal
• home-screen
• settings-app-page
• settings-page

settings-secrets

• secrets-add-form-filled
• secrets-add-form
• secrets-after-save
• secrets-delete-confirm
• secrets-list

settings-verification

• condenser-settings
• verification-settings-off
• verification-settings-on

sidebar

• sidebar-collapsed
• sidebar-conversation-panel
• sidebar-filter-menu

skills-page

• skills-empty
• skills-loaded
• skills-no-match
• skills-search-filtered
• skills-type-filter

---

Generated by the Snapshot Tests workflow. This comment was created by an AI agent (OpenHands) on behalf of the repo maintainers.