Clarify LLM profile deletion vs settings auto-migration behavior

[enyst]

Context

When testing LLM profiles, this sequence is surprising:

1. Create one or more LLM profiles.
2. Activate a profile. Activation copies that profile's LLM config into persisted user settings (active_profile + agent_settings.llm).
3. Delete all profiles.
4. Call GET /api/profiles.
5. A profile is automatically recreated from the still-populated LLM settings, even though the user explicitly deleted every profile.

The auto-create behavior appears intentional as a migration path from pre-profile LLM settings. The endpoint auto-creates a profile when there are no profiles but settings.llm_api_key_is_set is true. The problem is that the same migration also runs after an explicit delete-all action because the old active settings/secret remain in user settings.

Investigation: can agent-canvas users enter local LLM settings without a profile?

Short answer: in the normal standalone agent-canvas UI for a local backend, no.

Relevant findings from OpenHands/agent-canvas:

• src/routes/llm-settings.tsx renders different UIs by backend kind:
  • local backend -> <LlmSettingsLocalView /> (profile management)
  • cloud backend -> <LlmSettingsScreen /> (raw/settings form, no profiles)
• src/hooks/use-settings-nav-items.ts renames the local /settings entry to "LLM Profiles", reinforcing that profiles are the local user-facing abstraction.
• LlmSettingsLocalView only exposes a profile list plus create/edit flows. Its create/edit form embeds LlmSettingsScreen, but saves through ProfilesService.saveProfile(...), not as standalone raw LLM settings.
• The onboarding LLM step embeds LlmSettingsScreen, but for local backends it then best-effort saves and activates the entered config as a profile, with the code comment saying the profiles list is the user-facing source of truth.
• There is a public library export named LLMSettings that points to LlmSettingsScreen, and cloud mode still uses LlmSettingsScreen; those are not the normal local standalone agent-canvas path.
• The profile editor does not support clearing a stored API key: an empty api_key in edit mode means "no change" and preserves the existing encrypted key; in create mode it omits the key. The Slack thread also notes the settings endpoint rejects empty API keys, so there may be no reliable UI/API path to clear the legacy llm.api_key once it exists.

So in local agent-canvas, profiles effectively replaced the direct LLM settings UI. That makes the delete-all -> auto-recreate behavior more confusing: the user deletes all visible LLM configuration, but hidden raw settings still preserve enough data to recreate it.

Why this matters

If profiles are intended to be the source of truth, deleting an active profile should probably not leave a hidden copy of the same credentials/config in user settings that can resurrect a profile on the next list call.

If raw LLM settings are still intended to be a first-class fallback, we should expose that model clearly in the UI and give users a way to clear those settings. Today the local UI does not make that fallback visible.

Design questions

• Is GET /api/profiles allowed to have this side effect forever, or should auto-create run only once as a migration?
• Should deleting the active profile clear active_profile only, or also clear/reset the copied agent_settings.llm values?
• Should deleting the last active profile prompt the user with a choice like "also clear LLM credentials/settings"?
• Should there be a persisted migration sentinel (for example profiles_migrated_from_settings=true) so explicit delete-all does not trigger migration again?
• Can/should SaaS, agent-canvas, and old OSS/local GUI converge on one profile implementation under agent-server so semantics do not drift?

My take

I do not love silently deleting settings as a side effect of deleting a profile, because historically settings and profiles may have been separate concepts and credentials are valuable/user-entered data.

But in local agent-canvas specifically, the UI no longer gives users a normal way to manage raw LLM settings outside profiles. In that mental model, profile activation makes agent_settings.llm look like an implementation detail/cache. If so, deleting the active profile and leaving that cache populated is surprising and makes the delete operation feel ineffective.

A safer direction might be:

1. Make profile auto-creation a one-time migration, not a permanent GET /api/profiles side effect.
   • Once profiles have existed, or once the user explicitly deletes all profiles, do not auto-create again from settings.
2. When deleting an active profile:
   • clear active_profile immediately;
   • if other profiles remain, require/select another active profile before starting new conversations, or fall back to no active profile explicitly;
   • if no profiles remain, do not auto-create from legacy settings on the next list call.
3. Decide separately whether to clear agent_settings.llm secrets/config:
   • conservative option: leave raw settings in place but do not auto-create from them after explicit deletion;
   • stricter/profile-source-of-truth option: ask the user on active-profile deletion whether to also clear the copied LLM settings/API key;
   • if we choose stricter behavior, the delete modal should explain this because credentials may be difficult to recover.
4. Add regression coverage at the API layer for:
   • legacy no-profiles + API key -> migration creates first profile;
   • explicit delete-all -> subsequent GET /api/profiles returns empty list and does not recreate;
   • deleting active profile clears/deactivates active_profile consistently.

This issue was created by an AI agent (OpenHands) on behalf of the requester.

[enyst]

HUMAN:

Very interesting question! It's maybe one of the reasons why it kinda starts to make sense to not keep an llm settings object around, alongside profiles, it will lead to strange behaviour.

Tiny detail:
I would consider turning LLM API keys into Provider keys; and either include them in secrets, or make a bundle of them just like secrets.

After all, they really are per provider, not per LLM. It would be easier, also, to create quick a new profile from the same providers you already have.

Then it would maybe make sense UX wise to have a different tab for entering/CRUD these keys?

Then the mental model would seem to become: you can delete profiles, and that won't delete the bucket of keys; you can delete any or all keys when you want. 🤔

Just some thoughts. Today it's a bit weird.