feat(providers): make Providers v2 the only provider system

[johntmyers]

Problem Statement

OpenShell currently has two provider behaviors: legacy credential-only providers and Providers v2 profile-backed providers. The split makes provider behavior harder to reason about: profile-owned policy is gated behind providers_v2_enabled, CLI and TUI discovery still depend on the legacy registry, and profile-only providers that contribute policy without credentials are not consistently first-class.

We should make Providers v2 the only provider system, even if that breaks older workflows. Provider profiles should define provider semantics: credential requirements, discovery, endpoint and binary policy, refresh behavior, and policy-only providers. Provider instances should bind a name to a profile plus concrete credential/config values when the profile needs them.

Related: #1978 should be folded into this work. Credential-less policy-only profiles, such as pypi, must be creatable without dummy credentials like --credential NOOP=.

Proposed Design

Make provider profiles authoritative and remove the compatibility switch.

• Remove providers_v2_enabled from registered runtime settings and CLI/docs examples.
• Always compose attached provider profile policy into the effective sandbox policy.
• Stop accepting profileless provider types for new provider records.
• Treat provider records as instances of built-in or imported provider profiles.
• Replace public legacy registry discovery with profile-backed discovery.
• Update CLI and TUI provider creation to drive type selection, credential labels, discovery, and validation from provider profiles.
• Allow provider creation with no credential source when the profile declares no required static credentials or declares required credentials that are runtime-resolvable through refresh/token-grant metadata.
• Fail clearly when a profile requires static credentials and none are supplied or discoverable.

Credential-less providers should be normal, not a workaround. For a policy-only profile such as pypi, this should work:

openshell provider create --name pypi --type pypi
openshell sandbox create --provider pypi -- <command>

The resulting provider record should have no dummy credential entries, while the attached profile still contributes its endpoint and binary policy layer.

Affected Components

Component	Key files	Required work
Provider profiles	providers/*.yaml, crates/openshell-providers/src/profiles.rs	Ensure every supported provider type is represented as a profile and that profile metadata can express static, runtime-resolvable, and no-credential providers.
Gateway provider API	crates/openshell-server/src/grpc/provider.rs	Validate provider creation/update against profile requirements. Allow empty credentials when the profile permits it. Reject profileless provider types.
Gateway policy config	crates/openshell-server/src/grpc/policy.rs	Remove the providers_v2_enabled gate and always compose provider profile policy for attached providers. Revisit global-policy interaction so provider layers are not accidentally suppressed in the v2-only model unless that remains intentional.
CLI	crates/openshell-cli/src/main.rs, crates/openshell-cli/src/run.rs	Remove the unconditional provider credential-source requirement. Fetch the profile to decide whether credentials are required. Remove legacy registry fallback and update sandbox provider auto-create behavior.
TUI	crates/openshell-tui/src/app.rs, crates/openshell-tui/src/lib.rs	Replace legacy ProviderRegistry type/env-var selection with profile RPCs and profile credential metadata.
Docs	docs/sandboxes/manage-providers.mdx, docs/sandboxes/providers-v2.mdx, quickstarts, man pages	Collapse the two-provider-system story into one profile-backed provider model. Remove enable/disable docs. Document credential-less policy-only providers.
Tests	provider, policy, CLI, TUI-adjacent integration tests	Remove disabled-v2 expectations. Add coverage for profile-required credentials, runtime-resolvable credentials, policy-only providers, and profileless provider rejection.

Alternatives Considered

Only change the default for providers_v2_enabled. This preserves legacy behavior and keeps two provider systems alive. It does not satisfy the goal of making Providers v2 the only provider system.

Keep profileless credential-only providers as an escape hatch. This reduces migration pain, but it preserves the ambiguity that v2 is meant to remove. If a provider has no profile, OpenShell cannot know its policy, credential requirements, refresh model, or user-facing semantics.

Add a --no-credential flag for policy-only providers. This is less desirable than letting the profile declare whether credentials are needed. A pypi-style profile has no credential semantics, so the CLI should not require the user to opt out of credentials manually.

Agent Investigation

The local checkout shows the compatibility split clearly:

• providers_v2_enabled is registered as a runtime setting in crates/openshell-core/src/settings.rs.
• GetSandboxConfig reads that setting and only composes provider profile policy when it is enabled in crates/openshell-server/src/grpc/policy.rs.
• Provider profile policy composition already exists and is just-in-time derived data; it does not persist provider-generated rules back into the sandbox-authored policy.
• openshell provider create --from-existing switches between profile discovery and the legacy provider registry in crates/openshell-cli/src/run.rs.
• openshell sandbox create currently suppresses command-derived provider inference when Providers v2 is enabled.
• The TUI still uses openshell_providers::ProviderRegistry directly for provider type lists and credential env-var labels.
• Server-side provider creation currently rejects empty credentials unless the provider type allows empty credentials for gateway-managed refresh bootstrap.

Remote main has advanced provider-profile work that should be treated as the current target shape:

• Built-in profiles now include pypi.yaml and several other profile-backed providers.
• pypi.yaml has endpoints and binaries but no credentials, making it a concrete policy-only provider case.
• The CLI has expanded credential-source options, including --from-gcloud-adc and runtime credentials, but the create command still has an unconditionally required cred_source arg group.
• The server uses a generic provider_type_allows_empty_credentials() helper backed by profile metadata, but the current profile helper still requires at least one runtime-resolvable credential. A zero-credential policy-only profile still needs explicit support.

Definition of Done

• providers_v2_enabled is removed from user-facing settings and no longer gates provider policy composition.
• New provider records must reference a built-in or imported provider profile.
• Profile-backed discovery is the only provider discovery path exposed by CLI/TUI workflows.
• Credential-less policy-only profiles can be instantiated without dummy credentials.
• Profiles with required static credentials still fail clearly when no credential source is supplied.
• Runtime-resolvable credential profiles keep working for refresh/token-grant bootstrap flows.
• TUI provider creation uses provider profile metadata rather than the legacy registry.
• Docs describe one provider model and remove Providers v2 opt-in instructions.
• Tests cover policy-only providers, required credentials, runtime credentials, profileless rejection, and unconditional provider policy composition.

Notes

This is intentionally a breaking change. Existing profileless provider records may need a migration story, a one-time warning, or a release note explaining that unsupported provider types must be recreated from imported profiles.

Related issues:

• feat(providers): support credential-less provider creation for policy-only profiles #1978 — credential-less provider creation for policy-only profiles.
• bug: reserve _provider_* policy key prefix and enforce at gateway boundaries #1982 — reserved _provider_* policy key hardening; relevant because unconditional provider policy composition makes generated provider rule namespaces more important.