Skip to content

feat(acp): connectivity test for managed API key credentials - #1013

Draft
ChrAlpha wants to merge 6 commits into
mainfrom
worktree-acp-api-key-probe
Draft

feat(acp): connectivity test for managed API key credentials#1013
ChrAlpha wants to merge 6 commits into
mainfrom
worktree-acp-api-key-probe

Conversation

@ChrAlpha

Copy link
Copy Markdown
Member

Why

Model providers have a connectivity test, but ACP API-key setups (Codex, Claude Code) had none: a mistyped or revoked key, or a broken custom base URL, only surfaced at chat time as the opaque acp_runtime_start_failed — the ACP settings green dot merely means "form fields filled". A spawn+initialize probe would not help here either, since adapters validate credentials lazily on the first model call; the honest cheap check is to validate the stored key directly against the provider endpoint.

What

  • profile.APIKeyProbeTarget (internal/agent/runtime/acp/profile/credential_probe.go) resolves the probe target from the agent's managed config, mirroring what the runtime materializes: Codex → openai-responses @ https://api.openai.com/v1 (config.toml uses wire_api = "responses"), Claude Code → anthropic-messages @ https://api.anthropic.com; base_url overrides apply. OAuth/self modes, missing keys, and unsupported agents return sentinel errors.
  • providers.TestSDKProvider extracts the existing provider-probe classification (including the __ping__ sentinel that catches auth failures behind unauthenticated /models listings) into a reusable function; providers.Service.Test now delegates to it, behavior unchanged.
  • POST /bots/{bot_id}/acp/agents/{agent_id}/credentials/test (internal/handlers/acp_credentials.go) gates on AuthorizeBotAccess, probes the saved managed credentials, and returns the same {status, reachable, latency_ms, message} shape as the provider test. Swagger + SDK regenerated.
  • Web: the ACP agent settings page shows a "Test Connection" row in API Key mode for Codex/Claude Code — success shows latency, failure shows a sanitized message. The HTML-stripping error formatter from provider-form.vue moved to a shared utils/probe-error.ts (now unit-tested) and both call sites use it.
  • Drive-by: pinned the pnpm 11 allowBuilds placeholder (protobufjs: set this to true or falsefalse) that made every fresh pnpm install exit non-zero and blocked sdk-generate.

Out of scope (discussed, follow-ups): self mode needs an end-to-end micro-turn probe; Hermes custom provider is not covered.

Verification

  • go test ./internal/handlers/ ./internal/providers/ ./internal/agent/runtime/acp/profile/ ./internal/arch/ — all pass (17 new test cases, written test-first against httptest fakes).
  • go build ./cmd/... ./internal/..., golangci-lint clean on the touched packages.
  • pnpm vitest run — 1118 passed; the 5 failures are pre-existing on main (verified identical with this branch's changes stashed).

@ChrAlpha

Copy link
Copy Markdown
Member Author

Live E2E pass against a real Anthropic-compatible proxy (valid key + custom base URL), exercising testACPManagedCredentials end to end. It caught one real bug, fixed in 221ce5e: Claude Code's ANTHROPIC_BASE_URL excludes /v1 (the agent appends it), while the Twilight anthropic provider expects the base to include it — the probe was hitting {base}/models and 404ing even on valid credentials. The probe now appends /v1 for Claude Code targets (default and override), mirroring the runtime.

Observed feedback after the fix (backend message → what the settings row renders):

Scenario Status UI message
valid key + proxy base ok Credentials verified · 1510ms
wrong key auth_error api error 401: 401 Unauthorized · {"error":"Invalid API key"}
base URL points at a website error api error 404: 404 Not Found · Example Domain This domain is for use… (HTML stripped, 220-char cap)
unreachable host error (reachable: false) connection failed: request failed: Get "https://…/v1/models?limit=1": EOF
oauth/self mode HTTP 400 acp agent is not configured with api key setup
key not filled HTTP 400 acp agent api key is not configured

The UI strings were pinned by running the raw backend messages through formatProbeError in vitest.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants