Skip to content

Stale hardcoded model catalogue: doctor advertises nonexistent Anthropic models; agent init defaults to openai/gpt-4o regardless of configured providers #103

Description

@ambiorix2099

Summary

The CLI ships a hardcoded model catalogue that has drifted out of date. Two concrete consequences:

  1. conductor doctor advertises Anthropic models that do not exist — both return 404 from the
    Anthropic API.
  2. conductor agent init always writes model: openai/gpt-4o regardless of which providers are
    actually configured, so the documented initrun flow fails out of the box for anyone
    without an OpenAI key.

Found while regression-testing CLI branch cleanup/remove-skill-command (#100) against server
3.32.0-rc.23.

1. doctor advertises nonexistent models

cmd/doctor.go hardcodes per-provider model lists in aiProviders:

{name: "Anthropic", envVars: []string{"ANTHROPIC_API_KEY"},
 models: []string{"anthropic/claude-sonnet-4-20250514", "anthropic/claude-3-5-sonnet-20241022"}},

doctor presents these as available:

$ conductor doctor
AI Providers
  ok  Anthropic (ANTHROPIC_API_KEY)
        anthropic/claude-sonnet-4-20250514
        anthropic/claude-3-5-sonnet-20241022

Both are gone. Verified against the live Anthropic API with a valid key, with a control:

Model GET /v1/models/<id>
claude-sonnet-4-20250514 404
claude-3-5-sonnet-20241022 404
claude-haiku-4-5-20251001 (control) 200

Copying a model straight out of doctor output produces a failed agent execution:

reasonForIncompletion: Task ... failed: Anthropic Messages API failed with status 404:
{"type":"error","error":{"type":"not_found_error","message":"model: claude-sonnet-4-20250514"}}

Note this failure is server-side at execution time, so the user gets a FAILED workflow rather than
a validation error — an expensive way to learn the model name was wrong.

The other providers' lists are likely stale too (openai/gpt-4o-mini,
google_gemini/gemini-1.5-pro, azure_openai/gpt-4o); I only verified Anthropic, which is the key
I had.

2. agent init default ignores configured providers

cmd/agent.go:38:

defaultInitModel = "openai/gpt-4o"

agent init writes that unconditionally:

$ conductor doctor | grep -A1 'AI Providers' -A14 | grep '  ok'
  ok  Anthropic (ANTHROPIC_API_KEY)          # only Anthropic configured

$ conductor agent init probe && cat probe.yaml
model: openai/gpt-4o                          # unusable here

$ conductor agent run --config probe.yaml "hi"
# fails at execution time — no OPENAI_API_KEY

Nothing cross-checks the init default against what doctor already knows is configured, even though
both live in the same binary. The command even prints
Run with: conductor agent run --config probe.yaml "your prompt here" — an instruction that cannot
succeed.

Impact

First-run experience for the agent feature is broken in the common case where a user has exactly one
provider key and it isn't OpenAI. The two defects compound: doctor says Anthropic is ready and
names two models, both invalid; agent init then picks a third model from a provider that isn't
configured at all.

Fix options

Model catalogue

  • A. Drop the model lists from doctor. It only needs to report which providers are configured;
    naming specific models is a maintenance liability that will drift again. Smallest, most durable
    fix.
  • B. Fetch models from the provider at runtime (e.g. Anthropic's GET /v1/models). Accurate, but
    adds network calls and per-provider code to a diagnostic command.
  • C. Refresh the hardcoded lists. Restores correctness today; guarantees a repeat of this issue.

Recommend A, or A plus a pointer to each provider's model docs.

agent init default

  • D. Derive the default from configured providers — reuse the aiProviders detection doctor
    already performs, and pick a model for the provider whose key is present.
  • E. Require --model when the default provider isn't configured, with an error naming what is
    configured.

Recommend D with E as the fallback when nothing is configured.

Test coverage

cmd/doctor_test.go exists but does not assert model-string validity (it can't, offline). Worth
noting the model lists are untestable-by-construction while hardcoded, which is an argument for
option A.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions