Skip to content

v3 Slice 5: Provider breadth + per-vendor native-config policy + parallel runner #18

Description

@mark-allwyn

Parent

#13

What to build

Extend the provider layer to the full vendor roster so the whole model set can be evaluated, each returning the same structured result with correct status and thinking-token mapping.

  • Port OpenAI (finish_reason, reasoning_tokens), Google (finishReason, thoughtsTokenCount), OpenAI-compatible endpoints, Bedrock (stopReason), and Ollama - each mapping into the shared structured result and the status taxonomy.
  • Per-vendor best-native-config policy resolved from the model registry: adaptive thinking for Claude 4.6+/Fable, reasoning defaults for OpenAI/Gemini reasoning models, max_tokens >= 16000 everywhere; declarative per-model overrides in config.
  • Parallelize the runner across providers (thread pool) for multi-model sweeps.
  • API keys only via env vars; errors sanitized before storage.

Acceptance criteria

  • Mocked-HTTP tests per provider assert status (incl. refusal) and thinking-token extraction
  • Policy unit tests: correct request body per vendor (e.g. no temperature for Fable, adaptive thinking present, reasoning defaults for OpenAI/Gemini)
  • A live smoke passes for at least one model per configured provider family, recording stop_reason and thinking_tokens
  • Parallel runner observably caps concurrency and never corrupts the results file
  • pytest green

Blocked by

Metadata

Metadata

Assignees

No one assigned

    Labels

    ready-for-agentReady for autonomous agent pickup

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions