LLMStrategy capabilities (serializable + registerable) + pydantic-ai v2

[jduerholt]

Summary

Brings the LLM recommender to the next level: a serializable + registerable capability system for LLMStrategy, plus migration to pydantic-ai v2. Capabilities are typed, serializable config; a mapper reconstructs the live pydantic-ai capability (including its tool functions) at runtime — mirroring the existing LLM-provider triplet (data model + tagged union + mapper + register).

As the first capability, experiment presentation moves from a prompt-dump (with n_recent/n_top knobs) to a tool-only ExperimentAccessCapability the model queries on demand.

What's included

Capability triplet (mirrors AnyLLMProvider + LLM_MAP + register_llm_provider)

• bofire/data_models/llm/capability.py — LLMCapability base + ExperimentAccessCapability
• AnyLLMCapability union + register_llm_capability(); LLMStrategy.capabilities field (default-on via default_factory, replace-semantics documented)
• bofire/llm/capabilities_mapper.py — CAPABILITY_MAP + register/map
• bofire/llm/experiment_tools.py — FunctionToolset of inspect_recent/inspect_top/inspect_near/summary_stats/list_pending_candidates; per-query logic in unit-testable pure functions

Tool-only experiment access

• Removed n_recent_experiments / n_top_experiments and the prompt-dump path
• The capability instructions surface only counts (metadata, not a predefined view) to motivate tool use
• Strategy deps refactored _LLMDeps → LLMContext (domain + experiments + pending candidates)
• The output_validator feasibility gate is unchanged — capabilities enrich, never bypass domain.validate_candidates

pydantic-ai v2 migration

• output_retries → retries={"output": N}, OpenAIModel → OpenAIChatModel, pin pydantic-ai>=2,<3

Design notes

• Serialization boundary: config serializes; the mapper is the code that rebuilds tools. Custom capabilities round-trip only where their mapper is registered (same as custom providers).
• Why capabilities, not a bare tool system: v2's capability bundles instructions + tools + model settings + hooks; "skill" and "tool" collapse into one primitive, one registry, one mapper.
• Future capabilities (MCPCapability, RAG, BO-as-a-tool, sandboxed code analysis) drop into the same slot without redesign.

Verification

• 1418 passed, 5 skipped across tests/bofire/data_models, test_llm, bofire/llm, test_register (incl. exact serialization round-trip for the default capability)
• ruff check clean, formatted; ty check bofire exits 0 (new modules produce zero diagnostics)

Follow-ups (out of scope)

• asyncio.run() in _ask throws inside a running event loop (Jupyter); matters more once a deep/agentic harness lands.

🤖 Generated with Claude Code