Skip to content

feat: add data_model parameter to .to_solver() for structured output in evals#264

Merged
cpsievert merged 5 commits intomainfrom
feat/structured-solver
Jan 6, 2026
Merged

feat: add data_model parameter to .to_solver() for structured output in evals#264
cpsievert merged 5 commits intomainfrom
feat/structured-solver

Conversation

@cpsievert
Copy link
Collaborator

Summary

  • Adds a data_model parameter to .to_solver() that accepts a Pydantic model
  • When provided, the solver uses .chat_structured_async() instead of .chat_async()
  • The structured output is serialized to JSON and set as state.output.completion
  • Enables structured data extraction tasks in Inspect AI evaluations

Example usage

from chatlas import ChatOpenAI
from pydantic import BaseModel
from inspect_ai import Task, task
from inspect_ai.dataset import Sample

class Person(BaseModel):
    name: str
    age: int

chat = ChatOpenAI(system_prompt="Extract person info from text.")

@task
def extract_eval():
    return Task(
        dataset=[Sample(input="John is 42", target='{"name": "John", "age": 42}')],
        solver=chat.to_solver(data_model=Person),  # NEW
        scorer=...,
    )

Test plan

  • Added test test_structured_output in tests/test_inspect.py
  • Type checking passes
  • All existing inspect tests pass
  • VCR cassette recorded for the new test

🤖 Generated with Claude Code

cpsievert and others added 2 commits January 6, 2026 11:33
…n evals

When a data_model (Pydantic model) is provided to .to_solver(), the solver
uses .chat_structured_async() instead of .chat_async() to generate responses.
The resulting Pydantic model instance is serialized to JSON and set as the
completion text in state.output.completion.

This allows using chatlas for structured data extraction tasks in Inspect AI
evaluations, where scorers can parse and validate the JSON output.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@cpsievert cpsievert changed the title feat: add data_model parameter to to_solver() for structured output in evals feat: add data_model parameter to .to_solver() for structured output in evals Jan 6, 2026
Move all completion text determination to one place, right before
setting state.output.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@cpsievert cpsievert requested a review from Copilot January 6, 2026 17:48

This comment was marked as resolved.

@cpsievert cpsievert merged commit c9898f4 into main Jan 6, 2026
9 checks passed
@cpsievert cpsievert deleted the feat/structured-solver branch January 6, 2026 17:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant