feat: add openai-compatible executor - #315
Conversation
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds an openai-compatible execution path so evaluations can run against OpenAI-compatible Chat Completions endpoints (e.g., local LM Studio), and documents/configures the new options.
Changes:
- Introduces
openai-compatibleexecutor with endpoint normalization and API key handling (OPENAI_API_KEYfallback). - Extends JSON schemas + project defaults to support
endpointandapi_key, and updates docs accordingly. - Adds unit/integration-style tests for schema validation, spec loading, engine execution, and
waza runbehavior.
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| site/src/content/docs/reference/waza-yaml.mdx | Documents defaults.endpoint in .waza.yaml. |
| site/src/content/docs/reference/schema.mdx | Documents new openai-compatible executor and endpoint/api_key. |
| site/src/content/docs/reference/cli.mdx | Documents OPENAI_API_KEY env var behavior for the new executor. |
| site/src/content/docs/guides/eval-yaml.mdx | Updates eval YAML guide with openai-compatible options and semantics. |
| schemas/eval.schema.json | Extends eval schema for openai-compatible, makes model conditional. |
| schemas/config.schema.json | Extends project config schema with new engine and defaults.endpoint. |
| internal/validation/schema_test.go | Adds schema-validation test cases for openai-compatible configs. |
| internal/projectconfig/config_test.go | Validates new defaults.endpoint parsing/merging. |
| internal/projectconfig/config.go | Adds Defaults.Endpoint and merges it. |
| internal/models/spec_test.go | Adds tests for parsing endpoint/api_key into the spec model. |
| internal/models/spec.go | Adds Config.Endpoint and Config.APIKey. |
| internal/execution/openai_compatible_test.go | New tests covering endpoint normalization, auth, workspace cleanup, and skill-context injection. |
| internal/execution/openai_compatible.go | Implements the OpenAICompatibleEngine. |
| internal/execution/copilot.go | Refactors skill-dir selection into reusable helper for new engine. |
| cmd/waza/cmd_run_test.go | Adds waza run tests for openai-compatible and .waza.yaml defaults. |
| cmd/waza/cmd_run.go | Applies .waza.yaml defaults to eval specs and wires in openai-compatible engine creation. |
| README.md | Adds docs for openai-compatible executor and related env vars. |
| if cfg, err := projectconfig.Load(filepath.Dir(specPath)); err == nil && cfg != nil { | ||
| applyProjectDefaultsToSpec(spec, cfg) | ||
| } |
| workspaceDir, err = os.MkdirTemp("", "waza-openai-*") | ||
| if err != nil { | ||
| return nil, fmt.Errorf("failed to create openai-compatible workspace: %w", err) | ||
| } | ||
| if err := setupWorkspaceResources(workspaceDir, req.Resources); err != nil { | ||
| _ = os.RemoveAll(workspaceDir) | ||
| return nil, fmt.Errorf("failed to setup openai-compatible workspace resources: %w", err) | ||
| } | ||
| e.mu.Lock() | ||
| e.workspaces = append(e.workspaces, workspaceDir) | ||
| e.mu.Unlock() | ||
| } | ||
| if _, err := ResolveWorkDir(workspaceDir, req.WorkDir); err != nil { | ||
| return nil, err | ||
| } |
| case "openai-compatible": | ||
| engine, err = execution.NewOpenAICompatibleEngine(spec.Config.Endpoint, spec.Config.ModelID, spec.Config.APIKey) | ||
| if err != nil { | ||
| return nil, err | ||
| } |
| defaultedEndpoint := `name: test-eval | ||
| skill: test-skill | ||
| version: "1.0" | ||
| config: | ||
| trials_per_task: 1 | ||
| timeout_seconds: 60 | ||
| executor: openai-compatible | ||
| metrics: | ||
| - name: accuracy | ||
| weight: 1.0 | ||
| threshold: 0.8 | ||
| tasks: | ||
| - "tasks/*.yaml" | ||
| ` | ||
| require.Empty(t, ValidateEvalBytes([]byte(defaultedEndpoint)), "openai-compatible should allow endpoint from .waza.yaml defaults") |
Shayne Boyer (spboyer)
left a comment
Shayne Boyer (spboyer)
left a comment
There was a problem hiding this comment.
The OpenAI-compatible executor introduces workspace and multi-turn contract regressions that need correction before merge.
Issues to address:
- internal/execution/openai_compatible.go:122 - Git resources are not materialized in the execution workspace
- internal/execution/openai_compatible.go:177 - Follow-up turns discard the prior conversation history
| _ = os.RemoveAll(workspaceDir) | ||
| return nil, fmt.Errorf("failed to setup openai-compatible workspace resources: %w", err) | ||
| } | ||
| e.mu.Lock() |
There was a problem hiding this comment.
[MEDIUM] API Contract: This workspace setup passes only req.Resources, while the other engines also materialize req.GitResources. Evaluations that supply repositories will execute in a workspace without those repositories and may grade unrelated failures. Include req.GitResources when preparing the workspace, matching the existing engine behavior.
| func (e *OpenAICompatibleEngine) sendChatCompletion(ctx context.Context, modelID string, req *ExecutionRequest, sourceDir string) (string, *models.UsageStats, string, bool, error) { | ||
| payload := openAICompatibleChatRequest{ | ||
| Model: modelID, | ||
| Messages: buildOpenAICompatibleMessages(req, sourceDir), |
There was a problem hiding this comment.
[MEDIUM] API Contract: A non-empty SessionID identifies a follow-up turn, but every Chat Completions request is rebuilt from only the current prompt and system context. Multi-turn evaluations therefore lose the prior user and assistant messages despite reusing a session. Persist per-session message history and append it to the next request, or explicitly reject multi-turn requests for this engine.
|
David Honig (@davidhonig) checking in on this PR. It still appears to need a rebase/conflict resolution and follow-up on the requested changes before it can move forward. Could you share an update on whether you plan to continue with it? |
Summary
Summary
Adds an
openai-compatibleexecutor for running Waza evals against OpenAI-compatible Chat Completions APIs such as local LM Studio. The executor supports endpoint normalization, optional per-evalapi_key,OPENAI_API_KEYfallback, workspace capture, usage mapping, and skill-context injection..waza.yamlcan now providedefaults.engine: openai-compatibleanddefaults.endpoint, so eval specs can stay minimal while using a local endpoint.Related issue
Related to #66
Agent handoff
internal/execution/openai_compatible.go,cmd/waza/cmd_run.go,internal/models/spec.go,internal/projectconfig/config.go,schemas/eval.schema.json,schemas/config.schema.json, README andsite/docs..waza.yaml; useOPENAI_API_KEYfor auth.modelis optional foropenai-compatibleand defaults tolocal-model. Skill context is injected using the existing skill-system-message path.Type of change
Validation
go test ./...make lintorgolangci-lint runweb/changedhttp://127.0.0.1:1234Documentation
site/docs updated, if CLI, YAML, dashboard, or validator behavior changedRisk and rollback
bb0fd568to remove the new executor, schema fields, project defaults, and docs. Existingmockandcopilot-sdkexecutor paths should remain unaffected.Notes for reviewers
Please focus on
internal/execution/openai_compatible.gorequest/response handling, skill-context injection, and.waza.yamldefault application incmd/waza/cmd_run.go. The intended local config isdefaults.engine: openai-compatibleplusdefaults.endpoint: http://127.0.0.1:1234; auth should come fromOPENAI_API_KEY.