Skip to content

Consolidate eval-runner/ai-runner duplicates against main's runner-lib and local-runners #65

Description

@schmug

While #47 was in flight, main landed 71 commits including three modules that overlap it. The merge resolved the one hard conflict (auc) toward main, but three duplicates are still present and should be consolidated before the eval CLI (#48) is built on top of them.

Already resolved in #47

scripts/eval-vec.ts dropped its own auc/cohensD and re-exports src/metrics.ts. Main had independently made the same average-rank tie correction, and additionally returns NaN for an empty group. There is exactly one AUC again, which is main's stated rule.

Still duplicated

in #47 main's equivalent notes
restAiRunner (src/ai-runner.ts) cloudflareRunner (scripts/runner-lib.ts) same Workers AI REST call
localAiRunner (src/ai-runner.ts) createOpenAiRunner (src/local-runners.ts) main's also ships createOllamaRunner for the native API
resolveEvalRunner (scripts/eval-runner.ts) resolveBackend + parseArgs (scripts/runner-lib.ts) eval-runner.ts is imported by nothing

Recommended shape

  1. Delete scripts/eval-runner.ts and its tests. It is dead code duplicating runner-lib.ts. Its 64 tests encode real edge cases (empty flag values, space-separated flags, prototype-named flags, last-wins) — port the valuable ones onto runner-lib.ts's parser rather than discarding them, since resolveBackend has not been probed for those cases.
  2. Make localAiRunner delegate to createOpenAiRunner, keeping selectAiRunner and the LOCAL_AI_* env contract. The Worker-side wiring is genuinely additive — main's DO is still Workers-AI-only — but it should not carry a second OpenAI client.
  3. Drop restAiRunner once nothing imports it. calibrate.ts already moved to runner-lib in the merge.

Affects #48

#48 tells N6/N7 to use resolveEvalRunner. If this consolidation lands first, they should use resolveBackend from runner-lib.ts instead. Sequence this before #48, or update #48's briefs.

Acceptance criteria

  • Exactly one OpenAI-compatible runner, one Workers AI REST runner, one backend resolver, one AUC.
  • No behaviour change to the Worker: /api/debug/ai still reports which backend answered, and the DO still honours LOCAL_AI_BASE_URL.
  • Full suite green, typecheck clean over both tsc projects.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions