Skip to content

Refactor Jev judgment transports into run-bound drivers - #16

Merged
jkudish merged 8 commits into
mainfrom
transport-drivers
Sep 23, 2026
Merged

jkudish merged 8 commits into
mainfrom
transport-drivers

Conversation

@jkudish

@jkudish jkudish commented Sep 23, 2026

Copy link
Copy Markdown
Owner

What and why

Make the four existing Jev judgment transports self-contained drivers selected by one registry. Resolve the driver once per navigation run, validate untrusted answers before actions or usage accounting, and let library consumers supply a JevTransport without changing the separate typing system. This keeps provider-specific wire details out of the navigation loop and makes malformed judgments fail closed.

PlanMode task: #3982.

Design

  • Drivers for TypeSafe, OpenRouter, Cloudflare, and Vercel retain their request envelopes, headers, model slugs, response adaptation, and token behavior. TypeSafe uses a run-bound SDK client; Vercel uses a run-bound gateway model and an injectable evaluate factory for tests.
  • The facade selects in TypeSafe > OpenRouter > Cloudflare > Vercel order and validates exact answer IDs and types, choice/score distributions and levels, Noul range, confidence, usage, and effective model. Invalid responses cannot credit usage or execute an action.
  • NavigateOptions.transport overrides the judgment provider registry and credentials, with the transport name and effective model reported in results. The package entry exports transport and result types. est_cost_usd is still a Jev-token estimate, not carrier billing for injected transports.
  • OpenRouter and Cloudflare errors report status, category, and response byte count without raw response text. Other built-in errors also omit response content. Each adapter distinguishes absent usage counters (zero) from present malformed usage (error).

Deliberate behavior changes

  • Unknown forced JEV_PROVIDER values now reject instead of falling through to auto-detection; the no-provider diagnostic now includes AI_GATEWAY_API_KEY.
  • Missing or malformed answers now stop navigation instead of synthesizing {} or choosing a default dropdown option. The first-option select fallback is removed.
  • Programmatic injection takes precedence over forced provider settings, even invalid settings, but does not alter typing configuration.

Verification

  • npm run typecheck: passed.
  • npm run build: passed.
  • npm test: 71 passed, 0 failed, 0 skipped.
  • TYPESAFE_API_KEY="$TYPESAFE_API_KEY" npm run test:e2e: 39 passed, 0 failed, 0 skipped on final head; the live tests are unchanged.
  • README.md and CHANGELOG.md em-dash counts: 0 and 0. git diff --check: passed.
  • Oracle review: round 1 requested rejecting present malformed usage rather than defaulting it; remediated in the second commit with four-driver regression coverage. Round 2 APPROVE, no findings.

Two earlier full E2E runs had a transient payment-submit status: error before any step; an isolated test, six direct page runs, and subsequent full suites passed. The error detail was not captured, so the cause is unresolved; validation was not relaxed.

Left out intentionally: typing-provider refactoring, a generic chat-carrier helper, and merge/release. Joey gates the merge.

jkudish and others added 8 commits September 23, 2026 07:22
…I job

Oracle review of the post-APPROVE delta: the new skip guard meant the
injected-transport integration test never ran in CI (unit job has no
browser, e2e job ran only e2e.test.mjs). Run the transports file in the
browser-installed job too. Also qualify the changelog claim to built-in
providers, since injected transports can throw their own error text.

Amp-Thread-ID: https://ampcode.com/threads/T-01a0ad0f-c784-73ba-a506-4705ebd912c0
Co-authored-by: Amp <amp@ampcode.com>
Step-level env does not carry between steps, so the previous
env.TYPESAFE_API_KEY condition was always empty and the step never ran
(Oracle round 2). Gate on the install step outcome instead; the test
itself is hermetic and needs no key.

Amp-Thread-ID: https://ampcode.com/threads/T-01a0ad0f-c784-73ba-a506-4705ebd912c0
Co-authored-by: Amp <amp@ampcode.com>
@jkudish
jkudish merged commit 0784138 into main Sep 23, 2026
1 check passed
@jkudish
jkudish deleted the transport-drivers branch September 23, 2026 20:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant