Jev fast path for open/computer intents

[wassgha]

Summary

• Before the LLM tool loop, classify desktop intents with TypeSafe Jev (typesafe-ai/jev) via the AI Gateway /v4/ai/evaluation-model HTTP API (no AI SDK 7 upgrade — keeps the realtime canary intact).
• Clear open_app / open_url / open_path commands with a parseable target skip the language model and run the Open skill directly (still permission-gated). This is the common slow path today when "open Safari" goes through a full vision/tool model.
• High-confidence but unparseable open intents constrain tools to Open only; computer intents add a system-prompt bias so the LLM doesn't waste a step on openApp.
• Settings → Skills: Fast open (Jev) toggle (skills.jevFastPath, default on). Disabled in the UI when no Gateway key is present; no-ops without AI_GATEWAY_API_KEY.

Limits (intentional)

Jev cannot drive computer-use clicks/typing (no vision / no free-form tool args). Real multi-step UI control still uses the configured LLM; this PR only accelerates open/launch and steers routing. Mid-utterance speculative open (à la Andy Gao's demo) is a follow-up — this path still waits for endpointing.

Review follow-ups

• Confirmed both reported "blockers" are false positives: onDelta/onTool* handlers are the guarded emitters from index.ts; buildToolSet({ include }) was already used by realtime.
• Hardened missing probabilities → treat chosen option as confident (?? 1).
• Dimmed Settings toggle without a Gateway key; expanded extractor smoke cases.

Test plan

• pnpm typecheck
• pnpm smoke:jev (local extractors)
• pnpm smoke:jev --live with a Gateway key (route latency + labels)
• In-app: say/type "open Safari" with Gateway key + Open skill always-allow → should confirm without a long thinking pause
• "click the Submit button" with computer enabled → still goes through the LLM (Jev only biases the prompt)
• Toggle Fast open off → behavior falls back to the full LLM loop

  

[github-actions]

🤖 AI review

◇ injected env (0) from .env // tip: ⌘ custom filepath { path: '/custom/path/.env' }

Summary

Adds a "Jev fast path" that calls TypeSafe Jev's evaluation API to classify voice utterances (open_app / open_url / open_path / computer / other) before the LLM loop, short-circuiting the slow path for clear "open Safari"-style commands, and wires up config, settings UI, and a smoke-test script.

---

Findings

[blocker] fast-open.ts – null return treated as handled
In executeOpenTarget, when tool is not found or the signal is aborted, null is returned. Back in tryJevDesktopFastPath the caller checks if (handled) but a null return falls through to the hint path with skillFilter: ["open"], not to a proper handled/skip. That's probably fine for the not-found case, but the abort case is wrong: the user got no response and the LLM is then restricted to Open tools:

if (target) {
  const handled = await executeOpenTarget({ ... });
  if (handled) { return { kind: "handled", ... }; }
  // null falls through to hint ← abort + no response, then LLM fires
}

After an abort the outer ac.signal.aborted guard in index.ts will catch it eventually, but the hint path still runs buildToolSet/resolveModel. Should return { kind: "skip" } after the if (handled) check when the signal is aborted.

[blocker] fast-open.ts – double onDelta when tool execution fails but permission was denied
executeOpenTarget calls opts.handlers.onDelta(spoken) unconditionally and then appends a final role: "assistant" text message to the returned messages[]. Back in index.ts the handled branch sends IPC.chatDone(requestId, fast.messages). The renderer will render the spoken text twice — once from the chatDelta event, once from the assistant message in chatDone. Check how the existing non-fast-path keeps these separated, and either drop the trailing role: "assistant" content item or don't call onDelta.

[warning] evaluate.ts – unvalidated cast of API response

answers: json.answers as EvaluateResult<Q>["answers"],

Only json.answers existence is checked; individual answer .type fields and the choice string are not validated before being used in desktop-route.ts as answer.type !== "choice" / answer.choice as DesktopRoute. A mal/differently-shaped response from the Gateway (or a future API change) would silently produce a DesktopRoute value that isn't in DESKTOP_ROUTES, bypassing the guard:

const route = answer.choice as DesktopRoute;
if (!DESKTOP_ROUTES.includes(route)) return null; // this guard is fine

The guard saves you here, but answer.probabilities?.[answer.choice] ?? 1 will return undefined ?? 1 = 1, making every misclassified route look like 100 % confidence. Worth adding a minimal runtime check on answer.type.

[warning] fast-open.ts – permission gate bypass on re-route
When the fast path returns hint and the LLM fallthrough runs, makePermissionRequester was already consumed for the session (requestPermission is the same instance). The session-grant set means a prior "allow once" in the fast-path attempt could silently carry over to the LLM tool call. This is an existing design of the session grant, but the new code makes the surface area larger — worth a comment.

[warning] desktop-route.ts – regex for open_path misses paths without a verb
extractOpenTarget for open_path only matches open|show|reveal + a ~/ or / path. Utterances like "~/Documents" or "/tmp/foo.pdf" (dictated paths) return null and fall to the LLM. This is conservative (safe), but AGENTS.md says "open a file or folder by path" — a bare path with no verb should probably also be handled; at minimum add a test case to smoke-jev confirming the null.

[warning] index.ts – skillFilter! non-null assertion

include: skillFilter
  ? (skill) => skillFilter!.includes(skill.id)
  : undefined,

The ! is needed because the closure captures skillFilter as string[] | undefined even inside the truthy branch. This is a TypeScript scoping quirk, not a real risk, but it means buildToolSet now needs an include parameter that wasn't in the original signature — confirm registry.ts actually exposes it (the diff only changes a comment, not the signature; if include was already there this is fine, otherwise it's a compile error not caught in the diff).

[nit] evaluate.ts – AI_GATEWAY_API_KEY read inside the function
evaluateWithJev reads process.env.AI_GATEWAY_API_KEY at call time. routeDesktopIntent also checks it before calling. Two places to update if the env var name changes; a shared constant or a single check would be cleaner.

[nit] smoke-jev.ts – go to github.com test expects https://github.com (no trailing slash)
The domain regex in extractOpenTarget captures github.com without a trailing slash and prepends https://, producing https://github.com — that matches. But if the utterance ends in punctuation (e.g., go to github.com.) stripTrailingPunct is applied after the regex capture on the raw match, so the period would already be outside the capture group in that regex. Low risk, but the smoke test doesn't cover the punctuation edge case.

---

Questions

1. Does the renderer's chatDone handler append the messages array to conversation history, or only use it for display? If it appends, the fast-path's synthetic role: "tool" + role: "assistant" messages need to exactly match whatever shape streamChat normally produces, otherwise the next turn's context window will be malformed.
2. Is AI_GATEWAY_API_KEY the same credential as the one used for the gateway LLM provider, or a separate scoped key? The Settings UI implies they're the same (secrets.AI_GATEWAY_API_KEY check), but the AGENTS note says "same as the gateway LLM provider" — worth making this explicit in the schema comment since Jev calls a different endpoint and a misconfigured key scope could cause silent routing failures (falls back to LLM, no user-visible error).