Resolve and report the key of the configured judgment backend

CHANGELOG.md

@@ -4,6 +4,21 @@
 
 <!-- Empty. Next release starts here. -->
 
+## 0.7.0
+
+### Fixed
+
+- Key reporting and login can name the judgment backend: `keySituation(backend)`, `resolveApiKey(backend)`, `authState({ backend })`, and `ensureApiKey(ctx, { backend })` read the backend's own environment variable, `describeAuth` labels the key by backend and names the variable to set, and `AuthState` carries `backend`. Before, every surface reported the TypeSafe key, so an OpenRouter user saw "TypeSafe key: missing" while judgments ran, and `ensureApiKey` opened the TypeSafe login and verified the pasted key against api.typesafe.ai (#9).
+- `createTypeSafe({ backend: "openrouter" })` no longer falls back to `TYPESAFE_API_KEY` or the login store when `OPENROUTER_API_KEY` is unset; a TypeSafe key was being sent to OpenRouter.
+
+### Added
+
+- `DEFAULT_BACKEND` export and a `label` on every `DECISIONS_BACKENDS` entry.
+
+### Docs
+
+- Document the `backend` option, `DECISIONS_BACKENDS`, and which key each backend reads in the README and the API reference; the OpenRouter backend shipped in 0.6.0 without either.
+
 ## 0.6.2
 
 ### Fixed

README.md

@@ -138,7 +138,9 @@ const answer = await ask(typesafe, {
 if (!answer.ok) return { skipped: answer.errorCode === "budget" };  // never throws
 ```
 
-Your extension owns its own user consent and budget; `/typesafe enable` applies only to this package's tool. Check `authState()` rather than your own consent flag before you report that judgments are on. Every export — the client, `ask`, batching, the usage ledger, auth state, and the `pi-typesafe/calibrate` and `pi-typesafe/ui` entry points — is in [docs/api.md](docs/api.md).
+Your extension owns its own user consent and budget; `/typesafe enable` applies only to this package's tool. Check `authState()` rather than your own consent flag before you report that judgments are on.
+
+Judgments can also go through OpenRouter: `createTypeSafe({ backend: "openrouter" })` sends them to `openrouter.ai` with the key from `OPENROUTER_API_KEY`. That backend has no login store, so `/typesafe login` does not apply to it. Pass the same `backend` to `authState`, `keySituation`, and `ensureApiKey`, or the status you report describes the TypeSafe key while the requests use another one. The `/typesafe` commands and the `typesafe_evaluate` tool always use the TypeSafe backend. Every export — the client, `ask`, batching, the usage ledger, auth state, and the `pi-typesafe/calibrate` and `pi-typesafe/ui` entry points — is in [docs/api.md](docs/api.md).
 
 ## Development
 

docs/api.md

@@ -23,8 +23,9 @@ result.answers.severity.score;   // 0..2, may be fractional
 
 | Option | Default | Meaning |
 | --- | --- | --- |
-| `apiKey` | `TYPESAFE_API_KEY`, else the `/typesafe login` store | Never returned |
-| `model` | `jev-latest` | No model is inferred from submitted content |
+| `apiKey` | the backend's key (below) | Never returned |
+| `backend` | `typesafe` | `typesafe` or `openrouter`; picks the host, the request path, the default model, and the key |
+| `model` | `jev-latest` (`typesafe/jev-1.13` on OpenRouter) | No model is inferred from submitted content |
 | `timeoutMs` | `15000` | Per request; no automatic retries |
 | `maxInputBytes` | `65536` | UTF-8 JSON bytes, not tokens |
 | `maxRequests` | `20` | Attempts per client instance, failures included |
@@ -33,6 +34,8 @@ result.answers.severity.score;   // 0..2, may be fractional
 | `ledger` | the store next to the key | Inject a ledger in tests |
 | `fetch` | global fetch | Inject a transport for offline tests |
 
+`DECISIONS_BACKENDS` is the registry behind `backend`: each entry carries `label`, `host`, `keyEnv`, and, when the service does not serve the SDK's own path, `path`. `DEFAULT_BACKEND` is `"typesafe"`. The TypeSafe backend takes its key from `TYPESAFE_API_KEY`, then the `/typesafe login` store. Every other backend reads only its own environment variable (`OPENROUTER_API_KEY` for OpenRouter): the store holds a TypeSafe key, and a login verifies against api.typesafe.ai, so neither applies elsewhere. Pass the same `backend` to `authState`, `keySituation`, and `ensureApiKey` so what you report matches what you send.
+
 `evaluate(request, { signal })` validates before sending and rejects with `TypeSafeIntegrationError`. `code` is one of `configuration`, `validation`, `budget`, `aborted`, `timeout`, `http`, `connection`, `response`; messages never contain upstream bodies, headers, keys, or your submitted state. `listModels()` verifies the key without counting toward `maxRequests`.
 
 ## Admission
@@ -70,11 +73,11 @@ The environment may lower an explicit cap, never raise it. A reached cap raises
 
 ## Auth state
 
-`authState()` never throws. It reports `kind` (`environment`, `stored`, `missing`, `unusable`), `keyName`, `path`, `reason`, `verified`, `verifiedAt`, `lastFailure`, and `usable` — `usable` is false when no key is present or the last authentication outcome was an HTTP 401/403 rejection.
+`authState({ backend })` never throws. It reports `backend`, `kind` (`environment`, `stored`, `missing`, `unusable`), `keyName`, `path`, `reason`, `verified`, `verifiedAt`, `lastFailure`, and `usable` — `usable` is false when no key is present or the last authentication outcome was an HTTP 401/403 rejection. `backend` defaults to `typesafe`; name the backend you pass to `createTypeSafe`, or the report describes a key you do not send. The verification and failure record is one file shared by every backend, so after switching backends the last outcome stands until the next request.
 
 `describeAuth(state)` turns that into `{ level: "ok" | "warning" | "error", text }` for a status line or a log. The extension calls both at session start and after a rejection, so an enabled-but-unusable setup is never reported as working.
 
-`recordAuthVerified()` is called by `listModels()` and by the first successful request; `recordAuthFailure(error)` records what degraded TypeSafe; `clearAuthState()` forgets both, and `/typesafe logout` calls it. `keySituation()` and `keySourceLabel(situation)` remain the lower-level, frozen-for-existing-callers pair, and `resolveApiKey()` the pre-0.4.0 one.
+`recordAuthVerified()` is called by `listModels()` and by the first successful request; `recordAuthFailure(error)` records what degraded TypeSafe; `clearAuthState()` forgets both, and `/typesafe logout` calls it. `keySituation(backend)` and `keySourceLabel(situation)` remain the lower-level, frozen-for-existing-callers pair, and `resolveApiKey(backend)` the pre-0.4.0 one; the `backend` argument is optional and defaults to `typesafe`. An environment situation names the variable it read in `keyEnv`.
 
 ## Asking without throwing
 
@@ -111,7 +114,7 @@ console.log(formatCalibration(calibrate("action guard", samplesOf(results).sampl
 
 ## Login helpers: `pi-typesafe/ui`
 
-`ensureApiKey(ctx)`, `loginWithPrompt(ctx)`, and `promptForApiKey(ctx)` use the same hidden input as `/typesafe login`. `ensureApiKey(ctx)` returns the existing key source, or prompts, verifies, and stores a new key (`undefined` when the user cancels). These need Pi's TUI, so call them only from extension command handlers.
+`ensureApiKey(ctx, { backend })`, `loginWithPrompt(ctx)`, and `promptForApiKey(ctx)` use the same hidden input as `/typesafe login`. `ensureApiKey(ctx)` returns the existing key source, or prompts, verifies, and stores a new TypeSafe key (`undefined` when the user cancels). For any other backend it returns the environment source or throws `configuration` naming the variable to set; it never opens the prompt, because the prompt verifies against api.typesafe.ai and writes the TypeSafe store. These need Pi's TUI, so call them only from extension command handlers.
 
 ## One agent tool
 

package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-typesafe",
-  "version": "0.6.2",
+  "version": "0.7.0",
   "description": "TypeSafe AI (Jev) decisions for Pi: batched Choice/Score/Noul evaluation tool, terminal playground, and a typed API other extensions build on.",
   "type": "module",
   "license": "MIT",

src/auth.ts

@@ -1,5 +1,7 @@
 import { chmodSync, mkdirSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs";
 import { join } from "node:path";
+import { DEFAULT_BACKEND, TYPESAFE_KEY_ENV, backendConfig, usesTypesafeKey } from "./backends.js";
+import type { TypeSafeBackend } from "./backends.js";
 import { credentialsPath, keySituation, keySourceLabel, piTypesafeDir } from "./credentials.js";
 import type { KeySource } from "./credentials.js";
 import { TypeSafeIntegrationError } from "./errors.js";
@@ -26,16 +28,21 @@ export interface AuthFailure {
  * that judgments will happen — an enabled extension with no key used to look identical to a working one.
  */
 export interface AuthState {
+  /** The judgment backend this state describes; each backend has its own key. */
+  readonly backend: TypeSafeBackend;
   /** Same kinds as KeySituation: where the key in effect comes from. */
   readonly kind: "environment" | "stored" | "missing" | "unusable";
   readonly source?: KeySource;
   /** Where the key would be read from. */
   readonly path: string;
   /** Why a stored key cannot be used, when that is the case. */
   readonly reason?: string;
-  /** Short human label for the key source: `TYPESAFE_API_KEY`, `/typesafe login`, `no key`, `unusable key`. */
+  /** Short human label for the key source: `TYPESAFE_API_KEY`, `OPENROUTER_API_KEY`, `/typesafe login`, `no key`, `unusable key`. */
   readonly keyName: string;
-  /** True when the key in effect was accepted by api.typesafe.ai (login verifies it; a successful request proves it). */
+  /**
+   * True when the key in effect was accepted by the backend (login verifies a TypeSafe key; a successful request proves
+   * any key). The record is shared across backends: switching backends keeps the last outcome until the next request.
+   */
   readonly verified: boolean;
   readonly verifiedAt?: string;
   /** The last failure, cleared by the next successful request. */
@@ -82,15 +89,17 @@ function writeState(path: string, state: { verifiedAt?: string; lastFailure?: Au
   }
 }
 
-/** What the key situation, the last outcome, and the clock add up to. Never throws. */
-export function authState(options: { path?: string } = {}): AuthState {
+/** What the key situation, the last outcome, and the clock add up to for one backend. Never throws. */
+export function authState(options: { path?: string; backend?: TypeSafeBackend } = {}): AuthState {
   const path = options.path ?? authStatePath();
-  const situation = keySituation();
+  const backend = options.backend ?? DEFAULT_BACKEND;
+  const situation = keySituation(backend);
   const stored = readState(path);
   const source: KeySource | undefined = situation.kind === "environment" ? "environment" : situation.kind === "stored" ? "stored" : undefined;
   const rejected = stored.lastFailure?.code === "http" && stored.lastFailure.status !== undefined && REJECTED_STATUSES.has(stored.lastFailure.status);
   const usable = source !== undefined && !rejected;
   return {
+    backend,
     kind: situation.kind,
     ...(source === undefined ? {} : { source }),
     path: situation.kind === "unusable" ? situation.path : credentialsPath(),
@@ -137,19 +146,22 @@ export interface AuthReport {
  * state instead of reporting "enabled".
  */
 export function describeAuth(state: AuthState = authState()): AuthReport {
+  const config = backendConfig(state.backend ?? DEFAULT_BACKEND);
+  const label = `${config.label} key`;
   const since = state.lastFailure ? ` Last failure: ${state.lastFailure.message}${state.lastFailure.at ? ` (${state.lastFailure.at})` : ""}` : "";
   if (state.kind === "missing") {
-    return { level: "error", text: `TypeSafe key: missing — every Jev judgment is skipped until a key is configured (/typesafe login or TYPESAFE_API_KEY).${since}` };
+    const how = usesTypesafeKey(config) ? `a key is configured (/typesafe login or ${TYPESAFE_KEY_ENV})` : `${config.keyEnv} is set in the environment`;
+    return { level: "error", text: `${label}: missing — every Jev judgment is skipped until ${how}.${since}` };
   }
   if (state.kind === "unusable") {
-    return { level: "error", text: `TypeSafe key: unusable (${state.reason ?? "unknown reason"}) — judgments are skipped until the key is fixed.${since}` };
+    return { level: "error", text: `${label}: unusable (${state.reason ?? "unknown reason"}) — judgments are skipped until the key is fixed.${since}` };
   }
   const rejected = state.lastFailure?.code === "http" && state.lastFailure.status !== undefined && REJECTED_STATUSES.has(state.lastFailure.status);
   if (rejected) {
-    return { level: "error", text: `TypeSafe key: ${state.keyName} was rejected.${since}` };
+    return { level: "error", text: `${label}: ${state.keyName} was rejected.${since}` };
   }
   if (!state.verified) {
-    return { level: "warning", text: `TypeSafe key: ${state.keyName} (not verified yet — the first request proves it).${since}` };
+    return { level: "warning", text: `${label}: ${state.keyName} (not verified yet — the first request proves it).${since}` };
   }
-  return { level: "ok", text: `TypeSafe key: ${state.keyName} (verified${state.verifiedAt ? ` ${state.verifiedAt}` : ""}).${since}` };
+  return { level: "ok", text: `${label}: ${state.keyName} (verified${state.verifiedAt ? ` ${state.verifiedAt}` : ""}).${since}` };
 }

src/backends.ts

@@ -0,0 +1,40 @@
+import { TypeSafeIntegrationError } from "./errors.js";
+
+export type TypeSafeBackend = "typesafe" | "openrouter";
+
+export interface BackendConfig {
+  /** Human name for status lines: "TypeSafe", "OpenRouter". */
+  label: string;
+  host: string;
+  /** The environment variable that carries this backend's key. Absent means the TypeSafe key resolution applies. */
+  keyEnv?: string;
+  /** Request path, when the backend does not serve the SDK's own `/v1/systemone`. */
+  path?: string;
+}
+
+/** The backend every key and auth function assumes when none is named. */
+export const DEFAULT_BACKEND: TypeSafeBackend = "typesafe";
+
+/** The environment variable and login store that the default backend reads. */
+export const TYPESAFE_KEY_ENV = "TYPESAFE_API_KEY";
+
+/** Registry of known judgment backends. Extendable by callers. */
+export const DECISIONS_BACKENDS: Record<TypeSafeBackend, BackendConfig> = {
+  typesafe: { label: "TypeSafe", host: "https://api.typesafe.ai", keyEnv: TYPESAFE_KEY_ENV },
+  openrouter: { label: "OpenRouter", host: "https://openrouter.ai", keyEnv: "OPENROUTER_API_KEY", path: "/api/alpha/decisions" },
+};
+
+/** The registry entry for a backend name; a `configuration` error for a name the registry does not know. */
+export function backendConfig(name: TypeSafeBackend): BackendConfig {
+  const backend = DECISIONS_BACKENDS[name];
+  if (!backend) throw new TypeSafeIntegrationError("configuration", `Unknown judgment backend "${name}". Valid backends: ${Object.keys(DECISIONS_BACKENDS).join(", ")}.`);
+  return backend;
+}
+
+/**
+ * Whether a backend's key comes from the TypeSafe resolution (`TYPESAFE_API_KEY`, then the login store) or only from
+ * its own environment variable. Only the TypeSafe backend has a login store; every other backend is environment-only.
+ */
+export function usesTypesafeKey(backend: BackendConfig): boolean {
+  return (backend.keyEnv ?? TYPESAFE_KEY_ENV) === TYPESAFE_KEY_ENV;
+}

src/client.ts

@@ -1,6 +1,8 @@
 import { TypeSafeClient } from "@typesafe-ai/sdk";
 import type { Fetch, Questions, SystemOneRequest, SystemOneResult } from "@typesafe-ai/sdk";
 import { recordAuthFailure, recordAuthVerified } from "./auth.js";
+import { DEFAULT_BACKEND, TYPESAFE_KEY_ENV, backendConfig, usesTypesafeKey } from "./backends.js";
+import type { TypeSafeBackend } from "./backends.js";
 import type { BatchEvaluation, BatchOptions } from "./batch.js";
 import { evaluateAll, evaluateMany } from "./batch.js";
 import { keySituation } from "./credentials.js";
@@ -9,24 +11,12 @@ import { DEFAULT_MAX_INPUT_BYTES, assertWithinByteLimit, prepareEvaluationReques
 import { DEFAULT_USD_PER_MTOK, capsFromEnvironment, estimateUsd, mergeCaps, openUsageLedger } from "./usage.js";
 import type { BlockedCap, SpendCaps, UsageLedger, UsageReport } from "./usage.js";
 
-export type TypeSafeBackend = "typesafe" | "openrouter";
-
-export interface BackendConfig {
-  host: string;
-  keyEnv?: string;
-  /** Request path, when the backend does not serve the SDK's own `/v1/systemone`. */
-  path?: string;
-}
+export { DECISIONS_BACKENDS, DEFAULT_BACKEND } from "./backends.js";
+export type { BackendConfig, TypeSafeBackend } from "./backends.js";
 
 /** The path the SDK appends to whatever base URL it is given. */
 const SDK_PATH = "/v1/systemone";
 
-/** Registry of known judgment backends. Extendable by callers. */
-export const DECISIONS_BACKENDS: Record<TypeSafeBackend, BackendConfig> = {
-  typesafe: { host: "https://api.typesafe.ai", keyEnv: "TYPESAFE_API_KEY" },
-  openrouter: { host: "https://openrouter.ai", keyEnv: "OPENROUTER_API_KEY", path: "/api/alpha/decisions" },
-};
-
 /** Send the SDK's fixed path to the backend's own, preserving any caller-supplied transport. */
 function backendFetch(path: string, inner: Fetch = fetch): Fetch {
   return (input, init) => inner(String(input).replace(SDK_PATH, path), init);
@@ -155,34 +145,20 @@ function capsDescription(caps: SpendCaps): string {
 /** A bounded, server-side TypeSafe client independent of Pi's runtime. */
 export function createTypeSafe(options: TypeSafeOptions = {}): TypeSafe {
   let apiKey = options.apiKey?.trim();
-  // Resolve backend and host.
-  const backendName = options.backend;
-  let baseURL = "https://api.typesafe.ai";
-  let keyEnv = "TYPESAFE_API_KEY";
-  let backendPath: string | undefined;
-  if (backendName !== undefined) {
-    const backend = DECISIONS_BACKENDS[backendName];
-    if (!backend) throw new TypeSafeIntegrationError("configuration", `Unknown judgment backend "${backendName}". Valid backends: ${Object.keys(DECISIONS_BACKENDS).join(", ")}.`);
-    baseURL = backend.host;
-    keyEnv = backend.keyEnv ?? "TYPESAFE_API_KEY";
-    backendPath = backend.path;
-  }
+  const backendName: TypeSafeBackend = options.backend ?? DEFAULT_BACKEND;
+  const backend = backendConfig(backendName);
+  const baseURL = backend.host;
+  const backendPath = backend.path;
 
   if (!apiKey) {
-    // Backend-specific env var first (e.g. OPENROUTER_API_KEY), then fall back to
-    // the standard TYPESAFE_API_KEY / stored-key resolution.
-    if (backendName !== undefined && keyEnv !== "TYPESAFE_API_KEY") {
-      const fromEnv = process.env[keyEnv]?.trim();
-      if (fromEnv) apiKey = fromEnv;
-    }
-    if (!apiKey) {
-      const situation = keySituation();
-      if (situation.kind === "unusable") throw new TypeSafeIntegrationError("configuration", situation.reason);
-      if (situation.kind === "environment" || situation.kind === "stored") apiKey = situation.key;
-    }
+    // The same resolution that authState() and ensureApiKey() report, so the status line and the request agree.
+    const situation = keySituation(backendName);
+    if (situation.kind === "unusable") throw new TypeSafeIntegrationError("configuration", situation.reason);
+    if (situation.kind === "environment" || situation.kind === "stored") apiKey = situation.key;
   }
   if (!apiKey) {
-    throw new TypeSafeIntegrationError("configuration", `No API key. Run /typesafe login in Pi, or set ${keyEnv} in the environment.`);
+    const how = usesTypesafeKey(backend) ? `Run /typesafe login in Pi, or set ${TYPESAFE_KEY_ENV}` : `Set ${backend.keyEnv}`;
+    throw new TypeSafeIntegrationError("configuration", `No API key. ${how} in the environment.`);
   }
   const timeout = positiveInteger(options.timeoutMs ?? 15_000, "timeoutMs");
   const maxInputBytes = positiveInteger(options.maxInputBytes ?? DEFAULT_MAX_INPUT_BYTES, "maxInputBytes");