From 945d61b366fca1dd699fa0f15e4cc9b388a5c249 Mon Sep 17 00:00:00 2001 From: Ryan Gapac Date: Tue, 22 Sep 2026 22:16:25 +0800 Subject: [PATCH 1/2] fix: resolve and report the key of the configured judgment backend keySituation, resolveApiKey, authState, and ensureApiKey take the backend that createTypeSafe uses, read that backend's own environment variable, and describeAuth labels the key by backend. A backend without a login store never opens the TypeSafe prompt, which verified the pasted key against api.typesafe.ai. createTypeSafe no longer sends a TypeSafe key to another backend when that backend's variable is unset. The registry moves to src/backends.ts so the key functions and the client share it. Closes #9 --- CHANGELOG.md | 13 +++++++++- README.md | 4 ++- docs/api.md | 13 ++++++---- src/auth.ts | 32 ++++++++++++++++-------- src/backends.ts | 40 ++++++++++++++++++++++++++++++ src/client.ts | 52 +++++++++++---------------------------- src/credentials.ts | 26 +++++++++++++------- src/index.ts | 2 +- src/login.ts | 26 ++++++++++++++------ tests/auth.test.ts | 28 +++++++++++++++++++++ tests/client.test.ts | 19 ++++++++++++++ tests/credentials.test.ts | 29 ++++++++++++++++++++-- tests/login.test.ts | 18 ++++++++++++++ 13 files changed, 227 insertions(+), 75 deletions(-) create mode 100644 src/backends.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index d9f6e1c..5c23ca9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,18 @@ ## Unreleased - +### Fixed + +- Key reporting and login can name the judgment backend: `keySituation(backend)`, `resolveApiKey(backend)`, `authState({ backend })`, and `ensureApiKey(ctx, { backend })` read the backend's own environment variable, `describeAuth` labels the key by backend and names the variable to set, and `AuthState` carries `backend`. Before, every surface reported the TypeSafe key, so an OpenRouter user saw "TypeSafe key: missing" while judgments ran, and `ensureApiKey` opened the TypeSafe login and verified the pasted key against api.typesafe.ai (#9). +- `createTypeSafe({ backend: "openrouter" })` no longer falls back to `TYPESAFE_API_KEY` or the login store when `OPENROUTER_API_KEY` is unset; a TypeSafe key was being sent to OpenRouter. + +### Added + +- `DEFAULT_BACKEND` export and a `label` on every `DECISIONS_BACKENDS` entry. + +### Docs + +- Document the `backend` option, `DECISIONS_BACKENDS`, and which key each backend reads in the README and the API reference; the OpenRouter backend shipped in 0.6.0 without either. ## 0.6.2 diff --git a/README.md b/README.md index c55b0db..03b1375 100644 --- a/README.md +++ b/README.md @@ -138,7 +138,9 @@ const answer = await ask(typesafe, { if (!answer.ok) return { skipped: answer.errorCode === "budget" }; // never throws ``` -Your extension owns its own user consent and budget; `/typesafe enable` applies only to this package's tool. Check `authState()` rather than your own consent flag before you report that judgments are on. Every export — the client, `ask`, batching, the usage ledger, auth state, and the `pi-typesafe/calibrate` and `pi-typesafe/ui` entry points — is in [docs/api.md](docs/api.md). +Your extension owns its own user consent and budget; `/typesafe enable` applies only to this package's tool. Check `authState()` rather than your own consent flag before you report that judgments are on. + +Judgments can also go through OpenRouter: `createTypeSafe({ backend: "openrouter" })` sends them to `openrouter.ai` with the key from `OPENROUTER_API_KEY`. That backend has no login store, so `/typesafe login` does not apply to it. Pass the same `backend` to `authState`, `keySituation`, and `ensureApiKey`, or the status you report describes the TypeSafe key while the requests use another one. The `/typesafe` commands and the `typesafe_evaluate` tool always use the TypeSafe backend. Every export — the client, `ask`, batching, the usage ledger, auth state, and the `pi-typesafe/calibrate` and `pi-typesafe/ui` entry points — is in [docs/api.md](docs/api.md). ## Development diff --git a/docs/api.md b/docs/api.md index 1f5acd5..ec24250 100644 --- a/docs/api.md +++ b/docs/api.md @@ -23,8 +23,9 @@ result.answers.severity.score; // 0..2, may be fractional | Option | Default | Meaning | | --- | --- | --- | -| `apiKey` | `TYPESAFE_API_KEY`, else the `/typesafe login` store | Never returned | -| `model` | `jev-latest` | No model is inferred from submitted content | +| `apiKey` | the backend's key (below) | Never returned | +| `backend` | `typesafe` | `typesafe` or `openrouter`; picks the host, the request path, the default model, and the key | +| `model` | `jev-latest` (`typesafe/jev-1.13` on OpenRouter) | No model is inferred from submitted content | | `timeoutMs` | `15000` | Per request; no automatic retries | | `maxInputBytes` | `65536` | UTF-8 JSON bytes, not tokens | | `maxRequests` | `20` | Attempts per client instance, failures included | @@ -33,6 +34,8 @@ result.answers.severity.score; // 0..2, may be fractional | `ledger` | the store next to the key | Inject a ledger in tests | | `fetch` | global fetch | Inject a transport for offline tests | +`DECISIONS_BACKENDS` is the registry behind `backend`: each entry carries `label`, `host`, `keyEnv`, and, when the service does not serve the SDK's own path, `path`. `DEFAULT_BACKEND` is `"typesafe"`. The TypeSafe backend takes its key from `TYPESAFE_API_KEY`, then the `/typesafe login` store. Every other backend reads only its own environment variable (`OPENROUTER_API_KEY` for OpenRouter): the store holds a TypeSafe key, and a login verifies against api.typesafe.ai, so neither applies elsewhere. Pass the same `backend` to `authState`, `keySituation`, and `ensureApiKey` so what you report matches what you send. + `evaluate(request, { signal })` validates before sending and rejects with `TypeSafeIntegrationError`. `code` is one of `configuration`, `validation`, `budget`, `aborted`, `timeout`, `http`, `connection`, `response`; messages never contain upstream bodies, headers, keys, or your submitted state. `listModels()` verifies the key without counting toward `maxRequests`. ## Admission @@ -70,11 +73,11 @@ The environment may lower an explicit cap, never raise it. A reached cap raises ## Auth state -`authState()` never throws. It reports `kind` (`environment`, `stored`, `missing`, `unusable`), `keyName`, `path`, `reason`, `verified`, `verifiedAt`, `lastFailure`, and `usable` — `usable` is false when no key is present or the last authentication outcome was an HTTP 401/403 rejection. +`authState({ backend })` never throws. It reports `backend`, `kind` (`environment`, `stored`, `missing`, `unusable`), `keyName`, `path`, `reason`, `verified`, `verifiedAt`, `lastFailure`, and `usable` — `usable` is false when no key is present or the last authentication outcome was an HTTP 401/403 rejection. `backend` defaults to `typesafe`; name the backend you pass to `createTypeSafe`, or the report describes a key you do not send. The verification and failure record is one file shared by every backend, so after switching backends the last outcome stands until the next request. `describeAuth(state)` turns that into `{ level: "ok" | "warning" | "error", text }` for a status line or a log. The extension calls both at session start and after a rejection, so an enabled-but-unusable setup is never reported as working. -`recordAuthVerified()` is called by `listModels()` and by the first successful request; `recordAuthFailure(error)` records what degraded TypeSafe; `clearAuthState()` forgets both, and `/typesafe logout` calls it. `keySituation()` and `keySourceLabel(situation)` remain the lower-level, frozen-for-existing-callers pair, and `resolveApiKey()` the pre-0.4.0 one. +`recordAuthVerified()` is called by `listModels()` and by the first successful request; `recordAuthFailure(error)` records what degraded TypeSafe; `clearAuthState()` forgets both, and `/typesafe logout` calls it. `keySituation(backend)` and `keySourceLabel(situation)` remain the lower-level, frozen-for-existing-callers pair, and `resolveApiKey(backend)` the pre-0.4.0 one; the `backend` argument is optional and defaults to `typesafe`. An environment situation names the variable it read in `keyEnv`. ## Asking without throwing @@ -111,7 +114,7 @@ console.log(formatCalibration(calibrate("action guard", samplesOf(results).sampl ## Login helpers: `pi-typesafe/ui` -`ensureApiKey(ctx)`, `loginWithPrompt(ctx)`, and `promptForApiKey(ctx)` use the same hidden input as `/typesafe login`. `ensureApiKey(ctx)` returns the existing key source, or prompts, verifies, and stores a new key (`undefined` when the user cancels). These need Pi's TUI, so call them only from extension command handlers. +`ensureApiKey(ctx, { backend })`, `loginWithPrompt(ctx)`, and `promptForApiKey(ctx)` use the same hidden input as `/typesafe login`. `ensureApiKey(ctx)` returns the existing key source, or prompts, verifies, and stores a new TypeSafe key (`undefined` when the user cancels). For any other backend it returns the environment source or throws `configuration` naming the variable to set; it never opens the prompt, because the prompt verifies against api.typesafe.ai and writes the TypeSafe store. These need Pi's TUI, so call them only from extension command handlers. ## One agent tool diff --git a/src/auth.ts b/src/auth.ts index 8031c71..8a11de2 100644 --- a/src/auth.ts +++ b/src/auth.ts @@ -1,5 +1,7 @@ import { chmodSync, mkdirSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs"; import { join } from "node:path"; +import { DEFAULT_BACKEND, TYPESAFE_KEY_ENV, backendConfig, usesTypesafeKey } from "./backends.js"; +import type { TypeSafeBackend } from "./backends.js"; import { credentialsPath, keySituation, keySourceLabel, piTypesafeDir } from "./credentials.js"; import type { KeySource } from "./credentials.js"; import { TypeSafeIntegrationError } from "./errors.js"; @@ -26,6 +28,8 @@ export interface AuthFailure { * that judgments will happen — an enabled extension with no key used to look identical to a working one. */ export interface AuthState { + /** The judgment backend this state describes; each backend has its own key. */ + readonly backend: TypeSafeBackend; /** Same kinds as KeySituation: where the key in effect comes from. */ readonly kind: "environment" | "stored" | "missing" | "unusable"; readonly source?: KeySource; @@ -33,9 +37,12 @@ export interface AuthState { readonly path: string; /** Why a stored key cannot be used, when that is the case. */ readonly reason?: string; - /** Short human label for the key source: `TYPESAFE_API_KEY`, `/typesafe login`, `no key`, `unusable key`. */ + /** Short human label for the key source: `TYPESAFE_API_KEY`, `OPENROUTER_API_KEY`, `/typesafe login`, `no key`, `unusable key`. */ readonly keyName: string; - /** True when the key in effect was accepted by api.typesafe.ai (login verifies it; a successful request proves it). */ + /** + * True when the key in effect was accepted by the backend (login verifies a TypeSafe key; a successful request proves + * any key). The record is shared across backends: switching backends keeps the last outcome until the next request. + */ readonly verified: boolean; readonly verifiedAt?: string; /** The last failure, cleared by the next successful request. */ @@ -82,15 +89,17 @@ function writeState(path: string, state: { verifiedAt?: string; lastFailure?: Au } } -/** What the key situation, the last outcome, and the clock add up to. Never throws. */ -export function authState(options: { path?: string } = {}): AuthState { +/** What the key situation, the last outcome, and the clock add up to for one backend. Never throws. */ +export function authState(options: { path?: string; backend?: TypeSafeBackend } = {}): AuthState { const path = options.path ?? authStatePath(); - const situation = keySituation(); + const backend = options.backend ?? DEFAULT_BACKEND; + const situation = keySituation(backend); const stored = readState(path); const source: KeySource | undefined = situation.kind === "environment" ? "environment" : situation.kind === "stored" ? "stored" : undefined; const rejected = stored.lastFailure?.code === "http" && stored.lastFailure.status !== undefined && REJECTED_STATUSES.has(stored.lastFailure.status); const usable = source !== undefined && !rejected; return { + backend, kind: situation.kind, ...(source === undefined ? {} : { source }), path: situation.kind === "unusable" ? situation.path : credentialsPath(), @@ -137,19 +146,22 @@ export interface AuthReport { * state instead of reporting "enabled". */ export function describeAuth(state: AuthState = authState()): AuthReport { + const config = backendConfig(state.backend ?? DEFAULT_BACKEND); + const label = `${config.label} key`; const since = state.lastFailure ? ` Last failure: ${state.lastFailure.message}${state.lastFailure.at ? ` (${state.lastFailure.at})` : ""}` : ""; if (state.kind === "missing") { - return { level: "error", text: `TypeSafe key: missing — every Jev judgment is skipped until a key is configured (/typesafe login or TYPESAFE_API_KEY).${since}` }; + const how = usesTypesafeKey(config) ? `a key is configured (/typesafe login or ${TYPESAFE_KEY_ENV})` : `${config.keyEnv} is set in the environment`; + return { level: "error", text: `${label}: missing — every Jev judgment is skipped until ${how}.${since}` }; } if (state.kind === "unusable") { - return { level: "error", text: `TypeSafe key: unusable (${state.reason ?? "unknown reason"}) — judgments are skipped until the key is fixed.${since}` }; + return { level: "error", text: `${label}: unusable (${state.reason ?? "unknown reason"}) — judgments are skipped until the key is fixed.${since}` }; } const rejected = state.lastFailure?.code === "http" && state.lastFailure.status !== undefined && REJECTED_STATUSES.has(state.lastFailure.status); if (rejected) { - return { level: "error", text: `TypeSafe key: ${state.keyName} was rejected.${since}` }; + return { level: "error", text: `${label}: ${state.keyName} was rejected.${since}` }; } if (!state.verified) { - return { level: "warning", text: `TypeSafe key: ${state.keyName} (not verified yet — the first request proves it).${since}` }; + return { level: "warning", text: `${label}: ${state.keyName} (not verified yet — the first request proves it).${since}` }; } - return { level: "ok", text: `TypeSafe key: ${state.keyName} (verified${state.verifiedAt ? ` ${state.verifiedAt}` : ""}).${since}` }; + return { level: "ok", text: `${label}: ${state.keyName} (verified${state.verifiedAt ? ` ${state.verifiedAt}` : ""}).${since}` }; } diff --git a/src/backends.ts b/src/backends.ts new file mode 100644 index 0000000..84dcee8 --- /dev/null +++ b/src/backends.ts @@ -0,0 +1,40 @@ +import { TypeSafeIntegrationError } from "./errors.js"; + +export type TypeSafeBackend = "typesafe" | "openrouter"; + +export interface BackendConfig { + /** Human name for status lines: "TypeSafe", "OpenRouter". */ + label: string; + host: string; + /** The environment variable that carries this backend's key. Absent means the TypeSafe key resolution applies. */ + keyEnv?: string; + /** Request path, when the backend does not serve the SDK's own `/v1/systemone`. */ + path?: string; +} + +/** The backend every key and auth function assumes when none is named. */ +export const DEFAULT_BACKEND: TypeSafeBackend = "typesafe"; + +/** The environment variable and login store that the default backend reads. */ +export const TYPESAFE_KEY_ENV = "TYPESAFE_API_KEY"; + +/** Registry of known judgment backends. Extendable by callers. */ +export const DECISIONS_BACKENDS: Record = { + typesafe: { label: "TypeSafe", host: "https://api.typesafe.ai", keyEnv: TYPESAFE_KEY_ENV }, + openrouter: { label: "OpenRouter", host: "https://openrouter.ai", keyEnv: "OPENROUTER_API_KEY", path: "/api/alpha/decisions" }, +}; + +/** The registry entry for a backend name; a `configuration` error for a name the registry does not know. */ +export function backendConfig(name: TypeSafeBackend): BackendConfig { + const backend = DECISIONS_BACKENDS[name]; + if (!backend) throw new TypeSafeIntegrationError("configuration", `Unknown judgment backend "${name}". Valid backends: ${Object.keys(DECISIONS_BACKENDS).join(", ")}.`); + return backend; +} + +/** + * Whether a backend's key comes from the TypeSafe resolution (`TYPESAFE_API_KEY`, then the login store) or only from + * its own environment variable. Only the TypeSafe backend has a login store; every other backend is environment-only. + */ +export function usesTypesafeKey(backend: BackendConfig): boolean { + return (backend.keyEnv ?? TYPESAFE_KEY_ENV) === TYPESAFE_KEY_ENV; +} diff --git a/src/client.ts b/src/client.ts index 78c8606..985df03 100644 --- a/src/client.ts +++ b/src/client.ts @@ -1,6 +1,8 @@ import { TypeSafeClient } from "@typesafe-ai/sdk"; import type { Fetch, Questions, SystemOneRequest, SystemOneResult } from "@typesafe-ai/sdk"; import { recordAuthFailure, recordAuthVerified } from "./auth.js"; +import { DEFAULT_BACKEND, TYPESAFE_KEY_ENV, backendConfig, usesTypesafeKey } from "./backends.js"; +import type { TypeSafeBackend } from "./backends.js"; import type { BatchEvaluation, BatchOptions } from "./batch.js"; import { evaluateAll, evaluateMany } from "./batch.js"; import { keySituation } from "./credentials.js"; @@ -9,24 +11,12 @@ import { DEFAULT_MAX_INPUT_BYTES, assertWithinByteLimit, prepareEvaluationReques import { DEFAULT_USD_PER_MTOK, capsFromEnvironment, estimateUsd, mergeCaps, openUsageLedger } from "./usage.js"; import type { BlockedCap, SpendCaps, UsageLedger, UsageReport } from "./usage.js"; -export type TypeSafeBackend = "typesafe" | "openrouter"; - -export interface BackendConfig { - host: string; - keyEnv?: string; - /** Request path, when the backend does not serve the SDK's own `/v1/systemone`. */ - path?: string; -} +export { DECISIONS_BACKENDS, DEFAULT_BACKEND } from "./backends.js"; +export type { BackendConfig, TypeSafeBackend } from "./backends.js"; /** The path the SDK appends to whatever base URL it is given. */ const SDK_PATH = "/v1/systemone"; -/** Registry of known judgment backends. Extendable by callers. */ -export const DECISIONS_BACKENDS: Record = { - typesafe: { host: "https://api.typesafe.ai", keyEnv: "TYPESAFE_API_KEY" }, - openrouter: { host: "https://openrouter.ai", keyEnv: "OPENROUTER_API_KEY", path: "/api/alpha/decisions" }, -}; - /** Send the SDK's fixed path to the backend's own, preserving any caller-supplied transport. */ function backendFetch(path: string, inner: Fetch = fetch): Fetch { return (input, init) => inner(String(input).replace(SDK_PATH, path), init); @@ -155,34 +145,20 @@ function capsDescription(caps: SpendCaps): string { /** A bounded, server-side TypeSafe client independent of Pi's runtime. */ export function createTypeSafe(options: TypeSafeOptions = {}): TypeSafe { let apiKey = options.apiKey?.trim(); - // Resolve backend and host. - const backendName = options.backend; - let baseURL = "https://api.typesafe.ai"; - let keyEnv = "TYPESAFE_API_KEY"; - let backendPath: string | undefined; - if (backendName !== undefined) { - const backend = DECISIONS_BACKENDS[backendName]; - if (!backend) throw new TypeSafeIntegrationError("configuration", `Unknown judgment backend "${backendName}". Valid backends: ${Object.keys(DECISIONS_BACKENDS).join(", ")}.`); - baseURL = backend.host; - keyEnv = backend.keyEnv ?? "TYPESAFE_API_KEY"; - backendPath = backend.path; - } + const backendName: TypeSafeBackend = options.backend ?? DEFAULT_BACKEND; + const backend = backendConfig(backendName); + const baseURL = backend.host; + const backendPath = backend.path; if (!apiKey) { - // Backend-specific env var first (e.g. OPENROUTER_API_KEY), then fall back to - // the standard TYPESAFE_API_KEY / stored-key resolution. - if (backendName !== undefined && keyEnv !== "TYPESAFE_API_KEY") { - const fromEnv = process.env[keyEnv]?.trim(); - if (fromEnv) apiKey = fromEnv; - } - if (!apiKey) { - const situation = keySituation(); - if (situation.kind === "unusable") throw new TypeSafeIntegrationError("configuration", situation.reason); - if (situation.kind === "environment" || situation.kind === "stored") apiKey = situation.key; - } + // The same resolution that authState() and ensureApiKey() report, so the status line and the request agree. + const situation = keySituation(backendName); + if (situation.kind === "unusable") throw new TypeSafeIntegrationError("configuration", situation.reason); + if (situation.kind === "environment" || situation.kind === "stored") apiKey = situation.key; } if (!apiKey) { - throw new TypeSafeIntegrationError("configuration", `No API key. Run /typesafe login in Pi, or set ${keyEnv} in the environment.`); + const how = usesTypesafeKey(backend) ? `Run /typesafe login in Pi, or set ${TYPESAFE_KEY_ENV}` : `Set ${backend.keyEnv}`; + throw new TypeSafeIntegrationError("configuration", `No API key. ${how} in the environment.`); } const timeout = positiveInteger(options.timeoutMs ?? 15_000, "timeoutMs"); const maxInputBytes = positiveInteger(options.maxInputBytes ?? DEFAULT_MAX_INPUT_BYTES, "maxInputBytes"); diff --git a/src/credentials.ts b/src/credentials.ts index 6c591ba..e8a379a 100644 --- a/src/credentials.ts +++ b/src/credentials.ts @@ -1,13 +1,16 @@ import { chmodSync, mkdirSync, readFileSync, renameSync, rmSync, statSync, writeFileSync } from "node:fs"; import { homedir } from "node:os"; import { dirname, join } from "node:path"; +import { DEFAULT_BACKEND, TYPESAFE_KEY_ENV, backendConfig, usesTypesafeKey } from "./backends.js"; +import type { TypeSafeBackend } from "./backends.js"; import { TypeSafeIntegrationError } from "./errors.js"; export type KeySource = "environment" | "stored"; /** The complete, never-throwing answer to "which key is in effect". */ export type KeySituation = - | { readonly kind: "environment"; readonly key: string } + /** `keyEnv` names the variable that was read; absent means `TYPESAFE_API_KEY`. */ + | { readonly kind: "environment"; readonly key: string; readonly keyEnv?: string } | { readonly kind: "stored"; readonly key: string; readonly path: string } | { readonly kind: "missing" } | { readonly kind: "unusable"; readonly path: string; readonly reason: string }; @@ -56,12 +59,17 @@ export function readStoredApiKey(): string | undefined { } /** - * What the environment, the login store, and file permissions add up to right now. Never throws; the "unusable" kind - * carries the user-facing reason (a stored key that other local users can read). + * What the environment, the login store, and file permissions add up to right now for one judgment backend. Never + * throws; the "unusable" kind carries the user-facing reason (a stored key that other local users can read). + * The TypeSafe backend reads `TYPESAFE_API_KEY`, then the login store. Every other backend reads only its own + * environment variable, because the store holds a TypeSafe key and a login verifies against api.typesafe.ai. */ -export function keySituation(): KeySituation { - const fromEnvironment = process.env.TYPESAFE_API_KEY?.trim(); - if (fromEnvironment) return { kind: "environment", key: fromEnvironment }; +export function keySituation(backend: TypeSafeBackend = DEFAULT_BACKEND): KeySituation { + const config = backendConfig(backend); + const keyEnv = config.keyEnv ?? TYPESAFE_KEY_ENV; + const fromEnvironment = process.env[keyEnv]?.trim(); + if (fromEnvironment) return { kind: "environment", key: fromEnvironment, keyEnv }; + if (!usesTypesafeKey(config)) return { kind: "missing" }; const path = credentialsPath(); try { const key = readStoredApiKey(); @@ -76,7 +84,7 @@ export function keySituation(): KeySituation { /** Short phrase naming the source; full sentences stay with the caller. */ export function keySourceLabel(situation: KeySituation): string { switch (situation.kind) { - case "environment": return "TYPESAFE_API_KEY"; + case "environment": return situation.keyEnv ?? TYPESAFE_KEY_ENV; case "stored": return "/typesafe login"; case "missing": return "no key"; case "unusable": return "unusable key"; @@ -89,8 +97,8 @@ export function keySourceLabel(situation: KeySituation): string { * store must not be read. New code should call keySituation() instead: same precedence, never throws, and the * "must not be read" case arrives as `unusable` with the reason. */ -export function resolveApiKey(): { key: string; source: KeySource } | undefined { - const situation = keySituation(); +export function resolveApiKey(backend: TypeSafeBackend = DEFAULT_BACKEND): { key: string; source: KeySource } | undefined { + const situation = keySituation(backend); if (situation.kind === "unusable") throw new TypeSafeIntegrationError("configuration", situation.reason); return situation.kind === "environment" || situation.kind === "stored" ? { key: situation.key, source: situation.kind } : undefined; } diff --git a/src/index.ts b/src/index.ts index bcb3282..b362610 100644 --- a/src/index.ts +++ b/src/index.ts @@ -1,4 +1,4 @@ -export { createTypeSafe, DEFAULT_MAX_REQUESTS, DECISIONS_BACKENDS } from "./client.js"; +export { createTypeSafe, DEFAULT_MAX_REQUESTS, DECISIONS_BACKENDS, DEFAULT_BACKEND } from "./client.js"; export type { TypeSafe, TypeSafeOptions, EvaluationOptions, Evaluation, UsageSnapshot, SpendReport, TypeSafeBackend, BackendConfig, diff --git a/src/login.ts b/src/login.ts index 004c069..66b99ce 100644 --- a/src/login.ts +++ b/src/login.ts @@ -1,4 +1,6 @@ import type { ExtensionCommandContext } from "@earendil-works/pi-coding-agent"; +import { DEFAULT_BACKEND, backendConfig, usesTypesafeKey } from "./backends.js"; +import type { TypeSafeBackend } from "./backends.js"; import { createTypeSafe } from "./client.js"; import { keySituation, normalizeApiKey, storeApiKey } from "./credentials.js"; import type { KeySource } from "./credentials.js"; @@ -13,9 +15,10 @@ export interface LoginResult { } /** - * Prompt for a key (hidden input), verify it with a model listing, and store it for every pi-typesafe consumer. - * Resolves to undefined when the user cancels. Rejects with TypeSafeIntegrationError for an invalid or unverifiable key. - * Refuses when TYPESAFE_API_KEY is set, because the environment would shadow the stored key. + * Prompt for a TypeSafe key (hidden input), verify it against api.typesafe.ai with a model listing, and store it for + * every pi-typesafe consumer. Resolves to undefined when the user cancels. Rejects with TypeSafeIntegrationError for an + * invalid or unverifiable key. Refuses when TYPESAFE_API_KEY is set, because the environment would shadow the stored + * key. Other backends have no login: their key is an environment variable only. */ export async function loginWithPrompt(ctx: ExtensionCommandContext): Promise { if (process.env.TYPESAFE_API_KEY?.trim()) { @@ -37,14 +40,21 @@ export type EnsureApiKeyResult = | { source: "stored"; login: LoginResult }; /** - * Use the configured key if there is one; otherwise run the login prompt. `undefined` means the user cancelled. - * A store that must not be read throws `configuration` with the reason instead of prompting, so a permissions - * problem stays visible; the result shape is frozen for existing callers. + * Use the configured key for the backend if there is one; otherwise run the login prompt. `undefined` means the user + * cancelled. A store that must not be read throws `configuration` with the reason instead of prompting, so a + * permissions problem stays visible; the result shape is frozen for existing callers. A backend without a login store + * (every backend except TypeSafe) throws `configuration` naming its environment variable instead of prompting, because + * the prompt would verify the key against the wrong service and store it where the TypeSafe key lives. */ -export async function ensureApiKey(ctx: ExtensionCommandContext): Promise { - const situation = keySituation(); +export async function ensureApiKey(ctx: ExtensionCommandContext, options: { backend?: TypeSafeBackend } = {}): Promise { + const backend = options.backend ?? DEFAULT_BACKEND; + const situation = keySituation(backend); if (situation.kind === "environment" || situation.kind === "stored") return { source: situation.kind }; if (situation.kind === "unusable") throw new TypeSafeIntegrationError("configuration", situation.reason); + const config = backendConfig(backend); + if (!usesTypesafeKey(config)) { + throw new TypeSafeIntegrationError("configuration", `No ${config.label} key. Set ${config.keyEnv} in the environment; /typesafe login stores a TypeSafe key only.`); + } const login = await loginWithPrompt(ctx); return login ? { source: "stored", login } : undefined; } diff --git a/tests/auth.test.ts b/tests/auth.test.ts index 5fe2a44..6456360 100644 --- a/tests/auth.test.ts +++ b/tests/auth.test.ts @@ -30,6 +30,34 @@ test("no key at all is an error the consumer cannot mistake for a working setup" assert.ok(report.text.includes("every Jev judgment is skipped")); }); +test("another backend reports its own key and never the TypeSafe login hint", () => { + const savedOpenRouter = process.env.OPENROUTER_API_KEY; + try { + delete process.env.OPENROUTER_API_KEY; + process.env.TYPESAFE_API_KEY = "env-key-0123456789abcdef"; + assert.equal(authState().backend, "typesafe"); + const missing = authState({ backend: "openrouter" }); + assert.equal(missing.backend, "openrouter"); + assert.equal(missing.kind, "missing"); + assert.equal(missing.usable, false); + const report = describeAuth(missing); + assert.equal(report.level, "error"); + assert.ok(report.text.startsWith("OpenRouter key: missing")); + assert.ok(report.text.includes("OPENROUTER_API_KEY is set")); + assert.ok(!report.text.includes("/typesafe login")); + + process.env.OPENROUTER_API_KEY = "sk-or-0123456789abcdef"; + const present = authState({ backend: "openrouter" }); + assert.equal(present.kind, "environment"); + assert.equal(present.keyName, "OPENROUTER_API_KEY"); + assert.equal(present.usable, true); + assert.ok(describeAuth(present).text.startsWith("OpenRouter key: OPENROUTER_API_KEY")); + } finally { + delete process.env.TYPESAFE_API_KEY; + if (savedOpenRouter === undefined) delete process.env.OPENROUTER_API_KEY; else process.env.OPENROUTER_API_KEY = savedOpenRouter; + } +}); + test("an environment key is usable but unverified until something proves it", () => { process.env.TYPESAFE_API_KEY = "env-key-0123456789abcdef"; try { diff --git a/tests/client.test.ts b/tests/client.test.ts index 13044b6..1edcf97 100644 --- a/tests/client.test.ts +++ b/tests/client.test.ts @@ -311,6 +311,25 @@ test("openrouter backend uses default model typesafe/jev-1.13", async () => { assert.equal(sentModel, "typesafe/jev-1.13"); }); +test("a TypeSafe key is never sent to another backend", () => { + const originalKey = process.env.TYPESAFE_API_KEY; + const originalOR = process.env.OPENROUTER_API_KEY; + process.env.TYPESAFE_API_KEY = "ts-test-key-1234567890123456"; + delete process.env.OPENROUTER_API_KEY; + try { + assert.throws(() => createTypeSafe({ backend: "openrouter" }), (error: unknown) => { + assert.ok(error instanceof TypeSafeIntegrationError); + assert.equal(error.code, "configuration"); + assert.match(error.message, /OPENROUTER_API_KEY/); + assert.doesNotMatch(error.message, /typesafe login/); + return true; + }); + } finally { + if (originalKey === undefined) delete process.env.TYPESAFE_API_KEY; else process.env.TYPESAFE_API_KEY = originalKey; + if (originalOR === undefined) delete process.env.OPENROUTER_API_KEY; else process.env.OPENROUTER_API_KEY = originalOR; + } +}); + test("key resolution picks the right env var per backend", () => { // With no key set, openrouter backend should complain about OPENROUTER_API_KEY. const originalKey = process.env.TYPESAFE_API_KEY; diff --git a/tests/credentials.test.ts b/tests/credentials.test.ts index 7875279..2750a2d 100644 --- a/tests/credentials.test.ts +++ b/tests/credentials.test.ts @@ -104,15 +104,40 @@ test("keySituation is total and names every kind", { skip: process.platform === assert.throws(() => resolveApiKey(), (error: unknown) => error instanceof TypeSafeIntegrationError && error.message === situation.reason); process.env.TYPESAFE_API_KEY = ` ${validKey} `; - assert.deepEqual(keySituation(), { kind: "environment", key: validKey }); + assert.deepEqual(keySituation(), { kind: "environment", key: validKey, keyEnv: "TYPESAFE_API_KEY" }); // Environment values are trusted as-is; a wrong key fails at the API with its own advice. process.env.TYPESAFE_API_KEY = "short"; - assert.deepEqual(keySituation(), { kind: "environment", key: "short" }); + assert.deepEqual(keySituation(), { kind: "environment", key: "short", keyEnv: "TYPESAFE_API_KEY" }); +}); + +test("keySituation for another backend reads only that backend's variable, never the TypeSafe store", () => { + const savedOpenRouter = process.env.OPENROUTER_API_KEY; + try { + delete process.env.OPENROUTER_API_KEY; + storeApiKey(validKey); + process.env.TYPESAFE_API_KEY = validKey; + // A stored or TypeSafe-environment key is not an OpenRouter key. + assert.deepEqual(keySituation("openrouter"), { kind: "missing" }); + assert.equal(resolveApiKey("openrouter"), undefined); + + process.env.OPENROUTER_API_KEY = " sk-or-test-0123456789abcdef "; + const situation = keySituation("openrouter"); + assert.deepEqual(situation, { kind: "environment", key: "sk-or-test-0123456789abcdef", keyEnv: "OPENROUTER_API_KEY" }); + assert.equal(keySourceLabel(situation), "OPENROUTER_API_KEY"); + assert.deepEqual(resolveApiKey("openrouter"), { key: "sk-or-test-0123456789abcdef", source: "environment" }); + // The OpenRouter variable does not leak into the TypeSafe resolution either. + delete process.env.TYPESAFE_API_KEY; + assert.equal(keySituation().kind, "stored"); + assert.throws(() => keySituation("bogus" as never), (error: unknown) => error instanceof TypeSafeIntegrationError && error.code === "configuration" && /Unknown judgment backend/.test(error.message)); + } finally { + if (savedOpenRouter === undefined) delete process.env.OPENROUTER_API_KEY; else process.env.OPENROUTER_API_KEY = savedOpenRouter; + } }); test("keySourceLabel names each source", () => { assert.equal(keySourceLabel({ kind: "environment", key: "k" }), "TYPESAFE_API_KEY"); + assert.equal(keySourceLabel({ kind: "environment", key: "k", keyEnv: "OPENROUTER_API_KEY" }), "OPENROUTER_API_KEY"); assert.equal(keySourceLabel({ kind: "stored", key: "k", path: "/tmp/auth.json" }), "/typesafe login"); assert.equal(keySourceLabel({ kind: "missing" }), "no key"); assert.equal(keySourceLabel({ kind: "unusable", path: "/tmp/auth.json", reason: "r" }), "unusable key"); diff --git a/tests/login.test.ts b/tests/login.test.ts index 9cbe25c..329a535 100644 --- a/tests/login.test.ts +++ b/tests/login.test.ts @@ -81,3 +81,21 @@ test("ensureApiKey reports an existing key without prompting, otherwise logs in" assert.deepEqual(await ensureApiKey(ctx()), { source: "stored" }, "second call reuses the stored key"); assert.equal(modelListCalls, 1); }); + +test("ensureApiKey for another backend uses its environment variable and never opens the TypeSafe login", async () => { + const savedOpenRouter = process.env.OPENROUTER_API_KEY; + try { + delete process.env.OPENROUTER_API_KEY; + // A stored TypeSafe key does not satisfy OpenRouter, and the prompt must not run: it would verify against api.typesafe.ai. + customResult = "ts_live_key_0123456789abcdef"; + assert.deepEqual(await ensureApiKey(ctx()), { source: "stored", login: { path: storedPath(), models: 2 } }); + await assert.rejects(ensureApiKey(ctx(), { backend: "openrouter" }), (error: unknown) => error instanceof TypeSafeIntegrationError && error.code === "configuration" && /OPENROUTER_API_KEY/.test(error.message)); + assert.equal(modelListCalls, 1, "no second verification"); + + process.env.OPENROUTER_API_KEY = "sk-or-0123456789abcdef"; + assert.deepEqual(await ensureApiKey(ctx(), { backend: "openrouter" }), { source: "environment" }); + assert.equal(modelListCalls, 1); + } finally { + if (savedOpenRouter === undefined) delete process.env.OPENROUTER_API_KEY; else process.env.OPENROUTER_API_KEY = savedOpenRouter; + } +}); From 3f64f490b736ff036ac454de7692bf911686086b Mon Sep 17 00:00:00 2001 From: Ryan Gapac Date: Tue, 22 Sep 2026 22:16:45 +0800 Subject: [PATCH 2/2] chore: version bump to 0.7.0, finalize CHANGELOG --- CHANGELOG.md | 4 ++++ package-lock.json | 4 ++-- package.json | 2 +- 3 files changed, 7 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5c23ca9..1056af8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,10 @@ ## Unreleased + + +## 0.7.0 + ### Fixed - Key reporting and login can name the judgment backend: `keySituation(backend)`, `resolveApiKey(backend)`, `authState({ backend })`, and `ensureApiKey(ctx, { backend })` read the backend's own environment variable, `describeAuth` labels the key by backend and names the variable to set, and `AuthState` carries `backend`. Before, every surface reported the TypeSafe key, so an OpenRouter user saw "TypeSafe key: missing" while judgments ran, and `ensureApiKey` opened the TypeSafe login and verified the pasted key against api.typesafe.ai (#9). diff --git a/package-lock.json b/package-lock.json index 33c1d1b..a9089fe 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "pi-typesafe", - "version": "0.6.2", + "version": "0.7.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "pi-typesafe", - "version": "0.6.2", + "version": "0.7.0", "license": "MIT", "dependencies": { "@typesafe-ai/sdk": "^0.6.0", diff --git a/package.json b/package.json index 9292521..f89eba9 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "pi-typesafe", - "version": "0.6.2", + "version": "0.7.0", "description": "TypeSafe AI (Jev) decisions for Pi: batched Choice/Score/Noul evaluation tool, terminal playground, and a typed API other extensions build on.", "type": "module", "license": "MIT",