feat: configurable judgment endpoint (custom base URL, key and model id) instead of two hardcoded backends

[ocean-flux]

Summary

This is about which host serves the Jev model, not about replacing Jev with another model. Two different things get mixed together and they should stay apart:

1. Jev from another source (this issue). Exactly two hosts are baked into the code today. A third host that serves the same Jev model -- a partner or reseller, an internal gateway in front of api.typesafe.ai, an egress-controlled corporate deployment -- cannot be used without a code change and a release.
2. A different model acting as judge (that is Use Laya and other fast local models instead of paid APIs #34). A local model (Ollama, llama.cpp, MLX, vLLM) or any OpenAI-compatible provider answering the guards' questions instead of Jev. That is not "another source of Jev": it is a different model doing the job, it needs a protocol adapter, and feat: laya-mlx local judge backend #28 showed the quality risk is real.

I am asking for (1) first, with a config shape that does not preclude (2).

A working example: Command Code already serves Jev

Verified 2026-09-28 from https://commandcode.ai/models/jev:

• Model id: typesafe/jev
• Endpoint: POST https://api.commandcode.ai/provider/v1/systemone with Authorization: Bearer <Command Code key>
• Pricing: $0.04/M input tokens at cost; output is free.
• Plans: "Available on GOAT and above, the plans with Provider API access."
• Not listed in the chat-model catalog, by design: "Jev is a decision model, not a chat model. It answers typed questions about a state with probabilities and never produces text ... It is not in /model and cannot drive an interactive session."

The wire shape is the same one @typesafe-ai/sdk already speaks (/v1/systemone, noul/choice/score questions):

curl https://api.commandcode.ai/provider/v1/systemone \
  -H "Authorization: Bearer $COMMANDCODE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "typesafe/jev", "state": "Payments failed for three days.",
       "questions": {"urgent": {"type": "noul", "instructions": "Does this need urgent attention?"}}}'
# {"model":"typesafe/jev","answers":{"urgent":{"type":"noul","noul":0.95}},
#  "usage":{"input_tokens":41,"output_tokens":12}}

So this is not a protocol change: pi-typesafe already rewrites the SDK's fixed paths per backend (backendFetch with the registry's path), which is how OpenRouter works. Serving Jev through Command Code should be a registry entry, not an adapter:

commandcode: {
    label: "Command Code",
    host: "https://api.commandcode.ai",
    keyEnv: "COMMANDCODE_API_KEY",
    path: "/provider/v1/systemone",
}

That is the whole argument of this issue: the model is already reachable, and the only reason a Pi user with a Command Code plan cannot use it is that host and keyEnv are constants.

Current state (0.73.0)

• ~/.pi/agent/pi-warden/config.json typesafeBackend accepts two values; anything else silently falls back to typesafe (pi-warden/dist/backend.js, resolveBackend).

• The registry is hardcoded in pi-typesafe/dist/backends.js:

  typesafe:   { label: "TypeSafe",   host: "https://api.typesafe.ai", keyEnv: TYPESAFE_KEY_ENV },   // model: jev-latest
  openrouter: { label: "OpenRouter", host: "https://openrouter.ai",   keyEnv: "OPENROUTER_API_KEY",
                path: "/api/alpha/decisions", modelsPath: "/api/v1/models",
                modelsField: "data", modelsIdField: "id", modelsVerifyKey: false }                     // model: typesafe/jev-1.13

  host and keyEnv are per-entry constants, and there is no config path to the registry (DECISIONS_BACKENDS is exported and its comment says "Extendable by callers", but nothing reads it from config).

• Requests go through @typesafe-ai/sdk's TypeSafeClient, which appends fixed paths (/v1/systemone, /v1/models); backendFetch rewrites them per backend.

• The consent text is built from the host: pi-warden/dist/backend.js reads DECISIONS_BACKENDS[backend].host and substitutes the destination into the disclosure. A configurable host has to flow into that text and into /warden status.

Proposal

Let the user file name the endpoint:

{
  "typesafeBackend": {
    "label": "commandcode",
    "baseUrl": "https://api.commandcode.ai",
    "path": "/provider/v1/systemone",
    "apiKeyEnv": "COMMANDCODE_API_KEY",
    "model": "typesafe/jev"
  }
}

The invariants that already exist for this key must hold:

• User file only. A project file must not be able to redirect judgments (the same rule mode and typesafeBackend have today). Otherwise a checked-out repository could aim the judge at a host it controls and read everything the guards send.
• Consent names the real destination. The disclosure text and /warden status are built from the backend host today; a configurable host must appear there.
• The answer is still validated. A configurable endpoint does not relax the Choice/Score/Noul schema check, and a malformed reply must fail the way a network error does (fail-open per action.failOpen, with the existing judge cooldown and the one-time notice).

One detail worth deciding while adding a third entry: modelsVerifyKey. A key check against a model list proves something only when the list is private. Command Code's /provider/v1/models answers anonymously (82 chat models, verified), so a key check against it would pass with a bad key -- and typesafe/jev is not in that list at all, so the list cannot confirm that the configured model is reachable either.

If (2) is picked up later

The same config could grow an api: "openai-completions" field plus a model id for a model that is not Jev. That is #34's scope and it needs the adapter described there: build the prompt from the question set, ask for JSON, validate it. Two notes for that path:

• Keep the two cases visibly different in the status line and the trace, so it is clear whether Jev or a stand-in answered.
• A per-backend guard scope may be the honest default (a small local model should not decide holds), with the per-rule threshold: header already available to compensate.

Open questions

• Is the decisions protocol documented for third parties (Command Code's own docs describe it for one host)? If it is not, a third host has to reverse-engineer it, and (2) becomes the only open path.
• Should the registry become extensible at runtime, or should a config value be the only route? The latter is easier to keep user-file-only, which is the invariant that matters.

Alternatives

• Add hosts one at a time to DECISIONS_BACKENDS (what feat: add OpenRouter as a configurable judgment backend #12 did for OpenRouter). Each addition needs a code change and a release in pi-typesafe, and a private gateway can never be covered this way.
• Run the model inside the extension (feat: laya-mlx local judge backend #28, laya-mlx). Pins one model, one platform and one runtime, and makes the extension carry model downloads and a Python bridge.

Related: #34 (a different model as judge, open), #12 (OpenRouter added as a backend, closed), #28 (laya-mlx local backend, closed).

Correction: an earlier revision of this issue said Command Code does not serve Jev. That was wrong -- I had checked the chat-model catalog, where decision models are deliberately absent.

[DevMortimer]

This is in pi-warden 0.74.0 with pi-typesafe 0.8.0 (#136). typesafeBackend in the user config (~/.pi/agent/pi-warden/config.json) now takes "commandcode" or an endpoint object, so you can start your own server and point the backend at it. For example, with upstream Laya's server:

pip install "laya[serve]"
LAYA_HOST=127.0.0.1 LAYA_API_KEY=<any-secret> laya-serve

{
  "typesafeBackend": {
    "label": "Laya (local)",
    "host": "http://127.0.0.1:8000",
    "path": "/v1/systemone",
    "keyEnv": "LAYA_API_KEY",
    "defaultModel": "english"
  }
}

A project's .pi/pi-warden.json cannot set it, /warden status and the consent text name the host, and an invalid value turns judgments off with a notice instead of falling back to TypeSafe.

From my findings, however: I ran pi-warden's own judge benches through this path, against Laya on an M5 (laya-mlx) and on a Linux machine (upstream laya-serve 0.3.21). Both gave the same answers:

Guard	Jev	Laya
Rules: violations caught at the 0.7 cutoff (161 cases)	91%	0% (18% even at 0.3)
Stuck: accuracy (40 cases)	0.99	0.53 (calls every case stuck)
Done: accuracy (40 cases)	0.85–0.88	0.68–0.75

The multilingual checkpoint with an 8,192-token limit did not change the rules result, so the input length is not the cause.

Speed on the Linux machine: about 190 ms per rules request (about 3,300 tokens, 12 questions), 90 ms per stuck request and 50–76 ms per done request, against 245–600 ms for Jev. 1,042 requests ran in 117 s with no failures.

As I said in #28, it's... not good. Laya's only positive, in my opinion, is speed. The endpoint setting is there for anyone who wants to try another server, but I am not recommending Laya for the guards.