RFC: llm_route — suite-wide local-first model routing with auditable escalation

[bedardandy]

Problem

Across the 16-repo suite there are 9 different base-URL env conventions
(MCF_LLM_ENDPOINTS, AUDIT_QWEN_BASE_URL, ROUTER_BASE_URL, HALLUCHECK_BASE_URL,
INSPECTOR_BASE_URL, a hardcoded VLM_API_BASE, LOCAL_VL_ENDPOINTS,
OPENAI_BASE_URL, ANTHROPIC_BASE_URL), 20+ hardcoded model-name literals, and the
OpenAI-compatible client is hand-rolled ≥6 times. There is no shared abstraction for
picking a model per task or for escalating when a cheap/local model is likely wrong —
each repo reinvents retry-on-empty, enum-reject, and consensus logic, and a model choice
is never recorded as an auditable decision.

Design

A new optional, stdlib-only module maine_forms_engine.llm_route
(urllib / json / dataclasses — the sibling maine-forms-router proves the pattern)
that adds zero dependencies:

• Canonical env trio LLM_ROUTE_<TIER>_BASE_URL / _MODEL / _API_KEY with tiers
  LOCAL and FRONTIER (extensible), falling back to OPENAI_BASE_URL/OPENAI_MODEL/
  OPENAI_API_KEY so single-endpoint users need no new env. No hardcoded hosts; any
  OpenAI-compatible endpoint (Ollama / vLLM / llama.cpp / LiteLLM / any cloud).
• TaskClass{EXTRACT,CLASSIFY,DRAFT,VERIFY}, Signals, ModelTier, Router.
• Router.choose(task, signals) — LOCAL-first for EXTRACT/CLASSIFY/structured; escalate
  one rung per prior_failures>0, ambiguity>0.7, context overflow, or vision-without-
  local-vision. VERIFY starts at FRONTIER (high-liability path — mirrors the suite's
  Qwen-draft → Opus-adjudicate split).
• Router.complete(...) — walks the ladder; a caller-supplied validate(text)->bool
  failure triggers escalation (generalizes retry-on-empty + enum-reject); budget guard
  (max escalations + optional cost ceiling); never silent — the Completion carries
  tier_used, attempts, escalated, and a per-hop route_log. Unconfigured tier =
  skipped with a recorded reason; all tiers unconfigured = raise NoTierConfigured
  (the caller owns its deterministic fallback).
• on_decision callback designed to plug straight into
  legal_logic_layer.Ledger.record(source="llm", ...) — its schema already has an
  llm_inference backing type and an unused "route" applies-to kind. No import of
  legal_logic_layer (stays zero-dep); the callback is the only seam.

Full rationale, the 9-convention inventory table, the escalation-signal catalog, the
ledger wiring, and the migration plan are in
docs/model-routing.md. The doc notes the module can graduate
to its own repo if maintainers prefer.

What this deliberately does NOT do

• No provider SDKs — stdlib urllib, OpenAI-compatible chat-completions only.
• No change to deterministic fill — nothing in maine_forms_engine.fill imports this;
  the engine still consults no model at fill time.
• No hardcoded hosts / private-infra references; no hidden fallback; no streaming /
  function-calling / token accounting in this cut (additive, out of scope).

Migration plan (effort)

1. maine-forms-router (S) → 2. maine-government-feeds (S, also off raw httpx) →
2. LLM_Hallucination_Checker (M, reference VERIFY route, keep fail-closed) →
3. corp audit tools (M) → 5. maine-probate-forms (L, un-hardcode config.py) →
4. maine-court-forms (L, most call sites).

Tests

24 new tests in tests/test_llm_route.py (urllib mocked — no network): choose() policy
matrix, escalation-on-validate-fail, budget stop, all-unconfigured raise, unconfigured-tier
skip, transport-error escalation, on_decision / ledger-shaped wiring, and route_log
completeness. Full suite green: 145 passed, 2 skipped (existing + new).

Request for maintainer feedback

This is framed as an RFC. Specifically seeking a call on: (a) live here as an optional
engine module vs. graduate to a standalone llm-route package; (b) the LOCAL/FRONTIER
tier names and the canonical env trio; (c) whether VERIFY-starts-at-FRONTIER is the right
default; (d) the ledger callback shape before consumers wire it in.

Part of the 2026-07-06 suite-wide audit.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 088ed06e3c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Apply signal-driven escalation only once per signal

When an EXTRACT/CLASSIFY call has a tier-independent signal such as prior_failures=1 on a custom ladder like [LOCAL, MID, FRONTIER], this loop re-evaluates the same pressure after moving to MID and advances again, so MID is skipped entirely. Resetting start afterward also makes the successful FRONTIER completion report escalated=False with no route-log entry explaining why lower tiers were bypassed, which breaks the advertised one-rung, auditable escalation behavior for extensible ladders.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Keep budget-stop route log entries schema-compatible

When validation or transport failures exhaust the escalation budget, _advance appends and emits this stop record via on_decision, but it omits the standard hop fields (task, index, configured, validated, cost). Consumers are told every decision dict has those keys, so a ledger/audit callback that works for normal hops can raise KeyError only on the budget-stop path.

Useful? React with 👍 / 👎.