jevmlx

Typed decisions from a local model on Apple Silicon. One batched forward pass, every field at once, with a probability per field.

Asking an LLM for JSON means parsing what it wrote and retrying until it parses. jevmlx takes a schema of booleans, enums, and multi-selects, scores every allowed answer for every field in one forward pass, and assembles the JSON itself — valid by construction, every field with a probability.

pip install git+https://github.com/bnsd55/jevmlx
jevmlx decide --preset fintech_fraud --json

{
  "is_fraudulent": {"value": true, "prob": 0.6077},
  "risk_tier": {"value": "HIGH", "prob": 0.8392},
  "recommended_action": {"value": "CHALLENGE_2FA", "prob": 0.8891}
}

Install

pip install git+https://github.com/bnsd55/jevmlx   # library
uv tool install git+https://github.com/bnsd55/jevmlx   # CLI only
git clone https://github.com/bnsd55/jevmlx && cd jevmlx && ./setup.sh   # dev

Requires an Apple Silicon Mac (M1 or later) and Python 3.12+.

Model aliases

Alias	Resolves to	Use
quality	mlx-community/Qwen2.5-7B-Instruct-4bit	default — best accuracy
fast	mlx-community/Qwen2.5-3B-Instruct-4bit	lower latency
test	mlx-community/Qwen2.5-1.5B-Instruct-4bit	tests only (too small for production)

jevmlx decide --model quality --schema ticket.json --context ticket.txt   # or --model fast

A full Hub id also works (--model mlx-community/Llama-3.2-3B-Instruct-4bit); first use downloads the weights (~4.5 GB quality, ~2 GB fast).

Use it from Python

from typing import Literal
from pydantic import BaseModel, Field
from jevmlx import decide


class Ticket(BaseModel):
    urgent: bool
    category: Literal["BILLING", "TECHNICAL", "FEEDBACK"] = Field(
        description="What the ticket is about",
        json_schema_extra={"choice_descriptions": {"BILLING": "invoices, charges, refunds"}},
    )
    tags: list[Literal["refund", "login", "performance"]] = []


result = decide(Ticket, "Customer was charged twice and wants the duplicate refunded.")
for name, f in result.fields.items():
    print(f"{name}: {f.value} (p={f.probability})")

Read the result. decide(...) returns a Decision: .value (a validated Ticket), .latency_ms, .fields — one FieldResult per field:

• value, probability, alternatives (top 3, winner first; multi: per-option P(yes)), score, model ("slots"/"labels"), calibrated, legal_mass (probability mass in allowed continuations at the branch points — a leakage signal when low despite a confident decision).
• Margins, one per field: log_score_margin / probability_margin (scalar, top1-top2 gap in log/probability units), threshold_distance (multi, how close the closest yes/no call sat to the cut). Multi fields carry probability=None — only per-option decisions are claimed.
• reason: None, "none_of_above" (allow_none_of_above=True adds an explicit NONE_OF_ABOVE choice that maps to None; fields must be Optional), or "abstain" (abstain_below_margin=X withholds fields whose margin falls below the cut; the raw value stays on the FieldResult).
• Options: decide_many(model_cls, contexts) for batches (one prior pass, one merged scoring pass per context group — results match separate decide calls within PARITY_ATOL); constraints=[...] reconciles the joint answer by constrained MAP (implies / excludes / requires_parent / exclusivity; plus depends_on / set_constraints in the schema); calibration=<path-or-dict> (from jevmlx calibrate --out) selects multi options by fitted log-odds with the always-on count row reconciling to top-k above COUNT_MARGIN_MIN (0.7 nats); prior_correction=True subtracts the neutral-context prior; timing: latency_ms here, the full split (prior / prefill / plan compile / cache broadcast / suffix eval / lm-head gather / second pass) plus peak_active_bytes/peak_incremental_bytes and failed_attempts on the engine result, per-combo timing.json in bench output.

Use it from any OpenAI-compatible server

Instead of loading a model on this Mac, jevmlx can send the same prompts to a chat server that returns logprobs: Ollama, oMLX, MTPLX, vLLM.

jevmlx decide --backend openai --base-url http://localhost:11434/v1 --api-model llama3.2 --schema ticket.json --context ticket.txt

Two tradeoffs: one request per field (slower than one pass), and only the server's top-k logprobs are visible — options missing from that list get a floor probability and the telemetry flags truncated: true.

Leaderboard

Agreement with the TypeSafe public eval consensus. Official rows are cited from TypeSafe's page; local rows are measured by contributors on the 20 public examples.

Model	Source	Scorer	Machine	Accuracy	Customer service	Agent trace	Security	Invoices	Time per case	Cost per case	Cases
TypeSafe official (cited, retrieved 2026-09-17)
Jev	official (cited)	—	—	67.8%	76.0%	71.6%	61.7%	61.8%	0.4s	$0.0004	—
GPT-5.6 Terra	official (cited)	—	—	67.9%	—	—	—	—	10.1s	$0.0304	—
Claude Sonnet 5	official (cited)	—	—	67.8%	—	—	—	—	78.1s	$0.1174	—
Claude Opus 5	official (cited)	—	—	73.1%	—	—	—	—	37.8s	$0.1761	—
GPT-5.6 Sol	official (cited)	—	—	74.1%	—	—	—	—	23.3s	$0.0836	—
Claude Haiku 4.5	official (cited)	—	—	53.6%	—	—	—	—	12.5s	$0.0195	—

Official accuracies: TypeSafe's full private eval; ours: the 20 public examples — indicative, not the same test. Consensus = GPT-6 Astra + Claude Fable 5.1.

No local results yet — contribute one with jevmlx bench.

Run the benchmark on your Mac

git clone https://github.com/bnsd55/jevmlx && cd jevmlx && ./setup.sh
.venv/bin/jevmlx bench --model quality   # commit the results folder, open a PR

BENCHMARKING.md has the model list, what the command does, and the PR checklist.

CLI

Command	What it does
decide	Decide a preset or schema + context, print the JSON with probabilities; --model fast/quality/…, --scoring slots/labels, --constraints, --calibration, --prior-correction, --backend openai
serve	Serve decisions over HTTP (POST /decide, one Metal GPU, serial)
validate	Lint a schema for engine-visible problems (no model download)
calibrate	Fit a temperature + pooled multi calibrator on labeled JSONL, report ECE, --out writes what decide --calibration reads
eval	Run labeled cases through a track (parallel/naive_local/api_baseline/openai_slots), with optional permutations; writes predictions.jsonl + run.json + per-combo timing.json
report	Build a JSON + markdown eval report from predictions.jsonl (offline)
bench	Full benchmark: all (track, scorer, dataset) combos for one or more models, parity gate + one SUMMARY.md
doctor	Environment checks (platform, versions, venv/conda + subprocess hang, editable-install checkout, memory, power, Metal, model cache, network): run before filing an issue or a bench run

-v for progress logs; JEVMLX_LOG=json for machine-readable logs.

How it works

The schema compiles per tokenizer: a bounded codebook search picks the neutral alias codes whose candidate rows tokenize most cleanly, and the prompt renders FROM the compiled plan — the model is taught exactly the protocol the scorer judges. The context is fenced with a per-context nonce, so no interior line can impersonate the closing fence.

The model prefills once and the KV cache is shared. One scoring row per field (extra trie rows for multi-token options), a restricted softmax over each field's allowed options, and the JSON is assembled from the winners — with a probability per field. Chunks are sized by a measured active-memory budget (the B=1/B=2 tiling slope is probed at engine load; Metal allocation failures halve the chunk and retry, counted in failed_attempts). Temperature is applied once at the end; ties resolve deterministically. Batched decide_many runs one prior pass and one merged scoring pass per context group. ARCHITECTURE.md has the full picture.

Contributing

Code and docs: CONTRIBUTING.md · benchmark results: BENCHMARKING.md.

Credits and license

jevmlx started from rorshopping/jev-on-a-laptop (parallel constrained decoding on a laptop) and descends from harshatheg/Qwen-2.5-1B-RLCD, an MLX demo of the technique. jevmlx is the maintained, generic version — any MLX instruct model or OpenAI-compatible server, multi-select, none-of-the-above and abstention, per-field probability, an eval harness.

Not affiliated with TypeSafe. MIT — see LICENSE; third-party credits in NOTICE.