memra is a Rust + CUDA inference engine built for NVIDIA RTX PRO 6000 Blackwell and RTX 5090.
It serves GGUF models over an OpenAI-compatible API, and every default it ships was measured on
those two cards — both Blackwell sm_120a, with a separately compile-gated Hopper/H100 (sm_90a)
lane alongside.
That focus is the point, not a limitation. The mainstream serving stacks are tuned for datacenter Hopper and Blackwell; memra is tuned for the Blackwell cards people actually own — the 96 GB PRO 6000 Workstation/Server Edition and the 5090 — on the assumption that you want yours saturated rather than merely supported. Concretely, on ONE PRO 6000 it serves a 35B MoE at a 262,144-token context, holds a ~48 GiB in-VRAM prefix cache, and sheds excess load with clean 429s instead of degrading everyone's latency; the 5090 carries the same engine at its own measured settings.
Both cards are first-class, and neither is tuned at the other's expense. Arms are selected per device: when a mechanism wins on the 5090 and loses on the PRO 6000, or the reverse, it becomes a per-hardware default keyed on the device rather than a compromise applied to both. Detection drives that choice, so a naked command runs at full speed on whichever card it lands on.
Its center of gravity is OpenAI-compatible serving: streaming chat and text completions, prefix and session reuse, VRAM-aware admission, MTP speculative decoding, and PP-2 for models that need two cards.
The deployment philosophy is specialization: one model on one GPU, with tuned paths selected by default. PP-2 extends that shape to one model across two GPUs when the weights do not fit on one card. GGUF remains the primary runtime and delivery format.
Use memra if you are serving on RTX PRO 6000 Blackwell (or a 50-series card) and want defaults that were measured on your hardware. Look elsewhere if you need broad model coverage, tensor parallelism, or a datacenter-H100 fleet — those are other projects' strengths, and Design boundaries says so plainly.
Measured speed is summarized in three generated same-rig samples; the complete boards and conditions live in docs/PERFORMANCE.md.
Releases are listed at releases/latest; the
current tag is v0.82.0, and main runs ahead of it. Recent waves: security hardening (serving
refuses unauthenticated non-loopback binds at boot, metrics endpoints require bearer auth whenever
keys are configured, API-key comparisons are constant-time, keyring rewrites are atomic, and the GGUF
parser returns contextual errors instead of panicking on truncated or malformed files — a truncated
multi-gigabyte download is an ordinary event, not a crash), then a correctness wave on the serving
path: a request may no longer produce tokens under two different numerical programs, which is now a
documented rule rather than a series of separate fixes. Details under
Recent fixes.
Current focus is per-hardware tuning on RTX PRO 6000 Blackwell: dedicated 3-card boxes for Qwen3.6-27B (ahead of Qwen3.8-27B) and for Step-3.7-Flash, covering speculative-decode shapes, prefix-cache capacity, concurrency headroom, and profiling. Numbers land in docs/PERFORMANCE.md as they are measured; a default only ships for a card class it was measured on.
Two results from that tuning are worth stating up front, because both are counter-intuitive and both were measured on one RTX PRO 6000 Blackwell Workstation serving a 35B MoE at a 4,860-token prompt shape with a shared prefix:
- Prefix-cache depth dominates. With the cache holding the full shared prefix, concurrency 16 ran at 8.50 req/s and a 1.87 s median. With a shallower entry covering the same prefix class, the same build on the same card ran 2.72 req/s at 5.84 s — a 3.1x swing in both throughput and latency from cache depth alone.
- Speculative decoding is not free on cache-carried shapes. It is numerically exact, and offline
it looks like a 1.5-1.7x win, but on this serving shape it cost 4x because a speculative session
gives up the cross-request prefix cache.
MEMRA_SERVE_SPEC=0is the serving default for that measured reason; see docs/FLAGS.md.
Raw per-run receipts for both are under research/, and the second is the kind of result this project publishes either way: the arm expected to win lost.
The release installer is the shortest path. It selects the published sm_120a, sm_90a, or
sm_89 prebuilt, verifies the release checksum, and installs memra-server, run-gen,
run-spec, and kernel-check.
curl -fsSL https://raw.githubusercontent.com/avifenesh/memra/main/tools/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"Prebuilt binaries require Linux x86_64, glibc 2.35 or newer, NVIDIA driver 580 or newer,
and the CUDA runtime libraries. They do not require nvcc. Set MEMRA_INSTALL_DIR to
override ~/.local/bin.
To build all workspace binaries from source:
git clone https://github.com/avifenesh/memra.git
cd memra
cargo build --release
export PATH="$PWD/target/release:$PATH"Source builds require Rust 1.85 or newer and the CUDA 13.1 toolkit. Architecture is detected
at build time; MEMRA_CUDA_ARCH is the documented override.
Point the command at a supported GGUF to run one chat-templated generation:
MEMRA_CHAT=1 run-gen /absolute/path/to/model.gguf --prompt "Explain KV caches in one sentence."run-gen also accepts a supported Hugging Face checkpoint directory or an
hf:owner/repo[:file] spec. The hf: form downloads and caches the selected artifact on first
use.
Start the OpenAI-compatible server in one terminal:
MODEL=/absolute/path/to/model.gguf
MEMRA_MODELS="qwen=$MODEL" memra-serverThe default bind address is 127.0.0.1:8080. From another terminal, stream a chat completion:
curl -sS -N http://127.0.0.1:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "qwen",
"messages": [{"role": "user", "content": "Explain KV caches in one sentence."}],
"max_tokens": 128,
"stream": true
}'The qwen value is the alias from MEMRA_MODELS. Bearer authentication, multiple model
aliases, MTP drafter attachment, and deployment probes are covered in
docs/SERVING.md.
The README carries only representative regression samples. The full boards, methodology, thermal regime, N, and open cells live in docs/PERFORMANCE.md.
| Model / scenario | memra tok/s | llama.cpp tok/s | Ratio |
|---|---|---|---|
| Qwen3.6-35B-A3B MoE — plain decode (the flagship serving model, tg128 at 512-token context) | 187.0 | 164.9 | 1.13x |
| Qwen3.5-9B — MTP speculative, short / medium / long-agentic prompt classes (both engines at their measured best) | 281.0 / 211.7 / 187.1 | 122.2 / 121.5 / 117.7 | 2.30x / 1.74x / 1.59x |
| Qwen3.6-35B-A3B MoE — plain decode at 6.3k-token context (the short-context lead holds at depth) | 177.1 | 156.3 | 1.13x |
Measured 2026-08-02 on the RTX 5090 Laptop — same-session interleaved medians, same exact prompts; memra at its naked defaults, llama.cpp at its swept best (docs/COMPETITOR-SETUP.md). The llama.cpp column is a frozen reference recorded through 2026-08-03 (benching stopped that day). N, thermal regime, and the full boards: docs/PERFORMANCE.md; raw per-run logs: research/tune-data/.
Support is specific to a model, quantization, and drafter combination. The generated table is numbers-free; per-model measurements stay in docs/PERFORMANCE.md.
- RTX PRO 6000 Blackwell (
sm_120a) — co-primary target. Workstation and Server Edition, 96 GB. Carries verification, final tuning, and serving, single-card and as PP-2 pairs. - RTX 5090 / 50-series (
sm_120a) — co-primary target. Same architecture, its own measured settings, and its own defaults where they differ. Local 5090 performance is never traded away to simplify a remote default; a perf claim needs numbers from both cards before it sets a global one, and one-card evidence sets a one-card default at most. - Hopper
sm_90a— separately compile-gated H100 lane with its own validation battery and evidence ledger in ARCHITECTURE-H100.md. Secondary: it does not change the nakedsm_120abuild or its defaults. - Ada
sm_89— portable prebuilt target, not a tuned performance target. - B200
sm_100a— source-only, compile-gated backend. Other architectures are not a tuned support promise.
| Model | Class | Quant | Drafter | Supported since |
|---|---|---|---|---|
| Qwen3.5-9B | dense | NVFP4 (5090), Q8_0 (H100) | MTP + own-gen trimmed draft | v0.1.0 |
| Qwen3.6-27B | dense | NVFP4, Q4_K_M MTP-baked | MTP + own-gen trimmed draft | v0.1.0 |
| Qwen3.6-35B-A3B | MoE | IQ4_XS | MTP + own-gen trimmed draft | v0.1.0 |
| Gemma-4 26B-A4B | MoE | QAT Q4_0 | Gemma assistant draft (CLI only) | v0.23.0 |
| Gemma-4 31B | dense | QAT Q4_0 | Gemma assistant draft (CLI only) | v0.35.0 |
| Gemma-4 E4B | dense | QAT Q4_0 | Gemma assistant draft (CLI only) | v0.35.0 |
| Gemma-4 12B | dense | QAT Q4_0 | Gemma assistant draft (CLI only) | v0.40.0 |
| Ornith-1.0-9B | dense | Q8_0 | own-gen donor-block draft | v0.63.0 |
| Ornith-1.0-35B | MoE | Q4_K_M | own-gen donor-block draft | v0.64.0 |
| Qwen-AgentWorld-35B-A3B | MoE | UD-IQ4_XS (avoid UD-Q4_K_M — its Q5_K expert mix sits outside fast-path coverage) | own-gen drafter | v0.66.0 |
| Step-3.7-Flash 196B-A11B | MoE | IQ4_XS + Q8_0 MTP head (two-card PP-2) | MTP (single-card); plain batched decode on PP-2 | v0.73.1 |
Step-3.7-Flash serves on a two-card PP-2 pair (MEMRA_PP_STAGES=2); its explicit
configuration and qualification boundaries are recorded under
bring-up notes. On PP-2 it decodes in a single
numeric class at every batch width — served bytes do not depend on load history.
The Gemma-4 drafter is a separate-assistant format, not a NextN/MTP head, and it is reachable
only from the gemma-gate CLI path — memra-server attaches drafters exclusively through the
generic NextN loader, which refuses this format (draft n_embd != model n_embd). Gemma-4 serves
with speculative decoding off; speculative numbers for these SKUs are CLI measurements, not
serving ones.
memra-server is the center of the current engine. The public surface includes:
- OpenAI-compatible HTTP —
/v1/chat/completionsand/v1/completions, blocking or SSE streaming;/v1/modelsreports loaded-model capabilities throughsupported_parameters. - Request reuse and admission — tenant-scoped prefix caching, same-window prefix dedup, rewritten-history session affinity, request-sized KV allocation, and VRAM-aware admission.
- Speculative decoding — embedded or attached MTP/NextN drafters, with request-conditioned depth and automatic plain-decode fallback where batching or PP-2 wins.
- Agent surfaces — tool schemas and streaming
tool_calls, separated reasoning output,reasoning_effortmapped through each model's chat template, and constrained decoding viaresponse_formatjson_objectorjson_schema. - Operations — optional keyed bearer auth, per-tenant cache boundaries and request caps, health/readiness probes, metrics, and graceful drain.
- Long context and larger models — model-capped 262,144-token serving, plus PP-2 to split one model across two cards. Step-3.7-Flash is the current large-MoE PP-2 trial.
See docs/SERVING.md for request fields, response shapes, capability gates, auth configuration, cache semantics, admission behavior, and deployment runbooks.
The latest serving wave closes several serving-contract failures:
- Serving fails closed on exposure: a non-loopback bind without a configured key source refuses
to boot,
/metricsand/yield/metricsrequire bearer auth whenever keys are configured or the bind is public, API-key comparisons are constant-time, and keyring rewrites are atomic (receipt). - The GGUF parser rejects truncated and malformed files with contextual errors instead of panicking the GPU worker — every byte-prefix of a valid file is covered by fixture tests (receipt).
- The GPU Gumbel sampler now keeps uniforms below 1.0, removing rare
+infwinners that injected arbitrary tokens during long sampled generations. The 262,144-token serving target remains in place (receipt). - Explicit
max_tokensis enforced exactly. Token events, usage, and scheduler accounting now follow visible token IDs rather than speculative rounds (receipt). - Plain-decode session affinity now checkpoints and resumes rewritten conversation histories, so later turns prime the new suffix instead of an ever-growing history (receipt).
- New admissions can end an in-flight speculative burst at its next round boundary instead of waiting for the whole burst; request-sized KV allocation remains VRAM-aware (receipt).
- Cache boundaries were hardened: release builds retire prefix pins and client cache salts are bounded and validated (receipt).
/v1/modelsnow advertises the parameters each loaded model actually supports (gateway contract).- Step-3.7-Flash PP-2 decodes in one numeric class at every batch width: a greedy request returns the same bytes whether it decodes solo, joins a batch mid-generation, or starts batched. The transition matrix that proved and closed the load-history divergence is the receipt (isolation, fix).
- Grouped expert prefill (
MEMRA_MOE_GROUPED=1) looked promotable on two-card Step serving in August 2026 — 4k streaming TTFT 10.96 s -> 7.26 s, N=5 per arm (receipt) — but a re-sweep on an RTX PRO 6000 pair withdrew it: the arm is slower there (2,687.5 vs 8,193.7 tok/s resident-KAT transfer, 0/5 paired wins) and it fails the Q35 mixed c=4 exact-token gate, truncating every request in the cell (receipt). It stays off by default and should not be enabled; seeMEMRA_MOE_GROUPEDin docs/FLAGS.md. - An opt-in SWA ring (
MEMRA_SWA_RING=1) right-sizes sliding-window KV for Step sessions: the 262k session KV component drops 3.6x — measured 2 -> 12 concurrent 262k sessions before the first defer; lapped checkpoints decline safely (receipt, flag-on validation). - A B=1 decode specialization for Step models recovers most of the one-class contract's cost: +4.4% sustained c=1 decode with byte-identical output (the batched walk was issuing 90 arithmetic-free device copies per token at B=1) (receipt).
- A host-staged PP boundary fallback (
MEMRA_PP_HOST_BOUNCE=1, default off) serves byte-correct on hosts whose GPU peer-copy path reports success but does not preserve bytes — a failure mode a capability flag or bandwidth test does not reveal. Peer paths should be byte-probed at provisioning (receipt). - Graceful shutdown joins the GPU worker thread before exit, closing a restart race where a new server could boot while the old worker still held device state.
- Tuned paths are defaults. Environment variables are runtime parameters, machine configuration, diagnostics, or rollback seams; the catalog is docs/FLAGS.md.
- The standing correctness battery is
kernel-check, therun-genargmax gate, andrun-specK=1..8 self-consistency. GitHub CI is compile-only; GPU evidence runs on the target rig. - memra is not a broad model-compatibility or multi-vendor runtime. Unsupported architectures, model families, and quantizations may compile or load without becoming supported configurations.
- Multi-GPU support is replica fleet or pipeline-parallel PP-2. Tensor parallelism is not implemented.
- This is a moving research engine. Public APIs and flags can change between releases.
| Read this | For |
|---|---|
| docs/SERVING.md | API contract, caching, auth, admission, PP-2, and operations |
| docs/PERFORMANCE.md | Full boards, methodology, rigs, gaps, and model qualification |
| docs/FLAGS.md | Audited environment-variable catalog |
| ARCHITECTURE.md | Engine structure and the sm_120a implementation ledger |
| ARCHITECTURE-H100.md | H100-specific mechanisms and evidence |
| CONTRIBUTING.md | Development workflow and GPU correctness gates |
| docs/RELEASING.md | Versioning, target-rig battery, tags, and publication |
Issues and PRs are welcome. Start with CONTRIBUTING.md; validation on a new GPU should use the hardware report template.
MIT — see LICENSE.