Quickstart | Trade-off | Storage | v0.2 evidence | Stage B | StateLease-H5 | Compatibility | Reproduce

I built RecurQuant to answer one practical question: Can we quantize only the recurrent memory path in Qwen3.5 Gated DeltaNet and keep token-level quality from a fixed byte budget?

I kept this repo narrow on purpose. It does not change model weights and it uses the normal eager Transformers forward with a pluggable cache. That keeps each experiment clean enough to reproduce.

The frozen v0.2 layout passed a 500-task held-out MBPP teacher-forced fidelity protocol. Compared with uniform INT4, task-macro excess NLL was 72.75% lower while the packed recurrent-state footprint was 2,564,096 bytes (packed payloads plus FP16 scales).

An experimental v0.3 path, RHT-CQER-32, also cleared its separate 32-task development gate: aligned excess NLL was 52.73% lower than CQER-32 at the same packed-state and selector-byte budget. That result is development-only.

This repo currently targets Qwen/Qwen3.5-0.8B-Base and does not quantize model weights or standard attention KV caches. The current Python implementation still dequantizes one recurrent state during the forward pass.

I built and maintain RecurQuant as an open research project. Muhammad Labeeb Aryan. Licensed under Apache-2.0.

Verified storage-fidelity frontier

Among the nearest-rounding quantized layouts evaluated in the held-out protocol, three form the non-dominated storage-fidelity frontier. Each trades more resident recurrent-state storage for lower teacher-forced excess NLL. The frozen v0.2 layout is the middle point: it adds 131,072 bytes (5.39%) over uniform INT4 while lowering task-macro excess NLL by 72.75%.

The chart is regenerated directly from the authenticated 500-task confirmation record, and CI rejects stale generated assets. It compares exact resident recurrent-state bytes with teacher-forced fidelity only. The matched FP32 state reference is off-plot at 18,874,368 bytes and zero excess NLL by definition; these are not speed, peak-memory, whole-model-memory, or generated-code results.

Quickstart

This installs the public v0.2 alpha from its frozen tag. The first model-backed run downloads the pinned model and tokenizer. Python 3.11 and a CUDA GPU match the evaluated path; recurquant demo uses synthetic states and does not download a model. RHT-CQER-32 remains an experimental research path rather than the default package policy.

Windows PowerShell:

git clone --branch v0.2.0a1 --depth 1 https://github.com/Labeeb2339/recurquant.git
cd recurquant
py -3.11 -m venv .venv
.\.venv\Scripts\python.exe -m pip install .
.\.venv\Scripts\recurquant.exe qwen35 --max-new-tokens 16

macOS or Linux:

git clone --branch v0.2.0a1 --depth 1 https://github.com/Labeeb2339/recurquant.git
cd recurquant
python3.11 -m venv .venv
.venv/bin/python -m pip install .
.venv/bin/recurquant qwen35 --max-new-tokens 16

To verify the installation without downloading a model, run recurquant demo with the platform-specific executable path above. It performs a deterministic synthetic state round-trip and reports physical payload bytes, compression ratio, and quantization error.

The installed command and examples/qwen35_quickstart.py call the same implementation. The default is the frozen v0.2 mixed policy: layer 0 at INT8 and the remaining recurrent layers at INT4. Uniform INT4 is retained only as an explicit stress baseline via --policy uniform-int4-stress. Add --json for one machine-readable result containing the generated text, pinned model provenance, selected policy, and raw storage counters. Read the compatibility contract before using a different model, Transformers version, device layout, or generation mode.

Use it in Python

This example uses the reusable frozen v0.2 helper, which keeps Gated DeltaNet layer 0 at INT8 and the other 17 recurrent layers at INT4. The generic create_qwen35_packed_cache() factory remains available for controlled policy experiments.

import warnings

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

from recurquant import create_qwen35_v02_mixed_cache

MODEL_ID = "Qwen/Qwen3.5-0.8B-Base"
REVISION = "dc7cdfe2ee4154fa7e30f5b51ca41bfa40174e68"

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
if device.type != "cuda":
    dtype = torch.float32
elif torch.cuda.is_bf16_supported():
    dtype = torch.bfloat16
else:
    warnings.warn(
        "CUDA BF16 is unavailable; falling back to FP16. RecurQuant's public "
        "full-model fidelity evidence has not been validated for FP16 weights.",
        RuntimeWarning,
        stacklevel=2,
    )
    dtype = torch.float16

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, revision=REVISION)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    revision=REVISION,
    dtype=dtype,
    attn_implementation="eager",
).to(device)
model.eval()

cache = create_qwen35_v02_mixed_cache(model)
inputs = tokenizer("Explain recurrent-state quantization simply.", return_tensors="pt")
inputs = inputs.to(device)
continuation = []

with torch.inference_mode():
    output = model(**inputs, past_key_values=cache, use_cache=True)
    for step in range(32):
        next_token = output.logits[:, -1, :].argmax(dim=-1, keepdim=True)
        continuation.append(next_token)
        reached_eos = (
            tokenizer.eos_token_id is not None
            and bool((next_token == tokenizer.eos_token_id).all().item())
        )
        if reached_eos or step == 31:
            break
        output = model(input_ids=next_token, past_key_values=cache, use_cache=True)

generated_ids = torch.cat(continuation, dim=1)
print(tokenizer.decode(generated_ids[0], skip_special_tokens=True))
print(cache.storage_summary())

create_qwen35_v02_mixed_cache() and create_qwen35_packed_cache() reject unsupported Transformers versions, non-eager attention, training mode, multi-device placement, and incompatible Qwen configurations early. The returned cache exposes exact live tensor byte accounting through storage_summary().

What is physically smaller

For batch-one Qwen3.5-0.8B-Base recurrent states, the frozen mixed layout stores 2,564,096 resident bytes instead of 18,874,368 FP32-state bytes. Uniform INT4 stores 2,433,024 bytes. These figures include packed payloads, FP16 group scales, and padding.

This is recurrent-state storage only. It is not a whole-model, peak-CUDA-memory, latency, or throughput result.

Held-out confirmation

The frozen layer-0 mixed layout passed every preregistered v0.2 quality and integrity gate on the untouched MBPP test split. Task-macro excess negative log-likelihood above the FP32-state reference fell from 2.949743 for uniform INT4 to 0.803713: a 72.75% reduction.

The confirmation covers 500 paired tasks and 30,244 teacher-forced reference- code tokens. The paired mixed-versus-uniform improvement was 2.1460 nats/token with a 95% bootstrap interval of [2.0922, 2.1999]. Against the mean of three exactly same-byte random high-precision layer placements, the paired improvement was 2.0332 with a 95% interval of [1.9802, 2.0861].

Token-weighted measure	Uniform INT4	Mixed L0 INT8
Mean KL	3.149969	0.914580
Worst-5% KL	9.002207	4.839139
Top-1 agreement	0.321155	0.665190

The earlier 90-task development result was a 74.14% reduction; it remains available in evidence/mbpp-v02-development.json and DEVELOPMENT_002.md.

Important boundaries:

• The accepted result is in evidence/mbpp-v02-confirmation.json, with the full decision and interruption record in CONFIRMATION_002.md.
• Tokens were scored teacher-forced. Candidate-generated code was not fed back, executed, or graded for correctness.
• The MSE selector also chose layer 0, so it is the exact same candidate and not independent evidence that the read-risk selector is novel or superior.
• This supports one pinned recurrent-state fidelity and resident-byte result. It does not support generated-code quality, speed, peak memory, whole-model memory, cross-model generality, or a breakthrough claim.

Experiment 009 Stage-B development

RHT-CQER-32 applies a deterministic right-side randomized Hadamard transform inside each existing recurrent-state row group before the same physical Q4/Q8 packing used by CQER-32. The transform does not change the frozen 1,976-row precision allocation or storage contract: both methods use 2,564,096 packed state bytes and 2,711,552 resident bytes including the query-energy selector.

On the preregistered 32-task ranked MBPP [32, 64) development window, RHT-CQER-32 passed all eight frozen advancement checks. Task-macro aligned excess NLL fell from 0.323944 to 0.153129, a 52.73% reduction. Aggregate local recurrent-state reconstruction SSE fell from 36,409.363073 to 15,345.844948, a 57.85% reduction.

RHT-CQER-32 had lower excess NLL on 27 of 32 tasks, with no ties. The paired CQER-minus-RHT improvement was 0.170815 nats/token and its frozen 10,000-sample paired 95% bootstrap interval was [0.116082, 0.229438].

This is positive development evidence on one pinned model and task window, not held-out confirmation for RHT-CQER-32. Randomized Hadamard and rotation quantization are prior art, and the current Python implementation has no fused-kernel, latency, peak-memory, cross-model, or independent external-reproduction result. See the full Stage-B result, verification receipt, and machine-readable release manifest.

Experiment 012: StateLease-H5 screen

StateLease-H5 passed all eight prespecified Stage-A screening checks on one previously opened MBPP calibration task (38 scored tokens). At exactly 3,454,664 allocated resident bytes, its excess NLL was 0.023349 versus 0.028442 for the strongest fixed-replay schedule, fixed_cut4_in5—a descriptive 17.90% reduction on this one trace.

It did not beat the two strongest equal-total-byte no-replay codecs. The Q4/Q6/Q8 comparator reached -0.000014 excess NLL and expanded Q4/Q8 reached 0.002461. This is therefore a falsification-screen pass, not a development, held-out, general-advantage, or breakthrough result.

The full Stage-A record is committed with file SHA-256 1e92b0bea176154496c7d5e45013bf051ef3f388352c1267d86910f81844fd22. The verifier was added after v0.2.0a1 and is not included in that release tag. Install the current main branch in a separate checkout, then run it:

git clone --branch main --depth 1 https://github.com/Labeeb2339/recurquant.git recurquant-statelease
cd recurquant-statelease
python -m pip install .
recurquant verify-statelease-stage-a evidence/experiment012-statelease-stage-a-666.json

This recomputes the metrics, storage contracts, and eight gate decisions offline. See the result note for the complete method table, storage breakdown, gate outcomes, and claim boundary.

Scope

The supported public surface is deliberately narrow:

• Python >=3.11 and exactly transformers==5.14.1 for this alpha;
• text-only Qwen3.5 hybrid models with linear_attention and full_attention layer types;
• physical INT4 or INT8 recurrent-state payloads; FP16 scales are the evaluated default, while FP32 scales are supported as an experimental, unevaluated option;
• eager, evaluation-only, single-device inference; and
• explicit past_key_values=cache model calls.

See docs/compatibility.md for the validated software, hardware, model revision, generation paths, and unsupported modes.

Claim boundary

Quantizing recurrent state is not new. I built RecurQuant to test one narrower question: can sensitivity-guided mixed precision preserve Gated DeltaNet recurrent-state fidelity better than simple equal-byte placements? For the pinned Qwen3.5-0.8B-Base teacher-forced MBPP protocol, the frozen policy passed the held-out confirmation and beat all three tested same-byte random placements.

That is a confirmed case study, not proof of novelty or general superiority. Q-Mamba already studies 4-bit persistent Mamba2 states, Quamba2 quantizes cached SSM states, and other mixed-precision and replay systems overlap parts of this design space. Experiment 009 adds a positive 32-task development result for a known right-RHT codec composed with CQER-32, but it is not a new confirmation result or evidence that Hadamard quantization is new. RecurQuant currently has no fused packed kernel or measured speed claim. I therefore do not present this release as a breakthrough, a whole-model memory reduction, or a cross-model result. See the claim boundary and prior-art review for the exact comparison.

Research record

• Independently validate the held-out decision with recurquant verify-confirmation; the reproduction guide pins both committed hashes and explains optional raw-checkpoint reconstruction.
• Held-out MBPP confirmation report and machine-readable evidence
• Frozen public evaluation protocol
• MBPP development report
• Current research status
• CORA-C2 development result
• Experiment 009 frozen protocol
• Experiment 009 Stage-A result
• Experiment 012 StateLease-H5 Stage-A result
• Experiment 009 Stage-B identity freeze
• Experiment 009 Stage-B result
• Experiment 009 verification receipt
• Experiment 009 release manifest
• Claim boundary and prior-art review
• Failed proxy signals and empirical-sensitivity pivot
• Scale-format correction and packed/QDQ parity
• Earlier pilot protocol and v0.1 diagnostic confirmation archive

Contributing

I welcome reproducible compatibility reports, additional model-family adapters, and work toward a fused packed recurrent kernel. Open an issue with a minimal reproducer and cache.storage_summary(); never include access tokens, private prompts, or authentication files.