Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
4be35df
feat(kev): add pinned CPU export and FP32 pointer head
psv666 Oct 24, 2026
76b764a
Merge current main into Kev export and head preparation
psv666 Oct 27, 2026
8f765fd
fix(kev): validate every pinned LoRA projection target
psv666 Oct 27, 2026
0102113
feat(kev): add eager native Rust CUDA text worker and parity harness
psv666 Oct 27, 2026
57e239b
test(kev): reject invalid parity evidence and report boundary flips
psv666 Oct 27, 2026
1d44e4f
docs(kev): record eager numerical and serving acceptance limits
psv666 Oct 27, 2026
490d80d
fix(kev): make reference model lifetime explicit for lint checks
psv666 Oct 27, 2026
33e1848
Kev: qualify against default BF16 service and fix Unicode tokenization
psv666 Oct 27, 2026
b446d6e
docs: mark Kev BF16 numerical qualification as failing
psv666 Oct 27, 2026
224fa3a
test(kev): verify rendered prompts across the frozen service corpus
psv666 Oct 27, 2026
0efa1d9
fix(kev): align eager native numerics with the pinned BF16 service
psv666 Oct 28, 2026
d850153
test(kev): qualify Noul boundaries using official service outputs
psv666 Oct 28, 2026
b7c94b3
docs(kev): record BF16 acceptance and shared Qwen regression evidence
psv666 Oct 28, 2026
2892608
Merge remote-tracking branch 'origin/main' into codex/issue-27-kev-ex…
psv666 Oct 31, 2026
85119d0
test(kev): cover complete export and late-write cleanup
psv666 Oct 31, 2026
82751ae
Merge refreshed Kev export slice and main prefix-cache support
psv666 Oct 31, 2026
9501ede
test(qwen): guard fixed-prefix entry points against Kev numerics
psv666 Oct 31, 2026
125251c
docs(kev): pin the qualified CUDA reference wheel
psv666 Nov 1, 2026
716ec4a
Merge main into Kev native worker and record SM120 qualification failure
psv666 Nov 18, 2026
3ada7c2
fix(kev): preserve service projection component arithmetic
psv666 Nov 19, 2026
17d4816
docs(kev): record L20 projection diagnosis and unchanged-gate acceptance
psv666 Nov 19, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -194,7 +194,7 @@ checkpoint. Public231 concurrency-1 latency for heads `1ba07d1`, `f968e576`,
| CLM-v0.1-8B | [External worker with a CPU stub encoder](recipe/clm/README.md); contract checks only, real Qwen3-8B decisions unverified by this recipe |
| LFM2.5-350M | [Transformers worker](recipe/lfm2/README.md); text state and independent choice questions |
| OmniJev-4B v1.1 | [Native Rust/CUDA worker](recipe/omnijev/native.md), eager: one image, Choice, Noul and Score; [reference comparison on sm_89](recipe/omnijev/native.md#comparison-with-the-reference) |
| Kev-4B | [Pinned CPU export and Rust pointer head](recipe/kev/README.md); no worker or full-model parity yet ([#27](https://github.com/InferMatrixLab/system1-omni/issues/27)) |
| Kev-4B | [Eager native Rust/CUDA text worker](recipe/kev/native.md); Choice/Noul/Score; experimental; [full-model qualification pending](recipe/kev/native.md#primary-acceptance-status) ([#27](https://github.com/InferMatrixLab/system1-omni/issues/27)) |
| D3 (Decision 3.0) | [Python reference worker](recipe/d3/reference.md) on the checkpoint's pinned runtime; `choice`/`noul`/`score` over text, image and video state; author-run on A100 for text and media serving (see the integration and media PRs) |
| Vela 2.0 0.3B | [Python reference worker](recipe/vela/reference.md) on the checkpoint's pinned runtime; all five kinds (`choice`/`noul`/`score`/`span`/`set`) for text state; CPU FP32 validated (author-run, evidence in the integration PR) |
| Vela 2.0 0.8B hybrid | [Python reference worker](recipe/vela/reference.md) on the checkpoint's pinned runtime; same surface plus `span_heads`; CPU FP32 parity recorded in the integration PR |
Expand Down
18 changes: 16 additions & 2 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ design. Concrete input/output types follow each executor's supported layout.
## Implementation status

The [Rust frontend](../src/frontend/README.md) currently forwards HTTP requests
to separately running workers. Cua-S1, Open-Jev, Decider, and StartLux Instant
to separately running workers. Cua-S1, Open-Jev, Decider, Kev-4B, and StartLux Instant
4B have native Rust/CUDA workers that share the
[Qwen3.5/3.8 executor](../src/models/qwen3_5/native/), which accepts
single prompts and bounded packed prefill. StartLux Instant uses a
Expand Down Expand Up @@ -93,6 +93,14 @@ FP32 adapters and framework-reference arithmetic. It computes selected BF16
label-head logits on CUDA, then calibrates typed answers on the CPU. JEMM admits
one complete text or 1–4-image request and synchronizes both model streams before
releasing its resources. See its [model contract](../src/models/jemm/README.md).
Kev uses merged BF16 weights and an optional Torch 2.8 numerical profile. It
recomputes state then question in eager mode, matching the pinned official
padding and projection shapes, and downloads selected option/decision hidden
rows for its FP32 pointer head. Temporary GDN/KV state belongs to one question
call and is drained before release; it is not shared between requests. Its
processor owns prompt tokens and question geometry, while complete-question
calibration and typed response reconstruction remain in postprocessing.
The existing Qwen default, fixed and reference profiles keep their own paths.
Decider projects selected tied-embedding rows on CUDA with BF16 output rounding
before FP32 calibration and CPU response assembly. Laya computes its scorer and
action head on CUDA. Its fixed-shape
Expand Down Expand Up @@ -158,6 +166,7 @@ and candidate identity, usage, and response metadata outside the executor.
| Cua-S1 | One unpadded token-ID vector and option count per question, in request order. | One FP32 answer-letter logit vector per question. | Per-question softmax, choice/confidence, ordered answers, and token usage. |
| Open-Jev | Token-ID vectors grouped by question, then independent candidate, in request order. | One FP32 learned scalar per candidate in the same grouping. | Add the `noul` false logit of zero, calibrate across each complete question, and restore typed answers, usage, and metadata. |
| Decider | Complete independent token-ID rows in question order; Score expands one no/yes row per level. | One BF16-projection-rounded FP32 selected-label logit vector per row. | Per-type calibration, whole-Score normalization, ordered typed answers and unique-prefix token usage. |
| Kev-4B | Complete text rows, consecutive positions and ordered option/decision readouts, one row per question. | One FP32 pointer-logit vector per complete question. | Saved-temperature softmax over all candidates, ordered Choice/Noul/Score answers and official token billing. |
| Laya | One padded request: token IDs, true lengths, question types and ordered option markers; at most 16 questions, 512 tokens per row and 2048 markers. | Per-question FP32 option logits and two action logits copied back after GPU heads. | Calibrate and decode ordered `choice`, `score` and `noul` answers, usage and metadata. |
| JEMM | Ordered token IDs, three-axis positions, candidate counts and optional shared image patches for one complete request. | Per-question selected BF16 label logits copied back as FP32. | Apply checksum-covered text/image calibration and reconstruct ordered `choice`, `score` and `noul` answers and usage. |
| StartLux Instant 4B | One unpadded token-ID prompt per question, or per knockout group and then a final prompt, in request order. | FP32 letter logits from a CPU dot of the tied embedding rows at the last position. | Per-type temperature, none-check split, wide-choice residual, and ordered `choice`, `noul`, and `score` answers with token usage. |
Expand All @@ -172,7 +181,7 @@ retain original batch row counts; output/down GEMMs retain candidate lengths.
Request-local prefixes run as exclusive items. It restores question/candidate
grouping before normalization. Laya batches questions within one request. Shared runtime
admission precedes blocking dispatch: Cua-S1 admits one question forward at a
time; serial Open-Jev modes, Decider, Laya, StartLux Instant, and Clef admit one complete request. Clef's admitted work is the frame exchange with its executor process. StartLux runs each prompt inside that admission one at a time. Cua-S1's CPU letter projection
time; serial Open-Jev modes, Decider, Kev, Laya, StartLux Instant, and Clef admit one complete request. Kev runs all question rows and its FP32 head inside that admission; listener readiness follows a real mixed-question warmup. Clef's admitted work is the frame exchange with its executor process. StartLux runs each prompt inside that admission one at a time. Cua-S1's CPU letter projection
stays outside admission; Open-Jev's scalar heads and Laya's GPU heads and
synchronized readback remain inside their request unit. Decider includes every
backbone row and CUDA head projection, synchronizes both streams and retires
Expand Down Expand Up @@ -235,6 +244,11 @@ semantics.
Preserve whole-request failure behavior, error status/body, and usage fields
when splitting or regrouping work.

Kev prepares one complete causal row per question, reads every candidate closing
marker plus the final decision marker through the shared Qwen selected-position
readout, and applies its FP32 pointer head. Its eager executor recomputes the state
for each question and admits one whole request through the serial scheduler.

Cua-S1 prepares one prompt per question and reads option-letter logits.
Open-Jev prepares independent candidate prompts and normalizes across the
complete question's candidates. Laya pads prepared questions into one request
Expand Down
8 changes: 5 additions & 3 deletions docs/supported-models.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,11 @@ Models that are being added are also tracked in issues labeled [new model](https
| Clef (27B) | Same `omni-clef` server and text contract as Clef-flash, selected with `--variant clef` | Contract tests only; no inference | Not run | Not supported | Same stack; pinned `Cloudflare/clef` snapshot (~55.0 GB). Weights and joint-head config differ from flash |
| pplx-decider-v1.1-27b | [Python reference worker](../recipe/pplx_decider/README.md) on the checkpoint's own runtime. Text and up to four images; `choice`/`noul`/`score`. Native CUDA executor is not this worker. Lab L20X-labeled GPUs are the H200 target. The causal-conv1d build keeps sm_90 and sm_89. Capability 8.9 is ready on FLA chunk. Capability 9.0 uses all-sequence FlashInfer. The kept primary rotates 48 `cu_seqlens` buffers per device and length, passes `use_cp` false, skips the q/k repeat, and packs q and k before the L2-norm view. `801ef7b` fused the 48 linear gated RMSNorms and missed the median (p50 89.172 ms, p95 274.790 ms), so that kernel stays off. `2bbbe4e` replaced `silu(gate) * up` with FLA `swiglu_fwd` and the owner closeout rejected it (p50 97.613 ms, p95 292.795 ms), so that kernel stays off. Gate and up GEMMs stay separate. Residual RMSNorm and the gated output norm stay eager. An unknown `PPLX_GDN_PREFILL` fails closed on every device, including capability 8.9. `3ecdf5a` crashed in warmup (0/231). Always-on fusion `3afb472` and the 512-token gate `705ae089` missed the median. The 48-slot FlashInfer state ring `17a7b120` missed the gate. `PPLX_GDN_PREFILL=fla` reproduces the prior `f968e576` chunk. `PPLX_GDN_PREFILL=short` is the rejected ≤256 dispatch. SDPA stays on the checkpoint order. The fused gate-up GEMM is refused. The forward stays eager. `PPLX_COMPILE=dynamic` and `PPLX_COMPILE=dynamic-tail` are rejected opt-ins | Contract checks only; no inference | Reported load on 1× lab L20X (sm_89, ~140 GiB) at head `1ba07d1`: BF16 resident ~50333 MiB, no OOM. Historical primary: 231/231, p50 98.777 ms, p95 341.202 ms. Prior primary is the `f968e576` remeasure on the lab L20X-labeled GPU, `compute_capability=9.0`, `fla-chunk`, `flashinfer` null: 231/231, p50 94.077 ms, p95 300.200 ms (p50 −4.76%, p95 −12.02%). Prior primary `ffae182`: 231/231, p50 92.844 ms, p95 296.458 ms (p50 −1.31%, p95 −1.25%). Prior primary `4b99b73`: 231/231, p50 88.942 ms, p95 295.797 ms (p50 −4.20%, p95 −0.22%), 48-slot `cu_seqlens`, quality `choice_acc=0.9065` (139), `noul_acc=0.8784` (74), `score_mae=0.2035` (18). Kept primary is `fcf5e2b`: 231/231, p50 87.782 ms, p95 291.183 ms (p50 −1.30%, p95 −1.56%), grouped-value FlashInfer with q and k packed, same quality digits. Uncached all-sequence FlashInfer `3c72f02`: p50 88.808 ms, p95 300.751 ms, gate failed. Short-sequence FlashInfer `a50b9aa`: p50 95.446 ms, p95 300.871 ms, gate failed. cuDNN-first SDPA `0dcaa0ac`: p50 138.501 ms, p95 332.859 ms, gate failed. Fused gate-up `8b4ee326`: p50 102.437 ms, p95 304.970 ms, gate failed. Always-on dynamic compile `e36b32ed`: p50 97.751 ms, p95 263.124 ms, gate failed (p95 improved). 512-token compile `7e3b1be`: p50 100.037 ms, p95 257.084 ms, gate failed. Isolated-median compile `34ba5de`: p50 97.084 ms, p95 256.679 ms, gate failed (median probe was eager). FlashInfer state-workspace ring `17a7b120`: p50 89.165 ms, p95 295.723 ms, gate failed. Always-on FLA residual RMSNorm `3afb472`: p50 93.935 ms, p95 274.902 ms, gate failed. Length-gated FLA residual RMSNorm `705ae089`: p50 94.604 ms, p95 274.835 ms, gate failed. `use_cp=auto` `a6f4d9a`: p50 112.294 ms, p95 299.419 ms, gate failed. Grouped-value `3ecdf5a` crashed in warmup (0/231, no p50 or p95). The fused linear gated-norm candidate has no sample. Does not enter the H800 board | Not supported | Python 3.12, pinned checkpoint revision `14c9d90137962d300f3de95633f7b6535e80c5ce`, checkpoint `requirements.txt`, `causal-conv1d` 1.7.0, `flash-linear-attention` 0.5.2, optional `flashinfer-python` 0.6.16.post3. Shard sizes in the release manifest sum with the readout to about 48.6 GiB before activations |

Kev-4B has [CPU checkpoint preparation and a Rust pointer head](../recipe/kev/README.md)
only. No Kev worker, backbone parity, GPU serving or full-model accuracy is
established by that preparation slice; native integration remains [#27](https://github.com/InferMatrixLab/system1-omni/issues/27).
Kev-4B has an experimental [eager native Rust/CUDA text worker](../recipe/kev/native.md)
for Choice, Noul and Score. The integrated projection repair passes the frozen
5,662-question replay on NVIDIA L20 (`sm_89`) under the original 0.01 unrounded
probability gate. The earlier RTX 6000D (SM120) failure has not been rerun with
the repair. See the [configuration and replay results](../recipe/kev/native.md#primary-acceptance-status).

- **Validated:** covered by the recipe on `main` or by the checks in the linked merged pull request.
- **Author-run:** recorded by the recipe author on the stated hardware and not independently reproduced; the row names its owner.
Expand Down
4 changes: 4 additions & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,10 @@ nav:
- JEV-27B-VL validation status: recipe/jev_vl/validation.md
- OmniJev preparation: recipe/omnijev/README.md
- Kev-4B preparation: recipe/kev/README.md
- Kev-4B native worker: recipe/kev/native.md
- Kev-4B service acceptance: recipe/kev/service-acceptance.md
- Kev-4B numerical repair: recipe/kev/numerical-diagnosis.md
- Kev-4B Noul boundary qualification: recipe/kev/noul-boundary.md
- OmniJev native worker: recipe/omnijev/native.md
- CLM stub-encoder contract: recipe/clm/README.md
- LFM2.5-350M choice worker: recipe/lfm2/README.md
Expand Down
3 changes: 2 additions & 1 deletion recipe/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,8 @@ For a first real decision, follow the [complete CPU walkthrough](../docs/getting
regenerate the CPU fixtures; [native worker](omnijev/native.md): serve it on CUDA
and compare it with the reference.
- [Kev-4B preparation](kev/README.md): export all LoRA projections and validate
the Rust FP32 pointer head on CPU; worker integration remains pending.
the Rust FP32 pointer head on CPU; [native worker](kev/native.md) serves eager
text Choice/Noul/Score with explicit numerical acceptance gates.
- [CLM behind the frontend](clm/README.md): run CLM's own server behind the frontend on
CPU with a stub encoder, and what the response comparison has to allow for.
- [pplx-decider reference worker](pplx_decider/README.md): pin
Expand Down
37 changes: 21 additions & 16 deletions recipe/kev/README.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,20 @@
# Kev-4B: CPU export and pointer-head validation
# Kev-4B: checkpoint export and pointer-head validation

This is a preparation recipe for [#27](https://github.com/InferMatrixLab/system1-omni/issues/27),
not a runnable Kev server. The [model contract](../../src/models/kev/README.md)
records the pinned artifacts, numerical boundaries and missing worker pieces.
All commands below run from the repository root. No GPU is needed.
with a separate [native worker recipe](native.md). The [model contract](../../src/models/kev/README.md)
records the pinned artifacts, numerical boundaries and worker ownership.
All commands below run from the repository root. Native serving requires a CUDA
FP32 LoRA merge matching the official default service. Head/unit tests remain CPU-only.

## Environment and downloads

Use Python 3.12 and the stable Rust toolchain used by this repository. The CPU
Use Python 3.12 and the stable Rust toolchain used by this repository. The
export reads about 9.5 GB of inputs and writes about 8.5 GB; reserve at least 25 GB
of free disk and 24 GB of host RAM. It processes one source shard at a time.

```sh
python3.12 -m venv .venv-kev
.venv-kev/bin/pip install torch==2.8.0 --index-url https://download.pytorch.org/whl/cpu
.venv-kev/bin/pip install torch==2.8.0 --index-url https://download.pytorch.org/whl/cu128
.venv-kev/bin/pip install -r recipe/kev/requirements.txt huggingface-hub==1.33.0

.venv-kev/bin/hf download Qwen/Qwen3.5-4B-Base \
Expand All @@ -25,10 +26,17 @@ python3.12 -m venv .venv-kev
--include 'adapter_config.json' 'adapter_model.safetensors' 'head.pt' \
--local-dir weights/kev-adapter

.venv-kev/bin/python recipe/kev/export.py \
--base weights/kev-base --checkpoint weights/kev-adapter --out weights/kev-native
gpu run --gpus 1 --wait 2h --timeout 30m --note 'Kev pinned CUDA export' -- \
.venv-kev/bin/python recipe/kev/export.py \
--merge-device cuda --base weights/kev-base --checkpoint weights/kev-adapter --out weights/kev-native
```

CPU export remains the default for pointer-head validation. Run the Python
export command directly with `--merge-device cpu` and a separate output directory. That artifact is deliberately not accepted by the worker:
nine BF16 weights can differ from official GPU merging despite both merges using
FP32. CUDA exports record the device and exact Torch wheel in `merge_numerics`. The CUDA export
requires Torch 2.8.0+cu128 and disables TF32 for the LoRA matmul.

Every required input must match its pinned SHA-256. The output directory must not
already exist. `kev_export.json` lists merged targets, output hashes, FP32 head
dimensions and calibration, plus the base tokenizer's five delimiter IDs.
Expand Down Expand Up @@ -74,12 +82,9 @@ upstream `PointerHead` class, and uses CPU PyTorch. It does not need Transformer
PEFT or the backbone. The fixture's Torch version includes the wheel's local
suffix, so a different wheel can change provenance even when numbers agree.

## Next integration slice
## Native worker

There is no worker binary, frontend launch command, health endpoint or example
HTTP request yet. The next slice adds exact prompt/row preparation and typed
answers, verified export loading, shared Qwen execution and serial admission.
It must qualify the BF16 merge against the full reference, then test real warmup
and first inference directly and through the frontend. Record hardware, full
dependency pins and numerical gates before those runs. No latency or accuracy
claim follows from export success or isolated head agreement.
The [native worker recipe](native.md) builds and serves the eager Rust/CUDA worker,
reproduces exact processing fixtures, and runs the three-way full-model and HTTP
acceptance checks. Export/head checks alone do not establish full-model parity.
The export/head and complete worker remain separate review boundaries.
Loading
Loading