Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -193,7 +193,7 @@ checkpoint. Public231 concurrency-1 latency for heads `1ba07d1`, `f968e576`,
| autotrust/JEV-27B-VL | [Experimental Rust/CUDA worker](recipe/jev_vl/README.md); single-question text, online inline PNG/JPEG and offline-preencoded image inputs; [validation scope and limits](recipe/jev_vl/validation.md) |
| CLM-v0.1-8B | [External worker with a CPU stub encoder](recipe/clm/README.md); contract checks only, real Qwen3-8B decisions unverified by this recipe |
| LFM2.5-350M | [Transformers worker](recipe/lfm2/README.md); text state and independent choice questions |
| OmniJev-4B v1.1 | [Native Rust/CUDA worker](recipe/omnijev/native.md), eager: one image, Choice, Noul and Score; [reference comparison on sm_89](recipe/omnijev/native.md#comparison-with-the-reference) |
| OmniJev-4B v1.1 | [Native Rust/CUDA worker](recipe/omnijev/native.md), one image, Choice/Noul/Score; merged default and explicit [unmerged one-shot profile](recipe/omnijev/native.md#explicit-unmerged-one-shot-profile), whose full numerical acceptance remains incomplete; [historical sm_89 comparison](recipe/omnijev/native.md#comparison-with-the-reference) |
| Kev-4B | [Pinned CPU export and Rust pointer head](recipe/kev/README.md); no worker or full-model parity yet ([#27](https://github.com/InferMatrixLab/system1-omni/issues/27)) |
| D3 (Decision 3.0) | [Python reference worker](recipe/d3/reference.md) on the checkpoint's pinned runtime; `choice`/`noul`/`score` over text, image and video state; author-run on A100 for text and media serving (see the integration and media PRs) |
| Vela 2.0 0.3B | [Python reference worker](recipe/vela/reference.md) on the checkpoint's pinned runtime; all five kinds (`choice`/`noul`/`score`/`span`/`set`) for text state; CPU FP32 validated (author-run, evidence in the integration PR) |
Expand Down
17 changes: 17 additions & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,23 @@ call.
The Cua-S1 text worker uses it
([#188](https://github.com/InferMatrixLab/system1-omni/issues/188)).

OmniJev's separate unmerged one-shot profile uses
`forward_multimodal_reference_readout`: one complete unpadded multimodal row,
the existing reference norm/projection/attention and FP32 GDN layer loop, then
selected final-norm hidden rows in caller order. It synchronizes before returning
or reusing scratch, including error paths. It creates no prefix state and rejects
Graph and LM targets; the fixed/window/reuse guards above remain in effect.
The complete-request executor owns selected BF16 hidden values independently of
scratch and supplies explicit projection groups through `ReferenceLmTarget` /
`reference_lm_logprobs`. OmniJev maps request chunks, candidate ownership and
option tokens; Qwen executes each declared ordered group as one full-vocabulary
GEMM without inferring question semantics or splitting M. It normalizes with the
existing measured CPU FP64 reduction, an approximation to reference FP32. Both
streams drain before releasing admission, including errors. The optional
`OMNIJEV_PROFILE=unmerged-reference` worker uses the existing all-type warmup and
serial scheduler. Full numerical parity remains a separate acceptance gate.
See the [OmniJev preparation recipe](../recipe/omnijev/README.md#separate-unmerged-reference-artifact).

## Native worker boundaries

The native workers separate `processing.rs` from `executor.rs`; their worker
Expand Down
2 changes: 1 addition & 1 deletion docs/supported-models.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ Models that are being added are also tracked in issues labeled [new model](https
| CLM-v0.1-8B | [External `clm-serve` recipe](../recipe/clm/README.md) with a CPU stub embeddings server | **Stub-encoder contract checks only** ([#23](https://github.com/ThinkFlowLab/system1-omni/pull/23)); not real Qwen3-8B decisions | Real encoder unverified by the merged recipe | Unverified | Python, upstream CLM and head checkpoint; a real encoder requires a separate embeddings server |
| LFM2.5-350M | [Transformers choice worker](../recipe/lfm2/README.md) | Tiny-model regression tests; full checkpoint unverified | L40S FP16 [recorded execution checks](../recipe/lfm2/RESULTS.md) | Unverified | Python 3.12, [pinned dependencies](../src/models/lfm2/requirements.lock.txt), explicit candidate batch size |
| CLM-v0.1-8B | [Native Rust engine](../src/models/clm/README.md), `clm-run`, in front of a Qwen3-8B `/v1/embeddings` server | Engine-only agreement **4.5e-06** on identical vectors ([#29](https://github.com/ThinkFlowLab/system1-omni/pull/29), [validation](../recipe/clm/native/VALIDATION.md)); contract checks documented ([#23](https://github.com/ThinkFlowLab/system1-omni/pull/23)) | **Validated on compute capability 8.9** ([#29](https://github.com/ThinkFlowLab/system1-omni/pull/29), [validation](../recipe/clm/native/VALIDATION.md)) | Unverified | Rust to build and a separate Qwen3-8B `/v1/embeddings` server for the encoder; the engine itself has no device-specific code |
| OmniJev-4B v1.1 | [Native Rust/CUDA worker](../recipe/omnijev/native.md) on the shared Qwen3.5 vision and language path, eager | Contract, preparation, pixels, heads and finishing checked against the reference; no CPU inference | Compared with the reference on compute capability 8.9 for [140 answers](../recipe/omnijev/native.md#comparison-with-the-reference) | Not supported | Compute capability 8.0 or newer, CUDA toolkit to build, the export from `export.py` |
| OmniJev-4B v1.1 | [Native Rust/CUDA worker](../recipe/omnijev/native.md), merged default or explicit [unmerged one-shot profile](../recipe/omnijev/native.md#explicit-unmerged-one-shot-profile), one image and all three types | Contract, preparation, pixels, heads and finishing checked against the reference; no CPU inference | Historical compute capability 8.9 [140-answer comparison](../recipe/omnijev/native.md#comparison-with-the-reference); L20 unmerged same-hidden LM/features checked, full numerical acceptance incomplete | Not supported | Compute capability 8.0 or newer, CUDA toolkit and appropriate reference operators for the selected profile; matching export from `export.py` |
| D3 (Decision 3.0) | [Python reference worker](../recipe/d3/reference.md) on the checkpoint's pinned runtime — text, image and video state over `choice`/`noul`/`score`, with the deliberate transport differences documented in the recipe; the native worker is in progress ([#136](https://github.com/ThinkFlowLab/system1-omni/issues/136)) | Unverified | Author-run on NVIDIA A100 (sm_80): serving smoke and reference parity for text and media (exact decisions, max delta 0.0); evidence in the integration and media PRs. Owner: @MrDongsls. | Unverified | Python 3.12, the [pinned runtime requirements](../recipe/d3/requirements.txt), the [pinned package](https://huggingface.co/vllm-sr/d3) (~52 GB); [media budgets and the 64 MiB total request cap](../recipe/d3/reference.md) |
| Vela 2.0 0.3B | [Python reference worker](../recipe/vela/reference.md) running the checkpoint's pinned runtime | **CPU FP32 validated** (author-run, x86 WSL2, 24 cores): startup with all-kind warmup, the four example requests plus five synthetic rejections, and exact-match reference parity against the pinned upstream server on all nine (four 200s byte-equal incl. key order, agreed 422/413); byte-identical frontend forwarding; evidence in the integration PR. Owner: @MrDongsls. | Unverified (bf16-autocast is an untested deployment option) | Not supported | Python 3.12, the [pinned requirements](../recipe/vela/requirements.txt), the [pinned package](https://huggingface.co/vllm-sr/Vela-2.0-0.3B) (1.24 GB FP32 weights; ~4.4 GB with ONNX/assets) |
| Vela 2.0 0.8B hybrid | [Python reference worker](../recipe/vela/reference.md) running the checkpoint's pinned runtime (same wire layer as 0.3B) | Same surface as 0.3B plus the `span_heads` field (router/broad dispatch); **CPU FP32 parity** (author-run; evidence in the integration PR) | Unverified (bf16-autocast is the evaluated upstream setting but untested here) | Not supported | Python 3.12, the [pinned requirements](../recipe/vela/requirements.txt), the [pinned package](https://huggingface.co/vllm-sr/Vela-2.0-0.8B) (2.02 GB shard + 4.5 MB broad head; ~3 GB FP32 parameters) |
Expand Down
4 changes: 3 additions & 1 deletion recipe/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,9 @@ For a first real decision, follow the [complete CPU walkthrough](../docs/getting
image decisions; bounded H800 validation with explicit deployment limits.
- [OmniJev-4B v1.1 preparation](omnijev/README.md): export the checkpoint and
regenerate the CPU fixtures; [native worker](omnijev/native.md): serve it on CUDA
and compare it with the reference.
and compare it with the reference. The explicit unmerged one-shot profile
supports complete Choice/Noul/Score requests; full numerical acceptance remains
incomplete.
- [Kev-4B preparation](kev/README.md): export all LoRA projections and validate
the Rust FP32 pointer head on CPU; worker integration remains pending.
- [CLM behind the frontend](clm/README.md): run CLM's own server behind the frontend on
Expand Down
78 changes: 77 additions & 1 deletion recipe/omnijev/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,12 +57,88 @@ under a minute.

## Fixtures and tests

### Separate unmerged reference artifact

For the eager one-shot unmerged profile, prepare a separate directory:

```bash
.venv-omnijev/bin/python recipe/omnijev/export.py --base qwen3.5-4b \
--checkpoint omnijev-v1.1 --out omnijev-unmerged-reference \
--language unmerged-reference
```

This writes `omnijev-unmerged-reference/1`: the original BF16 base language
weights, resized tied embedding and saved special rows, plus all 400 separate
FP32 rank-32 adapter tensors at alpha 64 / scale 2. Vision, heads, tokenizer and
calibration retain the same identity. It does not merge or overwrite an existing
artifact. Both loaders reject the other format. Select the matching worker
profile explicitly as described in the [native recipe](native.md#explicit-unmerged-one-shot-profile).

Use `Executor::load_one_shot_reference` and `execute(&prepared)` after preparing
the **complete original request**. `execute_rows` and
`head_outputs(&prepared, false)` also support all three types. Keep every question in the
processor input, since selecting a question before compilation changes its
longest common prefix. The returned selected hidden rows, ordinal probabilities
and calibrated answer from `score_one_shot(&prepared, index)` remain available for
one selected Score's boundary validation. Prefix reuse and language Graphs are
rejected. Set `CUA_S1_GRAPH=0`. Whole-request LM groups include Score rows;
this profile limits all targets together to 16,384 and validates that budget
before dispatch. It fixes the official default chunk size at 64 candidate rows.

The root real-weight checks are opt-in and must run serially on a reserved GPU:

```bash
CUA_S1_GRAPH=0 CUA_S1_CUDA_LIB=/absolute/libqwen3_5_cuda.so \
QWEN3_5_CHECKPOINT=/absolute/omnijev-unmerged-reference \
QWEN3_5_ADAPTER=/absolute/omnijev-unmerged-reference/adapter_model.safetensors \
cargo test --release --locked -p omni-qwen3-5-native --test readout \
reference_selected_rows_preserve_full_row_and_recover -- --ignored --exact --test-threads=1
```

`tests/omnijev/worker.rs::unmerged_reference_original_q2_gate` consumes the original
12-question PNG request and an official full-request oracle supplied through
`OMNIJEV_ORIGINAL_REQUEST` and `OMNIJEV_REFERENCE_ORACLE` (the observer JSON's
`official_full_answer.q2`). Also set `OMNIJEV_REFERENCE_EXPORT` and
`OMNIJEV_CUDA_LIB` to absolute paths. The test checks prefix 581, input usage
1214, mode rejection, repeated hidden/head outputs, the Score decision, and every
returned probability/confidence at the original 0.01 gate. Optional
`OMNIJEV_REFERENCE_RESULT` saves its selected hidden/head output outside the repo.
Passing this bounded check does not establish full-280 or JPEG parity.

`reference_whole_request_outputs_and_recovery` uses the same original request
and checks all-type cardinality, repeated raw head outputs, reuse rejection,
an invalid target followed by recovery, and whole-request finishing.
`reference_actual_lm_groups_and_features` additionally consumes the external
`OMNIJEV_LM_CAPTURE` safetensors and `OMNIJEV_LM_LAYOUT` JSON from an actual
official capture of that complete request. The capture hash and the layout's
original-request identity are checked by the test;
the private original image and checkpoint tensors are not committed. It runs
the production Rust 112-row/31-row projections and compares picked
log-probabilities (absolute 1e-5), all six features (2e-5), and all twelve
same-hidden head outputs (1e-5) against the official branch capture. Set
`OMNIJEV_REFERENCE_EXPORT`, `OMNIJEV_CUDA_LIB` and `OMNIJEV_ORIGINAL_REQUEST` as
above, then name that test with `--ignored --exact --nocapture --test-threads=1`.
`OMNIJEV_LM_RESULT` optionally saves diagnostics outside the source tree.
The Qwen selected-row test also checks owned-hidden lifetime across forwards and
scratch growth, duplicate targets, empty groups, bad width/row/token and the group
row budget. The old fixed/window/FP32GDN guards remain.

On one L20, the original full-context PNG `video.png/q2` check passes: selected
`zq` and three `u` rows exactly match the pinned official unmerged plain-row
control; native confidence is 0.5640 against the normative original branch's
0.5613, within the unchanged 0.01 gate. The branch oracle remains authoritative;
the plain-row control only verifies that path. The whole-request and same-hidden
LM tests above cover the subsequent all-type implementation; original branch
numerical acceptance still needs the complete original corpus.

### Ordinary CPU fixtures

```bash
.venv-omnijev/bin/python tests/omnijev/generate_fixtures.py \
--reference OmniJev --checkpoint omnijev-v1.1 --base qwen3.5-4b --out tests/omnijev/data
cargo test --locked -p omni-omnijev-native
OMNIJEV_CHECKPOINT=$PWD/omnijev-v1.1 OMNIJEV_EXPORT=$PWD/omnijev-export \
cargo test --locked -p omni-omnijev-native -- --include-ignored
cargo test --locked -p omni-omnijev-native --test processing --test heads -- --include-ignored
```

The generator runs the reference's image loading and processor, preparation, layout,
Expand Down
30 changes: 25 additions & 5 deletions recipe/omnijev/export.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
"""Export OmniJev-4B v1.1 for the native worker on CPU.

Merges the language LoRA into the pinned Qwen3.5-4B in BF16, restores the option
By default merges the language LoRA into the pinned Qwen3.5-4B in BF16, restores the option
tokens' embeddings, and writes the base vision tensors, the FP32 heads, the tokenizer
and the calibration next to it. Use the reference environment in README.md; no GPU
is needed. Needs about 18 GB of host RAM and writes 9 GB.
is needed. --language unmerged-reference preserves the original BF16 base and
separate FP32 adapter for the eager one-shot Score library API. Needs about 18 GB
of host RAM and writes 9 GB.
"""

import argparse
Expand Down Expand Up @@ -48,6 +50,8 @@ def main():
parser.add_argument("--base", required=True, type=Path, help=f"{BASE_MODEL_ID} @ {BASE_REVISION}")
parser.add_argument("--checkpoint", required=True, type=Path, help=f"{MODEL_ID} @ {CHECKPOINT_REVISION}")
parser.add_argument("--out", required=True, type=Path)
parser.add_argument("--language", choices=("merged", "unmerged-reference"), default="merged",
help="unmerged-reference preserves base BF16 weights and the separate FP32 adapter")
args = parser.parse_args()
if args.out.exists():
raise ValueError("output already exists; choose a new export directory")
Expand Down Expand Up @@ -94,14 +98,28 @@ def main():
if model.get_output_embeddings().weight.data_ptr() != embedding.data_ptr():
raise ValueError("expected tied input and output embeddings")

model = PeftModel.from_pretrained(model, args.checkpoint).merge_and_unload(safe_merge=True)
unmerged = args.language == "unmerged-reference"
adapter = None
if unmerged:
from safetensors.torch import load_file
adapter = load_file(args.checkpoint / "adapter_model.safetensors")
# This pinned release has 200 projection pairs, all FP32, rank32.
# Do not silently cast an unexpected checkpoint into the reference format.
if len(adapter) != 400 or any(v.dtype != torch.float32 or v.ndim != 2
or v.shape[0 if ".lora_A." in k else 1] != 32
for k, v in adapter.items()):
raise ValueError("expected all 400 FP32 rank32 OmniJev adapter tensors")
else:
model = PeftModel.from_pretrained(model, args.checkpoint).merge_and_unload(safe_merge=True)
# Write next to the output and rename at the end, so a failed or killed export
# leaves no output behind; the next run removes what it left.
staging = args.out.with_name(f".{args.out.name}.partial")
if staging.exists():
shutil.rmtree(staging)
model.model.language_model.save_pretrained(staging, max_shard_size="5GB")
model.config.to_json_file(staging / "config.json")
if unmerged:
save_file(adapter, staging / "adapter_model.safetensors")
vision = {f"model.visual.{k}": v.contiguous() for k, v in model.model.visual.state_dict().items()}
save_file(vision, staging / "vision.safetensors")

Expand All @@ -118,10 +136,12 @@ def main():
outputs = {p.name: sha256(p) for p in sorted(staging.iterdir()) if p.is_file()}
# Written last: the worker refuses incomplete exports.
(staging / "omnijev_export.json").write_text(json.dumps({
"format": "omnijev-merged/1",
"format": "omnijev-unmerged-reference/1" if unmerged else "omnijev-merged/1",
"model_id": MODEL_ID, "checkpoint_revision": CHECKPOINT_REVISION,
"base_model_id": BASE_MODEL_ID, "base_revision": BASE_REVISION,
"lora": {"rank": 32, "alpha": 64, "merged": "bfloat16"},
"lora": ({"rank": 32, "alpha": 64, "scale": 2, "dtype": "float32",
"file": "adapter_model.safetensors", "tensors": 400}
if unmerged else {"rank": 32, "alpha": 64, "merged": "bfloat16"}),
"vocabulary": VOCABULARY, "option_tokens": OPTION_TOKENS, "max_pixels": MAX_PIXELS,
"heads": {"norm": meta["norm"], "lm_features": True, "ordinal": True},
"calibration": {"temperatures": temperatures, "noul_bias": noul_bias},
Expand Down
Loading
Loading