Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,15 @@ call.
The Cua-S1 text worker uses it
([#188](https://github.com/InferMatrixLab/system1-omni/issues/188)).

OmniJev's separate unmerged one-shot Score library profile uses
`forward_multimodal_reference_readout`: one complete unpadded multimodal row,
the existing reference norm/projection/attention and FP32 GDN layer loop, then
selected final-norm hidden rows in caller order. It synchronizes before returning
or reusing scratch, including error paths. It creates no prefix state and rejects
Graph and LM targets; the fixed/window/reuse guards above remain in effect.
This is a bounded library capability, with full worker/all-type parity unverified.
See the [OmniJev preparation recipe](../recipe/omnijev/README.md#separate-unmerged-reference-artifact).

## Native worker boundaries

The native workers separate `processing.rs` from `executor.rs`; their worker
Expand Down
58 changes: 57 additions & 1 deletion recipe/omnijev/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,12 +57,68 @@ under a minute.

## Fixtures and tests

### Separate unmerged reference artifact

For the eager one-shot Score **library API**, prepare a separate directory:

```bash
.venv-omnijev/bin/python recipe/omnijev/export.py --base qwen3.5-4b \
--checkpoint omnijev-v1.1 --out omnijev-unmerged-reference \
--language unmerged-reference
```

This writes `omnijev-unmerged-reference/1`: the original BF16 base language
weights, resized tied embedding and saved special rows, plus all 400 separate
FP32 rank-32 adapter tensors at alpha 64 / scale 2. Vision, heads, tokenizer and
calibration retain the same identity. It does not merge or overwrite an existing
artifact. Both loaders reject the other format; the merged worker does not serve
this artifact.

Use `Executor::load_one_shot_reference` and `score_one_shot(&prepared, index)`
after preparing the **complete original request**. Keep every question in the
processor input, since selecting a question before compilation changes its
longest common prefix. The returned selected hidden rows, ordinal probabilities
and calibrated answer support boundary validation. This API supports one Score
question at a time; whole-request execution, Choice/Noul, LM targets, prefix
reuse and Graphs are rejected. Set `CUA_S1_GRAPH=0`. There is no worker switch
for this incomplete all-type profile.

The root real-weight checks are opt-in and must run serially on a reserved GPU:

```bash
CUA_S1_GRAPH=0 CUA_S1_CUDA_LIB=/absolute/libqwen3_5_cuda.so \
QWEN3_5_CHECKPOINT=/absolute/omnijev-unmerged-reference \
QWEN3_5_ADAPTER=/absolute/omnijev-unmerged-reference/adapter_model.safetensors \
cargo test --release --locked -p omni-qwen3-5-native --test readout \
reference_selected_rows_preserve_full_row_and_recover -- --ignored --exact --test-threads=1
```

`tests/omnijev/worker.rs::unmerged_reference_original_q2_gate` consumes the original
12-question PNG request and an official full-request oracle supplied through
`OMNIJEV_ORIGINAL_REQUEST` and `OMNIJEV_REFERENCE_ORACLE` (the observer JSON's
`official_full_answer.q2`). Also set `OMNIJEV_REFERENCE_EXPORT` and
`OMNIJEV_CUDA_LIB` to absolute paths. The test checks prefix 581, input usage
1214, mode rejection, repeated hidden/head outputs, the Score decision, and every
returned probability/confidence at the original 0.01 gate. Optional
`OMNIJEV_REFERENCE_RESULT` saves its selected hidden/head output outside the repo.
Passing this bounded check does not establish all-type/full-280, JPEG or HTTP
parity; those require separate complete execution evidence.

On one L20, the original full-context PNG `video.png/q2` check passes: selected
`zq` and three `u` rows exactly match the pinned official unmerged plain-row
control; native confidence is 0.5640 against the normative original branch's
0.5613, within the unchanged 0.01 gate. The branch oracle remains authoritative;
the plain-row control only verifies this library path. This coverage is limited
to that observed Score case and the selected-row state/recovery tests.

### Ordinary CPU fixtures

```bash
.venv-omnijev/bin/python tests/omnijev/generate_fixtures.py \
--reference OmniJev --checkpoint omnijev-v1.1 --base qwen3.5-4b --out tests/omnijev/data
cargo test --locked -p omni-omnijev-native
OMNIJEV_CHECKPOINT=$PWD/omnijev-v1.1 OMNIJEV_EXPORT=$PWD/omnijev-export \
cargo test --locked -p omni-omnijev-native -- --include-ignored
cargo test --locked -p omni-omnijev-native --test processing --test heads -- --include-ignored
```

The generator runs the reference's image loading and processor, preparation, layout,
Expand Down
30 changes: 25 additions & 5 deletions recipe/omnijev/export.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
"""Export OmniJev-4B v1.1 for the native worker on CPU.

Merges the language LoRA into the pinned Qwen3.5-4B in BF16, restores the option
By default merges the language LoRA into the pinned Qwen3.5-4B in BF16, restores the option
tokens' embeddings, and writes the base vision tensors, the FP32 heads, the tokenizer
and the calibration next to it. Use the reference environment in README.md; no GPU
is needed. Needs about 18 GB of host RAM and writes 9 GB.
is needed. --language unmerged-reference preserves the original BF16 base and
separate FP32 adapter for the eager one-shot Score library API. Needs about 18 GB
of host RAM and writes 9 GB.
"""

import argparse
Expand Down Expand Up @@ -48,6 +50,8 @@ def main():
parser.add_argument("--base", required=True, type=Path, help=f"{BASE_MODEL_ID} @ {BASE_REVISION}")
parser.add_argument("--checkpoint", required=True, type=Path, help=f"{MODEL_ID} @ {CHECKPOINT_REVISION}")
parser.add_argument("--out", required=True, type=Path)
parser.add_argument("--language", choices=("merged", "unmerged-reference"), default="merged",
help="unmerged-reference preserves base BF16 weights and the separate FP32 adapter")
args = parser.parse_args()
if args.out.exists():
raise ValueError("output already exists; choose a new export directory")
Expand Down Expand Up @@ -94,14 +98,28 @@ def main():
if model.get_output_embeddings().weight.data_ptr() != embedding.data_ptr():
raise ValueError("expected tied input and output embeddings")

model = PeftModel.from_pretrained(model, args.checkpoint).merge_and_unload(safe_merge=True)
unmerged = args.language == "unmerged-reference"
adapter = None
if unmerged:
from safetensors.torch import load_file
adapter = load_file(args.checkpoint / "adapter_model.safetensors")
# This pinned release has 200 projection pairs, all FP32, rank32.
# Do not silently cast an unexpected checkpoint into the reference format.
if len(adapter) != 400 or any(v.dtype != torch.float32 or v.ndim != 2
or v.shape[0 if ".lora_A." in k else 1] != 32
for k, v in adapter.items()):
raise ValueError("expected all 400 FP32 rank32 OmniJev adapter tensors")
else:
model = PeftModel.from_pretrained(model, args.checkpoint).merge_and_unload(safe_merge=True)
# Write next to the output and rename at the end, so a failed or killed export
# leaves no output behind; the next run removes what it left.
staging = args.out.with_name(f".{args.out.name}.partial")
if staging.exists():
shutil.rmtree(staging)
model.model.language_model.save_pretrained(staging, max_shard_size="5GB")
model.config.to_json_file(staging / "config.json")
if unmerged:
save_file(adapter, staging / "adapter_model.safetensors")
vision = {f"model.visual.{k}": v.contiguous() for k, v in model.model.visual.state_dict().items()}
save_file(vision, staging / "vision.safetensors")

Expand All @@ -118,10 +136,12 @@ def main():
outputs = {p.name: sha256(p) for p in sorted(staging.iterdir()) if p.is_file()}
# Written last: the worker refuses incomplete exports.
(staging / "omnijev_export.json").write_text(json.dumps({
"format": "omnijev-merged/1",
"format": "omnijev-unmerged-reference/1" if unmerged else "omnijev-merged/1",
"model_id": MODEL_ID, "checkpoint_revision": CHECKPOINT_REVISION,
"base_model_id": BASE_MODEL_ID, "base_revision": BASE_REVISION,
"lora": {"rank": 32, "alpha": 64, "merged": "bfloat16"},
"lora": ({"rank": 32, "alpha": 64, "scale": 2, "dtype": "float32",
"file": "adapter_model.safetensors", "tensors": 400}
if unmerged else {"rank": 32, "alpha": 64, "merged": "bfloat16"}),
"vocabulary": VOCABULARY, "option_tokens": OPTION_TOKENS, "max_pixels": MAX_PIXELS,
"heads": {"norm": meta["norm"], "lm_features": True, "ordinal": True},
"calibration": {"temperatures": temperatures, "noul_bias": noul_bias},
Expand Down
54 changes: 52 additions & 2 deletions recipe/omnijev/native.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,50 @@ cargo build --release --locked -p omni-omnijev-native
OMNIJEV_MODEL=$PWD/omnijev-export target/release/omni-omnijev-native
```

## Explicit BF16 reference vision

`OMNIJEV_VISION_NUMERICS=reference` selects the unadapted OmniJev-4B vision
arithmetic. The default, `legacy`, retains the existing worker output. Unknown
values fail startup. The reference selection needs the optional cuDNN 9 build
and additive OmniJev CUDA symbols; requesting it with an old library fails with
a rebuild instruction. Build and launch it explicitly:

```sh
CUDNN_INCLUDE_DIR=/absolute/cudnn/include CUDNN_LIB_DIR=/absolute/cudnn/lib \
src/backends/cuda/qwen3_5/build.sh target/release 89
OMNIJEV_VISION_NUMERICS=reference OMNIJEV_MODEL=$PWD/omnijev-export \
target/release/omni-omnijev-native
```

This path uses a cuDNN BF16 patch convolution, vectorized Welford LayerNorm,
the existing framework GEMM policy with Torch's 1 MiB heuristic budget
(including its reduction schemes), FP32
rotary angles/trigonometry, and head-64 reference attention. It retains BF16
operator outputs and the separate patch-bias addition. It selects vision only:
the language weights still contain the merged LoRA, and language/readout and
calibration arithmetic are unchanged. The official unmerged BF16 model and the
original probability/decision gates remain the acceptance reference. Vision
alignment alone does not establish whole-model numerical acceptance.

The profile uses the existing four-grid FIFO, rejects vision adapters and
explicit bounded-cache policies, and must be selected before the first forward.
`CUA_S1_VISION_GRAPH=1` can capture the vision path after warmup; descriptors,
GEMM plans and workspace are prepared before capture. Validation is specific to
the tested GPU/library stack. See the [stage and operator regressions](../../tests/qwen3_5/README.md#omnijev-4b-reference-vision).

On one L20 (sm_89), Rust 1.100/CUDA 12.8 with cuDNN 9.24 and the documented
reference environment, the original `video.png/noul_region` case matches all
40 vision boundaries exactly, including the 24 blocks and merger. The complete
original 28-request/280-answer comparison still fails the unchanged acceptance
gate: 177 answers are within 0.01, maximum returned-field drift is 0.1758,
and three protected decisions differ (`board.jpg/q8`, `arcade.jpg/long_start`,
`web.jpg/one_token`). All 140 PNG decisions agree, but 20 PNG answers still
exceed 0.01. Unrounded values retain the same three protected failures. The
JPEG failure set changes from the legacy path; do not infer a uniform
whole-model improvement. JPEG preparation and language/LM/unmerged-LoRA
differences still need separate controlled validation. This profile stays
opt-in and does not complete #114. No performance result is claimed for it.

`CUA_S1_GRAPH=1` also enables language window Graphs in this worker, through the
shared `Model::readout_window` implementation. It defaults off. Request and
question prefixes and option continuations can capture and replay; current token,
Expand Down Expand Up @@ -68,11 +112,16 @@ CUDA library, as absolute paths:
```sh
cargo test --locked -p omni-omnijev-native
OMNIJEV_EXPORT=$PWD/omnijev-export OMNIJEV_CUDA_LIB=$PWD/target/release/libqwen3_5_cuda.so \
cargo test --release --locked -p omni-omnijev-native --test worker -- --ignored --test-threads=1
cargo test --release --locked -p omni-omnijev-native --test worker -- --ignored --test-threads=1 --skip reference
QWEN3_5_CHECKPOINT=$PWD/omnijev-export CUA_S1_CUDA_LIB=$PWD/target/release/libqwen3_5_cuda.so \
cargo test --release --locked -p omni-qwen3-5-native --test readout -- --ignored --test-threads=1
cargo test --release --locked -p omni-qwen3-5-native --test readout -- --ignored --test-threads=1 --skip reference
```

These commands select the merged worker's tests. The separate unmerged one-shot
Score library tests require their own artifact and Graph-off settings; see the
[preparation recipe](README.md#separate-unmerged-reference-artifact). They do not
establish a replacement HTTP worker profile.

The direct OmniJev window-Graph regression selects eager and Graph models without
changing process-wide environment variables. It compares raw FP32 head bits and
answers (excluding timing fields), requires actual capture and warm replay, and
Expand Down Expand Up @@ -108,6 +157,7 @@ python3 recipe/omnijev/validate.py compare reference.json native.json
options that differ in one token or share a long start, a Choice where no option fits, a
32-option Choice, a 10-level Score, and plain and region Noul.

The historical comparison and latency figures below use the legacy vision path.
On one RTX 6000 Ada (sm_89, CUDA 13.2), with the reference's seven v1.1 demo stills and
both question sets (140 answers), the input-token counts equal the reference's. The
largest probability difference per answer, against the reference in float32 (`--fp32`)
Expand Down
21 changes: 21 additions & 0 deletions src/backends/cuda/qwen3_5/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -225,3 +225,24 @@ run the matched projection regression in `tests/qwen3_5/kernels.rs` and the
[Cua screenshot qualification](../../../../recipe/cua_s1/native_screenshot.md#full-prompt-matched-algorithms)
for each supported GPU/library/model combination. Other numerical profiles,
vision GEMMs and the existing default/fixed policies stay on their own handles.

## Optional OmniJev-4B vision arithmetic

The additive ABI 9 `cs1_omnijev_vision_*` group is resolved only when explicitly
requested by `VisionModel::enable_omnijev_numerics`. Older libraries remain
loadable by existing callers. The group is separate from JEMM's guarded 27B
entry points: it accepts only unadapted 1024-wide/head-64 single-image vision,
and reuses the existing framework GEMM handle with the pinned Torch vision
1 MiB heuristic budget, without changing generic policies.
`cs1_reference_patch_create_v2(rows, hidden, stream)` adds a width-aware cuDNN
plan constructor; the original constructor still fixes hidden size to 1152.
Execute/destroy and synchronization ownership are shared. Build with cuDNN 9
and rebuild the library before selecting the profile.

The 4B attention workspace query returns FP32 elements for the exact bounded
row count (up to 4608), without the head-72 padding region. Q/K are compact;
V stays interleaved in QKV. Rust owns descriptor/workspace/Graph lifetimes and
rejects a bounded cache because opaque cuDNN allocations are not counted by
that cache. The default shared vision path and JEMM numerics stay unchanged.
See the [OmniJev recipe](../../../../recipe/omnijev/native.md#explicit-bf16-reference-vision)
and [regressions](../../../../tests/qwen3_5/README.md#omnijev-4b-reference-vision).
10 changes: 10 additions & 0 deletions src/backends/cuda/qwen3_5/ops.h
Original file line number Diff line number Diff line change
Expand Up @@ -264,6 +264,16 @@ void* cs1_reference_patch_create(int rows,void* stream);
int cs1_reference_patch(void* plan,const void* x,const void* weight,void* out);
void cs1_reference_patch_destroy(void* plan);

// Optional OmniJev-4B vision arithmetic (additive ABI 9). Select explicitly;
// legacy/default and JEMM symbols keep their existing layouts and policies.
// Patch plans use the existing execute/destroy calls; width is 1024 or 1152.
void* cs1_reference_patch_create_v2(int rows,int hidden,void* stream);
int cs1_omnijev_vision_norm(const void* x,const void* w,const void* b,void* y,int rows,int d,void* stream);
// co contains FP32 angles, si is unused, as in the 27B reference rotary API.
int cs1_omnijev_vision_rope(const void* qkv,const float* co,const float* si,void* q,void* k,int n,int hidden,int head_dim,void* stream);
size_t cs1_omnijev_vision_attention_workspace_floats(int rows,int heads);
int cs1_omnijev_vision_attention(const void* q,const void* k,const void* v,void* out,int rows,int heads,int head_dim,void* workspace,void* stream);

#ifdef __cplusplus
}
#endif
14 changes: 9 additions & 5 deletions src/backends/cuda/qwen3_5/reference_patch.cu
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,9 @@ extern "C" int cs1_reference_available() {
return 0;
#endif
}
extern "C" void* cs1_reference_patch_create(int rows,void* stream) {
extern "C" void* cs1_reference_patch_create_v2(int rows,int hidden,void* stream) {
#ifdef CS1_REFERENCE_CUDNN
if(rows<=0 || rows>65536) return nullptr;
if(rows<=0 || rows>65536 || (hidden!=1024 && hidden!=1152)) return nullptr;
PatchPlan* p=new(std::nothrow) PatchPlan;
if(!p) return nullptr;
bool ready=false;
Expand All @@ -45,8 +45,8 @@ extern "C" void* cs1_reference_patch_create(int rows,void* stream) {
CHECK_CUDNN(cudnnCreateFilterDescriptor(&p->filter));
CHECK_CUDNN(cudnnCreateConvolutionDescriptor(&p->conv));
int xs[]={rows,3,2,16,16},xt[]={1536,512,256,16,1};
int ys[]={rows,1152,1,1,1},yt[]={1152,1,1,1,1};
int ws[]={1152,3,2,16,16},pad[]={0,0,0},step[]={2,16,16},dilation[]={1,1,1};
int ys[]={rows,hidden,1,1,1},yt[]={hidden,1,1,1,1};
int ws[]={hidden,3,2,16,16},pad[]={0,0,0},step[]={2,16,16},dilation[]={1,1,1};
CHECK_CUDNN(cudnnSetTensorNdDescriptor(p->input,CUDNN_DATA_BFLOAT16,5,xs,xt));
CHECK_CUDNN(cudnnSetTensorNdDescriptor(p->output,CUDNN_DATA_BFLOAT16,5,ys,yt));
CHECK_CUDNN(cudnnSetFilterNdDescriptor(p->filter,CUDNN_DATA_BFLOAT16,CUDNN_TENSOR_NCHW,5,ws));
Expand All @@ -60,9 +60,13 @@ extern "C" void* cs1_reference_patch_create(int rows,void* stream) {
if(!ready) {delete p;return nullptr;}
return p;
#else
(void)rows;(void)stream;return nullptr;
(void)rows;(void)hidden;(void)stream;return nullptr;
#endif
}
// Preserve the original 27B descriptor layout and symbol.
extern "C" void* cs1_reference_patch_create(int rows,void* stream) {
return cs1_reference_patch_create_v2(rows,1152,stream);
}
extern "C" int cs1_reference_patch(void* plan,const void* x,const void* w,void* y) {
#ifdef CS1_REFERENCE_CUDNN
auto* p=static_cast<PatchPlan*>(plan);
Expand Down
13 changes: 13 additions & 0 deletions src/backends/cuda/qwen3_5/reference_vision.cu
Original file line number Diff line number Diff line change
Expand Up @@ -98,3 +98,16 @@ extern "C" int cs1_reference_vision_attention(const void* q,const void* k,const
reference_vision::unpack_attention_kernel<<<((size_t)n*heads*dh+255)/256,256,0,(cudaStream_t)stream>>>(po,(bf16*)out,(size_t)n*heads*dh,dh);
return cudaGetLastError();
}

// Additive, explicit OmniJev-4B operators. The 27B guards and callers stay unchanged.
extern "C" int cs1_omnijev_vision_norm(const void* x,const void* w,const void* b,void* y,int rows,int d,void* stream) {
if(rows<=0 || d!=1024) return cudaErrorInvalidValue;
reference_vision::reference_norm_kernel<<<rows,128,0,(cudaStream_t)stream>>>((const bf16*)x,(const bf16*)w,(const bf16*)b,(bf16*)y,d);
return cudaGetLastError();
}
extern "C" int cs1_omnijev_vision_rope(const void* qkv,const float* co,const float* si,void* q,void* k,int n,int hidden,int dh,void* stream) {
if(n<=0 || hidden!=1024 || dh!=64) return cudaErrorInvalidValue;
const size_t count=(size_t)n*hidden;
reference_vision::rope_v2_kernel<<<(count+255)/256,256,0,(cudaStream_t)stream>>>((const bf16*)qkv,co,si,(bf16*)q,(bf16*)k,count,hidden,dh);
return cudaGetLastError();
}
Loading
Loading