Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions recipe/omnijev/native.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,50 @@ cargo build --release --locked -p omni-omnijev-native
OMNIJEV_MODEL=$PWD/omnijev-export target/release/omni-omnijev-native
```

## Explicit BF16 reference vision

`OMNIJEV_VISION_NUMERICS=reference` selects the unadapted OmniJev-4B vision
arithmetic. The default, `legacy`, retains the existing worker output. Unknown
values fail startup. The reference selection needs the optional cuDNN 9 build
and additive OmniJev CUDA symbols; requesting it with an old library fails with
a rebuild instruction. Build and launch it explicitly:

```sh
CUDNN_INCLUDE_DIR=/absolute/cudnn/include CUDNN_LIB_DIR=/absolute/cudnn/lib \
src/backends/cuda/qwen3_5/build.sh target/release 89
OMNIJEV_VISION_NUMERICS=reference OMNIJEV_MODEL=$PWD/omnijev-export \
target/release/omni-omnijev-native
```

This path uses a cuDNN BF16 patch convolution, vectorized Welford LayerNorm,
the existing framework GEMM policy with Torch's 1 MiB heuristic budget
(including its reduction schemes), FP32
rotary angles/trigonometry, and head-64 reference attention. It retains BF16
operator outputs and the separate patch-bias addition. It selects vision only:
the language weights still contain the merged LoRA, and language/readout and
calibration arithmetic are unchanged. The official unmerged BF16 model and the
original probability/decision gates remain the acceptance reference. Vision
alignment alone does not establish whole-model numerical acceptance.

The profile uses the existing four-grid FIFO, rejects vision adapters and
explicit bounded-cache policies, and must be selected before the first forward.
`CUA_S1_VISION_GRAPH=1` can capture the vision path after warmup; descriptors,
GEMM plans and workspace are prepared before capture. Validation is specific to
the tested GPU/library stack. See the [stage and operator regressions](../../tests/qwen3_5/README.md#omnijev-4b-reference-vision).

On one L20 (sm_89), Rust 1.100/CUDA 12.8 with cuDNN 9.24 and the documented
reference environment, the original `video.png/noul_region` case matches all
40 vision boundaries exactly, including the 24 blocks and merger. The complete
original 28-request/280-answer comparison still fails the unchanged acceptance
gate: 177 answers are within 0.01, maximum returned-field drift is 0.1758,
and three protected decisions differ (`board.jpg/q8`, `arcade.jpg/long_start`,
`web.jpg/one_token`). All 140 PNG decisions agree, but 20 PNG answers still
exceed 0.01. Unrounded values retain the same three protected failures. The
JPEG failure set changes from the legacy path; do not infer a uniform
whole-model improvement. JPEG preparation and language/LM/unmerged-LoRA
differences still need separate controlled validation. This profile stays
opt-in and does not complete #114. No performance result is claimed for it.

`CUA_S1_GRAPH=1` also enables language window Graphs in this worker, through the
shared `Model::readout_window` implementation. It defaults off. Request and
question prefixes and option continuations can capture and replay; current token,
Expand Down Expand Up @@ -108,6 +152,7 @@ python3 recipe/omnijev/validate.py compare reference.json native.json
options that differ in one token or share a long start, a Choice where no option fits, a
32-option Choice, a 10-level Score, and plain and region Noul.

The historical comparison and latency figures below use the legacy vision path.
On one RTX 6000 Ada (sm_89, CUDA 13.2), with the reference's seven v1.1 demo stills and
both question sets (140 answers), the input-token counts equal the reference's. The
largest probability difference per answer, against the reference in float32 (`--fp32`)
Expand Down
21 changes: 21 additions & 0 deletions src/backends/cuda/qwen3_5/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -225,3 +225,24 @@ run the matched projection regression in `tests/qwen3_5/kernels.rs` and the
[Cua screenshot qualification](../../../../recipe/cua_s1/native_screenshot.md#full-prompt-matched-algorithms)
for each supported GPU/library/model combination. Other numerical profiles,
vision GEMMs and the existing default/fixed policies stay on their own handles.

## Optional OmniJev-4B vision arithmetic

The additive ABI 9 `cs1_omnijev_vision_*` group is resolved only when explicitly
requested by `VisionModel::enable_omnijev_numerics`. Older libraries remain
loadable by existing callers. The group is separate from JEMM's guarded 27B
entry points: it accepts only unadapted 1024-wide/head-64 single-image vision,
and reuses the existing framework GEMM handle with the pinned Torch vision
1 MiB heuristic budget, without changing generic policies.
`cs1_reference_patch_create_v2(rows, hidden, stream)` adds a width-aware cuDNN
plan constructor; the original constructor still fixes hidden size to 1152.
Execute/destroy and synchronization ownership are shared. Build with cuDNN 9
and rebuild the library before selecting the profile.

The 4B attention workspace query returns FP32 elements for the exact bounded
row count (up to 4608), without the head-72 padding region. Q/K are compact;
V stays interleaved in QKV. Rust owns descriptor/workspace/Graph lifetimes and
rejects a bounded cache because opaque cuDNN allocations are not counted by
that cache. The default shared vision path and JEMM numerics stay unchanged.
See the [OmniJev recipe](../../../../recipe/omnijev/native.md#explicit-bf16-reference-vision)
and [regressions](../../../../tests/qwen3_5/README.md#omnijev-4b-reference-vision).
10 changes: 10 additions & 0 deletions src/backends/cuda/qwen3_5/ops.h
Original file line number Diff line number Diff line change
Expand Up @@ -264,6 +264,16 @@ void* cs1_reference_patch_create(int rows,void* stream);
int cs1_reference_patch(void* plan,const void* x,const void* weight,void* out);
void cs1_reference_patch_destroy(void* plan);

// Optional OmniJev-4B vision arithmetic (additive ABI 9). Select explicitly;
// legacy/default and JEMM symbols keep their existing layouts and policies.
// Patch plans use the existing execute/destroy calls; width is 1024 or 1152.
void* cs1_reference_patch_create_v2(int rows,int hidden,void* stream);
int cs1_omnijev_vision_norm(const void* x,const void* w,const void* b,void* y,int rows,int d,void* stream);
// co contains FP32 angles, si is unused, as in the 27B reference rotary API.
int cs1_omnijev_vision_rope(const void* qkv,const float* co,const float* si,void* q,void* k,int n,int hidden,int head_dim,void* stream);
size_t cs1_omnijev_vision_attention_workspace_floats(int rows,int heads);
int cs1_omnijev_vision_attention(const void* q,const void* k,const void* v,void* out,int rows,int heads,int head_dim,void* workspace,void* stream);

#ifdef __cplusplus
}
#endif
14 changes: 9 additions & 5 deletions src/backends/cuda/qwen3_5/reference_patch.cu
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,9 @@ extern "C" int cs1_reference_available() {
return 0;
#endif
}
extern "C" void* cs1_reference_patch_create(int rows,void* stream) {
extern "C" void* cs1_reference_patch_create_v2(int rows,int hidden,void* stream) {
#ifdef CS1_REFERENCE_CUDNN
if(rows<=0 || rows>65536) return nullptr;
if(rows<=0 || rows>65536 || (hidden!=1024 && hidden!=1152)) return nullptr;
PatchPlan* p=new(std::nothrow) PatchPlan;
if(!p) return nullptr;
bool ready=false;
Expand All @@ -45,8 +45,8 @@ extern "C" void* cs1_reference_patch_create(int rows,void* stream) {
CHECK_CUDNN(cudnnCreateFilterDescriptor(&p->filter));
CHECK_CUDNN(cudnnCreateConvolutionDescriptor(&p->conv));
int xs[]={rows,3,2,16,16},xt[]={1536,512,256,16,1};
int ys[]={rows,1152,1,1,1},yt[]={1152,1,1,1,1};
int ws[]={1152,3,2,16,16},pad[]={0,0,0},step[]={2,16,16},dilation[]={1,1,1};
int ys[]={rows,hidden,1,1,1},yt[]={hidden,1,1,1,1};
int ws[]={hidden,3,2,16,16},pad[]={0,0,0},step[]={2,16,16},dilation[]={1,1,1};
CHECK_CUDNN(cudnnSetTensorNdDescriptor(p->input,CUDNN_DATA_BFLOAT16,5,xs,xt));
CHECK_CUDNN(cudnnSetTensorNdDescriptor(p->output,CUDNN_DATA_BFLOAT16,5,ys,yt));
CHECK_CUDNN(cudnnSetFilterNdDescriptor(p->filter,CUDNN_DATA_BFLOAT16,CUDNN_TENSOR_NCHW,5,ws));
Expand All @@ -60,9 +60,13 @@ extern "C" void* cs1_reference_patch_create(int rows,void* stream) {
if(!ready) {delete p;return nullptr;}
return p;
#else
(void)rows;(void)stream;return nullptr;
(void)rows;(void)hidden;(void)stream;return nullptr;
#endif
}
// Preserve the original 27B descriptor layout and symbol.
extern "C" void* cs1_reference_patch_create(int rows,void* stream) {
return cs1_reference_patch_create_v2(rows,1152,stream);
}
extern "C" int cs1_reference_patch(void* plan,const void* x,const void* w,void* y) {
#ifdef CS1_REFERENCE_CUDNN
auto* p=static_cast<PatchPlan*>(plan);
Expand Down
13 changes: 13 additions & 0 deletions src/backends/cuda/qwen3_5/reference_vision.cu
Original file line number Diff line number Diff line change
Expand Up @@ -98,3 +98,16 @@ extern "C" int cs1_reference_vision_attention(const void* q,const void* k,const
reference_vision::unpack_attention_kernel<<<((size_t)n*heads*dh+255)/256,256,0,(cudaStream_t)stream>>>(po,(bf16*)out,(size_t)n*heads*dh,dh);
return cudaGetLastError();
}

// Additive, explicit OmniJev-4B operators. The 27B guards and callers stay unchanged.
extern "C" int cs1_omnijev_vision_norm(const void* x,const void* w,const void* b,void* y,int rows,int d,void* stream) {
if(rows<=0 || d!=1024) return cudaErrorInvalidValue;
reference_vision::reference_norm_kernel<<<rows,128,0,(cudaStream_t)stream>>>((const bf16*)x,(const bf16*)w,(const bf16*)b,(bf16*)y,d);
return cudaGetLastError();
}
extern "C" int cs1_omnijev_vision_rope(const void* qkv,const float* co,const float* si,void* q,void* k,int n,int hidden,int dh,void* stream) {
if(n<=0 || hidden!=1024 || dh!=64) return cudaErrorInvalidValue;
const size_t count=(size_t)n*hidden;
reference_vision::rope_v2_kernel<<<(count+255)/256,256,0,(cudaStream_t)stream>>>((const bf16*)qkv,co,si,(bf16*)q,(bf16*)k,count,hidden,dh);
return cudaGetLastError();
}
20 changes: 20 additions & 0 deletions src/backends/cuda/qwen3_5/reference_vision_attention.cu
Original file line number Diff line number Diff line change
Expand Up @@ -194,3 +194,23 @@ extern "C" int cs1_reference_vision_attention_padded(const void* q,const void* k
}

extern "C" size_t cs1_reference_vision_attention_workspace_floats(int capacity,int heads) { return cs1::reference::workspace_floats(capacity,heads,80,128); }

// The pinned 4B SDPA uses noncausal head-64 FlashAttention with 128-key tiles,
// reverse key traversal and unscaled maxima. Keep Q/K compact and V in QKV.
extern "C" size_t cs1_omnijev_vision_attention_workspace_floats(int rows,int heads) {
if(rows<=0 || rows>4608 || heads!=16) return 0;
return (size_t)rows*heads*65;
}
extern "C" int cs1_omnijev_vision_attention(const void* q,const void* k,const void* v,void* out,int rows,int heads,int dh,void* workspace,void* stream) {
using namespace cs1;namespace f=vision_reference;
if(rows<=0 || rows>4608 || heads!=16 || dh!=64 || !workspace) return cudaErrorInvalidValue;
float* memory=(float*)workspace;float* lse=memory+(size_t)rows*heads*64;
cudaFuncSetAttribute(f::flash_kernel<64,128>,cudaFuncAttributeMaxDynamicSharedMemorySize,f::SMEM_BYTES<64,128>);
f::flash_kernel<64,128><<<dim3((rows+63)/64,heads),128,f::SMEM_BYTES<64,128>,(cudaStream_t)stream>>>((const bf16*)q,(const bf16*)k,(const bf16*)v,heads*64*3,memory,lse,rows,heads,heads,0.125f,1);
auto status=cudaGetLastError();
if(status==cudaSuccess) {
f::combine<<<((size_t)rows*heads*64+127)/128,128,0,(cudaStream_t)stream>>>(memory,lse,(bf16*)out,rows,heads,64,1);
status=cudaGetLastError();
}
return status;
}
15 changes: 15 additions & 0 deletions src/models/omnijev/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,21 @@ plus row up to rounding, execution may choose its own split points.

## Execution

`OMNIJEV_VISION_NUMERICS` explicitly selects `legacy` (the unchanged default)
or `reference`. The latter selects `VisionModel::enable_omnijev_numerics` for
the unadapted 1024-wide, head-64 vision tower. It requires a cuDNN 9 backend
build and the optional OmniJev symbols. Library callers can instead use
`Executor::load_with_vision_numerics` with `VisionNumerics::Reference`.
The [native recipe](../../../recipe/omnijev/native.md#explicit-bf16-reference-vision)
describes the build and scope. This is a vision arithmetic selection: it does
not replace the official unmerged BF16 reference, modify exported tensors, or
change language/LM/LoRA execution or the original numerical gates. Whole-model
precision remains a separate validation requirement.
The L20 visual-stage regression is exact for the pinned protected PNG case;
the original full 280-answer gate still fails (177 within 0.01 and three
protected JPEG disagreements). See the recipe for the measured scope and
remaining language/LM/LoRA and JPEG validation gaps.

[`executor.rs`](native/src/executor.rs) runs the vision tower once per request, then the
request's prefix up to its last multiple of 64 tokens once, each question's text up to the
last multiple of 64 before its first option once, and each option's block with the
Expand Down
36 changes: 35 additions & 1 deletion src/models/omnijev/native/src/executor.rs
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,23 @@ use crate::processing::{IMAGE_TOKEN, PreparedRequest, Processor, Row};
#[path = "../../../../../tests/omnijev/window_graph.rs"]
mod window_graph_tests;

/// Vision arithmetic only; language weights/readout and finishing are unchanged.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum VisionNumerics {
Legacy,
Reference,
}
impl std::str::FromStr for VisionNumerics {
type Err = anyhow::Error;
fn from_str(value: &str) -> Result<Self> {
match value {
"legacy" => Ok(Self::Legacy),
"reference" => Ok(Self::Reference),
_ => anyhow::bail!("OMNIJEV_VISION_NUMERICS must be legacy or reference"),
}
}
}

pub struct Executor {
vision: VisionModel,
language: Model,
Expand Down Expand Up @@ -100,6 +117,20 @@ impl Executor {
/// Check the export at `dir`, then load the processor, the vision tower, the
/// language model and the heads.
pub fn load(dir: &Path, library: &Path) -> Result<(Self, Processor)> {
let numerics = std::env::var("OMNIJEV_VISION_NUMERICS")
.or_else(|error| match error {
std::env::VarError::NotPresent => Ok("legacy".to_owned()),
error => Err(error),
})?
.parse()?;
Self::load_with_vision_numerics(dir, library, numerics)
}
/// An explicit selection for callers that do not use worker environment settings.
pub fn load_with_vision_numerics(
dir: &Path,
library: &Path,
numerics: VisionNumerics,
) -> Result<(Self, Processor)> {
let export = export::load(dir)?;
let processor = Processor::load(dir)?;
let language = Model::load(dir, library)?;
Expand All @@ -108,7 +139,10 @@ impl Executor {
&& language.cfg.hidden == heads::HIDDEN,
"export config does not match OmniJev-4B"
);
let vision = VisionModel::load(dir, library)?;
let mut vision = VisionModel::load(dir, library)?;
if numerics == VisionNumerics::Reference {
vision.enable_omnijev_numerics()?;
}
let heads = Heads::load(&dir.join("heads.safetensors"))?;
Ok((
Self {
Expand Down
13 changes: 11 additions & 2 deletions src/models/qwen3_5/native/src/cuda.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ use std::sync::OnceLock;
use anyhow::{Context, Result, bail, ensure};

mod reference;
pub use reference::Reference;
pub use reference::{OmniJevVision, Reference};

/// `CS1_ABI_VERSION` in ops.h.
const ABI_VERSION: u32 = 9;
Expand Down Expand Up @@ -258,6 +258,7 @@ macro_rules! api {
_lib: libloading::Library,
vision_v2: Option<VisionV2>,
reference: Option<Reference>,
omnijev_vision: Option<OmniJevVision>,
$(pub $name: unsafe extern "C" fn($($ty),*) $(-> $ret)?,)*
}

Expand All @@ -275,6 +276,13 @@ macro_rules! api {
pub fn vision_v2(&self) -> Result<&VisionV2> {
self.vision_v2.as_ref().context("27B vision needs additive v2 CUDA symbols; rebuild libqwen3_5_cuda.so")
}
pub fn omnijev_vision(&self) -> Result<&OmniJevVision> {
let vision = self.omnijev_vision.as_ref().context("OmniJev-4B requires optional vision CUDA symbols; rebuild libqwen3_5_cuda.so")?;
let reference = self.language_reference()?;
// SAFETY: the capability query does no device work.
ensure!(unsafe { (reference.available)() } == 1, "OmniJev-4B vision requires a cuDNN 9 build; set CUDNN_INCLUDE_DIR and CUDNN_LIB_DIR");
Ok(vision)
}
fn resolve(lib: libloading::Library) -> Result<Self> {
$(
// SAFETY: the signature is the one ops.h declares for this symbol.
Expand All @@ -288,7 +296,8 @@ macro_rules! api {
)*
let vision_v2=VisionV2::resolve(&lib);
let reference=Reference::resolve(&lib);
Ok(Self { _lib: lib, vision_v2, reference, $($name,)* })
let omnijev_vision=OmniJevVision::resolve(&lib);
Ok(Self { _lib: lib, vision_v2, reference, omnijev_vision, $($name,)* })
}
}
};
Expand Down
54 changes: 54 additions & 0 deletions src/models/qwen3_5/native/src/cuda/reference.rs
Original file line number Diff line number Diff line change
Expand Up @@ -137,3 +137,57 @@ impl Reference {
}
}
}

/// Optional, separately selected unadapted OmniJev-4B vision operators.
/// Old ABI-9 libraries remain loadable; requesting this group requires a rebuild.
pub struct OmniJevVision {
pub patch_create: unsafe extern "C" fn(c_int, c_int, Stream) -> *mut c_void,
pub norm: unsafe extern "C" fn(
*const c_void,
*const c_void,
*const c_void,
*mut c_void,
c_int,
c_int,
Stream,
) -> c_int,
pub rope: unsafe extern "C" fn(
*const c_void,
*const f32,
*const f32,
*mut c_void,
*mut c_void,
c_int,
c_int,
c_int,
Stream,
) -> c_int,
pub attention_workspace_floats: unsafe extern "C" fn(c_int, c_int) -> usize,
pub attention: unsafe extern "C" fn(
*const c_void,
*const c_void,
*const c_void,
*mut c_void,
c_int,
c_int,
c_int,
*mut c_void,
Stream,
) -> c_int,
}
impl OmniJevVision {
pub(super) fn resolve(lib: &libloading::Library) -> Option<Self> {
// SAFETY: these optional signatures match ops.h. Resolve the complete group.
unsafe {
Some(Self {
patch_create: *lib.get(b"cs1_reference_patch_create_v2\0").ok()?,
norm: *lib.get(b"cs1_omnijev_vision_norm\0").ok()?,
rope: *lib.get(b"cs1_omnijev_vision_rope\0").ok()?,
attention_workspace_floats: *lib
.get(b"cs1_omnijev_vision_attention_workspace_floats\0")
.ok()?,
attention: *lib.get(b"cs1_omnijev_vision_attention\0").ok()?,
})
}
}
}
Loading
Loading