Skip to content

omnijev: add opt-in BF16 reference vision arithmetic - #209

Draft
psv666 wants to merge 1 commit into
InferMatrixLab:mainfrom
psv666:codex/omnijev114-vision-20261120
Draft

psv666 wants to merge 1 commit into
InferMatrixLab:mainfrom
psv666:codex/omnijev114-vision-20261120

Conversation

@psv666

@psv666 psv666 commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator

Purpose

OmniJev's legacy BF16 vision arithmetic drifts from the pinned official tower through normalization, attention and GEMM reduction boundaries. Add an explicit OMNIJEV_VISION_NUMERICS=reference path that matches all 40 observed visual boundaries on the original protected PNG case, while retaining legacy as the default. This is the bounded vision stage of #114; the original full 280-answer precision gate still fails, so this PR stays draft.

Required human contributor answers

The contributor confirmed these answers and the draft-PR decision during preparation; the public scope is RFC #114. Private preparation records are retained with the contributor.

  1. Outcome: Advance reliable native model support to a reproducible state with real-weight validation and complete numerical evidence.
  2. Highest-leverage variable: Trustworthy support coverage and numerical consistency, before performance optimization.
  3. How this moves it: Reuse existing implementations, resolve reference/contract/numerical gaps, and make the smallest evidence-backed change rather than add another worker or unsupported model claims.
  4. User and actual need: Model/application/agent developers using System1-Omni need stable interfaces, reliable output and reproducible deployment on the target hardware.

Is a PR needed? Yes: the contributor authorized necessary implementation, L20 validation, self-review and a reviewable draft PR for this stage.

Observed failure: Original video.png/noul_region, unchanged question/region: official unmerged BF16 0.5443, legacy native 0.4997, drift 0.0446 and a protected true→false decision. The strict gate actually exits 1. Raw before/after results and failing intermediate stage regression are linked below.

Summary

OmniJev Executor (legacy default)
  reference opt-in -> shared VisionModel, unadapted 4B only
    cuDNN BF16 patch -> Welford norm -> framework GEMMs (1 MiB)
    FP32 rotary angles -> head-64 reference attention -> BF16 merger
  existing merged language/readout -> existing heads/calibration

Reuse the existing reference kernels/GEMM handle; the 1 MiB heuristic budget matches the actual Torch addmm boundary. A 32 MiB budget selects a different split-K in the merger: the intermediate exact-stage regression fails there, and the single budget control makes it exact. No change to gemm.cu, language/LoRA execution, weights, calibration, corpus or gates.

Add separately resolved optional ABI-9 symbols, retaining old-library loading and JEMM's 1152/head-72 guards. The original patch constructor still fixes 1152. Reject adapted/incorrect layout, bounded-cache accounting and late profile selection. Keep the existing trace events; add opt-in selected-block operator tracing and root tests/oracle generator. This builds on the native worker and reuse implementation from merged #129/#134, preserving their author attribution.

Test Plan

Base / head: f1e674f2d324f5bb4e1ba2162551d0b633f20be0 / 6410e7634fd5e077000810ae07dc2f251b4f044d. 17 files; 730 changed authored-code lines, 859 total. No generated run dumps or matrices committed.

Pinned artifacts: official code 14dbec4f71e194852c8d7b88ab36ef639493f400; checkpoint ffe5f436eaf22e20e2f041f8e74e121fd057a6cb; Qwen3.5-4B 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a; export manifest SHA256 2bab1a7d6ac96859fa07af932f1bafb448acd4abe21a36d9b204e5e2a56c6dfe. Original BF16 unmerged reference, 248079 vocabulary, 602112 pixels, calibration and all original 28 requests/280 answers remain fixed.

Hardware/runtime: one L20 GPU0 (sm_89), driver 580.178.04; Rust 1.100, nvcc 12.8, native cuBLASLt 120803 and cuDNN 92400. Actual native maps also load cuDNN's bundled cuBLASLt13. Official Python 3.10.12, Torch 2.14.0+cu130, torchvision 0.29.0+cu130, Transformers 5.17.0, PEFT 0.19.1, Pillow 12.3.0, no FLA. Full lock/library hashes are included. These stacks are not claimed library-identical. Cooperative GPU lock and occupancy checks were used; no performance campaign.

Test Result

  • Exact-head format, workspace Clippy (--locked --all-targets --features omni-laya/serve -- -D warnings), ordinary workspace tests, workspace release build, test discovery and strict MkDocs pass. Ordinary tests: 427 passed, 122 ignored; skipped device tests are not counted as executed passes. CUDA backend/checker tests: 27 + 68 pass; the existing Laya ABI-declaration warning remains.
  • Named GPU checks all pass: five Torch/operator/legacy compatibility tests, five existing head-72 vision tests, fixed-GEMM row invariance, exact real-weight stage/Graph regression, and two OmniJev worker order/reuse tests. The stage test includes actual capture/replay, changed pixels at the same grid, resident readback and error recovery. Existing 4B/JEMM operator bits match the pre-change library. The legacy minimum preserves every observed value/visual byte.
  • Six first-ready/direct/frontend parity and malformed-input recovery checks pass. The protected minimum now returns 0.5525 against official 0.5443 (drift 0.0082); all 40 visual boundaries, including all 24 blocks and merger, are bitwise exact on this case.
  • Original full corpus is freshly replayed with final native eager/reuse and FP32 head capture. All 28 eager/reuse responses agree apart from timing; image hashes, questions, fields and input usage match. Official full captures are reused read-only from the completed documented-stack prerequisite, not claimed freshly re-executed here.
Population Answers Within 0.01 Max returned drift Protected flips
demo PNG 84 68 0.0355 0
gate PNG 56 52 0.0151 0
demo JPEG 84 29 0.1758 1
gate JPEG 56 28 0.1456 2
Total 280 177 0.1758 3

Unrounded: 177/280 within 0.01, max 0.17586451768875122, same three protected failures; all raw results round back to production. The original strict full gate exits 1. The previous documented baseline (113/280 within 0.01, max 0.3066, four protected flips) remains separately retained; the JPEG failure set changes and does not uniformly improve.

Remaining protected cases: demo-jpg/board.jpg/q8 margin 0.0397 (check first→safe); gate-jpg/arcade.jpg/long_start margin 0.0935 (third→second); gate-jpg/web.jpg/one_token margin 0.0746 (click OK→click Close). All 140 PNG decisions agree, but 20 PNG answers still exceed 0.01. In the original 12-question demo-png/video.png/q2 Score request, confidence remains 0.5613460540771484 versus 0.5786595344543457 (raw drift 0.017313480377197266) with the same harmless decision. Its full context/raw/control pointers are preserved for the next stage. Remaining errors are not attributed wholesale to JPEG or LoRA; JPEG input factors and language/selected-readout/merged-vs-unmerged residuals need separate controls.

Demo / evidence

Numerical evidence archive, 485264 bytes, 197 inventoried files. SHA256 85e39a9d95762dec3cc34447ffb40d2e12562a0ae8d416810138ecc24fa786da. Contains the protocol, all original gate rows/raw numeric results, failures, source helpers, dependency lock, actual runtime/library hashes and reproduction commands. Screenshot bytes, embedded image requests, weights, credentials and private preparation discussions are omitted; complete replay inputs/BF16 tensors remain in the contributor's private archive. Maintained test commands and profile scope are documented in the recipe and root tests README.

Video: N/A, numerical backend change. Performance/startup/memory improvements: unmeasured. Other GPUs/Metal, full 27B/Cua checkpoints and whole unmerged-language parity: unverified. No ready/merge request or issue-closing claim.

Merge Danger

Door: two-way. Revert the change or select legacy; exports and request/response schema are unchanged.

Blast Radius: Vision. New arithmetic is selected explicitly for unadapted OmniJev4B; shared optional bindings and the patch-constructor wrapper have device compatibility coverage. This does not qualify real 27B models or enable the new profile by default. Whole #114 acceptance remains blocked by the disclosed original-gate failures.

Self-review

Agent self-review/precheck completed against the full pinned diff; contributor/maintainer review remains separate. Kept draft. Human checklist is intentionally left for the contributor:

  • I have reviewed the full diff and addressed the issues I found.
  • I have checked architecture and scope.
  • I have checked the reported validation and remaining gaps.
  • I have checked the description, documentation and numerical claims.

Align patch, LayerNorm, rotary, head-64 attention and vision GEMM boundaries with the pinned BF16 reference. Keep legacy/JEMM behavior and optional ABI compatibility. The original 280-answer whole-model precision gate still fails; this is the bounded vision stage for InferMatrixLab#114.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant