Skip to content

modernbert: reject zero attention heads before deriving geometry - #207

Draft
psv666 wants to merge 1 commit into
codex/vela152-main-sync-20261120from
codex/vela152-cpu-sync-20261120
Draft

psv666 wants to merge 1 commit into
codex/vela152-main-sync-20261120from
codex/vela152-cpu-sync-20261120

Conversation

@psv666

@psv666 psv666 commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator

Required human contributor answers

The contributor confirmed this stage on 2026-11-20: synchronize the existing Vela ModernBERT PR with main and validate its CPU structure/loader contracts, preserving the original author's work. The shared contributor intent was confirmed on 2026-11-19 and remains applicable; detailed confirmation is retained in the contributor's task record. Public parent context: #137 and #152.

  1. Outcome: Bring assigned model support to reproducible, reviewable acceptance so supported models work reliably.
  2. Highest-leverage variable: Reliable model coverage and numerical/contract consistency.
  3. How this moves it: Reuse the existing implementation, close contract and evidence gaps, and add only necessary focused corrections before performance work.
  4. User and need: Model, application and agent developers who need stable interfaces, reliable outputs and reproducible deployment.

Is a PR needed? Yes. The contributor authorized necessary small supplements and stage draft PRs while preserving others' commits. A real CPU loader failure was reproduced here.

Bug-fix evidence: On base 90a1beff, the new integration test fails with adapter.rs:90:13: attempt to divide by zero. The same command passes after this guard. Reproduction and verbatim excerpts are below.

Purpose

Reject num_attention_heads: 0 as a field-specific configuration error before deriving hidden_size / num_attention_heads. Without an explicit head_dim, the adapter currently panics before structure validation can reject the input.

This is a 2-file, +21-line stacked supplement, based on codex/vela152-main-sync-20261120 at 90a1beff35e4a8228593ae166073d8c7af9ce46d. That base preserves MrDongsls's #152 head 022c62af and main f1e674f2 as its two parents. Its sole manual merge resolution unions the workspace members in root Cargo.toml; Cargo.lock is auto-merged. Qwen ABI stays 9.

The base carries the original 74-file implementation so this draft's review diff stays limited to the guard and regression. Integration into main requires #152 to land, followed by retargeting and fresh checks. No changes were pushed to the author's branch.

Summary

 derive_head_dim(hidden, heads, config)
+  reject heads == 0 with num_attention_heads error
   derive or parse head_dim
   validate the complete structure

The root integration test covers the 1024-wide family fixture and Vela 0.3B fixture, each with derived and explicit head_dim. Existing golden fixtures, geometry, checkpoint census, precision classes and numerical tolerances are unchanged.

Evidence

Run from the repository root on Linux, with Rust 1.100.0 and no weights or GPU:

cargo test --locked -p omni-modernbert --test structure \
  zero_attention_heads_is_rejected_without_panicking -- --exact --nocapture

Before, base plus only the regression test:

thread 'zero_attention_heads_is_rejected_without_panicking' panicked at
src/models/modernbert/src/adapter.rs:90:13:
attempt to divide by zero
test result: FAILED. 0 passed; 1 failed

After, head a11335aeed99d30ebdef123d673b569eb8a1c299:

test zero_attention_heads_is_rejected_without_panicking ... ok
test result: ok. 1 passed; 0 failed

Test Plan

CPU-only on the L20 host, Rust 1.100.0, private cargo-home/target/Python environments; CUDA_VISIBLE_DEVICES empty. Dependencies came from verified local caches; Cargo checks used --locked --offline. No checkpoint download, forward pass or GPU execution.

  • cargo fmt --all --check
  • cargo clippy --workspace --locked --all-targets --features omni-laya/serve -- -D warnings
  • cargo test --workspace --locked --features omni-laya/serve
  • cargo build --workspace --release --locked --features omni-laya/serve
  • Component test discovery/tests for ModernBERT, Vela, Laya and shared Qwen; Vela cuda-feature compilation/CPU checks.
  • CUDA text contract and root Python tests; Vela Python/comparison-tool checks; benchmark tests; strict MkDocs.

Test Result

Check Personally executed result
Focused components 132 passed, 0 failed, 74 ignored
Full workspace, serialized follow-up 472 passed, 0 failed, 121 ignored
fmt, strict workspace Clippy, release build PASS
Vela Python / comparison tool 157 passed / PASS
Benchmark Python 133 passed; 185 subtests passed
CUDA text contract / root Python checks 0 errors, existing ABI-define warning / 27 passed
Strict MkDocs PASS using a separately copied and hash-verified docs environment

The first full workspace run failed the unchanged JEV-VL test stats_and_prefix_publication_finish_concurrently at its 10-second timeout. It subsequently passed in isolation on exact main (0.69 s) and this candidate (0.62 s); one serialized full-workspace retry passed. The original failure is retained; its cause remains unclassified and the timeout was not changed.

The initial docs attempt used a Python environment without MkDocs; that failure is retained. The final strict build used the existing independent docs environment, without installing or modifying shared packages.

Demo / evidence

The before/after CPU regression above is the evidence for this patch. Original Laya golden plans and the 20-configuration mock launch-trace comparison pass; these checks do not establish device numerical parity. The pinned Vela census (134 encoder + 18 head tensors) and YaRN assertions pass without fixture regeneration.

The Vela cuda-feature command reports 13 ok, but its one device test explicitly early-returns without VELA_NATIVE_CHECKPOINT/VELA_NATIVE_LIBRARY: 12 CPU checks, device execution unrun. L20 is sm_89 while the inherited bundle requires sm_90a; no L20 native Vela result is claimed. Historical H100 evidence belongs to its recorded source.

Raw logs, failures, source bundles, hashes and the component review map remain in the contributor-local/L20 evidence archive; there is no public raw-archive URL. The repository test and excerpts above are independently reproducible. Performance A/B and video are N/A for this invalid-config guard. The complete original #152 implementation still requires its own large-change and hardware review.

Merge Danger

Door: two-way; ordinary revert, no stored-data migration.

Blast Radius: loader.

Only malformed zero-head ModernBERT configurations change from panic to error. Valid structures retain their existing parsing, geometry and execution plans.

Self-review

Agent precheck reviewed the complete 21-line supplement, the caller through Vela read_structure, root test discovery, artifact placement and the unchanged positive-profile gates. No blocking finding was identified for this small guard. The full parent PR, future decoder path/crate integration and target-hardware qualification remain separate. Keep draft for contributor/maintainer review.

  • I have reviewed the full diff and addressed the issues I found.
  • I have checked that the change follows the project's architecture and stays focused on the stated purpose.
  • I have run the checks appropriate to this change and reported commands, results, and anything I could not verify above.
  • I have checked that the PR description, documentation, and any accuracy or performance claims match the implementation and available evidence.

A CPU loader regression test on the synced #152 tree reproduced a division-by-zero panic for num_attention_heads=0 without head_dim. Return a field-specific error before division; cover the 1024 family and Vela 0.3B HF layouts with derived and explicit head dimensions. Preserve all positive-profile geometry and numerical assertions.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant