[Feature] Add a Rust frontend for omni-modal, prefill-only Jev models

[hsliuustc0106]

Build a minimal, model-independent Rust frontend for Jev-style models that accept text, image, audio, video, and mixed-modality inputs and return typed decisions.

The first milestone covers the multimodal serving interface and request transport. Laya is the first backend for validating text inference. Actual modality support depends on the selected backend.

The initial implementation should:

• Use Axum, Tokio, and Reqwest in a single Rust binary.
• Expose POST /v1/systemone using the model, state, and questions envelope, with choice, score, and noul outputs. Jev API reference
• Forward multimodal request bodies unchanged, including media references, inline data, and mixed-input structures. Leave media loading, preprocessing, tokenization, and inference to the backend.
• Forward authorization when supplied and preserve backend response status, content type, and body.
• Expose GET /health backed by the model worker’s health endpoint.
• Reuse one HTTP client with a 60-second timeout and no automatic retries. Return 502 for connection failures and 504 for timeouts.
• Configure the frontend through OMNI_JEV_BIND and OMNI_JEV_BACKEND_URL, defaulting to 127.0.0.1:8080 and http://127.0.0.1:8000.
• Start the frontend and backend separately.

Acceptance criteria:

• Mock-backend tests verify unchanged forwarding of text, image, audio, video, and mixed-input payloads.
• Real Laya text requests cover all three decision types and match direct backend outputs.
• The public interface and configuration contain no Laya-specific assumptions.
• Backend errors, unavailable workers, timeouts, and health checks are tested.
• Formatting, Clippy, Cargo tests, and a CPU smoke test pass.
• Documentation distinguishes multimodal transport support from each backend’s inference capabilities.

Custom scheduling, Rust-side preprocessing, and additional model integrations are follow-up work. vLLM RFC #19038 provides a reference for future prefill scheduling and memory optimizations.

[linear3735]

Progress: reproduced Laya's outputs with plain HF transformers and served them behind the Rust frontend (responses byte-identical direct vs. proxied). On CPU, latency scales predictably with token count (~60 ms short → ~280 ms at 450 tokens); with 8 concurrent requests, short-request p99 rises from ~90 ms to ~1.2 s when long ones are mixed in. Next: open the frontend PR.

[hsliuustc0106]

Progress: reproduced Laya's outputs with plain HF transformers and served them behind the Rust frontend (responses byte-identical direct vs. proxied). On CPU, latency scales predictably with token count (~60 ms short → ~280 ms at 450 tokens); with 8 concurrent requests, short-request p99 rises from ~90 ms to ~1.2 s when long ones are mixed in. Next: open the frontend PR.

what if HF inference on GPU?

[linear3735]

Progress: reproduced Laya's outputs with plain HF transformers and served them behind the Rust frontend (responses byte-identical direct vs. proxied). On CPU, latency scales predictably with token count (~60 ms short → ~280 ms at 450 tokens); with 8 concurrent requests, short-request p99 rises from ~90 ms to ~1.2 s when long ones are mixed in. Next: open the frontend PR.

what if HF inference on GPU?

On a single H800, warm end-to-end latency through Rust for a 48-token request:

• FP32 eager: 10.5 ms
• FP32 + CUDA Graph: 5.8 ms
• Official fast (BF16 + Graph): 4.3 ms

Frontend PR: #2.