Skip to content

cli.md

Latest commit

 

History

History
294 lines (244 loc) · 14.5 KB

cli.md

File metadata and controls

294 lines (244 loc) · 14.5 KB
Raw

NInfer CLI

build/apps/ninfer runs one request against one registered .ninfer artifact. Build NInfer and download an artifact using the project README before following this guide.

The examples use Qwen3.8-27B NVFP4 with FP8 KV storage.

Text input

./build/apps/ninfer models/qwen3_8_27b_nvfp4.ninfer \
  --prompt "Summarize the difference between prefill and decode." \
  --max-context 32768 \
  --max-new 8192 \
  --kv-dtype fp8 \
  --spec mtp --draft-tokens 3 \
  --lm-head-draft

Exactly one of --prompt and --messages is required. The CLI normally omits --kv-capacity, so the shared Main Text KV pool follows the example's 32,768-token --max-context.

Answer content is streamed to stdout. Human-readable startup milestones and runtime errors are written to stderr without service timestamps. Reasoning and the CLI result report (timings, throughput, GPU memory, token IDs when requested, and speculative-decoding statistics) also use stderr as unprefixed product output, so stdout can be redirected independently. On a terminal, weight materialization is one transient progress line followed by a compact Engine-ready summary. Redirected stderr contains persistent readable progress for long loads and no carriage returns or ANSI escapes. --log-level debug exposes every startup phase. Option and local prompt/message input failures remain direct command diagnostics:

./build/apps/ninfer models/qwen3_8_27b_nvfp4.ninfer \
  --prompt "Return one sentence." \
  --max-context 4096 \
  --max-new 64 \
  --kv-dtype fp8 \
  > answer.txt 2> run.log

Thinking is enabled by default. If the chat template embedded in the loaded artifact exposes reasoning effort, --reasoning-effort low|medium|xhigh selects it; omitting the option uses the template's default. An artifact whose template does not expose effort rejects the option. Add --no-thinking for direct-response prompt rendering; it cannot be combined with --reasoning-effort. --greedy selects exact argmax decoding independently.

--thinking-budget N places a positive upper bound on accepted model-origin tokens while the new-turn Qwen thinking block remains open. If the model has not emitted </think> at that exact boundary, Engine appends Qwen's canonical early-close guidance and </think> to the same resident sequence without sampling, publishes the guidance through the reasoning stream, then resumes ordinary generation from the updated context. A natural thinking close, stop condition, cancellation, or total output/context limit at the boundary takes priority and suppresses this insertion. The option cannot be combined with --no-thinking, but it can be combined with --reasoning-effort.

--max-new counts every committed generated token, including internally inserted control tokens. When the effective output capacity extends beyond the thinking budget, it must have room for the complete tokenizer-derived control suffix plus one post-close model token; an undersized request is rejected rather than truncating the suffix. Normal output sends the inserted guidance to stderr as reasoning. --print-token-ids includes the inserted IDs, while --raw-output preserves the raw control representation.

For example, this allows at most 512 model-origin thinking tokens while retaining enough total output capacity for the inserted suffix and the answer:

./build/apps/ninfer models/qwen3_8_27b_nvfp4.ninfer \
  --prompt "Explain speculative decoding, then give a concise conclusion." \
  --max-context 4096 \
  --max-new 1024 \
  --thinking-budget 512 \
  --kv-dtype fp8 \
  --spec mtp --draft-tokens 3 \
  --lm-head-draft

Chat templates

By default, the Engine validates and uses frontend/chat_template.jinja embedded in the loaded artifact. Pass --chat-template-file PATH to replace that prompt renderer for this Engine startup:

./build/apps/ninfer models/qwen3_6_27b.ninfer \
  --chat-template-file /path/to/chat_template.jinja \
  --prompt "Return one sentence."

The file is read and compiled before prompt preparation. An unreadable, empty, malformed, or unsupported template stops startup; NInfer never silently falls back to the artifact template. The Qwen-Sharp chat template is a supported self-contained file override.

Custom templates use the bundled Minja LLM chat-template renderer. It supports the Qwen template features used by Qwen-Sharp, including macros, mutable namespaces, whitespace control, loops, filters, JSON conversion, and chained string operations. It does not support file includes, imports, inheritance, or arbitrary Python execution, so the supplied source must be self-contained. NInfer provides the standard messages, tools, add_generation_prompt, enable_thinking, add_vision_id, preserve_thinking, and chat_template_kwargs.preserve_thinking values. Text content is a string; multimodal content is an ordered array of text, image, and video parts.

--no-thinking remains available to a custom template through enable_thinking, but arbitrary Jinja does not declare reasoning-effort presets, so --reasoning-effort is rejected. Custom templates also do not provide the registered templates' response-replay checkpoint; prompts remain correct, but multi-turn prefix reuse may be less efficient.

Startup memory profile

GPU residency is frozen when the Engine starts:

  • no --spec omits MTP/DFlash weights and state and the optimized proposal head;
  • --spec mtp loads only MTP, while --spec dflash loads only the 35B-A3B DFlash backend;
  • a speculative backend with the full proposal head omits the optimized proposal head;
  • Vision is disabled by default, omitting its weights and Vision-specific unified-workspace extent;
  • --vision loads the weights, expands the one Program workspace for Vision encode/handoff, and enables image/video input.
  • the one-request CLI uses root-only context mode, so it does not reserve an extra Device checkpoint StateImage or capture a continuation that no later request could consume.

The complete .ninfer inventory is still validated. These choices are not lazy loading: an Engine started without Vision rejects media and cannot enable Vision later. DFlash and Vision may be enabled together; DFlash applies to generated-text decode after multimodal prefill and does not accelerate Vision encode. The default speculative and Vision settings produce the smallest resident profile.

Structured messages

--messages accepts either a non-empty JSON message array or an object containing messages and an optional tools array.

[
  {
    "role": "system",
    "content": "Answer concisely."
  },
  {
    "role": "user",
    "content": [
      {
        "type": "image",
        "image": "examples/cli/media/visual_chart.png"
      },
      {
        "type": "text",
        "text": "Describe the chart."
      }
    ]
  }
]

Run message files from the repository root when they contain repository-relative media paths:

./build/apps/ninfer models/qwen3_8_27b_nvfp4.ninfer \
  --messages examples/cli/messages/image_chart.json \
  --max-context 8192 \
  --max-new 128 \
  --kv-dtype fp8 \
  --vision \
  --spec mtp --draft-tokens 3 \
  --lm-head-draft

Supported roles are system, developer, user, assistant, and tool. System and developer messages retain their array positions; the Qwen family frontend renders both as system-class ChatML turns rather than moving later instructions to the beginning.

Message content may be a string or an ordered array containing:

Content type Source field Accepted source
text text string
image / image_url image or image_url local path, HTTP(S) URL, or base64 data URI
video / video_url video or video_url local path, HTTP(S) URL, or base64 data URI

image_url and video_url may be strings or objects containing a string url. Assistant history may include reasoning_content and tool_calls; a tool result uses role tool and tool_call_id.

See examples/cli/ for committed text, image, video, mixed-media, thinking, long-decode, and long-context inputs.

Speculative decoding

Speculative decoding is disabled by default. Select MTP with one to five draft positions, or the 35B-A3B DFlash backend with one to fifteen. DFlash may be combined with --vision. --lm-head-draft selects the optimized proposal head and requires a selected backend:

./build/apps/ninfer models/qwen3_6_35b_a3b.ninfer \
  --prompt "Write a short explanation of speculative decoding." \
  --max-context 16384 \
  --max-new 512 \
  --kv-dtype fp8 \
  --spec mtp --draft-tokens 3 \
  --lm-head-draft

For DFlash:

./build/apps/ninfer models/qwen3_6_35b_a3b.ninfer \
  --prompt "Write a short explanation of speculative decoding." \
  --max-context 16384 --max-new 512 \
  --kv-dtype fp8 \
  --spec dflash --draft-tokens 7 --lm-head-draft

MTP and DFlash cannot be enabled together. The published performance results use MTP with three draft tokens and DFlash with seven draft tokens (block length eight), both with the optimized proposal head. DFlash accepts one to fifteen draft tokens; seven forms the measured block length eight, while fifteen uses the full native block.

Common options

The table lists executable defaults. The examples above select FP8 KV and MTP3.

Option Meaning Default
--max-context N per-sequence logical context ceiling 2048
--kv-capacity N|auto explicit shared Main Text KV capacity, or maximize it from remaining GPU memory; omitted means --max-context 2048
--prefill-chunk N positive text-prefill chunk, in multiples of 128 1024
--max-new N requested output-token limit 128
--device N CUDA device index 0
--kv-dtype bf16|int8|fp8|nvfp4|k8v4 KV-cache storage bf16
--spec mtp|dflash speculative backend off
--draft-tokens N MTP 1..5; DFlash 1..15 unset
--lm-head-draft optimized proposal head off
--vision enable image/video input and load Vision GPU allocations off
--chat-template-file PATH self-contained Jinja prompt-template override loaded at startup artifact template
--no-cuda-graph disable CUDA Graph decode graphs on
--no-thinking disable thinking in prompt rendering thinking on
--thinking-budget N positive model-origin thinking-token cap; omitted means unlimited unset
--reasoning-effort low|medium|xhigh select an effort exposed by the loaded chat template template default
--greedy exact argmax decoding off
--temperature F sampling temperature override registered model/mode default
--top-p F nucleus-threshold override registered model/mode default
--top-k N top-k-threshold override (0..20; zero selects the top-20 cap) registered model/mode default
--min-p F min-p-threshold override registered model/mode default
--presence-penalty F presence-penalty override registered model/mode default
--frequency-penalty F frequency-penalty override registered model/mode default (0)
--seed N sampling seed 0

When a sampling flag is omitted, Engine selects the official general-task preset registered for the loaded model and the rendered prompt mode. The current presets are:

Model Prompt mode Temperature Top-p Top-k Min-p Presence penalty
Qwen3.6-27B thinking 1.0 0.95 20 0 0
Qwen3.6-27B non-thinking 0.7 0.80 20 0 1.5
Qwen3.8-27B thinking 1.0 0.95 20 0 0
Qwen3.8-27B non-thinking 0.7 0.80 20 0 1.5
Qwen3.6-35B-A3B thinking 1.0 0.95 20 0 1.5
Qwen3.6-35B-A3B non-thinking 0.7 0.80 20 0 1.5

Frequency penalty is 0 in every registered preset. Task-specific profiles such as Qwen's precise-coding profile use explicit sampling overrides.

Repeat --stop-token-id, --stop, or --reasoning-stop to add stop conditions. Use --raw-output to expose the frontend's raw output stream and --print-token-ids to include generated token IDs in diagnostics.

Run ./build/apps/ninfer --help for the exact option contract.

Context and memory

The registered model IDs have a native context limit of 262,144 tokens. The practical allocation on one RTX 5090 depends on the selected artifact, media workload, output budget, and KV-cache type. Artifact identity selects the weight profile; --kv-dtype selects runtime KV storage. The prepared prompt must fit --max-context; generation stops at the remaining context capacity when necessary. --kv-capacity N controls the shared physical Main Text KV pool independently and is rounded up to the 64-token page size. --kv-capacity auto loads the selected weights, measures the remaining GPU memory, and directly chooses the largest legal page capacity for the complete enabled runtime layout. This includes the selected speculative backend, fixed sequence state, unified workspace, and CUDA Graph allowance, while leaving the default 1 GiB automatic headroom unallocated. It does not probe allocations or resize the pool at request time. The single-request CLI normally leaves the option omitted so it follows --max-context; the distinction matters primarily to a concurrent Engine or server.

At Engine startup NInfer reserves model weights, persistent sequence state, one phase-reused Program workspace, and a separate CUDA Graph driver allowance. With Vision enabled, that one workspace contains a general execution prefix and a fixed item-output handoff region. Vision encode may reuse the full backing before producing the output; Text/MTP/decode work remains inside the general prefix while the handoff is live. The capacity is therefore the maximum legal simultaneous extent, not the sum of Text, Vision scratch, and Vision output allocations. Text prefill uses min(--prefill-chunk,--max-context); Vision keeps the existing 32,768-token aggregate prompt budget but plans Device execution for the registered 16,384-token maximum single item. Requests perform no project-owned device allocation or growth. Context-cache capacity controls are intentionally absent from this one-request interface; the persistent Engine and server routes own cross-request reuse and optional Host backing.

All weight, sequence, workspace, and graph allocations are released when the Engine is destroyed.