Skip to content

cli.md

Latest commit

 

History

History
175 lines (136 loc) · 7.76 KB

cli.md

File metadata and controls

175 lines (136 loc) · 7.76 KB
Raw

CLI reference

ft <command> [args]
Command Purpose
ft serve Start the API server (OpenAI /v1/*, Anthropic /v1/messages, Responses)
ft shell Chat with a server in the terminal
ft ctl Query and manage a running server over HTTP
ft launch Configure and launch a coding agent against a server
ft checkpoint Convert an HF checkpoint to the FTW fast-load format
ft bench bw Benchmark CPU vs PCIe bandwidth to calibrate the MoE backend

ft --version prints the installed version (torch-free; nightly wheels carry a +g<sha> build stamp, tagged releases a bare version). Every command supports --help.

ft serve

ft serve --model <path-or-hf-id> [options]

--model is the only required flag — dtype, attention backend, MoE backend, MoE cache size, KV capacity, CUDA-graph sizes and the tool-call/reasoning parsers all resolve automatically from the checkpoint and the GPU.

Model

Flag Default Meaning
--model-path, --model required Local dir, HF repo id, or an FTW dir (auto-detected)
--served-model-name basename of --model Model id reported by /v1/models

Server & runtime

Flag Default Meaning
--host 127.0.0.1 Bind address
--port 1919 Bind port
--gpu GPU 0 GPU to run on: a UUID from nvidia-smi -L or an nvidia-smi index; see below
--max-running-requests 4 Max concurrently running requests
--max-output-tokens 32768 Default output budget for requests that omit one
--max-seq-len-override from checkpoint Max sequence length
--max-prefill-length 8192 Chunked-prefill chunk size in tokens
--cuda-graph-max-bs, --graph = max running requests Max batch size captured as CUDA graphs
--decode-log-interval 40 Scheduler status line every N decode steps

Choosing a GPU

For example, a machine with an RTX 5090 and an RTX 3060 Ti:

$ nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 3060 Ti (UUID: GPU-2f3a9b1c-8d7e-4a05-b6c1-0e5f9a3d7b42)
GPU 1: NVIDIA GeForce RTX 5090 (UUID: GPU-9e8d7c6b-5a49-4f13-8207-c1b0a4e6d3f5)
ft serve --model ... --gpu 1             # by nvidia-smi index -- the 5090
ft serve --model ... --gpu GPU-9e8d7c6b  # the same card by UUID (a unique prefix is enough)

KV cache & memory

Flag Default Meaning
--memory-ratio 0.9 Fraction of free VRAM the engine may use (weights + MoE cache + KV)
--num-pages / --num-tokens auto KV capacity override in pages / tokens (mutually exclusive; auto sizes from VRAM left after weights and MoE cache)
--page-size 1 KV page size; DSV4 forces 128, the TRTLLM backend needs 16/32/64, SWA models require 1
--cache-type radix radix (prefix reuse; SWA/GDN-aware variants picked automatically) or naive
--attention-backend, --attn auto trtllm/fi/fa/triton/dsv4_sparse/dsa; prefill,decode pair allowed; auto picks per model + GPU

MoE offload

See models.md for what each backend does.

Flag Default Meaning
--moe-backend auto fused/offload/cpu/hybrid; auto → offload, or hybrid with a ft bench bw profile
--moe-cache-size / --moe-cache-rate / --moe-cache-auto auto GPU expert-cache size as slots / fraction of all experts / sized from free VRAM (mutually exclusive; auto is enabled by default for offload-family backends)
--kv-reserve-tokens 8192 KV token floor reserved before --moe-cache-auto fills experts
--moe-cpu-threads physical cores CPU worker threads for the cpu/hybrid executor
--moe-cpu-layers all on GPU With offload: which MoE layers decode on CPU (3,7,11, a count, or a fraction)
--moe-hybrid-max-fetch auto With hybrid: max experts fetched over PCIe per layer per step; rest computed on CPU
--moe-prefill-hit-d2d off Prefill: copy cache-hit experts device-side, stream only misses (CUDA >= 13)
--disable-moe-prefill-overlap overlap on Disable the two-buffer prefill copy overlap

API behaviour

Flag Default Meaning
--sampling-defaults model Fill unspecified sampling params from the checkpoint's generation_config.json (none = framework defaults)
--tool-call-parser auto Tool-call format; auto-inferred from the model family
--reasoning-parser auto Splits chain-of-thought into reasoning_content; auto-inferred; off disables
--enable-cache-report off Report prefix-cache hits in each response's usage block

ft shell

ft shell                                    # attach to a running server
ft shell --model ~/models/Qwen3.6-35B-A3B   # serve + chat in one process
  • Attach mode talks to --server URL (default http://127.0.0.1:1919)
  • /help inside the shell lists the commands (/think, /cache, /reset).

ft ctl

ft ctl [--base-url http://127.0.0.1:1919] [--timeout 10] [--json] <subcommand>
Subcommand Endpoint Purpose
health GET /health Server status, model, load progress
stats GET /v1/stats Throughput, latency, VRAM, pool occupancy
generate [prompt] [--max-tokens N] [--ignore-eos] POST /generate Raw completion smoke test (no chat template)
cache GET /v1/cache/status Cache pool table
cache --moe N | --kv N | --mamba N | --swa N [--wait 300] POST /v1/cache/rebuild Live pool resizing without a restart (k/m suffixes; --kv/--swa in tokens)
requests [--since N] [--limit N] GET /v1/requests Recent request ring

ft launch

ft launch {claude,codex,dsh,hermes,openclaw,opencode} [options] [-- <agent args>]

Discovers the served model via /v1/models, writes the agent's provider config, installs the agent CLI if missing, then launches it. Cloud API keys (ANTHROPIC_API_KEY, OPENAI_API_KEY, …) are cleared from the child environment so the agent cannot silently fall back to a paid endpoint.

Flag Meaning
--server URL Server to point the agent at (default http://127.0.0.1:1919)
--dry-run Print the planned config changes and command, touch nothing
-y, --yes Approve install/config prompts
--config Configure without launching
--install-only Just install the agent CLI (needs no server)
--force-reinstall Re-run the agent installer
-- <args> Forwarded verbatim to the agent

ft checkpoint

ft checkpoint --model <hf_dir> --out <ftw_dir> [--dtype bfloat16] [--moe-backend offload] [--shard-gib 8] [--gpu <uuid-or-index>]

Converts an HF safetensors checkpoint to FTW, FreeToken's self-contained fast-load format; point ft serve --model at the output dir. --moe-backend offload (default) packs experts into offload banks; --moe-backend triton keeps them dense for resident serving. See the FTW caveats in models.md.

ft bench bw

ft bench bw                       # once per GPU
ft bench bw --dtype nvfp4,bf16    # only the formats you serve
ft bench bw --gpu 1               # a specific GPU (UUID or nvidia-smi index, as for ft serve)

Measures host-RAM vs PCIe bandwidth with the real cpu/offload MoE kernels and writes a profile that ft serve --moe-backend auto and --moe-hybrid-max-fetch -1 then read.

  • One profile per GPU, at ~/.cache/freetoken/benchbw/<gpu-uuid>.json.
  • Keyed on expert format + GPU, so a profile from other hardware is ignored rather than misapplied. An older single benchbw.json still counts if its GPU name matches.
  • What to measure: --dtype, --model, --formats, --isa.
  • --threshold (default 2.0) sets the call: recommend hybrid when CPU bandwidth beats PCIe by that factor.