GLM-5.2 (4-bit) on Modal

Serves lowbitcoffee/GLM-5.2-W4A16 (744B MoE, ~40B active, 388GB INT4 weights) via vLLM 0.25.1 on 4x H200, tensor-parallel 4, 64K context.

Prerequisites

pip install modal openai
modal setup
modal secret create huggingface-secret HF_TOKEN=hf_xxxxxxxx

Steps

1. Download weights into a Modal Volume (CPU only, no GPU cost):

   modal run download_model.py
   

   Idempotent, safe to re-run if it's interrupted.

2. Deploy the vLLM server (4x H200):

   modal deploy serve.py
   

   Copy the printed endpoint URL (https://<workspace>--glm-5-2-serve-serve.modal.run).

3. Chat, either:

   • Web UI: open index.html in a browser (landing page), then "Open chat" for chat.html. No server or build step needed; vLLM's CORS allows the browser to call the endpoint directly. Endpoint URL is set in BASE_URL at the top of the <script> block in chat.html.
   • Terminal: python chat.py <endpoint-url>
   • Gradio (hosted on Modal): put the endpoint URL in GLM_VLLM_URL (env var, or edit VLLM_BASE_URL in gradio_ui.py) then modal deploy gradio_ui.py.

Web UI

File	What it is
index.html	Landing page. Cream/coral/dark-navy system from Design.md.
chat.html	Chat client. Streams content plus reasoning_content (the reasoning shows in a collapsible box), markdown rendering, stop button, /health status dot polled every 60s.

Cost

Item	Cost
4x H200	$18.16/hr
Budget	$200 total ~ 10-11 hours of serving
First cold boot	~29 min (weight load plus DeepGEMM kernel compile)
Cold boot, warm compile cache	~13 min
Volume storage	free under 1 TiB

Two volumes persist between runs and cost nothing at this size: glm-5-2-w4a16 (weights, mounted read-only at /models) and glm-5-2-vllm-cache (vLLM compile cache at /root/.cache/vllm), which is what cuts the second boot down.

No min_containers/keep-warm is set, and max_containers=1, so the container scales to zero after idle and you only pay while actively serving. startup_timeout is 55 min because the first boot is slow.

Traps hit already

• Image must be nvidia/cuda:12.8.1-devel-* with CUDA_HOME set. The runtime image lacks nvcc, so DeepGEMM fails to build its kernels.
• Do not pass --kv-cache-dtype fp8. It crashes this model.
• The first request after idle triggers the whole cold boot, so it looks like a hang. The status dot in chat.html stays red until /health answers.

Important

Stop the app when done to avoid burning idle GPU budget:

modal app stop glm-5-2-serve