Inference code for DeepSeek models

First convert huggingface model weight files to the format of this project.

export EXPERTS=256
export MP=4
export CONFIG=config.json
export HF_CKPT_PATH=/root/tingqli/DeepSeek-V4-Flash/
export SAVE_PATH=/root/tingqli/DeepSeek-V4-Flash/weights
export FILE=/root/tingqli/DeepSeek-V4-Flash/inference/input.txt
python convert.py --hf-ckpt-path ${HF_CKPT_PATH} --save-path ${SAVE_PATH} --n-experts ${EXPERTS} --model-parallel ${MP}

Then chat with DeepSeek model at will!

torchrun --nproc-per-node ${MP} generate.py --ckpt-path ${SAVE_PATH} --config ${CONFIG} --interactive

Or batch inference from file.

torchrun --nproc-per-node ${MP} generate.py --ckpt-path ${SAVE_PATH} --config ${CONFIG} --input-file ${FILE} --max-new-tokens 400

Or multi nodes inference.

torchrun --nnodes ${NODES} --nproc-per-node $((MP / NODES)) --node-rank $RANK --master-addr $ADDR generate.py --ckpt-path ${SAVE_PATH} --config ${CONFIG} --input-file ${FILE}

If you want to use fp8, just remove "expert_dtype": "fp4" in config.json and specify --expert-dtype fp8 in convert.py.