Context
DetoServe currently supports vLLM, Triton, and Dynamo/LLM-D runtimes. TensorRT-LLM is widely used for optimized NVIDIA GPU inference.
Task
- Add a SkyPilot service template under
services/tensorrt-llm.yaml
- Add TensorRT-LLM as a runtime option in the Function Manager
- Add it to the frontend dropdown
- Add example deployment manifest
Reference
services/vllm-llama70b.yaml — existing template to followcontrol-plane/function-manager/main.go — runtime validationfrontend/src/App.jsx — runtime dropdown
Context
DetoServe currently supports vLLM, Triton, and Dynamo/LLM-D runtimes. TensorRT-LLM is widely used for optimized NVIDIA GPU inference.
Task
services/tensorrt-llm.yamlReference
services/vllm-llama70b.yaml— existing template to followcontrol-plane/function-manager/main.go— runtime validationfrontend/src/App.jsx— runtime dropdown