Skip to content

Add SGLang serving backend support to ModelDeployment#5

Draft
devin-ai-integration[bot] wants to merge 1 commit into
mainfrom
devin/1777682718-sglang-serving-backend
Draft

Add SGLang serving backend support to ModelDeployment#5
devin-ai-integration[bot] wants to merge 1 commit into
mainfrom
devin/1777682718-sglang-serving-backend

Conversation

@devin-ai-integration
Copy link
Copy Markdown
Contributor

Adds SGLang as an alternative serving backend alongside vLLM for ModelConfig.serve().

Changes

modal_training_gym/common/serve_sglang.py (new)

  • build_sglang_serve_app() mirrors build_vllm_serve_app() but launches sglang.launch_server inside an SGLang Docker image
  • Accepts the same volume / GPU / scaling parameters as the vLLM builder
  • Configurable via sglang_image_tag (defaults to lmsysorg/sglang:v0.4.7-cu124) and extra_sglang_args

modal_training_gym/common/deploy.py

  • DeployConfig gains backend: Literal["vllm", "sglang"] (default "vllm" — fully backward-compatible)
  • New fields: extra_sglang_args, sglang_image_tag

modal_training_gym/common/models/base.py

  • ModelConfig.serve() dispatches to the correct builder based on deploy.backend

Usage (Qwen3-4B example)

from modal_training_gym.common.models import Qwen3_4B, DeployConfig

model = Qwen3_4B()
model.deploy = DeployConfig(
    backend="sglang",
    gpu="H100",
    n_gpu=1,
    extra_sglang_args=["--context-length", "8192", "--mem-fraction-static", "0.80"],
)
deployment = model.serve()
print(deployment.generate("Hello!"))

Checklist

  • Example is documented with comments throughout, in a Literate Programming style.
  • Example does not require third-party dependencies to be installed locally
  • Example pins its dependencies
    • Example pins container images to a stable tag, not a dynamic tag like latest
    • Example specifies a python_version for the base image, if it is used
    • Example pins all dependencies to at least minor version, ~=x.y.z or ==x.y
    • Example dependencies with version < 1 are pinned to patch version, ==0.y.z

Outside contributors

N/A — internal change.

Link to Devin session: https://app.devin.ai/sessions/6aac27bf99024337a1d27328aac58062
Requested by: @joyliu-q

- Add serve_sglang.py with build_sglang_serve_app() mirroring
  build_vllm_serve_app() but launching sglang.launch_server
- Add backend field to DeployConfig ('vllm' | 'sglang')
- Add extra_sglang_args and sglang_image_tag to DeployConfig
- Update ModelConfig.serve() to dispatch based on deploy.backend
- Fully backward compatible: defaults to vllm when unset

Co-Authored-By: Joy <joyliu.q@gmail.com>
@devin-ai-integration
Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant