Skip to content

feat: add TRT-LLM engine integration #280

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
zhengluo-nv wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat: add TRT-LLM engine integration #280

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions examples/p2p_transfer_k8s/client/trtllm/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

ARG TRTLLM_IMAGE=nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc11@sha256:d91c80ba8baf763782b1078267ed6b1e06363bebff4961094bf6e5679d371d04
FROM ${TRTLLM_IMAGE}

USER root

ENV NIXL_PREFIX=/opt/nvidia/nvda_nixl \
NIXL_LIB_DIR=/opt/nvidia/nvda_nixl/lib/x86_64-linux-gnu \
NIXL_PLUGIN_DIR=/opt/nvidia/nvda_nixl/lib/x86_64-linux-gnu/plugins
ENV LD_LIBRARY_PATH=${NIXL_LIB_DIR}:${NIXL_PLUGIN_DIR}:/usr/local/tensorrt/targets/x86_64-linux-gnu/lib:${LD_LIBRARY_PATH}

COPY examples/p2p_transfer_k8s/client/trtllm/install_modelexpress_client.sh /tmp/trtllm_modelexpress/install_modelexpress_client.sh
COPY examples/p2p_transfer_k8s/client/trtllm/fix_nixl_runpath.py /tmp/trtllm_modelexpress/fix_nixl_runpath.py

COPY modelexpress_client/python /tmp/modelexpress_client
COPY modelexpress_common/proto /tmp/proto

RUN bash /tmp/trtllm_modelexpress/install_modelexpress_client.sh && \
python3 /tmp/trtllm_modelexpress/fix_nixl_runpath.py

RUN python3 -c "import modelexpress; print('ModelExpress OK')" && \
test -f "$NIXL_PLUGIN_DIR/libplugin_UCX.so" && \
echo "TRT-LLM ModelExpress image OK" && \
rm -rf /tmp/trtllm_modelexpress /tmp/modelexpress_client /tmp/proto
78 changes: 0 additions & 78 deletions examples/p2p_transfer_k8s/client/trtllm/Dockerfile.ph3-gcp-gb200
View file
Open in desktop

This file was deleted.

Loading
Loading