fix: [pytorch] Resolve ROCm deps via RUNPATHs#6452
Open
venkat1361 wants to merge 4 commits into
Open
Conversation
✅ All Checks Passed — Ready for Review
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
|
🎉 All checks passed! This PR is ready for review. |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR changes PyTorch wheel packaging to resolve ROCm shared-library dependencies via relocatable ELF RUNPATHs (embedded in torch shared objects) instead of preloading rocprofiler-sdk at import/startup time, avoiding duplicate-provider initialization when users run PyTorch under external rocprofv3.
Changes:
- Removes
rocprofiler-sdkfrom PyTorch’s generated Linux preload list. - Post-processes the built torch wheel on Linux to strip absolute RUNPATH entries and append
$ORIGIN-relative RUNPATHs pointing at ROCm Python package library directories. - Extends package promotion to also rewrite ROCm version fragments embedded in torch ELF RUNPATHs (with unit tests for the string rewrite helper).
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| external-builds/pytorch/build_prod_wheels.py | Drops rocprofiler-sdk preload and adds wheel repack step that patches torch .so RUNPATHs via patchelf. |
| build_tools/packaging/promote_packages.py | Updates torch ELF RUNPATHs during version promotion so promoted wheels keep valid ROCm package paths. |
| build_tools/packaging/tests/promote_keep_list_test.py | Adds unit tests covering ROCm version substitution within RUNPATH strings. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Commit 6a55012 added rocprofiler-sdk to PyTorch’s Linux startup preload list to fix nightly smoke tests after PyTorch/Kineto gained a runtime dependency on
librocprofiler-sdk.so.1.When PyTorch is launched under an external
rocprofv3, the profiler may already have loaded and initialized one rocprofiler-sdk provider. PyTorch startup can then explicitly load the wheel-bundled rocprofiler-sdk, creating a duplicate-provider path in the same process. In that scenario, the rocprofv3 tool configuration path can be invoked after rocprofiler-sdk has already closed its configuration window, leading to:This PR treats the original issue as a packaging/runtime dependency issue instead of an import-time preload issue. PyTorch’s ROCm-linked shared libraries should resolve their
DT_NEEDEDdependencies through relocatable wheel RUNPATHs.JIRA ID: AIPROFSDK-969
Technical Details
This PR removes rocprofiler-sdk from PyTorch’s generated _rocm_init.py preload list.
After pytorch build step unpacks the generated torch wheel, finds ROCm-linked torch shared objects, removes non-relocatable absolute RUNPATH entries, appends $ORIGIN-relative RUNPATH entries pointing to the installed ROCm Python package library directories, and repacks the wheel before copying.
This lets dependencies such as
librocprofiler-sdk.so.1resolve through the dynamic loader instead of through explicitctypes/rocm_sdk.initialize_process()preloading. If an externalrocprofv3has already loaded a compatible SDK provider with the same SONAME, the loader can use the already-loaded provider instead of PyTorch forcing another absolute SDK load.This PR also updates package promotion logic so torch ELF RUNPATHs stay valid when ROCm package versions are promoted. Text files such as
_rocm_init.py,version.py, and metadata were already updated during promotion, but RUNPATH strings embedded in ELF files also need their ROCm version fragments updated when promoting from nightly/RC versions to release versions.Test Plan
build_tools/packaging/tests/promote_keep_list_test.pyTest Result
Submission Checklist