Skip to content

Fix deadlock in rocm-run - #560

Merged
umangyadav merged 1 commit into
developfrom
fix/deadlock-in-rocm-run
Oct 6, 2026
Merged

umangyadav merged 1 commit into
developfrom
fix/deadlock-in-rocm-run

Conversation

@pfultz2

@pfultz2 pfultz2 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

The paremeterSweeps-conv.py tests hangs.rocprofv3 profiles by LD_PRELOADing a library whose constructor spawns threads, and fork() in that state can deadlock the child on an inherited lock.

Technical Details

This rocm-run script must not fork before the final exec (no command substitutions, subshells, or pipes) due to rocprofv3. Paths are therefore built by parameter expansion and left unnormalized (they may contain ".." segments).

Test Plan

Test Result

Submission Checklist

@rocmlir-pr-reviewer rocmlir-pr-reviewer Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE -- submitted as COMMENT (automated reviews are advisory)  ·  Findings: 0 (0 Critical, 0 Major, 0 Minor)


Scope

Single-file change to mlir/utils/widgets/rocm-run, a developer convenience wrapper around mlir-runner. The PR removes every fork() that happened before the final exec, because rocprofv3 profiles the script by LD_PRELOADing a library whose constructor spawns threads, and forking from that state can deadlock the child on an inherited lock. Three fork sites are replaced: $(cd … && pwd) command substitutions become parameter expansion, the ( cd … && pwd ) subshell-echo return convention becomes global assignment (BASEDIR / LLVM_BUILD) with a documented contract comment on each function, and the awk pipeline over CMakeCache.txt becomes a while IFS= read -r loop with an input redirect.

Findings

No blocking issues found.

Notes

  • I walked the post-change file end to end for residual fork points and found none: case, [[ ]], read, local, and echo are all bash builtins, and done < "$cache" is a plain redirect (bash does not subshell a redirected while). The stated goal is fully achieved.
  • Behavior-preservation spot-checks: ${line#*=} strips the shortest …= prefix, matching the old sub(/^[^=]*=/, ""); the LLVM_DIR:*=* pattern still excludes LLVM_DIR-ADVANCED:INTERNAL=1 as the surrounding comment claims; and break inside case inside while correctly exits the loop. The repo_root fallback is slightly more robust than before, since the old cd … 2>/dev/null && pwd could silently yield an empty string and probe /build.
  • Tradeoff worth being aware of (not a defect): the old cd+pwd collapsed .. logically, whereas the now-unnormalized strings are resolved physically by the kernel. If a bin/ directory or $PWD component is a symlink into another tree, $SCRIPT_DIR/.. can land somewhere the old code would not. Unlikely for this script's install locations, and the file comment already flags the unnormalized paths — just noting the semantic delta is slightly wider than "cosmetic .. segments".
  • rocMLIR back-port: mlir/utils/widgets/ is not in the "shared with rocMLIR" path list in docs/PR_REVIEW_CHECKLIST.md, so no back-port note is strictly required. That said, the script's CMakeCache.txt/LLVM_DIR fallback targets the external-LLVM layout, which suggests a counterpart may exist upstream — worth a quick confirmation rather than a blocking finding.
  • The PR description leaves Test Plan and Test Result as empty placeholders. Since the motivation is a reproducible hang in parameterSweeps-conv.py, a one-line confirmation that the sweep now completes under rocprofv3 would make this self-documenting.

CI status

No checks in fail or cancel. py-checks, detect, Set System Property, and Kill old PR builds pass; Jenkins, Build and Test, MIGraphX, and Code coverage are still pending at the reviewed SHA. The review check is this auto-review pipeline itself.

@rocmlir-pr-reviewer rocmlir-pr-reviewer Bot removed the claude-review Request a Claude PR review label Oct 2, 2026
@umangyadav
umangyadav merged commit 5cd7ec6 into develop Oct 6, 2026
12 of 15 checks passed
@umangyadav
umangyadav deleted the fix/deadlock-in-rocm-run branch October 6, 2026 19:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants