Skip to content

fix(user_tools): handle zero-worker Databricks clusters - #2160

Closed
WilliamK112 wants to merge 1 commit into
NVIDIA:devfrom
WilliamK112:codex/handle-zero-worker-clusters
Closed

fix(user_tools): handle zero-worker Databricks clusters#2160
WilliamK112 wants to merge 1 commit into
NVIDIA:devfrom
WilliamK112:codex/handle-zero-worker-clusters

Conversation

@WilliamK112

Copy link
Copy Markdown
Contributor

Description

Normalize a missing Databricks executors field to an empty list in both the AWS and Azure cluster implementations. This lets zero-worker/single-node cluster initialization finish cleanly and reach the existing ClusterBase._verify_workers_exist validation instead of failing earlier with a TypeError while iterating over None.

The change deliberately preserves the current policy of rejecting driver-only clusters; adding support for that cluster shape remains out of scope.

The regression test covers both platforms using the existing Databricks fixtures with executors removed and num_workers set to zero, and verifies the intended "no worker nodes" error.

Closes #2143.

Validation

Run from user_tools on current dev (bcda54d):

  • PYTHONPATH=src python -m pytest tests/spark_rapids_tools_ut/test_cluster.py -q — 9 passed
  • Pylint on the two implementations and regression test — 10.00/10
  • Flake8 on the two implementations and regression test
  • py_compile on the two implementations and regression test
  • git diff --check upstream/dev...HEAD

Signed-off-by: Ching-Wei Kang <ckang53@wisc.edu>
@github-actions github-actions Bot added the user_tools Scope the wrapper module running CSP, QualX, and reports (python) label Sep 11, 2026
@greptile-apps

greptile-apps Bot commented Sep 11, 2026

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

The PR appears safe to merge, with both implementations preserving existing validation behavior while avoiding the earlier missing-executor iteration failure.

Summary

  • Normalizes missing or null executor collections to an empty list for AWS and Azure Databricks clusters.
  • Preserves the existing policy that driver-only clusters are rejected.
  • Adds a parameterized regression test covering both platform implementations.

Reviews (1) · Last reviewed commit: "fix(user_tools): handle zero-worker Data..."

@amahussein

Copy link
Copy Markdown
Collaborator

duplicate of #2148

@amahussein amahussein closed this Sep 11, 2026
@WilliamK112

Copy link
Copy Markdown
Contributor Author

Thanks for pointing that out — I missed #2148 when checking for overlap. It covers the same zero-worker handling, so closing this duplicate makes sense.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

user_tools Scope the wrapper module running CSP, QualX, and reports (python)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] --cluster on a zero-worker (single-node) Databricks cluster raises TypeError instead of the intended "no worker nodes" error

3 participants