Skip to content

fix(user_tools): replace deprecated prop containers - #2105

Open
WilliamK112 wants to merge 4 commits into
NVIDIA:devfrom
WilliamK112:codex/fix-deprecated-prop-container
Open

WilliamK112 wants to merge 4 commits into
NVIDIA:devfrom
WilliamK112:codex/fix-deprecated-prop-container

Conversation

@WilliamK112

@WilliamK112 WilliamK112 commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Description

Replace the remaining in-memory JSONPropertiesContainer usages in Dataproc and cluster inference with AbstractPropContainer, removing the runtime DeprecationWarning while preserving existing property access and JSON-string handling.

The migration now covers:

  • inferred cluster properties
  • Dataproc master and worker node properties
  • Dataproc pricing configuration
  • Dataproc instance descriptions

Regression coverage exercises all four paths, verifies the new container type and existing outputs, and confirms that the deprecated-container warning is not emitted.

Fixes #1898.

Tests

  • PYTHONPATH=src .venv/bin/python -m pytest -q tests/spark_rapids_tools_ut/test_deprecated_prop_container.py (4 passed)
  • .venv/bin/python -m pylint -d fixme --load-plugins pylint_pydantic --rcfile=../.pylintrc src/spark_rapids_pytools/cloud_api/dataproc.py src/spark_rapids_pytools/common/cluster_inference.py tests/spark_rapids_tools_ut/test_deprecated_prop_container.py (10.00/10)
  • .venv/bin/python -m flake8 src/spark_rapids_pytools/cloud_api/dataproc.py src/spark_rapids_pytools/common/cluster_inference.py tests/spark_rapids_tools_ut/test_deprecated_prop_container.py
  • .venv/bin/python -m py_compile src/spark_rapids_pytools/cloud_api/dataproc.py src/spark_rapids_pytools/common/cluster_inference.py tests/spark_rapids_tools_ut/test_deprecated_prop_container.py
  • git diff --check upstream/dev...HEAD

Signed-off-by: WilliamK112 <164879897+WilliamK112@users.noreply.github.com>
@github-actions github-actions Bot added the user_tools Scope the wrapper module running CSP, QualX, and reports (python) label Jun 23, 2026
@greptile-apps

greptile-apps Bot commented Jun 23, 2026

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge with no actionable correctness, security, or repository-rule issues identified.

Summary

  • Adds explicit parsing for JSON-string inputs before constructing AbstractPropContainer.
  • Migrates inferred clusters, Dataproc node properties, pricing configuration, and instance descriptions.
  • Adds regression coverage for container types, existing outputs, and suppression of the targeted deprecation warning.

Signed-off-by: WilliamK112 <164879897+WilliamK112@users.noreply.github.com>
…-prop-container

Signed-off-by: WilliamK112 <164879897+WilliamK112@users.noreply.github.com>
@parthosa

Copy link
Copy Markdown
Collaborator

One JSON string path still needs to be handled. generate_cluster_configuration() returns a rendered JSON string, while AbstractPropContainer stores it without parsing. This causes cluster inference to fail with string indices must be integers (same reason as premerge failure).

Could you parse cluster_conf before constructing AbstractPropContainer? The test should also have generate_cluster_configuration() return json.dumps instead of the dictionary

Signed-off-by: WilliamK112 <164879897+WilliamK112@users.noreply.github.com>
@WilliamK112

Copy link
Copy Markdown
Contributor Author

Addressed in 1a91b80. ClusterInference.infer_cluster() now parses the rendered JSON string with json.loads() before constructing AbstractPropContainer. The regression test now has generate_cluster_configuration() return json.dumps(...) and asserts that nested inferred-cluster values remain accessible.

Validation:

  • PYTHONPATH=src .venv/bin/python -m pytest -q tests/spark_rapids_tools_ut/test_deprecated_prop_container.py (4 passed)
  • Pylint on both modified files (10.00/10)
  • Flake8 on both modified files
  • git diff --check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

user_tools Scope the wrapper module running CSP, QualX, and reports (python)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Fix DeprecationWarning: Use AbstractPropContainer instead. This class does not support CspPaths.

3 participants