[SkipRecovery] Fix get_json_object runtime path semantics [databricks]

[wjxiz1992]

JaCoCo production line coverage: +83 lines (sql-plugin +83: shared parser +4, Spark 330 shim +18, Spark 353 shim +43, Spark 400 shim +18; JsonPathParserSuite fix-line intersection)

Fixes #14290.

Description

Dataproc backported SPARK-46761 while retaining Spark 3.5.3. A Spark version check alone therefore cannot distinguish vanilla Spark 3.5.3 from the Google-patched runtime, and the previous revision inferred the behavior by evaluating a Catalyst expression at runtime.

This revision removes that semantic probe and uses the mapping established by testing every supported Dataproc line:

• Spark 3 versions other than 3.5.3 use the legacy parser.
• The exact Spark 3.5.3 shim uses the fixed parser only when the spark.dataproc.engine platform marker identifies Dataproc. Vanilla Spark 3.5.3 remains legacy so CPU and GPU behavior stay aligned.
• Spark 4 uses the fixed parser.
• The legacy and fixed parsers are independent copies of their corresponding Spark sources so each remains directly diffable.
• Null and invalid literal paths, plus all-invalid multi-path batches, remain guarded without invoking JNI with zero paths.

The change is cold-path only. Non-3.5.3 shims select statically, and the 3.5.3 shim performs one lazy SparkConf marker lookup. No per-row, per-batch, or JNI work was added; parsed instructions remain cached.

Dataproc runtime matrix

The quoted question-mark key was evaluated with the CPU Spark expression on every supported Classic and Serverless line.

Environment	Dataproc runtime tested	Spark	Scala	Java	Result
Classic 2.1	2.1.118-ubuntu20	3.3.2	2.12.18	11	NULL — legacy
Classic 2.2	2.2-ubuntu22, resolved to 2.2.84	3.5.3	2.12.18	11	QUESTION — fixed
Classic 2.3	2.3.35-ubuntu22	3.5.3	2.12.18	11	QUESTION — fixed
Serverless 1.2	1.2.86	3.5.1	2.12.18	17.0.17	NULL — legacy
Serverless 2.2	2.2.86	3.5.3	2.13.12	17.0.17	QUESTION — fixed
Serverless 2.3	2.3.39	3.5.3	2.13.12	17.0.20	QUESTION — fixed
Serverless 3.0	current 3.0 line	4.0.1	2.13.16	21.0.12	QUESTION — fixed

The exact Classic 2.2.86 image is retired and was rejected before resource creation, so the Classic 2.2 check used the currently resolvable 2.2 image, 2.2.84. All submitted Classic probes reached DONE with YARN FINISHED; all Serverless batches reached SUCCEEDED.

Exact-head Dataproc GPU validation

A clean custom JAR built from commit bc7d89bfa5aba4b3a558441994d331a4846b97ab was validated on Dataproc Classic 2.2.84-debian12 with Spark 3.5.3 and two Tesla T4 workers.

• Artifact SHA-256: 69194905a24df75828273219dd9c72c55625b272a5dea6a49c6575cdf703ca90
• The exact artifact and SHA were verified on the master and both workers.
• Job 9fe997a7047f45e29cfa6f18663f07e0: DONE, YARN FINISHED.
• CPU result: QUESTION; GPU result: QUESTION; CPU_GPU_EQUAL=true.
• The executed GPU plan contained GpuProject with get_json_object and GpuRange; GPU_PLAN_EVIDENCE=PASS.
• The plugin loaded the Spark 3.5.3 shim from the exact commit above.

Local validation

• Shim coverage check: passed.
• JsonPathParserSuite: Spark build versions 330, 332, 340, 351, 353, 400, and 401 all passed with succeeded 4, failed 0; Spark 400 passed with ANSI both off and on.
• Spark 3.3 dist and integration-test artifact build: all reactor modules succeeded; BUILD SUCCESS.
• Spark 3.3 / Python 3.10 focused integration tests: 3 passed, 32630 deselected; CPU/GPU equality and GPU-plan capture passed.
• JaCoCo fix-line intersection: sql-plugin +83 distinct covered production lines.

AI assistance: This change and PR description were prepared with AI assistance.

Checklists

Documentation

• Updated for new or modified user-facing features or behaviors
• No user-facing change

Testing

• Added or modified tests to cover new code paths
• Covered by existing tests
  (Please provide the names of the existing tests in the PR description.)
• Not required

Performance

• Tests ran and results are added in the PR description
• Issue filed with a link in the PR description
• Not required

[greptile-apps]

Greptile Summary

This PR moves JSON-path parsing into version-specific shims so GPU get_json_object follows legacy Spark 3, Dataproc-patched Spark 3.5.3, and Spark 4 semantics.

• Adds dedicated runtime parser selection for the Spark 3.5.3 shim.
• Handles null and all-invalid combined paths without invoking JNI with an empty path list.
• Adds Scala parser tests and CPU/GPU integration coverage for quoted question-mark keys and invalid paths.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains.

No blocking failure remains.

Important Files Changed

Filename	Overview
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuGetJsonObject.scala	Delegates literal parsing to version shims, safely handles null literals, and avoids zero-path JNI calls.
sql-plugin/src/main/spark353/scala/com/nvidia/spark/rapids/shims/GetJsonObjectShim.scala	Adds dedicated Spark 3.5.3 legacy-versus-Dataproc parser selection based on the platform marker.
sql-plugin/src/main/spark330/scala/com/nvidia/spark/rapids/shims/GetJsonObjectShim.scala	Retains legacy quoted-name parsing for the mapped Spark 3 variants.
sql-plugin/src/main/spark400/scala/com/nvidia/spark/rapids/shims/GetJsonObjectShim.scala	Implements the SPARK-46761-compatible parser for Spark 4 and later mapped shims.
integration_tests/src/main/python/get_json_test.py	Adds GPU-plan-verified parity coverage for question-mark keys and all-invalid multi-path evaluation.
tests/src/test/scala/com/nvidia/spark/rapids/JsonPathParserSuite.scala	Tests platform-sensitive parser selection, malformed paths, null literals, and active CPU parser compatibility.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  P[Literal JSON path] --> V{Spark shim}
  V -->|Spark 3 except 3.5.3| L[Legacy parser]
  V -->|Spark 3.5.3| D{Dataproc marker present?}
  D -->|No| L
  D -->|Yes| F[Fixed parser]
  V -->|Spark 4| F
  L --> I[Path instructions]
  F --> I
  I --> E{Any valid paths?}
  E -->|Yes| J[JNI multi-path evaluation]
  E -->|No| N[Construct null output columns]

Loading

_{Reviews (7): Last reviewed commit: "Merge branch 'main' into fix/14290-get-j..." | Re-trigger Greptile}

[Copilot]

Pull request overview

This pull request fixes GPU/CPU divergence for get_json_object when JSON paths contain quoted ? keys on some Spark 3.5 runtimes with vendor backports (e.g., Dataproc), by detecting the runtime’s Catalyst semantics once and selecting the matching JSON-path parser dialect (or failing closed to CPU when unknown).

Changes:

• Added a shared GetJsonObjectRuntimeSemantics classifier and updated shims to expose quotedQuestionMarkSupport (Spark 4 hard-coded modern; pre-Spark-4 probes Catalyst at runtime).
• Updated GpuGetJsonObject / JsonPathParser to parse quoted names using the selected dialect, and fail closed to CPU when runtime semantics are unknown.
• Added Scala unit tests plus Python integration tests for quoted ? keys and invalid-path batches (with GPU plan capture assertions).

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/src/test/scala/com/nvidia/spark/rapids/JsonPathParserSuite.scala	New unit coverage for parser dialect behavior and runtime probe classification.
sql-plugin/src/main/spark400/scala/com/nvidia/spark/rapids/shims/GetJsonObjectShim.scala	Spark 4 shim now reports modern quoted-? support.
sql-plugin/src/main/spark330/scala/com/nvidia/spark/rapids/shims/GetJsonObjectShim.scala	Pre-Spark-4 shim probes Catalyst once to detect quoted-? support at runtime.
sql-plugin/src/main/scala/com/nvidia/spark/rapids/shims/GetJsonObjectRuntimeSemantics.scala	New shared helper to classify probe results (modern/legacy/unknown).
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuGetJsonObject.scala	Plumbs dialect into parsing and expression conversion; adds fail-closed behavior and fixes empty multi-path handling.
integration_tests/src/main/python/get_json_test.py	Adds/updates ITs for quoted ? keys and invalid paths with executed-plan capture.

Suppressed comments (1)

integration_tests/src/main/python/get_json_test.py:158

• This query projects multiple get_json_object expressions, so it is expected to be combined into GpuMultiGetJsonObject (and not necessarily contain GpuGetJsonObject nodes). Asserting GpuGetJsonObject here can cause a false failure if expression combining is enabled (default). Assert GpuMultiGetJsonObject instead.

    assert_cpu_and_gpu_are_equal_collect_with_capture(
        lambda spark: spark.createDataFrame(data,schema=schema).selectExpr(
            'get_json_object(jsonStr, CAST(NULL AS STRING)) AS null_path',
            'get_json_object(jsonStr, "$[") AS malformed_path',
            'get_json_object(jsonStr, "not_a_path") AS missing_root'),
        exist_classes='GpuProjectExec,GpuGetJsonObject')

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[wjxiz1992]

I kept the GpuGetJsonObject plan assertion. GpuEquivalentExpressions.replaceMultiExpressions runs inside GpuProjectExec.internalDoExecuteColumnar while binding a GpuTieredProject, so it does not rewrite the Spark executedPlan tree inspected by exist_classes. Changing both assertions to GpuMultiGetJsonObject made the focused Spark 3.3 GPU IT fail; restoring GpuGetJsonObject passed both tests (2 passed).

[wjxiz1992]

build

[firestarman]

Personally GetJsonObjectRuntimeSemantics is not necessary, we can merge it with object GetJsonObjectShim.
It is not a good idea to put the expected result "QUESTION" and the Json test literal string {"?":"QUESTION"} into different classes/objects.

If you want keep GetJsonObjectRuntimeSemantics common, maybe we need callers to provide the expect results instead of the hardcode. e.g. def classifyQuotedQuestionMarkResult(result: => Any, expected: Any): Option[Boolean]

[wjxiz1992]

Updated.

[wjxiz1992]

build

[revans2]

I also am not thrilled with us trying to combine the different parser versions. For me if we are copying code from Spark, it would be nice to make sure it stays copied, even if it means more duplicate code. It just makes it that much simpler to maintain because we can diff it with the original code. I get in this case the change is very small, so for me it is just a nit.

[wjxiz1992]

build

[wjxiz1992]

build

[wjxiz1992]

build

[wjxiz1992]

build

[wjxiz1992]

build

[wjxiz1992]

build

[wjxiz1992]

Hi @revans2, when you have a chance, could you please take another look at this PR? I’ve followed up on your review feedback. Thanks!