Skip to content

[LIVY-1083] Coerce batch/session Int fields sent as JSON strings - #558

Open
soumyadeeplogin wants to merge 2 commits into
apache:masterfrom
soumyadeeplogin:livy-item10-int-field-json-coercion
Open

soumyadeeplogin wants to merge 2 commits into
apache:masterfrom
soumyadeeplogin:livy-item10-int-field-json-coercion

Conversation

@soumyadeeplogin

@soumyadeeplogin soumyadeeplogin commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

CreateBatchRequest and CreateInteractiveRequest declare driverCores, executorCores and numExecutors as Option[Int]. Because of Scala type erasure, Option[Int] erases to Option[Object], so jackson-module-scala cannot tell the element type is Integer. When a client sends one of these fields as a JSON string (e.g. "4" instead of 4), Jackson stores it verbatim as Some("4") without error. The bad value only surfaces later as a ClassCastException when the code unboxes it to Int while building the Spark submit command, which the servlet returns to callers as an opaque HTTP 500.

This PR annotates the three fields on both request classes with @JsonDeserialize(contentAs = classOf[java.lang.Integer]) so Jackson coerces numeric strings to Int during deserialization, and rejects truly non-numeric strings with a client-facing 400 (JsonMappingException) instead of a 500 later during app creation.

Why are the changes needed?

To fail fast with a clear 400 at the API boundary when a client sends a malformed numeric field, instead of an opaque 500 deep in Spark app submission logic.

Does this PR introduce any user-facing change?

Yes, but only for previously-broken input. POST /batches and interactive session creation now accept JSON string values for driverCores/executorCores/numExecutors (e.g. "4") by coercing them to the equivalent int, and return a 400 instead of a 500 for genuinely non-numeric strings (e.g. "notanumber", "4.5").

How was this patch tested?

Added unit tests to CreateBatchRequestSpec and CreateInteractiveRequestSpec covering: numeric-as-JSON-number (unchanged behavior), numeric-as-JSON-string (coerced), non-numeric string (400/JsonMappingException), empty string (None), whitespace-padded numeric string (coerced), and fractional numeric string (rejected).

Ran mvn -pl server -am test -DwildcardSuites=org.apache.livy.server.batch.CreateBatchRequestSpec,org.apache.livy.server.interactive.CreateInteractiveRequestSpec; all 14 tests pass.

Was this patch authored or co-authored using generative AI tooling?

Yes, this patch was co-authored using Claude Code (Anthropic).

driverCores, executorCores and numExecutors are Option[Int], but Scala
type erasure means jackson-module-scala stores a JSON string such as "4"
verbatim as Some("4") without error. The bad value only surfaces later
as a ClassCastException during unboxing in BatchSession/InteractiveSession
app creation, which the servlet returns to callers as an opaque HTTP 500.

Annotate the three fields on CreateBatchRequest and CreateInteractiveRequest
with @JsonDeserialize(contentAs = classOf[Integer]) so Jackson coerces
numeric strings to Int and rejects truly non-numeric strings with a 400
instead of a 500.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@soumyadeeplogin

Copy link
Copy Markdown
Contributor Author

Filed LIVY-1083 for this change.

…RequestSpec

CI failed on a 103-character line introduced by the JSON-string coercion
test. Split the literal across two concatenated strings to fit under the
100-character limit.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@soumyadeeplogin

Copy link
Copy Markdown
Contributor Author

Hi @gyogal / @roczei — could one of you approve the pending CI workflow run for this PR (fork-contributor approval gate)? Happy to address any review feedback as well. Thanks!

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.10%. Comparing base (501e822) to head (015c296).
⚠️ Report is 12 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master     #558      +/-   ##
============================================
+ Coverage     68.68%   73.10%   +4.41%     
- Complexity     1218     1295      +77     
============================================
  Files           106      106              
  Lines          6815     6874      +59     
  Branches        836      855      +19     
============================================
+ Hits           4681     5025     +344     
+ Misses         1666     1344     -322     
- Partials        468      505      +37     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants