[LIVY-1083] Coerce batch/session Int fields sent as JSON strings - #558
Open
soumyadeeplogin wants to merge 2 commits into
Open
soumyadeeplogin wants to merge 2 commits into
soumyadeeplogin wants to merge 2 commits into
Conversation
driverCores, executorCores and numExecutors are Option[Int], but Scala
type erasure means jackson-module-scala stores a JSON string such as "4"
verbatim as Some("4") without error. The bad value only surfaces later
as a ClassCastException during unboxing in BatchSession/InteractiveSession
app creation, which the servlet returns to callers as an opaque HTTP 500.
Annotate the three fields on CreateBatchRequest and CreateInteractiveRequest
with @JsonDeserialize(contentAs = classOf[Integer]) so Jackson coerces
numeric strings to Int and rejects truly non-numeric strings with a 400
instead of a 500.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
Author
|
Filed LIVY-1083 for this change. |
…RequestSpec CI failed on a 103-character line introduced by the JSON-string coercion test. Split the literal across two concatenated strings to fit under the 100-character limit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
Author
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #558 +/- ##
============================================
+ Coverage 68.68% 73.10% +4.41%
- Complexity 1218 1295 +77
============================================
Files 106 106
Lines 6815 6874 +59
Branches 836 855 +19
============================================
+ Hits 4681 5025 +344
+ Misses 1666 1344 -322
- Partials 468 505 +37 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
CreateBatchRequestandCreateInteractiveRequestdeclaredriverCores,executorCoresandnumExecutorsasOption[Int]. Because of Scala type erasure,Option[Int]erases toOption[Object], sojackson-module-scalacannot tell the element type isInteger. When a client sends one of these fields as a JSON string (e.g."4"instead of4), Jackson stores it verbatim asSome("4")without error. The bad value only surfaces later as aClassCastExceptionwhen the code unboxes it toIntwhile building the Spark submit command, which the servlet returns to callers as an opaque HTTP 500.This PR annotates the three fields on both request classes with
@JsonDeserialize(contentAs = classOf[java.lang.Integer])so Jackson coerces numeric strings toIntduring deserialization, and rejects truly non-numeric strings with a client-facing 400 (JsonMappingException) instead of a 500 later during app creation.Why are the changes needed?
To fail fast with a clear 400 at the API boundary when a client sends a malformed numeric field, instead of an opaque 500 deep in Spark app submission logic.
Does this PR introduce any user-facing change?
Yes, but only for previously-broken input.
POST /batchesand interactive session creation now accept JSON string values fordriverCores/executorCores/numExecutors(e.g."4") by coercing them to the equivalent int, and return a 400 instead of a 500 for genuinely non-numeric strings (e.g."notanumber","4.5").How was this patch tested?
Added unit tests to
CreateBatchRequestSpecandCreateInteractiveRequestSpeccovering: numeric-as-JSON-number (unchanged behavior), numeric-as-JSON-string (coerced), non-numeric string (400/JsonMappingException), empty string (None), whitespace-padded numeric string (coerced), and fractional numeric string (rejected).Ran
mvn -pl server -am test -DwildcardSuites=org.apache.livy.server.batch.CreateBatchRequestSpec,org.apache.livy.server.interactive.CreateInteractiveRequestSpec; all 14 tests pass.Was this patch authored or co-authored using generative AI tooling?
Yes, this patch was co-authored using Claude Code (Anthropic).