[LIVY-1082] Add config to disable Kubernetes driver pod log polling - #557
Open
soumyadeeplogin wants to merge 1 commit into
Open
soumyadeeplogin wants to merge 1 commit into
soumyadeeplogin wants to merge 1 commit into
Conversation
Livy fetches the driver pod's Kubernetes log (pods/<driver>/log) on every poll cycle. On large clusters or when driver logs are already collected externally (e.g. via a centralized log pipeline), this is wasted load against the Kubernetes API server for log content that is never read. Add livy.server.kubernetes.driver-log-polling.enabled (default true) to skip this fetch while still polling pod state and diagnostics. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
Author
|
Filed LIVY-1082 for this change. |
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
SparkKubernetesAppcurrently fetches the driver pod's Kubernetes log (pods/<driver>/log) on every poll cycle to populate live log lines in/sessions/:id/logand/batches/:id/logresponses. On large clusters, or when driver logs are already collected externally (e.g. via a centralized log pipeline / Spark History Server), this is wasted request volume against the Kubernetes API server, and can be a source of transient errors during pod lifecycle transitions.This PR adds a new config,
livy.server.kubernetes.driver-log-polling.enabled(defaulttrue, preserving existing behavior), that when set tofalseskips thepods/<driver>/logfetch while Livy continues to poll pod state and diagnostics normally.Why are the changes needed?
To let operators opt out of the driver log poll when it's unnecessary, reducing Kubernetes API server load and avoiding transient log-fetch errors during pod lifecycle transitions, without losing pod state/diagnostics monitoring.
Does this PR introduce any user-facing change?
Yes. A new optional config
livy.server.kubernetes.driver-log-polling.enabledis added (defaulttrue). When explicitly set tofalse,/sessions/:id/logand/batches/:id/logwill no longer include live Kubernetes driver log lines; pod state and diagnostics continue to be polled and reported as before.How was this patch tested?
Added/updated unit tests in
SparkKubernetesAppSpec:should enable driver log polling by defaultresolveDriverAppLog-should skip fetching the log when driver log polling is disabledresolveDriverAppLog-should fetch the log when driver log polling is enabledRan
mvn -pl server -am test -Dtest=SparkKubernetesAppSpec -DwildcardSuites=org.apache.livy.utils.SparkKubernetesAppSpec; all 14 tests in the suite pass.Was this patch authored or co-authored using generative AI tooling?
Yes, this patch was co-authored using Claude Code (Anthropic).