Skip to content

[LIVY-1086][TEST] Pin Spark driver to loopback in remaining macOS test fixtures - #555

Open
nileshrathi345 wants to merge 1 commit into
apache:masterfrom
nileshrathi345:master_LIVY-1065
Open

[LIVY-1086][TEST] Pin Spark driver to loopback in remaining macOS test fixtures#555
nileshrathi345 wants to merge 1 commit into
apache:masterfrom
nileshrathi345:master_LIVY-1065

Conversation

@nileshrathi345

@nileshrathi345 nileshrathi345 commented Sep 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Extend the LIVY-1065 macOS test fixes by binding Spark's driver host and bind address to TestUtils.TEST_BIND_HOST in test fixtures that start Spark directly (REPL interpreter/session specs, RSC client, Scala API client, interactive server tests, Thrift server JDBC sessions, and MiniCluster).

What changes were proposed in this pull request?

This is a follow-up to #538

PR #538 introduced TestUtils.TEST_BIND_HOST and pinned Livy, RSC, YARN mini-cluster, and several unit-test services to loopback (127.0.0.1) so tests pass on macOS. However, some unit tests still start Spark directly without setting spark.driver.host and spark.driver.bindAddress. On macOS, Spark's driver can advertise the machine's LAN IP (from InetAddress.getLocalHost()), which peer processes on the same host cannot reach, causing connect timeouts and stage failures (e.g. Failed to connect to /192.168.x.x:...).

This PR extends the LIVY-1065 fix by:

  • client-common/TestUtils.java: Add SPARK_DRIVER_HOST and SPARK_DRIVER_BIND_ADDRESS constants so Spark driver config keys are not duplicated as raw strings across modules (same pattern as TEST_BIND_HOST).

  • Unit-test fixtures: Set spark.driver.host and spark.driver.bindAddress to TestUtils.TEST_BIND_HOST in fixtures that create Spark sessions/drivers directly:

  1. repl: BaseSessionSpec ,SessionSpec, ReplDriverSuite, PythonInterpreterSpec, ScalaInterpreterSpec, SQLInterpreterSpec, SparkRInterpreterSpec
  2. rsc: TestSparkClient
  3. scala-api: ScalaClientTest
  4. server: BaseInteractiveServletSpec, InteractiveSessionSpec
  5. thriftserver: ThriftServerBaseTest (inject driver host/bind via JDBC session conf in jdbcUri())
  • integration-test/MiniCluster.scala: Use TestUtils.SPARK_DRIVER_HOST instead of a hardcoded "spark.driver.host" key string (value remains TEST_BIND_HOST from #538).

Together with #538, this ensures both Livy/RSC services and Spark driver processes started from tests bind/advertise a locally reachable loopback address on macOS. Linux CI behavior is unchanged.

JIRA: https://issues.apache.org/jira/browse/LIVY-1086

How was this patch tested?

Tested locally on macOS (aarch64, JDK 17):
`$ mvn -Pthriftserver -B verify
...

[INFO] Rat check: Summary over all files. Unapproved: 0, unknown: 0, generated: 0, approved: 1 licenses.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Livy Project Parent POM 1.0.0-SNAPSHOT:
[INFO] 
[INFO] Livy Project Parent POM ............................ SUCCESS [  3.706 s]
[INFO] livy-api ........................................... SUCCESS [  5.891 s]
[INFO] livy-client-common ................................. SUCCESS [  2.877 s]
[INFO] livy-test-lib ...................................... SUCCESS [  3.400 s]
[INFO] multi-scala-project-root ........................... SUCCESS [  1.027 s]
[INFO] livy-core-parent ................................... SUCCESS [  0.295 s]
[INFO] livy-core_2.13 ..................................... SUCCESS [  1.394 s]
[INFO] livy-rsc ........................................... SUCCESS [ 29.710 s]
[INFO] livy-repl-parent ................................... SUCCESS [  3.695 s]
[INFO] livy-repl_2.13 ..................................... SUCCESS [01:29 min]
[INFO] livy-server ........................................ SUCCESS [03:33 min]
[INFO] livy-thriftserver-session .......................... SUCCESS [ 14.393 s]
[INFO] livy-thriftserver .................................. SUCCESS [05:34 min]
[INFO] livy-assembly ...................................... SUCCESS [  4.162 s]
[INFO] livy-client-http ................................... SUCCESS [ 12.459 s]
[INFO] livy-scala-api-parent .............................. SUCCESS [  1.066 s]
[INFO] livy-scala-api_2.13 ................................ SUCCESS [ 20.797 s]
[INFO] livy-integration-test .............................. SUCCESS [24:52 min]
[INFO] livy-coverage-report ............................... SUCCESS [  3.216 s]
[INFO] livy-examples ...................................... SUCCESS [  4.174 s]
[INFO] livy-python-api .................................... SUCCESS [  2.423 s]
[INFO] livy-beeline ....................................... SUCCESS [  2.968 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  37:27 min
[INFO] Finished at: 2026-09-17T19:24:42+05:30
[INFO] ------------------------------------------------------------------------

Was this patch authored or co-authored using generative AI tooling?

Coauthored-by: Claude Code (Opus 4.8)

@nileshrathi345

nileshrathi345 commented Sep 17, 2026

Copy link
Copy Markdown
Contributor Author

Hi @gyogal and @roczei ,
This is a follow-up to #538 (LIVY-1065).
Could you please take a look at this when you have a moment? Any feedback you provide would be greatly appreciated! Thank you..!!

@roczei

roczei commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Thanks a lot for this follow-up PR. On my end, I had this set in my environment variables, which is why I didn't notice it: export SPARK_LOCAL_IP="127.0.0.1". I've commented it out in my $HOME/.bashrc now.

It seems like it's still not quite working on macOS, because when I run this for example, I get an error like this:

mvn -Pthriftserver -B verify
...
[INFO] Running org.apache.livy.thriftserver.session.ThriftSessionTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.683 s - in org.apache.livy.thriftserver.session.ThriftSessionTest
[INFO] Running org.apache.livy.thriftserver.session.ColumnBufferTest
[ERROR] Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 76.203 s <<< FAILURE! - in org.apache.livy.thriftserver.session.ColumnBufferTest
[ERROR] testColumnBuffer(org.apache.livy.thriftserver.session.ColumnBufferTest)  Time elapsed: 76.2 s  <<< ERROR!
org.apache.spark.SparkException: 
Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1) (192.168.0.2 executor driver): org.apache.spark.executor.RemoteClassLoaderError: org.apache.spark.sql.catalyst.expressions.Object
	at org.apache.spark.executor.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:117)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:592)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:579)
	at org.apache.spark.util.ParentClassLoader.loadClass(ParentClassLoader.java:40)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
	at org.codehaus.janino.ClassLoaderIClassLoader.findIClass(ClassLoaderIClassLoader.java:75)

@nileshrathi345
[LIVY-1065] Pin Spark driver to loopback in remaining macOS tests
e02a1ea 
Extend the LIVY-1065 macOS test fixes by binding Spark's driver host and
bind address to TestUtils.TEST_BIND_HOST in test fixtures that start Spark
directly (REPL interpreter/session specs, RSC client, Scala API client,
interactive server tests, Thrift server JDBC sessions, and MiniCluster).
@gyogal

gyogal commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Thanks for your contribution @nileshrathi345 ! I am still going through the changes, but in the meantime, could you please create a new JIRA for this? It seems extensive enough to have its own ticket.

@nileshrathi345

nileshrathi345 commented Sep 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Thanks a lot for this follow-up PR. On my end, I had this set in my environment variables, which is why I didn't notice it: export SPARK_LOCAL_IP="127.0.0.1". I've commented it out in my $HOME/.bashrc now.

It seems like it's still not quite working on macOS, because when I run this for example, I get an error like this:

mvn -Pthriftserver -B verify
...
[INFO] Running org.apache.livy.thriftserver.session.ThriftSessionTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.683 s - in org.apache.livy.thriftserver.session.ThriftSessionTest
[INFO] Running org.apache.livy.thriftserver.session.ColumnBufferTest
[ERROR] Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 76.203 s <<< FAILURE! - in org.apache.livy.thriftserver.session.ColumnBufferTest
[ERROR] testColumnBuffer(org.apache.livy.thriftserver.session.ColumnBufferTest)  Time elapsed: 76.2 s  <<< ERROR!
org.apache.spark.SparkException: 
Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1) (192.168.0.2 executor driver): org.apache.spark.executor.RemoteClassLoaderError: org.apache.spark.sql.catalyst.expressions.Object
	at org.apache.spark.executor.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:117)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:592)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:579)
	at org.apache.spark.util.ParentClassLoader.loadClass(ParentClassLoader.java:40)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
	at org.codehaus.janino.ClassLoaderIClassLoader.findIClass(ClassLoaderIClassLoader.java:75)

Tested complete unit and integration tests and pushed changes

$ mvn -Pthriftserver -B verify
...

[INFO] Rat check: Summary over all files. Unapproved: 0, unknown: 0, generated: 0, approved: 1 licenses.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Livy Project Parent POM 1.0.0-SNAPSHOT:
[INFO] 
[INFO] Livy Project Parent POM ............................ SUCCESS [  3.706 s]
[INFO] livy-api ........................................... SUCCESS [  5.891 s]
[INFO] livy-client-common ................................. SUCCESS [  2.877 s]
[INFO] livy-test-lib ...................................... SUCCESS [  3.400 s]
[INFO] multi-scala-project-root ........................... SUCCESS [  1.027 s]
[INFO] livy-core-parent ................................... SUCCESS [  0.295 s]
[INFO] livy-core_2.13 ..................................... SUCCESS [  1.394 s]
[INFO] livy-rsc ........................................... SUCCESS [ 29.710 s]
[INFO] livy-repl-parent ................................... SUCCESS [  3.695 s]
[INFO] livy-repl_2.13 ..................................... SUCCESS [01:29 min]
[INFO] livy-server ........................................ SUCCESS [03:33 min]
[INFO] livy-thriftserver-session .......................... SUCCESS [ 14.393 s]
[INFO] livy-thriftserver .................................. SUCCESS [05:34 min]
[INFO] livy-assembly ...................................... SUCCESS [  4.162 s]
[INFO] livy-client-http ................................... SUCCESS [ 12.459 s]
[INFO] livy-scala-api-parent .............................. SUCCESS [  1.066 s]
[INFO] livy-scala-api_2.13 ................................ SUCCESS [ 20.797 s]
[INFO] livy-integration-test .............................. SUCCESS [24:52 min]
[INFO] livy-coverage-report ............................... SUCCESS [  3.216 s]
[INFO] livy-examples ...................................... SUCCESS [  4.174 s]
[INFO] livy-python-api .................................... SUCCESS [  2.423 s]
[INFO] livy-beeline ....................................... SUCCESS [  2.968 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  37:27 min
[INFO] Finished at: 2026-09-17T19:24:42+05:30
[INFO] ------------------------------------------------------------------------

@nileshrathi345

nileshrathi345 commented Sep 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Thanks for your contribution @nileshrathi345 ! I am still going through the changes, but in the meantime, could you please create a new JIRA for this? It seems extensive enough to have its own ticket.

Sure, I have created new Jira for this: https://issues.apache.org/jira/browse/LIVY-1086

@gyogal gyogal changed the title [LIVY-1065] Pin Spark driver to loopback in remaining macOS test fixtures [LIVY-1065][TEST] Pin Spark driver to loopback in remaining macOS test fixtures Sep 17, 2026
@gyogal gyogal changed the title [LIVY-1065][TEST] Pin Spark driver to loopback in remaining macOS test fixtures [LIVY-1086][TEST] Pin Spark driver to loopback in remaining macOS test fixtures Sep 17, 2026
@codecov-commenter

codecov-commenter commented Sep 17, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 53.10%. Comparing base (501e822) to head (e02a1ea).
⚠️ Report is 12 commits behind head on master.

Additional details and impacted files 
@@              Coverage Diff              @@
##             master     #555       +/-   ##
=============================================
- Coverage     68.68%   53.10%   -15.58%     
+ Complexity     1218      849      -369     
=============================================
  Files           106      106               
  Lines          6815     6871       +56     
  Branches        836      842        +6     
=============================================
- Hits           4681     3649     -1032     
- Misses         1666     2777     +1111     
+ Partials        468      445       -23

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@nileshrathi345 @roczei @gyogal @codecov-commenter