Skip to content

Fix broken spark-3.5-base-hadoop3.2.dockerfile #503

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
abhisheknath2011 merged 2 commits into from
Mar 19, 2026
Merged

Fix broken spark-3.5-base-hadoop3.2.dockerfile #503

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
31cd56a
[Fix] Fix broken spark-3.5-base-hadoop3.2.dockerfile
Mar 18, 2026
a492387
[Docs] Add --master flag to spark-shell command in SETUP.md
Mar 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion SETUP.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -234,7 +234,8 @@ docker exec -it local.spark-master /bin/bash
Start `spark-shell` with the following command: Available users are `openhouse` and `u_tableowner`.

```
bin/spark-shell --packages org.apache.iceberg:iceberg-spark-runtime-3.1_2.12:1.2.0 \
bin/spark-shell --master spark://spark-master:7077 \
--packages org.apache.iceberg:iceberg-spark-runtime-3.1_2.12:1.2.0 \
--jars openhouse-spark-runtime_2.12-*-all.jar \
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions,com.linkedin.openhouse.spark.extensions.OpenhouseSparkSessionExtensions \
--conf spark.sql.catalog.openhouse=org.apache.iceberg.spark.SparkCatalog \
Expand All @@ -245,6 +246,10 @@ bin/spark-shell --packages org.apache.iceberg:iceberg-spark-runtime-3.1_2.12:1.2
--conf spark.sql.catalog.openhouse.cluster=LocalHadoopCluster
```

> **Note:** `--master spark://spark-master:7077` connects to the Spark standalone cluster
> instead of using the default `local[*]` mode. Without this, Spark actions that scan
> HDFS (e.g. orphan file deletion) may hang.

If you are integrating with ADLS, use this `spark-shell` command instead:

```
Expand Down
12 changes: 7 additions & 5 deletions infra/recipes/docker-compose/common/spark/spark-3.5-base-hadoop3.2.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
FROM openjdk:11.0.11-jdk-slim-buster as builder
FROM eclipse-temurin:11-jdk-jammy as builder

RUN apt-get update && apt-get install -y \
git curl vim zip software-properties-common ssh net-tools ca-certificates \
# Update package lists and install packages with proper security
RUN apt-get update && apt-get install -y --no-install-recommends \
git curl vim zip unzip software-properties-common ssh net-tools ca-certificates \
# Add Dependencies for PySpark \
python3 python3-pip python3-numpy python3-matplotlib python3-scipy python3-pandas python3-simpy
python3 python3-pip python3-numpy python3-matplotlib python3-scipy python3-pandas python3-simpy && \
apt-get clean && rm -rf /var/lib/apt/lists/*

RUN update-alternatives --install "/usr/bin/python" "python" "$(which python3)" 1

Expand All @@ -25,7 +27,7 @@ RUN curl --no-verbose -o apache-spark.tgz \
&& rm apache-spark.tgz

# install maven to build apache livy
RUN curl --no-verbose -o maven.tgz https://dlcdn.apache.org/maven/maven-3/${MAVEN_VERSION}/binaries/apache-maven-${MAVEN_VERSION}-bin.tar.gz \
RUN curl --no-verbose -o maven.tgz https://archive.apache.org/dist/maven/maven-3/${MAVEN_VERSION}/binaries/apache-maven-${MAVEN_VERSION}-bin.tar.gz \
&& mkdir -p /opt/maven \
&& tar -xf maven.tgz -C /opt/maven --strip-components=1 \
&& rm maven.tgz
Expand Down