Spark + MinIO + Jupyter Learning Stack

A self-contained Docker Compose stack for learning PySpark against an S3-compatible object store (MinIO), with all the S3A/AWS jars baked into the images.

┌─────────────┐    s3a://    ┌──────────┐
│   Jupyter   │ ───────────► │  MinIO   │  (S3 storage)
│  (driver)   │              └──────────┘
└──────┬──────┘
       │ spark://spark-master:7077
       ▼
┌─────────────┐         ┌──────────────┐
│ Spark master│ ◄──────►│ worker  ×3   │  (executors, 6 cores / 6G)
└─────────────┘         └──────────────┘

Services

Service	Image	Port	Purpose
minio	minio/minio	9000	S3 API
		9001	Web console (minioadmin/minioadmin)
spark-master	spark-s3a:3.5.5 (built)	8080	Master Web UI
		7077	Master RPC
spark-worker	spark-s3a:3.5.5 (built)	8081	Worker Web UI (worker #1)
spark-worker-2	spark-s3a:3.5.5 (built)	8082	Worker Web UI (worker #2)
spark-worker-3	spark-s3a:3.5.5 (built)	8083	Worker Web UI (worker #3)
jupyter	jupyter-pyspark:3.5.5 (built)	8888	JupyterLab (token: spark)
(Spark app UI)	— served by the driver	4040	Per-app Web UI: DAGs, jobs, stages

The minio-init container runs once at startup to create the raw and processed buckets.

The jars

Connecting Spark to MinIO needs two jars (Hadoop's S3A filesystem + the AWS SDK it depends on). They must match the Hadoop version Spark ships with. Spark 3.5.5 uses Hadoop 3.3.4, so the matched pair is:

• hadoop-aws-3.3.4.jar
• aws-java-sdk-bundle-1.12.262.jar

Both live in the repo's jars/ folder. Download them once with the included script (the bundle jar is ~280 MB, which is too large to fetch reliably at build time):

bash jars/download_jars.sh

The Dockerfiles then COPY them onto Spark's default classpath:

• Spark image → /opt/spark/jars/ (master + all workers)
• Jupyter image → pip-installed pyspark's jars/ directory (the driver)

So no --packages / --jars flags are needed at runtime.

Base image note: the stack uses the official apache/spark:3.5.5. The free bitnami/spark images were deprecated in 2025 (tags removed from Docker Hub), so apache/spark is the supported replacement. apache/spark has no standalone-mode entrypoint, so master/worker are launched directly via spark-class <Master|Worker class> in docker-compose.yml.

Run it

bash jars/download_jars.sh     # one-time: fetch the two S3A jars (~280 MB)
docker compose up -d --build

Then open:

• Jupyter → http://localhost:8888 (token: spark) → open work/spark_minio_examples.ipynb
• MinIO console → http://localhost:9001 (minioadmin / minioadmin)
• Spark Master UI → http://localhost:8080

Run the notebook top to bottom. It will:

1. Create a SparkSession pointed at MinIO.
2. Generate 10,000,000 rows as a synthetic transactions dataset and write them as CSV to s3a://raw/transactions_csv/.
3. Generate 10 JSON files × 100,000 rows with pandas + s3fs and write them to s3://raw/events_json/file_01.json … file_10.json.
4. Read both back to verify row counts.

Notes

• Spark Web UIs (after you've run getOrCreate() in the notebook):
  • http://localhost:4040 — application UI for the notebook's SparkSession. This is where you inspect DAGs (Jobs → a job → DAG Visualization, or Stages → a stage → DAG Visualization), plus stages, tasks, storage, executors, and SQL queries. Served by the driver (Jupyter container).
  • http://localhost:8080 — master/cluster UI: workers, total resources, and the list of running/completed applications.
  • http://localhost:8081 / :8082 / :8083 — per-worker UIs.
  • The app UI is only live while the SparkSession exists. spark.stop() or a kernel restart takes it down; re-running cell 1 brings it back. Spark bumps the port (4041, …) if 4040 is still held — hence the exposed range.
• Build caching (BuildKit — default on Docker Desktop / Compose v2). Layers are ordered least-likely-to-change first, and downloads are cache-mounted so a busted layer rebuilds fast:
  • Editing jupyter/requirements.txt → re-runs pip (wheels served from the pip cache mount, no PyPI re-download).
  • Editing the Java install / JAVA_HOME → re-runs apt (.debs served from the apt cache mount); does not bust the pip layer above it.
  • Re-running jars/download_jars.sh with unchanged jars → cache hit on the COPY layers (content-hash match), so no rebuild past them.
  • Editing the mounted notebook → no rebuild at all (it's a bind mount).
  • Force a clean rebuild with docker compose build --no-cache, and clear the download caches with docker buildx prune (or docker builder prune).
• The notebook connects to the cluster via spark://spark-master:7077. To run purely inside the Jupyter container (no cluster needed), set MASTER_URL = "local[*]" in the first code cell, or change the SPARK_MASTER_URL env var in docker-compose.yml.
• Spark writes large datasets as a directory of part-files, not a single file — that's expected and lets the work parallelize. Use .coalesce(1) only if you genuinely need one file.
• Change the MinIO credentials by editing both the minio and jupyter environment blocks in docker-compose.yml (and minio-init, which uses the same defaults).

Stop

docker compose down            # stop containers
docker compose down -v         # also delete the MinIO data volume