Skip to content

[Build] Support Spark commit-based builds - #6898

Open
murali-db wants to merge 2 commits into
delta-io:masterfrom
murali-db:spark42_pin
Open

murali-db wants to merge 2 commits into
delta-io:masterfrom
murali-db:spark42_pin

Conversation

@murali-db

@murali-db murali-db commented May 28, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Add an opt-in sparkCommit hook that derives a local Spark artifact version for exact Spark SHA testing.
  • Keep normal Spark release matrices unchanged by excluding source-built Spark targets unless explicitly requested.
  • Add a manual workflow and helper script to build a Spark commit locally before running focused Delta build checks.

Test plan

  • bash -n project/scripts/build_spark_from_source.sh
  • git diff --check
  • build/sbt showSparkVersions

Full Spark commit build/test was not run because it requires building Apache Spark from source.

Add an opt-in Spark commit hook that keeps Spark version selection tied to compatibility profiles while resolving locally built Spark artifacts for a specific SHA.
~/.ivy2
~/.cache/coursier
~/.m2/repository
key: delta-spark-commit-${{ inputs.spark-commit }}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this should just be renamed to spark-source-ref

#
set -euo pipefail

SPARK_COMMIT="${SPARK_COMMIT:?SPARK_COMMIT is required}"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we make this generic instead and allow any refs, not just commits? Might make this script readily reusable.

SPARK_REPO="${SPARK_REPO:-https://github.com/apache/spark.git}"
SPARK_DIR="${SPARK_DIR:-/tmp/spark}"
SPARK_VERSION="${SPARK_VERSION:-4.2}"
if [[ -z "${SPARK_BASE_VERSION:-}" ]]; then

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to perform this version wrangling here as opposed to relying on version.sbt from whichever ref we're building from provide this information?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Grepping version.sbt has some precedence and might be more straightforward to do here.


if [[ -d "$SPARK_DIR/.git" ]]; then
echo "Using existing Spark checkout at $SPARK_DIR"
git -C "$SPARK_DIR" fetch "$SPARK_REPO" "$SPARK_COMMIT"

@seewishnew seewishnew May 28, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Maybe worth adding --depth 1 here and below in else branch. Spark is a fairly bulky repo to clone all its history when we're only ever interested in a single ref.

# Get all Spark versions as JSON array
python project/scripts/get_spark_version_info.py --all-spark-versions
# Output: ["4.0", "4.1"] or ["master", "4.0"] if master is present
# Output: ["4.0", "4.1"] (source-built commit targets are excluded by default)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's worth treating master as its own thing (iirc this had to be manually invoked for local builds) and keeping cross spark versions scala for master builds as is, and then separately carving a space for ourselves for parameterized arbitrary source-built refs of Spark


/** Spark master branch version (optional). Release branches should not build against master */
val MASTER: Option[SparkVersionSpec] = None
val MASTER: Option[SparkVersionSpec] = Some(spark42Commit)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's keep spark 42 separate from master. Related to this comment above: https://github.com/delta-io/delta/pull/6898/changes#r3320289230

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude seems to have an affinity to just club 4.2 with master. Needs constant explicit course correction ime.

writer.println(s""" "supportIceberg": "${spec.supportIceberg}",""")
writer.println(s""" "supportHudi": "${spec.supportHudi}"""")
writer.println(s""" "supportHudi": "${spec.supportHudi}",""")
writer.println(s""" "requiresSparkCommit": ${spec.requiresSparkCommit}""")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just for my understanding, if we're filtering that these specs do not require spark commit, why does the output need tweaking?

jacksonVersion = "2.18.2"
)

private val spark42Commit = SparkVersionSpec(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think spark42Commit should supersede the current spark42 that's tied to preview-5.

@murali-db

Copy link
Copy Markdown
Collaborator Author

Split this into two replacement PRs:

The split addresses the review feedback here: the workflows use spark-source-ref, the helper accepts arbitrary source refs and shallow-fetches Spark, the helper derives the artifact base version from Spark source when possible, and the consumer keeps Spark master separate from the Spark 4.2 compatibility line without adding source-built pseudo-versions to ordinary matrices.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants