Repository navigation
Conversation
Add an opt-in Spark commit hook that keeps Spark version selection tied to compatibility profiles while resolving locally built Spark artifacts for a specific SHA.
| ~/.ivy2 | ||
| ~/.cache/coursier | ||
| ~/.m2/repository | ||
| key: delta-spark-commit-${{ inputs.spark-commit }} |
There was a problem hiding this comment.
Nit: this should just be renamed to spark-source-ref
| # | ||
| set -euo pipefail | ||
|
|
||
| SPARK_COMMIT="${SPARK_COMMIT:?SPARK_COMMIT is required}" |
There was a problem hiding this comment.
Should we make this generic instead and allow any refs, not just commits? Might make this script readily reusable.
| SPARK_REPO="${SPARK_REPO:-https://github.com/apache/spark.git}" | ||
| SPARK_DIR="${SPARK_DIR:-/tmp/spark}" | ||
| SPARK_VERSION="${SPARK_VERSION:-4.2}" | ||
| if [[ -z "${SPARK_BASE_VERSION:-}" ]]; then |
There was a problem hiding this comment.
Do we need to perform this version wrangling here as opposed to relying on version.sbt from whichever ref we're building from provide this information?
There was a problem hiding this comment.
Grepping version.sbt has some precedence and might be more straightforward to do here.
|
|
||
| if [[ -d "$SPARK_DIR/.git" ]]; then | ||
| echo "Using existing Spark checkout at $SPARK_DIR" | ||
| git -C "$SPARK_DIR" fetch "$SPARK_REPO" "$SPARK_COMMIT" |
There was a problem hiding this comment.
Nit: Maybe worth adding --depth 1 here and below in else branch. Spark is a fairly bulky repo to clone all its history when we're only ever interested in a single ref.
| # Get all Spark versions as JSON array | ||
| python project/scripts/get_spark_version_info.py --all-spark-versions | ||
| # Output: ["4.0", "4.1"] or ["master", "4.0"] if master is present | ||
| # Output: ["4.0", "4.1"] (source-built commit targets are excluded by default) |
There was a problem hiding this comment.
I think it's worth treating master as its own thing (iirc this had to be manually invoked for local builds) and keeping cross spark versions scala for master builds as is, and then separately carving a space for ourselves for parameterized arbitrary source-built refs of Spark
|
|
||
| /** Spark master branch version (optional). Release branches should not build against master */ | ||
| val MASTER: Option[SparkVersionSpec] = None | ||
| val MASTER: Option[SparkVersionSpec] = Some(spark42Commit) |
There was a problem hiding this comment.
Let's keep spark 42 separate from master. Related to this comment above: https://github.com/delta-io/delta/pull/6898/changes#r3320289230
There was a problem hiding this comment.
Claude seems to have an affinity to just club 4.2 with master. Needs constant explicit course correction ime.
| writer.println(s""" "supportIceberg": "${spec.supportIceberg}",""") | ||
| writer.println(s""" "supportHudi": "${spec.supportHudi}"""") | ||
| writer.println(s""" "supportHudi": "${spec.supportHudi}",""") | ||
| writer.println(s""" "requiresSparkCommit": ${spec.requiresSparkCommit}""") |
There was a problem hiding this comment.
Just for my understanding, if we're filtering that these specs do not require spark commit, why does the output need tweaking?
| jacksonVersion = "2.18.2" | ||
| ) | ||
|
|
||
| private val spark42Commit = SparkVersionSpec( |
There was a problem hiding this comment.
I think spark42Commit should supersede the current spark42 that's tied to preview-5.
|
Split this into two replacement PRs:
The split addresses the review feedback here: the workflows use |
Summary
sparkCommithook that derives a local Spark artifact version for exact Spark SHA testing.Test plan
bash -n project/scripts/build_spark_from_source.shgit diff --checkbuild/sbt showSparkVersionsFull Spark commit build/test was not run because it requires building Apache Spark from source.