Repository navigation
Conversation
Add a manual workflow and helper script that publish a selected Apache Spark source ref to the runner's local Maven cache under a deterministic commit-qualified version.
Keep the cache producer helper focused on the generic source-ref implementation so the workflow does not retain the old commit-only path.
0a32bfc to
5e38dc3
Compare
| ~/.sbt | ||
| ~/.ivy2 | ||
| ~/.cache/coursier | ||
| key: delta-sbt-cache-spark-source-${{ steps.spark-ref.outputs.spark_version }} |
There was a problem hiding this comment.
What are we restoring here in this step exactly?
There was a problem hiding this comment.
This separate source-ref workflow has been removed. The remaining SBT cache restore is the existing Spark matrix cache for ~/.sbt, ~/.ivy2, and ~/.cache/coursier; the Spark Maven artifact restore now lives in the Spark 4.2 matrix lane and restores only ~/.m2/repository/org/apache/spark.
| paths-ignore: | ||
| - '**.md' | ||
| - '**.txt' | ||
| workflow_dispatch: |
There was a problem hiding this comment.
Just curious: would we need a manual workflow dispatch for this test?
There was a problem hiding this comment.
Removed. The consumer test no longer has workflow_dispatch because it now runs through the existing Spark matrix. The only manual workflow left is the producer/cache-warming workflow, where dispatch is the intended entry point.
Avoid deriving the Spark source artifact cache key from runner metadata so producer and consumer workflows share an explicit Ubuntu cache namespace.
Keep the source cache workflow generic by resolving default Spark refs from CrossSparkVersions while still allowing manual source-ref overrides. Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
7cdd305 to
5acaac5
Compare
5acaac5 to
2b10780
Compare
Add an opt-in Spark source ref overlay for dependency resolution and a non-blocking PR signal that restores prebuilt Spark artifacts from the cache instead of rebuilding them.
Source-built Spark refs replace sparkVersion.value with a commit-qualified Maven coordinate, so select shim directories from the configured compatibility spec instead of looking up that artifact version in ALL_SPECS.
When a Spark source ref is selected, include Maven local in the Spark-aware resolvers so restored commit-qualified Spark artifacts are visible to SBT.
Allow the source-ref workflow to opt in to Maven local while SBT repository override is enabled, and verify the restored Spark Maven artifacts before invoking SBT.
Only expose Maven local to source-built Spark runs, and make a missing source Spark cache fail once while the remaining Spark 4.2 shards skip until the cache is warmed.
Remove the standalone source Spark workflow so the non-blocking Spark 4.2 lane remains the only cache consumer. Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Preserve the source-built Spark helper references and shared property/env lookup after restacking the consumer branch. Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
2b10780 to
279bfa8
Compare
|
Superseded by #6907. I folded the consumer changes into the main PR and removed the manual warmer model, so normal CI now prepares the source Spark cache automatically before the 4.2 shards run. |
## Summary - Removes the separate manual Spark artifact-cache workflow and makes `spark_test.yaml` prepare source-built Spark Maven artifacts automatically on cache miss. - Changes Delta's Spark 4.2 metadata from `4.2.0-preview5` to `4.2.0-SNAPSHOT`, matching the current Spark 4.2 development line. - Splits the Spark test workflow into published-artifact shards for 4.0/4.1 and source-built Spark shards for 4.2, so normal shards do not wait on Spark source preparation. - Factors shared test setup into two composite actions (`prepare-spark-env`, `run-delta-spark-tests`) and moves the Spark Maven cache key and source-built artifact-version formatting into the version helper. - Adds manual workflow-dispatch cache-key invalidation without allowing workflow-level Spark source ref overrides. - Supersedes #6908; the useful consumer changes are folded into this PR. ## How this works - `generate-matrix` emits source-build and non-source-build Spark version lists from `CrossSparkVersions.scala` metadata. - `CrossSparkVersions.scala` is the source of truth for the source-built Spark ref (`sourceBuildDefaultRef`). Workflows do not override it. - `prepare_spark_source_cache` runs once per source-build Spark version. It uses `.github/actions/prepare-spark-env` to resolve the Spark SHA + exact cache key, restore `~/.m2/repository/org/apache/spark`, build Spark with `build_spark.sh` on cache miss, and save the cache for the sharded test jobs. - `test` (published 4.0/4.1 shards) and `source_spark_test` (source-built 4.2 shards) share two composite actions: - `.github/actions/prepare-spark-env`: installs the JDK and, for source-build rows, restores/verifies source-built Spark artifacts. It builds on miss by default when source Spark is opted in; source test shards explicitly disable build-on-miss so they restore the cache instead of rebuilding 8x. - `.github/actions/run-delta-spark-tests`: restores the SBT cache, publishes the pinned Unity Catalog build, and runs the sharded Scala/Java tests with `SPARK_COMMIT` / `SPARK_ARTIFACT_VERSION` from the prepared env. - `sparkVersion` remains the compatibility selector (shims, suffixes, JVM options). `sparkArtifactVersion` is the Maven dependency version used for Spark artifacts, so source-built Spark can resolve `4.2.0-<sha12>-SNAPSHOT` without changing the compatibility profile. - The Spark SHA, source-built Maven artifact version, and exact Maven cache key are computed in `get_spark_version_info.py`; Scala and shell code do not reformat the commit-qualified artifact version themselves. - Maven local is scoped to source-built Spark runs via `USE_MAVEN_LOCAL_FOR_SOURCE_SPARK=true`. The workflow exposes a temporary Maven repository containing only the restored source-built Spark artifacts, so normal published Spark builds and unrelated dependencies do not pick up stale local artifacts. - The source-built Spark lane is non-blocking end to end: prep/build/cache failures and source Spark test failures are surfaced through the non-blocking source Spark signal, while published Spark shards remain blocking. - Manual runs can pass `source-cache-key-suffix` to force a distinct cache key without changing the source ref. ## Test plan - Parsed `.github/workflows/spark_test.yaml`, `.github/workflows/changelog_tests_spark42.yaml`, and `.github/actions/prepare-spark-env/action.yml` with PyYAML; confirmed `.github/workflows/spark_commit_artifacts.yaml` is removed. - `python3.12 -m py_compile project/scripts/get_spark_version_info.py project/tests/test_cross_spark_publish.py`. - `bash -n build/sbt project/scripts/build_spark.sh`. - `build/sbt exportSparkVersionsJson`. - `python3.12 project/scripts/get_spark_version_info.py --get-field 4.2 fullVersion` returns `"4.2.0-SNAPSHOT"`. - `python3.12 project/scripts/get_spark_version_info.py --source-build-spark-versions` returns `["4.2"]` and `--non-source-build-spark-versions` returns `["4.0", "4.1"]`. - Ran targeted Delta `SparkVersionsScriptTest` metadata/helper checks, including no-network `--resolve-source-build` coverage, `compute_spark_m2_cache_key`, and source-built artifact-version formatting tests. - `USE_MAVEN_LOCAL_FOR_SOURCE_SPARK=true build/sbt -DsparkVersion=4.2 -DsparkCommit=b6bd005ac7549411ec4e7dc944d7a0e19fd56561 -DsparkArtifactVersion=4.2.0-b6bd005ac754-SNAPSHOT "show sparkVersion" "show sparkArtifactVersion"` shows compatibility Spark `4.2.0-SNAPSHOT` and Maven artifact version `4.2.0-b6bd005ac754-SNAPSHOT`. - `build/sbt -DsparkVersion=4.2 -DsparkCommit=b6bd005ac7549411ec4e7dc944d7a0e19fd56561 "show sparkArtifactVersion"` fails with an actionable error requiring explicit `sparkArtifactVersion`. - CI validates the published Spark shards and source-built Spark 4.2 shards. ## Known Spark 4.2 Changelog Compatibility Follow-up - The standalone `Changelog tests on Spark 4.2` check currently compiles Delta's Spark 4.2 changelog shims against an API where `ChangelogInfo` / `ChangelogInfoUtils` have changed to `ChangelogContext`. This is a Spark 4.2 shim compatibility update, not a source Spark cache producer/consumer failure: the source-built Spark matrix shards pass. --------- Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Stacked PR
This PR is stacked on #6907. Use this link to review only the consumer changes.
Stack:
Summary
Delta Sparkmatrix lane for Spark 4.2 instead of adding a separate source-ref workflow.project/scripts/get_spark_version_info.py --resolve-source-buildhelper as the producer, so cache producer and consumer derive identical keys fromCrossSparkVersions.scalametadata.SPARK_COMMITandSPARK_ARTIFACT_VERSIONinto the Spark tests.Review SHAs
e1fd3f6ebe9fe5f951e12f0c762f279c863b9b4f279bfa89b735a40bf2b2b111bc9ec0d0413db462How this works
CrossSparkVersions.scala; UC keeps the same kind of metadata inproject/spark-versions.json. Both stacks now expose the sameget_spark_version_info.pyhelper name to workflows.build_spark.shhash.SPARK_COMMITandSPARK_ARTIFACT_VERSIONset.Test plan
.github/workflows/spark_commit_artifacts.yamland.github/workflows/spark_test.yamlwith PyYAML.python3.11 -m py_compile project/scripts/get_spark_version_info.py project/tests/test_cross_spark_publish.py.bash -n project/scripts/build_spark.sh.SparkVersionsScriptTestmetadata/helper checks, including no-network--resolve-source-buildcoverage.build/sbt -DsparkVersion=4.2 -DsparkCommit=b6bd005ac7549411ec4e7dc944d7a0e19fd56561 -DsparkArtifactVersion=4.2.0-b6bd005ac754-SNAPSHOT "show spark / scalaVersion".