Skip to content

[Build] Test Delta against cached Spark source refs - #6908

Closed
murali-db wants to merge 16 commits into
delta-io:masterfrom
murali-db:stack/spark-source-cache-consumer
Closed

murali-db wants to merge 16 commits into
delta-io:masterfrom
murali-db:stack/spark-source-cache-consumer

Conversation

@murali-db

@murali-db murali-db commented May 29, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked PR

This PR is stacked on #6907. Use this link to review only the consumer changes.

Stack:

Summary

  • Reuses the existing non-blocking Delta Spark matrix lane for Spark 4.2 instead of adding a separate source-ref workflow.
  • Uses the same project/scripts/get_spark_version_info.py --resolve-source-build helper as the producer, so cache producer and consumer derive identical keys from CrossSparkVersions.scala metadata.
  • Restores cached Apache Spark artifacts, verifies the expected Maven modules, then passes SPARK_COMMIT and SPARK_ARTIFACT_VERSION into the Spark tests.
  • On cache miss, reports one non-blocking Spark 4.2 shard failure and skips the other Spark 4.2 shards until the cache is warmed.

Review SHAs

  • Producer head: e1fd3f6ebe9fe5f951e12f0c762f279c863b9b4f
  • Consumer head: 279bfa89b735a40bf2b2b111bc9ec0d0413db462

How this works

  • Delta keeps Spark metadata in CrossSparkVersions.scala; UC keeps the same kind of metadata in project/spark-versions.json. Both stacks now expose the same get_spark_version_info.py helper name to workflows.
  • The Spark 4.2 CI lane computes the same exact cache key as [Build] Test source-built Spark automatically in CI #6907: runner label, Scala binary line, normalized Spark compatibility line, artifact base version, resolved Spark SHA, and build_spark.sh hash.
  • On a cache hit, CI verifies the expected Spark Maven artifacts and runs the existing Spark matrix tests with SPARK_COMMIT and SPARK_ARTIFACT_VERSION set.
  • On a cache miss, only Spark 4.2 shard 0 reports the miss as the non-blocking signal; other Spark 4.2 shards skip to avoid noisy duplicate failures until the producer cache is warmed.

Test plan

  • Parsed .github/workflows/spark_commit_artifacts.yaml and .github/workflows/spark_test.yaml with PyYAML.
  • python3.11 -m py_compile project/scripts/get_spark_version_info.py project/tests/test_cross_spark_publish.py.
  • bash -n project/scripts/build_spark.sh.
  • Ran Delta SparkVersionsScriptTest metadata/helper checks, including no-network --resolve-source-build coverage.
  • build/sbt -DsparkVersion=4.2 -DsparkCommit=b6bd005ac7549411ec4e7dc944d7a0e19fd56561 -DsparkArtifactVersion=4.2.0-b6bd005ac754-SNAPSHOT "show spark / scalaVersion".
  • Verified changed OSS-facing files contain no internal Maven/proxy references.

murali-db added 2 commits May 29, 2026 09:48
Add a manual workflow and helper script that publish a selected Apache Spark source ref to the runner's local Maven cache under a deterministic commit-qualified version.
Keep the cache producer helper focused on the generic source-ref implementation so the workflow does not retain the old commit-only path.
~/.sbt
~/.ivy2
~/.cache/coursier
key: delta-sbt-cache-spark-source-${{ steps.spark-ref.outputs.spark_version }}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are we restoring here in this step exactly?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This separate source-ref workflow has been removed. The remaining SBT cache restore is the existing Spark matrix cache for ~/.sbt, ~/.ivy2, and ~/.cache/coursier; the Spark Maven artifact restore now lives in the Spark 4.2 matrix lane and restores only ~/.m2/repository/org/apache/spark.

Comment thread build/sbt Resolved
paths-ignore:
- '**.md'
- '**.txt'
workflow_dispatch:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious: would we need a manual workflow dispatch for this test?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed. The consumer test no longer has workflow_dispatch because it now runs through the existing Spark matrix. The only manual workflow left is the producer/cache-warming workflow, where dispatch is the intended entry point.

Comment thread project/CrossSparkVersions.scala Resolved
Comment thread project/CrossSparkVersions.scala Resolved
Avoid deriving the Spark source artifact cache key from runner metadata so producer and consumer workflows share an explicit Ubuntu cache namespace.
Keep the source cache workflow generic by resolving default Spark refs from CrossSparkVersions while still allowing manual source-ref overrides.

Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
@murali-db
murali-db force-pushed the stack/spark-source-cache-consumer branch from 7cdd305 to 5acaac5 Compare June 3, 2026 15:39
@murali-db
murali-db force-pushed the stack/spark-source-cache-consumer branch from 5acaac5 to 2b10780 Compare June 4, 2026 19:26
murali-db added 8 commits June 9, 2026 10:29
Add an opt-in Spark source ref overlay for dependency resolution and a non-blocking PR signal that restores prebuilt Spark artifacts from the cache instead of rebuilding them.
Source-built Spark refs replace sparkVersion.value with a commit-qualified Maven coordinate, so select shim directories from the configured compatibility spec instead of looking up that artifact version in ALL_SPECS.
When a Spark source ref is selected, include Maven local in the Spark-aware resolvers so restored commit-qualified Spark artifacts are visible to SBT.
Allow the source-ref workflow to opt in to Maven local while SBT repository override is enabled, and verify the restored Spark Maven artifacts before invoking SBT.
Only expose Maven local to source-built Spark runs, and make a missing source Spark cache fail once while the remaining Spark 4.2 shards skip until the cache is warmed.
Remove the standalone source Spark workflow so the non-blocking Spark 4.2 lane remains the only cache consumer.

Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Preserve the source-built Spark helper references and shared property/env lookup after restacking the consumer branch.

Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
@murali-db
murali-db force-pushed the stack/spark-source-cache-consumer branch from 2b10780 to 279bfa8 Compare June 9, 2026 17:33
@murali-db

Copy link
Copy Markdown
Collaborator Author

Superseded by #6907. I folded the consumer changes into the main PR and removed the manual warmer model, so normal CI now prepares the source Spark cache automatically before the 4.2 shards run.

@murali-db murali-db closed this Jun 9, 2026
murali-db added a commit that referenced this pull request Jul 15, 2026
## Summary
- Removes the separate manual Spark artifact-cache workflow and makes
`spark_test.yaml` prepare source-built Spark Maven artifacts
automatically on cache miss.
- Changes Delta's Spark 4.2 metadata from `4.2.0-preview5` to
`4.2.0-SNAPSHOT`, matching the current Spark 4.2 development line.
- Splits the Spark test workflow into published-artifact shards for
4.0/4.1 and source-built Spark shards for 4.2, so normal shards do not
wait on Spark source preparation.
- Factors shared test setup into two composite actions
(`prepare-spark-env`, `run-delta-spark-tests`) and moves the Spark Maven
cache key and source-built artifact-version formatting into the version
helper.
- Adds manual workflow-dispatch cache-key invalidation without allowing
workflow-level Spark source ref overrides.
- Supersedes #6908; the useful consumer changes are folded into this PR.

## How this works
- `generate-matrix` emits source-build and non-source-build Spark
version lists from `CrossSparkVersions.scala` metadata.
- `CrossSparkVersions.scala` is the source of truth for the source-built
Spark ref (`sourceBuildDefaultRef`). Workflows do not override it.
- `prepare_spark_source_cache` runs once per source-build Spark version.
It uses `.github/actions/prepare-spark-env` to resolve the Spark SHA +
exact cache key, restore `~/.m2/repository/org/apache/spark`, build
Spark with `build_spark.sh` on cache miss, and save the cache for the
sharded test jobs.
- `test` (published 4.0/4.1 shards) and `source_spark_test`
(source-built 4.2 shards) share two composite actions:
- `.github/actions/prepare-spark-env`: installs the JDK and, for
source-build rows, restores/verifies source-built Spark artifacts. It
builds on miss by default when source Spark is opted in; source test
shards explicitly disable build-on-miss so they restore the cache
instead of rebuilding 8x.
- `.github/actions/run-delta-spark-tests`: restores the SBT cache,
publishes the pinned Unity Catalog build, and runs the sharded
Scala/Java tests with `SPARK_COMMIT` / `SPARK_ARTIFACT_VERSION` from the
prepared env.
- `sparkVersion` remains the compatibility selector (shims, suffixes,
JVM options). `sparkArtifactVersion` is the Maven dependency version
used for Spark artifacts, so source-built Spark can resolve
`4.2.0-<sha12>-SNAPSHOT` without changing the compatibility profile.
- The Spark SHA, source-built Maven artifact version, and exact Maven
cache key are computed in `get_spark_version_info.py`; Scala and shell
code do not reformat the commit-qualified artifact version themselves.
- Maven local is scoped to source-built Spark runs via
`USE_MAVEN_LOCAL_FOR_SOURCE_SPARK=true`. The workflow exposes a
temporary Maven repository containing only the restored source-built
Spark artifacts, so normal published Spark builds and unrelated
dependencies do not pick up stale local artifacts.
- The source-built Spark lane is non-blocking end to end:
prep/build/cache failures and source Spark test failures are surfaced
through the non-blocking source Spark signal, while published Spark
shards remain blocking.
- Manual runs can pass `source-cache-key-suffix` to force a distinct
cache key without changing the source ref.

## Test plan
- Parsed `.github/workflows/spark_test.yaml`,
`.github/workflows/changelog_tests_spark42.yaml`, and
`.github/actions/prepare-spark-env/action.yml` with PyYAML; confirmed
`.github/workflows/spark_commit_artifacts.yaml` is removed.
- `python3.12 -m py_compile project/scripts/get_spark_version_info.py
project/tests/test_cross_spark_publish.py`.
- `bash -n build/sbt project/scripts/build_spark.sh`.
- `build/sbt exportSparkVersionsJson`.
- `python3.12 project/scripts/get_spark_version_info.py --get-field 4.2
fullVersion` returns `"4.2.0-SNAPSHOT"`.
- `python3.12 project/scripts/get_spark_version_info.py
--source-build-spark-versions` returns `["4.2"]` and
`--non-source-build-spark-versions` returns `["4.0", "4.1"]`.
- Ran targeted Delta `SparkVersionsScriptTest` metadata/helper checks,
including no-network `--resolve-source-build` coverage,
`compute_spark_m2_cache_key`, and source-built artifact-version
formatting tests.
- `USE_MAVEN_LOCAL_FOR_SOURCE_SPARK=true build/sbt -DsparkVersion=4.2
-DsparkCommit=b6bd005ac7549411ec4e7dc944d7a0e19fd56561
-DsparkArtifactVersion=4.2.0-b6bd005ac754-SNAPSHOT "show sparkVersion"
"show sparkArtifactVersion"` shows compatibility Spark `4.2.0-SNAPSHOT`
and Maven artifact version `4.2.0-b6bd005ac754-SNAPSHOT`.
- `build/sbt -DsparkVersion=4.2
-DsparkCommit=b6bd005ac7549411ec4e7dc944d7a0e19fd56561 "show
sparkArtifactVersion"` fails with an actionable error requiring explicit
`sparkArtifactVersion`.
- CI validates the published Spark shards and source-built Spark 4.2
shards.

## Known Spark 4.2 Changelog Compatibility Follow-up
- The standalone `Changelog tests on Spark 4.2` check currently compiles
Delta's Spark 4.2 changelog shims against an API where `ChangelogInfo` /
`ChangelogInfoUtils` have changed to `ChangelogContext`. This is a Spark
4.2 shim compatibility update, not a source Spark cache
producer/consumer failure: the source-built Spark matrix shards pass.

---------

Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants