Skip to content

[Build] Test source-built Spark automatically in CI - #6907

Merged
murali-db merged 27 commits into
delta-io:masterfrom
murali-db:stack/spark-source-cache
Jul 15, 2026
Merged

murali-db merged 27 commits into
delta-io:masterfrom
murali-db:stack/spark-source-cache

Conversation

@murali-db

@murali-db murali-db commented May 29, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Removes the separate manual Spark artifact-cache workflow and makes spark_test.yaml prepare source-built Spark Maven artifacts automatically on cache miss.
  • Changes Delta's Spark 4.2 metadata from 4.2.0-preview5 to 4.2.0-SNAPSHOT, matching the current Spark 4.2 development line.
  • Splits the Spark test workflow into published-artifact shards for 4.0/4.1 and source-built Spark shards for 4.2, so normal shards do not wait on Spark source preparation.
  • Factors shared test setup into two composite actions (prepare-spark-env, run-delta-spark-tests) and moves the Spark Maven cache key and source-built artifact-version formatting into the version helper.
  • Adds manual workflow-dispatch cache-key invalidation without allowing workflow-level Spark source ref overrides.
  • Supersedes [Build] Test Delta against cached Spark source refs #6908; the useful consumer changes are folded into this PR.

How this works

  • generate-matrix emits source-build and non-source-build Spark version lists from CrossSparkVersions.scala metadata.
  • CrossSparkVersions.scala is the source of truth for the source-built Spark ref (sourceBuildDefaultRef). Workflows do not override it.
  • prepare_spark_source_cache runs once per source-build Spark version. It uses .github/actions/prepare-spark-env to resolve the Spark SHA + exact cache key, restore ~/.m2/repository/org/apache/spark, build Spark with build_spark.sh on cache miss, and save the cache for the sharded test jobs.
  • test (published 4.0/4.1 shards) and source_spark_test (source-built 4.2 shards) share two composite actions:
    • .github/actions/prepare-spark-env: installs the JDK and, for source-build rows, restores/verifies source-built Spark artifacts. It builds on miss by default when source Spark is opted in; source test shards explicitly disable build-on-miss so they restore the cache instead of rebuilding 8x.
    • .github/actions/run-delta-spark-tests: restores the SBT cache, publishes the pinned Unity Catalog build, and runs the sharded Scala/Java tests with SPARK_COMMIT / SPARK_ARTIFACT_VERSION from the prepared env.
  • sparkVersion remains the compatibility selector (shims, suffixes, JVM options). sparkArtifactVersion is the Maven dependency version used for Spark artifacts, so source-built Spark can resolve 4.2.0-<sha12>-SNAPSHOT without changing the compatibility profile.
  • The Spark SHA, source-built Maven artifact version, and exact Maven cache key are computed in get_spark_version_info.py; Scala and shell code do not reformat the commit-qualified artifact version themselves.
  • Maven local is scoped to source-built Spark runs via USE_MAVEN_LOCAL_FOR_SOURCE_SPARK=true. The workflow exposes a temporary Maven repository containing only the restored source-built Spark artifacts, so normal published Spark builds and unrelated dependencies do not pick up stale local artifacts.
  • The source-built Spark lane is non-blocking end to end: prep/build/cache failures and source Spark test failures are surfaced through the non-blocking source Spark signal, while published Spark shards remain blocking.
  • Manual runs can pass source-cache-key-suffix to force a distinct cache key without changing the source ref.

Test plan

  • Parsed .github/workflows/spark_test.yaml, .github/workflows/changelog_tests_spark42.yaml, and .github/actions/prepare-spark-env/action.yml with PyYAML; confirmed .github/workflows/spark_commit_artifacts.yaml is removed.
  • python3.12 -m py_compile project/scripts/get_spark_version_info.py project/tests/test_cross_spark_publish.py.
  • bash -n build/sbt project/scripts/build_spark.sh.
  • build/sbt exportSparkVersionsJson.
  • python3.12 project/scripts/get_spark_version_info.py --get-field 4.2 fullVersion returns "4.2.0-SNAPSHOT".
  • python3.12 project/scripts/get_spark_version_info.py --source-build-spark-versions returns ["4.2"] and --non-source-build-spark-versions returns ["4.0", "4.1"].
  • Ran targeted Delta SparkVersionsScriptTest metadata/helper checks, including no-network --resolve-source-build coverage, compute_spark_m2_cache_key, and source-built artifact-version formatting tests.
  • USE_MAVEN_LOCAL_FOR_SOURCE_SPARK=true build/sbt -DsparkVersion=4.2 -DsparkCommit=b6bd005ac7549411ec4e7dc944d7a0e19fd56561 -DsparkArtifactVersion=4.2.0-b6bd005ac754-SNAPSHOT "show sparkVersion" "show sparkArtifactVersion" shows compatibility Spark 4.2.0-SNAPSHOT and Maven artifact version 4.2.0-b6bd005ac754-SNAPSHOT.
  • build/sbt -DsparkVersion=4.2 -DsparkCommit=b6bd005ac7549411ec4e7dc944d7a0e19fd56561 "show sparkArtifactVersion" fails with an actionable error requiring explicit sparkArtifactVersion.
  • CI validates the published Spark shards and source-built Spark 4.2 shards.

Known Spark 4.2 Changelog Compatibility Follow-up

  • The standalone Changelog tests on Spark 4.2 check currently compiles Delta's Spark 4.2 changelog shims against an API where ChangelogInfo / ChangelogInfoUtils have changed to ChangelogContext. This is a Spark 4.2 shim compatibility update, not a source Spark cache producer/consumer failure: the source-built Spark matrix shards pass.

@seewishnew seewishnew left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment thread project/scripts/build_spark.sh Resolved
Comment thread project/scripts/build_spark.sh Resolved
@murali-db murali-db changed the title [Build] Add Spark source artifact cache workflow [Build] Test source-built Spark automatically in CI Jun 9, 2026
@murali-db
murali-db force-pushed the stack/spark-source-cache branch from 68a924c to f53b4fd Compare June 9, 2026 19:41
@murali-db
murali-db force-pushed the stack/spark-source-cache branch from f0bf5ef to 39366f3 Compare June 22, 2026 15:50
@murali-db
murali-db requested a review from huan233usc as a code owner June 22, 2026 15:50

@huan233usc huan233usc left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Generated] Left one comment about source Spark cache preparation still being blocking.

Comment thread .github/workflows/spark_test.yaml Resolved
@murali-db
murali-db requested a review from huan233usc June 22, 2026 18:24
@huan233usc

Copy link
Copy Markdown
Collaborator

[AI Generated] Can you fix the failing Changelog tests on Spark 4.2 check as well? It still runs ./build/sbt -DsparkVersion=4.2 without the new source-built Spark setup, so CI is compiling the 4.2 changelog shims against Spark artifacts that are missing ChangelogInfo / ChangelogInfoUtils.

@huan233usc huan233usc left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Generated] LGTM. Assuming the remaining changelog check failure is unrelated to this PR, I do not see other blocking issues.

Comment thread .github/workflows/spark_test.yaml Resolved
Comment thread build/sbt Resolved
Comment thread project/scripts/build_spark.sh Resolved

@tdas tdas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

project/CrossSparkVersions.scala:512
[AI-assisted] back-compat · normal

sparkVersionKey now holds getSparkArtifactVersion() (possibly the commit-qualified 4.2.0-<sha>-SNAPSHOT) instead of the compatibility version. does anything downstream read sparkVersionKey expecting fullVersion?

a setting key whose meaning quietly shifts from compat-version to artifact-version is a trap for the next reader who assumes the old contract.

Comment thread project/CrossSparkVersions.scala Resolved
Comment thread build/sbt Resolved
Comment thread project/CrossSparkVersions.scala Resolved
Comment thread project/CrossSparkVersions.scala Resolved
murali-db added 25 commits July 15, 2026 12:11
Keep the source cache workflow generic by resolving default Spark refs from CrossSparkVersions while still allowing manual source-ref overrides.

Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Add an opt-in Spark source ref overlay for dependency resolution and a non-blocking PR signal that restores prebuilt Spark artifacts from the cache instead of rebuilding them.
Source-built Spark refs replace sparkVersion.value with a commit-qualified Maven coordinate, so select shim directories from the configured compatibility spec instead of looking up that artifact version in ALL_SPECS.
When a Spark source ref is selected, include Maven local in the Spark-aware resolvers so restored commit-qualified Spark artifacts are visible to SBT.
Allow the source-ref workflow to opt in to Maven local while SBT repository override is enabled, and verify the restored Spark Maven artifacts before invoking SBT.
Only expose Maven local to source-built Spark runs, and make a missing source Spark cache fail once while the remaining Spark 4.2 shards skip until the cache is warmed.
Remove the standalone source Spark workflow so the non-blocking Spark 4.2 lane remains the only cache consumer.

Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Preserve the source-built Spark helper references and shared property/env lookup after restacking the consumer branch.

Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Move the Spark Maven cache key into get_spark_version_info.py (--resolve-source-build
emits cache_key) and split the duplicated test/source_spark_test/prepare steps into two
composite actions (prepare-spark-env, run-delta-spark-tests). Keeps the generated matrix,
the dedicated prepare-cache build job, and sharding.

Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Make the source Spark cache producer non-blocking and always run the source Spark
test shards after matrix generation. If source checkout/build/cache-save fails,
the failure is reported through the already non-blocking source Spark lane instead
of blocking the workflow before shards run.

Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Make source Spark cache preparation build on miss by default when opted in, add manual dispatch controls for source refs/cache invalidation, simplify Maven local handling and Spark version extraction, and label source-build helper checks as non-blocking lane metadata.

Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Source-built Spark artifacts are restored under ~/.m2/repository, so keep Maven local
in the generated repository config for both normal and MAVEN_PROXY_URL paths.

Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Centralize source-built Spark artifact version formatting in get_spark_version_info.py,
keep sparkVersion as the compatibility version while using sparkArtifactVersion for Maven
dependencies, scope Maven local to source-built runs only, and run changelog Spark 4.2 tests
through the source-built Spark environment.

Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Use the sbt launcher's maven-local resolver name instead of a raw file repository
when source-built Spark rows opt into Maven local. This keeps local Spark artifacts
visible without making unrelated missing local transitive dependencies fatal.

Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Remove workflow Spark ref overrides and the sourceBuildArtifactBaseVersion field so
source-built Spark policy stays in CrossSparkVersions and artifact versions are formatted
only by get_spark_version_info.py. Keep sparkVersion as the compatibility selector while
using sparkArtifactVersion for Spark Maven coordinates.

Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Rely on the project-level source-built Spark resolver instead of adding maven-local to the
sbt launcher repository config. This keeps source-built Spark rows scoped while allowing
normal remote fallback for unrelated transitive dependencies.

Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Expose a temporary Maven repository containing only restored source-built Spark artifacts
instead of the whole local Maven cache. This lets source-built Spark rows resolve their
commit-qualified artifacts without allowing stale unrelated local artifacts to shadow remotes.

Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Keep the V2 changelog Spark 4.2 workflow out of the source-built Spark cache refactor.
Its current failure is a separate Spark 4.2 changelog API compatibility issue, not part
of the source Spark cache plumbing in this PR.

Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
@murali-db
murali-db force-pushed the stack/spark-source-cache branch from 1929adc to 258f509 Compare July 15, 2026 19:12
@murali-db

Copy link
Copy Markdown
Collaborator Author

the red changelog 4.2 check is a separate shim compatibility issue — current spark 4.2 uses ChangelogContext while the existing delta 4.2 changelog shims still reference ChangelogInfo/ChangelogInfoUtils. the source spark cache/test path itself is green; this needs a separate changelog shim update.

@murali-db
murali-db merged commit 879e490 into delta-io:master Jul 15, 2026
42 of 43 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants