Repository navigation
[Build] Test source-built Spark automatically in CI - #6907
Conversation
68a924c to
f53b4fd
Compare
f0bf5ef to
39366f3
Compare
huan233usc
left a comment
There was a problem hiding this comment.
[AI Generated] Left one comment about source Spark cache preparation still being blocking.
|
[AI Generated] Can you fix the failing |
huan233usc
left a comment
There was a problem hiding this comment.
[AI Generated] LGTM. Assuming the remaining changelog check failure is unrelated to this PR, I do not see other blocking issues.
tdas
left a comment
There was a problem hiding this comment.
project/CrossSparkVersions.scala:512
[AI-assisted] back-compat · normal
sparkVersionKey now holds getSparkArtifactVersion() (possibly the commit-qualified 4.2.0-<sha>-SNAPSHOT) instead of the compatibility version. does anything downstream read sparkVersionKey expecting fullVersion?
a setting key whose meaning quietly shifts from compat-version to artifact-version is a trap for the next reader who assumes the old contract.
Keep the source cache workflow generic by resolving default Spark refs from CrossSparkVersions while still allowing manual source-ref overrides. Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Add an opt-in Spark source ref overlay for dependency resolution and a non-blocking PR signal that restores prebuilt Spark artifacts from the cache instead of rebuilding them.
Source-built Spark refs replace sparkVersion.value with a commit-qualified Maven coordinate, so select shim directories from the configured compatibility spec instead of looking up that artifact version in ALL_SPECS.
When a Spark source ref is selected, include Maven local in the Spark-aware resolvers so restored commit-qualified Spark artifacts are visible to SBT.
Allow the source-ref workflow to opt in to Maven local while SBT repository override is enabled, and verify the restored Spark Maven artifacts before invoking SBT.
Only expose Maven local to source-built Spark runs, and make a missing source Spark cache fail once while the remaining Spark 4.2 shards skip until the cache is warmed.
Remove the standalone source Spark workflow so the non-blocking Spark 4.2 lane remains the only cache consumer. Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Preserve the source-built Spark helper references and shared property/env lookup after restacking the consumer branch. Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Move the Spark Maven cache key into get_spark_version_info.py (--resolve-source-build emits cache_key) and split the duplicated test/source_spark_test/prepare steps into two composite actions (prepare-spark-env, run-delta-spark-tests). Keeps the generated matrix, the dedicated prepare-cache build job, and sharding. Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Make the source Spark cache producer non-blocking and always run the source Spark test shards after matrix generation. If source checkout/build/cache-save fails, the failure is reported through the already non-blocking source Spark lane instead of blocking the workflow before shards run. Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Make source Spark cache preparation build on miss by default when opted in, add manual dispatch controls for source refs/cache invalidation, simplify Maven local handling and Spark version extraction, and label source-build helper checks as non-blocking lane metadata. Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Source-built Spark artifacts are restored under ~/.m2/repository, so keep Maven local in the generated repository config for both normal and MAVEN_PROXY_URL paths. Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Centralize source-built Spark artifact version formatting in get_spark_version_info.py, keep sparkVersion as the compatibility version while using sparkArtifactVersion for Maven dependencies, scope Maven local to source-built runs only, and run changelog Spark 4.2 tests through the source-built Spark environment. Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Use the sbt launcher's maven-local resolver name instead of a raw file repository when source-built Spark rows opt into Maven local. This keeps local Spark artifacts visible without making unrelated missing local transitive dependencies fatal. Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Remove workflow Spark ref overrides and the sourceBuildArtifactBaseVersion field so source-built Spark policy stays in CrossSparkVersions and artifact versions are formatted only by get_spark_version_info.py. Keep sparkVersion as the compatibility selector while using sparkArtifactVersion for Spark Maven coordinates. Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Rely on the project-level source-built Spark resolver instead of adding maven-local to the sbt launcher repository config. This keeps source-built Spark rows scoped while allowing normal remote fallback for unrelated transitive dependencies. Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Expose a temporary Maven repository containing only restored source-built Spark artifacts instead of the whole local Maven cache. This lets source-built Spark rows resolve their commit-qualified artifacts without allowing stale unrelated local artifacts to shadow remotes. Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Keep the V2 changelog Spark 4.2 workflow out of the source-built Spark cache refactor. Its current failure is a separate Spark 4.2 changelog API compatibility issue, not part of the source Spark cache plumbing in this PR. Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
1929adc to
258f509
Compare
|
the red changelog 4.2 check is a separate shim compatibility issue — current spark 4.2 uses ChangelogContext while the existing delta 4.2 changelog shims still reference ChangelogInfo/ChangelogInfoUtils. the source spark cache/test path itself is green; this needs a separate changelog shim update. |
Summary
spark_test.yamlprepare source-built Spark Maven artifacts automatically on cache miss.4.2.0-preview5to4.2.0-SNAPSHOT, matching the current Spark 4.2 development line.prepare-spark-env,run-delta-spark-tests) and moves the Spark Maven cache key and source-built artifact-version formatting into the version helper.How this works
generate-matrixemits source-build and non-source-build Spark version lists fromCrossSparkVersions.scalametadata.CrossSparkVersions.scalais the source of truth for the source-built Spark ref (sourceBuildDefaultRef). Workflows do not override it.prepare_spark_source_cacheruns once per source-build Spark version. It uses.github/actions/prepare-spark-envto resolve the Spark SHA + exact cache key, restore~/.m2/repository/org/apache/spark, build Spark withbuild_spark.shon cache miss, and save the cache for the sharded test jobs.test(published 4.0/4.1 shards) andsource_spark_test(source-built 4.2 shards) share two composite actions:.github/actions/prepare-spark-env: installs the JDK and, for source-build rows, restores/verifies source-built Spark artifacts. It builds on miss by default when source Spark is opted in; source test shards explicitly disable build-on-miss so they restore the cache instead of rebuilding 8x..github/actions/run-delta-spark-tests: restores the SBT cache, publishes the pinned Unity Catalog build, and runs the sharded Scala/Java tests withSPARK_COMMIT/SPARK_ARTIFACT_VERSIONfrom the prepared env.sparkVersionremains the compatibility selector (shims, suffixes, JVM options).sparkArtifactVersionis the Maven dependency version used for Spark artifacts, so source-built Spark can resolve4.2.0-<sha12>-SNAPSHOTwithout changing the compatibility profile.get_spark_version_info.py; Scala and shell code do not reformat the commit-qualified artifact version themselves.USE_MAVEN_LOCAL_FOR_SOURCE_SPARK=true. The workflow exposes a temporary Maven repository containing only the restored source-built Spark artifacts, so normal published Spark builds and unrelated dependencies do not pick up stale local artifacts.source-cache-key-suffixto force a distinct cache key without changing the source ref.Test plan
.github/workflows/spark_test.yaml,.github/workflows/changelog_tests_spark42.yaml, and.github/actions/prepare-spark-env/action.ymlwith PyYAML; confirmed.github/workflows/spark_commit_artifacts.yamlis removed.python3.12 -m py_compile project/scripts/get_spark_version_info.py project/tests/test_cross_spark_publish.py.bash -n build/sbt project/scripts/build_spark.sh.build/sbt exportSparkVersionsJson.python3.12 project/scripts/get_spark_version_info.py --get-field 4.2 fullVersionreturns"4.2.0-SNAPSHOT".python3.12 project/scripts/get_spark_version_info.py --source-build-spark-versionsreturns["4.2"]and--non-source-build-spark-versionsreturns["4.0", "4.1"].SparkVersionsScriptTestmetadata/helper checks, including no-network--resolve-source-buildcoverage,compute_spark_m2_cache_key, and source-built artifact-version formatting tests.USE_MAVEN_LOCAL_FOR_SOURCE_SPARK=true build/sbt -DsparkVersion=4.2 -DsparkCommit=b6bd005ac7549411ec4e7dc944d7a0e19fd56561 -DsparkArtifactVersion=4.2.0-b6bd005ac754-SNAPSHOT "show sparkVersion" "show sparkArtifactVersion"shows compatibility Spark4.2.0-SNAPSHOTand Maven artifact version4.2.0-b6bd005ac754-SNAPSHOT.build/sbt -DsparkVersion=4.2 -DsparkCommit=b6bd005ac7549411ec4e7dc944d7a0e19fd56561 "show sparkArtifactVersion"fails with an actionable error requiring explicitsparkArtifactVersion.Known Spark 4.2 Changelog Compatibility Follow-up
Changelog tests on Spark 4.2check currently compiles Delta's Spark 4.2 changelog shims against an API whereChangelogInfo/ChangelogInfoUtilshave changed toChangelogContext. This is a Spark 4.2 shim compatibility update, not a source Spark cache producer/consumer failure: the source-built Spark matrix shards pass.