chore: Add script to copy release artifacts to the same staging repo#18747
Conversation
hudi-agent
left a comment
hudi-agent
left a comment
There was a problem hiding this comment.
🤖 This review was generated by an AI agent and may contain mistakes. Please verify any suggestions before applying.
Thanks for the docs update! The new instructions clearly explain how to use copy_staging_repo.sh to consolidate Java 17 and Java 11 artifacts into a single Nexus staging repo, and the dry-run-first guidance is helpful. A couple of small clarifications around verification ordering and execution context would make the steps even safer for release managers — please defer to a Hudi committer or PMC member for the final call.
| ./scripts/release/copy_staging_repo.sh <java17-repo-id> <java11-repo-id> 2>&1 | tee -a "/tmp/${RELEASE_VERSION}-${RC_NUM}.copy_staging.log" | ||
| ``` | ||
| 4. The script reads Nexus credentials from `~/.m2/settings.xml` (server id `apache.releases.https`), downloads | ||
| every artifact from the source repo, and re-uploads them to the target repo. After it finishes, drop the |
There was a problem hiding this comment.
🤖 It might help to explicitly instruct the release manager to verify that all artifacts have landed in the Java 11 staging repo (e.g., by spot-checking in the Nexus UI or relying on the end-of-run success summary) before dropping the Java 17 staging repo. As written, the step sequences "run the copy" → "drop the Java 17 repo" → "review all staged artifacts" (step 6), so if the copy silently missed something, the source artifacts would already be gone when the review happens.
- AI-generated; verify before applying. React 👍/👎 to flag quality.
| 2. First do a dry-run to verify the list of artifacts to be copied: | ||
| ```shell | ||
| ./scripts/release/copy_staging_repo.sh --dry-run <java17-repo-id> <java11-repo-id> | ||
| ``` |
There was a problem hiding this comment.
🤖 Since both invocations use a relative path (./scripts/release/copy_staging_repo.sh), it could clarify that the commands should be run from the repository root, consistent with the convention used elsewhere in the guide.
- AI-generated; verify before applying. React 👍/👎 to flag quality.
| 5. Note that the artifacts from Java 17 build are uploaded to a separate staging repo. Use the | ||
| `copy_staging_repo.sh` script to copy all artifacts from the Java 17 staging repo into the Java 11 staging repo | ||
| so that all artifacts stay in the same repo. | ||
| 1. Identify both staging repo IDs from [Apache Nexus Staging Repositories](https://repository.apache.org/#stagingRepositories) |
There was a problem hiding this comment.
🤖 Worth confirming: does the script require the target (Java 11) staging repo to be in the "open" state as well? The note says "Make sure both repos are still in the 'open' state (not closed)" — if Nexus rejects uploads to a closed repo, it might be useful to mention what error the user would see, or whether the script surfaces a clear message in that case.
- AI-generated; verify before applying. React 👍/👎 to flag quality.
hudi-agent
left a comment
hudi-agent
left a comment
There was a problem hiding this comment.
🤖 This review was generated by an AI agent and may contain mistakes. Please verify any suggestions before applying.
Thanks for the docs update! The new step clearly walks through using copy_staging_repo.sh with a dry-run first, which is a nice improvement over the manual download/upload flow. Two concerns worth a look: the instruction to drop the Java 17 staging repo immediately after the copy (before the verification steps that follow) is irreversible if the copy was incomplete, and the doc could clarify whether the script also copies the .asc signatures and .md5/.sha1 checksums required for the release vote. Please defer to a Hudi committer or PMC member for final review.
| ``` | ||
| 4. The script reads Nexus credentials from `~/.m2/settings.xml` (server id `apache.releases.https`), downloads | ||
| every artifact from the source repo, and re-uploads them to the target repo. After it finishes, drop the | ||
| Java 17 staging repo on Apache Nexus. |
There was a problem hiding this comment.
🤖 Dropping the Java 17 staging repo right after the copy script finishes is irreversible — if any artifact failed to upload (or if signatures/checksums are missing), the original source is gone. This step could direct the release manager to first complete the verification in step 6 (confirming the Java 11 repo contains all expected artifacts, including .asc/.md5/.sha1 files for each jar/pom) and only then drop the Java 17 repo.
- AI-generated; verify before applying. React 👍/👎 to flag quality.
| 3. Then run the actual copy: | ||
| ```shell | ||
| ./scripts/release/copy_staging_repo.sh <java17-repo-id> <java11-repo-id> 2>&1 | tee -a "/tmp/${RELEASE_VERSION}-${RC_NUM}.copy_staging.log" | ||
| ``` |
There was a problem hiding this comment.
🤖 It might help to clarify what the script considers an "artifact" — specifically, does it also copy the accompanying .asc signature files and .md5/.sha1 checksum files that Apache releases require? If not, the resulting Java 11 staging repo would fail signature verification during the vote. Worth calling out explicitly in the doc (or confirming the script handles them).
- AI-generated; verify before applying. React 👍/👎 to flag quality.
yihua
force-pushed
the
add-script-staging-repo-merging
branch
from
fea7a23 to
da5a87e
Compare
yihua
force-pushed
the
add-script-staging-repo-merging
branch
from
da5a87e to
0cb543e
Compare
yihua
force-pushed
the
add-script-staging-repo-merging
branch
from
0cb543e to
d4a4a52
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #18747 +/- ##
============================================
- Coverage 54.01% 54.00% -0.01%
+ Complexity 12461 12457 -4
============================================
Files 1434 1434
Lines 72161 72161
Branches 8245 8245
============================================
- Hits 38979 38974 -5
- Misses 29686 29690 +4
- Partials 3496 3497 +1
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
5 participants
Describe the issue this Pull Request addresses
During an RC release, jars built with Java 17 (currently the Spark 4 bundle) are deployed to a separate Apache Nexus staging repo from the Java 11 build. To produce a single, consistent staging repo for the vote, the release manager today has to manually download every Java 17 artifact and re-upload it into the Java 11 staging repo, which is tedious and error-prone.
Summary and Changelog
scripts/release/copy_staging_repo.shthat, given a source and target Nexus staging repo ID, crawls the source repo via the Nexus content API, downloads every artifact, and re-uploads it to the target repo using credentials from~/.m2/settings.xml(server idapache.releases.https).--dry-runto list artifacts without downloading or uploading.release/release_guide.mdstep "Deploy maven artifacts and verify" to reference the new script in place of the manual merge instructions.Impact
Release-only tooling and documentation. No runtime or user-facing change.
Risk Level
none
Documentation Update
Updated
release/release_guide.mdto document the script.Contributor's checklist