Add yaml blueprints publishing to GCS #3056

derrickaw · 2025-12-04T21:19:20Z

Piggybacking the release process to include publishing the yaml blueprints to a GCS bucket for further processing later on.
Will set the publishYamlBlueprints parameter to true in the next PR.
mvn -pl plugins/templates-maven-plugin -Dtest=TemplatesReleaseMojoTest test

gemini-code-assist · 2025-12-04T21:19:39Z

Summary of Changes

Hello @derrickaw, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request extends the TemplatesReleaseMojo Maven plugin to include the capability of publishing YAML-based blueprints to Google Cloud Storage. This enhancement provides a unified release mechanism for both traditional Dataflow templates and new YAML blueprints, controlled by a dedicated configuration parameter. The changes involve adding necessary GCS dependencies, implementing the upload logic, and ensuring proper validation and testing of the new feature.

Highlights

YAML Blueprint Publishing: The Maven plugin now supports publishing YAML blueprints to Google Cloud Storage, alongside traditional Dataflow templates.
New Configuration Parameter: A new Maven parameter, publishYamlBlueprints, has been introduced to enable or disable this new publishing functionality.
Google Cloud Storage Integration: Added google-cloud-storage dependency and integrated GCS client logic to handle the upload of YAML files.
Unit Testing for YAML Publishing: A new unit test has been added to verify the correct behavior of the YAML blueprint publishing process, including mocking GCS interactions.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

codecov · 2025-12-04T21:33:53Z

Codecov Report

❌ Patch coverage is 50.98039% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 50.86%. Comparing base (9e360c9) to head (47d7f85).
⚠️ Report is 7 commits behind head on main.

Files with missing lines	Patch %	Lines
...ud/teleport/plugin/maven/TemplatesReleaseMojo.java	50.98%	19 Missing and 6 partials ⚠️

❌ Your patch check has failed because the patch coverage (50.98%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #3056      +/-   ##
============================================
+ Coverage     50.79%   50.86%   +0.06%     
- Complexity     5116     5125       +9     
============================================
  Files           976      976              
  Lines         59994    60031      +37     
  Branches       6555     6559       +4     
============================================
+ Hits          30476    30533      +57     
+ Misses        27380    27348      -32     
- Partials       2138     2150      +12

Components	Coverage Δ
spanner-templates	`71.00% <ø> (+<0.01%)`	⬆️
spanner-import-export	`68.98% <ø> (-0.02%)`	⬇️
spanner-live-forward-migration	`80.03% <ø> (ø)`
spanner-live-reverse-replication	`77.74% <ø> (+0.04%)`	⬆️
spanner-bulk-migration	`88.48% <ø> (ø)`

Files with missing lines	Coverage Δ
...ud/teleport/plugin/maven/TemplatesReleaseMojo.java	`49.36% <50.98%> (+49.36%)`	⬆️

... and 4 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

derrickaw · 2025-12-05T16:18:23Z

/gemini review

gemini-code-assist

Code Review

This pull request adds the capability to publish YAML blueprints to Google Cloud Storage during the release process. The changes are logical and well-contained, introducing a new Mojo parameter to control this feature and the corresponding implementation to handle the file uploads. New tests are also included to validate the happy path of this new functionality. My review focuses on improving the robustness, efficiency, and configurability of the new upload logic. I've identified a few areas for improvement, including a logic error that prevents graceful skipping, a potential resource leak, inefficient file handling, and the use of hardcoded paths. I've also provided suggestions to enhance the test coverage.

...-maven-plugin/src/main/java/com/google/cloud/teleport/plugin/maven/TemplatesReleaseMojo.java

...en-plugin/src/test/java/com/google/cloud/teleport/plugin/maven/TemplatesReleaseMojoTest.java

derrickaw · 2025-12-05T19:54:41Z

/gemini review

gemini-code-assist

Code Review

This pull request enhances the TemplatesReleaseMojo Maven plugin by adding functionality to publish YAML blueprints to Google Cloud Storage. The changes include introducing new Maven parameters (publishYamlBlueprints, yamlBlueprintsPath, yamlBlueprintsGCSBucket), adding google-cloud-storage and mockito-inline dependencies, and implementing logic to read YAML files from a local path and upload them to a specified GCS bucket. The existing template staging process was refactored to integrate with this new feature. A new test file, TemplatesReleaseMojoTest, was added to verify the YAML blueprint upload mechanism, utilizing mocked GCS services. A review comment suggests improving the test by using java.nio.charset.StandardCharsets.UTF_8 instead of the string literal "UTF-8" when converting bytes to a string, to avoid UnsupportedEncodingException and increase code robustness.

...en-plugin/src/test/java/com/google/cloud/teleport/plugin/maven/TemplatesReleaseMojoTest.java

derrickaw · 2025-12-05T20:00:05Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces the capability to publish YAML blueprints to Google Cloud Storage as part of the release process. The changes include adding the necessary GCS dependency, implementing the upload logic in TemplatesReleaseMojo, and adding comprehensive unit tests for the new functionality. The implementation is solid, but I've identified a potential resource leak with the Storage client object which should be addressed. I've provided a suggestion to fix this using a try-with-resources statement, which also simplifies the code structure.

...-maven-plugin/src/main/java/com/google/cloud/teleport/plugin/maven/TemplatesReleaseMojo.java

derrickaw · 2025-12-05T20:07:31Z

/gemini review

gemini-code-assist

Code Review

This pull request adds a new capability to the templates-maven-plugin to publish YAML blueprints to Google Cloud Storage during the release process. The changes include adding the google-cloud-storage dependency, introducing new parameters to control the publishing behavior, and implementing the upload logic in TemplatesReleaseMojo. Commendably, comprehensive unit tests have been added in TemplatesReleaseMojoTest to validate the new functionality, covering success cases, and scenarios where uploads should be skipped.

The implementation is solid, but I have a couple of suggestions to enhance robustness and code clarity:

In TemplatesReleaseMojo.java, the file filtering logic for YAML blueprints can be made more robust to correctly handle directories and be platform-independent.
In TemplatesReleaseMojoTest.java, the test code can be made more idiomatic by using AtomicReference for capturing state within a lambda expression.

Overall, this is a great addition. Please see my detailed comments.

...-maven-plugin/src/main/java/com/google/cloud/teleport/plugin/maven/TemplatesReleaseMojo.java

...en-plugin/src/test/java/com/google/cloud/teleport/plugin/maven/TemplatesReleaseMojoTest.java

derrickaw · 2025-12-05T20:22:14Z

/gemini review

gemini-code-assist

Code Review

This pull request adds the functionality to publish YAML blueprints to Google Cloud Storage as part of the release process. The changes include adding the google-cloud-storage dependency, introducing new Maven parameters to control the publishing, and implementing the upload logic in TemplatesReleaseMojo. A comprehensive set of unit tests has been added in TemplatesReleaseMojoTest to verify the new functionality, including success and failure scenarios. The implementation looks solid. I have one minor suggestion in the test code to improve its robustness across different environments.

...en-plugin/src/test/java/com/google/cloud/teleport/plugin/maven/TemplatesReleaseMojoTest.java

derrickaw · 2025-12-05T20:25:58Z

/gemini review

gemini-code-assist

Code Review

This pull request adds a feature to publish YAML blueprints to GCS as part of the templates-maven-plugin. The changes include adding necessary dependencies, new configuration parameters, and the implementation logic for uploading files. New tests are also added to cover the new functionality. The implementation looks solid and is well-tested. I've added a couple of minor suggestions to improve code style by using imports instead of fully qualified class names for better readability.

...-maven-plugin/src/main/java/com/google/cloud/teleport/plugin/maven/TemplatesReleaseMojo.java

...en-plugin/src/test/java/com/google/cloud/teleport/plugin/maven/TemplatesReleaseMojoTest.java

derrickaw · 2025-12-05T20:40:49Z

/gemini review

gemini-code-assist

Code Review

This pull request adds functionality to publish YAML blueprints to Google Cloud Storage as part of the release process. The changes include adding necessary dependencies, implementing the publishing logic in TemplatesReleaseMojo, and adding comprehensive unit tests for the new feature. The implementation is solid, and the tests cover success and failure cases well. I have one suggestion to improve the exception handling in the new publishing logic for better maintainability.

...-maven-plugin/src/main/java/com/google/cloud/teleport/plugin/maven/TemplatesReleaseMojo.java

chamikaramj

Thanks!

chamikaramj · 2025-12-17T04:18:04Z

...-maven-plugin/src/main/java/com/google/cloud/teleport/plugin/maven/TemplatesReleaseMojo.java

+
+  @Parameter(
+      defaultValue = "yaml-blueprints",
+      property = "yamlBlueprintsGCSBucket",


Are we always uploading to the top level of the GCS bucket ? May be this should be a GCS path ?

I believe we are. We have the standard: gs://bucketName/stagePrefix/yamlBlueprintsGCSBucket/fileName

So you would rather give a path at that point, so something like this that could be placed anymore under stagePrefix?
gs://bucketName/stagePrefix/yamlBlueprintsGCSBucketPath/fileName

Thanks

Went ahead with the GCSPath name, which is what I think you were alluding to. Thanks.

chamikaramj · 2025-12-17T04:20:16Z

...-maven-plugin/src/main/java/com/google/cloud/teleport/plugin/maven/TemplatesReleaseMojo.java

+          }
        }
+      } else {
+        LOG.warn("Not found templates to release in this module.");


Did not find ...

done, thanks

chamikaramj · 2025-12-17T04:21:21Z

...-maven-plugin/src/main/java/com/google/cloud/teleport/plugin/maven/TemplatesReleaseMojo.java

-          if (definition.isFlex()) {
-            generator.saveImageSpec(definition, imageSpec, targetDirectory);
+      if (publishYamlBlueprints) {
+        LOG.info("Trying to upload YAML blueprints to bucket '{}'...", bucketNameOnly(bucketName));


Job builder blueprints

done, thanks

chamikaramj · 2025-12-17T18:55:02Z

...-maven-plugin/src/main/java/com/google/cloud/teleport/plugin/maven/TemplatesReleaseMojo.java

+        LOG.info("Trying to upload YAML blueprints to bucket '{}'...", bucketNameOnly(bucketName));
+        Path yamlPath = Paths.get(project.getBasedir().getAbsolutePath(), yamlBlueprintsPath);
+        if (!Files.exists(yamlPath) || !Files.isDirectory(yamlPath)) {
+          LOG.warn("YAML blueprints directory not found, skipping upload: {}", yamlPath);


Just fail the script if no files are found ?

Valid point, done, thanks

chamikaramj · 2025-12-17T19:01:07Z

...-maven-plugin/src/main/java/com/google/cloud/teleport/plugin/maven/TemplatesReleaseMojo.java

+                    path ->
+                        Files.isRegularFile(path)
+                            && path.getFileName().toString().endsWith(".yaml"))
+                .forEach(


So to clarify, the YAML files under "yaml/src/main/yaml" uploaded to GCS here are directly consumable by Job builder without any transformations ?

The intention is to make the yaml format the standard, which may require some modifications on JB, but not the yaml files themselves.

derrickaw · 2025-12-30T03:24:16Z

Hi @damccorm, if you prefer to wait on Cham's final approval that is cool. Thanks.

damccorm · 2025-12-30T16:25:06Z

Yeah, I'll let Cham finish the review here since it is already in progress. Thanks

pull-request-size bot added the size/L label Dec 4, 2025

derrickaw added addition New feature or request improvement labels Dec 4, 2025

gemini-code-assist bot reviewed Dec 5, 2025

View reviewed changes

...en-plugin/src/test/java/com/google/cloud/teleport/plugin/maven/TemplatesReleaseMojoTest.java Outdated Show resolved Hide resolved

gemini-code-assist bot reviewed Dec 5, 2025

View reviewed changes

...-maven-plugin/src/main/java/com/google/cloud/teleport/plugin/maven/TemplatesReleaseMojo.java Outdated Show resolved Hide resolved

gemini-code-assist bot reviewed Dec 5, 2025

View reviewed changes

...-maven-plugin/src/main/java/com/google/cloud/teleport/plugin/maven/TemplatesReleaseMojo.java Outdated Show resolved Hide resolved

...en-plugin/src/test/java/com/google/cloud/teleport/plugin/maven/TemplatesReleaseMojoTest.java Outdated Show resolved Hide resolved

gemini-code-assist bot reviewed Dec 5, 2025

View reviewed changes

...en-plugin/src/test/java/com/google/cloud/teleport/plugin/maven/TemplatesReleaseMojoTest.java Outdated Show resolved Hide resolved

gemini-code-assist bot reviewed Dec 5, 2025

View reviewed changes

...-maven-plugin/src/main/java/com/google/cloud/teleport/plugin/maven/TemplatesReleaseMojo.java Outdated Show resolved Hide resolved

...en-plugin/src/test/java/com/google/cloud/teleport/plugin/maven/TemplatesReleaseMojoTest.java Show resolved Hide resolved

gemini-code-assist bot reviewed Dec 5, 2025

View reviewed changes

...-maven-plugin/src/main/java/com/google/cloud/teleport/plugin/maven/TemplatesReleaseMojo.java Show resolved Hide resolved

derrickaw marked this pull request as ready for review December 8, 2025 18:38

derrickaw requested a review from chamikaramj December 9, 2025 17:28

chamikaramj reviewed Dec 17, 2025

View reviewed changes

anowardear062-svg approved these changes Dec 29, 2025

View reviewed changes

derrickaw requested a review from chamikaramj December 29, 2025 18:04

derrickaw added 6 commits December 29, 2025 18:35

add dependencies needed

838b8cd

add support for pushing to GCS

fe78214

add test for publishing yaml to GCS

45ca162

fix spotless issue

4088e23

update based on gemini comments

8982e87

spotless

e9a5314

derrickaw added 8 commits December 29, 2025 18:35

change charset

b20d024

fix gemini comment

fc6ce7c

fix more gemini comments

3479f73

test refactoring

c7f9822

more comment revisions

74adc26

minor fixes

4983b41

fix minor comments

a6077e0

fix spotless

a776aa9

derrickaw force-pushed the addYamlBlueprintsToMojo branch from 1a00610 to a776aa9 Compare December 29, 2025 18:36

derrickaw added 2 commits December 29, 2025 19:58

fix unit test

bb6f735

switch to gcs path

47d7f85

derrickaw requested a review from damccorm December 30, 2025 03:23

Add yaml blueprints publishing to GCS #3056

Are you sure you want to change the base?

Add yaml blueprints publishing to GCS #3056

Conversation

derrickaw commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist bot commented Dec 4, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

codecov bot commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

derrickaw commented Dec 5, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

derrickaw commented Dec 5, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

derrickaw commented Dec 5, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

derrickaw commented Dec 5, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

derrickaw commented Dec 5, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

derrickaw commented Dec 5, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

derrickaw commented Dec 5, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

chamikaramj left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

derrickaw Dec 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

derrickaw commented Dec 4, 2025 •

edited

Loading

codecov bot commented Dec 4, 2025 •

edited

Loading

derrickaw Dec 29, 2025 •

edited

Loading

derrickaw Dec 29, 2025 •

edited

Loading