[DO NOT MERGE] Validate Beam 2.71.0rc1

[damccorm]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello @damccorm, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request is dedicated to validating the upcoming Apache Beam 2.71.0 release candidate. It involves a comprehensive update of both Java and Python dependencies to align with the new Beam version, ensuring compatibility and leveraging the latest features and fixes. The changes span across various project modules, primarily focusing on dependency version bumps and the introduction of new transitive dependencies in Python environments.

Highlights

• Apache Beam Version Update: The core Apache Beam and Beam Python versions have been updated from 2.70.0 to 2.71.0 in the pom.xml configuration.
• Maven Repository Configuration: A new Maven repository URL (https://repository.apache.org/content/repositories/orgapachebeam-1420) has been added to pom.xml to fetch the updated Beam artifacts.
• Python Dependency Pinning: All relevant Python requirements.txt files across various modules (e.g., default_base_python_requirements.txt, default_base_yaml_requirements.txt, job-builder-server, streaming-llm, word-count-python, yaml-template, and several v2 templates) now explicitly pin apache-beam to version 2.71.0rc1.
• Python Dependency Updates: Several Python dependencies have been updated to newer versions, including beartype (to 0.22.9), google-cloud-bigquery (to 3.40.0), google-genai (to 1.57.0), jsonschema (to 4.26.0), numpy (to 2.4.0), pymongo (to 4.16.0), testcontainers (to 4.14.0), and urllib3 (to 2.6.3).
• New Python Dependencies Introduced: New Python packages such as betterproto, envoy-data-plane, grpclib, h2, hpack, and hyperframe have been added to the dependency lists in various requirements.txt files, often as transitive dependencies of apache-beam or other updated packages.
• Build Command Adjustments: Comments within some requirements.txt files (job-builder-server, word-count-python, yaml-template, and v2 templates) have been updated to reflect changes in pip-compile command syntax, specifically removing the ./ prefix from output file paths.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 50.87%. Comparing base (7be9f75) to head (dae581c).

Additional details and impacted files

@@            Coverage Diff             @@
##               main    #3204    +/-   ##
==========================================
  Coverage     50.86%   50.87%            
- Complexity     5133     5513   +380     
==========================================
  Files           976      976            
  Lines         60090    60090            
  Branches       6572     6572            
==========================================
+ Hits          30564    30569     +5     
+ Misses        27374    27372     -2     
+ Partials       2152     2149     -3     

Components	Coverage Δ
spanner-templates	71.02% <ø> (+<0.01%)	⬆️
spanner-import-export	68.99% <ø> (ø)
spanner-live-forward-migration	80.12% <ø> (+0.01%)	⬆️
spanner-live-reverse-replication	77.77% <ø> (+0.01%)	⬆️
spanner-bulk-migration	88.44% <ø> (+0.01%)	⬆️
see 2 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[damccorm]

Currently what I am seeing is that all tests are passing except some spanner related tests. These are failing with:

Timeout in polling result file: gs://dataflow-staging-us-west2-269744978479/staging/template_launches/2026-01-08_12_48_39-9625519560852535969/operation_result.
Service account: 269744978479-compute@developer.gserviceaccount.com
Image URL: gcr.io/cloud-teleport-testing/spanner-changestreams-to-bigquery:2026-01-08-20-42-15_IT
Troubleshooting guide at https://cloud.google.com/dataflow/docs/guides/troubleshoot-templates#timeout-polling

Example: https://pantheon.corp.google.com/dataflow/jobs/us-west2/2026-01-08_12_48_39-9625519560852535969;logsSeverity=ERROR;graphView=1?project=cloud-teleport-testing&pageState=(%22dfTime%22:(%22l%22:%22dfJobMaxTime%22))&e=13802955&mods=dm_deploy_from_gcs

I'm not currently seeing anything in the logs that gives much of a hint about what is happening. I have confirmed that if I remove SpannerIO.ReadChangeStream from the pipeline, that allows it to succeed, so there is something odd going on there. There are no obvious changes in https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner

@Abacn have you seen anything like this while doing this upgrade before? If not, I will keep looking

[Abacn]

In the past there were situations that spanner tests due to breaking changes in Spanner client that get upgraded as GCP BOM upgrade happened in Beam.

"Timeout in polling result file: gs://dataflow-staging-us-west2-269744978479/staging/template_launches/2026-01-08_12_48_39-9625519560852535969/operation_result."

This sounds template launcher failed.

if I remove SpannerIO.ReadChangeStream from the pipeline

Sounds like pipeline expansion gets stuck. if we can reproduce it locally then could get a hint. Also reach out to Spanner CDC team to investigate.

[damccorm]

In the past there were situations that spanner tests due to breaking changes in Spanner client that get upgraded as GCP BOM upgrade happened in Beam.

Thanks, this is good context - was already starting on the other steps here, thanks!