[SourceDbToSpanner] Correcting Binary datatype handling in SourceDbToSpanner template on Spanner write errors

[darshan-sj]

Merged all the Bulk+Live retryDLQ test cases into a single IT.

Also, found and fixed the following bugs:

1. DLQ file overwrite issue was still there since the same dlqTransform with same UUID was used to write the Spanner write errors, transformation errors and filtered events and they were overwriting each others' files. - Fixed this bug.
2. Fixed BYTES and BYTES ARRAY handling for spanner write errored DLQ events.
3. Fixed an issue in GenericRecordTypeConverter for BIT datatype handling when there is custom transformation present in the pipeline.

[gemini-code-assist]

Summary of Changes

Hello @darshan-sj, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the SourceDbToSpanner template by improving the handling of binary data types when errors occur and data is routed to the Dead Letter Queue. Crucially, it also undertakes a significant refactoring and consolidation of integration tests for bulk and live migrations. The updated tests now incorporate more sophisticated failure injection scenarios, allowing for thorough validation of the template's resilience and error recovery capabilities across different stages of data processing, from custom transformations to Spanner writes.

Highlights

• Binary Data Handling in Dead Letter Queue: The DeadLetterQueue logic has been updated to correctly serialize BYTES data types from Spanner Mutation objects into hex-encoded strings when writing to the Dead Letter Queue (DLQ). This ensures that binary data is preserved and correctly represented in the DLQ, preventing potential data loss or corruption.
• Integration Test Consolidation and Refactoring: Multiple Bulk+Live retryDLQ integration test cases have been merged into a single, comprehensive integration test. The test file MySQLAllDataTypesCustomTransformationsBulkAndLiveFT.java was renamed to MySQLAllDataTypesBulkAndLiveIT.java and significantly refactored to cover a broader range of failure injection scenarios.
• Enhanced Failure Injection Scenarios: The refactored integration test now includes more granular failure injection, simulating errors at both the custom transformation phase (for AllDataTypes_CT table) and the Spanner write phase due to schema mismatches (for AllDataTypes_SWF, Authors, and Books tables). This provides more robust validation of the template's error handling and retry mechanisms.
• Selective Custom Transformation Failure: The CustomTransformationAllTypesWithException class was modified to selectively throw exceptions only for the AllDataTypes_CT table. This allows other tables to proceed through the custom transformation successfully, enabling the test to specifically target Spanner write failures for those tables.
• New Test Schemas for Detailed Scenarios: New MySQL and Spanner schema files (mysql-schema.sql and spanner-schema.sql under MySQLAllDataTypesBulkAndLiveIT) were added to support the consolidated test, defining multiple tables (AllDataTypes_CT, AllDataTypes_SWF, Authors, Books) with specific configurations to facilitate the various failure injection tests, including an interleaved table structure.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[codecov]

Codecov Report

❌ Patch coverage is 70.00000% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 54.76%. Comparing base (5de840a) to head (b060d50).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...ogle/cloud/teleport/v2/writer/DeadLetterQueue.java	65.38%	5 Missing and 4 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #3159      +/-   ##
============================================
+ Coverage     50.80%   54.76%   +3.95%     
+ Complexity     5118     2241    -2877     
============================================
  Files           976      490     -486     
  Lines         59996    28389   -31607     
  Branches       6555     2983    -3572     
============================================
- Hits          30482    15546   -14936     
+ Misses        27377    11898   -15479     
+ Partials       2137      945    -1192     

Components	Coverage Δ
spanner-templates	72.15% <70.00%> (+1.14%)	⬆️
spanner-import-export	∅ <ø> (∅)
spanner-live-forward-migration	80.02% <100.00%> (-0.02%)	⬇️
spanner-live-reverse-replication	77.72% <100.00%> (-0.02%)	⬇️
spanner-bulk-migration	88.41% <70.00%> (-0.07%)	⬇️

Files with missing lines	Coverage Δ
...er/migrations/avro/GenericRecordTypeConvertor.java	96.22% <100.00%> (+0.01%)	⬆️
...ogle/cloud/teleport/v2/writer/DeadLetterQueue.java	82.89% <65.38%> (-2.82%)	⬇️

... and 503 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[VardhanThigle]

Overall looks good to me. Please consider adding a case for bytesArray.