test(bigquery): add Storage Write API tests for flaky tables - #6849
alvarowolfx wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces flaky write integration tests for the BigQuery Storage Write API, covering sequential and parallel reconnects as well as initial connection failures. The feedback identifies critical issues with the generated table IDs in flaky.rs; the BigQuery test framework in us-east7 requires exact suffixes (_reconnect_on_close and _initial_connect_failure) to trigger simulated flakiness, meaning the parallel and default test table IDs must be adjusted to place their differentiators before these suffixes.
| let table_id = format!( | ||
| "{}_reconnect_on_close_parallel", | ||
| bigquery_samples::random_id_suffix() | ||
| ); |
There was a problem hiding this comment.
The table ID ends with _reconnect_on_close_parallel. However, the BigQuery Storage Write API test framework in us-east7 triggers simulated connection drops specifically for tables ending with the exact suffix _reconnect_on_close. Because of this, the table name used here will not trigger the simulated flakiness, and the test will run against a normal table.
Please change the table ID format so that the differentiator is placed before the suffix, ensuring it ends with _reconnect_on_close.
| let table_id = format!( | |
| "{}_reconnect_on_close_parallel", | |
| bigquery_samples::random_id_suffix() | |
| ); | |
| let table_id = format!( | |
| "{}_parallel_reconnect_on_close", | |
| bigquery_samples::random_id_suffix() | |
| ); |
| let table_id = format!( | ||
| "{}_initial_connect_failure_parallel", | ||
| bigquery_samples::random_id_suffix() | ||
| ); |
There was a problem hiding this comment.
The table ID ends with _initial_connect_failure_parallel. The BigQuery Storage Write API test framework in us-east7 triggers simulated initial connection failures specifically for tables ending with the exact suffix _initial_connect_failure.
Please adjust the table ID format to place the differentiator before the suffix so that it ends with _initial_connect_failure.
| let table_id = format!( | |
| "{}_initial_connect_failure_parallel", | |
| bigquery_samples::random_id_suffix() | |
| ); | |
| let table_id = format!( | |
| "{}_parallel_initial_connect_failure", | |
| bigquery_samples::random_id_suffix() | |
| ); |
| dataset_id: &str, | ||
| schema: TableSchema, | ||
| ) -> Result<()> { | ||
| let table_id = format!("{}_reconnect_default", bigquery_samples::random_id_suffix()); |
There was a problem hiding this comment.
The table ID ends with _reconnect_default, but the comment and the test's intent indicate it should target a table with the _reconnect_on_close suffix to simulate connection drops.
Please update the table ID format to end with _reconnect_on_close so that the simulated connection drops are properly triggered.
| let table_id = format!("{}_reconnect_default", bigquery_samples::random_id_suffix()); | |
| let table_id = format!("{}_default_reconnect_on_close", bigquery_samples::random_id_suffix()); |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #6849 +/- ##
=======================================
Coverage 97.18% 97.18%
=======================================
Files 328 328
Lines 109962 109962
=======================================
Hits 106867 106867
Misses 3095 3095 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Add integration tests for BigQuery Storage Write API exercising reconnection and retry behaviors on simulated failure tables in
us-east7:_reconnect_on_close: sequential and parallel appends to pending streamsverifying connection recovery on periodic drops (every 10 requests).
_initial_connect_failure: sequential and parallel appends verifyingretry logic during frequent initial stream connect failures.
Towards ???