Dev/v0.5.0 resilience - #1
Merged
Merged
Conversation
…eshold - Add failover_cooldown_secs (default: 30) — hold failover active for N seconds after trigger before allowing recovery - Add failover_min_recovery_checks (default: 3) — require N consecutive successful health checks before clearing failover - Detect flap events when failover re-triggers within cooldown*2 window - New Prometheus metric: turbineproxy_ha_failover_flap_total (counter) - Log [HA] cooldown active when recovery checks pass but timer hasn't - Log [HA] Failover FLAP detected on re-trigger within window - Record failover_triggered_at timestamp and recovery_checks in BackendPool - Apply same logic to both MySQL HealthChecker and PG PgHealthChecker - PG trigger_failover now also increments failover_events_total Docs: - Update ha-failover.md with Flapping Protection section + alert rule - Update configuration/reference.md with new fields in [ha] and [pgsql] - Update failure-modes.md recovery description with cooldown semantics - Update README feature table (14 metric families, flap protection) - Update turbineproxy.example.toml with commented new options
- New lock-free circuit breaker module (src/proxy/circuit_breaker.rs) using atomics for hot-path performance - States: Closed → Open (after N errors) → HalfOpen (probe after recovery_ms) - Config: circuit_breaker_threshold (default 5), circuit_breaker_recovery_ms (default 10000) - Integrated into BackendPool (replica_breakers + primary_breaker) - Replica routing filters out Open backends in try_weighted_replica() - Router records success/failure on both read and write paths - Prometheus gauge: turbineproxy_circuit_breaker_state per-backend - Log prefix [CB] for all state transitions - Updated docs: ha-failover.md, failure-modes.md, example TOML - Updated roadmap_v0.5.0.md — all CB items done
- New pool_wait_queue_size config (default 0 = reject-fast, legacy) - New pool_wait_timeout_ms config (default 5000ms) - When max_connections is hit and queue is enabled, requests wait on a tokio::sync::Semaphore instead of being rejected immediately - Slot released back on connection return (put_for_database) - Prometheus: turbineproxy_pool_wait_queue_length (gauge), turbineproxy_pool_wait_timeouts_total (counter) - Log: [pool] wait queue timeout for backend ... (WARN) - Config available for both MySQL and PostgreSQL listeners - Updated failure-modes.md, example TOML, roadmap_v0.5.0.md
- Replace std Mutex/RwLock with parking_lot (no lock poisoning) - Add per-backend circuit breaker (Closed/Open/Half-Open) - Add bounded connection wait queue with timeout - Add failover flap protection (cooldown + min recovery checks) - Add connection multiplexing 1:N (Phase A) - Add PostgreSQL HA parity (health, lag, failover, circuit breaker) - Add SCRAM-SHA-256 hardening + fuzz corpus (15 invariants) - Add POST /api/auth/refresh endpoint (atomic token rotation) - Add POST /api/auth/logout with readonly support - Add turbineproxy_dashboard_auth_failures_total Prometheus counter - Add auth failure tracking in login, middleware, and refresh - Add panic recovery unit tests for parking_lot Mutex and RwLock - Add PG TLS integration test (auto-skip without psql/infra) - Add clippy::unwrap_used CI lint for dashboard and analytics paths - Add chaos testing suite and chaos CI workflow - Bump version to 0.5.0 - Update README, CHANGELOG, and docs
set TURBINEPROXY_SKIP_DASHBOARD_BUILD=1 globally so build.rs does not attempt to run 'npm run build' (vite) in jobs that have no node.js. the dashboard is built separately in the frontend-tests job.
turbineproxy is a binary-only crate (no src/lib.rs), so cargo test --lib fails with 'no library targets found'.
- routes.rs: sort_by -> sort_by_key for p95/count sorts - postgres/mod.rs: collapse if into match guard for b'K' arm - app_analytics.rs: sort_by -> sort_by_key for queries_total - heatmap.rs: sort_by -> sort_by_key for queries - stmt_shadow.rs: collapse if into match guard for b'C' arm - tracer.rs: sort_by -> sort_by_key for pair counts
adds npm overrides to force a patched version of serialize-javascript, fixing GHSA-5c6j-r48x-rmvq (RCE via RegExp.flags) and GHSA-qj8w-gfj5-8c6v (cpu exhaustion dos) without downgrading docusaurus.
mysqladmin ping fails on mariadb 10.11/11.4 in github actions. use the official 'healthcheck.sh --connect --innodb_initialized' script (bundled in all official mariadb images) for mariadb variants. mysql variants keep mysqladmin ping as before. also add --health-start-period=30s and bump retries to 20 to allow more time for initialization in slow ci runners.
ci runners (especially with mysql 8.4) are slower to initialize. also inherit stderr so proxy startup errors surface in test output instead of being silently discarded.
two root causes: 1. mysql 8.4 disabled mysql_native_password by default. the proxy pool auth fails with 'early eof' on every backend connection attempt. fix: add a ci step that uses docker exec to reconfigure root user with mysql_native_password before running tests. container is named 'mysql-service' via --name in the service options to make it addressable by docker exec. 2. when start_proxy() panicked after the 30s timeout, the OnceLock was left uninitialized. subsequent test threads would re-run the closure, spawn a second proxy process and fail with 'address already in use'. fix: start_proxy() now returns bool instead of panicking — on timeout it kills the child and returns false, which sets OnceLock to false so no further proxy spawns occur and all tests are cleanly skipped via require_proxy!().
…ysql 8.4 mysql 8.4 completely removed the mysql_native_password plugin, so ALTER USER cannot re-enable it at runtime (error: plugin not loaded). the correct fix is to start mysqld with --mysql-native-passwords=ON. add a db-cmd field to every matrix entry and pass it as the service container command. for mariadb, db-cmd is empty so docker falls back to the image default CMD. removes the failing docker exec step added in the previous commit.
…b containers empty command: "" in github actions service containers overrides the image default CMD with an empty string, causing the container to fail. set db-cmd to 'mariadbd' (the actual default CMD for mariadb 10.11 and 11.4 official images) so the container starts with the same behaviour as without a command override.
… mysql 8.4 the mysqld option name has no trailing 's'. 'mysql-native-passwords=ON' is unknown; the correct flag is '--mysql-native-password=ON'.
--mysql-native-password=ON loads the plugin at startup, but the root user is still initialized with caching_sha2_password by the docker entrypoint init script. the proxy pool auth fails because it only implements mysql_native_password handshake. add a step that runs after the healthy service to ALTER USER and switch root@% and root@localhost to mysql_native_password. this now succeeds because the plugin is loaded (unlike the earlier attempt where the plugin was completely absent).
…fault --mysql-native-password=ON loads the plugin but does not change the default auth method. the docker init script still creates root with caching_sha2_password, and the proxy's pool auth fails because it only implements the mysql_native_password handshake (no AuthSwitch). mysql 8.4 removed --default-authentication-plugin. the replacement is --authentication-policy=mysql_native_password,, which sets native password as the default first-factor auth plugin. combined with --mysql-native-password=ON, this makes the docker init script create root with mysql_native_password from the start. removes the docker exec ALTER USER step (no longer needed since root is now created with the correct plugin).
COM_STMT_CLOSE and COM_STMT_SEND_LONG_DATA are fire-and-forget: MySQL sends NO response for these commands. the proxy's send_raw() always calls collect_response_tracked() which blocks on read_exact() forever waiting for bytes that never arrive — deadlocking the connection. add send_raw_no_response() to BackendConnection trait (write only, no read). use it in route_stmt_mysql_shadow for CLOSE and SEND_LONG_DATA. return an empty BackendResponse so the caller skips writing to the client (these commands also expect no client-side response). this fixes integration test hangs on MySQL 8.0 where prepared statement tests (test_prepared_insert_select and later) would block indefinitely when the mysql crate closes a prepared statement.
the response to com_stmt_prepare is a unique multi-packet sequence: 1. prepare_ok packet (status=0x00, stmt_id, num_columns, num_params, ...) 2. if num_params > 0: num_params column-def packets + eof 3. if num_columns > 0: num_columns column-def packets + eof collect_response_tracked() saw the leading 0x00 byte and treated it as a regular ok packet, parsed the stmt_id bytes as garbage lenenc fields, and returned after the first packet. the remaining param/column-def packets stayed in the tcp buffer, corrupting the protocol stream for the next command and causing the proxy to hang waiting for response bytes that were actually unrelated leftover payload. add collect_prepare_response() which reads the prepare_ok packet, extracts num_columns and num_params, then drains the exact number of column-def packets + eofs. send_raw() now detects the com_stmt_prepare command byte and uses this collector instead. this is the actual root cause of the prepared statement hang on mysql 8.0 (the previous fix for com_stmt_close was correct but not sufficient on its own — close was never reached because prepare's reply stream was already corrupted).
The transaction state was being set to false BEFORE routing, causing COMMIT/ROLLBACK to go to a fresh pool connection instead of the sticky tx_conn where the transaction was actually open. This resulted in committed data appearing lost (the original tx_conn was silently rolled back on drop). Now we defer set_in_transaction(false) until AFTER the COMMIT/ROLLBACK is sent through the tx_conn, then release the connection to the pool.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This pull request introduces TurbineProxy v0.5.0, a major release focused on security hardening, high availability improvements, connection pooling enhancements, observability, and CI quality. Key highlights include migration to non-poisoning locks, SCRAM-SHA-256 fuzzing, new dashboard authentication endpoints, per-backend circuit breakers, connection multiplexing, and expanded Prometheus metrics. It also adds chaos testing infrastructure and stricter CI linting for critical code paths.
Security & Hardening
parking_lot::{Mutex, RwLock}to eliminate lock poisoning, with panic-recovery tests added. [1] [2]turbineproxy_dashboard_auth_failures_total). [1] [2] [3]High Availability & Pooling
Dashboard Authentication & API
POST /api/auth/refreshfor token renewal, andPOST /api/auth/logoutfor explicit session revocation. Read-only users can now refresh/logout. [1] [2]token_ttl_secs = 0now means tokens never expire (breaking change).Observability & CI
clippy::unwrap_usedlint on critical code paths (src/dashboard/,src/analytics/). [1] [2]Other Notable Changes
Please review for breaking changes in dashboard token configuration and expanded CI requirements.
Type of Change
feat— new featurefix— bug fixperf— performance improvementrefactor— code change without feat/fixdocs— documentation onlytest— adding or updating testsci— CI/CD changeschore— maintenanceRelated Issue
Closes #
Checklist
cargo fmt --allpassescargo clippy --all-targets -- -D warningspassescargo test --libpasses locallyTesting