Conversation
Reactor lifecycle * Cluster#close_async stops the ione IO reactor itself in addition to the control connection doing so, and only resolves once the reactor has stopped, so a closed cluster can never leave an io_reactor thread behind. * ControlConnection#close_async used to return early when its status was :closed, which is also the initial status. Closing a cluster whose control connection never connected returned a future that never resolved and left the reactor running. It now keys off whether the close already completed. Cassandra::Cluster::IoReactor (subclass of Ione::Io::IoReactor) * The connect timeout now bounds the TLS handshake. ione only applied it to the TCP connect, so a proxy that accepts TCP but hangs TLS hung connects forever. * A TLS socket waiting for server bytes is selected for readability. ione selected every connecting socket for writability; a connected TCP socket is always writable, so the loop spun at 100% CPU while a handshake hung. * Closing a TLS connection closes the raw TCP socket too (sync_close). * The reactor sleeps until IO, the next timer, or an unblock instead of ticking every second; sockets whose fd was closed underneath it are evicted rather than re-selected. * The reactor thread is named io_reactor so leaked reactors can be counted. Reconnection * Reconnection::Policies::Exponential accepts jitter: (fraction) so a fleet does not reconnect in lockstep; the ceiling is always honoured. Requires ione ~> 1.3 (the reactor subclass relies on 1.3 internals). Specs only use loopback sockets opened by the spec itself. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A peer that accepts TCP but stalls TLS can leave
Cassandra.clusterand reconnects hanging while the reactor spins. This change bounds TLS handshakes byconnect_timeout, avoids polling idle sockets, and makes cluster shutdown and reactor restart release and restore their resources correctly.Runtime target for review and validation: Ruby 3.4+.
Based on
main(86b8ba00), retaining its Cassandra 5.x schema-fetcher mapping, vector support, and Ruby 3.4 compatibility updates.Changes
Cassandra::Cluster::IoReactor: share a deadline across TCP connect and TLS handshake, select pending TLS handshakes for readability when required, and close the underlying TCP socket with TLS.connect_timeout: Float::INFINITYremains supported.ControlConnection. A distinct initial:disconnectedstate allows closing a never-connected cluster; repeated close and explicit reconnect use the correct lifecycle future.~> 1.3.0, name the reactor threadio_reactor, and update the gem version and lockfile to 3.2.6.Validation
Validated on Ruby 3.4.5 using the checked-in lockfile, including the ione fork at
e665ea33(1.3.1).bundle exec rake compile)The full suite uses the real Snappy and LZ4 native extensions, with no compressor stubs. The two pending examples cover credentials-request equality and hashing. CCM integration tests and RuboCop were not run.
The extra schema-picker checks exercised driver dispatch and schema reconstruction with the existing
system_schemafixtures; they did not use a live Cassandra server.The shutdown/restart interleaving was reproduced deterministically before the fix and now passes. Regression coverage also checks an old stop future completing after a new run has started. Network tests use local loopback sockets.
Review findings resolved
All previously listed findings and the subsequent restart-race finding are addressed:
Restore the unblocker after stop and crash.
Use a fixed drain timeout tick even when timers are overdue.
Remove the redundant nested reactor stop from
Cluster#close_async.Guard a missing raw socket in TLS close.
Bound the jitter window before drawing.
Use a distinct initial control-connection state and fresh close promises after explicit reconnect.
Update the lockfile PATH version to 3.2.6.
Preserve infinite connect timeouts without constructing an infinite
Time.Skip redundant timer wake-ups on the reactor thread.
Bound select by the nearest finite connect deadline.
Probe descriptors to handle real
EBADFerrors.Handle select
TypeErrorfrom a missingto_io.Normalize invalid jitter errors to
ArgumentErrorand correct the randomness-source documentation.Restrict ione to the 1.3.x series.
Make pipe restoration and restart atomic with shutdown completion, preserving each run's promises.
Wake idle select after closing connections or listeners, including drain paths, before notifying connection close listeners.
Contain socket
IOErrorandEBADFduring dispatch, and tolerate transient accept errors.Prefer completed TCP connects and TLS handshakes over elapsed deadlines, including connections queued while stopped.
Follow-up
The reactor fixes are also proposed in ione PR #1. After that merges, update the ione dependency revision and replace the driver-local reactor with
Ione::Io::IoReactorin the same change.