Skip to content

Stop reactor leaks, TLS handshake hangs and reactor spin - #5

Open
SixiS wants to merge 4 commits into
mainfrom
reactor-lifecycle-fixes
Open

SixiS wants to merge 4 commits into
mainfrom
reactor-lifecycle-fixes

Conversation

@SixiS

@SixiS SixiS commented Sep 14, 2026

Copy link
Copy Markdown
Collaborator

A peer that accepts TCP but stalls TLS can leave Cassandra.cluster and reconnects hanging while the reactor spins. This change bounds TLS handshakes by connect_timeout, avoids polling idle sockets, and makes cluster shutdown and reactor restart release and restore their resources correctly.

Runtime target for review and validation: Ruby 3.4+.

Based on main (86b8ba00), retaining its Cassandra 5.x schema-fetcher mapping, vector support, and Ruby 3.4 compatibility updates.

Changes

  • Add Cassandra::Cluster::IoReactor: share a deadline across TCP connect and TLS handshake, select pending TLS handshakes for readability when required, and close the underlying TCP socket with TLS. connect_timeout: Float::INFINITY remains supported.
  • Sleep until IO, the next timer or connect deadline, or an explicit wake-up. Drain with a fixed tick and evict invalid descriptors without hiding unrelated select errors.
  • Wake the reactor when connections or listeners close or drain, so idle descriptors are released promptly. Preserve close-listener ordering and contain socket IO errors without stopping healthy connections.
  • Accept completed TCP connections and TLS handshakes when observed at the deadline; only still-pending attempts time out.
  • Restore the wake-up pipe atomically with the start transition. Each reactor run owns its start/stop promises, and completion callbacks run outside the state lock.
  • Keep reactor shutdown owned by ControlConnection. A distinct initial :disconnected state allows closing a never-connected cluster; repeated close and explicit reconnect use the correct lifecycle future.
  • Add optional reconnection jitter, sampled uniformly within the bounded interval so retries do not concentrate at the ceiling.
  • Pin ione to ~> 1.3.0, name the reactor thread io_reactor, and update the gem version and lockfile to 3.2.6.

Validation

Validated on Ruby 3.4.5 using the checked-in lockfile, including the ione fork at e665ea33 (1.3.1).

Run Result
Native extension compilation (bundle exec rake compile) Pass
Full unit suite 1,141 examples, 0 failures, 2 pending
Reactor specs, included in the full suite 50 examples, 0 failures
Isolated reactor specs on Linux, Ruby 3.4.5 50 examples, 0 failures
Additional local schema-picker checks for Cassandra 3.11.17, 4.1.7, and 5.0.9 3 examples, 0 failures

The full suite uses the real Snappy and LZ4 native extensions, with no compressor stubs. The two pending examples cover credentials-request equality and hashing. CCM integration tests and RuboCop were not run.

The extra schema-picker checks exercised driver dispatch and schema reconstruction with the existing system_schema fixtures; they did not use a live Cassandra server.

The shutdown/restart interleaving was reproduced deterministically before the fix and now passes. Regression coverage also checks an old stop future completing after a new run has started. Network tests use local loopback sockets.

Review findings resolved

All previously listed findings and the subsequent restart-race finding are addressed:

  • Restore the unblocker after stop and crash.

  • Use a fixed drain timeout tick even when timers are overdue.

  • Remove the redundant nested reactor stop from Cluster#close_async.

  • Guard a missing raw socket in TLS close.

  • Bound the jitter window before drawing.

  • Use a distinct initial control-connection state and fresh close promises after explicit reconnect.

  • Update the lockfile PATH version to 3.2.6.

  • Preserve infinite connect timeouts without constructing an infinite Time.

  • Skip redundant timer wake-ups on the reactor thread.

  • Bound select by the nearest finite connect deadline.

  • Probe descriptors to handle real EBADF errors.

  • Handle select TypeError from a missing to_io.

  • Normalize invalid jitter errors to ArgumentError and correct the randomness-source documentation.

  • Restrict ione to the 1.3.x series.

  • Make pipe restoration and restart atomic with shutdown completion, preserving each run's promises.

  • Wake idle select after closing connections or listeners, including drain paths, before notifying connection close listeners.

  • Contain socket IOError and EBADF during dispatch, and tolerate transient accept errors.

  • Prefer completed TCP connects and TLS handshakes over elapsed deadlines, including connections queued while stopped.

Follow-up

The reactor fixes are also proposed in ione PR #1. After that merges, update the ione dependency revision and replace the driver-local reactor with Ione::Io::IoReactor in the same change.

SixiS and others added 4 commits September 14, 2026 12:24
Reactor lifecycle
* Cluster#close_async stops the ione IO reactor itself in addition to the
  control connection doing so, and only resolves once the reactor has
  stopped, so a closed cluster can never leave an io_reactor thread behind.
* ControlConnection#close_async used to return early when its status was
  :closed, which is also the initial status. Closing a cluster whose
  control connection never connected returned a future that never resolved
  and left the reactor running. It now keys off whether the close already
  completed.

Cassandra::Cluster::IoReactor (subclass of Ione::Io::IoReactor)
* The connect timeout now bounds the TLS handshake. ione only applied it to
  the TCP connect, so a proxy that accepts TCP but hangs TLS hung connects
  forever.
* A TLS socket waiting for server bytes is selected for readability. ione
  selected every connecting socket for writability; a connected TCP socket
  is always writable, so the loop spun at 100% CPU while a handshake hung.
* Closing a TLS connection closes the raw TCP socket too (sync_close).
* The reactor sleeps until IO, the next timer, or an unblock instead of
  ticking every second; sockets whose fd was closed underneath it are
  evicted rather than re-selected.
* The reactor thread is named io_reactor so leaked reactors can be counted.

Reconnection
* Reconnection::Policies::Exponential accepts jitter: (fraction) so a fleet
  does not reconnect in lockstep; the ceiling is always honoured.

Requires ione ~> 1.3 (the reactor subclass relies on 1.3 internals).

Specs only use loopback sockets opened by the spec itself.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant