TLS I/O layer: event-driven core, backends as thin codecs (supersedes #101) - #105
Open
MrIron-no wants to merge 2 commits into
Open
TLS I/O layer: event-driven core, backends as thin codecs (supersedes #101)#105MrIron-no wants to merge 2 commits into
MrIron-no wants to merge 2 commits into
Conversation
Rework the TLS layer so all engine / sendq / IPcheck / flood logic lives once
in the core and the backends (openssl, gnutls, libtls, none) become thin
codecs. Squashes the refactor/tls-io-layer branch.
Core -- new ircd/tls_io.c, include/tls_io.h:
- socket interest is computed from connection state (level-triggered model),
never pushed ad hoc;
- the send drain owns the con_rexmit / msgq_excise partial-write handling;
- the receive path, the fatal-error teardown (mark the socket dead so we can
never fall back to plaintext), and peer-fingerprint storage;
- the handshake trust policy (cert-required / verifypeer / fingerprint pin).
Backends now only provide read / write / handshake / drop primitives plus
setup; they never touch msgq, con_rexmit, cli_tls_fingerprint, FLAG_*, or
socket_events (kept honest by a grep invariant).
Handshake: driven by ET_READ / ET_WRITE, bounded by a single handshake-deadline
timer for both directions, waiting on exactly the blocked direction so a
level-triggered writable socket cannot spin. sendq enforcement, IPcheck
(before make_client on accept), flood and throttle all apply to TLS via the
existing read_packet / send_queued paths -- no parallel I/O path.
Fixes carried in:
- s_bsd: never touch cptr after read_packet() may have freed it (CPTR_KILLED);
- s_bsd: report an outbound handshake failure that arrives as EOF/error;
- s_bsd: free the TLS session, Client and IPcheck count on rejected accepts;
- send_queued: a zero-credit TLS success (a rexmit drain that excised the last
queued message) is progress, not a block -- don't strand an empty
connection on send_queues;
- tls_openssl: harden every context (NO_RENEGOTIATION | NO_COMPRESSION);
- tls_gnutls: refuse client-initiated renegotiation; bound the handshake loop;
- engine_kqueue: deliver unread data before EOF.
Tests: misbehaving-peer harness (inbound and outbound), s2s TLS burst, real
TLS 1.3 KeyUpdate, REHASH cert rotation under a live connection, data-path
teardown regressions, and s2s latency / slow-handshake edge cases; TLS_BACKEND
(openssl / gnutls / libtls) selection documented. Full tls/ suite green on all
three backends.
Supersedes UndernetIRC#101.
Drive tls_io.c through a scripted fake tls_backend_* so the core's logic is
tested deterministically, with no TLS library, socket, or docker involved:
- socket-interest model: full 36-case truth table of cross-direction
blocking states x queued output x /LIST (the anti-spin and anti-stall
invariants);
- tls_io_sendv() drain: partial writes drained in-call, con_rexmit parking
and resumption across calls, the rexmit-bytes-never-credited rule, the
zero-credit success (a rexmit drain that excised the last queued message
is progress, not a block), priority-before-normal and full three-way
(rexmit, prio, normal) wire ordering, and fatal teardown from both the
queue walk and the rexmit drain;
- tls_io_recv() blocked-direction recording and fatal teardown;
- fingerprint storage edge cases (non-SHA-256 lengths, hex length limits,
Cloudflare-port suppression);
- ircd_tls_negotiate() trust policy matrix over scripted tls_peer material
(cert-required, verifypeer, backend reason fall-through, digest vs
pre-formatted hex fingerprint hand-off, no-session re-entry).
Scenarios mimic the real caller (msgq_delete of count_out between calls, as
send_queued does) against a real MsgQ, and the fake backend captures the
exact byte stream accepted so ordering and duplication bugs are caught, not
just return codes. The backend error-classification code itself
(SSL_get_error and friends) is intentionally out of scope -- the docker
suite's TLS_BACKEND matrix covers that with real handshakes.
Runs under make check (no docker), so it also runs on FreeBSD.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces #101. That PR fixed the inbound-handshake busy-loop and the kqueue
EOF ordering, but its review (10 findings) and the follow-up data-path review
kept pointing at the same root cause: engine, sendq, IPcheck and flood logic
was duplicated across the TLS backends and the socket-interest decisions were
scattered ad hoc through the event loop. Rather than keep patching that shape,
this PR restructures the TLS layer and carries the #101 fixes inside the new
design.
Design
Core — new
ircd/tls_io.c/include/tls_io.h:(level-triggered model), never pushed ad hoc. TLS breaks the plaintext
assumption that "readable = want to read, writable = want to send" — a TLS
write can block waiting to read and vice versa (renegotiation, TLS 1.3
KeyUpdate, a partial record).
tls_desired_events()/tls_want_writable()own that mapping, so interest can never drift out ofsync with what the TLS session needs, and a level-triggered writable socket
cannot spin.
tls_io_sendv) owns thecon_rexmit/msgq_excisepartial-write bookkeeping; the receive path records the blocked direction.
never fall back to plaintext.
(cert-required / verifypeer / fingerprint pin) live here too.
Backends (
tls_openssl,tls_gnutls,tls_libtls,tls_none) become thincodecs: read / write / handshake / drop primitives plus setup. They never
touch
msgq,con_rexmit,cli_tls_fingerprint,FLAG_*, orsocket_events— kept honest by a grep invariant.Handshake: driven by
ET_READ/ET_WRITE, bounded by a singlehandshake-deadline timer for both directions, waiting on exactly the blocked
direction. sendq enforcement, IPcheck (before
make_client()on accept),flood and throttle all apply to TLS through the existing
read_packet()/send_queued()paths — there is no parallel I/O path.Fixes carried in
s_bsd: never touchcptrafterread_packet()may have freed it(
CPTR_KILLED).s_bsd: report an outbound handshake failure that arrives as EOF/error —previously a silent teardown with no operator notice.
s_bsd: free the TLS session, Client and IPcheck count on rejected accepts(unbounded growth under a reconnect flood to a TLS port).
send_queued: a zero-credit TLS success (a rexmit drain that excised thelast queued message) is progress, not a block — don't strand an empty
connection on
send_queues.tls_openssl: harden every context (NO_RENEGOTIATION | NO_COMPRESSION).tls_gnutls: refuse client-initiated renegotiation (parity with OpenSSL);bound the handshake loop.
engine_kqueue: deliver unread data before EOF (from Drive the inbound TLS handshake by socket events; kqueue: deliver unread data before EOF #101; validated onFreeBSD).
Relation to the #101 findings
Findings 1–3 and 5–10 are addressed, most of them structurally rather than by
point fixes: the single interest model removes the WANT_WRITE misreporting and
the READABLE-during-WANT_WRITE spin (3, 5); the single deadline timer replaces
the three per-backend checks and the timer-dispatch copy (7, 10); the gnutls
retry loop no longer encodes "retry" as a fake socket direction (6); IPcheck
runs before allocation on accept (8); and the outbound path no longer
duplicates the negotiate dispatch (9). Findings 1 and 2 are the
s_bsd/kqueue fixes above.
Finding 4's remainder — every engine's
SO_ERRORpre-check drops a peer'sfinal line on RST, engine-independent — is deliberately not in this PR.
It is now reproduced deterministically on Linux/epoll and filed with a fix
sketch as #104 (the FIN case needs no epoll fix, confirmed by the same repro).
Testing
tests/tls/test_tls_bogus_peer.py): drivesOpenSSL through
ssl.MemoryBIOso each scenario controls which bytes hitthe wire and when — silent peers, stalled/dribbled ClientHello, garbage,
RST/FIN mid-handshake, a peer that never reads (write-blocked server
flight), post-handshake flood, plus outbound scenarios against a controlled
server sidecar. Every scenario checks the deadline, no CPU spin, and that a
healthy client keeps getting PONGs.
test_tls_keyupdate.py), REHASH certrotation under a live connection (
test_tls_rehash.py), s2s TLS burst withthe hub in both TLS roles (
test_tls_s2s_burst.py), and data-path teardownregressions (
test_tls_datapath_repro.py).tests/tls/suite green on all three backends (TLS_BACKEND=openssl / gnutls / libtls; selection documented in
tests/README.md).ircd/test/tls_io_t.c, runs undermake checkwith no docker or TLS library): a scripted fake backend drives
tls_io.cthrough a 36-case socket-interest truth table, the send drain (rexmit
parking/resumption, the rexmit-bytes-never-credited rule, zero-credit
success, priority and three-way wire ordering, fatal teardown), receive
direction recording, fingerprint edge cases, and the negotiate trust-policy
matrix. Backend error classification is deliberately left to the
TLS_BACKENDmatrix above.