Skip to content

TLS I/O layer: event-driven core, backends as thin codecs (supersedes #101) - #105

Open
MrIron-no wants to merge 2 commits into
UndernetIRC:mainfrom
MrIron-no:tls-io-refactor
Open

TLS I/O layer: event-driven core, backends as thin codecs (supersedes #101)#105
MrIron-no wants to merge 2 commits into
UndernetIRC:mainfrom
MrIron-no:tls-io-refactor

Conversation

@MrIron-no

@MrIron-no MrIron-no commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Summary

Replaces #101. That PR fixed the inbound-handshake busy-loop and the kqueue
EOF ordering, but its review (10 findings) and the follow-up data-path review
kept pointing at the same root cause: engine, sendq, IPcheck and flood logic
was duplicated across the TLS backends and the socket-interest decisions were
scattered ad hoc through the event loop. Rather than keep patching that shape,
this PR restructures the TLS layer and carries the #101 fixes inside the new
design.

Design

Core — new ircd/tls_io.c / include/tls_io.h:

  • Socket interest is computed from connection state in exactly one place
    (level-triggered model), never pushed ad hoc. TLS breaks the plaintext
    assumption that "readable = want to read, writable = want to send" — a TLS
    write can block waiting to read and vice versa (renegotiation, TLS 1.3
    KeyUpdate, a partial record). tls_desired_events() /
    tls_want_writable() own that mapping, so interest can never drift out of
    sync with what the TLS session needs, and a level-triggered writable socket
    cannot spin.
  • The send drain (tls_io_sendv) owns the con_rexmit / msgq_excise
    partial-write bookkeeping; the receive path records the blocked direction.
  • Fatal-error teardown marks the socket dead so a failed TLS connection can
    never fall back to plaintext.
  • Peer-fingerprint storage and the handshake trust policy
    (cert-required / verifypeer / fingerprint pin) live here too.

Backends (tls_openssl, tls_gnutls, tls_libtls, tls_none) become thin
codecs
: read / write / handshake / drop primitives plus setup. They never
touch msgq, con_rexmit, cli_tls_fingerprint, FLAG_*, or
socket_events — kept honest by a grep invariant.

Handshake: driven by ET_READ / ET_WRITE, bounded by a single
handshake-deadline timer for both directions, waiting on exactly the blocked
direction. sendq enforcement, IPcheck (before make_client() on accept),
flood and throttle all apply to TLS through the existing read_packet() /
send_queued() paths — there is no parallel I/O path.

Fixes carried in

  • s_bsd: never touch cptr after read_packet() may have freed it
    (CPTR_KILLED).
  • s_bsd: report an outbound handshake failure that arrives as EOF/error —
    previously a silent teardown with no operator notice.
  • s_bsd: free the TLS session, Client and IPcheck count on rejected accepts
    (unbounded growth under a reconnect flood to a TLS port).
  • send_queued: a zero-credit TLS success (a rexmit drain that excised the
    last queued message) is progress, not a block — don't strand an empty
    connection on send_queues.
  • tls_openssl: harden every context (NO_RENEGOTIATION | NO_COMPRESSION).
  • tls_gnutls: refuse client-initiated renegotiation (parity with OpenSSL);
    bound the handshake loop.
  • engine_kqueue: deliver unread data before EOF (from Drive the inbound TLS handshake by socket events; kqueue: deliver unread data before EOF #101; validated on
    FreeBSD).

Relation to the #101 findings

Findings 1–3 and 5–10 are addressed, most of them structurally rather than by
point fixes: the single interest model removes the WANT_WRITE misreporting and
the READABLE-during-WANT_WRITE spin (3, 5); the single deadline timer replaces
the three per-backend checks and the timer-dispatch copy (7, 10); the gnutls
retry loop no longer encodes "retry" as a fake socket direction (6); IPcheck
runs before allocation on accept (8); and the outbound path no longer
duplicates the negotiate dispatch (9). Findings 1 and 2 are the s_bsd /
kqueue fixes above.

Finding 4's remainder — every engine's SO_ERROR pre-check drops a peer's
final line on RST, engine-independent — is deliberately not in this PR.
It is now reproduced deterministically on Linux/epoll and filed with a fix
sketch as #104 (the FIN case needs no epoll fix, confirmed by the same repro).

Testing

  • Misbehaving-peer harness (tests/tls/test_tls_bogus_peer.py): drives
    OpenSSL through ssl.MemoryBIO so each scenario controls which bytes hit
    the wire and when — silent peers, stalled/dribbled ClientHello, garbage,
    RST/FIN mid-handshake, a peer that never reads (write-blocked server
    flight), post-handshake flood, plus outbound scenarios against a controlled
    server sidecar. Every scenario checks the deadline, no CPU spin, and that a
    healthy client keeps getting PONGs.
  • Real TLS 1.3 KeyUpdate mid-session (test_tls_keyupdate.py), REHASH cert
    rotation under a live connection (test_tls_rehash.py), s2s TLS burst with
    the hub in both TLS roles (test_tls_s2s_burst.py), and data-path teardown
    regressions (test_tls_datapath_repro.py).
  • Full tests/tls/ suite green on all three backends (TLS_BACKEND =
    openssl / gnutls / libtls; selection documented in tests/README.md).
  • C unit suite for the core (ircd/test/tls_io_t.c, runs under make check
    with no docker or TLS library): a scripted fake backend drives tls_io.c
    through a 36-case socket-interest truth table, the send drain (rexmit
    parking/resumption, the rexmit-bytes-never-credited rule, zero-credit
    success, priority and three-way wire ordering, fatal teardown), receive
    direction recording, fingerprint edge cases, and the negotiate trust-policy
    matrix. Backend error classification is deliberately left to the
    TLS_BACKEND matrix above.
  • Tested on FreeBSD (kqueue), including the unread-data-before-EOF ordering.

Rework the TLS layer so all engine / sendq / IPcheck / flood logic lives once
in the core and the backends (openssl, gnutls, libtls, none) become thin
codecs.  Squashes the refactor/tls-io-layer branch.

Core -- new ircd/tls_io.c, include/tls_io.h:
  - socket interest is computed from connection state (level-triggered model),
    never pushed ad hoc;
  - the send drain owns the con_rexmit / msgq_excise partial-write handling;
  - the receive path, the fatal-error teardown (mark the socket dead so we can
    never fall back to plaintext), and peer-fingerprint storage;
  - the handshake trust policy (cert-required / verifypeer / fingerprint pin).
Backends now only provide read / write / handshake / drop primitives plus
setup; they never touch msgq, con_rexmit, cli_tls_fingerprint, FLAG_*, or
socket_events (kept honest by a grep invariant).

Handshake: driven by ET_READ / ET_WRITE, bounded by a single handshake-deadline
timer for both directions, waiting on exactly the blocked direction so a
level-triggered writable socket cannot spin.  sendq enforcement, IPcheck
(before make_client on accept), flood and throttle all apply to TLS via the
existing read_packet / send_queued paths -- no parallel I/O path.

Fixes carried in:
  - s_bsd: never touch cptr after read_packet() may have freed it (CPTR_KILLED);
  - s_bsd: report an outbound handshake failure that arrives as EOF/error;
  - s_bsd: free the TLS session, Client and IPcheck count on rejected accepts;
  - send_queued: a zero-credit TLS success (a rexmit drain that excised the last
    queued message) is progress, not a block -- don't strand an empty
    connection on send_queues;
  - tls_openssl: harden every context (NO_RENEGOTIATION | NO_COMPRESSION);
  - tls_gnutls: refuse client-initiated renegotiation; bound the handshake loop;
  - engine_kqueue: deliver unread data before EOF.

Tests: misbehaving-peer harness (inbound and outbound), s2s TLS burst, real
TLS 1.3 KeyUpdate, REHASH cert rotation under a live connection, data-path
teardown regressions, and s2s latency / slow-handshake edge cases; TLS_BACKEND
(openssl / gnutls / libtls) selection documented.  Full tls/ suite green on all
three backends.

Supersedes UndernetIRC#101.
Drive tls_io.c through a scripted fake tls_backend_* so the core's logic is
tested deterministically, with no TLS library, socket, or docker involved:

  - socket-interest model: full 36-case truth table of cross-direction
    blocking states x queued output x /LIST (the anti-spin and anti-stall
    invariants);
  - tls_io_sendv() drain: partial writes drained in-call, con_rexmit parking
    and resumption across calls, the rexmit-bytes-never-credited rule, the
    zero-credit success (a rexmit drain that excised the last queued message
    is progress, not a block), priority-before-normal and full three-way
    (rexmit, prio, normal) wire ordering, and fatal teardown from both the
    queue walk and the rexmit drain;
  - tls_io_recv() blocked-direction recording and fatal teardown;
  - fingerprint storage edge cases (non-SHA-256 lengths, hex length limits,
    Cloudflare-port suppression);
  - ircd_tls_negotiate() trust policy matrix over scripted tls_peer material
    (cert-required, verifypeer, backend reason fall-through, digest vs
    pre-formatted hex fingerprint hand-off, no-session re-entry).

Scenarios mimic the real caller (msgq_delete of count_out between calls, as
send_queued does) against a real MsgQ, and the fake backend captures the
exact byte stream accepted so ordering and duplication bugs are caught, not
just return codes.  The backend error-classification code itself
(SSL_get_error and friends) is intentionally out of scope -- the docker
suite's TLS_BACKEND matrix covers that with real handshakes.

Runs under make check (no docker), so it also runs on FreeBSD.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant