Implement IRCv3 UTF8ONLY (ISUPPORT) with mixed-network semantics

[MrIron-no]

Purpose

Implement the IRCv3 UTF8ONLY ISUPPORT token so a server can promise its clients that everything it delivers is valid UTF-8, and reject anything they send that is not.

The complication is that this is a per-server feature on a network where not every server will enable it. Non-UTF-8 channel names, topics, keys, realnames, away text, kick/part reasons and message bodies are all legal on the wire today and will keep arriving from servers that have not enabled the feature. A UTF8ONLY server must relay those bytes untouched to other servers, but must never hand them to a client it made the promise to.

What the spec requires

• UTF8ONLY appears as a bare token in RPL_ISUPPORT (005).
• Servers MUST NOT relay non-UTF-8 content (message data, topics, realnames, ...) to clients.
• Handling of invalid input from clients is implementation-defined. The spec defines the INVALID_UTF8 standard-reply code for use with FAIL. Because the code is defined by this spec, sending FAIL for it does not require the standard-replies capability (that cap only covers standard replies "other than ones enabled by a specific specification").
• The spec says nothing about mixed networks, channel names, invites or nicks. Everything below on those topics is our design.

Design

The promise travels with the client, not the server

A new client flag (FLAG_UTF8ONLY) is set on a local, non-server connection at accept time when the UTF8ONLY feature is on. From then on the flag alone decides everything: whether 005 carries the token, whether input is rejected, whether output is sanitized, and the LIST/INVITE behaviour below. The feature is consulted exactly once per connection.

This gives sane runtime toggling:

• Toggling on leaves already-connected clients unflagged. They never saw the token, so they keep sending and receiving raw bytes. A Latin-1 client that was never told otherwise is not broken, and a client already sitting in a non-UTF-8 channel keeps working.
• Toggling off leaves flagged clients enforced until they reconnect. They were told UTF8ONLY and the promise is kept.
• Strict from the first byte: a client that will be told UTF8ONLY was never allowed to send anything else, including NICK/USER/PASS/CAP before registration. A Latin-1 realname in USER gets a FAIL and the client must retry in UTF-8 (same as Ergo).

Input rejection and output sanitization must be symmetric. Sanitizing output for a client whose non-UTF-8 input we accept is incoherent: it could join #chan\xff, and every JOIN echo, NAMES and message would come back under #chan�, a name it never used. This is why the flag, not the feature, gates both directions.

Input rejection

In parse_client(), after tag parsing and the length check and before command lookup, the whole line (tags and body) is validated for flagged clients. Invalid lines get:

FAIL <COMMAND> INVALID_UTF8 :Message rejected, not valid UTF-8

<COMMAND> is the command token if it is itself valid ASCII, otherwise *. The rejected line still charges the flood counter like any other line. No standard-replies capability is added.

Because a flagged client can never type a non-UTF-8 channel name, it can never join, invite to, or message such a channel. No channel-membership state ever needs reconciling.

Output sanitization: lazy, cached, per line

All delivery to a local client already funnels through send_buffer() in send.c, where the WebSocket framing hook lives. That is the single egress point. Sanitizing on ingress from server links is not an option: rewriting a channel name in transit would split the channel across the network.

Buffers are created raw, always. Server-bound buffers are therefore never touched and nothing in msgq creation changes.

• The first flagged recipient of a line scans it once. If clean, a bit on the MsgBuf records that and every later flagged recipient does a bit test. If dirty, a sanitized and truncated copy is built once and cached on the MsgBuf; every later flagged recipient reuses it.
• Unflagged recipients take the raw buffer as today.
• The cached copy lives on the real body buffer (the MsgBuf attach mechanism means several headers can reference one body) and is released once, on the body's final release.
• Tag prefixes are formatted per recipient, after creation, and client tags relayed from remote users can carry non-UTF-8 values. They are scanned per flagged recipient in the same function (almost always ASCII, so a fast-path check).
• WebSocket clients keep their existing unconditional sanitization of the final wire line, now via the shared helper. RFC 6455 requires it regardless of the feature.

Steady state with the feature on forever and every client flagged: one scan per line, and one extra buffer allocation only for lines that actually carry bad bytes, which only a non-UTF-8 server can produce.

Alternatives considered and rejected:

• Sanitize per recipient without caching — one allocation per flagged recipient of a dirty line on channel fan-out.
• Sanitize at MsgBuf creation when the feature is on — would serve sanitized lines to unflagged clients (the incoherence above), needs a destination rule so server-bound buffers are exempt, and needs an explicit opt-out for the iauth pipe, which shares the same creation function.
• Sanitize per field in command handlers — spreads over dozens of handlers, misses numerics, and every future handler is a new hole. "Must not relay" is a delivery guarantee and only a delivery choke point makes it one.

Line length

Each bad byte becomes three bytes (EF BF BD), so a 510-byte Latin-1 PRIVMSG can grow well past the limit. The sanitized body is truncated at a UTF-8 character boundary so the line fits in 512 bytes including CRLF. Tags are not counted, matching the input side. A line can only exceed the limit after expansion if its trailing parameter is long, so the cut always lands inside the trailing parameter and never changes parsing. This gets a unit test rather than an assumption.

Non-UTF-8 channels: hide what cannot be acted on

Display-only output (WHOIS channel lists, WHO, etc.) is simply sanitized. Two places turn a channel name into an action and are handled specially, because the sanitized name is a trap: JOIN #chan� is valid UTF-8, succeeds, and creates a brand-new look-alike channel with the user alone in it.

• INVITE: when a remote user invites a local flagged client to a channel whose name is not valid UTF-8, ms_invite() drops it silently: no add_invite(), no delivery. The inviter gets no feedback; they are on a remote server and the spec offers nothing here.
• LIST: channels whose names are not valid UTF-8 are skipped in list_next_channels() for flagged requesters.

Shared helper

The UTF-8 sanitizer currently private to websocket.c moves into ircd_string.c / ircd_string.h as three functions: validate a byte range, sanitize into an output buffer with U+FFFD replacement, and truncate at a character boundary to fit 512 with CRLF. ircd_string_t gets the unit cases (overlongs, surrogates, truncated tails, 512 boundary). No new source files.

Out of scope

• WebSocket inbound validation (RFC 6455 close code 1007 on invalid text frames). Separate compliance item that applies even with the feature off; with the flag, a WS client gets a FAIL like anyone else.
• Any ingress rewriting on server links, any S2S protocol change, any NETWORK_FEATURES gating.
• standard-replies capability.
• Nicks, idents, hosts (already ASCII-only).
• Per-field handling of realnames or anything else; the generic egress path covers them.

Tasks

1. Shared helper in ircd_string.c/.h, WebSocket framer switched to it, unit tests in ircd/test/ircd_string_t.c. (independent)
2. Feature and flag: UTF8ONLY boolean feature (default off), FLAG_UTF8ONLY set in the accept path in s_bsd.c, 005 token emitted from the flag, doc/readme.features entry with the toggle semantics. (independent)
3. Fake P10 server: raw-bytes send in tests/p10_server.py plus helpers for JOIN/PRIVMSG/TOPIC/INVITE with arbitrary bytes. (independent)
4. Input rejection in parse_client(). (needs 1, 2)
5. Egress sanitization: MsgBuf flags word and cached-copy pointer, lazy population in send_buffer(), release in msgq.c, tag-prefix scan. (needs 1, 2)
6. LIST and INVITE handling. (needs 1, 2)
7. Integration tests on the hub topology using the fake P10 server as the non-UTF-8 origin: 005 token; FAIL on bad input; sanitized PRIVMSG, TOPIC and realname; truncation of a long Latin-1 message; dropped invite; hidden LIST entry; a pre-toggle client keeping raw bytes both ways; buffer-pool stress with a flood of dirty lines. Feature enabled via oper SET and reset at the end, so no new docker topology. (needs 3–6)

Risks and edge cases on record

• A sanitized NAMES or WHOIS reply that grows past 512 loses names at the tail. Accepted: the alternative is an overlong line, and the names belong to a channel the client cannot use anyway.
• Each dirty line costs one extra buffer from the shared pool, so a hostile or broken peer flooding Latin-1 doubles buffer use for those lines. Bounded (released with the line, pool degrades gracefully when exhausted) but covered by a stress test.
• The cached copy must be freed exactly once on the real buffer's final release; the attach mechanism makes this the one place to get right.
• Opers on a flagged connection see sanitized STATS output and G-line masks like anyone else. No special casing.
• A flagged client's own echoed lines were validated on input, so they hit the clean-bit fast path.