From a19b4b5657c82ba4adc1b0941f17551d8eb11900 Mon Sep 17 00:00:00 2001 From: Bronek Kozicki Date: Tue, 25 Aug 2026 15:52:50 +0100 Subject: [PATCH 1/4] docs(network): glossary, ADRs 0001-0003 and extension points for the stream network MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Record the design for flux-network before the implementation stack lands: - CONTEXT.md: the workspace glossary (Tile, Spine, Signal, Waker; Stream, Group, Tenant, Raw group, Owned/External mode, Endpoint, Peer, Deadline, Draining, Lingering, Refused). - ADR 0001: poll ownership is a construction-time mode of the stream network, and protocol layers are tenants the network schedules — one driver owns deadline folding, routing by group, maintenance order and ticking; tenants expose their events by pull; hooks stay private behind an opaque TenantRef carrier. - ADR 0002: transports are the closed set Endpoint { Tcp, Unix }. - ADR 0003: Unix-domain socket files are probed (lstat, then connect) before a stale one is replaced, and removed on close. - docs/extension-points.md: eventual (streamed responses) and speculative (TLS, Unix socket file mode, QUIC) directions the design keeps open without deciding. All three ADRs are accepted. Assisted-by: Claude Code:claude-fable-5 --- CONTEXT.md | 90 ++++++++++++++ .../0001-poll-ownership-modes-and-tenancy.md | 109 +++++++++++++++++ docs/adr/0002-endpoint-closed-set.md | 24 ++++ docs/adr/0003-unix-socket-lifecycle.md | 34 ++++++ docs/extension-points.md | 111 ++++++++++++++++++ 5 files changed, 368 insertions(+) create mode 100644 CONTEXT.md create mode 100644 docs/adr/0001-poll-ownership-modes-and-tenancy.md create mode 100644 docs/adr/0002-endpoint-closed-set.md create mode 100644 docs/adr/0003-unix-socket-lifecycle.md create mode 100644 docs/extension-points.md diff --git a/CONTEXT.md b/CONTEXT.md new file mode 100644 index 0000000..5b69df2 --- /dev/null +++ b/CONTEXT.md @@ -0,0 +1,90 @@ +# Flux + +Flux runs latency-sensitive applications as pinned-core worker loops that exchange +messages over shared memory. This glossary fixes the words used across the workspace; +implementation detail belongs in code and ADRs, not here. + +## Language + +### Execution + +**Tile**: +One worker loop pinned to a core, with an init, a loop body, and a teardown. +_Avoid_: thread, worker, actor + +**Spine**: +The shared-memory queue fabric that connects tiles within a process. +_Avoid_: bus, channel + +**Signal**: +The process-wide sticky wake counter that idle tiles park on and that producers +increment. +_Avoid_: work signal, event, notify + +**Waker**: +The `mio::Waker` a tile registers with the Signal so that spine work interrupts the +tile's blocking poll. +_Avoid_: unparker, notifier + +### Networking + +**Stream**: +An ordered byte channel with one peer at each end — a TCP or Unix-domain connection — and the +unit a stream network drives. A response held open for appended writes is not a stream in this +sense. +_Avoid_: socket (the OS handle, not the channel), pipe + +**Group**: +A set of connections sharing one wire framing and one socket configuration inside a +network, with limits enforced per connection. +_Avoid_: pool, channel, protocol + +**Tenant**: +A protocol layer that owns exactly one Group inside a shared network and is scheduled by +that network; an HTTP server or client is a tenant. +_Avoid_: instance, service, sub-network + +**Raw group**: +A Group no Tenant owns; the caller consumes its events directly as they arrive, borrowing +each payload for the duration of the call. +_Avoid_: unmanaged group, bare group + +**Owned mode**: +A network that holds its own poll and drives it, with whatever timeout the caller passes. +_Avoid_: standalone, embedded + +**External mode**: +A network built over a poll the caller holds; the caller delivers readiness events and +drives timers, and may register its own sources alongside. +_Avoid_: injected, shared, hosted + +**Endpoint**: +The address a listener binds or an outbound connection targets: a TCP socket address or +a Unix-domain socket path. +_Avoid_: bind, addr, address, target + +**Peer**: +The identity of the remote end of an accepted connection: a TCP socket address, or +anonymous for a Unix-domain socket. +_Avoid_: client, remote + +**Deadline**: +The per-request timer on an outbound connection; expiry fails the request and closes the +connection. +_Avoid_: timeout (which names the idle sweep), TTL + +**Draining**: +The closing state of a connection whose request stream was fully consumed: the +connection closes as soon as its queued bytes are written. +_Avoid_: flushing, closing + +**Lingering**: +The closing state of a connection whose request stream was not fully consumed: after the +response is written the write side shuts, inbound bytes are read and discarded under +idle and total caps, then the connection closes. +_Avoid_: half-close, graceful close, linger-close + +**Refused**: +An accepted connection dropped immediately, without registration or bytes, because its +Group is at its connection cap. +_Avoid_: rejected, throttled diff --git a/docs/adr/0001-poll-ownership-modes-and-tenancy.md b/docs/adr/0001-poll-ownership-modes-and-tenancy.md new file mode 100644 index 0000000..585bdd8 --- /dev/null +++ b/docs/adr/0001-poll-ownership-modes-and-tenancy.md @@ -0,0 +1,109 @@ +--- +status: accepted +--- + +# Poll ownership is a network mode; protocols are tenants the network schedules + +A tile that hosts sockets must own exactly one `mio::Poll`, so that under `flux/park` it can +register one `Waker` with the Signal and block in `poll` with a non-zero timeout — two polls +in one tile means neither may ever block. `StreamNetwork` therefore chooses at construction who +holds the poll: **Owned mode** (the network creates and drives its own poll, and hands out a +`Waker` on a reserved token) or **External mode** (the network is built over a `Registry` +cloned from the caller's poll and a token base, and never polls). Protocol layers such as +`HttpTenant` do not own a network: each is a **Tenant** owning one `StreamGroup` inside a +shared `StreamNetwork`, and the network — not the caller — schedules them. In Owned mode one +call per iteration, `drive(max_timeout, tenants, raw_handler)`, folds every deadline, polls +once, routes each event to the tenant owning its group, runs network maintenance and ticks each +tenant. In External mode the caller makes only the three calls a caller-held poll inherently +requires — `next_deadline(tenants)` to fold into its own timeout, `handle_event(&event, +tenants, raw_handler) -> bool` per readiness event (false: not ours, route to your own +sources), and `tick(tenants)` once per iteration — and reconstructs nothing else. Tenants +expose their protocol events by pull (`next_event(&mut net)`). The scheduling hooks — group, +`on_event`, `tick`, `next_deadline` — live on a trait private to flux that only the network +calls; a tenant hands the network an opaque `TenantRef<'_>` (`beacon.as_tenant()`), so +`drive(max_timeout, &mut [beacon.as_tenant(), engine.as_tenant()], raw_handler)` is the whole +public contract, and nothing outside flux can implement or invoke a hook. Groups no tenant owns are **raw groups**: their +events reach `raw_handler` synchronously, lending the payload for the duration of the call. + +## Considered options + +- **Owned mode only, tenants inside it.** Rejected: users with their own mio sources need a + caller-held poll; a foreign-source API on an owned poll would duplicate External mode with + a worse contract. +- **Per-tenant token ranges.** Rejected: a tenant's token demand (accepted connections) is + unknowable up front. One network is one contiguous token space from its base; the caller + reserves its own tokens below it. +- **Type-state `StreamNetwork` / `StreamNetwork`.** Rejected: the parameter + would infect every tenant signature to catch a misuse that fires on the first poll in any + test. Polling an External-mode network panics with a clear message instead. +- **Individually driven tenants** — the caller folds deadlines, chains `on_event` calls and + orders ticks. Rejected: every tile becomes a slightly different implementation of the + network scheduler, and the scheduling invariants live nowhere. +- **A public `Tenant` trait as the carrier (`&mut [&mut dyn Tenant]`), sealed or not.** + Rejected: an openly implementable trait commits flux to far more than four methods before + third-party tenants are product scope, and sealing does not help — sealing prevents + implementation, not invocation. On a trait object, supertrait methods resolve as if they were + inherent even when the supertrait is unnameable, so hooks placed on a private supertrait are + still callable by every holder of the object. Only an opaque carrier over a private trait + keeps the hooks the network's alone. +- **Releasing a claim from the network side (`release_group`), or permanent claims.** + Rejected: the former leaves a stopped tenant's connections delivering HTTP-framed bytes to + `raw_handler` with nothing to parse them; the latter forces every tenant to live as long as + its network for no gain. A `Drop`-time check was rejected because it fires spuriously at + process teardown. +- **Pull-based delivery for raw groups too.** Rejected: a raw `Message` lends its payload for + the callback only; queueing it for a later pull would force a copy on a path that is + zero-copy today. + +## Consequences + +- One iteration is, in order: capture `now`; run network maintenance due at `now` (reconnect + attempts, pending disconnects); poll (Owned) or receive the caller's events (External); + route each event to its tenant or to `raw_handler`; deliver the lifecycle events those + operations produced; tick each tenant once in slice order, passing `now`; return so the + caller pulls protocol events. Transport events produced during maintenance reach a tenant + before that tenant's tick, so protocol state never lags transport state by an iteration. + Slice order may affect fairness and must never affect correctness. +- `drive`, `next_deadline` and `tick` first validate the supplied tenants against the groups + the network knows to be tenant-owned: every such group appears exactly once, or the call + panics before anything else happens — a deterministic configuration error at the first call, + never a timing-dependent one (an omitted tenant with a request deadline in flight would + otherwise never have that deadline folded). A tenant's claim is released only by `close(self, &mut StreamNetwork)`, which consumes the + tenant, hard-closes its group's connections and listeners, discards its un-pulled events + and returns the group to raw status, empty, with its handle still valid. Dropping a tenant + without closing it while the network is still driven is a programming error, and the next + driver call reports it through that same validation, naming the group; at teardown, dropping + tenants and network together is harmless because nothing is driven afterwards. Omission is + therefore never a lifecycle state. Routing is then a linear lookup by group. +- Deadlines are folded by the network from its own timers and every tenant's + `next_deadline()`; in External mode the caller folds only its own timers against the result. +- Response capability is offered only where a response is possible: `Request` and `Writable` + carry a `Responder` scoped to that connection, writing the body straight into the send + buffer; a request borrowed from the connection buffer is never copied to make a response + possible. Answering later by token remains available; a request is answered exactly once + and that is connection state, not caller choreography. +- A pulled event borrows the tenant and the network for as long as it lives, so a handler + reaches the network only through the `Responder` it was handed, and events cannot be stored + or cloned. Parsed request metadata is kept as byte ranges owned by the tenant so that an + event can outlive the parse. Dropping a `Responder` without responding defers the response + to the by-token path; a request never answered is closed by the idle sweep. +- Dynamic dispatch touches only the control path — routing, ticks, deadlines. Parsing, byte + handling and response generation stay concrete, and `next_event` runs in the same iteration + as the tick that made the connection ready. +- Readiness is tenant state, not per-iteration scratch: a caller may stop pulling after any + number of events and resume in a later iteration with nothing lost, which is what gives a + tile a per-iteration work cap. A tenant's `tick` returns `true` while it has pullable protocol + events — created by this tick or left un-pulled by a caller that stopped early — and + `drive`, `handle_event` and the network's `tick` fold that with their own actions into one + did-work result, so a tile can honour the park contract without inspecting tenant internals + and never parks on outstanding work. +- Configuration ownership follows the layers: stream transport and queue policy (socket + options, reconnect interval, framing, backlog caps, connection cap) live on the Group; HTTP + parsing and HTTP connection-state policy (head, body and header limits, idle timeout, linger + caps, request deadline) live on the tenant; nothing varies per operation until a consumer + demonstrates the need. A tenant claims a caller-created group and adds no transport settings + of its own, so two tenants with different caps coexist because they own different groups. +- The park contract this serves: a tile registers the network's (Owned) or its own + (External) `Waker` via `SpineAdapter::register_waker`, after which the tile runner stops + parking on the Signal and the tile blocks in its poll; the Signal wakes the poll on spine + work, socket readiness wakes it on I/O. diff --git a/docs/adr/0002-endpoint-closed-set.md b/docs/adr/0002-endpoint-closed-set.md new file mode 100644 index 0000000..cd126a0 --- /dev/null +++ b/docs/adr/0002-endpoint-closed-set.md @@ -0,0 +1,24 @@ +--- +status: accepted +--- + +# Transports are a closed set: `Endpoint { Tcp, Unix }` + +`StreamNetwork` listens and connects on an `Endpoint` enum — a TCP socket address or a +Unix-domain socket path — and reports accepted connections with a `Peer` enum (TCP address, +or anonymous for Unix-domain sockets), rather than being generic over a stream type. A tile +must be able to hold TCP and Unix-domain listeners in the same poll and the same group, which +a type parameter forbids; the set of transports is fixed and small, and the two mio stream +types differ only in construction, so an enum costs one match per operation and no +monomorphisation. TCP-only socket options (`TCP_NODELAY`, keepalive, `TCP_USER_TIMEOUT`) do +not exist for Unix-domain sockets; socket buffer sizes apply to both. Half-close +(`shutdown(Write)`) is required of both transports because the Lingering state depends on it. +Parsing user-facing address strings is the caller's job; the enum is the only form flux +accepts. + +## Considered options + +- **`StreamNetwork`.** Rejected: one network could not mix transports, so a tile + serving a TCP and a Unix-domain bind would need two polls. +- **Trait object per connection.** Rejected: a virtual call per read/write on the hot path + for a set that will not grow. diff --git a/docs/adr/0003-unix-socket-lifecycle.md b/docs/adr/0003-unix-socket-lifecycle.md new file mode 100644 index 0000000..f7da8e0 --- /dev/null +++ b/docs/adr/0003-unix-socket-lifecycle.md @@ -0,0 +1,34 @@ +--- +status: accepted +--- + +# Unix-domain socket files: probe then replace on bind, remove on close + +A listener binding an `Endpoint::Unix` whose path already exists first checks with `lstat` that +the existing object is a socket; anything else — a regular file, a directory, a symbolic link +even if it points at a socket — is left untouched and the bind fails with an error naming the +path, never a panic. For a socket it then connects to the path: a refused connection means the +file is a stale remnant of a process that did not clean up, so it is unlinked and the bind +proceeds; any other outcome means a live server owns the path and the bind fails with +`AddrInUse`. The `lstat` check is what makes the unlink safe, because a refused `connect` alone +proves nothing about the object's type: connecting to a regular file is refused too. Closing a +listener unlinks its path. The socket file is created with mode `0777` less the umask bits, and +a client needs write permission on it to connect, so the usual `022` umask yields `0755` — +owner-only connections — and an operator who wants group or world access sets the umask or +changes the mode; flux offers no mode or ownership setting. Outbound Unix endpoints reconnect +at the Group's interval exactly like TCP (`ENOENT` and `ECONNREFUSED` both retry). Removing the +file on close is what nginx and Go's `net.Listen` do; a stale file after a crash must not block +a restart; and an unconditional unlink would let a misconfigured second process silently take a +live node's path — the probe is the cheapest guard against that. + +## Considered options + +- **Bare `bind`, no unlink anywhere.** Rejected: every crash leaves a file that blocks the + next start until an operator removes it. +- **Unconditional unlink before bind.** Rejected: steals a live path on misconfiguration, + and the probe costs one connect. +- **Bind a temporary path and rename over the target.** Rejected: atomic, but the same steal + semantics as unconditional unlink with more code. +- **Mode and ownership settings on the listener.** Not offered: the operator controls both + through the umask and the directory. An explicit setting is recorded as a speculative + extension, not a decision. diff --git a/docs/extension-points.md b/docs/extension-points.md new file mode 100644 index 0000000..c7b7611 --- /dev/null +++ b/docs/extension-points.md @@ -0,0 +1,111 @@ +# Extension points + +Directions the networking design keeps open without deciding, in two tiers. An **eventual +feature** is wanted and unscheduled: the current design must accommodate it cheaply, so its +shape is worked out here in enough detail to check that nothing in the current stack +forecloses it. A **speculative feature** may never happen: the current design must merely not +make it impossible, and it does not drive any current decision, so it is recorded in a few +sentences. A section becomes an ADR only when it is decided, and may be dropped at any time. + +## Eventual features + +### Streamed responses + +**Why.** Two workloads need a response held open and appended to: an event stream +(server-sent events — a long-lived, mostly idle connection with small appended writes) and a +body too large to render at once (hundreds of megabytes of JSON produced in slices). The state +machine is the costly part to retrofit, so its shape is fixed here; the framing is not. + +**Shape.** The `Responder` carried by `HttpEvent::Request` gains `begin_stream(status, +headers)`; once a connection is in the **Streaming** state its `Responder` — carried by +`HttpEvent::Writable` — offers `stream(bytes)`, `stream_with(FnOnce(&mut Vec))`, +`end_stream()` and `abort_stream()`. Each is valid only in the state that admits it; a misuse +returns `false`, like a second `respond`. The same operations exist deferred by token. A +producer writes its first slices inline from the request that opened the stream, until the +connection reports full, and continues from each `Writable`. A Streaming connection stops +parsing inbound bytes as pipelined requests (they are read and discarded, as in Lingering) and +is exempt from the idle sweep, which would otherwise disconnect exactly the subscribers +behaving correctly; the exemption is per connection, so ordinary requests on the same listener +keep their timeout. The state ends when the tenant ends or aborts the stream, when the +network disconnects the peer (backlog cap or `send_timeout`), or when the peer disconnects. + +**Framing.** The caller supplies only the status and its own headers; the tenant writes the +message delimiting, chosen per stream, so the delimiting can change without touching callers. +A bulk body is chunked, because the terminal chunk is what lets a client tell a complete body +from a truncated one; after `end_stream` the connection returns to keep-alive, and a request +the client sends after the stream is served normally. An event stream is intended to be +close-delimited (no `Content-Length`, no `Transfer-Encoding`, `Connection: close`; `end_stream` +drains and closes), pending a per-client check of the event-stream consumers, with chunked as +the alternative. An HTTP/1.0 requester always gets close-delimited. + +**Backpressure to the producer.** Watermarks are Group queue policy and the network enforces +them: `StreamGroupConfig` carries a low and a high watermark below `max_backlog_bytes`; a +write that leaves a connection's backlog at or above the high watermark is reported as full, +and when a backlog that was reported full drains below the low watermark the network emits one +`StreamEvent::Writable` for that connection — an edge, never repeated while the backlog stays +low. Raw groups receive it through `raw_handler`; the HTTP tenant records it in `on_event` +and, only for a connection whose producer was refused, queues an `HttpEvent::Writable` to be +pulled. The producer keeps its own cursor and writes the next slice on each `Writable`, so +per-connection memory stays near one watermark whatever the body size, and the work done per +iteration is bounded by the slice — which is what lets a bulk render share a tile with +latency-sensitive tenants. The hard cap still disconnects a peer that stops draining; +`send_timeout`, also Group policy enforced by the network for every connection in the group, +disconnects a peer whose queue makes no progress for that long — the peer that drains too +slowly to trip the cap. + +**Abort is not end.** `abort_stream()` closes the connection without the terminal chunk, so the +client observes a truncated body. A producer whose source is invalidated mid-stream — a state +snapshot superseded while a body is half sent — cannot restart the body on the wire, and ending +the stream would forge completion. An event stream has nothing to abort; a bulk body does. + +**Consequences.** Slow event-stream consumers are disconnected by the backlog cap and reconnect +with `Last-Event-ID`; the tenant adds no stream-specific buffering. A half-open peer with an +empty backlog is invisible to both the backlog cap and `TCP_USER_TIMEOUT`, which only act with +data queued, so event-stream producers emit a periodic SSE comment line at an interval of +their choosing; the tenant does not do this for them. A client that gives up on a large body +mid-stream is the ordinary case, not an error — many consumers cap a whole request at a few +seconds — and the existing `Disconnected` handling covers it; the producer drops its cursor. +Un-pulled `Writable` events persist across iterations like any other readiness, so a tile's +per-iteration work cap applies to streams too. + +**What the current design must preserve.** The tenant, not the caller, writes +`Content-Length` and rejects caller-supplied `Content-Length` and `Transfer-Encoding`, so +delimiting stays the tenant's to choose. The `Responder` is scoped to the connection of the +event being pulled and can grow operations gated on connection state. The accepted-connection +state is an enum that can gain a variant, and the idle sweep decides per connection state. +Watermarks and `send_timeout` are `StreamGroupConfig` fields the network enforces, beside the +existing backlog cap and the send queue's age tracking; `StreamEvent` can gain a `Writable` +variant. None of this is built until the feature is scheduled; the compile-check prototype of +the `Responder` borrows covers the stream operations as well as `respond`. + +## Speculative features + +### TLS 1.3 + +Would be a per-group option on `StreamGroupConfig`, transparent to every tenant: the network +decrypts before emitting `Message` and encrypts inside its write path, emits `Accepted` and +`Connected` once the handshake completes, and sends `close_notify` before a write-side shutdown. +A sans-IO implementation behind a cargo feature; the crate is chosen if and when the work is +scheduled. Terminating TLS at a reverse proxy in front of a plain listener remains the +zero-cost alternative. Kept open by two properties the current design already has: `Message` +payloads are opaque byte chunks, and all writes go through one closure-based path. + +### Unix socket file mode + +Would be an explicit mode, and possibly owner and group, on a Unix `Endpoint`, applied at +bind instead of inherited from the process umask (ADR 0003). The umask is process-wide, so +changing it around a bind is not thread-safe; the race-free shapes are a mode applied after +bind, accepting a brief window at umask mode, or binding inside a private directory and +renaming into place. Kept open because the bind path is one function inside `StreamNetwork` +and `Endpoint::Unix` can grow options without disturbing callers that pass only a path. + +### QUIC + +Would be a sibling network beside `StreamNetwork`, sharing the poll through the same tenant +contract, never an `Endpoint` variant: QUIC multiplexes many connections over one UDP socket +and many streams per connection, which does not fit a network whose unit is an accepted byte +stream (ADR 0002). A flux application already runs a sans-IO QUIC transport in a tile of its +own to a latency standard the HTTP tenant does not need to meet; a flux `QuicNetwork` would +generalise that shape, and only if a second user needs it. Kept open by one discipline: a +token identifies the channel a request arrives on, not a TCP connection, so a multiplexing +transport can key the same router-facing events by request stream. From 4c5381837e1d515455c00d061954cfab4d7ec1d9 Mon Sep 17 00:00:00 2001 From: Bronek Kozicki Date: Wed, 26 Aug 2026 12:35:11 +0100 Subject: [PATCH 2/4] docs(network): define Service and ConnectionGroup in the glossary MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The networking glossary names a ConnectionGroup — the connections sharing one configuration and one owner, together with the listeners and outbound endpoints that produce them; a Service, the stateful protocol layer that owns one group inside a shared network and is scheduled by it; and an unclaimed ConnectionGroup, one no Service has claimed, whose events the caller takes inline through the closure it passes to drive. Owned poll and External poll name the two poll-ownership choices. ADR 0001 speaks of services with the same generality: a service is any protocol layer the network schedules, and HttpService is the first of them. Its decisions, options and consequences carry over untouched; only the vocabulary moves. ADR 0003 names the ConnectionGroup for the reconnect interval it configures. docs/extension-points.md leaves the tree. Its purpose was informational — directions the design keeps open, deciding nothing — and reviewers read that as scope instead. It stays in the maintainers' notes, where a direction can be written down without reading as a commitment, and ADR 0003 states the Unix file-mode setting as speculative in its own words. This answers the review on #135. Assisted-by: Claude Code:claude-opus-5 --- CONTEXT.md | 40 ++++--- .../0001-poll-ownership-modes-and-services.md | 113 ++++++++++++++++++ .../0001-poll-ownership-modes-and-tenancy.md | 109 ----------------- docs/adr/0003-unix-socket-lifecycle.md | 13 +- docs/extension-points.md | 111 ----------------- 5 files changed, 141 insertions(+), 245 deletions(-) create mode 100644 docs/adr/0001-poll-ownership-modes-and-services.md delete mode 100644 docs/adr/0001-poll-ownership-modes-and-tenancy.md delete mode 100644 docs/extension-points.md diff --git a/CONTEXT.md b/CONTEXT.md index 5b69df2..3c3150a 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -34,26 +34,30 @@ unit a stream network drives. A response held open for appended writes is not a sense. _Avoid_: socket (the OS handle, not the channel), pipe -**Group**: -A set of connections sharing one wire framing and one socket configuration inside a -network, with limits enforced per connection. -_Avoid_: pool, channel, protocol - -**Tenant**: -A protocol layer that owns exactly one Group inside a shared network and is scheduled by -that network; an HTTP server or client is a tenant. -_Avoid_: instance, service, sub-network - -**Raw group**: -A Group no Tenant owns; the caller consumes its events directly as they arrive, borrowing -each payload for the duration of the call. -_Avoid_: unmanaged group, bare group - -**Owned mode**: +**ConnectionGroup**: +The connections — inbound and outbound, TCP or Unix-domain — that share one configuration +(framing, socket options, backlog and connection caps) and one owner, together with the listeners +and outbound endpoints that produce them. Owned by one Service, or by the caller as an unclaimed +group. A Service requires one; you never use one bare. +_Avoid_: pool, channel, transport, stream group + +**Service**: +A stateful server, client, or both, for an application-layer protocol. It owns one ConnectionGroup +inside a shared network and is scheduled by that network. An HTTP server or client is a Service +(`HttpService`). +_Avoid_: tenant, handler, protocol + +**Unclaimed ConnectionGroup**: +A ConnectionGroup no Service has claimed; the caller is its protocol layer and receives its events +inline through the closure passed to `drive`. Claiming is a group-level decision made when a +Service is constructed and undone by `close`, never a per-connection state. +_Avoid_: raw group, unmanaged group, bare group + +**Owned poll**: A network that holds its own poll and drives it, with whatever timeout the caller passes. _Avoid_: standalone, embedded -**External mode**: +**External poll**: A network built over a poll the caller holds; the caller delivers readiness events and drives timers, and may register its own sources alongside. _Avoid_: injected, shared, hosted @@ -86,5 +90,5 @@ _Avoid_: half-close, graceful close, linger-close **Refused**: An accepted connection dropped immediately, without registration or bytes, because its -Group is at its connection cap. +ConnectionGroup is at its connection cap. _Avoid_: rejected, throttled diff --git a/docs/adr/0001-poll-ownership-modes-and-services.md b/docs/adr/0001-poll-ownership-modes-and-services.md new file mode 100644 index 0000000..868de0a --- /dev/null +++ b/docs/adr/0001-poll-ownership-modes-and-services.md @@ -0,0 +1,113 @@ +--- +status: accepted +--- + +# Poll ownership is a network mode; services are protocol layers the network schedules + +A tile that hosts sockets must own exactly one `mio::Poll`, so that under `flux/park` it can +register one `Waker` with the Signal and block in `poll` with a non-zero timeout — two polls +in one tile means neither may ever block. `StreamNetwork` therefore chooses at construction who +holds the poll: **Owned poll** (the network creates and drives its own poll, and hands out a +`Waker` on a reserved token) or **External poll** (the network is built over a `Registry` +cloned from the caller's poll and a token base, and never polls). Protocol layers do not own a +network: each is a **Service** owning one `ConnectionGroup` inside a shared `StreamNetwork`, and +the network — not the caller — schedules them. The shape is general — a service is any +protocol layer the network schedules — and `HttpService` is the first. Under Owned poll, one +call per iteration, `drive(max_timeout, services, unclaimed_handler)`, folds every deadline, polls +once, routes each event to the service owning its group, runs network maintenance and ticks each +service. Under External poll the caller makes only the three calls a caller-held poll inherently +requires — `next_deadline(services)` to fold into its own timeout, +`handle_event(&event, services, unclaimed_handler) -> bool` per readiness event (false: not ours, +route to your own sources), and `tick(services)` once per iteration — and reconstructs nothing +else. Services expose their protocol events by pull (`next_event(&mut net)`). The scheduling +hooks — group, `on_event`, `tick`, `next_deadline` — live on a trait private to flux that only +the network calls; a service hands the network an opaque `ServiceRef<'_>` (`beacon.as_service()`), +so `drive(max_timeout, &mut [beacon.as_service(), engine.as_service()], unclaimed_handler)` is the +whole public contract, and nothing outside flux can implement or invoke a hook. ConnectionGroups +no service claims are **unclaimed groups**: their events reach `unclaimed_handler` synchronously, +lending the payload for the duration of the call. + +## Considered options + +- **Owned poll only, services inside it.** Rejected: users with their own mio sources need a + caller-held poll; a foreign-source API on an owned poll would duplicate External poll with + a worse contract. +- **Per-service token ranges.** Rejected: a service's token demand (accepted connections) is + unknowable up front. One network is one contiguous token space from its base; the caller + reserves its own tokens below it. +- **Type-state `StreamNetwork` / `StreamNetwork`.** Rejected: the parameter + would infect every service signature to catch a misuse that fires on the first poll in any + test. Polling an External-poll network panics with a clear message instead. +- **Individually driven services** — the caller folds deadlines, chains `on_event` calls and + orders ticks. Rejected: every tile becomes a slightly different implementation of the + network scheduler, and the scheduling invariants live nowhere. +- **A public `Service` trait as the carrier (`&mut [&mut dyn Service]`), sealed or not.** + Rejected: an openly implementable trait commits flux to far more than four methods before + third-party services are product scope, and sealing does not help — sealing prevents + implementation, not invocation. On a trait object, supertrait methods resolve as if they were + inherent even when the supertrait is unnameable, so hooks placed on a private supertrait are + still callable by every holder of the object. Only an opaque carrier over a private trait + keeps the hooks the network's alone. +- **Releasing a claim from the network side (`release_group`), or permanent claims.** + Rejected: the former leaves a stopped service's connections delivering HTTP-framed bytes to + `unclaimed_handler` with nothing to parse them; the latter forces every service to live as long as + its network for no gain. A `Drop`-time check was rejected because it fires spuriously at + process teardown. +- **Pull-based delivery for unclaimed groups too.** Rejected: an unclaimed group's `Message` lends + its payload for the callback only; queueing it for a later pull would force a copy on a path + that is zero-copy today. + +## Consequences + +- One iteration is, in order: capture `now`; run network maintenance due at `now` (reconnect + attempts, pending disconnects); poll (Owned) or receive the caller's events (External); + route each event to its service or to `unclaimed_handler`; deliver the lifecycle events those + operations produced; tick each service once in slice order, passing `now`; return so the + caller pulls protocol events. Transport events produced during maintenance reach a service + before that service's tick, so protocol state never lags transport state by an iteration. + Slice order may affect fairness and must never affect correctness. +- `drive`, `next_deadline` and `tick` first validate the supplied services against the groups + the network knows to be service-owned: every such group appears exactly once, or the call + panics before anything else happens — a deterministic configuration error at the first call, + never a timing-dependent one (an omitted service with a request deadline in flight would + otherwise never have that deadline folded). A service's claim is released only by + `close(self, &mut StreamNetwork)`, which consumes the service, hard-closes its group's + connections and listeners, discards its un-pulled events and returns the group to unclaimed + status, empty, with its handle still valid. Dropping a service without closing it while the + network is still driven is a programming error, and the next driver call reports it through that + same validation, naming the group; at teardown, dropping services and network together is + harmless because nothing is driven afterwards. Omission is therefore never a lifecycle state. + Routing is then a linear lookup by group. +- Deadlines are folded by the network from its own timers and every service's + `next_deadline()`; under External poll the caller folds only its own timers against the result. +- Response capability is offered only where a response is possible: `Request` and `Writable` + carry a `Responder` scoped to that connection, writing the body straight into the send + buffer; a request borrowed from the connection buffer is never copied to make a response + possible. Answering later by token remains available; a request is answered exactly once + and that is connection state, not caller choreography. +- A pulled event borrows the service and the network for as long as it lives, so a handler + reaches the network only through the `Responder` it was handed, and events cannot be stored + or cloned. Parsed request metadata is kept as byte ranges owned by the service so that an + event can outlive the parse. Dropping a `Responder` without responding defers the response + to the by-token path; a request never answered is closed by the idle sweep. +- Dynamic dispatch touches only the control path — routing, ticks, deadlines. Parsing, byte + handling and response generation stay concrete, and `next_event` runs in the same iteration + as the tick that made the connection ready. +- Readiness is service state, not per-iteration scratch: a caller may stop pulling after any + number of events and resume in a later iteration with nothing lost, which is what gives a + tile a per-iteration work cap. A service's `tick` returns `true` while it has pullable protocol + events — created by this tick or left un-pulled by a caller that stopped early — and + `drive`, `handle_event` and the network's `tick` fold that with their own actions into one + did-work result, so a tile can honour the park contract without inspecting service internals + and never parks on outstanding work. +- Configuration ownership follows the layers: stream transport and queue policy (socket + options, reconnect interval, framing, backlog caps, connection cap) live on the ConnectionGroup; + HTTP parsing and HTTP connection-state policy (head, body and header limits, idle timeout, + linger caps, request deadline) live on the service; nothing varies per operation until a + consumer demonstrates the need. A service claims a caller-created group and adds no transport + settings of its own, so two services with different caps coexist because they own different + groups. +- The park contract this serves: a tile registers the network's (Owned) or its own + (External) `Waker` via `SpineAdapter::register_waker`, after which the tile runner stops + parking on the Signal and the tile blocks in its poll; the Signal wakes the poll on spine + work, socket readiness wakes it on I/O. diff --git a/docs/adr/0001-poll-ownership-modes-and-tenancy.md b/docs/adr/0001-poll-ownership-modes-and-tenancy.md deleted file mode 100644 index 585bdd8..0000000 --- a/docs/adr/0001-poll-ownership-modes-and-tenancy.md +++ /dev/null @@ -1,109 +0,0 @@ ---- -status: accepted ---- - -# Poll ownership is a network mode; protocols are tenants the network schedules - -A tile that hosts sockets must own exactly one `mio::Poll`, so that under `flux/park` it can -register one `Waker` with the Signal and block in `poll` with a non-zero timeout — two polls -in one tile means neither may ever block. `StreamNetwork` therefore chooses at construction who -holds the poll: **Owned mode** (the network creates and drives its own poll, and hands out a -`Waker` on a reserved token) or **External mode** (the network is built over a `Registry` -cloned from the caller's poll and a token base, and never polls). Protocol layers such as -`HttpTenant` do not own a network: each is a **Tenant** owning one `StreamGroup` inside a -shared `StreamNetwork`, and the network — not the caller — schedules them. In Owned mode one -call per iteration, `drive(max_timeout, tenants, raw_handler)`, folds every deadline, polls -once, routes each event to the tenant owning its group, runs network maintenance and ticks each -tenant. In External mode the caller makes only the three calls a caller-held poll inherently -requires — `next_deadline(tenants)` to fold into its own timeout, `handle_event(&event, -tenants, raw_handler) -> bool` per readiness event (false: not ours, route to your own -sources), and `tick(tenants)` once per iteration — and reconstructs nothing else. Tenants -expose their protocol events by pull (`next_event(&mut net)`). The scheduling hooks — group, -`on_event`, `tick`, `next_deadline` — live on a trait private to flux that only the network -calls; a tenant hands the network an opaque `TenantRef<'_>` (`beacon.as_tenant()`), so -`drive(max_timeout, &mut [beacon.as_tenant(), engine.as_tenant()], raw_handler)` is the whole -public contract, and nothing outside flux can implement or invoke a hook. Groups no tenant owns are **raw groups**: their -events reach `raw_handler` synchronously, lending the payload for the duration of the call. - -## Considered options - -- **Owned mode only, tenants inside it.** Rejected: users with their own mio sources need a - caller-held poll; a foreign-source API on an owned poll would duplicate External mode with - a worse contract. -- **Per-tenant token ranges.** Rejected: a tenant's token demand (accepted connections) is - unknowable up front. One network is one contiguous token space from its base; the caller - reserves its own tokens below it. -- **Type-state `StreamNetwork` / `StreamNetwork`.** Rejected: the parameter - would infect every tenant signature to catch a misuse that fires on the first poll in any - test. Polling an External-mode network panics with a clear message instead. -- **Individually driven tenants** — the caller folds deadlines, chains `on_event` calls and - orders ticks. Rejected: every tile becomes a slightly different implementation of the - network scheduler, and the scheduling invariants live nowhere. -- **A public `Tenant` trait as the carrier (`&mut [&mut dyn Tenant]`), sealed or not.** - Rejected: an openly implementable trait commits flux to far more than four methods before - third-party tenants are product scope, and sealing does not help — sealing prevents - implementation, not invocation. On a trait object, supertrait methods resolve as if they were - inherent even when the supertrait is unnameable, so hooks placed on a private supertrait are - still callable by every holder of the object. Only an opaque carrier over a private trait - keeps the hooks the network's alone. -- **Releasing a claim from the network side (`release_group`), or permanent claims.** - Rejected: the former leaves a stopped tenant's connections delivering HTTP-framed bytes to - `raw_handler` with nothing to parse them; the latter forces every tenant to live as long as - its network for no gain. A `Drop`-time check was rejected because it fires spuriously at - process teardown. -- **Pull-based delivery for raw groups too.** Rejected: a raw `Message` lends its payload for - the callback only; queueing it for a later pull would force a copy on a path that is - zero-copy today. - -## Consequences - -- One iteration is, in order: capture `now`; run network maintenance due at `now` (reconnect - attempts, pending disconnects); poll (Owned) or receive the caller's events (External); - route each event to its tenant or to `raw_handler`; deliver the lifecycle events those - operations produced; tick each tenant once in slice order, passing `now`; return so the - caller pulls protocol events. Transport events produced during maintenance reach a tenant - before that tenant's tick, so protocol state never lags transport state by an iteration. - Slice order may affect fairness and must never affect correctness. -- `drive`, `next_deadline` and `tick` first validate the supplied tenants against the groups - the network knows to be tenant-owned: every such group appears exactly once, or the call - panics before anything else happens — a deterministic configuration error at the first call, - never a timing-dependent one (an omitted tenant with a request deadline in flight would - otherwise never have that deadline folded). A tenant's claim is released only by `close(self, &mut StreamNetwork)`, which consumes the - tenant, hard-closes its group's connections and listeners, discards its un-pulled events - and returns the group to raw status, empty, with its handle still valid. Dropping a tenant - without closing it while the network is still driven is a programming error, and the next - driver call reports it through that same validation, naming the group; at teardown, dropping - tenants and network together is harmless because nothing is driven afterwards. Omission is - therefore never a lifecycle state. Routing is then a linear lookup by group. -- Deadlines are folded by the network from its own timers and every tenant's - `next_deadline()`; in External mode the caller folds only its own timers against the result. -- Response capability is offered only where a response is possible: `Request` and `Writable` - carry a `Responder` scoped to that connection, writing the body straight into the send - buffer; a request borrowed from the connection buffer is never copied to make a response - possible. Answering later by token remains available; a request is answered exactly once - and that is connection state, not caller choreography. -- A pulled event borrows the tenant and the network for as long as it lives, so a handler - reaches the network only through the `Responder` it was handed, and events cannot be stored - or cloned. Parsed request metadata is kept as byte ranges owned by the tenant so that an - event can outlive the parse. Dropping a `Responder` without responding defers the response - to the by-token path; a request never answered is closed by the idle sweep. -- Dynamic dispatch touches only the control path — routing, ticks, deadlines. Parsing, byte - handling and response generation stay concrete, and `next_event` runs in the same iteration - as the tick that made the connection ready. -- Readiness is tenant state, not per-iteration scratch: a caller may stop pulling after any - number of events and resume in a later iteration with nothing lost, which is what gives a - tile a per-iteration work cap. A tenant's `tick` returns `true` while it has pullable protocol - events — created by this tick or left un-pulled by a caller that stopped early — and - `drive`, `handle_event` and the network's `tick` fold that with their own actions into one - did-work result, so a tile can honour the park contract without inspecting tenant internals - and never parks on outstanding work. -- Configuration ownership follows the layers: stream transport and queue policy (socket - options, reconnect interval, framing, backlog caps, connection cap) live on the Group; HTTP - parsing and HTTP connection-state policy (head, body and header limits, idle timeout, linger - caps, request deadline) live on the tenant; nothing varies per operation until a consumer - demonstrates the need. A tenant claims a caller-created group and adds no transport settings - of its own, so two tenants with different caps coexist because they own different groups. -- The park contract this serves: a tile registers the network's (Owned) or its own - (External) `Waker` via `SpineAdapter::register_waker`, after which the tile runner stops - parking on the Signal and the tile blocks in its poll; the Signal wakes the poll on spine - work, socket readiness wakes it on I/O. diff --git a/docs/adr/0003-unix-socket-lifecycle.md b/docs/adr/0003-unix-socket-lifecycle.md index f7da8e0..19b0a3e 100644 --- a/docs/adr/0003-unix-socket-lifecycle.md +++ b/docs/adr/0003-unix-socket-lifecycle.md @@ -16,10 +16,10 @@ listener unlinks its path. The socket file is created with mode `0777` less the a client needs write permission on it to connect, so the usual `022` umask yields `0755` — owner-only connections — and an operator who wants group or world access sets the umask or changes the mode; flux offers no mode or ownership setting. Outbound Unix endpoints reconnect -at the Group's interval exactly like TCP (`ENOENT` and `ECONNREFUSED` both retry). Removing the -file on close is what nginx and Go's `net.Listen` do; a stale file after a crash must not block -a restart; and an unconditional unlink would let a misconfigured second process silently take a -live node's path — the probe is the cheapest guard against that. +at the ConnectionGroup's interval exactly like TCP (`ENOENT` and `ECONNREFUSED` both retry). +Removing the file on close is what nginx and Go's `net.Listen` do; a stale file after a crash must +not block a restart; and an unconditional unlink would let a misconfigured second process silently +take a live node's path — the probe is the cheapest guard against that. ## Considered options @@ -29,6 +29,5 @@ live node's path — the probe is the cheapest guard against that. and the probe costs one connect. - **Bind a temporary path and rename over the target.** Rejected: atomic, but the same steal semantics as unconditional unlink with more code. -- **Mode and ownership settings on the listener.** Not offered: the operator controls both - through the umask and the directory. An explicit setting is recorded as a speculative - extension, not a decision. +- **Mode and ownership settings on the listener.** Not offered: the operator controls both through + the umask and the directory. An explicit setting is a speculative extension, not a decision. diff --git a/docs/extension-points.md b/docs/extension-points.md deleted file mode 100644 index c7b7611..0000000 --- a/docs/extension-points.md +++ /dev/null @@ -1,111 +0,0 @@ -# Extension points - -Directions the networking design keeps open without deciding, in two tiers. An **eventual -feature** is wanted and unscheduled: the current design must accommodate it cheaply, so its -shape is worked out here in enough detail to check that nothing in the current stack -forecloses it. A **speculative feature** may never happen: the current design must merely not -make it impossible, and it does not drive any current decision, so it is recorded in a few -sentences. A section becomes an ADR only when it is decided, and may be dropped at any time. - -## Eventual features - -### Streamed responses - -**Why.** Two workloads need a response held open and appended to: an event stream -(server-sent events — a long-lived, mostly idle connection with small appended writes) and a -body too large to render at once (hundreds of megabytes of JSON produced in slices). The state -machine is the costly part to retrofit, so its shape is fixed here; the framing is not. - -**Shape.** The `Responder` carried by `HttpEvent::Request` gains `begin_stream(status, -headers)`; once a connection is in the **Streaming** state its `Responder` — carried by -`HttpEvent::Writable` — offers `stream(bytes)`, `stream_with(FnOnce(&mut Vec))`, -`end_stream()` and `abort_stream()`. Each is valid only in the state that admits it; a misuse -returns `false`, like a second `respond`. The same operations exist deferred by token. A -producer writes its first slices inline from the request that opened the stream, until the -connection reports full, and continues from each `Writable`. A Streaming connection stops -parsing inbound bytes as pipelined requests (they are read and discarded, as in Lingering) and -is exempt from the idle sweep, which would otherwise disconnect exactly the subscribers -behaving correctly; the exemption is per connection, so ordinary requests on the same listener -keep their timeout. The state ends when the tenant ends or aborts the stream, when the -network disconnects the peer (backlog cap or `send_timeout`), or when the peer disconnects. - -**Framing.** The caller supplies only the status and its own headers; the tenant writes the -message delimiting, chosen per stream, so the delimiting can change without touching callers. -A bulk body is chunked, because the terminal chunk is what lets a client tell a complete body -from a truncated one; after `end_stream` the connection returns to keep-alive, and a request -the client sends after the stream is served normally. An event stream is intended to be -close-delimited (no `Content-Length`, no `Transfer-Encoding`, `Connection: close`; `end_stream` -drains and closes), pending a per-client check of the event-stream consumers, with chunked as -the alternative. An HTTP/1.0 requester always gets close-delimited. - -**Backpressure to the producer.** Watermarks are Group queue policy and the network enforces -them: `StreamGroupConfig` carries a low and a high watermark below `max_backlog_bytes`; a -write that leaves a connection's backlog at or above the high watermark is reported as full, -and when a backlog that was reported full drains below the low watermark the network emits one -`StreamEvent::Writable` for that connection — an edge, never repeated while the backlog stays -low. Raw groups receive it through `raw_handler`; the HTTP tenant records it in `on_event` -and, only for a connection whose producer was refused, queues an `HttpEvent::Writable` to be -pulled. The producer keeps its own cursor and writes the next slice on each `Writable`, so -per-connection memory stays near one watermark whatever the body size, and the work done per -iteration is bounded by the slice — which is what lets a bulk render share a tile with -latency-sensitive tenants. The hard cap still disconnects a peer that stops draining; -`send_timeout`, also Group policy enforced by the network for every connection in the group, -disconnects a peer whose queue makes no progress for that long — the peer that drains too -slowly to trip the cap. - -**Abort is not end.** `abort_stream()` closes the connection without the terminal chunk, so the -client observes a truncated body. A producer whose source is invalidated mid-stream — a state -snapshot superseded while a body is half sent — cannot restart the body on the wire, and ending -the stream would forge completion. An event stream has nothing to abort; a bulk body does. - -**Consequences.** Slow event-stream consumers are disconnected by the backlog cap and reconnect -with `Last-Event-ID`; the tenant adds no stream-specific buffering. A half-open peer with an -empty backlog is invisible to both the backlog cap and `TCP_USER_TIMEOUT`, which only act with -data queued, so event-stream producers emit a periodic SSE comment line at an interval of -their choosing; the tenant does not do this for them. A client that gives up on a large body -mid-stream is the ordinary case, not an error — many consumers cap a whole request at a few -seconds — and the existing `Disconnected` handling covers it; the producer drops its cursor. -Un-pulled `Writable` events persist across iterations like any other readiness, so a tile's -per-iteration work cap applies to streams too. - -**What the current design must preserve.** The tenant, not the caller, writes -`Content-Length` and rejects caller-supplied `Content-Length` and `Transfer-Encoding`, so -delimiting stays the tenant's to choose. The `Responder` is scoped to the connection of the -event being pulled and can grow operations gated on connection state. The accepted-connection -state is an enum that can gain a variant, and the idle sweep decides per connection state. -Watermarks and `send_timeout` are `StreamGroupConfig` fields the network enforces, beside the -existing backlog cap and the send queue's age tracking; `StreamEvent` can gain a `Writable` -variant. None of this is built until the feature is scheduled; the compile-check prototype of -the `Responder` borrows covers the stream operations as well as `respond`. - -## Speculative features - -### TLS 1.3 - -Would be a per-group option on `StreamGroupConfig`, transparent to every tenant: the network -decrypts before emitting `Message` and encrypts inside its write path, emits `Accepted` and -`Connected` once the handshake completes, and sends `close_notify` before a write-side shutdown. -A sans-IO implementation behind a cargo feature; the crate is chosen if and when the work is -scheduled. Terminating TLS at a reverse proxy in front of a plain listener remains the -zero-cost alternative. Kept open by two properties the current design already has: `Message` -payloads are opaque byte chunks, and all writes go through one closure-based path. - -### Unix socket file mode - -Would be an explicit mode, and possibly owner and group, on a Unix `Endpoint`, applied at -bind instead of inherited from the process umask (ADR 0003). The umask is process-wide, so -changing it around a bind is not thread-safe; the race-free shapes are a mode applied after -bind, accepting a brief window at umask mode, or binding inside a private directory and -renaming into place. Kept open because the bind path is one function inside `StreamNetwork` -and `Endpoint::Unix` can grow options without disturbing callers that pass only a path. - -### QUIC - -Would be a sibling network beside `StreamNetwork`, sharing the poll through the same tenant -contract, never an `Endpoint` variant: QUIC multiplexes many connections over one UDP socket -and many streams per connection, which does not fit a network whose unit is an accepted byte -stream (ADR 0002). A flux application already runs a sans-IO QUIC transport in a tile of its -own to a latency standard the HTTP tenant does not need to meet; a flux `QuicNetwork` would -generalise that shape, and only if a second user needs it. Kept open by one discipline: a -token identifies the channel a request arrives on, not a TCP connection, so a multiplexing -transport can key the same router-facing events by request stream. From 1d31f8d9139983214c15e8bfd5b53bd86c09e7ab Mon Sep 17 00:00:00 2001 From: Bronek Kozicki Date: Tue, 25 Aug 2026 17:00:26 +0100 Subject: [PATCH 3/4] docs(adr): name the probe outcomes ADR 0003 maps to AddrInUse The probe connects without blocking, so a live owner whose accept queue is full is reported rather than waited for: a connection that completes or is left pending fails the bind with AddrInUse, and any other probe error is returned as it is, naming the path, instead of being read as a live owner. Assisted-by: Claude Code:claude-fable-5 --- docs/adr/0003-unix-socket-lifecycle.md | 28 ++++++++++++++------------ 1 file changed, 15 insertions(+), 13 deletions(-) diff --git a/docs/adr/0003-unix-socket-lifecycle.md b/docs/adr/0003-unix-socket-lifecycle.md index 19b0a3e..c41397e 100644 --- a/docs/adr/0003-unix-socket-lifecycle.md +++ b/docs/adr/0003-unix-socket-lifecycle.md @@ -7,19 +7,21 @@ status: accepted A listener binding an `Endpoint::Unix` whose path already exists first checks with `lstat` that the existing object is a socket; anything else — a regular file, a directory, a symbolic link even if it points at a socket — is left untouched and the bind fails with an error naming the -path, never a panic. For a socket it then connects to the path: a refused connection means the -file is a stale remnant of a process that did not clean up, so it is unlinked and the bind -proceeds; any other outcome means a live server owns the path and the bind fails with -`AddrInUse`. The `lstat` check is what makes the unlink safe, because a refused `connect` alone -proves nothing about the object's type: connecting to a regular file is refused too. Closing a -listener unlinks its path. The socket file is created with mode `0777` less the umask bits, and -a client needs write permission on it to connect, so the usual `022` umask yields `0755` — -owner-only connections — and an operator who wants group or world access sets the umask or -changes the mode; flux offers no mode or ownership setting. Outbound Unix endpoints reconnect -at the ConnectionGroup's interval exactly like TCP (`ENOENT` and `ECONNREFUSED` both retry). -Removing the file on close is what nginx and Go's `net.Listen` do; a stale file after a crash must -not block a restart; and an unconditional unlink would let a misconfigured second process silently -take a live node's path — the probe is the cheapest guard against that. +path, never a panic. For a socket it then connects to the path without blocking: a refused +connection means the file is a stale remnant of a process that did not clean up, so it is +unlinked and the bind proceeds; a connection that completes or is left pending — a live owner, +even one whose accept queue is full — fails the bind with `AddrInUse`; any other error from the +probe is returned as it is, naming the path. The `lstat` check is what makes the unlink safe, +because a refused `connect` alone proves nothing about the object's type: connecting to a +regular file is refused too. Closing a listener unlinks its path. The socket file is created +with mode `0777` less the umask bits, and a client needs write permission on it to connect, so +the usual `022` umask yields `0755` — owner-only connections — and an operator who wants group +or world access sets the umask or changes the mode; flux offers no mode or ownership setting. +Outbound Unix endpoints reconnect at the ConnectionGroup's interval exactly like TCP (`ENOENT` and +`ECONNREFUSED` both retry). Removing the file on close is what nginx and Go's `net.Listen` do; a +stale file after a crash must not block a restart; and an unconditional unlink would let a +misconfigured second process silently take a live node's path — the probe is the cheapest guard +against that. ## Considered options From f0410b4e11829cbd7b8b096144d95aec2c98bff6 Mon Sep 17 00:00:00 2001 From: Bronek Kozicki Date: Tue, 25 Aug 2026 22:34:26 +0100 Subject: [PATCH 4/4] docs(adr): ticks see the time after the poll wait An iteration captures the time once for validation, maintenance and the deadline fold, but the poll wait that follows may last the whole timeout, so the service ticks are given the time the wait ended: a timer a tick starts runs from the end of the wait, and one that expired during it is due in the same iteration rather than the next. Assisted-by: Claude Code:claude-fable-5 --- docs/adr/0001-poll-ownership-modes-and-services.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/adr/0001-poll-ownership-modes-and-services.md b/docs/adr/0001-poll-ownership-modes-and-services.md index 868de0a..64fc773 100644 --- a/docs/adr/0001-poll-ownership-modes-and-services.md +++ b/docs/adr/0001-poll-ownership-modes-and-services.md @@ -62,8 +62,10 @@ lending the payload for the duration of the call. - One iteration is, in order: capture `now`; run network maintenance due at `now` (reconnect attempts, pending disconnects); poll (Owned) or receive the caller's events (External); route each event to its service or to `unclaimed_handler`; deliver the lifecycle events those - operations produced; tick each service once in slice order, passing `now`; return so the - caller pulls protocol events. Transport events produced during maintenance reach a service + operations produced; tick each service once in slice order, passing the time the poll wait + ended — the wait is where an iteration spends its time, so a timer a tick starts runs from + the end of the wait, and one that expired during it is due in the same iteration; return so + the caller pulls protocol events. Transport events produced during maintenance reach a service before that service's tick, so protocol state never lags transport state by an iteration. Slice order may affect fairness and must never affect correctness. - `drive`, `next_deadline` and `tick` first validate the supplied services against the groups