diff --git a/CONTEXT.md b/CONTEXT.md new file mode 100644 index 0000000..3c3150a --- /dev/null +++ b/CONTEXT.md @@ -0,0 +1,94 @@ +# Flux + +Flux runs latency-sensitive applications as pinned-core worker loops that exchange +messages over shared memory. This glossary fixes the words used across the workspace; +implementation detail belongs in code and ADRs, not here. + +## Language + +### Execution + +**Tile**: +One worker loop pinned to a core, with an init, a loop body, and a teardown. +_Avoid_: thread, worker, actor + +**Spine**: +The shared-memory queue fabric that connects tiles within a process. +_Avoid_: bus, channel + +**Signal**: +The process-wide sticky wake counter that idle tiles park on and that producers +increment. +_Avoid_: work signal, event, notify + +**Waker**: +The `mio::Waker` a tile registers with the Signal so that spine work interrupts the +tile's blocking poll. +_Avoid_: unparker, notifier + +### Networking + +**Stream**: +An ordered byte channel with one peer at each end — a TCP or Unix-domain connection — and the +unit a stream network drives. A response held open for appended writes is not a stream in this +sense. +_Avoid_: socket (the OS handle, not the channel), pipe + +**ConnectionGroup**: +The connections — inbound and outbound, TCP or Unix-domain — that share one configuration +(framing, socket options, backlog and connection caps) and one owner, together with the listeners +and outbound endpoints that produce them. Owned by one Service, or by the caller as an unclaimed +group. A Service requires one; you never use one bare. +_Avoid_: pool, channel, transport, stream group + +**Service**: +A stateful server, client, or both, for an application-layer protocol. It owns one ConnectionGroup +inside a shared network and is scheduled by that network. An HTTP server or client is a Service +(`HttpService`). +_Avoid_: tenant, handler, protocol + +**Unclaimed ConnectionGroup**: +A ConnectionGroup no Service has claimed; the caller is its protocol layer and receives its events +inline through the closure passed to `drive`. Claiming is a group-level decision made when a +Service is constructed and undone by `close`, never a per-connection state. +_Avoid_: raw group, unmanaged group, bare group + +**Owned poll**: +A network that holds its own poll and drives it, with whatever timeout the caller passes. +_Avoid_: standalone, embedded + +**External poll**: +A network built over a poll the caller holds; the caller delivers readiness events and +drives timers, and may register its own sources alongside. +_Avoid_: injected, shared, hosted + +**Endpoint**: +The address a listener binds or an outbound connection targets: a TCP socket address or +a Unix-domain socket path. +_Avoid_: bind, addr, address, target + +**Peer**: +The identity of the remote end of an accepted connection: a TCP socket address, or +anonymous for a Unix-domain socket. +_Avoid_: client, remote + +**Deadline**: +The per-request timer on an outbound connection; expiry fails the request and closes the +connection. +_Avoid_: timeout (which names the idle sweep), TTL + +**Draining**: +The closing state of a connection whose request stream was fully consumed: the +connection closes as soon as its queued bytes are written. +_Avoid_: flushing, closing + +**Lingering**: +The closing state of a connection whose request stream was not fully consumed: after the +response is written the write side shuts, inbound bytes are read and discarded under +idle and total caps, then the connection closes. +_Avoid_: half-close, graceful close, linger-close + +**Refused**: +An accepted connection dropped immediately, without registration or bytes, because its +ConnectionGroup is at its connection cap. +_Avoid_: rejected, throttled diff --git a/docs/adr/0001-poll-ownership-modes-and-services.md b/docs/adr/0001-poll-ownership-modes-and-services.md new file mode 100644 index 0000000..64fc773 --- /dev/null +++ b/docs/adr/0001-poll-ownership-modes-and-services.md @@ -0,0 +1,115 @@ +--- +status: accepted +--- + +# Poll ownership is a network mode; services are protocol layers the network schedules + +A tile that hosts sockets must own exactly one `mio::Poll`, so that under `flux/park` it can +register one `Waker` with the Signal and block in `poll` with a non-zero timeout — two polls +in one tile means neither may ever block. `StreamNetwork` therefore chooses at construction who +holds the poll: **Owned poll** (the network creates and drives its own poll, and hands out a +`Waker` on a reserved token) or **External poll** (the network is built over a `Registry` +cloned from the caller's poll and a token base, and never polls). Protocol layers do not own a +network: each is a **Service** owning one `ConnectionGroup` inside a shared `StreamNetwork`, and +the network — not the caller — schedules them. The shape is general — a service is any +protocol layer the network schedules — and `HttpService` is the first. Under Owned poll, one +call per iteration, `drive(max_timeout, services, unclaimed_handler)`, folds every deadline, polls +once, routes each event to the service owning its group, runs network maintenance and ticks each +service. Under External poll the caller makes only the three calls a caller-held poll inherently +requires — `next_deadline(services)` to fold into its own timeout, +`handle_event(&event, services, unclaimed_handler) -> bool` per readiness event (false: not ours, +route to your own sources), and `tick(services)` once per iteration — and reconstructs nothing +else. Services expose their protocol events by pull (`next_event(&mut net)`). The scheduling +hooks — group, `on_event`, `tick`, `next_deadline` — live on a trait private to flux that only +the network calls; a service hands the network an opaque `ServiceRef<'_>` (`beacon.as_service()`), +so `drive(max_timeout, &mut [beacon.as_service(), engine.as_service()], unclaimed_handler)` is the +whole public contract, and nothing outside flux can implement or invoke a hook. ConnectionGroups +no service claims are **unclaimed groups**: their events reach `unclaimed_handler` synchronously, +lending the payload for the duration of the call. + +## Considered options + +- **Owned poll only, services inside it.** Rejected: users with their own mio sources need a + caller-held poll; a foreign-source API on an owned poll would duplicate External poll with + a worse contract. +- **Per-service token ranges.** Rejected: a service's token demand (accepted connections) is + unknowable up front. One network is one contiguous token space from its base; the caller + reserves its own tokens below it. +- **Type-state `StreamNetwork` / `StreamNetwork`.** Rejected: the parameter + would infect every service signature to catch a misuse that fires on the first poll in any + test. Polling an External-poll network panics with a clear message instead. +- **Individually driven services** — the caller folds deadlines, chains `on_event` calls and + orders ticks. Rejected: every tile becomes a slightly different implementation of the + network scheduler, and the scheduling invariants live nowhere. +- **A public `Service` trait as the carrier (`&mut [&mut dyn Service]`), sealed or not.** + Rejected: an openly implementable trait commits flux to far more than four methods before + third-party services are product scope, and sealing does not help — sealing prevents + implementation, not invocation. On a trait object, supertrait methods resolve as if they were + inherent even when the supertrait is unnameable, so hooks placed on a private supertrait are + still callable by every holder of the object. Only an opaque carrier over a private trait + keeps the hooks the network's alone. +- **Releasing a claim from the network side (`release_group`), or permanent claims.** + Rejected: the former leaves a stopped service's connections delivering HTTP-framed bytes to + `unclaimed_handler` with nothing to parse them; the latter forces every service to live as long as + its network for no gain. A `Drop`-time check was rejected because it fires spuriously at + process teardown. +- **Pull-based delivery for unclaimed groups too.** Rejected: an unclaimed group's `Message` lends + its payload for the callback only; queueing it for a later pull would force a copy on a path + that is zero-copy today. + +## Consequences + +- One iteration is, in order: capture `now`; run network maintenance due at `now` (reconnect + attempts, pending disconnects); poll (Owned) or receive the caller's events (External); + route each event to its service or to `unclaimed_handler`; deliver the lifecycle events those + operations produced; tick each service once in slice order, passing the time the poll wait + ended — the wait is where an iteration spends its time, so a timer a tick starts runs from + the end of the wait, and one that expired during it is due in the same iteration; return so + the caller pulls protocol events. Transport events produced during maintenance reach a service + before that service's tick, so protocol state never lags transport state by an iteration. + Slice order may affect fairness and must never affect correctness. +- `drive`, `next_deadline` and `tick` first validate the supplied services against the groups + the network knows to be service-owned: every such group appears exactly once, or the call + panics before anything else happens — a deterministic configuration error at the first call, + never a timing-dependent one (an omitted service with a request deadline in flight would + otherwise never have that deadline folded). A service's claim is released only by + `close(self, &mut StreamNetwork)`, which consumes the service, hard-closes its group's + connections and listeners, discards its un-pulled events and returns the group to unclaimed + status, empty, with its handle still valid. Dropping a service without closing it while the + network is still driven is a programming error, and the next driver call reports it through that + same validation, naming the group; at teardown, dropping services and network together is + harmless because nothing is driven afterwards. Omission is therefore never a lifecycle state. + Routing is then a linear lookup by group. +- Deadlines are folded by the network from its own timers and every service's + `next_deadline()`; under External poll the caller folds only its own timers against the result. +- Response capability is offered only where a response is possible: `Request` and `Writable` + carry a `Responder` scoped to that connection, writing the body straight into the send + buffer; a request borrowed from the connection buffer is never copied to make a response + possible. Answering later by token remains available; a request is answered exactly once + and that is connection state, not caller choreography. +- A pulled event borrows the service and the network for as long as it lives, so a handler + reaches the network only through the `Responder` it was handed, and events cannot be stored + or cloned. Parsed request metadata is kept as byte ranges owned by the service so that an + event can outlive the parse. Dropping a `Responder` without responding defers the response + to the by-token path; a request never answered is closed by the idle sweep. +- Dynamic dispatch touches only the control path — routing, ticks, deadlines. Parsing, byte + handling and response generation stay concrete, and `next_event` runs in the same iteration + as the tick that made the connection ready. +- Readiness is service state, not per-iteration scratch: a caller may stop pulling after any + number of events and resume in a later iteration with nothing lost, which is what gives a + tile a per-iteration work cap. A service's `tick` returns `true` while it has pullable protocol + events — created by this tick or left un-pulled by a caller that stopped early — and + `drive`, `handle_event` and the network's `tick` fold that with their own actions into one + did-work result, so a tile can honour the park contract without inspecting service internals + and never parks on outstanding work. +- Configuration ownership follows the layers: stream transport and queue policy (socket + options, reconnect interval, framing, backlog caps, connection cap) live on the ConnectionGroup; + HTTP parsing and HTTP connection-state policy (head, body and header limits, idle timeout, + linger caps, request deadline) live on the service; nothing varies per operation until a + consumer demonstrates the need. A service claims a caller-created group and adds no transport + settings of its own, so two services with different caps coexist because they own different + groups. +- The park contract this serves: a tile registers the network's (Owned) or its own + (External) `Waker` via `SpineAdapter::register_waker`, after which the tile runner stops + parking on the Signal and the tile blocks in its poll; the Signal wakes the poll on spine + work, socket readiness wakes it on I/O. diff --git a/docs/adr/0002-endpoint-closed-set.md b/docs/adr/0002-endpoint-closed-set.md new file mode 100644 index 0000000..cd126a0 --- /dev/null +++ b/docs/adr/0002-endpoint-closed-set.md @@ -0,0 +1,24 @@ +--- +status: accepted +--- + +# Transports are a closed set: `Endpoint { Tcp, Unix }` + +`StreamNetwork` listens and connects on an `Endpoint` enum — a TCP socket address or a +Unix-domain socket path — and reports accepted connections with a `Peer` enum (TCP address, +or anonymous for Unix-domain sockets), rather than being generic over a stream type. A tile +must be able to hold TCP and Unix-domain listeners in the same poll and the same group, which +a type parameter forbids; the set of transports is fixed and small, and the two mio stream +types differ only in construction, so an enum costs one match per operation and no +monomorphisation. TCP-only socket options (`TCP_NODELAY`, keepalive, `TCP_USER_TIMEOUT`) do +not exist for Unix-domain sockets; socket buffer sizes apply to both. Half-close +(`shutdown(Write)`) is required of both transports because the Lingering state depends on it. +Parsing user-facing address strings is the caller's job; the enum is the only form flux +accepts. + +## Considered options + +- **`StreamNetwork`.** Rejected: one network could not mix transports, so a tile + serving a TCP and a Unix-domain bind would need two polls. +- **Trait object per connection.** Rejected: a virtual call per read/write on the hot path + for a set that will not grow. diff --git a/docs/adr/0003-unix-socket-lifecycle.md b/docs/adr/0003-unix-socket-lifecycle.md new file mode 100644 index 0000000..c41397e --- /dev/null +++ b/docs/adr/0003-unix-socket-lifecycle.md @@ -0,0 +1,35 @@ +--- +status: accepted +--- + +# Unix-domain socket files: probe then replace on bind, remove on close + +A listener binding an `Endpoint::Unix` whose path already exists first checks with `lstat` that +the existing object is a socket; anything else — a regular file, a directory, a symbolic link +even if it points at a socket — is left untouched and the bind fails with an error naming the +path, never a panic. For a socket it then connects to the path without blocking: a refused +connection means the file is a stale remnant of a process that did not clean up, so it is +unlinked and the bind proceeds; a connection that completes or is left pending — a live owner, +even one whose accept queue is full — fails the bind with `AddrInUse`; any other error from the +probe is returned as it is, naming the path. The `lstat` check is what makes the unlink safe, +because a refused `connect` alone proves nothing about the object's type: connecting to a +regular file is refused too. Closing a listener unlinks its path. The socket file is created +with mode `0777` less the umask bits, and a client needs write permission on it to connect, so +the usual `022` umask yields `0755` — owner-only connections — and an operator who wants group +or world access sets the umask or changes the mode; flux offers no mode or ownership setting. +Outbound Unix endpoints reconnect at the ConnectionGroup's interval exactly like TCP (`ENOENT` and +`ECONNREFUSED` both retry). Removing the file on close is what nginx and Go's `net.Listen` do; a +stale file after a crash must not block a restart; and an unconditional unlink would let a +misconfigured second process silently take a live node's path — the probe is the cheapest guard +against that. + +## Considered options + +- **Bare `bind`, no unlink anywhere.** Rejected: every crash leaves a file that blocks the + next start until an operator removes it. +- **Unconditional unlink before bind.** Rejected: steals a live path on misconfiguration, + and the probe costs one connect. +- **Bind a temporary path and rename over the target.** Rejected: atomic, but the same steal + semantics as unconditional unlink with more code. +- **Mode and ownership settings on the listener.** Not offered: the operator controls both through + the umask and the directory. An explicit setting is a speculative extension, not a decision.