Flux-network re-design

CONTEXT.md

@@ -0,0 +1,94 @@
+# Flux
+
+Flux runs latency-sensitive applications as pinned-core worker loops that exchange
+messages over shared memory. This glossary fixes the words used across the workspace;
+implementation detail belongs in code and ADRs, not here.
+
+## Language
+
+### Execution
+
+**Tile**:
+One worker loop pinned to a core, with an init, a loop body, and a teardown.
+_Avoid_: thread, worker, actor
+
+**Spine**:
+The shared-memory queue fabric that connects tiles within a process.
+_Avoid_: bus, channel
+
+**Signal**:
+The process-wide sticky wake counter that idle tiles park on and that producers
+increment.
+_Avoid_: work signal, event, notify
+
+**Waker**:
+The `mio::Waker` a tile registers with the Signal so that spine work interrupts the
+tile's blocking poll.
+_Avoid_: unparker, notifier
+
+### Networking
+
+**Stream**:
+An ordered byte channel with one peer at each end — a TCP or Unix-domain connection — and the
+unit a stream network drives. A response held open for appended writes is not a stream in this
+sense.
+_Avoid_: socket (the OS handle, not the channel), pipe
+
+**ConnectionGroup**:
+The connections — inbound and outbound, TCP or Unix-domain — that share one configuration
+(framing, socket options, backlog and connection caps) and one owner, together with the listeners
+and outbound endpoints that produce them. Owned by one Service, or by the caller as an unclaimed
+group. A Service requires one; you never use one bare.
+_Avoid_: pool, channel, transport, stream group
+
+**Service**:
+A stateful server, client, or both, for an application-layer protocol. It owns one ConnectionGroup
+inside a shared network and is scheduled by that network. An HTTP server or client is a Service
+(`HttpService`).
+_Avoid_: tenant, handler, protocol
+
+**Unclaimed ConnectionGroup**:
+A ConnectionGroup no Service has claimed; the caller is its protocol layer and receives its events
+inline through the closure passed to `drive`. Claiming is a group-level decision made when a
+Service is constructed and undone by `close`, never a per-connection state.
+_Avoid_: raw group, unmanaged group, bare group
+
+**Owned poll**:
+A network that holds its own poll and drives it, with whatever timeout the caller passes.
+_Avoid_: standalone, embedded
+
+**External poll**:
+A network built over a poll the caller holds; the caller delivers readiness events and
+drives timers, and may register its own sources alongside.
+_Avoid_: injected, shared, hosted
+
+**Endpoint**:
+The address a listener binds or an outbound connection targets: a TCP socket address or
+a Unix-domain socket path.
+_Avoid_: bind, addr, address, target
+
+**Peer**:
+The identity of the remote end of an accepted connection: a TCP socket address, or
+anonymous for a Unix-domain socket.
+_Avoid_: client, remote
+
+**Deadline**:
+The per-request timer on an outbound connection; expiry fails the request and closes the
+connection.
+_Avoid_: timeout (which names the idle sweep), TTL
+
+**Draining**:
+The closing state of a connection whose request stream was fully consumed: the
+connection closes as soon as its queued bytes are written.
+_Avoid_: flushing, closing
+
+**Lingering**:
+The closing state of a connection whose request stream was not fully consumed: after the
+response is written the write side shuts, inbound bytes are read and discarded under
+idle and total caps, then the connection closes.
+_Avoid_: half-close, graceful close, linger-close
+
+**Refused**:
+An accepted connection dropped immediately, without registration or bytes, because its
+ConnectionGroup is at its connection cap.
+_Avoid_: rejected, throttled

docs/adr/0001-poll-ownership-modes-and-services.md

@@ -0,0 +1,115 @@
+---
+status: accepted
+---
+
+# Poll ownership is a network mode; services are protocol layers the network schedules
+
+A tile that hosts sockets must own exactly one `mio::Poll`, so that under `flux/park` it can
+register one `Waker` with the Signal and block in `poll` with a non-zero timeout — two polls
+in one tile means neither may ever block. `StreamNetwork` therefore chooses at construction who
+holds the poll: **Owned poll** (the network creates and drives its own poll, and hands out a
+`Waker` on a reserved token) or **External poll** (the network is built over a `Registry`
+cloned from the caller's poll and a token base, and never polls). Protocol layers do not own a
+network: each is a **Service** owning one `ConnectionGroup` inside a shared `StreamNetwork`, and
+the network — not the caller — schedules them. The shape is general — a service is any
+protocol layer the network schedules — and `HttpService` is the first. Under Owned poll, one
+call per iteration, `drive(max_timeout, services, unclaimed_handler)`, folds every deadline, polls
+once, routes each event to the service owning its group, runs network maintenance and ticks each
+service. Under External poll the caller makes only the three calls a caller-held poll inherently
+requires — `next_deadline(services)` to fold into its own timeout,
+`handle_event(&event, services, unclaimed_handler) -> bool` per readiness event (false: not ours,
+route to your own sources), and `tick(services)` once per iteration — and reconstructs nothing
+else. Services expose their protocol events by pull (`next_event(&mut net)`). The scheduling
+hooks — group, `on_event`, `tick`, `next_deadline` — live on a trait private to flux that only
+the network calls; a service hands the network an opaque `ServiceRef<'_>` (`beacon.as_service()`),
+so `drive(max_timeout, &mut [beacon.as_service(), engine.as_service()], unclaimed_handler)` is the
+whole public contract, and nothing outside flux can implement or invoke a hook. ConnectionGroups
+no service claims are **unclaimed groups**: their events reach `unclaimed_handler` synchronously,
+lending the payload for the duration of the call.
+
+## Considered options
+
+- **Owned poll only, services inside it.** Rejected: users with their own mio sources need a
+  caller-held poll; a foreign-source API on an owned poll would duplicate External poll with
+  a worse contract.
+- **Per-service token ranges.** Rejected: a service's token demand (accepted connections) is
+  unknowable up front. One network is one contiguous token space from its base; the caller
+  reserves its own tokens below it.
+- **Type-state `StreamNetwork<Owned>` / `StreamNetwork<External>`.** Rejected: the parameter
+  would infect every service signature to catch a misuse that fires on the first poll in any
+  test. Polling an External-poll network panics with a clear message instead.
+- **Individually driven services** — the caller folds deadlines, chains `on_event` calls and
+  orders ticks. Rejected: every tile becomes a slightly different implementation of the
+  network scheduler, and the scheduling invariants live nowhere.
+- **A public `Service` trait as the carrier (`&mut [&mut dyn Service]`), sealed or not.**
+  Rejected: an openly implementable trait commits flux to far more than four methods before
+  third-party services are product scope, and sealing does not help — sealing prevents
+  implementation, not invocation. On a trait object, supertrait methods resolve as if they were
+  inherent even when the supertrait is unnameable, so hooks placed on a private supertrait are
+  still callable by every holder of the object. Only an opaque carrier over a private trait
+  keeps the hooks the network's alone.
+- **Releasing a claim from the network side (`release_group`), or permanent claims.**
+  Rejected: the former leaves a stopped service's connections delivering HTTP-framed bytes to
+  `unclaimed_handler` with nothing to parse them; the latter forces every service to live as long as
+  its network for no gain. A `Drop`-time check was rejected because it fires spuriously at
+  process teardown.
+- **Pull-based delivery for unclaimed groups too.** Rejected: an unclaimed group's `Message` lends
+  its payload for the callback only; queueing it for a later pull would force a copy on a path
+  that is zero-copy today.
+
+## Consequences
+
+- One iteration is, in order: capture `now`; run network maintenance due at `now` (reconnect
+  attempts, pending disconnects); poll (Owned) or receive the caller's events (External);
+  route each event to its service or to `unclaimed_handler`; deliver the lifecycle events those
+  operations produced; tick each service once in slice order, passing the time the poll wait
+  ended — the wait is where an iteration spends its time, so a timer a tick starts runs from
+  the end of the wait, and one that expired during it is due in the same iteration; return so
+  the caller pulls protocol events. Transport events produced during maintenance reach a service
+  before that service's tick, so protocol state never lags transport state by an iteration.
+  Slice order may affect fairness and must never affect correctness.
+- `drive`, `next_deadline` and `tick` first validate the supplied services against the groups
+  the network knows to be service-owned: every such group appears exactly once, or the call
+  panics before anything else happens — a deterministic configuration error at the first call,
+  never a timing-dependent one (an omitted service with a request deadline in flight would
+  otherwise never have that deadline folded). A service's claim is released only by
+  `close(self, &mut StreamNetwork)`, which consumes the service, hard-closes its group's
+  connections and listeners, discards its un-pulled events and returns the group to unclaimed
+  status, empty, with its handle still valid. Dropping a service without closing it while the
+  network is still driven is a programming error, and the next driver call reports it through that
+  same validation, naming the group; at teardown, dropping services and network together is
+  harmless because nothing is driven afterwards. Omission is therefore never a lifecycle state.
+  Routing is then a linear lookup by group.
+- Deadlines are folded by the network from its own timers and every service's
+  `next_deadline()`; under External poll the caller folds only its own timers against the result.
+- Response capability is offered only where a response is possible: `Request` and `Writable`
+  carry a `Responder` scoped to that connection, writing the body straight into the send
+  buffer; a request borrowed from the connection buffer is never copied to make a response
+  possible. Answering later by token remains available; a request is answered exactly once
+  and that is connection state, not caller choreography.
+- A pulled event borrows the service and the network for as long as it lives, so a handler
+  reaches the network only through the `Responder` it was handed, and events cannot be stored
+  or cloned. Parsed request metadata is kept as byte ranges owned by the service so that an
+  event can outlive the parse. Dropping a `Responder` without responding defers the response
+  to the by-token path; a request never answered is closed by the idle sweep.
+- Dynamic dispatch touches only the control path — routing, ticks, deadlines. Parsing, byte
+  handling and response generation stay concrete, and `next_event` runs in the same iteration
+  as the tick that made the connection ready.
+- Readiness is service state, not per-iteration scratch: a caller may stop pulling after any
+  number of events and resume in a later iteration with nothing lost, which is what gives a
+  tile a per-iteration work cap. A service's `tick` returns `true` while it has pullable protocol
+  events — created by this tick or left un-pulled by a caller that stopped early — and
+  `drive`, `handle_event` and the network's `tick` fold that with their own actions into one
+  did-work result, so a tile can honour the park contract without inspecting service internals
+  and never parks on outstanding work.
+- Configuration ownership follows the layers: stream transport and queue policy (socket
+  options, reconnect interval, framing, backlog caps, connection cap) live on the ConnectionGroup;
+  HTTP parsing and HTTP connection-state policy (head, body and header limits, idle timeout,
+  linger caps, request deadline) live on the service; nothing varies per operation until a
+  consumer demonstrates the need. A service claims a caller-created group and adds no transport
+  settings of its own, so two services with different caps coexist because they own different
+  groups.
+- The park contract this serves: a tile registers the network's (Owned) or its own
+  (External) `Waker` via `SpineAdapter::register_waker`, after which the tile runner stops
+  parking on the Signal and the tile blocks in its poll; the Signal wakes the poll on spine
+  work, socket readiness wakes it on I/O.

docs/adr/0002-endpoint-closed-set.md

@@ -0,0 +1,24 @@
+---
+status: accepted
+---
+
+# Transports are a closed set: `Endpoint { Tcp, Unix }`
+
+`StreamNetwork` listens and connects on an `Endpoint` enum — a TCP socket address or a
+Unix-domain socket path — and reports accepted connections with a `Peer` enum (TCP address,
+or anonymous for Unix-domain sockets), rather than being generic over a stream type. A tile
+must be able to hold TCP and Unix-domain listeners in the same poll and the same group, which
+a type parameter forbids; the set of transports is fixed and small, and the two mio stream
+types differ only in construction, so an enum costs one match per operation and no
+monomorphisation. TCP-only socket options (`TCP_NODELAY`, keepalive, `TCP_USER_TIMEOUT`) do
+not exist for Unix-domain sockets; socket buffer sizes apply to both. Half-close
+(`shutdown(Write)`) is required of both transports because the Lingering state depends on it.
+Parsing user-facing address strings is the caller's job; the enum is the only form flux
+accepts.
+
+## Considered options
+
+- **`StreamNetwork<S: Stream>`.** Rejected: one network could not mix transports, so a tile
+  serving a TCP and a Unix-domain bind would need two polls.
+- **Trait object per connection.** Rejected: a virtual call per read/write on the hot path
+  for a set that will not grow.

docs/adr/0003-unix-socket-lifecycle.md

@@ -0,0 +1,35 @@
+---
+status: accepted
+---
+
+# Unix-domain socket files: probe then replace on bind, remove on close
+
+A listener binding an `Endpoint::Unix` whose path already exists first checks with `lstat` that
+the existing object is a socket; anything else — a regular file, a directory, a symbolic link
+even if it points at a socket — is left untouched and the bind fails with an error naming the
+path, never a panic. For a socket it then connects to the path without blocking: a refused
+connection means the file is a stale remnant of a process that did not clean up, so it is
+unlinked and the bind proceeds; a connection that completes or is left pending — a live owner,
+even one whose accept queue is full — fails the bind with `AddrInUse`; any other error from the
+probe is returned as it is, naming the path. The `lstat` check is what makes the unlink safe,
+because a refused `connect` alone proves nothing about the object's type: connecting to a
+regular file is refused too. Closing a listener unlinks its path. The socket file is created
+with mode `0777` less the umask bits, and a client needs write permission on it to connect, so
+the usual `022` umask yields `0755` — owner-only connections — and an operator who wants group
+or world access sets the umask or changes the mode; flux offers no mode or ownership setting.
+Outbound Unix endpoints reconnect at the ConnectionGroup's interval exactly like TCP (`ENOENT` and
+`ECONNREFUSED` both retry). Removing the file on close is what nginx and Go's `net.Listen` do; a
+stale file after a crash must not block a restart; and an unconditional unlink would let a
+misconfigured second process silently take a live node's path — the probe is the cheapest guard
+against that.
+
+## Considered options
+
+- **Bare `bind`, no unlink anywhere.** Rejected: every crash leaves a file that blocks the
+  next start until an operator removes it.
+- **Unconditional unlink before bind.** Rejected: steals a live path on misconfiguration,
+  and the probe costs one connect.
+- **Bind a temporary path and rename over the target.** Rejected: atomic, but the same steal
+  semantics as unconditional unlink with more code.
+- **Mode and ownership settings on the listener.** Not offered: the operator controls both through
+  the umask and the directory. An explicit setting is a speculative extension, not a decision.