Skip to content

Handle EMFILE so connection pools can grow to what the machine supports, instead of capping at an arbitrary number #222

Description

@bman654

The problem with having a cap at all

clodex retains a reusable WebSocket per live conversation so later turns send only the new message.
Those connections sit in two fixed-size pools (CLODEX_WS_MAX_NURSERY_CONNECTIONS,
CLODEX_WS_MAX_CONNECTIONS). When a pool is full, the least-recently-used idle connection is dropped
and that conversation loses its cache — its next turn resends the full history over a new connection.

Every value we could pick for those caps is wrong for somebody:

  • It has to be guessed per machine (descriptor limits, memory) and per workload (how many agents the
    user actually runs). fix(oauth): keep more cached conversations alive when many subagents run at once #221 raised the nursery cap to 48 explicitly as a safety valve rather than a
    measured requirement, because nothing about the observed data identifies a correct number.
  • Getting it wrong degrades silently. Until fix(oauth): keep more cached conversations alive when many subagents run at once #221 a cap eviction logged nothing identifying it as
    one; even now, a user has to read logs to discover that a cap — not their own behaviour — cost them
    a cached conversation and a full prompt resend.
  • The user's actual intent is legible: if they start 200 agents, they want 200 agents. The resource
    cost of that concurrency is real and it is theirs to pay. Churning connections underneath the number
    they asked for costs more — in cache misses and in new-connection pressure — than holding them.

An arbitrary ceiling turns a resource question the OS can answer authoritatively into a tuning
parameter nobody can set correctly.

Proposal

Let the pools grow organically and let the operating system be the limit, reporting it clearly when it
is reached rather than quietly degrading before it.

  1. Handle EMFILE (and ENFILE) on the connection-open path. There is currently no handling
    anywhere in src/grep -rn "EMFILE" src/ returns nothing — so exhausting descriptors is an
    unhandled failure rather than a degraded mode. This is the blocking prerequisite.
  2. On EMFILE, shed load deliberately instead of failing the request: evict idle pooled
    connections (oldest first) and retry the open. That converts descriptor exhaustion into exactly the
    behaviour the caps provide today, but triggered by the real limit rather than a guess.
  3. Tell the user, once, in terms they can act on — that the machine's open-file limit was reached,
    how many connections were held, and that ulimit -n is the knob. Today the equivalent condition is
    invisible unless they go trawling diagnostics.
  4. Then remove the caps as a sizing mechanism. Keep the env vars as an escape hatch for anyone who
    wants a hard bound, but let the default be unbounded-until-the-OS-objects. The idle TTLs (5 minutes
    nursery, 30 minutes established) remain the real retention policy, and they already are: replaying
    a 27.6-hour ledger, head reuse was identical at every cap from 8 to unlimited.

Why this is safe to aim for

From the analysis in #219 and #221:

  • Head reuse over a 27.6-hour real ledger was cap-invariant — 8, 16, 24, 64, unlimited all gave
    the same reuse rate. No modelled miss at any cap was caused by an eviction.
  • The ten real cap evictions in that ledger displaced connections idle 217–284 seconds, already at
    the five-minute nursery timeout. The caps were doing nothing useful.
  • Ordinary pool occupancy was 1–4 nursery for 22 of 24 hours and peaked at 28 established.
    Real demand sits far below any cap we would set, which is precisely why a cap is the wrong place to
    express the limit.

Notes for whoever picks this up

  • Neither cap is a hard ceiling today: only idle entries are evictable, so a generation already
    exceeds its cap while every head is busy (a 16-conversation fan-out reached 11 against a cap of 8),
    and isolated sockets are never registered or counted. So "unbounded" is closer to the current
    behaviour than it sounds.
  • Memory, not just descriptors, scales with retention: each retained head holds its conversation plus
    a canonical copy for prefix comparison (fix(oauth): keep prompt caching when many subagents start at the same moment #219 added a second copy while a response is in flight).
    Descriptor exhaustion may not be the first limit reached on a memory-constrained box, so consider
    whether a heap-pressure signal belongs alongside EMFILE.
  • Descriptor limits vary enormously — a stock macOS shell commonly carries a 256 soft limit while a
    tuned one reports over a million — which is itself the argument for asking the OS instead of
    guessing.
  • The pools are per process. Several clodex processes on one machine multiply held descriptors,
    and each has its own connection pacer, so there is no account-wide or machine-wide limit today.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions