runner.py lambda orchestrator: default max_workers exhausts client file descriptors, silently dropping cells

[espg]

Problem

_run_lambda (src/zagg/runner.py:452) fans out via ThreadPoolExecutor(max_workers=max_workers), where each worker holds an open socket to the Lambda endpoint for a synchronous invoke. The lambda-backend default is max_workers=1700 (and the CLI accepts arbitrary values).

On stock macOS / many Linux login shells the open-file limit (ulimit -n) defaults to 256. Once concurrent workers exceed it, invocations fail with:

[Errno 24] Too many open files
Could not connect to the endpoint URL: ".../functions/process-shard/invocations"

This is a client-side failure — AWS never receives those cells, so they're dropped from the output while the run still "completes." In practice --max-workers 900 lost ~430/1330 cells; even --max-workers 500 silently caps at ~246 concurrent (≈ the 256 FD limit) if ulimit isn't raised.

Why it's a footgun

• The default (1700) is far above any common ulimit -n, so the default invocation can fail on a fresh machine.
• The error reads like an AWS/network fault, not a local FD limit.
• Wall-clock is gated by the slowest single cell anyway, so high concurrency has little upside — the risky default buys nothing.

Proposed fixes (some combination)

1. Saner default — clamp default max_workers to something safe on a 256 FD limit (e.g. 128–200), or derive from resource.getrlimit(RLIMIT_NOFILE) minus headroom.
2. Bounded connection reuse — share one boto3 client with botocore.config.Config(max_pool_connections=...) across threads so FD use is bounded regardless of worker count.
3. Pre-flight — read ulimit -n at startup; if max_workers exceeds it, raise the soft limit via resource.setrlimit (up to hard) or clamp + warn clearly.
4. Clear error — catch OSError: [Errno 24] and surface "raise ulimit -n or lower --max-workers" instead of the raw connection error.

Workaround (documented)

ulimit -n 8192 before running; keep --max-workers ≤ min(ulimit -n − headroom, account concurrency). See docs/deployment/lambda.md → "Concurrency, workers, and file descriptors".

[espg]

the orchestrator needs a heavier rewrite; we should be changing for concurrency and running lambdas before submitting workers, so we can know if other processes are eating into the overall lambda concurrent workers that are allowed

[espg]

🤖 from Claude

@espg's comment shifts the scope from the FD-limit fix to a concurrency-aware orchestrator redesign. Three approaches to "know how many concurrent workers are available before submitting":

Option A: Pre-flight account concurrency query

• Call lambda.get_account_settings() for the account-level ConcurrentExecutions ceiling (no CloudWatch needed for the limit itself)
• Query CloudWatch GetMetricStatistics on AWS/Lambda:ConcurrentExecutions for a recent window (e.g., last 60s) to estimate current account-wide Lambda usage across all functions
• Compute available = ceiling - current_concurrent and set max_workers = min(requested, available - headroom)
• Re-check mid-run (e.g., every N completions) and throttle if concurrent use has grown
• Tradeoff: CloudWatch metric has ~1 min lag, so a burst of external invocations in that window can still collide. Adds cloudwatch:GetMetricStatistics IAM dependency.

Option B: Reactive backpressure (no pre-flight)

• Replace ThreadPoolExecutor with a token-based semaphore (initial capacity derived from resource.getrlimit(RLIMIT_NOFILE) and/or a safe default)
• On TooManyRequestsException (HTTP 429): return the token immediately, re-queue the cell with exponential backoff and jitter
• Track throttle rate; if it exceeds a threshold, reduce effective in-flight count dynamically
• Tradeoff: No pre-flight — relies on Lambda throttle responses as the concurrency signal. Doesn't see other processes' usage until it hits a throttle. Simpler (no CloudWatch), but the FD-limit issue must also be fixed (bounded boto3 connection pool, resource.setrlimit).

Option C: Wave-gated dispatch (synchronous batches)

• Dispatch N cells per wave, await all completions, then dispatch the next wave
• N derived from resource.getrlimit(RLIMIT_NOFILE) minus headroom (also fixes the FD issue)
• Tradeoff: Maximally safe; no Lambda throttling risk. Wall-clock is now gated by the slowest cell in each wave. Works well when per-cell latency is uniform (ATL06 HEALPix ~2–5s each); degrades if one shard hangs.

Clarifying questions before planning

1. Scope of "other processes": concern specifically other zagg runs (same function ARN), or any Lambda function in the account? get_function_concurrency vs get_account_settings differ here.
2. CloudWatch availability: does the runtime IAM role include cloudwatch:GetMetricStatistics? Many Lambda execution roles don't by default.
3. FD fix baseline: should bounded connection pool + resource.setrlimit be part of this rewrite as a baseline fix, or tracked separately from the concurrency-aware logic?
4. Probe invocation: "running lambdas before submitting workers" — does this mean a warmup/validation probe (invoke 1 cell, confirm response) before full dispatch, to validate the function is deployed and healthy before committing thousands of invocations?

Recommendation: If CloudWatch is available (Option A), it gives the strongest pre-flight guarantee and directly answers "how much quota is other processes consuming right now." If not, Option B (backpressure) combined with the FD fix is more robust than the current approach and doesn't require account-level metric read access. Option C is the fallback for environments where concurrency can't be queried at all.

---

Generated by Claude Code

[espg]

Option A is cleanest, and is how we check workers now (if we do it interactively). So let's go with A.

Scope of "other processes": concern specifically other zagg runs (same function ARN), or any Lambda function in the account? get_function_concurrency vs get_account_settings differ here.

We want both. We want to track the total number of our function that's running in the system... but we also need to make sure that we don't saturate the lambda pool, since that we cause failures (here, and for the other lambda runs). So we should report get_function_concurrency , and calculate capacity from get_account_settings + cloudwatch. The actual scaling comes from the latter (with appropriate padding).

CloudWatch availability: does the runtime IAM role include cloudwatch:GetMetricStatistics? Many Lambda execution roles don't by default.

We can modify our deplot/aws cloud formation template if needed to provision this into the role if needed-- and can check if thats even needed in the current env

FD fix baseline: should bounded connection pool + resource.setrlimit be part of this rewrite as a baseline fix, or tracked separately from the concurrency-aware logic?

we should do this: "Clear error — catch OSError: [Errno 24] and surface "raise ulimit -n or lower --max-workers" instead of the raw connection error." and also update our docs to have setting the ulimit as part of the example workflow.

Probe invocation: "running lambdas before submitting workers" — does this mean a warmup/validation probe (invoke 1 cell, confirm response) before full dispatch, to validate the function is deployed and healthy before committing thousands of invocations?

No, we're not waiting for a sample cell to complete before submitting-- these are fire and forget. We should check that we can fire the amount of them that we want. and we could do a (cheap) check on the function being deployed/callable so we don't spam the console... but not waiting for completion

[espg]

🤖 from Claude

Your answers settle it — Option A (pre-flight), report get_function_concurrency but scale from get_account_settings + CloudWatch with padding, FD fix = actionable error + docs, and a cheap deploy-exists check (no warmup wait). Implementation plan below.

Implementation plan — concurrency-aware orchestrator (#28)

Phase 1 — FD-exhaustion baseline (independent, lands first)

• Bound the boto3 pool: construct the Lambda client with botocore.config.Config(max_pool_connections=...) sized to the worker count, so connections can't outrun file descriptors.
• Catch OSError: [Errno 24] (too many open files) at dispatch and re-raise with an actionable message — "raise ulimit -n or lower --max-workers" — instead of the raw connection error.
• Docs: add ulimit -n <N> to the example run workflow.

Phase 2 — pre-flight concurrency probe (Option A)

New helper (src/zagg/concurrency.py) computing available headroom before fan-out:

• lambda.get_account_settings() → account ConcurrentExecutions ceiling + UnreservedConcurrentExecutions.
• CloudWatch GetMetricStatistics on AWS/Lambda ConcurrentExecutions (account-wide, ~last 60s, Maximum) → current usage.
• lambda.get_function_concurrency(FunctionName=...) → report our function's reserved/used (informational, per your note — not the limiter).
• available = account_limit − current_concurrent − padding; max_workers = min(requested, available). Pad to absorb the ~1-min CloudWatch lag.

Phase 3 — cheap deploy/health check (not a warmup wait)

• One get_function(FunctionName=...) before fan-out to confirm the function exists and is callable (fail fast, don't spam the console with thousands of doomed invokes). Invocations stay fire-and-forget — no waiting on a sample cell to complete.

Phase 4 — mid-run re-check

• Re-query available concurrency every N completions; throttle the in-flight count down if account-wide usage has grown. Keep reactive TooManyRequestsException backoff as the safety net beneath the pre-flight estimate.

Phase 5 — IAM (gated; touches deploy infra)

• cloudwatch:GetMetricStatistics must be on the dispatch identity's role. Per your note we may add it to deployment/aws/template.yaml only if the current role lacks it — I'll verify against the deployed role first. This is the one infra touch and runner.py lambda orchestrator: default max_workers exhausts client file descriptors, silently dropping cells #28 explicitly authorizes it (CLAUDE.md §1); it lands as its own flagged commit, not silently.

Files

src/zagg/runner.py (_run_lambda/_invoke_lambda_cell dispatch), new src/zagg/concurrency.py, deployment/aws/template.yaml (Phase 5, gated), docs, tests (mock boto3/CloudWatch).

Acceptance criteria

Dispatcher computes available concurrency pre-flight and caps workers (padded for CloudWatch lag); FD errno-24 surfaces the actionable message; missing/undeployed function fails fast; docs show ulimit -n; reactive 429 backoff retained.

Open questions

1. Padding/headroom default — fixed N workers, or a % of the account ceiling?
2. CloudWatch metric — confirm Maximum statistic over a 60s period is the right read (vs Average)?
3. Helper placement — new concurrency.py vs inline in runner.py?

---

Generated by Claude Code

[espg]

Padding/headroom default

5% of account ceiling with a floor of at least 100 unused lambda capacity

CloudWatch metric

Yes, max is fine

Helper placement — new concurrency.py vs inline in runner.py?

Unsure. One thing to keep in mind is that we'll also allow for step functions in the future, to do additional horizontal scale out. Does that planned architecture piece impact / complicate this decision, or suggest one route over the other?