OUT-4099 + OUT-4100: Assembly 429 backpressure + per-direction fan-out with per-portal fairness

[SandipBajracharya]

Implements two throughput improvements from the Performance & Reliability Improvements milestone.

OUT-4099 — Recover quickly from Assembly rate-limit errors

Assembly returns no retry-after header and recovers in ~1s, but a 429 previously fell through to blind exponential backoff and escalated to the 2–5 min task-level retry, parking files.

• A Copilot 429 now waits ~1s before retrying, and only before an actual retry (not on the final give-up).
• The Copilot path retries up to 6 times so a burst self-corrects inside withRetry.
• withRetry gains a configurable retries option; Dropbox callers keep their existing 3-retry behaviour (no blanket change).

OUT-4100 — Fan out files per direction, keyed per portal

Replaces the per-page batchTriggerAndWait convoy.

• Each direction lists all pages first, then fans out files in chunks (BATCH_CHUNK_SIZE = 150, within Trigger.dev's 500-item batch limit).
• Chunks are awaited sequentially because Trigger.dev forbids parallel waits (Promise.all around wait functions throws).
• Each file run is keyed by concurrencyKey = portalId, so one busy portal can no longer starve others in the shared 25-slot pool.
• The A→D-before-D→A ordering barrier is preserved (each direction awaits its own fan-out), and the channel is marked synced only after the fan-out completes.

Testing

• New unit tests: withRetry.test.ts (429 wait, no-wait-on-final-attempt, Dropbox-429 gating, configurable retries) and fanOut.test.ts (chunking, per-item concurrencyKey, empty short-circuit, and a guard that batches never wait in parallel — proven to fail if reverted to Promise.all).
• Full suite green: 268 unit + 179 integration, pnpm typecheck and pnpm lint clean.

Known trade-offs / follow-ups (not in this PR)

• Per-portal keying removes the global 25 concurrency cap (intended fairness change). Right-sizing the per-portal limit / overall ceiling and confirming the DB pool holds up is OUT-4102.
• At BATCH_CHUNK_SIZE = 150, heavy channels (>150 files) still have a between-batch convoy (~same granularity as main's per-page). A two-level batch fan-out would remove it — noted as a possible follow-up if heavy channels prove common.

Validation still needed

The integration harness can't model Trigger.dev's queue concurrency, batch limits, or the parallel-wait guard (inline SDK double). A heavy-file channel run against a real Trigger.dev instance is needed to confirm BATCH_CHUNK_SIZE is safe and to measure real throughput (see the manual load-testing plan).

🤖 Generated with Claude Code

[linear-code]

OUT-4099

OUT-4100

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
dropbox-integration	Ready	Preview	Aug 25, 2026 2:22pm

[greptile-apps]

Greptile Summary

The PR improves rate-limit recovery for Assembly operations and replaces page-level synchronization batches with sequential, portal-keyed fan-out.

• Adds configurable retry counts and short backpressure waits for Assembly 429 responses.
• Collects paginated work before dispatching file tasks in bounded sequential chunks.
• Preserves batch-item scheduling options while applying per-portal concurrency keys.
• Prevents malformed or incomplete listings from advancing synchronization state while retaining valid work discovered earlier.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
src/trigger/processFileSync.ts	Reworks both initial-sync directions into all-page collection followed by sequential portal-keyed fan-out, while correctly withholding completion state after listing failures.
src/lib/fanOut.ts	Adds bounded sequential batch dispatch and preserves existing item options when injecting the portal concurrency key.
src/lib/withRetry.ts	Adds configurable retry counts and Assembly-specific 429 waiting without changing the default Dropbox retry budget.
src/lib/copilot/CopilotAPI.ts	Raises the Copilot retry budget so short Assembly rate-limit bursts can recover within the API wrapper.
src/features/webhook/dropbox/lib/webhook.service.ts	Prevents Dropbox cursor advancement when the channel-change child task reports failure.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Assembly-to-Dropbox listing] --> B[Collect all valid pages]
    B --> C[Portal-keyed sequential fan-out]
    C --> D[Dropbox-to-Assembly listing]
    D --> E[Collect all valid pages]
    E --> F[Portal-keyed sequential fan-out]
    F --> G[Persist cursor and completed state]
    B -->|listing failure| H[Dispatch discovered work]
    D -->|listing failure| H
    H --> I[Fail without advancing state]

Loading

_{Reviews (3): Last reviewed commit: "fix(OUT-4100): don't advance sync state ..." | Re-trigger Greptile}

[SandipBajracharya]

@greptileai review again

[SandipBajracharya]

@greptileai review again