Environment
- SKU: Dedicated, 3 CU (rated 2,000 actions/sec/CU → 6,000 actions/sec ceiling)
- Region: West US 2
- SDK: .NET out-of-process worker (
Microsoft.DurableTask.*), tested across SDK 1.20.1 → 1.24.2
- Workload shape: high-fanout orchestrations (~12-18 actions per workflow execution — orchestration start + activities + timers), sustained via load testing at fixed steady request rates
Summary
Over several weeks of structured load testing against a dedicated (non-shared) Durable Task Scheduler instance, we have repeatedly observed the scheduler backend itself collapsing at load levels well under its documented capacity — typically 50-85% of the rated actions/sec ceiling for the provisioned CU tier. This is not a client-side, throttling (429), or infrastructure-sizing issue on our end. We are filing this to consolidate the evidence and ask Microsoft to investigate the scheduler's own dispatch/worker-lease subsystem.
What we observed
Across at least 6 independent multi-hour load test runs (constant, unchanging offered load — no ramps, no spikes triggering the incidents), the scheduler would spontaneously degrade:
ConnectedWorkers drops to 0 (total worker-fleet disconnection from the scheduler's perspective) even though our worker fleet had 0 pod restarts, 0 crashes, and normal CPU/memory throughout.OrchestratorPendingItems balloons from ~0 to tens of thousands (one run: 0 → 19,668; another: 7 separate episodes crossed 1,000+ pending, peaking at 2,417) while offered load never changed.- Worker-side gRPC errors during these windows: high volumes of
Unavailable (StatusCode 14), NotFound ("work item not found" — implying the server lost its own work-item lease), Internal, and Cancelled. Zero DeadlineExceeded and zero ResourceExhausted on the client side, and zero throttling/429 responses — ruling out client slowness or hitting the documented rate limit.
- One run captured the scheduler returning HTTP 500 ("Bad gRPC response") on both
ScheduleNewOrchestrationInstanceAsync (create) and CompleteOrchestratorTaskAsync (complete) for a sustained multi-minute window, at only ~76% of rated capacity, while a fully healthy 30/30-pod worker fleet was capped by the server to 12-16 connected workers.
- Recovery is usually spontaneous (self-heals without any client-side restart), but recovery time is highly variable: mean ~8 minutes, median ~5 minutes, worst observed ~55 minutes, across 47 distinct episodes in one 12-hour run.
- In the worst-observed 12-hour run at ~53% of rated ceiling: workers were below full connected count for 35% of the total run duration, including 60 cumulative minutes at zero connected workers.
Why we believe this is scheduler-side, not client/infra-side
For each incident window we independently ruled out:
- Worker fleet health: 0 pod restarts, 0 crashes, flat CPU/memory, no node preemption or scale events, verified via cluster events and logs.
- Storage: DataUsedInBytes flat, nowhere near the documented per-CU storage ceiling.
- Throttling: zero
ResourceExhausted responses; the scheduler's own backend logs (reviewed with Microsoft support in one incident) confirmed no throttling occurred.
- Capacity: collapses recurred at 53%, 76%, and 85% of rated CU-based actions/sec ceiling — not just near/at 100%.
- Our request pattern: offered load was flat/constant through every collapse; the scheduler did not degrade proportionally to load, it degraded suddenly at a fixed point in time.
In one incident, Microsoft support reviewed the scheduler's own backend logs and confirmed internal failures coincident with the collapse: Orleans connectivity failures, PDP (policy decision point) authorization failures, and persistence-layer errors — all internal to the managed scheduler backend, none attributable to the client.
A secondary, self-inflicted-but-scheduler-triggered failure mode
We also identified that once the scheduler starts recovering, a client fleet resuming from a large backlog can overshoot well above steady-state throughput (in one case 1.3–2.3x baseline), which appears to re-trigger the same backend failure within 1-3 minutes — producing a sawtooth of collapse → partial recovery → re-collapse, 3 cycles in one run before finally stabilizing. This amplifies the outage duration, but the initial trigger in every case was the backend failure described above, not the client-side backlog drain.
Ask
- Guidance on the actual sustained dispatch ceiling per CU for high-fanout orchestration shapes (our measurements suggest real sustainable throughput is well under the documented 2,000 actions/sec/CU figure — closer to 50-85% before instability, and that upper end is not reliably safe either).
- Root-cause investigation into the Orleans/PDP/persistence-layer failures inside the scheduler backend that are being triggered at these load levels.
- Documentation/guidance on expected recovery-time bounds and whether client-side backoff strategies exist that are known to avoid re-triggering the cascade during recovery.
Happy to share detailed timelines, Azure Monitor metric exports, and correlated worker-side log signatures for any of the incidents referenced above if useful for investigation.
Environment
Microsoft.DurableTask.*), tested across SDK 1.20.1 → 1.24.2Summary
Over several weeks of structured load testing against a dedicated (non-shared) Durable Task Scheduler instance, we have repeatedly observed the scheduler backend itself collapsing at load levels well under its documented capacity — typically 50-85% of the rated actions/sec ceiling for the provisioned CU tier. This is not a client-side, throttling (429), or infrastructure-sizing issue on our end. We are filing this to consolidate the evidence and ask Microsoft to investigate the scheduler's own dispatch/worker-lease subsystem.
What we observed
Across at least 6 independent multi-hour load test runs (constant, unchanging offered load — no ramps, no spikes triggering the incidents), the scheduler would spontaneously degrade:
ConnectedWorkersdrops to 0 (total worker-fleet disconnection from the scheduler's perspective) even though our worker fleet had 0 pod restarts, 0 crashes, and normal CPU/memory throughout.OrchestratorPendingItemsballoons from ~0 to tens of thousands (one run: 0 → 19,668; another: 7 separate episodes crossed 1,000+ pending, peaking at 2,417) while offered load never changed.Unavailable(StatusCode 14),NotFound("work item not found" — implying the server lost its own work-item lease),Internal, andCancelled. ZeroDeadlineExceededand zeroResourceExhaustedon the client side, and zero throttling/429 responses — ruling out client slowness or hitting the documented rate limit.ScheduleNewOrchestrationInstanceAsync(create) andCompleteOrchestratorTaskAsync(complete) for a sustained multi-minute window, at only ~76% of rated capacity, while a fully healthy 30/30-pod worker fleet was capped by the server to 12-16 connected workers.Why we believe this is scheduler-side, not client/infra-side
For each incident window we independently ruled out:
ResourceExhaustedresponses; the scheduler's own backend logs (reviewed with Microsoft support in one incident) confirmed no throttling occurred.In one incident, Microsoft support reviewed the scheduler's own backend logs and confirmed internal failures coincident with the collapse: Orleans connectivity failures, PDP (policy decision point) authorization failures, and persistence-layer errors — all internal to the managed scheduler backend, none attributable to the client.
A secondary, self-inflicted-but-scheduler-triggered failure mode
We also identified that once the scheduler starts recovering, a client fleet resuming from a large backlog can overshoot well above steady-state throughput (in one case 1.3–2.3x baseline), which appears to re-trigger the same backend failure within 1-3 minutes — producing a sawtooth of collapse → partial recovery → re-collapse, 3 cycles in one run before finally stabilizing. This amplifies the outage duration, but the initial trigger in every case was the backend failure described above, not the client-side backlog drain.
Ask
Happy to share detailed timelines, Azure Monitor metric exports, and correlated worker-side log signatures for any of the incidents referenced above if useful for investigation.