Recover transient Plane-API failures during batch-merge bursts

[abwagner]

Problem

A rapid batch-merge fans out many concurrent GitHub webhooks, each driving a state PATCH plus module-reconciliation reads. That burst can push a self-hosted Plane CE into 429s/timeouts.

Previously a transient failure became permanent state drift: the per-ref try/except in handle_pull_request logged-and-swallowed the exception, the webhook still answered HTTP 200, and GitHub (which only retries on non-2xx / timeout) never re-delivered — so the merged → Done transition vanished silently. The interleaved-within-seconds merge timestamps (adjacent PRs ~5s apart getting opposite outcomes) point to concurrent processing, not a Plane outage.

Fix — three defensive layers

1. PlaneClient retries 429/5xx/timeout responses with jittered exponential backoff, honouring Retry-After when present.
2. Non-2xx on handler error — if a ref still fails, handle_pull_request re-raises (PullRequestHandlerError) and the webhook route returns 503, so GitHub's built-in delivery retry re-runs the event. Mutations are idempotent on replay (state PATCH is set-to-target, link attach dedups). Module-reconciliation failures stay non-fatal — the ticket-state work that matters already landed.
3. Resolver cold-cache lock — asyncio.Lock(s) guard cache population so a concurrent first-hit burst collapses to one list_projects/list_states fetch instead of a self-inflicted herd on the same API the handler is trying not to overrun.

Tests

New/updated coverage for: ref-failure propagation, the non-fatal module path, the 503-vs-200 webhook mapping (test_main.py), client retry/exhaustion/Retry-After, and concurrent cold-cache single-fetch.

Full suite: 160 passed; ruff check clean.

Not included (separate feature)

Remediation #4 — a periodic merged-PR-vs-ticket-state reconciliation sweep for self-healing — is left out deliberately: it needs GitHub API access plus interval/scope design. These three layers prevent recurrence; #4 would be belt-and-suspenders.

🤖 Generated with Claude Code