Skip to content

Fix map-tile cold-render timeouts on Cloud Run - #28

Merged
ellmkay merged 1 commit into
mainfrom
fix/map-tile-cold-render-performance
Sep 24, 2026
Merged

ellmkay merged 1 commit into
mainfrom
fix/map-tile-cold-render-performance

Conversation

@ellmkay

@ellmkay ellmkay commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

Summary

Follow-up to the colour-stack release (#26), which took the /api/map-tile path down on Cloud Run revision oxygen-00036-vwj (504s at 299.99 s, Cloud SQL connector failures). Full write-up: docs/bugfix-map-tile-cold-render-cloud-timeouts.md.

  • Stop reading the OCAD blob per tile request. ensureEventMapRenderKey / ensureClubMapRenderKey / refresh* now select metadata only and fetch file_data solely when file_hash is still null. It was selected unconditionally and called three times per cache miss — ~30 MB from Cloud SQL per tile, which is what pinned the connection pool and starved tRPC.
  • Resolve map settings once per request and thread them through getMapSource / preCacheChunk / maybePreCache; preCacheConsidered is marked before the first await so bursts can't all slip past it.
  • Bounded render queue → fast 503. Semaphore.run takes maxQueue; beyond MAP_RENDER_MAX_QUEUE (default 4) waiting foreground blocks the route answers 503 Retry-After: 5 instead of sitting until Cloud Run's 300 s cap. TileLayer already honours Retry-After. Background pre-cache is never refused.
  • Composite + ink rasterised in parallel under one permit (block ~3.6 s vs ~8.5 s locally; resvg's SVG parse dominates and can't be shared).
  • Sweep: course.mapFileInfo used octet_length instead of downloading the blob for .length; course.mapMetadata memoises the auto profile classification per render key instead of re-parsing the OCAD on every page load.
  • Fixed an order-dependent assertion in ppen-import.test.ts (picked whichever imported control the DB returned first).

Local numbers on the same 9.8 MB map that failed in prod: cache hit 7–9 ms (was ~250 ms + 9.8 MB); 20-tile burst over five uncached blocks 10.8 s, all 200.

Test plan

  • pnpm build, pnpm test (605 api / 867 web / 292 shared)
  • Integration suite: 336 passed (new: render-cache blob-read guard, semaphore maxQueue, 503 + retry path, mapFileInfo.size, profile memo)
  • Full sharded E2E: 274 passed, 2 pre-existing skips
  • Docker oxygen-api / oxygen-web rebuilt and started
  • After deploy: watch /api/map-tile on a cold map — expect brief 503 bursts that resolve within ~5 s, no 504s, CPU actually busy
  • Optional pre-warm: copy map_tiles rows for render_key = dee0e6b476af702521403cf4f96e0890 from a dev DB that has viewed the same map

Made with Cursor

Every tile request pulled the whole OCAD through Cloud SQL (three times
per miss) and uncached blocks queued without limit until Cloud Run's
300 s cap killed them, taking tRPC down with them. Read metadata only,
resolve settings once per request, refuse with 503 + Retry-After when
the queue is full, and rasterise composite and ink in parallel.

Co-authored-by: Cursor <cursoragent@cursor.com>
@ellmkay
ellmkay merged commit 269e8d3 into main Sep 24, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants