Skip to content

feat(worker): add ordering lock, metrics, and health endpoints - #1196

Open
realcodesiman wants to merge 5 commits into
mainfrom
perf/worker-observability-ordering
Open

realcodesiman wants to merge 5 commits into
mainfrom
perf/worker-observability-ordering

Conversation

@realcodesiman

Copy link
Copy Markdown
Contributor

Implements P1/P2 items B3, B6, B7, B8, B9 from the system-design-review audit.

B3 — conversation ordering lock

The integration queue (concurrency 10-200) could process two webhook deliveries for the same conversation concurrently, risking out-of-order message persistence (a documented race already referenced by an isEchoOfOwnSend comment). Wraps the conversation-mutating section of apps/worker/src/integration/handlers/received-message.ts's message-ingest entry points in distributedLock.runExclusive({ key: "conversation-ingest:<id>" }), keyed by the resolved conversationId — a concurrency safety net only, no behavior change for the uncontended case.

B6 — recommend isolated production deployments

docker-entrypoint.sh already supports worker <name>, but the default is worker all (one container, one crash kills every queue). Documents the recommended production split (at least chat, integration, heavy as separate deployments) — docs only, no runtime change.

B7 — prom-client metrics

Added apps/worker/src/lib/metrics.ts (shared Registry, failed_jobs_total Counter, job_duration_seconds Histogram) and wired it into every one of the 9 apps/worker/src/*/worker.ts entrypoints' on("failed")/on("completed") handlers.

B8 — jobId in failed/completed logs

Standardized every worker's failed/completed log to carry jobId: job.id as a structured field (matching the naming already used in ai-agent/heavy), instead of only interpolating it into the message text — lets log aggregators correlate a job's failure and completion by exact field match.

B9 — health + metrics HTTP endpoints

Added apps/worker/src/lib/health-server.ts: GET /health (200 while the process's BullMQ Worker is running, 503 once closing) and GET /metrics (prom-client registry). worker all runs each queue as its own OS process, so each worker gets its own default port (CHAT_WORKER_HEALTH_PORT=3301DEFAULT_WORKER_HEALTH_PORT=3309) to avoid an EADDRINUSE collision — a worker <name> deployment only ever binds its own port. Also added an unauthenticated GET /health to apps/realtime/src/server.ts via PartyKit's onFetch (catches requests that don't match any /parties/:party/:id route), so it never touches the existing onBeforeRequest 403-by-default guard on real party traffic.

Verification

  • pnpm --filter worker check-types && test — 2340/2340 pass
  • pnpm --filter realtime check-types — clean
  • Live smoke test: started apps/worker/src/chat/worker.ts against local Postgres/Redis, curl localhost:3301/health200 ok, curl localhost:3301/metrics → real prom-client output, curl localhost:3301/nope404

@github-actions github-actions Bot added the feature New feature or request label Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant