Skip to content

依存関係を更新し、Redis依存をCloudflare Queuesへ移行 - #312

Open
koh110 wants to merge 50 commits into
mainfrom
dev
Open

koh110 wants to merge 50 commits into
mainfrom
dev

Conversation

@koh110

@koh110 koh110 commented Apr 24, 2026 •

Copy link
Copy Markdown
Member

概要

依存関係を更新し、削除済みRedisに依存していたevent配信・lockを、Cloudflare QueuesとMongoDB transactional outboxへ移行します。

browser
  → Cloudflare event gateway(api.mzm.dev / auth.mzm.dev)
  → Cloud Run backend/auth
  → MongoDB transaction
      ├─ business mutation
      ├─ idempotency operation
      └─ outbox events
  → gateway / scheduled relay
  → Cloudflare Queue(at-least-once)
  → queue consumer Worker
  → Cloud Run callback
  → MongoDB receipt + idempotent side effect

詳細仕様: docs/spec/cloudflare-queues-outbox-gateway.md
本番手順: docs/production-deployment.md

主な変更

Transactional outbox / idempotency

  • backend/authのbusiness mutation・operation result・outbox eventを同じMongoDB transactionでcommit
  • operation単位のstable IDとrequest hashを保存
  • 同じIdempotency-Keyの同一requestは保存済みresponseを返却
  • 同じkeyを別requestへ再利用した場合は409
  • gateway即時publishを確認できない場合は503 + Retry-After
  • scheduled relayがpending/期限切れleaseを再送
  • Queue deliveryはat-least-once、DB side effectは永続receiptで冪等化
  • DLQをR2へarchiveし、Cloudflare Access認証付きreplay endpointを追加

Cloudflare Workers

  • Event Gateway Worker
    • api.mzm.dev/* / auth.mzm.dev/*をCloud Runへstreaming proxy
    • event mutationへgateway認証とidempotency keyを付与
    • public /internal到達をpath normalization後に拒否
    • outbox claim → Queue send → ackとscheduled relay
  • Queue Worker
    • mzm-events consumer
    • retry上限後はmzm-events-dlq
    • DLQをmzm-events-dlq-archiveへ保存
    • Access JWTを検証するmanual replay endpoint
    • queue.mzm.dev custom domainで公開し、workers.devは無効化

backend / auth / frontend

  • Redis Streams / Redis lock / ioredisを削除
  • MongoDB replica setをlocal/CI/production要件化
  • callbackとoutbox内部routeをshared secretで保護
  • event-producing mutationはgateway経由を必須化
  • room名のunique indexとatomic upsertでRedis lockを置換
  • consumer side effectをtransaction + durable receiptで冪等化
  • SSE event IDのclient-side重複排除を追加

CI/CD・本番運用

  • GitHub Actions → Google Cloudを長期SA keyからOIDC/WIFへ移行
  • WIF trustを数値repository/owner ID、workflow_dispatch、対象workflow path、branch refへ限定
  • CI workflowsを全branchのpush/PRで実行可能に変更
  • Cloud Run deploy workflowsを任意branchから手動実行可能に変更し、tagは拒否
  • deploy SA権限をArtifact Registry、Cloud Run、runtime SA impersonationへ限定
  • Cloud Run imageのNode.jsをCIと同じ24へ統一
  • image内でentrypoint、Node major、backend undici importをdeploy前検証
  • bootstrap、verify-wif、internal secret初期設定などの一時operationを削除
  • 未参照repository Secretsと空のGitHub Environmentを削除
  • 通常deployとsecret rotation/DRを分離

検証

go run github.com/rhysd/actionlint/cmd/actionlint@latest .github/workflows/*.yml
npm run lint
npm run build
npm test
npm run lint -w packages/event-gateway-worker
npm test -w packages/event-gateway-worker
npm run build -w packages/event-gateway-worker
npm run lint -w packages/queue-worker
npm test -w packages/queue-worker
npm run build -w packages/queue-worker
gh pr checks 312 --repo mzmessenger/mzm

確認結果:

  • actionlint: pass
  • repository lint/build: pass
  • Worker/frontend/shared local tests: 79 passed
  • backend/authを含むGitHub Actions: 4/4 checks pass
  • Workers Builds deploy script: 9 tests passed
  • Worker tests: Event Gateway 10、Queue Worker 5 passed
  • Worker production dry-run: pass
  • Workers Builds: commit 3c9ab78b5d105da3caf4056a91b0c74c6db53149をQueue→Gatewayの順に本番反映し、check成功
  • Build token: 新規tokenは作成せず、既存tokenへWorkers R2 Storage Readのみ追加
  • Cloud Run image: Node 24、entrypoint、backend undici@8.3.0 import pass
  • branch固定scan: branches: / refs/heads/dev 0件
  • WIF provider condition: 任意branch ref、workflow_dispatch、backend/auth workflow pathをread-back済み
  • 独立security review: provider未更新の一時的不整合のみ指摘され、provider更新・read-back後に解消

本番構築状況

完了済み:

  • Cloudflare Queue mzm-events
  • Cloudflare Queue mzm-events-dlq
  • R2 bucket mzm-events-dlq-archive
  • Event Gateway Workerとapi.mzm.dev/* / auth.mzm.dev/* routes
  • Queue Workerとqueue.mzm.dev
  • Cloudflare Access operator policy / service-auth policy分離
  • service token付きpositive probeと未認証negative probe
  • GCP WIF、Artifact Registry、deploy/runtime SA分離
  • Cloud Run backend/auth Ready、traffic 100%
  • Secret Manager internal secretsとruntime SAのsecret単位access
  • Queue delivery確認、main/DLQ backlog 0
  • 旧workers.dev endpointの404確認
  • 旧SA key、bootstrap経路、一時Secret Manager admin権限の廃止

残余リスク・後続

  • Queue transportはexactly-onceではなくat-least-once。永続side effectはreceiptで冪等化する。
  • SSE messageはprocess-local fan-outのため、一時的重複があり得る。clientがevent IDで重複排除する。
  • SSEの水平scale/再接続時replayは今回のscope外。後続: koh110/memo#12
  • DLQ/prolonged relay failureの外部自動通知は未実装。
  • production deployは任意branchから可能だが、workflow_dispatch、repository/owner numeric ID、対象workflow path、branch refでWIF trustを限定する。

Merge方針

GitHub差分確認後の明示承認を受けてからmergeします。

koh110 and others added 2 commits April 24, 2026 19:44
- Updated @types/express, @types/node, concurrently, prettier in package.json
- Updated dependencies in packages/auth/package.json including connect-mongo, cors, express, express-session, ioredis, jsonwebtoken, mongodb, zod
- Updated devDependencies in packages/auth/package.json including eslint, typescript, vite, vitest
- Refactored mock imports in authorize.test.ts and github.test.ts
- Updated globalSetup.ts in packages/auth and packages/backend to use TestProject type
- Updated tsconfig.json in packages/auth and packages/backend to include rootDir and types
- Updated vitest.config.ts in packages/auth and packages/backend to simplify pool options
- Updated dependencies in packages/backend/package.json including @aws-sdk/client-s3, cors, express, ioredis, jsonwebtoken, mongodb, multer, undici, zod
- Updated test utilities in packages/backend/test/testUtil.ts for database naming conventions
- Updated tsconfig.json in packages/backend and packages/shared to include rootDir and types
- Updated vitest.config.ts in packages/backend to change pool to threads
- Updated dependencies in packages/frontend/package.json including @types/react, @vitejs/plugin-react, eslint, typescript, vitest, and others
- Refactored messageConverter.ts in packages/frontend for type safety
- Updated tsconfig.json in packages/frontend to include strict mode and adjust module resolution
- Updated dependencies in packages/shared/package.json including express, jsonwebtoken, path-to-regexp
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented May 28, 2026 •

Copy link
Copy Markdown

Deploying mzm with  Cloudflare Pages  Cloudflare Pages

Latest commit: bc5ea27
Status:🚫  Build failed.

View logs

@koh110 koh110 changed the title chore: update dependencies and devDependencies across packages 依存関係を更新し、Redis依存をCloudflare Queuesへ移行 Jul 13, 2026
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 1, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
mzm-event-gateway 3c9ab78 Aug 02 2026, 10:20 AM

This branch was previously deployed

1 inactive (outdated) deployment
production — d83b6009 Deployed Jul 31, 2026 by koh110 via configure-internal-secrets #47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant