Skip to content

feat(cloud): move storage to D1; Durable Object becomes broadcast-only - #28

Merged
JarenKempton merged 3 commits into
mainfrom
t3code/d1-cutover
Oct 12, 2026
Merged

JarenKempton merged 3 commits into
mainfrom
t3code/d1-cutover

Conversation

@JarenKempton

@JarenKempton JarenKempton commented Oct 3, 2026 •

Copy link
Copy Markdown
Owner

Warning

Merge only right after the final data pass. A merge to main deploys production. On 2026-10-12 production D1 got its schema and a full copy of the DO data (2 companies, 1 device, 47 sessions; VERIFY OK: 12 checks match). Anything clocked after that copy is only in the DO. Right before merging, pause the switch and dashboard, re-run export → import → verify (about a minute), then merge.

What

  • The Worker now reads and writes D1 directly through drizzle-orm/d1. time-clock.ts went from 1,330 lines to a 50-line hub. Queries and routes now live in feature modules (sessions/, companies/, devices/, reports/, retrievals/, live/).
  • The TimeClock Durable Object holds no tables and only broadcasts over hibernated WebSockets. The Worker calls broadcast() on it after each successful D1 write. The class name is unchanged, so its legacy SQLite storage stays reachable for 1B: Migrate DO data into D1 #21.
  • A partial unique index idx_sessions_one_open (ended_at IS NULL) enforces one open session. Concurrent starts retry, so a device flip never gets a 4xx that firmware would drop. A unique index on (company, period_start) blocks duplicate concurrent retrievals.
  • Access service tokens (CF-Access-Client-Id/Secret) can open only the read-only GET /api/v1/live stream. Every other route returns 403 service_token_read_only. Browser Access is unchanged.
  • The device API (/device/v1/*, HMAC contract) is unchanged, so firmware 0.2.5 needs no reflash.
  • Migration 0005 was generated with drizzle-kit. Deploy now runs wrangler d1 migrations apply DB --remote before wrangler deploy.

Why

Refs #20 (part of #18). Data was locked inside the DO, and the one-open-session rule only held because the DO handled requests one at a time.

How verified

Tests run against D1 (miniflare): cd cloud && pnpm test: 7 files, 33 passed. That's the 25 existing tests unchanged, plus 8 new ones:

  • A second open session inserted with raw SQL fails with UNIQUE constraint failed: index 'idx_sessions_one_open'.
  • Two concurrent dashboard starts return [201, 409 session_already_active], leaving one open row.
  • Two concurrent device flips both return 201, leaving one open row.
  • A device flip broadcasts session.stopped, then session.started, over WebSocket. The hub's sqlite_master has no tables.
  • A service token gets a 101 and a snapshot on /api/v1/live, and a 403 on GET /companies, GET /status and POST /sessions. A user JWT still gets a 200. A token with no identity gets a 401.

Mutation checks: I dropped the index from 0005, and 3 of the 4 D1 tests failed. I set the start retry to 1 attempt, and the concurrent device-flip test failed.

Local wrangler dev repro of a clock-in:

pnpm exec wrangler d1 migrations apply DB --local --persist-to /tmp/ts-d1-state   # 0000–0005 ✅
pnpm exec wrangler dev --local --port 8799 --persist-to /tmp/ts-d1-state
node repro.mjs   # creates company + device, provisions, opens ws /api/v1/live, sends HMAC-signed POST /device/v1/sessions/start

Result: device start 201. The WebSocket received snapshot (active: null), then session.started (session = the new id, active = the new id), then device.changed. Reading local D1 with wrangler d1 execute DB --local --command "SELECT id, ended_at FROM sessions WHERE ended_at IS NULL" returned [{"id":"1c6d710d-…","ended_at":null}].

Full cutover rehearsal, run locally on 2026-10-12 (current main, i.e. the production code, against the code in this PR):

# main (DO storage) on wrangler dev :8791, seeded through the real API: 2 companies (one "O'Brien & Co"),
# a device provisioned as firmware 0.2.5, 2 device flips, a dashboard stop, a pay period, then an open session
curl localhost:8791/api/v1/admin/export -o export.json        # 200: companies 2, devices 1, sessions 3, hour_retrievals 1
# this branch: fresh local D1
wrangler d1 migrations apply time-switch --local --persist-to cut-state     # 0000–0005 ✅
node scripts/d1-migrate.mjs import export.json time-switch --local --persist-to cut-state
node scripts/d1-migrate.mjs verify export.json time-switch --local --persist-to cut-state   # VERIFY OK: 12 checks match

Then I started this branch on wrangler dev :8792 against that D1, and diffed its API output with main's, ignoring only the server clock:

status IDENTICAL · sessions IDENTICAL · retrievals IDENTICAL · companies IDENTICAL · devices IDENTICAL

Signed with the device secret carried over in the export (no re-provisioning), a device start returned 201, then stop 200, then heartbeat 200. The WebSocket received snapshot, session.stopped, session.started, device.changed, session.stopped, device.changed, …. D1 now holds the 3 migrated sessions plus the new one; the open session the switch flipped away from was closed at the flip moment.

The merge commit drops the DO export endpoint from #27 on this branch: after the cutover the DO has no tables, and the export only has to run against the pre-cutover deploy.

Other gates: pnpm format:check ✅, pnpm check ✅, pnpm migrate:generate reports no schema changes ✅, pnpm deploy:check dry run ✅ (bindings include env.DB (time-switch) D1 Database).

Not verified:

  • Production deploy and the remote migrations step. I didn't run either.
  • A real Access service-token request. The JWT path is tested with a locally signed RS256 token and a mocked JWKS.
  • The physical switch on firmware 0.2.5. The HMAC contract and paths are untouched, and the signed-request tests pass.

Go-live (approved by Jaren 2026-10-12; steps 1–2 done)

Run from cloud/ on this branch. Production keeps serving from the DO until step 4.

  1. wrangler d1 migrations apply time-switch --remote: schema only, into the new empty database.
  2. Rehearsal: cloudflared access login https://time.jarenkempton.dev, then cloudflared access curl https://time.jarenkempton.dev/api/v1/admin/export -o do-export.json, node scripts/d1-migrate.mjs import do-export.json time-switch --remote, and … verify …. It must print VERIFY OK.
  3. Freeze: don't touch the switch or the dashboard. Re-export, import and verify again (it's re-runnable), then shred -u do-export.json.
  4. Merge this PR. Deploy applies migrations (a no-op by now) and switches traffic to D1. The smoke test checks D1.

Rollback: revert the merge. The DO's SQLite tables are untouched, so the old code serves them again; writes made on D1 in between would need copying back.

Risk / rollout

  • Order: see Go-live. The production deploy's CLOUDFLARE_API_TOKEN secret needs D1 Edit permission, or the new migration step fails before the deploy (safe failure: nothing ships).
  • Service tokens: the Worker trusts any service token the Access policy lets through; it doesn't keep a client-id allowlist. Add a Service Auth rule for the Mac mini's token (docs updated).
  • Behavior change: a wrong HTTP method on a browser /api route now returns 404 instead of 405. The device routes still return 405.
  • Size: about 2.2k lines added. About 1k of that is the drizzle snapshot JSON, and most of the rest is code moved out of time-clock.ts. It can't be split cleanly, because the storage swap touches every query.

🤖 Generated with Claude Code

JarenKempton and others added 3 commits October 2, 2026 22:50
…-only

The Worker now reads and writes D1 through drizzle-orm/d1. The TimeClock
Durable Object keeps its name (so its legacy storage stays addressable for
the data migration) but holds no tables; the Worker notifies it after each
successful D1 write and it fans out to hibernated WebSockets.

- Partial unique index enforces one open session; concurrent starts retry
  so a device flip never gets a 4xx the firmware would drop.
- Unique (company, period start) rejects duplicate concurrent retrievals.
- Access service tokens may open only the read-only /api/v1/live stream.
- time-clock.ts split into feature modules; device API unchanged.
- Deploy applies D1 migrations before uploading the Worker.

Refs #20

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The DO export endpoint is dropped here: after the cutover the DO holds no
tables, and the export only has to run against the pre-cutover deploy.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@JarenKempton
JarenKempton marked this pull request as ready for review October 12, 2026 18:52
@JarenKempton
JarenKempton merged commit b02038c into main Oct 12, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant