feat(replication): subscription health + app_versions count canary - #2822
feat(replication): subscription health + app_versions count canary#2822riderx wants to merge 4 commits into
Conversation
Slot lag alone missed dead Google apply workers. Check subscription health on the replica and compare app_versions counts (cached 5m). Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
📝 WalkthroughWalkthroughThe replication endpoint now checks subscription-worker health and ChangesReplication health monitoring
Estimated code review effort: 4 (Complex) | ~45 minutes Suggested labels: Suggested reviewers: Sequence Diagram(s)sequenceDiagram
participant AdminDashboard
participant replicationEndpoint
participant PrimaryDatabase
participant ReadReplica
AdminDashboard->>replicationEndpoint: Request replication status
replicationEndpoint->>PrimaryDatabase: Query subscription workers and app_versions count
replicationEndpoint->>ReadReplica: Validate source and query app_versions count
replicationEndpoint-->>AdminDashboard: Return slot, subscription, and canary results
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
Comment |
Visual diff passedVisual changesGenerated at 2026-08-01T14:45:24.574Z. Threshold: 0.1% pixel difference.
Commit: Open |
Merging this PR will not alter performance
Comparing Footnotes
|
Typed Hono context only exposes requestId via c.get(); use the X-Database-Source header set by getPgClient instead. Co-authored-by: Cursor <cursoragent@cursor.com>
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_ae0b7b3c-f209-4fa3-83ec-9e45c0638a0a) |
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/pages/admin/dashboard/replication.vue`:
- Around line 363-367: Update the status badge classes in the replication
dashboard template to use the configured DaisyUI `d-` prefix: change the static
`badge` class and both conditional `badge-success`/`badge-error` values in the
status span.
In `@supabase/functions/_backend/public/replication.ts`:
- Around line 534-535: Update the admin request flow around
querySubscriptionHealth and getCachedDataCanary to start both independent
operations concurrently and await their combined results, preserving the
existing subscription and dataCanary assignments and behavior.
In `@tests/replication-data-canary.unit.test.ts`:
- Around line 57-74: Move the vi.useRealTimers() cleanup from the test body into
the existing afterEach hook so it runs even when assertions in the “marks
enabled subscription without apply worker as ko” test fail; keep fake-timer
setup and system-time configuration within the test.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 312dd825-4f0d-4184-93a6-b6789b7703a4
📒 Files selected for processing (4)
docs/pr-screenshots/replication-data-canary.webpsrc/pages/admin/dashboard/replication.vuesupabase/functions/_backend/public/replication.tstests/replication-data-canary.unit.test.ts
🔗 Linked repositories identified
CodeRabbit considers these linked repositories for cross-repo context during reviews:
Cap-go/capacitor-updater(manual)
There was a problem hiding this comment.
7 issues found across 4 files
Confidence score: 3/5
- In
supabase/functions/_backend/public/replication.ts, subscription aggregation can hide an unhealthy enabled subscription when a sibling is healthy, so/replicationmay report green while an apply worker is actually down; this is the highest regression risk because it can mask real outages—treat every enabled subscription as individually required for health. - In
supabase/functions/_backend/public/replication.ts, the Cache API write is awaited without a timeout, so a cache stall can block the endpoint response even when canary counts succeeded—make the cache write explicitly best-effort by applying the existing timeout/fallback path. - In
supabase/functions/_backend/public/replication.ts,has_apply_workercurrently mixespidwith receipt state, which can classify a live worker as missing when receipt timestamps are absent and trigger misleading operator alerts—derive worker presence frompidand report receipt freshness separately. - In
src/pages/admin/dashboard/replication.vue, status rendering can mislead users (SKIPPEDshown asCache fresh) and the new badge uses unprefixed DaisyUI classes that may not style correctly, reducing dashboard trust during incidents—renderskipped/-accurately and switch badges to the project’sd-prefixed classes.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="tests/replication-data-canary.unit.test.ts">
<violation number="1" location="tests/replication-data-canary.unit.test.ts:58">
P3: The fake timers here are unnecessary — evaluateSubscriptionHealth is called with the default checkedAt and the assertions only inspect status/reasons. Because vi.useRealTimers() runs only after the assertion, a failed assertion leaves fake timers active for subsequent tests in this file (afterEach only clears the canary cache and never restores timers). Drop vi.useFakeTimers()/vi.setSystemTime()/vi.useRealTimers(), or restore timers in afterEach for isolation.</violation>
</file>
<file name="src/pages/admin/dashboard/replication.vue">
<violation number="1" location="src/pages/admin/dashboard/replication.vue:364">
P3: This project uses the DaisyUI `d-` class prefix (see the existing `d-btn` usage elsewhere in this file), but the new status badge uses unprefixed `badge`/`badge-success`/`badge-error` classes. These won't match any generated DaisyUI class, so the status cell will render unstyled.</violation>
<violation number="2" location="src/pages/admin/dashboard/replication.vue:403">
P2: When no replica connection exists, the canary card reports `Cache fresh` even though the API marks the check as `SKIPPED` and did not run a count. Rendering `skipped` (and `-` when the object is absent) would keep the cache state consistent with the health status.</violation>
</file>
<file name="supabase/functions/_backend/public/replication.ts">
<violation number="1" location="supabase/functions/_backend/public/replication.ts:152">
P1: An unhealthy enabled subscription is hidden whenever any sibling is healthy, so `/replication` can report green while an apply worker is down. Disabled rows can be ignored, but every enabled subscription should be healthy for the aggregate check to pass.</violation>
<violation number="2" location="supabase/functions/_backend/public/replication.ts:302">
P2: The subscription result can misdiagnose a live worker with no receipt timestamp as a missing worker because `has_apply_worker` combines `pid` with receipt state. Computing worker presence from `pid` and reporting a separate missing-receipt reason would preserve the requested live-PID signal and make the failure actionable.</violation>
<violation number="3" location="supabase/functions/_backend/public/replication.ts:409">
P2: A Cache API stall can keep `/replication` from responding even after the canary counts succeed because the cache write is awaited without a timeout. The cache write is best-effort here, so bounding it with the existing timeout keeps caching from becoming a health-check latency dependency.</violation>
<violation number="4" location="supabase/functions/_backend/public/replication.ts:534">
P3: The subscription health check and data canary check are independent but awaited sequentially, adding the subscription query latency to every uncached canary request. Consider running them with `Promise.all` to reduce latency on the admin request path.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
Require every enabled subscription to be healthy, split worker/receipt signals, parallelize replica checks, bound cache writes, and fix DaisyUI badge classes. Co-authored-by: Cursor <cursoragent@cursor.com>
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_db789043-67cf-4923-a635-e37d75eed8b6) |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/pages/admin/dashboard/replication.vue`:
- Line 367: Update the subscription status cell in the replication dashboard to
check sub.enabled before health status: render a neutral DISABLED badge for
disabled subscriptions, while preserving the existing success/error rendering
for enabled subscriptions.
In `@supabase/functions/_backend/public/replication.ts`:
- Around line 546-551: Update the response construction around
querySubscriptionHealth and getCachedDataCanary so their failures do not alter
the existing top-level status or HTTP status code, which must remain based on
slot health. Expose subscription and canary failure state through a separate
aggregate field, or gate any response-shape change behind an explicit version
mechanism, while preserving existing /replication clients’ behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 3b9addf6-d93b-4521-b7be-ddc6a93399a6
📒 Files selected for processing (3)
src/pages/admin/dashboard/replication.vuesupabase/functions/_backend/public/replication.tstests/replication-data-canary.unit.test.ts
🔗 Linked repositories identified
CodeRabbit considers these linked repositories for cross-repo context during reviews:
Cap-go/capacitor-updater(manual)
| {{ sub.apply_lag_seconds == null ? '-' : formatNumberValue(sub.apply_lag_seconds, { maximumFractionDigits: 1 }) }} | ||
| </td> | ||
| <td class="whitespace-nowrap px-4 py-3"> | ||
| <span class="d-badge" :class="sub.status === 'ok' ? 'd-badge-success' : 'd-badge-error'"> |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Show disabled subscriptions as disabled.
The backend excludes disabled subscriptions from aggregate health, but this cell renders every non-ok row as a red KO badge. A healthy summary can therefore show a failed-looking disabled row. Render a neutral DISABLED state when sub.enabled is false, or omit disabled subscriptions from this table.
Proposed fix
- <span class="d-badge" :class="sub.status === 'ok' ? 'd-badge-success' : 'd-badge-error'">
- {{ sub.status.toUpperCase() }}
+ <span
+ class="d-badge"
+ :class="!sub.enabled ? 'd-badge-neutral' : sub.status === 'ok' ? 'd-badge-success' : 'd-badge-error'"
+ >
+ {{ sub.enabled ? sub.status.toUpperCase() : 'DISABLED' }}
</span>📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| <span class="d-badge" :class="sub.status === 'ok' ? 'd-badge-success' : 'd-badge-error'"> | |
| <span | |
| class="d-badge" | |
| :class="!sub.enabled ? 'd-badge-neutral' : sub.status === 'ok' ? 'd-badge-success' : 'd-badge-error'" | |
| > | |
| {{ sub.enabled ? sub.status.toUpperCase() : 'DISABLED' }} | |
| </span> |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/pages/admin/dashboard/replication.vue` at line 367, Update the
subscription status cell in the replication dashboard to check sub.enabled
before health status: render a neutral DISABLED badge for disabled
subscriptions, while preserving the existing success/error rendering for enabled
subscriptions.
| const [subscriptionResult, canaryResult] = await Promise.all([ | ||
| querySubscriptionHealth(replicaDrizzle, thresholdSeconds), | ||
| getCachedDataCanary(c, drizzleClient, replicaDrizzle, replicaSource), | ||
| ]) | ||
| subscription = subscriptionResult | ||
| dataCanary = canaryResult |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Find all in-repository consumers and status assumptions for this endpoint.
rg -n -C 5 \
--glob '*.ts' --glob '*.vue' --glob '*.js' --glob '*.md' \
'/replication|ReplicationStatusResponse|data_canary|subscription' \
src supabase tests docsRepository: Cap-go/capgo.app
Length of output: 50373
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- replication implementation ---'
sed -n '1,220p' supabase/functions/_backend/public/replication.ts
sed -n '480,620p' supabase/functions/_backend/public/replication.ts
printf '%s\n' '--- status and route references ---'
rg -n -C 4 --glob '*.ts' --glob '*.tsx' --glob '*.vue' --glob '*.js' --glob '*.md' \
'ReplicationStatusResponse|replication.*status|status.*replication|dataCanary|data_canary|/replication' \
src supabase tests docs 2>/dev/null | head -n 1000
printf '%s\n' '--- current diff summary and file diff ---'
git diff --stat
git diff -- supabase/functions/_backend/public/replication.ts | sed -n '1,260p'Repository: Cap-go/capgo.app
Length of output: 39163
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- route handler and helpers ---'
sed -n '430,620p' supabase/functions/_backend/public/replication.ts
printf '%s\n' '--- recent commits ---'
git log -5 --oneline -- supabase/functions/_backend/public/replication.ts
printf '%s\n' '--- parent comparison ---'
parent="$(git rev-parse HEAD^)"
git diff --unified=20 "$parent" HEAD -- supabase/functions/_backend/public/replication.ts | sed -n '1,360p'
printf '%s\n' '--- all direct status consumers ---'
rg -n -C 5 --glob '*.ts' --glob '*.tsx' --glob '*.vue' --glob '*.js' --glob '*.md' \
'\bstatus\b' src/pages/admin/dashboard/replication.vue supabase/functions/_backend/public/replication.ts \
supabase/functions/replication/index.ts tests/replication-data-canary.unit.test.tsRepository: Cap-go/capgo.app
Length of output: 50372
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- feature commit history ---'
git show --stat --oneline d8ba617bc
git show --stat --oneline d426070ce
printf '%s\n' '--- change that introduced subscription/canary aggregate status ---'
git diff --unified=12 d8ba617bc^ d8ba617bc -- supabase/functions/_backend/public/replication.ts | sed -n '1,420p'
printf '%s\n' '--- endpoint response before the feature ---'
git show d8ba617bc^:supabase/functions/_backend/public/replication.ts | sed -n '470,590p'
printf '%s\n' '--- endpoint response at feature commit ---'
git show d8ba617bc: supabase/functions/_backend/public/replication.ts 2>/dev/null | sed -n '470,610p' || trueRepository: Cap-go/capgo.app
Length of output: 15038
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- pre-feature route file size and status logic ---'
git show d8ba617bc^:supabase/functions/_backend/public/replication.ts | wc -l
git show d8ba617bc^:supabase/functions/_backend/public/replication.ts | rg -n -C 12 \
'overallStatus|status:|return c.json|const slots|app.get'
printf '%s\n' '--- remainder of feature diff around response/status ---'
git diff --unified=20 d8ba617bc^ d8ba617bc -- supabase/functions/_backend/public/replication.ts | sed -n '380,760p'
printf '%s\n' '--- complete pre-feature response section ---'
git show d8ba617bc^:supabase/functions/_backend/public/replication.ts | tail -n 150Repository: Cap-go/capgo.app
Length of output: 16255
Preserve /replication response compatibility.
Keep the top-level status and HTTP status code based on slot health. The new subscription and canary failures must use a separate aggregate field or a version-gated response.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@supabase/functions/_backend/public/replication.ts` around lines 546 - 551,
Update the response construction around querySubscriptionHealth and
getCachedDataCanary so their failures do not alter the existing top-level status
or HTTP status code, which must remain based on slot health. Expose subscription
and canary failure state through a separate aggregate field, or gate any
response-shape change behind an explicit version mechanism, while preserving
existing /replication clients’ behavior.
Source: Coding guidelines
|
There was a problem hiding this comment.
3 issues found across 4 files
Confidence score: 3/5
- In
supabase/functions/_backend/public/replication.ts, the health check can mark an empty replica as healthy when the primary is also empty, so/replicationmay return 200 during an explicitly failing replica-empty state and mask a real outage signal — classifyreplicaCount === 0asreplica_emptybefore other health evaluation. - In
supabase/functions/_backend/public/replication.ts, disabled leftover subscriptions are currently treated as failing, which can produce false-negative replication status even when all enabled subscriptions are healthy — keep disabled leftovers non-failing (or exclude them from failing checks/display). - In
src/pages/admin/dashboard/replication.vue, the Cache row showsfreshwhendata_canaryis missing, which can mislead operators into thinking a cache count probe succeeded — preserve a missing indicator (for example-) before mapping to freshness states.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="src/pages/admin/dashboard/replication.vue">
<violation number="1" location="src/pages/admin/dashboard/replication.vue:406">
P3: When `data_canary` is absent, the Cache row reports `fresh` even though the other canary fields show no data, which can imply that a count query ran successfully. Preserve the missing state (for example, `-`) before selecting `cached` or `fresh`.</violation>
</file>
<file name="supabase/functions/_backend/public/replication.ts">
<violation number="1" location="supabase/functions/_backend/public/replication.ts:106">
P2: An empty replica is reported healthy when the primary is also empty, so `/replication` can return 200 for the explicitly failing empty-replica condition. Treat `replicaCount === 0` as `replica_empty` before evaluating the percentage drift.</violation>
<violation number="2" location="supabase/functions/_backend/public/replication.ts:137">
P2: Disabled leftover subscriptions are displayed as failing even when all enabled subscriptions are healthy, contradicting the intended “disabled leftovers are ignored” behavior. Keep disabled rows non-failing or omit them from the displayed `subscriptions` list while preserving the aggregate check for enabled subscriptions.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| const subscriptions = rows.map((row) => { | ||
| const reasons: string[] = [] | ||
| if (!row.subenabled) { | ||
| reasons.push('subscription_disabled') |
There was a problem hiding this comment.
P2: Disabled leftover subscriptions are displayed as failing even when all enabled subscriptions are healthy, contradicting the intended “disabled leftovers are ignored” behavior. Keep disabled rows non-failing or omit them from the displayed subscriptions list while preserving the aggregate check for enabled subscriptions.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At supabase/functions/_backend/public/replication.ts, line 137:
<comment>Disabled leftover subscriptions are displayed as failing even when all enabled subscriptions are healthy, contradicting the intended “disabled leftovers are ignored” behavior. Keep disabled rows non-failing or omit them from the displayed `subscriptions` list while preserving the aggregate check for enabled subscriptions.</comment>
<file context>
@@ -34,6 +89,112 @@ function toNumber(value: unknown): number | null {
+ const subscriptions = rows.map((row) => {
+ const reasons: string[] = []
+ if (!row.subenabled) {
+ reasons.push('subscription_disabled')
+ }
+ else {
</file context>
| const diffPercent = baseline === 0 ? 0 : diff / baseline | ||
| const reasons: string[] = [] | ||
|
|
||
| if (primaryCount > 0 && replicaCount === 0) |
There was a problem hiding this comment.
P2: An empty replica is reported healthy when the primary is also empty, so /replication can return 200 for the explicitly failing empty-replica condition. Treat replicaCount === 0 as replica_empty before evaluating the percentage drift.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At supabase/functions/_backend/public/replication.ts, line 106:
<comment>An empty replica is reported healthy when the primary is also empty, so `/replication` can return 200 for the explicitly failing empty-replica condition. Treat `replicaCount === 0` as `replica_empty` before evaluating the percentage drift.</comment>
<file context>
@@ -34,6 +89,112 @@ function toNumber(value: unknown): number | null {
+ const diffPercent = baseline === 0 ? 0 : diff / baseline
+ const reasons: string[] = []
+
+ if (primaryCount > 0 && replicaCount === 0)
+ reasons.push('replica_empty')
+ else if (diffPercent > thresholdPercent)
</file context>
| if (primaryCount > 0 && replicaCount === 0) | |
| if (replicaCount === 0) |
| </div> | ||
| <div class="flex justify-between gap-4"> | ||
| <span>Cache</span> | ||
| <span class="font-semibold">{{ data.data_canary?.status === 'skipped' ? 'skipped' : (data.data_canary?.cached ? 'cached' : 'fresh') }}</span> |
There was a problem hiding this comment.
P3: When data_canary is absent, the Cache row reports fresh even though the other canary fields show no data, which can imply that a count query ran successfully. Preserve the missing state (for example, -) before selecting cached or fresh.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/pages/admin/dashboard/replication.vue, line 406:
<comment>When `data_canary` is absent, the Cache row reports `fresh` even though the other canary fields show no data, which can imply that a count query ran successfully. Preserve the missing state (for example, `-`) before selecting `cached` or `fresh`.</comment>
<file context>
@@ -230,6 +317,102 @@ displayStore.defaultBack = '/dashboard'
+ </div>
+ <div class="flex justify-between gap-4">
+ <span>Cache</span>
+ <span class="font-semibold">{{ data.data_canary?.status === 'skipped' ? 'skipped' : (data.data_canary?.cached ? 'cached' : 'fresh') }}</span>
+ </div>
+ <div class="flex justify-between gap-4">
</file context>
| <span class="font-semibold">{{ data.data_canary?.status === 'skipped' ? 'skipped' : (data.data_canary?.cached ? 'cached' : 'fresh') }}</span> | |
| <span class="font-semibold">{{ !data.data_canary ? '-' : data.data_canary.status === 'skipped' ? 'skipped' : (data.data_canary.cached ? 'cached' : 'fresh') }}</span> |



Summary (AI generated)
/replicationto check Google SQL subscription apply workers (enabled + livepid+ receipt lag), not only primary slot lagapp_versionsrow counts between primary and read replica, fail when replica is empty or drift exceeds 1%COUNT(*)runs at most once per 5 minutesMotivation (AI generated)
Primary slot lag can stay green while the subscriber apply worker is dead or pointed at the wrong subscription name. That left OTA updates reading stale replica data with no alert from
/replication.Business Impact (AI generated)
Faster detection of dead/stale read replicas reduces production OTA outages and support load when replication silently stops applying.
Test Plan (AI generated)
bunx vitest run tests/replication-data-canary.unit.test.ts/replicationas platform admin with replica bindings: expectsubscription.status=okanddata_canarywith primary/replica counts/replicationtwice within 5 minutes: second response should showdata_canary.cached=true503andstatus=ko/admin/dashboard/replicationshows Subscription + Data canary cardsGenerated with AI
Made with Cursor
Summary by CodeRabbit
New Features
Documentation
Tests