Skip to content

feat(kit): async product sync with job queue, dry-run, and purge-local #127

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hyochan merged 3 commits into from
May 5, 2026
Merged

feat(kit): async product sync with job queue, dry-run, and purge-local #127

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
83d18fc
feat(kit): async product sync with job queue, dry-run, and purge-local
hyochan May 5, 2026
acaaa8e
fix(kit): address Copilot/Gemini/CodeRabbit review on async product sync
hyochan May 5, 2026
3c9421f
fix(kit): round 2 review — atomic dedup, USD guard, Modal a11y, contrast
hyochan May 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 40 additions & 5 deletions packages/docs/src/pages/docs/kit-backend.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,14 +183,49 @@ if status.active:
</li>
</ul>
<p>
The{' '}
Sync is asynchronous —{' '}
<code>
POST /v1/products/&#123;apiKey&#125;/sync/&#123;ios|android&#125;
</code>{' '}
endpoint returns the count of pulled / pushed rows plus per-product
failure messages so the dashboard surfaces upstream rejections
(price-tier conflicts, locale issues, missing review notes) without
dropping silent failures.
enqueues a job and returns <code>&#123; jobId, deduped &#125;</code>{' '}
with HTTP 202 immediately. The actual catalog walk (App Store Connect
REST or Play Developer API) runs in the background as a Convex
internalAction, writing progress and the final result back to a{' '}
<code>productSyncJobs</code> row. Earlier kit versions held the HTTP
connection open for the entire sync, which iOS Safari aborted on
cellular / backgrounded tabs as <code>TypeError: Load failed</code>.
</p>
<p>Clients poll the job state:</p>
<ul>
<li>
<code>
GET /v1/products/&#123;apiKey&#125;/sync/jobs/&#123;jobId&#125;
</code>{' '}
— current status (<code>queued</code> / <code>running</code> /{' '}
<code>succeeded</code> / <code>failed</code>),{' '}
<code>progress.phase</code>, and on terminal status the{' '}
<code>result</code> object with <code>pulled</code> /{' '}
<code>pushed</code> counts and per-product <code>failures</code>{' '}
(price-tier conflicts, locale issues, missing review notes). When
more than 200 products fail, <code>failuresTruncated: true</code> is
set and the array is capped.
</li>
<li>
<code>
POST
/v1/products/&#123;apiKey&#125;/sync/jobs/&#123;jobId&#125;/cancel
</code>{' '}
— request a cancel; the worker checks at phase boundaries (PULL.iaps
→ PULL.subscriptions → PUSH.drafts) and stops within seconds.
</li>
</ul>
<p>
Backoff polls at ~3s intervals until <code>status</code> is{' '}
<code>succeeded</code> or <code>failed</code>. Most catalogs finish in
tens of seconds; large ones in 1–2 minutes. A{' '}
<code>reapStaleProductSyncJobs</code> cron flips workers stuck past
the 9-minute deadline to <code>failed</code> so a crashed action can't
pin the project's "active job" slot.
Comment thread
hyochan marked this conversation as resolved.
</p>
</section>

Expand Down
53 changes: 53 additions & 0 deletions packages/kit/CONVENTION.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,59 @@ boots it on port 3100, and probes `/health`, `/`, `/api/v1` — catches
startup regressions (missing env, bind conflicts, missing
`dist/index.html`).

## Long-running operations

Convex actions cap at ~10 minutes; the browser fetch holding an
action result open is bounded much more aggressively (iOS Safari
aborts pending fetches when a tab backgrounds or the network
flips, surfacing as `TypeError: Load failed`). Anything that walks
an external catalog or fans out per-product API calls — App Store
Connect / Play Console sync, Meta Horizon reconciliation, future
Stripe price sync — must run as a background job, not as a
synchronous public action the dashboard awaits.

Pattern (mirrors `convex/products/jobs.ts` + `runProductSyncIOS` /
`runProductSyncAndroid`):

1. **Schema**: a `*Jobs` table with `status`
(`queued | running | succeeded | failed`), `progress` (`{phase,
current?, total?, failuresCount?}`), `result?`, `error?`,
`cancelRequested?`, `expectedDeadline?`, `createdBy?`,
`startedAt?`, `completedAt?`, `createdAt`. Indexes:
`(projectId, platform, status)` for active-job lookup,
`(status, expectedDeadline)` for the reaper,
`(status, completedAt)` for the pruner.
2. **Enqueue mutation**: validates membership, dedups against an
existing `queued`/`running` row for the same `(projectId,
platform)`, inserts the row, schedules the worker via
`ctx.scheduler.runAfter(0, internal.<module>.runX, { jobId })`.
Returns `{ jobId, deduped }`.
3. **Worker internalAction**: `args: { jobId }`. Reads job →
resolves project → runs the work, calling
`updateJobProgress` at phase boundaries and
`isCancelRequested` between phases. Wraps the body in
try/catch and finishes via `markJobSucceeded` /
`markJobFailed` so a thrown error never leaves the row in
`running` forever.
4. **Cron pair**: `reapStaleProductSyncJobs` (5 min, flips
`running` rows past `expectedDeadline + grace` to `failed`)
and `pruneProductSyncJobs` (6 h, deletes `succeeded` rows
older than 7 d / `failed` rows older than 30 d).
5. **Dashboard**: `useQuery(getActiveSyncJob)` for the reactive
button state + progress label; `useMutation(enqueue*)` to
start; `useMutation(cancel*)` to stop. The completion toast
fires once via a `useRef`-gated `useEffect` so reactive
updates don't re-toast.
6. **HTTP**: `POST .../sync/...` returns 202 with `{ jobId,
deduped }`; `GET .../sync/jobs/{jobId}` polls; `POST
.../sync/jobs/{jobId}/cancel` cancels. Clients backoff at ~3 s
intervals.

Failures arrays should pass through `truncateFailures` (cap 200,
sets `failuresTruncated: true`) so a runaway sync where every
product fails for the same reason can't blow past Convex's
per-document size budget.

## Commit messages

Follow the monorepo-wide convention from the root
Expand Down
2 changes: 2 additions & 0 deletions packages/kit/convex/_generated/api.d.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ import type * as organizations_mutation from "../organizations/mutation.js";
import type * as organizations_query from "../organizations/query.js";
import type * as plans from "../plans.js";
import type * as products_asc from "../products/asc.js";
import type * as products_jobs from "../products/jobs.js";
import type * as products_jwt from "../products/jwt.js";
import type * as products_mutation from "../products/mutation.js";
import type * as products_play from "../products/play.js";
Expand Down Expand Up @@ -106,6 +107,7 @@ declare const fullApi: ApiFromModules<{
"organizations/query": typeof organizations_query;
plans: typeof plans;
"products/asc": typeof products_asc;
"products/jobs": typeof products_jobs;
"products/jwt": typeof products_jwt;
"products/mutation": typeof products_mutation;
"products/play": typeof products_play;
Expand Down
19 changes: 19 additions & 0 deletions packages/kit/convex/crons.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,4 +81,23 @@ crons.interval(
{ batchSize: 50 },
);

// Mark stuck product-sync jobs as failed. Convex caps actions at
// ~10min; the worker sets `expectedDeadline = startedAt + 9min`,
// and this reaper flips anything still `running` past
// `deadline + 1min` to failed("worker timed out"). Without it, a
// crashed action permanently pins the project's "active job" slot
// and the dashboard's button stays disabled forever.
crons.interval(
"reap stale product sync jobs",
{ minutes: 5 },
internal.products.jobs.reapStaleProductSyncJobs,
);

// Drop succeeded jobs after 7d, failed after 30d.
crons.interval(
"prune product sync jobs past retention",
{ hours: 6 },
internal.products.jobs.pruneProductSyncJobs,
);

export default crons;
26 changes: 23 additions & 3 deletions packages/kit/convex/files/internal.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -179,10 +179,30 @@ export const readFileAsBase64 = internalAction({
// accessCount: (file.accessCount || 0) + 1,
// });

// Convert blob to base64 via Buffer (Convex actions run on Node/Bun
// — Buffer is available globally and avoids the per-byte loop).
// Convex `internalAction`s without `"use node"` run in the V8
// isolate runtime where `Buffer` is NOT a global — using it
// throws `ReferenceError: Buffer is not defined` at request time
// (the prior `Buffer.from(...)` shipped here was the bug behind
// the dashboard's "download .p8" failing). Encode via `btoa` on
// chunked binary strings so the path stays portable to either
// runtime; chunking keeps the call-stack bound below the
// `String.fromCharCode` argument limit for files of any size,
// and accumulating the chunks in an array before `join("")`
// avoids the O(n²) string-concatenation behavior of `binary +=`
// on multi-megabyte uploads (Copilot review on PR #127).
const arrayBuffer = await blob.arrayBuffer();
const base64 = Buffer.from(arrayBuffer).toString("base64");
const bytes = new Uint8Array(arrayBuffer);
const CHUNK = 0x8000;
const chunks: string[] = [];
for (let i = 0; i < bytes.length; i += CHUNK) {
chunks.push(
String.fromCharCode.apply(
null,
bytes.subarray(i, i + CHUNK) as unknown as number[],
),
);
}
Comment on lines +199 to +204

@gemini-code-assist gemini-code-assist Bot May 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

While String.fromCharCode.apply is a common pattern for converting byte arrays to strings, it is subject to stack size limits in many JavaScript engines (including V8). Although the CHUNK size of 0x8000 (32,768) is generally safe, very large files could still potentially hit limits if the environment's stack is constrained. Using TextDecoder (if available in the runtime) or a manual loop to build the string from the Uint8Array is more robust for arbitrary file sizes.

    for (let i = 0; i < bytes.length; i += CHUNK) {
      const subarray = bytes.subarray(i, i + CHUNK);
      let chunk = "";
      for (let j = 0; j < subarray.length; j++) {
        chunk += String.fromCharCode(subarray[j]);
      }
      chunks.push(chunk);
    }

const base64 = btoa(chunks.join(""));

return {
fileId: file._id,
Expand Down
Loading