From 030a52a1e013a9f02bf8aa4035283ca80dce9805 Mon Sep 17 00:00:00 2001 From: Nitesh Rijal Date: Wed, 20 May 2026 09:54:07 -0500 Subject: [PATCH 1/2] docs: storage limits, Cloudflare deploy, and storage error reference Add three operational docs that cover production gaps in JSON admin: - docs/storage-limits.md: GitHub Contents API 1 MB inline limit, Blobs API 100 MB ceiling, sizing table by row size, write amplification, and when to move off single-file array storage. - docs/deploying-cloudflare-pages.md: per-request token-injection middleware pattern for runtimes where runtimeConfig is frozen and secrets live on event.context.cloudflare.env instead of process.env. - docs/error-reference.md: every createError thrown by server/utils/githubContents.ts and server/utils/jsonStorage/* with status code, exact message, cause, and remediation. Reflects the post-fix Blobs API fallback path. Cross-link the new pages from README.md and docs/json-admin.md. --- README.md | 6 ++ docs/deploying-cloudflare-pages.md | 135 +++++++++++++++++++++++++++++ docs/error-reference.md | 51 +++++++++++ docs/json-admin.md | 8 ++ docs/storage-limits.md | 90 +++++++++++++++++++ 5 files changed, 290 insertions(+) create mode 100644 docs/deploying-cloudflare-pages.md create mode 100644 docs/error-reference.md create mode 100644 docs/storage-limits.md diff --git a/README.md b/README.md index 77007ef..0251035 100644 --- a/README.md +++ b/README.md @@ -122,6 +122,12 @@ Run the project and open `/admin` to access the admin interfaces for users and p JSON-backed settings and lists (GitHub or local files), separate from Drizzle tables. **Start here:** [docs/json-admin.md](./docs/json-admin.md) — quick start, registration, storage, env vars, and UI options. +Operating JSON admin in production: + +- [docs/storage-limits.md](./docs/storage-limits.md) — GitHub 1 MB / 100 MB limits and sizing guidance. +- [docs/deploying-cloudflare-pages.md](./docs/deploying-cloudflare-pages.md) — per-request token-injection middleware for Cloudflare Pages and other serverless runtimes. +- [docs/error-reference.md](./docs/error-reference.md) — every storage-layer error, with cause and remediation. + ## Specifying Custom Field Types While AutoAdmin infers types from your Drizzle schema, you can override them for more control over the UI. For example, you may want to change a text field to a textarea, a rich-text editor, or an image uploader. Use the fields option during registration. diff --git a/docs/deploying-cloudflare-pages.md b/docs/deploying-cloudflare-pages.md new file mode 100644 index 0000000..689bfb3 --- /dev/null +++ b/docs/deploying-cloudflare-pages.md @@ -0,0 +1,135 @@ +# Deploying to Cloudflare Pages (and other serverless runtimes) + +JSON admin's GitHub storage adapter needs `NUXT_AUTOADMIN_GITHUB_TOKEN` at **request time**. On long-running Node servers this is automatic — `useRuntimeConfig()` is populated from `process.env` at boot. On **Cloudflare Pages / Workers** and other V8-isolate runtimes, the story is different and the naïve setup fails in production with a 500. + +This page documents the failure mode and the per-request token-injection pattern that fixes it. + +--- + +## Why the default setup breaks on Cloudflare + +On Cloudflare Pages, secrets configured in the dashboard are **not** present on `process.env` when Nitro builds `runtimeConfig`. They are delivered per request on `event.context.cloudflare.env`. + +Two symptoms follow: + +1. **At boot**, `runtimeConfig.autoadmin.github.token` resolves to an empty string. The factory in `server/utils/jsonStorage/factory.ts` sees no token and either falls back to local storage (dev) or throws **500** (production): + + > GitHub token missing for this JSON resource in non-dev environment. Set NUXT_AUTOADMIN_GITHUB_TOKEN / runtimeConfig.autoadmin.github.token, or use local storage. + +2. **At request time**, trying to patch `useRuntimeConfig().autoadmin.github.token = …` may throw, because the config object is **deep-frozen** in some serverless builds. You can't simply re-assign it from middleware. + +The fix is to inject the token from the Cloudflare per-request env into the **registry's stored resource configs**, where the storage factory already looks first. + +--- + +## Per-request token-injection middleware + +Drop this in `server/middleware/admin-auth.ts` of the **consuming** Nuxt project (not in the autoadmin layer). It runs before the autoadmin API handlers, so every JSON-admin call sees a token in its resource's `storage` block. + +```ts +// server/middleware/admin-auth.ts +import { createError } from 'h3' +import { useJsonResourceRegistry } from '#layers/autoadmin/server/utils/jsonResourceRegistry' + +export default defineEventHandler((event) => { + const path = event.path || '' + if (!path.startsWith('/api/autoadmin')) { + return + } + + // 1. Resolve the token for this request. + // On Cloudflare Pages the secret lives on event.context.cloudflare.env, + // not on process.env at boot. + const token + = event.context.cloudflare?.env?.NUXT_AUTOADMIN_GITHUB_TOKEN + ?? process.env.NUXT_AUTOADMIN_GITHUB_TOKEN + + if (!token) { + if (process.env.NODE_ENV === 'production') { + throw createError({ + statusCode: 500, + statusMessage: + 'NUXT_AUTOADMIN_GITHUB_TOKEN is not configured. Set it in the ' + + 'Cloudflare Pages dashboard (or your serverless platform).', + }) + } + return // dev falls back to local storage in the factory + } + + // 2. Inject the token into the registered JSON resources. + // The storage factory checks `storage.token` before runtimeConfig, + // so this wins even if runtimeConfig is frozen. + const registry = useJsonResourceRegistry() + for (const resource of registry.all()) { + if (resource.storage.kind === 'github') { + resource.storage.token = token + } + } + + // 3. Best-effort: also update runtimeConfig for any code that reads it + // directly. Wrapped in try/catch because some runtimes freeze it. + try { + const config = useRuntimeConfig() + config.autoadmin = config.autoadmin || {} + config.autoadmin.github = config.autoadmin.github || {} + config.autoadmin.github.token = token + } + catch { + // runtimeConfig is frozen in production on some serverless platforms. + // Step 2 already covered the JSON-admin path. + } +}) +``` + +> Add your own auth check (session cookie, JWT, etc.) at the top of this middleware. Token injection alone does not authenticate the request — it only makes GitHub storage work. + +--- + +## Why this works + +`server/utils/jsonStorage/factory.ts` resolves the GitHub token in this order: + +1. `storage.token` on the **per-resource** config (set by the middleware above). +2. `runtimeConfig.autoadmin.github.token`. +3. Empty → throw 500 (or fall back to local storage in dev). + +Because step 1 is a plain object property on the registry's in-memory store, you can mutate it from any request handler. Step 2 may be frozen; step 1 isn't. + +--- + +## Configuration checklist + +In the Cloudflare Pages dashboard, under **Settings → Environment variables** for the production environment: + +| Variable | Required | Notes | +|----------|:--------:|-------| +| `NUXT_AUTOADMIN_GITHUB_TOKEN` | yes | Fine-grained PAT with `contents:read` and `contents:write` on the target repo. | +| `NUXT_AUTOADMIN_GITHUB_OWNER` | yes* | Or set `storage.owner` per-resource. | +| `NUXT_AUTOADMIN_GITHUB_REPO` | yes* | Or set `storage.repo` per-resource. | +| `NUXT_AUTOADMIN_GITHUB_REF` | no | Defaults to the repo's default branch. | +| `NUXT_DATABASE_URL` | yes | For Drizzle admin. D1 binding is auto-detected if the binding is named `DB`. | + +\* required unless every JSON resource sets its own `storage.owner` / `storage.repo`. + +Make sure the variables are set on **both** the Production and Preview environments if you use preview deployments. + +--- + +## Verifying + +After deploy, a quick sanity check from a logged-in admin session: + +``` +GET /api/autoadmin/json// → 200 with the current document +PUT /api/autoadmin/json// → 200, returns a new revision +``` + +A `500` with `GitHub token missing for this JSON resource …` means the middleware didn't see the env var — check the dashboard variable name (it must be exactly `NUXT_AUTOADMIN_GITHUB_TOKEN`). + +A `500` with `Cannot assign to read only property` or similar in logs means a code path is still trying to mutate `useRuntimeConfig()` directly. The middleware above wraps that in `try/catch`; any other call site should do the same. + +--- + +## Applies to + +This pattern is not Cloudflare-specific. Any runtime that delivers secrets per-request rather than at boot (Vercel Edge, Deno Deploy, some Lambda configurations) needs the same shape: read the secret from the request-scoped env, write it into the registry's resource storage, don't trust `runtimeConfig` to be mutable. diff --git a/docs/error-reference.md b/docs/error-reference.md new file mode 100644 index 0000000..b9b0e23 --- /dev/null +++ b/docs/error-reference.md @@ -0,0 +1,51 @@ +# Error reference: JSON admin storage + +Every error thrown by the JSON admin storage layer, with the exact message, the likely cause, and how to recover. Errors are listed by source file. + +For limits and sizing context, see [storage-limits.md](./storage-limits.md). For the Cloudflare-specific token-injection issue, see [deploying-cloudflare-pages.md](./deploying-cloudflare-pages.md). + +--- + +## `server/utils/githubContents.ts` — GitHub Contents + Blobs API + +| Status | Message | Likely cause | Remediation | +|:------:|---------|--------------|-------------| +| **404** | `` or `GitHub API error (404)` | File at `path` does not exist on `ref`. | First-write flows treat this as "create new". For reads, verify `owner`, `repo`, `ref`, and `path` exactly match the file in the repo. | +| **400** | `` or `GitHub API error ()` | Any 4xx from Contents API other than 404 — most often `401`/`403` (bad/missing token, repo permissions), or `422` (invalid request). | Confirm the token has `contents:read` (and `contents:write` for saves) on the repo. Check the token isn't expired. | +| **502** | `` or `GitHub API error (<5xx>)` | GitHub returned 5xx (rare outage, secondary rate limit). | Retry. If persistent, check status.github.com and your token's rate-limit headers. | +| **500** | `GitHub response is not a single file with content.` | The path resolved to a directory, submodule, or symlink instead of a regular file. | Point `path` at a regular `.json` file, not a directory. | +| **400** | `` or `GitHub Blobs API error ()` | The Contents API said the file is too big for inline content (>1 MB), so JSON admin fell back to the Git Blobs API and that call returned 4xx. Usually means the token can read repo metadata but not raw blobs, or the blob `sha` was rotated mid-request. | Confirm the token has `contents:read`. If `sha` rotation is the cause, retry — JSON admin re-reads `sha` on each call. | +| **502** | `` or `GitHub Blobs API error (<5xx>)` | 5xx on Blobs fallback. | Retry. Persistent failures here usually mean the file is at or above the 100 MB Blobs hard ceiling. | +| **500** | `GitHub Blobs API response is missing base64 content.` | Blobs API returned successfully but with a non-`base64` encoding or no string content. Effectively unreachable for normal JSON files. | File the issue with a repro — this indicates a corrupted blob or an API change. | +| **422** | `File is not valid JSON.` | Decoded file contents could not be `JSON.parse`d. Usually means a human edited the file in Git directly and broke it. | Open the file on the configured `ref` and fix the JSON. JSON admin will not overwrite an unparseable file. | +| **409** | `` or `GitHub file changed on the server (sha conflict). Refresh and try again.` | Optimistic concurrency: another writer updated the file between this request's read and write. The CRUD service already retried once. | Reload the admin UI and re-apply the change. Frequent 409s mean too many concurrent editors for the single-file model — see [storage-limits.md](./storage-limits.md). | +| **400** | `` or `GitHub API error ()` | PUT-side 4xx other than 409 — branch protection, missing write scope, payload too large. | Check token scopes, branch protection rules, and whether the file is above the 100 MB hard ceiling. | +| **502** | `` or `GitHub API error (<5xx>)` | 5xx on write. | Retry. | + +> **About the 1 MB → 100 MB fallback.** When the Contents API returns `encoding: "none"` (file >1 MB), JSON admin transparently re-reads via `GET /repos/{owner}/{repo}/git/blobs/{sha}`. Reads still succeed up to 100 MB. **Writes** still go through the Contents API, which accepts up to ~100 MB but is slow and memory-heavy above a few MB. + +--- + +## `server/utils/jsonStorage/factory.ts` — adapter selection + +| Status | Message | Likely cause | Remediation | +|:------:|---------|--------------|-------------| +| **500** | `GitHub token missing for this JSON resource in non-dev environment. Set NUXT_AUTOADMIN_GITHUB_TOKEN / runtimeConfig.autoadmin.github.token, or use local storage.` | A resource is registered with `storage: { kind: 'github' }` but no token was found on the per-resource config, on `runtimeConfig.autoadmin.github.token`, or on `NUXT_AUTOADMIN_GITHUB_TOKEN`. Dev mode silently falls back to local storage; production refuses. | Set the env var. On serverless platforms where `runtimeConfig` is populated from `process.env` at boot but secrets are delivered per-request, use the middleware pattern in [deploying-cloudflare-pages.md](./deploying-cloudflare-pages.md). | + +--- + +## `server/utils/jsonStorage/localJsonRepository.ts` — local files + +| Status | Message | Likely cause | Remediation | +|:------:|---------|--------------|-------------| +| **422** | `Local JSON file is not valid JSON.` | The on-disk file at `absolutePath` could not be `JSON.parse`d. | Open the file and fix it. JSON admin will not overwrite an unparseable file. | +| **409** | `Local JSON file changed on disk. Refresh and try again.` | The file's `mtime` changed between the read and the write — concurrent edit. The CRUD service already retried once. | Reload the admin UI and re-apply the change. | +| **409** | `Local JSON file was created concurrently. Refresh and try again.` | The client expected the file to not exist yet (`revision === '0'`), but it was created by another writer in the meantime. | Reload the admin UI; you'll now see the other writer's content as the baseline. | + +--- + +## Cross-cutting notes + +- **Where the messages surface.** The admin UI shows `statusMessage` as the toast text on failed requests. If you proxy these errors through your own handler, preserve `statusCode` and `statusMessage` so the UI behaves correctly. +- **Logging.** None of these throw paths log by default. Wrap calls at the route layer if you need structured logs. +- **`createError` source.** All errors are thrown via Nitro's `createError` and serialize as standard `H3Error` shapes. diff --git a/docs/json-admin.md b/docs/json-admin.md index 75d3ce2..c114ec8 100644 --- a/docs/json-admin.md +++ b/docs/json-admin.md @@ -105,6 +105,14 @@ Per-resource **`githubToken`** / `storage.token` overrides the global token when --- +## Related docs + +- **[storage-limits.md](./storage-limits.md)** — GitHub 1 MB / 100 MB limits, sizing table, when to move off single-file `array` storage. +- **[deploying-cloudflare-pages.md](./deploying-cloudflare-pages.md)** — per-request token-injection middleware for Cloudflare Pages / serverless runtimes where `runtimeConfig` is frozen. +- **[error-reference.md](./error-reference.md)** — every error the storage layer can throw, with cause and remediation. + +--- + ## Other notes - **Auth:** Optional per-resource `roles` — same as Drizzle admin; see [autoadmin-roles.md](./autoadmin-roles.md). diff --git a/docs/storage-limits.md b/docs/storage-limits.md new file mode 100644 index 0000000..e05c033 --- /dev/null +++ b/docs/storage-limits.md @@ -0,0 +1,90 @@ +# Storage limits & scaling + +JSON admin stores each resource in a **single file**. That model is simple and Git-friendly, but it imposes hard upper bounds — especially when the storage backend is the GitHub Contents API. Read this page **before** picking the `array` kind for anything that will grow. + +--- + +## GitHub limits at a glance + +| Limit | Value | Source | +|-------|-------|--------| +| **Inline content in Contents API response** | **1 MB** per file | GitHub returns `encoding: "none"` and omits `content` above this. JSON admin falls back to the **Git Blobs API** automatically. | +| **Blobs API content size** | **100 MB** | Hard ceiling. Reads of larger files fail. | +| **Recommended repo file size** | **50 MB** | GitHub warns above this and rejects pushes above 100 MB through normal Git. | +| **Per-write API call** | 1 commit per save | Each save is `read → merge → write`, which means one commit on every CRUD action. | +| **Authenticated rate limit** | 5,000 req/hr (PAT) | Shared across all readers and writers using the same token. | + +Files between 1 MB and 100 MB still work — JSON admin transparently reads them via the Blobs API — but they pay a second round-trip on every read and the full blob is held in memory while it is base64-decoded. + +--- + +## Sizing table (single-file `array` mode) + +Approximate row count where you should start worrying. Numbers assume modest rows with short string fields and one or two relations; rich-text or embedded media drives the row size up fast. + +| Avg. row size | Comfortable (<256 KB) | Slow but works (1–10 MB) | Approaching ceiling (>50 MB) | +|---------------|----------------------:|-------------------------:|-----------------------------:| +| 500 B | < 500 rows | ~ 2k–20k rows | ~ 100k rows | +| 2 KB | < 130 rows | ~ 500–5k rows | ~ 25k rows | +| 10 KB | < 25 rows | ~ 100–1k rows | ~ 5k rows | +| 50 KB (rich text) | < 5 rows | ~ 20–200 rows | ~ 1k rows | + +Treat the **"Comfortable"** column as a soft target. Above it, every save rewrites the whole file and every read transfers the whole file. + +--- + +## Write amplification + +Every CRUD action on an `array` resource is: + +1. `GET` the entire file (1 round trip, or 2 if it exceeds 1 MB). +2. Parse it, mutate one row in memory. +3. `PUT` the entire file with the new contents (1 commit). + +Implications: + +- **A 10 MB array file pays 10 MB of read + 10 MB of write to flip a single boolean.** +- **Concurrent edits collide.** The server uses GitHub's blob `sha` for optimistic concurrency and retries **once** on 409 conflict. Heavy editing by multiple users on the same large file will surface 409s to the UI. +- **Commit history grows fast.** A 500-row file edited 50 times a day produces 50 commits per day against the same path. + +--- + +## When `array` becomes a problem + +Move off single-file `array` storage when **any** of these are true: + +- File size in repo exceeds **~1 MB** (you're now on the Blobs API fallback path, with extra latency and memory cost). +- More than ~2 concurrent editors regularly touch the same resource. +- Row count exceeds a few hundred and rows are individually large (rich text, base64 images, long arrays). +- You need partial reads, search, sort, or filter pushdown — JSON admin always loads the whole array. +- You need an audit trail finer than "the commit that touched this file". + +For those workloads, use **Drizzle admin** with a real database. JSON admin is for **configuration and small editorial lists**, not user-generated content at scale. + +--- + +## When `object` mode is fine + +`object` resources are bounded by your schema, not by row count. Site settings, feature flags, and similar config files stay well under any limit. They have the same write-amplification on save, but the file is small so it doesn't matter. + +--- + +## Mitigations short of switching backends + +If you must stay on GitHub storage with a growing `array`: + +- **Split by domain.** Register multiple `array` resources backed by separate files (`content/banners-home.json`, `content/banners-marketing.json`) instead of one mega-file. +- **Trim historical rows.** Move closed/archived rows out of the live file. +- **Avoid storing media inline.** Reference S3/R2 URLs instead of base64-encoded blobs. +- **Cache reads at the edge** for public consumers. JSON admin only owns the editing path; downstream readers don't have to go through it. + +--- + +## Error surfaces tied to these limits + +See [error-reference.md](./error-reference.md) for the full catalog. The ones most often caused by size: + +- **502** "GitHub Blobs API error (…)" — fallback failed; the file is likely over 100 MB or the token lacks `contents:read`. +- **500** "GitHub Blobs API response is missing base64 content." — unexpected encoding from the Blobs API; usually a corrupted upload or non-blob object. +- **409** "GitHub file changed on the server (sha conflict). Refresh and try again." — concurrent edit beat your save; the server already retried once. +- **422** "File is not valid JSON." — the file was edited outside JSON admin and broke parse. Fix the file directly in the repo. From f40730a2f208f57188f29fbadd1ac6f0baf1872e Mon Sep 17 00:00:00 2001 From: Nitesh Rijal Date: Wed, 20 May 2026 13:51:47 -0500 Subject: [PATCH 2/2] =?UTF-8?q?docs:=20replace=20'na=C3=AFve'=20with=20'de?= =?UTF-8?q?fault'=20in=20cloudflare=20deploy=20guide?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/deploying-cloudflare-pages.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deploying-cloudflare-pages.md b/docs/deploying-cloudflare-pages.md index 689bfb3..3e0ae2f 100644 --- a/docs/deploying-cloudflare-pages.md +++ b/docs/deploying-cloudflare-pages.md @@ -1,6 +1,6 @@ # Deploying to Cloudflare Pages (and other serverless runtimes) -JSON admin's GitHub storage adapter needs `NUXT_AUTOADMIN_GITHUB_TOKEN` at **request time**. On long-running Node servers this is automatic — `useRuntimeConfig()` is populated from `process.env` at boot. On **Cloudflare Pages / Workers** and other V8-isolate runtimes, the story is different and the naïve setup fails in production with a 500. +JSON admin's GitHub storage adapter needs `NUXT_AUTOADMIN_GITHUB_TOKEN` at **request time**. On long-running Node servers this is automatic — `useRuntimeConfig()` is populated from `process.env` at boot. On **Cloudflare Pages / Workers** and other V8-isolate runtimes, the story is different and the default setup fails in production with a 500. This page documents the failure mode and the per-request token-injection pattern that fixes it.