Skip to content

feat(seo): serve the IndexNow key file on grabcaramel.com - #235

Open
BSalaeddin wants to merge 1 commit into
mainfrom
seo/indexnow-key
Open

BSalaeddin wants to merge 1 commit into
mainfrom
seo/indexnow-key

Conversation

@BSalaeddin

Copy link
Copy Markdown
Collaborator

What

Serve an IndexNow host-ownership key file on grabcaramel.com:

  • apps/caramel-app/src/lib/seo/indexnow.tsINDEXNOW_KEY (d5c0ad54cc4dc724a423fa3e9d273f7f), INDEXNOW_KEY_PATH, and indexNowKeyResponse() (body = the key verbatim, text/plain; charset=utf-8, Cache-Control: public, max-age=86400). Doc comment covers what IndexNow is, why the key is intentionally public (it only proves control of the host), and that submission is done by the SEO owner via the fleet seo-mcp (indexing_submit method=index_now), never from the app.
  • apps/caramel-app/src/app/d5c0ad54cc4dc724a423fa3e9d273f7f.txt/route.ts — the literal <key>.txt route folder (same family as src/app/llms.txt/route.ts; deliberately not withRoute, which owns /api). Serves https://grabcaramel.com/d5c0ad54cc4dc724a423fa3e9d273f7f.txt.
  • apps/caramel-app/tests/unit/indexnow-key-file.test.ts — CI pin: key matches IndexNow's [A-Za-z0-9-]{8,128} shape; a src/app/<INDEXNOW_KEY>.txt/route.ts folder exists, is the ONLY hex-key .txt route (no stale twin after a rotation), and imports the constant; the response is 200 / text/plain; charset=utf-8 / max-age=86400 with the key as the whole body; INDEXNOW_KEY_PATH === /<key>.txt.
  • apps/caramel-app/e2e/seo-regression.spec.ts — one new ungated (deployment-safe) test: GET /<key>.txt → 200, text/plain, trimmed body === INDEXNOW_KEY. Imports the constant relatively (../src/lib/seo/indexnow) — the module is alias-free and env-free on purpose, since Playwright specs do not resolve the @/ alias.

No change to robots.ts (the path is not disallowed) and the key is NOT added to the sitemap.

Why

IndexNow is the legitimate push channel (Bing, Yandex, Seznam, Naver) for the 4,000+ per-store /coupons/<store> pages that Google/Bing are slow to discover organically. The protocol validates a submission by fetching https://<host>/<key>.txt and comparing the body to the submitted key, so the site must serve that file before any URL can be pushed. Context: C:\Users\alaed\Documents\Github\caramel-artifact\seo-2026-09-11 (SEO program artifact dir). Reference shape: BioFlow PR #241.

This PR submits nothing to any engine. It only serves the ownership proof. Fresh key generated for this project (node -e "crypto.randomBytes(16).toString('hex')"); never reused from another project.

Verification done (local, worktree C:\wt-caramel-indexnow)

Dotted App Router folder segment confirmed to serve (no public/ fallback needed). next dev -p 58000 with the CI dummy .env, then:

GET http://localhost:58000/d5c0ad54cc4dc724a423fa3e9d273f7f.txt
=== STATUS 200
content-type: text/plain; charset=utf-8
cache-control: public, max-age=86400
body=[d5c0ad54cc4dc724a423fa3e9d273f7f] len=32
=== llms.txt control STATUS 200

Server log: GET /d5c0ad54cc4dc724a423fa3e9d273f7f.txt 200 in 3.5s. Server stopped afterwards (port 58000 listeners after kill: 0).

Gates (turbo ones with --force, since the turbo cache is shared across worktrees):

##### GATE lint :: pnpm lint --force                      → EXIT 0
##### GATE lint:oxlint :: pnpm lint:oxlint                → EXIT 0
##### GATE prettier-check :: pnpm prettier-check --force  → EXIT 0
##### GATE knip :: pnpm --filter caramel-app knip         → EXIT 0
##### GATE type-check :: pnpm -r run type-check           → EXIT 0   (after `pnpm --filter caramel-app db:generate`; a fresh worktree has no generated Prisma client, and the first attempt failed only on `@prisma/client has no exported member 'PrismaClient'` + its implicit-any cascade — none in the changed files)
##### GATE test :: pnpm --filter caramel-app test
 ✓ tests/unit/indexnow-key-file.test.ts (4 tests) 40ms
 Test Files  70 passed (70)
      Tests  692 passed (692)
 + 9 jsdom (.tsx) files hit "[vitest-pool]: Failed to start forks worker" on the shared box (163 node processes from 11 parallel agents), never ran, and were rerun alone:
   --maxWorkers=1            → 3 files / 11 tests passed (620s), 6 still "Failed to start forks worker"
   --pool=threads            → 5 files / 15 tests passed (588s), coupon-card.test.tsx still "Failed to start threads worker"
   coupon-card.test.tsx alone, threads → "Failed to start threads worker" again (60s worker-start limit)
 => 78 of 79 unit files proven green locally (70 + 3 + 5); tests/unit/coupon-card.test.tsx is UNVERIFIED LOCALLY
    (a component-render test whose source this PR does not touch; the CI `test` job runs it on an unloaded runner).

Noise ruled out, not hand-waved: three-lazy-boundary and userInitial first showed Test timed out in 5000ms under load; rerun with my files present they pass in 1.2s, and a control with my files stashed behaves identically (raw walk+read of src = 250ms). Full-repo pnpm test --force additionally reds 7 tests in apps/caramel-extension, which this branch does not touch (git diff origin/main -- apps/caramel-extension is empty) — pre-existing/environmental by construction, and out of this PR's gate (pnpm --filter caramel-app test).

What CI covers

  • tests/unit/indexnow-key-file.test.ts pins folder name ↔ constant ↔ body on every PR.
  • e2e/seo-regression.spec.ts fetches the live path in both e2e contexts (hermetic e2e-pr and deployed e2e-push), so a deploy that stops serving the file goes red.
  • knip is green: the constant is consumed by the route, the unit pin and the e2e spec.

Owner note

After this deploys, the SEO owner submits URL changes via the fleet seo-mcp (indexing_submit method=index_now, key d5c0ad54cc4dc724a423fa3e9d273f7f, key location https://grabcaramel.com/d5c0ad54cc4dc724a423fa3e9d273f7f.txt). Nothing is submitted by this PR. Do not merge on my behalf.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant