fix(seo): canonical store sitemap with lastmod behind one indexability policy

[BSalaeddin]

Why

grabcaramel.com: 920 clicks / 16,409 impressions in 90 days, but GSC's sitemap audit says 4,289 of 4,317 sitemap URLs are unknown to Google and the submitted sitemap was last read 2025-07-12 (14 months ago). Root causes, from the 2026-09-11 audit in C:\Users\alaed\Documents\Github\caramel-artifact\seo-2026-09-11\ (route-inventory.md §3 "Sitemap composition", audit-findings.md §"Sitemap" and §"Store-page sample"):

• sitemap.ts emitted raw DISTINCT coupons.site values (via listStoreOptions), not the canonical store slug the page canonicalizes to (resolveStoreDomain). Served sitemap 2026-09-11: 4,311 store slugs collapse to 4,262 canonical bases; 86 slugs differ from their base (84 subdomains — athleta.gap.com → gap.com, au.shein.com → shein.com; 2 mixed-case — Brooklinen.com, eNasco.com); 40 bases appear under several slugs (gap.com ×4, shein.com ×4, att.com ×3); 38 canonical targets (mattel.com, nfl.com, enasco.com, w3schools.com, timeout.com, …) were not in the sitemap at all; 1 slug is not a domain (dhl.com-us-en-home.html). Every one of those <loc>s carried a <link rel="canonical"> pointing somewhere else.
• Mixed-case sites were unreachable: listStoreCoupons / listCoupons match site = $base OR site LIKE '%.' || $base case-sensitively, resolveStoreDomain lowercases, so /coupons/enasco.com AND /coupons/eNasco.com both rendered the empty state with noindex while the eNasco.com rows exist — and the sitemap listed that noindexed page.
• No <lastmod> — Google had no freshness signal to re-read a sitemap it last read 14 months ago.
• /support is indexable, canonical'd and header-linked but was absent from the sitemap; /sources was in the sitemap but prod has 0 active sources (/api/sources → []), so it is an empty shell GSC reports as crawled-not-indexed.

What (the BioFlow shape: ONE pure policy consulted by BOTH the sitemap and page metadata)

Policy contract — src/lib/seo/storeIndexability.ts, pure (no DB, no env):

evaluateStorePageIndexability({ base, visibleCouponCount }): { indexable: boolean; reason: 'not-a-store' | 'no-coupons' | null }

base = resolveStoreDomain() output; null/'' → not-a-store; count < 1 → no-coupons; otherwise indexable. A store page is in the sitemap iff its own metadata would not noindex it — same function, both sides.

1. src/lib/seo/storeIndexability.ts + tests/unit/storeIndexability.test.ts (real prod cases: gap/roborock/pandora/enasco, dhl.com-us-en-home.html, co.uk, mixed-case slugs through the real resolveStoreDomain).
2. src/lib/seo/sitemapStores.ts — pure collapseStoreRows(rows): groups by resolveStoreDomain(site) (drops nulls such as dhl.com-us-en-home.html, bare public suffixes like co.uk), sums counts, keeps the max last_updated, applies the policy, returns sorted { base, couponCount, lastModified }[]. tests/unit/sitemapStores.test.ts: athleta.gap.com + gap.com + bananarepublic.gap.com → one gap.com with summed count + max date; Brooklinen.com → brooklinen.com; dhl.com-us-en-home.html / co.uk / null / 0-count dropped.
3. src/lib/couponsRepo.ts — new read listStoreSitemapEntries(limit): SELECT site, COUNT(*)::int AS coupon_count, MAX(updated_at) AS last_updated FROM coupons WHERE <visibleCouponsWhere()> AND site IS NOT NULL GROUP BY site ORDER BY site ASC LIMIT $limit, parsed through the new SiteAggregateRowSchema in src/lib/couponsDb.ts (z.coerce.date() like RecentStoreRowSchema.added_at). listStoreOptions unchanged (autocomplete). Unit pin in tests/unit/couponsRepo.test.ts (query shape + bound LIMIT + drift throw), integration pin in tests/integration/coupons-read.itest.ts (real pg: sorted, ::int, Date, count equals listCoupons total for the same base, LIMIT bound).
4. src/app/sitemap.ts — stores from listStoreSitemapEntries → collapseStoreRows, emitting ${origin}/coupons/${encodeURIComponent(base)} with lastModified when present; /support added to STATIC_ROUTES (monthly, 0.5); /sources emitted only when listActiveSources() returns ≥1 row (Promise.all with the store read). 5000 cap + force-dynamic comment kept. New tests/unit/sitemap.test.ts (mocks @/lib/couponsRepo + @/lib/env.client): static routes incl. /support; /sources absent when empty / present otherwise; store URLs canonical + lowercase + deduped; lastModified carried through (and omitted when null); no entry for a 0-count or unresolvable site; cap passed as 5000.
5. (marketing)/coupons/[store]/page.tsx generateMetadata — both ad-hoc branches now take their verdict from the policy (robots: indexable ? undefined : { index: false, follow: true }); tests/unit/coupons-store-page.test.ts passes unchanged (every existing assertion holds).
6. (marketing)/sources/page.tsx — generateMetadata sets robots: { index: false, follow: true } when the active-source list is empty; the read is wrapped in React cache() so the body shares it (same pattern as the store page's fetchStoreCoupons). tests/unit/sources-page.test.ts.
7. Case normalization so the canonical (lowercase) sitemap URL always renders the coupons:
   • (a) src/lib/catalog/applyCatalogRows.ts lowercases site on write (normalizeSite, the one write-side normalization; null stays null; status/discount_type/code stay RAW per ingestSchemas.ts). Covers both the ingest route and bridge:sync (same engine). Unit pin tests/unit/applyCatalogRows-site-case.test.ts (asserts the bound INSERT values); integration pin in tests/integration/ingest-catalog.itest.ts (mixed-case push stored lowercase, found by listStoreCoupons under both spellings and by listStoreSitemapEntries under the lowercase site).
   • (b) data-only migration prisma/migrations/20260912120000_lowercase_coupon_sites/migration.sql: UPDATE "public"."coupons" SET site = lower(site) WHERE site IS NOT NULL AND site <> lower(site); — no schema change; updated_at deliberately NOT bumped (the ingest only-if-newer rule keys on it).
   • (c) read queries unchanged in shape (plain equality keeps coupons_site_idx); listStoreCoupons / listCoupons lowercase the incoming base before binding it (one-line guard, pinned by asserting the bound values in tests/unit/couponsRepo.test.ts).
8. e2e: e2e/seo-a11y.spec.ts "Coupon pages — crawler-visible SEO" gains (a) every store <loc> in /sitemap.xml equals its own resolveStoreDomain and is lowercase, and there are no duplicate <loc>s (imports ../src/lib/storeDomain relatively — pure, tldts only, no prisma, no @/ alias, so collection survives the deployed context); (b) /support is in the sitemap. e2e/seo-regression.spec.ts: { path: '/support', minVisibleChars: 380 } added to ROUTES (547 visible chars measured on prod, −30%), and the sitemap test now asserts /sources conditionally against /api/sources (present iff ≥1 active source) instead of unconditionally, since the deployed site may legitimately have 0.

Migration blast radius

UPDATE coupons SET site = lower(site) WHERE site IS NOT NULL AND site <> lower(site) touches only rows whose site carries uppercase — 2 known sites on prod as of 2026-09-11 (Brooklinen.com, eNasco.com), i.e. their coupon rows. Idempotent; a re-run is a no-op. No DDL, so CI's schema-drift job is unaffected — verified locally with the CI-exact commands (below). It applies cleanly to an empty DB (prisma migrate deploy on a fresh compose Postgres, then both migrate diff variants → "No difference detected").

Dry-run numbers

From the served sitemap on 2026-09-11 (the DB's DISTINCT visible sites): 4,311 raw slugs → 4,262 canonical entries after this PR (−49 duplicates, −1 non-domain; the +38 previously-missing canonical targets are already counted inside the 4,262), and the 2 mixed-case slugs become reachable after the migration. Prod DB read access was unavailable for a direct dry-run against live rows — the numbers are derived from the served sitemap, not from a query against prod.

Verification done locally (worktree C:\wt-caramel-index-policy, fresh pnpm install --frozen-lockfile, compose Postgres on :58005 brought up for this run)

Command	Result
pnpm turbo run lint --force	pass — "Tasks: 2 successful, 2 total", exit 0 (3 pre-existing no-console warnings in apps/caramel-extension files untouched here). Note: pnpm lint -- --force forwards --force into the extension's eslint and exits 2 — that is the invocation, not the code.
pnpm lint:oxlint	pass — exit 0 (3 pre-existing no-underscore-dangle warnings in extension files, untouched)
pnpm turbo run prettier-check --force	pass — "All matched files use Prettier code style!" for both packages, exit 0
pnpm --filter caramel-app knip	pass — exit 0, no new ignores
pnpm -r run type-check	pass — caramel-ui / caramel-app / caramel-extension all "Done", exit 0 (the first run caught two es5-target iterator spreads, fixed with Array.from)
pnpm --filter caramel-app test (vitest unit)	every test green; exit code 1 is environmental — full run: "Test Files 77 passed (77) / Tests 741 passed (741)" plus 6 × [vitest-pool] Failed to start forks worker … Timeout waiting for worker to respond on jsdom .tsx files (support-form, recently-added-section, coupon-card, coupons-section, faq-section, error — none touched by this PR; transform alone took 325 s on a saturated box). Those 6 files run alone: "Test Files 6 passed (6) / Tests 23 passed (23)", exit 0. An earlier full run on the same tree reported "78 passed (78) / 749 passed (749)". The 8 touched/new unit files run together: 8 passed / 67 tests. Flagged verbatim rather than hidden; CI's runner should see the clean exit.
pnpm --filter caramel-app exec vitest run --config vitest.integration.config.ts (real pg :58005, after prisma migrate deploy)	pass — 7 files / 31 tests, incl. the new listStoreSitemapEntries and lowercase-site cases
prisma migrate deploy (fresh DB, all 9 migrations incl. 20260912120000_lowercase_coupon_sites)	pass — "All migrations have been successfully applied"
prisma migrate diff --from-migrations prisma/migrations --to-schema-datamodel prisma/schema.prisma --shadow-database-url … --exit-code (CI-exact)	pass — "No difference detected", exit 0
prisma migrate diff --from-migrations prisma/migrations --to-url $DATABASE_URL --shadow-database-url … --exit-code (CI-exact)	pass — "No difference detected", exit 0
pnpm --filter caramel-app exec prisma validate	pass
Playwright e2e/seo-a11y.spec.ts + e2e/seo-regression.spec.ts against this branch's own next dev on :58271 (compose pg :58005, migrated + seeded; PLAYWRIGHT_START_SERVER=false, --timeout=120000)	pass — 25 passed / 2 read ECONNRESET (dev server dropped the socket under load), then the 2 re-run alone: "2 passed", exit 0 — so 27/27 green including the new "every store <loc> is its own canonical base" and "/support is listed" tests and the conditional /sources assertion. Direct curl on the same server: sitemap = 7 static (incl. /support, /sources because the seed has 2 ACTIVE sources) + 5 canonical store <loc>s, 5 <lastmod> (e.g. <loc>…/coupons/codecademy.com</loc><lastmod>2026-07-13T07:45:00.000Z</lastmod>), 0 duplicate / 0 uppercase <loc>; /coupons/CodeCademy.com → canonical …/coupons/codecademy.com, "currently lists 5 active"; /coupons/no-coupons-here-zz.example → noindex, follow; robots non-prod branch. Caveat: the first attempt reused a stranger's next start already on :58000 (another worktree, reuseExistingServer) and is not evidence either way; the rest of the e2e suite was not run locally.

What CI covers

checks-app.yml: lint / oxlint / prettier / knip / type-check / unit; the schema-drift job (both migrate diff variants + "Apply migrations" on an empty DB); the integration job (real Postgres); e2e-pr (hermetic seeded DB — exercises the new sitemap/canonical//support assertions and, because the seed has 2 ACTIVE sources, the /sources-present branch) and e2e-push (deployed dev site — exercises the /sources-absent branch if dev has 0 active sources). Note for e2e-push: the new "every store <loc> is its own canonical" assertion only holds once the deployed dev site is running this code (it fails by design against the current sitemap).

Owner questions

• Should stores with exactly 1 code stay indexable? This PR keeps them: a page with a single live code is still a real answer to "<store> coupon code", and counts move daily — raising the bar would churn thousands of URLs in and out of the sitemap as codes expire and reappear, worse for crawl budget than a few thin-but-true pages. The threshold is one number in evaluateStorePageIndexability if you want it higher.
• After deploy, re-submit the sitemap in GSC and Bing Webmaster so Google re-reads it (last read 2025-07-12); the new <lastmod> values give it a reason to keep coming back.
• /sources will drop out of the sitemap and carry noindex on prod until at least one source is ACTIVE — intended; flip a source to ACTIVE and both revert automatically.