fix(seo): AI-crawler allow-list, cf-visitor https redirect, HSTS preload, llms-full.txt

[BSalaeddin]

What / why

grabcaramel.com is the fleet's second-best organic asset (920 clicks / 16,409 impressions in 90 days). The 2026-09-11/12 SEO audit (C:\Users\alaed\Documents\Github\caramel-artifact\seo-2026-09-11\ — route-inventory.md §10 "Gaps vs the fleet standard", audit-findings.md "Host hygiene", ci-inventory.md §4) measured these host-hygiene gaps; this PR closes the app-side ones and pins each in a test.

Measured gap (2026-09-11/12)	Change
http://grabcaramel.com/ serves 200 with the full page (Cloudflare Always-Use-HTTPS off; GSC already indexes the http twin with clicks)	src/middleware.ts: http→https 308 keyed ONLY on Cloudflare's cf-visitor: {"scheme":"http"}. Never on x-forwarded-proto (Next synthesises it, Traefik rewrites it — a fallback loops). No header = serve. Folded into the existing www→apex 308 so a http://www. request redirects once to the https apex.
HSTS = max-age=31536000; includeSubDomains (no preload)	next.config.mjs SECURITY_HEADERS → …; includeSubDomains; preload. dev.grabcaramel.com is already https-only, so preload is safe.
robots.txt has a single * group, no AI-crawler allow-list	src/lib/seo/aiCrawlers.ts (18-agent readonly const, fleet list) + a second prod rule in src/app/robots.ts { userAgent: [...AI_CRAWLERS], allow: '/', disallow: DISALLOWED_PATHS } (same disallow set). Non-prod branch unchanged (blanket disallow, one rule).
/llms-full.txt 404; nothing links /llms.txt	New src/app/llms-full.txt/route.ts (what/how/browsers/FAQ/key pages/install/privacy summary → /privacy). FAQ strings come from the SAME array as the landing accordion + FAQPage JSON-LD, moved verbatim to src/lib/faqItems.ts (wording untouched). llms.txt gains a pointer to the full doc. Discoverability: <link rel="alternate" type="text/plain" href="/llms.txt" title="llms.txt"> via alternates.types (root layout + home page — Next replaces alternates wholesale per level, verified in resolve-metadata.js, so home's canonical block re-states it through one shared constant src/lib/seo/llmsTxtAlternate.ts) + a tiny footer <a href="/llms.txt"> under Legal.
Organization JSON-LD has @id/alternateName/sameAs but nothing pins them	parentOrganization: { Organization, "Devino Solutions", https://devino.ca } added (devino.ca → 200 via curl 2026-09-12); shape pinned in e2e. No rating/review markup added.
/login, /signup, /verify rely on robots.txt Disallow only	robots: { index: false, follow: false } metadata on the three pages. /forgot-password, /reset-password untouched.

Not touched (owned by seo/index-policy / seo/claim-integrity): sitemap.ts, couponsRepo.ts, coupon pages, marketing copy. Nothing needed from them.

Tests

• tests/unit/robots-env-contract.test.ts — rewritten: non-prod = exactly ONE rule; prod = exactly TWO rules; the AI group's userAgent equals a literal 18-agent list (spelled out, not imported, so a dropped/misspelled agent goes red) with the same disallow set; AI_CRAWLERS itself is compared to that literal.
• tests/unit/middleware.test.ts — new (NextRequest): http cf-visitor → 308 https same path+query; https cf-visitor → served; no header → served; www → 308 apex; www+http → one 308 to https apex; x-forwarded-proto: http alone → NO redirect; garbage cf-visitor → served; matcher still excludes _next/static, _next/image, favicon.ico.
• tests/unit/llms-txt-routes.test.ts — new: llms-full.txt carries every FAQ question+answer verbatim, points at /privacy, no review markup; llms.txt links llms-full.txt.
• e2e/seo-regression.spec.ts — extended (same runner, all deployment-safe, no DATABASE_URL assumptions): (a) /llms-full.txt 200 text/plain + 'Caramel' + a FAQ question; (b) home raw HTML has the rel=alternate type=text/plain link + the crawlable footer <a href="/llms.txt">; (c) Organization JSON-LD @id ends #organization, alternateName includes 'Caramel coupon extension', non-empty https sameAs, parentOrganization.url === 'https://devino.ca'; (d) strict-transport-security on / contains preload (set by next.config, so it holds in both contexts); (e) /login, /signup, /verify raw HTML carries noindex + nofollow meta. The prod robots allow-list is pinned at unit level because e2e only ever sees the non-prod branch.

Verification (worktree C:\wt-caramel-hygiene, fresh pnpm install --frozen-lockfile + prisma generate)

pnpm exec turbo run lint --force        → Tasks: 2 successful, 2 total (0 errors; 3 pre-existing extension warnings)
pnpm lint:oxlint                        → exit 0 (pre-existing extension no-underscore-dangle warnings only)
pnpm prettier-check --force             → All matched files use Prettier code style! (2/2)
pnpm --filter caramel-app knip          → exit 0
pnpm -r run type-check                  → caramel-app Done, caramel-ui Done, caramel-extension Done (exit 0)
pnpm --filter caramel-app test          → Test Files 79 passed (79), Tests 729 passed (729)
vitest run robots-env-contract middleware faq-section → 3 files, 21 tests passed
vitest run llms-txt-routes test-quality-guardrails repo-integrity eval-files-out-of-unit-glob → 4 files, 9 tests passed
curl https://devino.ca                  → 200
curl -I https://grabcaramel.com/        → strict-transport-security: max-age=31536000; includeSubDomains   (the gap, before this PR)

Not run locally: Playwright e2e and Lighthouse (no local server stood up; the e2e additions are type-checked only via tsc --noEmit, which includes e2e/). CI's e2e-pr (hermetic) runs them on this PR; e2e-push re-runs them against the deployed site after merge (it waits on /api/version first, so the HSTS/redirect assertions see the new build).

What CI covers

checks-app.yml: lint/oxlint/prettier/knip/type-check/unit (the gates above) + e2e-pr runs the extended seo-regression.spec.ts against the hermetic build. The footer gains one small link, so the visual-regression job may flag the home/footer snapshot — expected, approve the new baseline.

Owner items (not code)

• 🟡 Cloudflare → SSL/TLS → Edge Certificates → "Always Use HTTPS" = ON for the grabcaramel.com zone. The middleware 308 closes the gap at the origin, but the edge toggle is the proper fix and saves a hop.
• Optional: submit grabcaramel.com at https://hstspreload.org/ once this is live (header now qualifies: 1y + includeSubDomains + preload). Irreversible-ish — only do it if every current/future subdomain will stay https-only.
• After deploy: curl -sI http://grabcaramel.com/ should show 308 + location: https://grabcaramel.com/; curl -s https://grabcaramel.com/robots.txt should show the second User-Agent: GPTBot … group.