Add agent-readable docs discovery

[yashau]

Summary

• publish llms.txt, compact/full variants, and source-aligned Markdown for every public docs page
• serve Markdown from canonical docs URLs through content negotiation while keeping static assets asset-first
• add robots.txt and verify robots, sitemap, LLM indexes, and per-page Markdown during the docs build
• document release verification for the new agent-facing endpoints

The Worker implementation is compatible with the site's current Cloudflare Free plan; it does not depend on the paid native Markdown for Agents feature.

Validation

• mise run node:check
• mise run node:lint
• mise run node:format-check
• mise run docs:lint
• mise run docs:build
• pnpm --dir apps/docs exec tsc -p worker/tsconfig.json --noEmit
• pnpm --dir apps/docs exec wrangler deploy --dry-run
• local Wrangler HTTP smoke covering HTML/Markdown negotiation, q-values, HEAD, robots, sitemap, and LLM indexes