feat: add sitemap.xml and robots.txt for SEO and AI crawler protectio… #23#24
feat: add sitemap.xml and robots.txt for SEO and AI crawler protectio… #23#24ankitrraj wants to merge 2 commits into
Conversation
WalkthroughAdds two Next.js app-router metadata files: Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes
Possibly related issues
Poem
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (2)
app/sitemap.ts (2)
9-9: Consider caching lastModified timestamps.Using
new Date()forlastModifiedon every build means the timestamp changes even when content hasn't, which can signal to search engines that pages have been updated when they haven't. This may trigger unnecessary recrawls and could impact crawl budget.For a static sitemap, consider using a fixed date or moving lastModified generation to build time. For a more sophisticated approach, fetch actual last modification dates from your data source:
// Example: Use a fixed build-time constant const LAST_BUILD = new Date('2025-12-13'); // Or fetch from your data layer const lastAuctionUpdate = await getLastAuctionUpdateDate();Also applies to: 15-15, 21-21, 27-27
6-31: Consider adding dynamic auction routes to sitemap.This sitemap is static and only includes 4 top-level routes. If your application has individual auction detail pages (e.g.,
/auctions/[id]), they should also be included in the sitemap for better SEO and discoverability.Consider making the sitemap dynamic by fetching auction IDs:
export default async function sitemap(): Promise<MetadataRoute.Sitemap> { // Static routes const routes = [ { url: BASE_URL, lastModified: new Date(), changeFrequency: 'daily' as const, priority: 1, }, // ... other static routes ]; // Fetch auction IDs and add to sitemap const auctions = await fetchAuctionIds(); // implement this const auctionRoutes = auctions.map((id) => ({ url: `${BASE_URL}/auctions/${id}`, lastModified: new Date(), changeFrequency: 'daily' as const, priority: 0.8, })); return [...routes, ...auctionRoutes]; }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
app/robots.ts(1 hunks)app/sitemap.ts(1 hunks)
🔇 Additional comments (1)
app/robots.ts (1)
13-32: Good approach to blocking AI training crawlers.The specific blocking of AI training bots (GPTBot, ChatGPT-User, CCBot, anthropic-ai, Google-Extended) aligns with the PR objectives and is a common practice for sites wanting to opt out of AI training datasets.
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (2)
app/sitemap.ts (2)
13-18: Good fix:/auctionschangeFrequency is no longer “hourly”.
Matches the earlier feedback; “daily” is a safer default hint.
6-25: Good fix:/dashboardis no longer in the sitemap (aligns with robots blocking it).
Keeps sitemap and crawler directives consistent.
🧹 Nitpick comments (2)
app/sitemap.ts (2)
3-4: NormalizeBASE_URLto avoid accidental double-slashes / non-canonical URLs.
IfNEXT_PUBLIC_SITE_URLends with/,${BASE_URL}/auctionsbecomes//auctions; also consider enforcinghttps://and stripping any path component.-const BASE_URL = process.env.NEXT_PUBLIC_SITE_URL || 'https://hah.app' +const BASE_URL = (process.env.NEXT_PUBLIC_SITE_URL || 'https://hah.app').replace(/\/+$/, '')Also applies to: 8-21
6-25:lastModified: new Date()will churn and can trigger unnecessary recrawls.
Because it changes on every render/build, bots may treat everything as “just updated” constantly. Prefer a stable build-time value or real content timestamps per route.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
app/robots.ts(1 hunks)app/sitemap.ts(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- app/robots.ts
🔇 Additional comments (1)
app/sitemap.ts (1)
1-26: No changes required. MetadataRoute.Sitemap in Next.js 15.3.3 fully supports bothchangeFrequencyandpriorityproperties. The code correctly uses validchangeFrequencyvalues ('daily' and 'weekly' are both allowed).
2 participants
Implemented sitemap.xml and robots.txt using Next.js App Router conventions to improve SEO and block AI crawlers from scraping site content.
Changes
Added app/sitemap.ts - Auto-generates sitemap with all public routes
Added app/robots.ts - Controls crawler access and blocks AI bots
Features
Sitemap (/sitemap.xml)
Includes home, auctions, create, and dashboard pages
Proper priority and change frequency settings
Auto-updates lastModified timestamp
Robots (/robots.txt)
Allows standard search engine crawlers
Blocks /api/ and /dashboard/ from indexing
Blocks AI crawlers:
GPTBot (OpenAI)
ChatGPT-User
CCBot (Common Crawl)
anthropic-ai
Google-Extended (Bard/Gemini training)
Summary by CodeRabbit
New Features
Chores
✏️ Tip: You can customize this high-level summary in your review settings.