Add SEO metadata, robots.txt/sitemap, and expanded PyPI keywords#485
Conversation
- Homepage: Open Graph + Twitter Card tags, canonical URL, SVG favicon, SoftwareApplication JSON-LD, and a keyword-forward title (closes #479) - Pages build: emit robots.txt pointing at both sitemaps and a root sitemap.xml covering the homepage; publish og-card.png (closes #480) - pyproject: expand PyPI keywords for discovery (closes #483) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mf6GAt4wpvSXPTDKGu2q6B
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mf6GAt4wpvSXPTDKGu2q6B
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a992aa6b26
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| site_url="https://hexsecs.github.io/canarchy" | ||
|
|
||
| cat > "$site_root/robots.txt" <<EOF |
There was a problem hiding this comment.
In the Pages workflow I checked, the site artifact is published at https://hexsecs.github.io/canarchy/, so this generated file is served as /canarchy/robots.txt. Crawlers only treat robots.txt as valid at the protocol/host root (/robots.txt; Google documents subdirectory robots files as invalid: https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec), so the Sitemap: directives here will not be discovered from robots.txt for this project site. Unless the sitemap is submitted or linked another way, the crawler signposting added for the published site remains ineffective.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Good catch — you're right that /canarchy/robots.txt isn't authoritative for a project page, so its Sitemap: directives won't be auto-discovered. I've kept the file (it still serves SEO tooling that probes it, and it becomes authoritative if the site ever moves to a custom domain at the host root) but documented the limitation in the build script, and I'm adding "submit the sitemaps directly to Google Search Console / Bing Webmaster Tools" to the reach tracking issue (#484) since that's the discovery path that actually works here. The homepage's og: tags and JSON-LD are unaffected by this.
Generated by Claude Code
Crawlers only honor robots.txt at the host root, so the generated /canarchy/robots.txt cannot advertise the sitemaps on its own; they must be submitted to search engines directly. Noted in the build script per PR review. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mf6GAt4wpvSXPTDKGu2q6B
2 participants
Summary
Implements the quick-win items from the project SEO review so that shared links unfurl properly and crawlers can find the site structure. Covers the homepage metadata, Pages-build crawler signposting, and PyPI keyword expansion.
Closes #479
Closes #480
Closes #483
Changes
<head>now carries Open Graph and Twitter Card tags backed by a generated 1200×630 brand card (og-card.png, rendered in the site's yellow/black brutalist style), a canonical URL, an inline SVG favicon, and aSoftwareApplicationJSON-LD block<title>reworded to front-load search terms: "CANarchy — CAN bus & J1939 security research toolkit (CLI, TUI, MCP)"robots.txtreferencing both the root and docs sitemaps, plus a rootsitemap.xmlcovering the homepage, and publishes the social card alongsideindex.htmlautomotive-security,car-hacking,socketcan,candump,dbc,doip,iso-tp,mcp, …); takes effect on the next releaseNot in scope: pre-rendering the homepage (#478) — swapping the React dev builds for production builds needs new SRI hashes and belongs with that issue.
Test plan
uv run python -m unittest discover -s tests -v— 1149 tests, OK (1 skipped)uv run ruff check— cleanbash scripts/build_pages_site.shend-to-end; verifiedsite/robots.txt,site/sitemap.xml, andsite/og-card.pngare emitted, the JSON-LD block parses as valid JSON, and OG/Twitter/canonical tags are present in the builtsite/index.html;pyproject.tomlvalidated withtomllibDocumentation
CHANGELOG.mdupdated under[Unreleased]docs/command_spec.mdupdated (command surface unchanged)docs/event-schema.mdupdated (structured-output shape unchanged)docs/design/spec updated (no design specs touched)docs/tests/spec updated (no test specs touched)AGENTS.md/docs/agents.mdupdated (agent workflows unchanged)mkdocs.ymlnav updated (no new docs pages)Safety
Not applicable — no active-bus behaviour changed.
🤖 Generated with Claude Code
https://claude.ai/code/session_01Mf6GAt4wpvSXPTDKGu2q6B
Generated by Claude Code