From c46db3dbca38868e695f2c1a1ae4781f0f08ebc3 Mon Sep 17 00:00:00 2001 From: Mohammad Saddam Hossain Date: Sun, 19 Jul 2026 11:18:05 +0530 Subject: [PATCH] docs: add robots.txt for docs.seleniumboot.com MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The docs site served no robots.txt (404). Not blocking — crawlers treat an absent file as allow-all — but it meant the sitemap was only discoverable via a direct Search Console submission. Declares allow-all plus the sitemap location so any crawler can find the 53 doc URLs without going through a console. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs-site/static/robots.txt | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 docs-site/static/robots.txt diff --git a/docs-site/static/robots.txt b/docs-site/static/robots.txt new file mode 100644 index 0000000..7dee054 --- /dev/null +++ b/docs-site/static/robots.txt @@ -0,0 +1,4 @@ +User-agent: * +Allow: / + +Sitemap: https://docs.seleniumboot.com/sitemap.xml