From f4db8e32732cf3c9ed24bb5656cc6badc9ee8ae1 Mon Sep 17 00:00:00 2001 From: giswqs Date: Mon, 10 Aug 2026 22:41:44 -0400 Subject: [PATCH] ci: keep studio.geolibre.app out of search results MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The gated instance is meant to be reached by invitation, and nothing on it is useful to a crawler — every route renders the sign-in card until a session exists. Write a robots.txt disallowing everything, and append a noindex meta to for crawlers that ignore it. The two deliberately do not compose: a crawler honouring the Disallow never fetches the page and so never reads the noindex, which is why a URL linked from elsewhere can still be listed bare. Noted inline, with the switch to make (relax robots.txt to Allow) if studio is ever linked publicly. The meta is injected into generated markup, so the step greps for it afterwards and fails rather than publishing an indexable page if a future index.html stops matching. --- .github/workflows/studio-deploy.yml | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/.github/workflows/studio-deploy.yml b/.github/workflows/studio-deploy.yml index 14ec32b8a9..2614ba4ee7 100644 --- a/.github/workflows/studio-deploy.yml +++ b/.github/workflows/studio-deploy.yml @@ -129,6 +129,24 @@ jobs: touch .nojekyll # Keeps the custom domain if the Pages settings are ever reset. echo "studio.geolibre.app" > CNAME + # Keep the gated instance out of search results. Nothing here is + # useful to a crawler — every route renders the sign-in card until a + # session exists — and the deployment is meant to be reached by + # invitation, not found. + printf 'User-agent: *\nDisallow: /\n' > robots.txt + # A second signal for crawlers that ignore robots.txt. Note the two + # do not compose: a crawler that honours the Disallow above never + # fetches this page and so never reads the noindex, which is why a + # URL linked from elsewhere can still be listed (bare, no title). If + # studio is ever linked publicly and that matters, relax robots.txt + # to "Allow: /" so the noindex is the rule that applies. + # Appended to the end of rather than the start, so the charset + # declaration keeps its place as the first thing in the document. + sed -i '0,/<\/head>/s// \n <\/head>/' index.html + # The tag is injected into generated markup, so fail loudly if a + # future index.html no longer matches rather than publishing an + # indexable page. + grep -q 'name="robots"' index.html # Published Pages sites may be no larger than 1 GB. The build is # ~200 MB today, most of it the two DuckDB-WASM binaries; fail early # rather than publish a site Pages will reject.