feat(web): migrate apps/web to static Astro app - #9
Merged
Conversation
…ontent Four markdown drafts in apps/web/content/ for the SEO pages researchers actually need before they download. Numbers are grounded in live API and DB queries (verified 2026-05-19), not brand.md (some figures there are stale). Limitations are first-class per brand voice. Drafts include frontmatter (title, description, canonicalPath, status, lastVerified, primaryQuery, secondaryQueries) so the Astro migration can consume them as page source rather than rewriting from scratch. Not wired into the build. Routes don't exist yet; these are content input for the upcoming Astro migration (PR 3).
Replaces the static-HTML + generate-pages.ts setup with a static Astro
build. Output stays purely static so deployment can keep using R2 +
Cloudflare without a Worker between users and raw /documents/* files.
What's new
- 4 SEO content pages (/dataset, /classification, /quality, /download)
rendered from the markdown drafts in apps/web/content/
- /types/{type} and /topics/{topic} dynamic routes generated at build
time from live Neon queries via @neondatabase/serverless
- Generated /sitemap.xml, /robots.txt, /llms.txt routes (live numbers,
no more stale hand-edited copies)
- Shared Layout.astro with canonical/OG/Twitter/JSON-LD/breadcrumb
plumbing in one place
What's preserved
- The existing client-rendered homepage (apps/web/public/index.html)
copies through Astro's public/ into dist/ unchanged; the explorer
port is out of scope for this PR
- The api.docxcorp.us worker (apps/web/worker/) is untouched
- /documents/* and /extracted/* keep noindex via Cloudflare Transform
Deploy workflow
- bun install --frozen-lockfile then bun run --cwd apps/web build
- aws s3 sync uploads non-HTML assets with their natural extensions
- A find loop uploads HTML to extensionless keys (/classification.html
-> /classification) to match the canonical URLs; index.html is the
one exception and stays at /
- No --delete flag: the bucket also holds scraper-managed /documents/
and /extracted/
Cleanup
- Removed legacy static apps/web/{robots.txt, llms.txt, sitemap.xml};
Astro routes replace them
- Removed apps/web/scripts/{generate-pages,generate-stats,preview}.ts
- Top-level dev:web now runs astro dev
The Astro app's tsconfig.json used 'include: ["**/*"]', which pulled in apps/web/worker/src/index.ts and ran the worker's code through the Astro app's typecheck. The worker has its own runtime types and deployment; it should not be checked under the Astro app's config. Restrict the include to src/ only and exclude worker explicitly so each package owns its own type boundary. Also drop an unused import from types/[type].astro that the stricter pass surfaced.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
The web Astro build was only happening at deploy time, after merge. That made 'CI green' misleading for PRs touching apps/web because the build that actually ships was never run. Add two steps: - Astro check (fast, no env) - Astro build with DATABASE_URL (the dynamic /types/* and /topics/* routes need live counts at build time) Now every PR runs the same build that production runs.
Astro 6 requires Node >=22.12.0. The runner ships Node 20 by default, and setup-bun doesn't install Node, so astro check / astro build fail even when run via 'bun run'. Add actions/setup-node@v4 with node 22 to both ci.yml and deploy-site.yml before the Astro steps.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the static-HTML + generate-pages.ts setup with a static Astro build. Output stays purely static so deployment can keep using R2 + Cloudflare without a Worker between users and raw
/documents/*files. The api.docxcorp.us worker is untouched, and the existing client-rendered homepage rides through as a public/ asset, no port in this PR.Three commits in order:
apps/web/content/{dataset,classification,quality,download}.md, grounded in live/statsand DB queries (verified 2026-05-19)/types/{type}+/topics/{topic}dynamic routes generated at build time from Neon, generated/sitemap.xml+/robots.txt+/llms.txtroutes, sharedLayout.astrowith canonical/OG/JSON-LD/breadcrumb plumbingMust stay the same: the explorer JS on the homepage (
apps/web/public/index.html);/documents/*and/extracted/*keepX-Robots-Tag: noindexfrom the Cloudflare Transform Rule; api.docxcorp.us worker.Review: check the deploy workflow rewrite (
.github/workflows/deploy-site.yml) — extensionless HTML keys via a find loop,aws s3 syncwith no--deleteso scraper-managed/documents/and/extracted/keys are not touched. Skip the lefthook + format chatter in the diff; it's noise.Verified locally:
bun run --cwd apps/web typecheck-> 0 errorsDATABASE_URL=... bun run --cwd apps/web build-> 25 pages built in ~17sindex.htmlstays at/, everything else loses.html/public/favicon.ico->/favicon.ico(4 lines, only change to the legacy homepage)Heads up: merging this auto-deploys to production R2 via the existing push trigger. There is no staging environment. After merge, the workflow will run on main; smoke test:
curl -sI https://docxcorp.us/-> 200, homepage HTMLcurl -sI https://docxcorp.us/dataset-> 200, text/htmlcurl -s https://docxcorp.us/sitemap.xml | head -5-> shows new routescurl -sI https://docxcorp.us/documents/<hash>.docx-> still 200 +x-robots-tag: noindexcurl -s https://docxcorp.us/llms.txt | head -10-> live countsIf anything regresses, revert the merge and dispatch the workflow against main; R2 will be re-uploaded from the previous good state on the next deploy.
Out of scope (follow-ups):