fix: make standalone checkouts work and repair page.evaluate under tsx

[siva01c]

Found by cloning this repo fresh from GitHub into a sandbox with no surrounding deployment tree — no ops-proxy, no agentic-ops network, no sibling repositories — and running the documented workflow end to end.

1. docker compose was unusable in any standalone checkout

The mcp service referenced ../.env.shared, a file that only exists in the deployment tree this repo is normally checked out into. Compose validates env_file for every service when it loads the project, not just the one being started, so a clean clone failed on every compose command — config, build, run --rm app, the test profile:

env file /path/to/../.env.shared not found

Marked optional with the long-form env_file syntax. The other two lkv couplings turn out not to block anything, verified rather than assumed:

coupling	verdict
../.env.shared	was a hard blocker for the whole file — now optional
agentic-ops external network	not a blocker — only required by services that attach to it, and app uses network_mode: host. Confirmed by repointing it at a network that does not exist and running app anyway
../ludekkvapil/public/seo	not a blocker for app/test/typecheck; only the mcp service mounts it

README gains a short "Standalone checkouts" section stating what a fresh clone does and does not need.

2. page.evaluate() threw on every page, so auto-scrolling never ran

npm run crawl is tsx, which transforms through esbuild, whose keepNames pass rewrites named function expressions as __name(fn, "name"). That rewrite also lands inside page.evaluate() callbacks, which Playwright serialises and executes in the browser — where __name is undefined. The headless scroll callback declares const scrollStep, so every page produced:

⚠️ Scrolling failed: page.evaluate: ReferenceError: __name is not defined

Each evaluate has its own try/catch, so this only ever surfaced as a warning: lazy-loaded content was never triggered before extraction and nobody noticed. Shimmed in a preNavigationHook rather than by banning named inner functions, so future callbacks cannot regress. Unused under node dist/ — tsc emits no __name.

3. …which then exposed that the scroll itself was pathological

With the callback finally running, the original walk showed its true cost: 50px every 800ms plus a 3-second pause every 15 steps — minutes on a long page. Crawlee bounds the handler by requestTimeoutSecs (60s), so every page aborted as a timeout and got retried: a crawl capped at 8 pages was still on Processing page 13/8 after ten minutes. More load on the target, not less — the same failure mode the load-shaping docs warn about for --rate-limit.

Scrolling exists to trigger lazy-loaded content, not to satisfy a bot detector, so it is now bounded three ways: reaching the bottom, a 40-step cap (backstop for infinite-scroll pages whose scrollHeight grows as you go), and a 5s wall-clock budget — all far below the handler timeout.

	before	after
8-page crawl	>10 min, killed	20 s
__name errors	every page	0
handler timeouts / retries	page 13/8	0, exactly 8/8

Browser globals are now hoisted to single-line consts: an eslint-disable comment covers only the following line, and Prettier is free to wrap a long expression across several, which silently uncovered them.

4. Manifest cosmetic

_crawl-meta.json wrote "previousCrawlDate": "" for full crawls. The field means "the crawl this one was diffed against", so it is omitted instead.

Verification

Fresh clone of this branch into an empty directory with no siblings present:

• all three docker compose config targets resolve; build, run, and the test profile work
• 8-page crawl of a real site in 20 s, 0 errors, 0 retries
• _crawl-meta.json written with no previousCrawlDate key
• report:404 and seo-audit produce correctly scoped output
• 275 tests pass in the sandbox image; tsc --noEmit clean; lint back at its 90-warning baseline, 0 errors

🤖 Generated with Claude Code

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.