fix(reports): default report scripts to the latest crawl, not the historical union - #5
Merged
Merged
Conversation
Adds --concurrency, --delay-min/--delay-max, --max-retries and --block-assets to src/main.ts, overriding config/crawler.yml for a single run so a deliberately gentle crawl doesn't require editing the shared config. --block-assets skips CSS/images/fonts/JS subresource requests during navigation via Crawlee's playwrightUtils.blockRequests — the main document is never blocked, so extractors still see full server-rendered markup (title, meta, canonical, JSON-LD, links); JS-rendered content will not appear. Documents in AGENTS.md why --concurrency/--delay-min/--delay-max/ --block-assets are preferable to --rate-limit for low-load crawling: the rate limiter's sleep happens inside the request handler, which Crawlee bounds by requestHandlerTimeoutSecs (60s) — a throttle longer than that aborts as a timeout and gets retried, increasing load instead of reducing it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…torical union storage/datasets/<domain>/<domain>.jsonl accumulates every crawl of a domain into one merged JSONL. dedupePagesByUrl() correctly keeps only the newest record per URL, but never drops a URL the latest crawl didn't visit at all — a page retired months ago (now redirected, so nothing links to it and the crawler never reaches it again) keeps its stale record forever, and that record is still "newest" for that URL. Reports built on the full union therefore described a site state that had stopped existing: on ludekkvapil.cz this surfaced as 9 phantom 404s and 40 phantom missing-JSON-LD findings, all last crawled 2026-06-14, none of them true against the 2026-08-28 crawl. Adds splitByCrawlSnapshot()/selectSnapshot()/resolveSnapshotMode() to scripts/page-records.ts and wires them into report-404s.ts, report-seo-issues.ts, report-link-graph-issues.ts, generate-title-description-fixes.ts and seo-audit.ts. Default is now the newest crawl only; --all-crawls opts back into the historical union. Every report prints how many pages it's covering and how many stale URLs it excluded, so nothing disappears silently. report-404s.ts additionally keeps excluded 404s in a separate `stale_not_in_latest_crawl` field (and a snapshot/last_seen CSV column pair) instead of dropping them — a genuinely dead legacy URL stays visible for manual verification without being counted as a current issue. Adds src/tests/services/pageRecords.test.ts covering the new helpers. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
siva01c
added a commit
that referenced
this pull request
Aug 31, 2026
fix: address code review findings from PR #5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
storage/datasets/<domain>/<domain>.jsonlaccumulates every crawl of a domain into one merged JSONL.dedupePagesByUrl()correctly keeps only the newest record per URL, but never drops a URL the latest crawl didn't visit at all — a page retired months ago (now redirected, so nothing links to it and the crawler never reaches it again) keeps its stale record forever, and that record is still "newest" for that URL.Reports built on the full union therefore described a site state that had stopped existing. Verified against
ludekkvapil.cz: 9 phantom 404s and 40 phantom missing-JSON-LD findings, all last crawled2026-06-14, none of them true against the2026-08-28crawl (0 and 0 respectively).Fix
splitByCrawlSnapshot()/selectSnapshot()/resolveSnapshotMode()helpers inscripts/page-records.ts.report-404s.ts,report-seo-issues.ts,report-link-graph-issues.ts,generate-title-description-fixes.ts,seo-audit.ts.--all-crawlsopts back into the historical union.report-404s.tskeeps excluded 404s in a separatestale_not_in_latest_crawlfield (plus asnapshot/last_seenCSV column pair) instead of dropping them, so a genuinely dead legacy URL stays visible for manual verification without counting as a current issue.--concurrency,--delay-min/max,--max-retries,--block-assets) tosrc/main.tsfor gentler third-party crawls, with matchingAGENTS.mddocs — as its own commit.Verification
src/tests/services/pageRecords.test.ts(13 tests) covering the snapshot-splitting logic.npm test→ 260 passed,npm run styleclean,tsc --noEmitclean.npm run report:404 -- --domain ludekkvapil.cz→ 311 pages, 0 findings, 47 stale excluded,--all-crawlsrestores 358/9/40.Note: the branch was originally cut from
security/remove-internal-topology, which has since been merged and deleted — rebased cleanly onto currentmain.🤖 Generated with Claude Code