fix(pipeline): dedupe npm search pagination drift - #11
Merged
Merged
Conversation
The npm search API's `total` drifts between page requests (packages are published/unpublished mid-enumeration), so the same package can land on two adjacent pages. enumeratePackages collected every object without deduping by name, producing duplicate rows in the published index (~250 dups in the Jul 14 run). - enumeratePackages: track seen names in a Set, skip duplicates at the source - buildIndex: defensive dedup by name (keeps first occurrence) so a stale search cache predating the fix can never produce duplicate rows either - add pipeline/normalize.test.ts (node:test via tsx) covering both dedup paths - wire `pnpm test:run`
Corrective dedup of the committed index so the live site serves clean data on the next deploy instead of waiting for the 04:00 UTC cron to regenerate from scratch. No API access needed — pure row dedup (keeps first occurrence). Generated by the same dedup logic now in buildIndex; count field realigned to 5250. The next pipeline cron will refresh from npm with the pagination fix.
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Resolve data/packages.json + .min.json conflict by taking origin/main's latest CI-generated data (6000 rows, 750 dups — root cause still unfixed on main) and applying the same dedup: 6000 -> 5250. readmes.json merged cleanly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The live index (
?q=pi-subagents) showed duplicate rows — the same package appearing 2–4× with identical stats. Root cause:data/packages.jsoncarried 250 duplicate package names (5500 rows, 5250 unique).The npm search API paginates with
from/size=250, and itstotaldrifts between page requests (packages published/unpublished mid-enumeration). Whentotalshifts, the same package lands on two adjacent pages.enumeratePackagescollected every object without deduping by name → duplicates flowed into the committedpackages.jsonand onto the live site. CI is vulnerable every cron run because itsdata/.cache/is gitignored + ephemeral.Fix (3 layers)
pipeline/npm.ts— dedup at the source.enumeratePackagestracks aSet<string>of seen names and skips duplicates as pages arrive.pipeline/normalize.ts— defensive dedup inbuildIndex. Keeps first occurrence by name, so a stale search cache predating the fix can never produce duplicate rows either.data/packages.json+packages.min.json— corrected in place. Dropped the 250 duplicate rows immediately so the live site is clean on the next deploy instead of waiting for the 04:00 UTC cron.Tests
Added
pipeline/normalize.test.ts(node:test viatsx, no new deps) + wiredpnpm test:run:Every consumer reads the now-deduped source: SSG pages (
src/lib/data.ts),/api/packagesroute (getIndex()), andpublic/data/*.json(scripts/copy-data.mjsprebuild).Commits split: code+tests, then corrective data dedup.