fix(data): raise the user cap so a healthy refresh clears the retention floor - #164
Merged
Merged
Conversation
…on floor The 2026-09-03 scheduled run failed, and the guard was right while the cap was wrong. That run fetched all 8,000 users with ZERO failures, collected 197,991 records against the 244,950 committed, and tripped the 95% floor at 80.8%. DEFAULT_MAX_USERS was sized on an assumed ~90% contribution rate. The measured rate is 68.2%: 5,460 of 8,000 users contributed records and 2,540 have no contest history at all, so roughly a third of usernames.json belongs to accounts that never entered a contest. At that rate 8,000 attempts cannot reach the committed record count, so every scheduled run would have failed indefinitely. Raised to 12,000: ~8,190 contributors, ~297K records, ~65 MB. That clears the committed dataset by 21%, leaving headroom so ordinary variance does not abort, and stays well below GitHub's 100 MB per-file limit (which allows ~18,300 attempts). The constant now carries the measured table so the next adjustment starts from data instead of a guess. Also fixes the abort message, which blamed throttling on a run with no failures: "0 of 8000 fetches failed, which usually means LeetCode throttled the run". It now branches. With failures it points at throttling and suggests a re-run. With none it states that throttling is not the cause, that --users is too low, and explicitly says to raise the cap rather than lower the floor. The floor itself is unchanged at 95%. Lowering it would have let the dataset shrink permanently, which is the outcome it exists to prevent. Verified with 11 checks covering both message branches: a healthy undershoot aborts, leaves data.json untouched, and names the cap; a throttled run names throttling and suggests a re-run. ruff and black clean, uv.lock relocked for the version bump so `uv sync --locked` still passes.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The 2026-09-05 scheduled run failed. The guard was right; my cap was wrong.
What actually happened
Zero fetch failures. Nothing was throttled, nothing errored -- the run was completely healthy and still could not reach the committed record count.
Root cause: I sized the cap on a guess
DEFAULT_MAX_USERSassumed a ~90% contribution rate. Measured from this run:Roughly a third of
usernames.jsonbelongs to accounts that never entered a contest. At 68.2%, 8,000 attempts caps out around 198K records against 245K committed -- so every scheduled run would have failed indefinitely, not just this one.Fix: 12,000, chosen from the measured numbers
12,000 leaves enough room above the floor that ordinary variance will not abort a run, and stays well under GitHub's 100 MB per-file hard limit (which allows ~18,300 attempts). The constant now carries this table inline so the next adjustment starts from data.
The 95% floor is unchanged. Lowering it would let the dataset shrink permanently, which is the exact outcome it exists to prevent.
Also: the abort message blamed the wrong thing
It said "0 of 8000 fetches failed, which usually means LeetCode throttled the run" -- on a run with no failures. That sends you chasing a phantom throttle. It now branches:
--usersis too low, and that the fix is raising the cap rather than lowering the floorVerification
11 checks, all passing:
data.jsonuntouched, message names the cap and explicitly does not claim a throttleDEFAULT_MAX_USERSis 12000,DEFAULT_MIN_RETENTIONstill 0.95ruffandblackclean.uv.lockrelocked for the version bump souv sync --lockedstill passes (that bit me on the last PR).