fix(data): stop the weekly refresh from silently halving the training set

[Sagargupta16]

Closes the hole that produced #162, which proposed deleting 121,437 training records with every check green.

What #162 actually was

It was the refresh job's first successful run since 2026-08-17, and it looked clean. Counted directly with git show <ref>:data/data.json | wc -l:

	Records	Contributing users	Records/user
main	244,950	6,830	35.9
#162	123,513	3,440	35.9

Identical records-per-user, so per-user data quality was fine. The entire 49.6% shortfall was fewer users getting through. (Users counted via records seeded at f1 == 1500.0, each user's first attended contest.)

Three independent causes

1. CI silently processed 11.6% of the list. main() prompted for a user count. Under Actions there is no stdin, so input() raised EOFError and the handler fell back to min(5000, len(usernames)) out of 43,158. Since the committed dataset came from 6,830 successful users, the cron could never reproduce it: even a flawless 5,000-user run lands ~27% short. Now --users, defaulting to 8,000.

2. Throttling was indistinguishable from an empty account. fetch_user_contest_history returned [] on any non-200, exactly like a user with no contest history. The run log shows Failed/No data: 1560 out of 5,000, all silently treated as "nothing to contribute". Now retryable statuses (429, 5xx) and network errors retry with exponential backoff honouring Retry-After, and a failed fetch returns None so the summary reports failures separately from empty accounts.

3. Nothing compared new against old. main() wrote data.json unconditionally. A run retaining less than --min-retention (default 95%) now aborts non-zero, which fails the job and skips PR creation. --force overrides for an intended shrink.

Contest histories only grow and usernames.json is fixed, so a much smaller result means a broken fetch, not smaller data.

Why the cap is not simply "fetch everyone"

data/data.json is committed, so it lives under GitHub's 100 MB per-file hard limit. At ~231 bytes/record and ~35.9 records/user, that is ~8.3 KB per contributing user:

Users attempted (at 90% success)	Records	Size
5,000 (old CI behaviour)	161k	35.6 MB
8,000 (new default)	258k	57.0 MB
12,000	387k	85.4 MB
43,158 (all)	1.39M	307 MB, unpushable

8,000 clears the 6,830 users behind the current dataset without approaching the limit. The script logs the written size and warns past 90 MB.

Also

• Writes via a temp file plus os.replace, so an interrupted run cannot truncate the committed dataset.
• Drops a time.sleep(0.05) in the result-consuming loop. Every future is submitted up front, so it throttled nothing and only added latency: 250s of pure sleep per 5,000 users.

Verification

25 offline checks against a fake session: 429 exhaustion returns None not []; a 404 is not retried; Retry-After is honoured and capped; retry-then-succeed takes exactly 3 calls; collect_records counts failed and empty separately.

18 end-to-end checks driving main() against a throwaway repo layout:

• shrinking run exits 1 and leaves data.json at its original count, no .tmp residue
• --force overrides and writes
• growth and healthy runs write normally
• an all-failed run never writes an empty file
• --min-retention is tunable, --users caps the slice

The blocked run reports:

ERROR - Refusing to overwrite data/data.json: 20 new records against 1000 on
disk (2.0%, under the 95% floor). 0 of 4 fetches failed, which usually means
LeetCode throttled the run. Re-run it, or pass --force if the shrink is intended.

Repo-wide ruff check . clean, black/isort clean, 59/59 pytest passing, all three workflow YAMLs parse.

Once this lands, #162 should be closed rather than merged; the next scheduled run regenerates it against the same branch.

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud