Stop the import silently discarding plays and whole files - #73
Merged
Conversation
Four separate defects found by grouping scrobblify_error events in PostHog and tracing each back through the import path. 1. A track whose Last.fm entry has a missing, empty or non-numeric duration was silently thrown away. parseInt returned NaN, and because every comparison against NaN is false it slipped past the "pretend it is 2 minutes" fallback *and* the minimum-length check, only to fail msListened > NaN at the end. The stated intent of that code is to err on the side of giving the user the scrobble. 2. Track durations were looked up once per *play* rather than once per distinct track, each with 250ms of enforced rate buffer. A listening history is mostly repeat plays, so this multiplied validation time by the user's average play count. Now cached per (artist, track), including permanent "not found" answers; rate limits and network errors stay uncached. 3. user.getrecenttracks was sent fractional timestamps (from=1784563202.848). Last.fm documents UNIX timestamps. Same class of bug as the one already fixed for track.scrobble. Window ends now round outwards, so a duplicate check can only ever look too far, never miss. 4. One unreadable or malformed history file abandoned the entire import. Large exports genuinely do come back truncated or corrupted on memory-constrained mobile browsers, seen across three users. Damaged files are now skipped with a visible warning and the rest is imported; only a ZIP where every file fails is an error. Each of the five new tests was confirmed to fail with the source changes stashed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f0df23dc-ed73-4b56-9b7c-4d2846ddf2a2
taurheim
added a commit
that referenced
this pull request
Jul 27, 2026
`on: pull_request: branches: [master]` only triggers for PRs whose *base* is master, so a PR stacked on another PR silently ran no checks at all. That is the PR most in need of them, since it is reviewed against a base that has not landed yet. Verified against PR #73, which targets a feature branch: "no checks reported on the 'fix/import-data-loss' branch". ci.yml still handles push-to-master, so this does not double up. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f0df23dc-ed73-4b56-9b7c-4d2846ddf2a2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #72 — base is
fix/scrobble-timestamp-collapse, so review that one first. GitHub will retarget this tomasterautomatically once #72 merges.I found these by grouping
scrobblify_errorevents in PostHog over the last 60 days and tracing each back through the import path. All four silently lose user data.1. A track with no duration on Last.fm was silently discarded
getTrackTimeMsdidparseInt(response.track.duration, 10). Last.fm happily returns tracks with a missing, empty or non-numeric duration, which gaveNaN. Because every comparison againstNaNis false, it slipped past the=== 0fallback and the minimum-length check, then failed the finalmsListened > NaN— so the play was dropped.The comment directly above that code says the opposite is intended: "It's better to err on the side of giving them the scrobble as it will help populate their last.fm history." Unknown durations now normalise to
0, which that fallback already handles.2. Durations were fetched once per play, not once per track
Each lookup carries 250ms of enforced rate buffer. A listening history is overwhelmingly repeat plays — that is the entire premise of the app — so this multiplied every "Validate track lengths" run by the user's average play count. Now cached per
(artist, track)on theLastFminstance.Permanent "track not found" answers are cached too, since a track played 40 times was otherwise looked up and rejected 40 times. Rate limits and network errors are deliberately not cached.
3. Fractional timestamps sent to
user.getrecenttracksdateToSecondsStringdid(date.getTime() / 1000).toString(). Last.fm documents UNIX timestamps; this is the same class of bug already fixed fortrack.scrobble. The two ends now round outwards, so a duplicate-check window can only ever look slightly too far — a window that misses writes a real duplicate into the user's library, one that overshoots costs nothing.4. One bad file abandoned the whole import
A Spotify export is split across many
Streaming_History_Audio_*.jsonfiles, and any single one failing to extract or parse returned early and discarded everything. Observed on three separate users:JSON Parse error: Unrecognized token ' 'Expected double-quoted property name in JSON at position 2468006 (line 76959 column 26)Unexpected token ' ', " Ma"... is not valid JSONThe position deep inside the file points at truncated or partially-read data, which the existing code comments already attribute to mobile Safari memory limits on large exports. Damaged files are now skipped with a visible warning naming the file and stating that part of the history is missing; only a ZIP where every history file fails is an error.
Testing
41/41 pass. Five new tests, and I verified each one fails with the source changes stashed so they are genuine regression tests rather than tests written to match the new code:
Not fixed here
vue.errorHandler: Cannot read properties of undefined (reading 'has')— one occurrence, one user. I traced every.has(call site inSelectStepand could not construct a path where the non-reactive Sets are unset by the time a computed reads them. Left alone rather than guessed at.spotifyJsonToListenssplits the artist on', 'and keeps the first element, which mangles artists whose names contain a comma (Tyler, The CreatorbecomesTyler). Real, but there is no reliable way to tell that apart from a genuine multi-artist list without another lookup, so it needs a design decision rather than a patch.