Skip to content

Stop the import silently discarding plays and whole files - #73

Merged
taurheim merged 1 commit into
fix/scrobble-timestamp-collapsefrom
fix/import-data-loss
Jul 27, 2026
Merged

Stop the import silently discarding plays and whole files#73
taurheim merged 1 commit into
fix/scrobble-timestamp-collapsefrom
fix/import-data-loss

Conversation

@taurheim

Copy link
Copy Markdown
Owner

Stacked on #72 — base is fix/scrobble-timestamp-collapse, so review that one first. GitHub will retarget this to master automatically once #72 merges.

I found these by grouping scrobblify_error events in PostHog over the last 60 days and tracing each back through the import path. All four silently lose user data.

1. A track with no duration on Last.fm was silently discarded

getTrackTimeMs did parseInt(response.track.duration, 10). Last.fm happily returns tracks with a missing, empty or non-numeric duration, which gave NaN. Because every comparison against NaN is false, it slipped past the === 0 fallback and the minimum-length check, then failed the final msListened > NaN — so the play was dropped.

parsed: NaN
=== 0?          false   <- missed the "pretend it is 2 minutes" fallback
< 30000?        false   <- missed the minimum-length rejection
isValid?        false   <- dropped anyway, at the last comparison

The comment directly above that code says the opposite is intended: "It's better to err on the side of giving them the scrobble as it will help populate their last.fm history." Unknown durations now normalise to 0, which that fallback already handles.

2. Durations were fetched once per play, not once per track

Each lookup carries 250ms of enforced rate buffer. A listening history is overwhelmingly repeat plays — that is the entire premise of the app — so this multiplied every "Validate track lengths" run by the user's average play count. Now cached per (artist, track) on the LastFm instance.

Permanent "track not found" answers are cached too, since a track played 40 times was otherwise looked up and rejected 40 times. Rate limits and network errors are deliberately not cached.

3. Fractional timestamps sent to user.getrecenttracks

"from":"1784563202.848","to":"1784563622.848"

dateToSecondsString did (date.getTime() / 1000).toString(). Last.fm documents UNIX timestamps; this is the same class of bug already fixed for track.scrobble. The two ends now round outwards, so a duplicate-check window can only ever look slightly too far — a window that misses writes a real duplicate into the user's library, one that overshoots costs nothing.

4. One bad file abandoned the whole import

A Spotify export is split across many Streaming_History_Audio_*.json files, and any single one failing to extract or parse returned early and discarded everything. Observed on three separate users:

  • JSON Parse error: Unrecognized token ' '
  • Expected double-quoted property name in JSON at position 2468006 (line 76959 column 26)
  • Unexpected token ' ', " Ma"... is not valid JSON

The position deep inside the file points at truncated or partially-read data, which the existing code comments already attribute to mobile Safari memory limits on large exports. Damaged files are now skipped with a visible warning naming the file and stating that part of the history is missing; only a ZIP where every history file fails is an error.

Testing

41/41 pass. Five new tests, and I verified each one fails with the source changes stashed so they are genuine regression tests rather than tests written to match the new code:

5 failed
  one malformed history file does not throw away the whole import
  a ZIP where every history file is unreadable still reports an error
  a repeated track is looked up once, not once per play
  a track with no duration on Last.fm is kept, not silently discarded
  duplicate-check requests use integer timestamps
    Expected pattern: /^\d+$/
    Received string:  "1785028584.929"

Not fixed here

  • vue.errorHandler: Cannot read properties of undefined (reading 'has') — one occurrence, one user. I traced every .has( call site in SelectStep and could not construct a path where the non-reactive Sets are unset by the time a computed reads them. Left alone rather than guessed at.
  • spotifyJsonToListens splits the artist on ', ' and keeps the first element, which mangles artists whose names contain a comma (Tyler, The Creator becomes Tyler). Real, but there is no reliable way to tell that apart from a genuine multi-artist list without another lookup, so it needs a design decision rather than a patch.

Four separate defects found by grouping scrobblify_error events in
PostHog and tracing each back through the import path.

1. A track whose Last.fm entry has a missing, empty or non-numeric
   duration was silently thrown away. parseInt returned NaN, and because
   every comparison against NaN is false it slipped past the "pretend
   it is 2 minutes" fallback *and* the minimum-length check, only to
   fail msListened > NaN at the end. The stated intent of that code is
   to err on the side of giving the user the scrobble.

2. Track durations were looked up once per *play* rather than once per
   distinct track, each with 250ms of enforced rate buffer. A listening
   history is mostly repeat plays, so this multiplied validation time by
   the user's average play count. Now cached per (artist, track),
   including permanent "not found" answers; rate limits and network
   errors stay uncached.

3. user.getrecenttracks was sent fractional timestamps
   (from=1784563202.848). Last.fm documents UNIX timestamps. Same class
   of bug as the one already fixed for track.scrobble. Window ends now
   round outwards, so a duplicate check can only ever look too far,
   never miss.

4. One unreadable or malformed history file abandoned the entire
   import. Large exports genuinely do come back truncated or corrupted
   on memory-constrained mobile browsers, seen across three users.
   Damaged files are now skipped with a visible warning and the rest is
   imported; only a ZIP where every file fails is an error.

Each of the five new tests was confirmed to fail with the source changes
stashed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f0df23dc-ed73-4b56-9b7c-4d2846ddf2a2
@taurheim
taurheim merged commit 589e9ea into fix/scrobble-timestamp-collapse Jul 27, 2026
taurheim added a commit that referenced this pull request Jul 27, 2026
`on: pull_request: branches: [master]` only triggers for PRs whose *base*
is master, so a PR stacked on another PR silently ran no checks at all.
That is the PR most in need of them, since it is reviewed against a base
that has not landed yet.

Verified against PR #73, which targets a feature branch:
"no checks reported on the 'fix/import-data-loss' branch".

ci.yml still handles push-to-master, so this does not double up.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f0df23dc-ed73-4b56-9b7c-4d2846ddf2a2
@taurheim
taurheim deleted the fix/import-data-loss branch August 4, 2026 03:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant