Skip to content

Fix repeat listens being silently dropped, and rate-limit handling - #72

Merged
taurheim merged 6 commits into
masterfrom
fix/scrobble-timestamp-collapse
Jul 27, 2026
Merged

Fix repeat listens being silently dropped, and rate-limit handling#72
taurheim merged 6 commits into
masterfrom
fix/scrobble-timestamp-collapse

Conversation

@taurheim

Copy link
Copy Markdown
Owner

The bug

Last.fm keys a scrobble on (user, artist, track, timestamp) and silently discards any repeat of that tuple. It still returns accepted=1, ignored=0, so the loss is undetectable from the response. This is undocumented — I verified it experimentally against the live API (three scrobbles, two distinct timestamps, two plays landed).

reTagOldListens assigned the same Date object to every play moved into the 14-day window. So if you listened to a song five times, four of those were destroyed.

That path runs whenever "Scrobble tracks older than 2 weeks" is ticked: 300 of 386 parses, 80 of 119 people over the last 60 days.

Separately, scrobblePlay discarded the API response, so scrobbles Last.fm explicitly did report as ignored were counted as successes.

The fix

Timestamps for re-tagged plays are allocated at send time from a cursor, not baked in at parse time:

reTagCursorSec = min(nowSec, max(reTagCursorSec + 1, nowSec - RETAG_BACKFILL_SECONDS))

Send-time allocation rather than parse-time matters for two reasons beyond uniqueness:

  • Spreading by real track duration only fits ~5,700 tracks in the 14-day window. At 1s spacing, 80,000 fits in 22 hours.
  • Absolute timestamps baked in at parse time expire. A queue saved today and resumed three weeks later would be rejected wholesale with ignore code 3.

The cursor persists as a high-water mark (lastReTagTimestampSec) so a resumed run can't reuse seconds an earlier run already sent, and is allocated once per track, not once per attempt — if the original request reached Last.fm and only the response was lost, an identical resend is deduplicated away, whereas a fresh second would become a phantom play in the user's library.

scrobblePlay now returns a ScrobbleResult. Parsing defaults to accepted on an unrecognised shape, so a surprise response can never invent failures.

Also in this PR

  • Rate limiting reworked (RateLimitTracker): rolling window of send timestamps with a limit learned adaptively from observed 429s, escalating backoff (5/15/30 min), then an honest give-up-and-save. The old flat 1-minute-forever retry produced 33 recoveries across 2,135 observed rate limits, median 130 minutes — two users sat through 200+ consecutive retries watching a countdown that was never going to work.
  • Resume was silently broken: a 2-second delayed emit in AuthenticateStep undid a resume the user had already started.
  • Completion telemetry: original_total_tracks, total_succeeded, completion_pct, is_resumed, previously_scrobbled. Completion was previously measured against the remaining chunk, which shrinks on resume — real cases went 81,313 -> 573 and 9,924 -> 40, so a barely-started import looked nearly finished.
  • Migration for in-flight users: a saved session whose timestamps are >=90% identical (across >=20 tracks) is re-stamped on load.
  • filterDuplicates skips re-tagged listens, whose provisional timestamps would otherwise be matched against real listening history and silently drop import rows.

Testing

36/36 Playwright tests pass, including four new ones covering distinct timestamps, ignored-scrobble accounting, and retry idempotency. Lint 0 errors, build clean.

Known unfixed (pre-existing)

  • failedIndices is always written empty, so failed tracks are never retried on resume.
  • total_succeeded is an upper bound: re-scrobbling history a user already has is indistinguishable from a fresh play.

taurheim and others added 6 commits July 26, 2026 14:40
Documents the architecture for a backend that continues large Last.fm
imports while the user's browser is closed, offered only when a selection
exceeds the 2,700/day limit.

Key decisions:
- Cloudflare Workers free tier to start, with explicit portability rules
  and a ~24-concurrent-job ceiling as the trigger to move to a VM.
- Auth handoff via a second Last.fm round-trip so the client's session
  key is never transmitted; server holds its own, deleted on completion.
- Duplicate safety by reconciling against Last.fm rather than relying on
  cursor granularity, since the ambiguity window is irreducible.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2c093209-9749-4d93-990f-c970f26d570e
…baseline

Reconciling against user-supplied timestamps does not work: reTagOldListens
assigns one identical Date instance to every listen, so re-tagged imports
carry a single timestamp across tens of thousands of tracks. A 37-day job
also outlives Last.fm's 14-day window, so client-assigned timestamps expire
mid-run regardless.

The worker now assigns scrobble timestamps at send time and persists the
index-to-timestamp mapping before sending, giving a unique idempotency key
we control and narrowing reconciliation to one batch.

Also records the PostHog baseline: median user completes 2.1% of their
import, reactive rate limits outnumber preventive pauses ~9:1, and the
1-minute cooldown clears the limit in 33 of 2,135 cases. Fixing the client
cooldown and thresholds should ship before this feature.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2c093209-9749-4d93-990f-c970f26d570e
…g spec

Background mode is now gated on tracks *remaining* rather than selected, and
offered at four entry points: new import, resume from saved state, resume from
file, and the mid-scrobble pause screen. The pause screen matters most, since
telemetry shows that is where users abandon.

Adds rules for handing off partial progress atomically, halting the client loop
on handoff to avoid double-scrobbling, tolerating old state files, and falling
back to the client-side flow.

Beta feedback goes to niko@savas.ca via the prefilled-mailto pattern already in
ErrorDialog.vue, including the job ID, and must stay reachable on failed jobs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2c093209-9749-4d93-990f-c970f26d570e
Verified against Last.fm's published API docs and corrected the design:

- track.scrobble batches up to 50 scrobbles per request. Rewrote the
  capacity section and withdrew the ~24-concurrent-user ceiling, which
  was computed without batching.
- Error 29 is an IP-level limit; the per-user daily cap arrives as
  ignoredMessage code 5 on an HTTP 200. The client conflates the two,
  which explains why half of all first rate limits occur at
  burst_count = 0. Added error 26 as a global-halt condition.
- savas.ca/lastwave shares the origin, so Domain=savas.ca gives no
  isolation. Moved to a __Host- cookie on api.savas.ca plus explicit
  CSRF tokens, and raised a dedicated origin as an open question.
- Specified the auth handoff as a transaction: persist before redirect,
  single-use nonce bound to the expected username, digest-validated
  upload, finalize before clearing client state, reap abandoned jobs.
- Replaced lease-based locking with fencing tokens, since an expired
  lease does not stop an in-flight tick from writing.
- Withdrew the exactly-once claim in favour of an explicit
  at-least-once choice, and documented why reconciliation is imperfect.
- Reframed timestamp reassignment as history rewriting, with an
  explicit ordering trade-off and a decision to go recent-first.
- Fixed internal contradictions around threshold reuse and retention.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2c093209-9749-4d93-990f-c970f26d570e
Round two found that several round-one fixes were themselves wrong:

- Recent-first ordering was backwards. A track played today has 14 days
  of slack before its timestamp expires; one played 13 days ago has one.
  Sending recent tracks first spends slack on the tracks needing it
  least. Now earliest-deadline-first over two queues.
- The __Host- cookie plus bearer token did not isolate LastWave.
  Cookies attach by destination, not initiator, so a same-site fetch
  from savas.ca carries the cookie anyway; and a token the SPA can read
  LastWave can also read. Replaced with an explicit trust-boundary
  decision rather than ineffective crypto.
- The handoff never committed the digest server-side that step 4
  claimed to verify. Added a preflight step and a compare-and-set state
  machine covering seven previously unenumerated ambiguous states.
- A single cursor cannot represent a 50-entry batch containing accepted,
  permanently failed, rate-limited and unknown entries. Progress is now
  per-entry, with the cursor advancing only over a terminal prefix.
- Fencing tokens protect D1 but cannot revoke an in-flight Last.fm
  request; documented as part of the at-least-once window.
- An immutable payload does not make behaviour immutable. Jobs now pin
  an algorithm version for parsing, ordering and timestamps.
- The concurrency cap still cited the withdrawn capacity ceiling. It is
  now blocked on a feasibility spike, which is a design gate.

Also added chunk upload limits against compression bombs, admission
control for jobs too large to finish within the credential TTL, and a
correction to the claim that listening history is not personal data.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2c093209-9749-4d93-990f-c970f26d570e
Last.fm keys a scrobble on (user, artist, track, timestamp) and silently
discards any repeat of that tuple, still reporting accepted=1 ignored=0.
Verified experimentally against the live API; it is undocumented and
undetectable from the response.

reTagOldListens assigned the *same* Date object to every play moved into
the 14-day window, so every repeat listen of a track collapsed into one
scrobble. That path runs on ~78% of imports (300/386 parses, 80/119
people over 60 days).

Timestamps for re-tagged plays are now allocated at send time from a
cursor rather than baked in at parse time, which also fixes queues saved
today and resumed weeks later being rejected wholesale with ignore code 3.
The cursor persists as a high-water mark so a resumed run cannot reuse
seconds an earlier run already sent, and is allocated once per track so a
retry re-sends an identical tuple instead of creating a phantom play.

scrobblePlay now parses the response: a 200 did not mean the play was
stored, so ignored scrobbles were being counted as successes.

Also in this change:

- Client-side rate limiting reworked (RateLimitTracker): rolling window of
  send timestamps with an adaptive limit learned from observed 429s,
  escalating backoff, and an honest give-up-and-save instead of an
  infinite 1-minute retry. Across 2,135 observed rate limits the old
  behaviour produced 33 recoveries, median 130 minutes to recover.
- Fixed a 2-second delayed emit in AuthenticateStep that silently undid a
  resume the user had already started.
- Resume telemetry: original_total_tracks, total_succeeded, completion_pct,
  is_resumed and previously_scrobbled, so completion can be measured
  against the original import rather than the shrinking remaining chunk.
- Legacy saved sessions with collapsed timestamps are migrated on load for
  users who are mid-import.
- filterDuplicates skips re-tagged listens, whose provisional timestamps
  would otherwise be matched against real history.

Known unfixed, pre-existing: failedIndices is always written empty, so
failed tracks are never retried on resume. total_succeeded remains an
upper bound, since re-scrobbling history a user already has is
indistinguishable from a fresh play.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f0df23dc-ed73-4b56-9b7c-4d2846ddf2a2
@taurheim
taurheim merged commit df6eceb into master Jul 27, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant