Fix repeat listens being silently dropped, and rate-limit handling - #72
Merged
Conversation
Documents the architecture for a backend that continues large Last.fm imports while the user's browser is closed, offered only when a selection exceeds the 2,700/day limit. Key decisions: - Cloudflare Workers free tier to start, with explicit portability rules and a ~24-concurrent-job ceiling as the trigger to move to a VM. - Auth handoff via a second Last.fm round-trip so the client's session key is never transmitted; server holds its own, deleted on completion. - Duplicate safety by reconciling against Last.fm rather than relying on cursor granularity, since the ambiguity window is irreducible. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2c093209-9749-4d93-990f-c970f26d570e
…baseline Reconciling against user-supplied timestamps does not work: reTagOldListens assigns one identical Date instance to every listen, so re-tagged imports carry a single timestamp across tens of thousands of tracks. A 37-day job also outlives Last.fm's 14-day window, so client-assigned timestamps expire mid-run regardless. The worker now assigns scrobble timestamps at send time and persists the index-to-timestamp mapping before sending, giving a unique idempotency key we control and narrowing reconciliation to one batch. Also records the PostHog baseline: median user completes 2.1% of their import, reactive rate limits outnumber preventive pauses ~9:1, and the 1-minute cooldown clears the limit in 33 of 2,135 cases. Fixing the client cooldown and thresholds should ship before this feature. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2c093209-9749-4d93-990f-c970f26d570e
…g spec Background mode is now gated on tracks *remaining* rather than selected, and offered at four entry points: new import, resume from saved state, resume from file, and the mid-scrobble pause screen. The pause screen matters most, since telemetry shows that is where users abandon. Adds rules for handing off partial progress atomically, halting the client loop on handoff to avoid double-scrobbling, tolerating old state files, and falling back to the client-side flow. Beta feedback goes to niko@savas.ca via the prefilled-mailto pattern already in ErrorDialog.vue, including the job ID, and must stay reachable on failed jobs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2c093209-9749-4d93-990f-c970f26d570e
Verified against Last.fm's published API docs and corrected the design: - track.scrobble batches up to 50 scrobbles per request. Rewrote the capacity section and withdrew the ~24-concurrent-user ceiling, which was computed without batching. - Error 29 is an IP-level limit; the per-user daily cap arrives as ignoredMessage code 5 on an HTTP 200. The client conflates the two, which explains why half of all first rate limits occur at burst_count = 0. Added error 26 as a global-halt condition. - savas.ca/lastwave shares the origin, so Domain=savas.ca gives no isolation. Moved to a __Host- cookie on api.savas.ca plus explicit CSRF tokens, and raised a dedicated origin as an open question. - Specified the auth handoff as a transaction: persist before redirect, single-use nonce bound to the expected username, digest-validated upload, finalize before clearing client state, reap abandoned jobs. - Replaced lease-based locking with fencing tokens, since an expired lease does not stop an in-flight tick from writing. - Withdrew the exactly-once claim in favour of an explicit at-least-once choice, and documented why reconciliation is imperfect. - Reframed timestamp reassignment as history rewriting, with an explicit ordering trade-off and a decision to go recent-first. - Fixed internal contradictions around threshold reuse and retention. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2c093209-9749-4d93-990f-c970f26d570e
Round two found that several round-one fixes were themselves wrong: - Recent-first ordering was backwards. A track played today has 14 days of slack before its timestamp expires; one played 13 days ago has one. Sending recent tracks first spends slack on the tracks needing it least. Now earliest-deadline-first over two queues. - The __Host- cookie plus bearer token did not isolate LastWave. Cookies attach by destination, not initiator, so a same-site fetch from savas.ca carries the cookie anyway; and a token the SPA can read LastWave can also read. Replaced with an explicit trust-boundary decision rather than ineffective crypto. - The handoff never committed the digest server-side that step 4 claimed to verify. Added a preflight step and a compare-and-set state machine covering seven previously unenumerated ambiguous states. - A single cursor cannot represent a 50-entry batch containing accepted, permanently failed, rate-limited and unknown entries. Progress is now per-entry, with the cursor advancing only over a terminal prefix. - Fencing tokens protect D1 but cannot revoke an in-flight Last.fm request; documented as part of the at-least-once window. - An immutable payload does not make behaviour immutable. Jobs now pin an algorithm version for parsing, ordering and timestamps. - The concurrency cap still cited the withdrawn capacity ceiling. It is now blocked on a feasibility spike, which is a design gate. Also added chunk upload limits against compression bombs, admission control for jobs too large to finish within the credential TTL, and a correction to the claim that listening history is not personal data. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2c093209-9749-4d93-990f-c970f26d570e
Last.fm keys a scrobble on (user, artist, track, timestamp) and silently discards any repeat of that tuple, still reporting accepted=1 ignored=0. Verified experimentally against the live API; it is undocumented and undetectable from the response. reTagOldListens assigned the *same* Date object to every play moved into the 14-day window, so every repeat listen of a track collapsed into one scrobble. That path runs on ~78% of imports (300/386 parses, 80/119 people over 60 days). Timestamps for re-tagged plays are now allocated at send time from a cursor rather than baked in at parse time, which also fixes queues saved today and resumed weeks later being rejected wholesale with ignore code 3. The cursor persists as a high-water mark so a resumed run cannot reuse seconds an earlier run already sent, and is allocated once per track so a retry re-sends an identical tuple instead of creating a phantom play. scrobblePlay now parses the response: a 200 did not mean the play was stored, so ignored scrobbles were being counted as successes. Also in this change: - Client-side rate limiting reworked (RateLimitTracker): rolling window of send timestamps with an adaptive limit learned from observed 429s, escalating backoff, and an honest give-up-and-save instead of an infinite 1-minute retry. Across 2,135 observed rate limits the old behaviour produced 33 recoveries, median 130 minutes to recover. - Fixed a 2-second delayed emit in AuthenticateStep that silently undid a resume the user had already started. - Resume telemetry: original_total_tracks, total_succeeded, completion_pct, is_resumed and previously_scrobbled, so completion can be measured against the original import rather than the shrinking remaining chunk. - Legacy saved sessions with collapsed timestamps are migrated on load for users who are mid-import. - filterDuplicates skips re-tagged listens, whose provisional timestamps would otherwise be matched against real history. Known unfixed, pre-existing: failedIndices is always written empty, so failed tracks are never retried on resume. total_succeeded remains an upper bound, since re-scrobbling history a user already has is indistinguishable from a fresh play. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f0df23dc-ed73-4b56-9b7c-4d2846ddf2a2
This was referenced Jul 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
Last.fm keys a scrobble on (user, artist, track, timestamp) and silently discards any repeat of that tuple. It still returns
accepted=1, ignored=0, so the loss is undetectable from the response. This is undocumented — I verified it experimentally against the live API (three scrobbles, two distinct timestamps, two plays landed).reTagOldListensassigned the sameDateobject to every play moved into the 14-day window. So if you listened to a song five times, four of those were destroyed.That path runs whenever "Scrobble tracks older than 2 weeks" is ticked: 300 of 386 parses, 80 of 119 people over the last 60 days.
Separately,
scrobblePlaydiscarded the API response, so scrobbles Last.fm explicitly did report as ignored were counted as successes.The fix
Timestamps for re-tagged plays are allocated at send time from a cursor, not baked in at parse time:
Send-time allocation rather than parse-time matters for two reasons beyond uniqueness:
The cursor persists as a high-water mark (
lastReTagTimestampSec) so a resumed run can't reuse seconds an earlier run already sent, and is allocated once per track, not once per attempt — if the original request reached Last.fm and only the response was lost, an identical resend is deduplicated away, whereas a fresh second would become a phantom play in the user's library.scrobblePlaynow returns aScrobbleResult. Parsing defaults to accepted on an unrecognised shape, so a surprise response can never invent failures.Also in this PR
RateLimitTracker): rolling window of send timestamps with a limit learned adaptively from observed 429s, escalating backoff (5/15/30 min), then an honest give-up-and-save. The old flat 1-minute-forever retry produced 33 recoveries across 2,135 observed rate limits, median 130 minutes — two users sat through 200+ consecutive retries watching a countdown that was never going to work.AuthenticateStepundid a resume the user had already started.original_total_tracks,total_succeeded,completion_pct,is_resumed,previously_scrobbled. Completion was previously measured against the remaining chunk, which shrinks on resume — real cases went 81,313 -> 573 and 9,924 -> 40, so a barely-started import looked nearly finished.filterDuplicatesskips re-tagged listens, whose provisional timestamps would otherwise be matched against real listening history and silently drop import rows.Testing
36/36 Playwright tests pass, including four new ones covering distinct timestamps, ignored-scrobble accounting, and retry idempotency. Lint 0 errors, build clean.
Known unfixed (pre-existing)
failedIndicesis always written empty, so failed tracks are never retried on resume.total_succeededis an upper bound: re-scrobbling history a user already has is indistinguishable from a fresh play.