Conversation
The app fetched the entire +/-24h EPG on every boot, on every page, and then re-fetched it every 5 minutes for the lifetime of the tab. On a large provider that is a ~70 MB uncompressed, ~10 second query per fetch, and nothing on the sidebar needed more than the currently-airing programme title. Measured on a real 111K-channel / 461K-programme catalogue: boot EPG payload 9781 ms / 69.9 MB -> 7 ms / 0.4 MB (1388x, 171x) full boot path >10 s / ~96 MB -> 0.7 s / 1.7 MB gzipped Changes: - Add compression middleware. The 26 MB channel list ships as 1.5 MB. - Add GET /api/proxy/epg/:sourceId/now returning one row per channel, and load that at startup instead of the full guide. The full guide is now fetched lazily by GuidePage.show(), which already had the branch for it. - Extend idx_epg_source_start to (source_id, start_time, end_time, channel_id, title) so the now-playing query is served by a COVERING index - including description forces a per-row table lookup and costs ~125x. The (source_id, start_time) prefix also fixes the full guide query, which had been falling back to idx_epg_cleanup and scanning nearly the whole table. - Bound the now-playing query below on start_time; unbounded, SQLite walks every programme the source has ever had. 48h keeps multi-day placeholder entries that real providers emit. - Scope the 5-minute background refresh to now-playing only, skip it for hidden tabs, and catch up on visibilitychange. - Drop the unused `data` column from the guide SELECT and send epoch ms instead of ISO strings. Coerce in showProgramDetails, where the values round-trip through dataset as strings. - Index programmes by channel id. getCurrentProgram() linear-scanned all programmes once per rendered channel row. - ChannelList: replace categories.find() inside streams.map() with a Map, and fetch categories/streams and all sources concurrently. - Delete the shadowed duplicate GET /epg/:sourceId and DELETE /cache/:sourceId. Express is first-match-wins so neither had ever run, which is why ?refresh=1 and ?maxAge=N silently did nothing. - Reject non-numeric source ids in cache.clearSource() and assert path containment. The id reached fs.rmSync() URL-decoded and unvalidated, so DELETE /api/proxy/cache/..%2f..%2f.. escaped the cache directory.
This was referenced Jul 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Opening the app on a large Xtream provider took over ten seconds. The cause is that
app.jsfetched the entire ±24h EPG at every boot, on every page, and then re-fetched it every 5 minutes for the lifetime of the tab —EpgGuide.startBackgroundRefresh()had no visibility check and no page scoping. Nothing in the channel sidebar needed more than the currently-airing programme title.Measured on a real provider with 111K live channels, 354K movies and 461K EPG programmes:
Changes
Serve the sidebar what it actually needs. New
GET /api/proxy/epg/:sourceId/nowreturns one row per channel. Startup loads that; the full guide is now fetched lazily byGuidePage.show(), which already had the branch for it.Index.
idx_epg_source_startbecomes(source_id, start_time, end_time, channel_id, title). The extra columns make it a covering index for the now-playing query — includingdescriptionforces a per-row table lookup and costs ~125x (10.4 s → 83 ms in isolation). The(source_id, start_time)prefix also fixes the full-guide query, which had been falling back toidx_epg_cleanupand scanning nearly the whole table.The now-playing query is bounded below on
start_time; unbounded, SQLite walks every programme the source has ever had (~1400x slower measured). The 48h bound is deliberate — at 24h it dropped a real 103-hour placeholder entry that providers do emit.gzip.
compressionmiddleware; the channel list ships as 1.5 MB instead of 26 MB.Remove two shadowed routes. A second
GET /epg/:sourceIdandDELETE /cache/:sourceIdwere defined later inproxy.js. Express is first-match-wins, so neither had ever run — which is why?refresh=1and?maxAge=Nsilently did nothing. Callers updated to stop sending them.Smaller fixes.
getCurrentProgram()linear-scanned all programmes once per rendered channel row (now indexed by channel id).ChannelListhadcategories.find()insidestreams.map()(now aMap) and loaded sources strictly serially (now concurrent). The guide SELECT no longer pulls the unuseddatacolumn and sends epoch ms rather than ISO strings.Path traversal in the cache clear.
sourceIdreachedfs.rmSync(dir, {recursive:true})URL-decoded and unvalidated, soDELETE /api/proxy/cache/..%2f..%2f..escaped the cache directory. Now rejects non-numeric ids and asserts path containment. Unauthenticated before this change.Compatibility
No API removals —
/epg/:sourceIdkeeps its shape and itsdescriptionfield. The one client-side subtlety:start/stopare now epoch ms, and they round-trip throughdatasetas strings in the guide grid, soshowProgramDetailscoerces withNumber()(new Date("1784908800000")is an Invalid Date).Testing
There is no test suite in the repo, so this was verified manually against a real large provider: query plans confirmed via
EXPLAIN QUERY PLAN, payload sizes and timings measured over HTTP, and the sidebar lookup chain (stream.epg_channel_id→channelMap→programme.channelId) checked end-to-end — 6,127 channels resolve to a now-playing title. Fresh-database path confirmed: the index is created correctly on first boot.The full guide is still ~7.8 s / 22 MB when the Guide page is opened. That is unchanged by this PR beyond no longer running at startup; paginating it by day/category is the follow-up.