Skip to content

perf: cut Live TV load time from ~10s to <1s on large providers - #145

Open
Hujairi wants to merge 1 commit into
technomancer702:mainfrom
Hujairi:main
Open

Hujairi wants to merge 1 commit into
technomancer702:mainfrom
Hujairi:main

Conversation

@Hujairi

@Hujairi Hujairi commented Jul 25, 2026

Copy link
Copy Markdown

Problem

Opening the app on a large Xtream provider took over ten seconds. The cause is that app.js fetched the entire ±24h EPG at every boot, on every page, and then re-fetched it every 5 minutes for the lifetime of the tab — EpgGuide.startBackgroundRefresh() had no visibility check and no page scoping. Nothing in the channel sidebar needed more than the currently-airing programme title.

Measured on a real provider with 111K live channels, 354K movies and 461K EPG programmes:

before after
Boot EPG query 9,781 ms / 69.9 MB 7 ms / 0.4 MB
Full boot path >10 s / ~96 MB 0.7 s / 1.7 MB gzipped
Channel list payload 26 MB 1.54 MB gzipped

Changes

Serve the sidebar what it actually needs. New GET /api/proxy/epg/:sourceId/now returns one row per channel. Startup loads that; the full guide is now fetched lazily by GuidePage.show(), which already had the branch for it.

Index. idx_epg_source_start becomes (source_id, start_time, end_time, channel_id, title). The extra columns make it a covering index for the now-playing query — including description forces a per-row table lookup and costs ~125x (10.4 s → 83 ms in isolation). The (source_id, start_time) prefix also fixes the full-guide query, which had been falling back to idx_epg_cleanup and scanning nearly the whole table.

The now-playing query is bounded below on start_time; unbounded, SQLite walks every programme the source has ever had (~1400x slower measured). The 48h bound is deliberate — at 24h it dropped a real 103-hour placeholder entry that providers do emit.

gzip. compression middleware; the channel list ships as 1.5 MB instead of 26 MB.

Remove two shadowed routes. A second GET /epg/:sourceId and DELETE /cache/:sourceId were defined later in proxy.js. Express is first-match-wins, so neither had ever run — which is why ?refresh=1 and ?maxAge=N silently did nothing. Callers updated to stop sending them.

Smaller fixes. getCurrentProgram() linear-scanned all programmes once per rendered channel row (now indexed by channel id). ChannelList had categories.find() inside streams.map() (now a Map) and loaded sources strictly serially (now concurrent). The guide SELECT no longer pulls the unused data column and sends epoch ms rather than ISO strings.

Path traversal in the cache clear. sourceId reached fs.rmSync(dir, {recursive:true}) URL-decoded and unvalidated, so DELETE /api/proxy/cache/..%2f..%2f.. escaped the cache directory. Now rejects non-numeric ids and asserts path containment. Unauthenticated before this change.

Compatibility

No API removals — /epg/:sourceId keeps its shape and its description field. The one client-side subtlety: start/stop are now epoch ms, and they round-trip through dataset as strings in the guide grid, so showProgramDetails coerces with Number() (new Date("1784908800000") is an Invalid Date).

Testing

There is no test suite in the repo, so this was verified manually against a real large provider: query plans confirmed via EXPLAIN QUERY PLAN, payload sizes and timings measured over HTTP, and the sidebar lookup chain (stream.epg_channel_idchannelMapprogramme.channelId) checked end-to-end — 6,127 channels resolve to a now-playing title. Fresh-database path confirmed: the index is created correctly on first boot.

The full guide is still ~7.8 s / 22 MB when the Guide page is opened. That is unchanged by this PR beyond no longer running at startup; paginating it by day/category is the follow-up.

The app fetched the entire +/-24h EPG on every boot, on every page, and then
re-fetched it every 5 minutes for the lifetime of the tab. On a large provider
that is a ~70 MB uncompressed, ~10 second query per fetch, and nothing on the
sidebar needed more than the currently-airing programme title.

Measured on a real 111K-channel / 461K-programme catalogue:

  boot EPG payload   9781 ms / 69.9 MB  ->  7 ms / 0.4 MB   (1388x, 171x)
  full boot path     >10 s / ~96 MB     ->  0.7 s / 1.7 MB gzipped

Changes:

- Add compression middleware. The 26 MB channel list ships as 1.5 MB.
- Add GET /api/proxy/epg/:sourceId/now returning one row per channel, and
  load that at startup instead of the full guide. The full guide is now
  fetched lazily by GuidePage.show(), which already had the branch for it.
- Extend idx_epg_source_start to (source_id, start_time, end_time,
  channel_id, title) so the now-playing query is served by a COVERING index -
  including description forces a per-row table lookup and costs ~125x. The
  (source_id, start_time) prefix also fixes the full guide query, which had
  been falling back to idx_epg_cleanup and scanning nearly the whole table.
- Bound the now-playing query below on start_time; unbounded, SQLite walks
  every programme the source has ever had. 48h keeps multi-day placeholder
  entries that real providers emit.
- Scope the 5-minute background refresh to now-playing only, skip it for
  hidden tabs, and catch up on visibilitychange.
- Drop the unused `data` column from the guide SELECT and send epoch ms
  instead of ISO strings. Coerce in showProgramDetails, where the values
  round-trip through dataset as strings.
- Index programmes by channel id. getCurrentProgram() linear-scanned all
  programmes once per rendered channel row.
- ChannelList: replace categories.find() inside streams.map() with a Map,
  and fetch categories/streams and all sources concurrently.
- Delete the shadowed duplicate GET /epg/:sourceId and DELETE
  /cache/:sourceId. Express is first-match-wins so neither had ever run,
  which is why ?refresh=1 and ?maxAge=N silently did nothing.
- Reject non-numeric source ids in cache.clearSource() and assert path
  containment. The id reached fs.rmSync() URL-decoded and unvalidated, so
  DELETE /api/proxy/cache/..%2f..%2f.. escaped the cache directory.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant