diff --git a/CHANGELOG.md b/CHANGELOG.md index 8775743..172bb22 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,80 @@ # Changelog +## 0.9.4 + +**After upgrading, restart Dispatcharr, then press Test portals and Re-fetch +all.** The restart is what puts every worker on the new code, since plugins are +loaded once per process. Sync on its own would report every portal as unchanged +and fetch nothing — it compares your settings against what was last published, +and nothing in them changed, only the code did. + +- **Stalker compatibility improvements.** +- **Security improvements in the Settings panel, which now masks credentials.** + + + +**Talking to more portals** + +- **A refusal is recognised however the portal words it.** Ministra answers a + rejected session with HTTP 200 and a bare line of text, so only one exact + phrase was ever understood; the others arrived as "portal returned non-JSON + response" and cost a channel its failover. `Access denied.`, `Unauthorized + request.`, the stock server's numeric debug counter and the refusals + non-Ministra panels put inside the JSON envelope are all read now, and each + says which of the three things is actually wrong: the session, the account, + or the MAC. +- **A portal is looked for on every path one is served from.** Two were probed; + five are now — the URL as written first, then `/server/load.php`, + `/c/portal.php`, `/portal.php` and `/stalker_portal/server/load.php`. An + explicit `.php` in your own URL is no longer swapped for a guess, since that + path is the address the provider handed out. +- **The box describes itself the way a real one does.** The handshake and + profile now carry `prehash`, `client_type`, `video_out` and the metrics blob + a MAG sends, which is what some panels authenticate on and what the admin + panel reads to show a box as connected. +- **A command reaches the portal decoded exactly once.** A `%` inside a channel + command was being encoded twice, so those channels asked for a link that + never existed. +- **Channels the portal marks as needing no temporary link now play without one** + — the portal's own player skips `create_link` for them, and so does this. It + removes the one request known to go wrong on those providers. +- **The portal's own streams are fetched as the box that authenticated**, with + the session and MAC that minted the link. Never to anyone else's CDN: a + stream on another host gets no credentials, which is what keeps a + subscriber's MAC out of a request that has no business carrying it. +- Three things the portal was already saying are now read: the expiry date + where Ministra puts it, a device-conflict message with the setting that fixes + it named, and a channel listed twice in one response counted once. + +**Credentials are no longer displayed** + +- **The Portals box hides them once saved.** The MAC, the password and the rest + of the account identity come back as `••••`: the real line lives in + `/data/distalker/portals.txt`, which the settings panel cannot read. Names, + URLs and the tuning keys stay, so the box still reads as your own + configuration — and a portal is still deleted by deleting its line. +- This matters beyond the screen. Dispatcharr serves a plugin's settings row to + every account on the install, so the credentials were in an API response + anyone could read. +- **Editing works through the bullets.** Renaming a portal and repointing its + URL both keep the MAC you cannot see; pasting a line back in full still + works. Change a line's name *and* its URL in one edit and there is nothing + left to recognise it by — the plugin then quotes that line back and asks you + to retype it, rather than parsing a MAC address made of bullets. +- **Keep your own copy of each portal line in a password manager.** The box will + not give a credential back, and if `/data/distalker` is ever lost the last + copy goes with it. Nothing is redacted until that file is known to hold the + same list, so a registry that could not be written leaves the credentials + where they were. +- Hiding is not encryption. The resolver reads the MAC on every tune in a + process with no database, so `portals.txt` and the state mirrors go on + holding it in the clear at `0600` — the README says which files those are. +- **Settings the panel no longer has a field for are dropped.** Removing a field + from the manifest stops the panel rendering it and nothing else, so the + Add-portal form retired in 0.4.0 left a MAC, a password and a portal URL in + that same API response — for a portal you may have deleted months ago. They + are cleared on the first action after upgrading. + ## 0.9.3 **After upgrading, restart Dispatcharr, then press Test portals and Re-fetch diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ab98fa9..ab1e92c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -139,10 +139,10 @@ added or deleted would silently rebind existing channels to the other portal. It runs inside the container with no persistence, so it comes back empty from every restart — which used to kill every channel until someone pressed Sync, -twice observed before it was understood. Everything `save_portal` and -`save_fallback` publish is therefore mirrored to `/data/distalker/state/*.json` -(`0600`: credentials), read only when Redis has nothing, and written back to -Redis on first use. Reads go through `_client_or_none`, so an unreachable Redis +twice observed before it was understood. Everything `save_portal`, +`save_static_cmds` and `save_fallback` publish is therefore mirrored to +`/data/distalker/state/*.json` (`0600`: credentials), read only when Redis has +nothing, and written back to Redis on first use. Reads go through `_client_or_none`, so an unreachable Redis degrades rather than raising. The session token is deliberately *not* mirrored: it expires within the hour, and losing it costs one handshake. @@ -151,6 +151,31 @@ before the first sync, or a lost data volume, and there is again nothing to read. `Plugin._republish` runs on the assign path instead, which the button, every `m3u_refresh` and every `channel_error` all reach. +### Some channels never reach the portal at tune time + +A Stalker row carries `use_http_tmp_link` and `use_load_balancing`, and the +portal's own `player.js` calls `create_link` only when one of them is set — +everything else plays the `cmd` the listing already gave. pvr.stalker does the +same and cites that line for it (`ChannelManager::GetStreamURL`). + +Sync evaluates `plays_without_create_link` per row and publishes the commands +that qualify as a set (`save_static_cmds`); `resolver.resolve` checks it before +doing anything else, and on a hit returns the command's own URL with no +handshake and no `create_link`. On nearly every portal the set is empty — a +stock row is a loopback marker, which only the portal can resolve whatever its +flags say. The family it exists for is the one `undoubled_link` was written +for: providers that answer the listing with a resolved link and then mangle it +when handed it back. + +Four guards narrow it, and every one of them falls back to asking the portal, +so none can break an install that works today. Absent flags mean *no evidence* +rather than "no". The scheme must be one ffmpeg opens (a portal's own +`ffrt4://` pseudo-URL parses like an address and plays as nothing). The host +must not be loopback in any of its spellings. And — this one is ours, not the +reference behaviour — the command must carry **no query string**: a link that +expires keeps its token there, and by the time a stream fails the resolver has +already become ffmpeg and Dispatcharr has spent this channel's failover. + ### The ffmpeg defaults carry no `-reconnect`, and that is load-bearing ffmpeg's own reconnection retries the URL it was handed, which for Stalker is a @@ -259,6 +284,44 @@ Consequences that shape the code: (`Plugin._report`). Anything more urgent goes to Dispatcharr's notification centre, which does reach an open browser (`sync.announce`). +### The panel is shown a redacted list + +The settings row is served to every account on the install and painted straight +into a textarea, so it is the one copy of the portal list that must not hold a +credential. `Plugin._save_settings` therefore writes two different things: the +whole list to `portals.txt`, and `stalker_api.mask_portals()` of it to the row. +The MAC, `username`, `password`, `device_id`, `device_id2`, `serial` and +`signature` become `••••`; names, URLs and the tuning keys stay, because a box +of nothing but bullets is one nobody can recognise their own portals in. + +Redaction is a property of the **write path and nothing else**. +`_reconcile_registry` runs `unmask_portals()` on whatever the panel sent and +always returns the real list, so every migration, action and sync downstream +goes on reading plain lines and never has to know. + +Four things are easy to break here: + +- **Never mirror redacted text.** `_save_settings` checks `is_masked()` before + writing `portals.txt`, because `_failed()` can be reached with settings that + never passed through `_reconcile_registry` — and mirroring a row of tokens + would write them over the only copy of the credentials. +- **Never redact before the file holds the list.** The guard is + `digest(stored) == digest(text)`. A registry that could not be written leaves + the row as the only copy there is, and hiding the only copy loses it. +- **The token stands where the MAC stands.** `split_portal_line` decides which + field is which by *where the MAC sits*, so `_mac_index()` counts the token as + one; without that, redacting an unnamed line's MAC shifts its fields by one. + For the same reason `mask_portals` writes the derived name out. +- **A redacted line is paired back up by slug, then by URL.** Renaming a portal + and repointing it are both ordinary edits, and each changes the half the other + is recognised by. Change both at once and nothing matches: the token survives + `unmask_portals`, and `Plugin._portals` quotes the line back and asks for it + to be retyped rather than parsing a MAC address made of bullets. + +None of this is encryption at rest, and the README says so: the resolver reads +the MAC on every tune with no Django, so `portals.txt` and the state mirrors go +on holding it in the clear at `0600`. + ### Actions `plugin.json` is the single definition of the UI; `plugin.py` reads it at import diff --git a/README.md b/README.md index 076b5b9..5d9eec6 100644 --- a/README.md +++ b/README.md @@ -59,6 +59,7 @@ A plugin that writes credentials to disk should say so plainly: | --- | --- | | `/data/uploads/m3us/distalker-.m3u` | The generated playlist | | `/data/distalker/portals.txt` | Your portal list verbatim, **credentials included** | +| Dispatcharr's plugin settings row | The same list with every credential **redacted** — this is the copy the panel renders and the API serves | | `/data/distalker/state/*.json` | What the resolver reads at tune time, **credentials included**, `0600` | | Redis `distalker:*` | The same, plus the session token | @@ -156,13 +157,14 @@ URL: | `http://host:8080/c/` | `http://host:8080/c/portal.php` | | `host:8080/c/` | `http://host:8080/c/portal.php` | | `http://host` | `http://host/portal.php` | -| `http://host/…/load.php` | unchanged — explicit endpoints are preserved | -| `http://host/c/other.php` | `http://host/c/portal.php` | +| `http://host/…/load.php` | unchanged — any explicit `.php` is preserved | +| `http://host/cp/api.php` | unchanged — a panel's own path is an address, not a typo | If that path turns out not to be where the portal answers, Distalker tries the -other one Ministra uses — `…/c/portal.php` and `…/server/load.php` are swapped -for each other — and logs which one worked. Putting the working one on the -portal line saves a failed request on every sync. +others in turn — `…/server/load.php`, `…/c/portal.php`, `…/portal.php` and +`…/stalker_portal/server/load.php`, built from the same install root — and logs +which one worked. Putting the working one on the portal line saves a failed +request on every sync. Anything unusual goes in trailing `key=value` pairs, separated by spaces or further `|` characters, quoted where a value contains spaces @@ -182,9 +184,16 @@ further `|` characters, quoted where a value contains spaces > Raise it only on what your provider told you: exceeding it is the quickest > route to a blocked MAC. -> **Credentials are visible in this box**, and stored unencrypted in the -> Dispatcharr database like every plugin setting. Treat your backups -> accordingly. +> **Credentials are hidden once saved.** The MAC, the password and the rest of +> the account identity come back as `••••` the moment the list is stored: the +> real line lives in `/data/distalker/portals.txt`, which the settings panel +> cannot read. Edit around the bullets and what you leave alone is left alone — +> renaming a portal or repointing its URL both work with the MAC still hidden. +> +> **Keep your own copy of each portal line in a password manager.** The box will +> not give a credential back, and if `/data/distalker` is ever lost the only +> remaining copy goes with it. Hidden is also not encrypted — see +> [What it writes, and where](#what-it-writes-and-where). ### STB identity @@ -391,8 +400,10 @@ deleting it. a stock Dispatcharr can run, so *Refresh every (hours)* drives the M3U accounts' own refresh interval and answers the event that follows. It works, and it is why the interval cannot usefully go below an hour. -- **Credentials are stored unencrypted**, in the Dispatcharr database and on - disk — see [What it writes, and where](#what-it-writes-and-where). +- **Credentials are stored unencrypted** on disk. The settings panel no longer + shows them, but `/data/distalker/portals.txt` and the state mirrors hold them + in the clear at `0600`: the resolver reads them on every tune, in a process + with no database — see [What it writes, and where](#what-it-writes-and-where). - **No session keep-alive.** A cached token is reused and re-issued on demand. Portals that drop idle sessions are untested. - **One `ffmpeg` per tuned channel**, which is normal for any non-proxy stream diff --git a/plugin.json b/plugin.json index 27325d0..4e87d10 100644 --- a/plugin.json +++ b/plugin.json @@ -1,6 +1,6 @@ { "name": "Distalker", - "version": "0.9.3", + "version": "0.9.4", "description": "Stalker/MAG portal support for Dispatcharr. Syncs portal channels into a native M3U account and resolves short-lived stream links at tune time -- no extra containers, no extra ports.", "author": "PiloUnk", "license": "AGPL-3.0-only", @@ -19,7 +19,7 @@ "type": "text", "default": "", "placeholder": "http://portal.example.com:8080/c/ | 00:1A:79:AA:BB:CC", - "help_text": "Portal URL | MAC address, one line each. The name is taken from the host, so put one in front only if you want a different label -- or if two portals share a host, which the sync will then ask you to do. Trailing key=value pairs cover the rest: username, password, max_streams, model, serial, device_id, device_id2, timezone, and epg=1 to fetch this portal's programme guide (epg_hours=48 for more than a day -- a guide is by far the largest thing a sync downloads, which is why it is off unless asked for). A line starting with '#' is ignored, which is how you suspend a portal without losing its channels. Credentials are stored unencrypted and are visible in this box." + "help_text": "Portal URL | MAC address, one line each. The name is taken from the host, so put one in front only if you want a different label -- or if two portals share a host, which the sync will then ask you to do. Trailing key=value pairs cover the rest: username, password, max_streams, model, serial, device_id, device_id2, timezone, and epg=1 to fetch this portal's programme guide (epg_hours=48 for more than a day -- a guide is by far the largest thing a sync downloads, which is why it is off unless asked for). A line starting with '#' is ignored, which is how you suspend a portal without losing its channels. Once saved, the MAC, the password and the rest of the account identity are replaced by bullets here and kept in a file only this plugin reads; edit around the bullets and what you leave alone stays as it was. That hides them from this page, it does not encrypt them -- and the box will not give one back, so keep your own copy of each line in a password manager." }, { "id": "status", diff --git a/plugin.py b/plugin.py index e0e430a..5d27a30 100644 --- a/plugin.py +++ b/plugin.py @@ -42,13 +42,16 @@ claim_auto_sync, forget_portal, format_portal_line, + is_masked, is_superseded_ffmpeg_args, load_portal, + mask_portals, parse_portals, published_slugs, save_portal, split_portal_line, sync_lock_age, + unmask_portals, ) from .sync import ( @@ -128,6 +131,7 @@ def run(self, action: str, params: Dict[str, Any], context: Dict[str, Any]) -> D settings = self._reconcile_registry(settings, logger) settings = self._migrate_legacy_globals(settings, logger) settings = self._migrate_ffmpeg_args(settings, logger) + settings = self._drop_unknown_settings(settings, logger) self._drop_legacy_schedule(logger) result = handler(params or {}, settings, logger) @@ -293,6 +297,41 @@ def _migrate_ffmpeg_args(self, settings: Dict[str, Any], logger) -> Dict[str, An self._save_settings(updated) return updated + def _drop_unknown_settings(self, settings: Dict[str, Any], logger) -> Dict[str, Any]: + """Delete stored settings no field in the manifest declares any more. + + Taking a field out of plugin.json only stops the panel *rendering* it. + The value stays in PluginConfig.settings, which Dispatcharr serves to + every account on the install -- so the Add-portal form that went in + 0.4.0 left a MAC, a password and a portal URL sitting in the API + response for months, belonging to a portal the user had since dropped. + Redacting the portal list and leaving those behind would have been + half a job. + + The manifest is the single definition of the panel, so anything it does + not declare is dead by construction; test_manifest.py pins that every + setting the code reads is declared. Runs after the migrations, which + read keys of their own that this would otherwise take first. + + The open panel goes on replaying the stale keys until the page is + reloaded -- it PUTs the state it fetched, same as with the portal list + -- so this runs on every action rather than once. + """ + known = {field["id"] for field in self.fields} + stale = sorted(key for key in self._raw_settings() if key not in known) + if not stale: + return settings + + updated = {k: v for k, v in settings.items() if k in known} + logger.info( + "distalker: dropped %d stored setting(s) the panel no longer has a " + "field for: %s", + len(stale), + ", ".join(stale), + ) + self._save_settings(updated) + return updated + def _settings_with_defaults(self) -> Dict[str, Any]: """Stored settings over the manifest's defaults. @@ -329,18 +368,42 @@ def _write_settings(self, settings: Dict[str, Any]) -> None: cfg.save(update_fields=["settings", "updated_at"]) def _save_settings(self, settings: Dict[str, Any]) -> None: - """Persist settings the plugin changed itself.""" - self._write_settings(settings) + """Persist settings the plugin changed itself. + + The portal list is stored twice and deliberately not the same way. The + registry file takes it whole, because the next action and the resolver + need the credentials; the settings row takes the redacted rendering, + because Dispatcharr serves that row to every account on the install and + the panel paints it straight into a textarea. + ``settings`` always carries the real list here -- redaction happens on + the way out and nowhere else, so every caller in between goes on + reading and rewriting plain lines. + """ # Mirror the list somewhere the settings panel cannot reach, and flag it # as a write the open panel has no way of knowing about. Saves that # leave the list alone -- recording a status, clearing the form -- skip # this entirely: rewriting the same text would renew the marker and go # on distrusting a panel that is in fact still in step with the list. + # + # Redacted text is never mirrored. _failed() can be reached with + # settings that never went through _reconcile_registry -- an error + # raised inside it, for one -- and mirroring a row full of tokens would + # write the tokens over the only copy of the credentials. text = settings.get("portals") or "" - if digest(text) != digest(load_registry()): + if not is_masked(text) and digest(text) != digest(load_registry()): save_registry(text, pending=True) + stored = load_registry() + row = dict(settings) + # Redact only once the file is known to hold the same list. A registry + # that could not be written leaves this row as the only copy there is, + # and hiding the only copy is how a configuration gets lost. + if stored is not None and digest(stored) == digest(text): + row["portals"] = mask_portals(text) + + self._write_settings(row) + @staticmethod def _worth_recording(params: Dict[str, Any], result: Dict[str, Any]) -> bool: """Whether this run should overwrite the Last action box. @@ -397,11 +460,21 @@ def _reconcile_registry(self, settings: Dict[str, Any], logger) -> Dict[str, Any what the plugin wrote, so the file wins. Once the panel quotes the current list back -- which happens as soon as it is reopened -- the marker clears and the textarea is authoritative again. + + Returns settings carrying the *real* list in every case, whatever the + row and the panel hold. Everything downstream -- the migrations, the + actions, the sync -- reads plain lines and never has to know that the + stored copy is redacted. """ stored = load_registry() raw = self._raw_settings() panel_value = raw.get("portals") + def adopt(text: str) -> Dict[str, Any]: + updated = dict(settings) + updated["portals"] = text or "" + return updated + if panel_value is None: if stored: logger.info( @@ -409,14 +482,36 @@ def _reconcile_registry(self, settings: Dict[str, Any], logger) -> Dict[str, Any "restoring it from %s", REGISTRY_PATH, ) - settings = dict(settings) - settings["portals"] = stored + settings = adopt(stored) self._save_settings(settings) return settings - if (panel_value or "").strip() == (stored or "").strip(): + # The panel can only send back what it was shown, and what it was shown + # is redacted. Undo that before anything compares or parses it. + plain = unmask_portals(panel_value, stored or "") + + # "Unchanged" is judged against whatever the panel was actually holding. + # A redacted list has to be compared with the redaction: a line that + # never named its portal comes back with the derived name written out, + # and that is the rendering's doing rather than an edit. Comparing the + # plain text instead would read every reopened panel as a hand edit and + # rewrite the file for nothing. The redaction is *not* used to judge a + # list typed in full: it hides the MAC, so a hand-corrected MAC would + # compare equal to the one it replaced. + if is_masked(panel_value): + unchanged = panel_value.strip() == mask_portals(stored or "").strip() + else: + unchanged = plain.strip() == (stored or "").strip() + + if unchanged: # The panel has caught up, so whatever it sends next can be trusted. clear_pending() + settings = adopt(stored) + # An install upgrading into this version still has its credentials + # in the settings row, and a panel that never edits the box would + # never trigger a save. This is where those rows get redacted, once. + if panel_value != mask_portals(stored or ""): + self._save_settings(settings) return settings if is_pending(): @@ -426,19 +521,36 @@ def _reconcile_registry(self, settings: Dict[str, Any], logger) -> Dict[str, Any "panel to edit the list by hand.", REGISTRY_PATH, ) - settings = dict(settings) - settings["portals"] = stored or "" + settings = adopt(stored) self._save_settings(settings) return settings + # A token that survived unmask_portals() belongs to a line nothing in + # the registry answers for -- a lost /data/distalker, or a line renamed + # *and* moved in one edit. Adopting it would write bullets over the + # only copy of the credentials, so the list is left exactly as it is + # and _portals() names the offending lines and refuses. + if is_masked(plain): + logger.error( + "distalker: the settings panel sent back portal lines whose " + "credentials are not in %s; leaving the stored list alone", + REGISTRY_PATH, + ) + return adopt(plain) + # Hand-edited in the textarea, or the very first run. - if not save_registry(panel_value): + settings = adopt(plain) + if not save_registry(plain): logger.warning( "distalker: could not write %s; the portal list is only " "stored in the plugin settings and may be lost", REGISTRY_PATH, ) + # Left in the row as it came, credentials and all: with no file to + # read them back from, redacting them here would erase them. + return settings + self._save_settings(settings) return settings # -- actions ---------------------------------------------------------- @@ -449,7 +561,25 @@ def _portals(self, settings: Dict[str, Any]) -> List[PortalConfig]: Partially applying a mistyped config is worse than doing nothing: it would leave half the accounts pointing at stale files. """ - portals, errors = parse_portals(settings.get("portals") or "") + text = settings.get("portals") or "" + + # A line still holding a redaction token got here without its + # credentials, and _reconcile_registry could not find them: either the + # registry is gone, or the line was renamed *and* moved in one edit, + # which leaves nothing to recognise it by. Parsing on would report a + # MAC address made of bullets, which explains nothing. + if is_masked(text): + lines = [ + line.strip() for line in text.splitlines() if is_masked(line) + ] + raise PortalError( + "these portal lines are still hidden and their credentials " + f"could not be read back from {REGISTRY_PATH}: " + + "; ".join(lines) + + " -- retype each of them in full, URL, MAC and password" + ) + + portals, errors = parse_portals(text) if errors: raise PortalError("invalid portal configuration -- " + "; ".join(errors)) if not portals: @@ -747,6 +877,7 @@ def run_sync_now(self, full: bool = False, logger=None) -> Dict[str, Any]: settings = self._reconcile_registry(settings, logger) settings = self._migrate_legacy_globals(settings, logger) settings = self._migrate_ffmpeg_args(settings, logger) + settings = self._drop_unknown_settings(settings, logger) try: result = self._sync_portals(settings, logger, full=full) diff --git a/resolver.py b/resolver.py index e10407b..f5d0c05 100644 --- a/resolver.py +++ b/resolver.py @@ -46,8 +46,13 @@ def log(message: str) -> None: print(f"[distalker] {message}", file=sys.stderr, flush=True) -def resolve(slug: str, cmd: str) -> tuple[str, stalker_api.PortalConfig]: - """Return a playable URL for ``cmd``, refreshing the session if needed.""" +def resolve(slug: str, cmd: str) -> tuple[str, stalker_api.PortalConfig, str]: + """Return a playable URL for ``cmd``, refreshing the session if needed. + + The token comes back with it: the stream is fetched with the session when + it is served by the portal itself, and only the session that minted the + link is the one it will accept. + """ try: client = stalker_api.get_redis() except Exception as exc: @@ -65,6 +70,26 @@ def resolve(slug: str, cmd: str) -> tuple[str, stalker_api.PortalConfig]: ) cached = stalker_api.get_cached_token(slug, client) + + # A channel the portal itself marked as needing no temporary link is played + # from the command the listing gave, with no request to the portal at all. + # That is what the portal's own player does, what pvr.stalker does, and + # here it also skips the one request that is known to go wrong: the + # providers undoubled_link exists for answer create_link by gluing their + # base in front of a command that was already a link, and the reply is + # thrown away again a moment later. + # + # The token is whatever was already cached -- it may be nothing, and that + # is not worth a handshake to fix. It only ever feeds a header, and a + # command with no query string is not a link the portal minted for a + # session in the first place. + if cmd in stalker_api.load_static_cmds(slug, client): + link = stalker_api.extract_link(cmd) + if link: + log(f"{slug}: the portal marks this channel as needing no " + "temporary link; playing its command as it stands") + return link, cfg, cached or "" + portal = stalker_api.Portal(cfg, token=cached or "") if cached: @@ -83,7 +108,7 @@ def resolve(slug: str, cmd: str) -> tuple[str, stalker_api.PortalConfig]: else: for warning in portal.warnings: log(f"{slug}: {warning}") - return link, cfg + return link, cfg, portal.token portal.login() # Cached before the link is asked for, not after: the token is good either @@ -95,15 +120,21 @@ def resolve(slug: str, cmd: str) -> tuple[str, stalker_api.PortalConfig]: # answer is reported too, and not only what login() found. for warning in portal.warnings: log(f"{slug}: {warning}") - return link, cfg + return link, cfg, portal.token -def build_ffmpeg_command(cfg: stalker_api.PortalConfig, url: str) -> list: +def build_ffmpeg_command( + cfg: stalker_api.PortalConfig, url: str, token: str = "" +) -> list: """Expand the portal's ffmpeg template into an argv list. Referer and Origin are derived from the **stream** URL, not the portal. Providers routinely serve the stream from a different host or port than the portal API, and expect the request to look like it came from there. + + The session goes with them when the stream is the portal's own -- see + :func:`stalker_api.stream_credential_safe`, which is what keeps a + subscriber's MAC out of a request to somebody else's CDN. """ import shlex from urllib.parse import urlparse @@ -111,8 +142,7 @@ def build_ffmpeg_command(cfg: stalker_api.PortalConfig, url: str) -> list: parsed = urlparse(url) origin = f"{parsed.scheme}://{parsed.netloc}" - headers = dict(stalker_api.stream_headers(cfg.model)) - headers["Origin"] = origin + headers = stalker_api.stream_headers(cfg, url, token) header_blob = "".join(f"{k}: {v}\r\n" for k, v in headers.items()) template = cfg.ffmpeg_args or stalker_api.DEFAULT_FFMPEG_ARGS @@ -229,7 +259,7 @@ def probe(pseudo_url: str) -> int: return 2 try: - link, cfg = resolve(slug, cmd) + link, cfg, token = resolve(slug, cmd) except Exception as exc: log(f"resolve failed: {exc}") return 1 @@ -242,10 +272,11 @@ def probe(pseudo_url: str) -> int: parsed = urlparse(link) origin = f"{parsed.scheme}://{parsed.netloc}" - headers = dict(stalker_api.stream_headers(cfg.model)) + headers = stalker_api.stream_headers(cfg, link, token) headers["User-Agent"] = stalker_api.USER_AGENT headers["Referer"] = origin + "/" - headers["Origin"] = origin + print("session : " + ("sent with the stream" + if "Cookie" in headers else "not this host's to send")) try: resp = requests.get(link, headers=headers, stream=True, timeout=20) @@ -293,7 +324,7 @@ def main(argv: list) -> int: return 2 try: - link, cfg = resolve(slug, cmd) + link, cfg, token = resolve(slug, cmd) except PortalError as exc: log(f"{slug}: {exc}") return 1 @@ -303,7 +334,7 @@ def main(argv: list) -> int: log(f"{slug}: resolved -> {link.split('?', 1)[0]}") - command = build_ffmpeg_command(cfg, link) + command = build_ffmpeg_command(cfg, link, token) executable = shutil.which(command[0]) or command[0] try: os.execv(executable, command) diff --git a/stalker_api.py b/stalker_api.py index abaa5b5..a93d4fa 100644 --- a/stalker_api.py +++ b/stalker_api.py @@ -19,6 +19,8 @@ from __future__ import annotations +import hashlib +import ipaddress import json import os import re @@ -66,10 +68,46 @@ STB_HW_VERSION = "1.7-BD-00" STB_NUM_BANKS = 1 -# What a portal answers with, in plain text and with no JSON around it, once -# the token it was given is no longer good. Matched exactly because it is a -# fixed string in Ministra rather than something a reseller writes. -AUTH_FAILED_BODY = "authorization failed." +# The refusals Ministra answers with in plain text, HTTP 200 attached and no +# JSON around them, and what each of them actually means. Matched against the +# whole body rather than searched for inside it: these are bare phrases, so a +# proxy or a WAF answering 'Access denied' -- 38 +# characters, under any length a cap would catch -- would otherwise be read as +# the portal speaking, and send the resolver off to re-authenticate against +# something that never answered at all. +# +# Only the first carries a trailing number, and it is a debug counter rather +# than anything to read. Matching without it is most of the point of this +# table: 'Authorization failed. 75' is what the stock server actually sends, +# and the exact-string comparison this replaces did not recognise it -- so the +# one refusal the resolver exists to recover from arrived typed as an endpoint +# failure, which is never re-authenticated on and costs a second request to the +# other path of a portal that is already refusing us. +AUTH_REFUSALS = ( + ( + re.compile(r"^authorization\s+failed[.!]*(?:\s+\d+)?$", re.I), + "portal says the session is no longer authorised", + ), + ( + re.compile(r"^access\s+denied[.!]*$", re.I), + "portal says this account is denied access", + ), + ( + re.compile(r"^unauthorized\s+request[.!]*$", re.I), + "portal did not receive the MAC address it authorises on", + ), +) + +# Wording accepted inside a JSON envelope's 'error' field, beyond the three +# above. Wider on purpose, and applied in one narrow place: a panel that fills +# in 'error' has said something went wrong deliberately -- a reply that worked +# carries no 'error' at all -- so 'Invalid token' and a bare 'unauthorized' are +# worth reading there, where the same breadth against an arbitrary body would +# match half the error pages on the internet. +ENVELOPE_REFUSAL = re.compile( + r"authorization|access\s+denied|unauthorized|auth\s+failed|invalid\s+token", + re.I, +) # Answers worth asking again for. Everything else is the portal having made up # its mind: a 404 is not going to become a 200, and a 403 is the subject of @@ -185,18 +223,64 @@ def is_superseded_ffmpeg_args(value: str) -> bool: current = " ".join((value or "").split()) return any(current == " ".join(old.split()) for old in SUPERSEDED_FFMPEG_ARGS) +def stream_credential_safe(portal_url: str, link: str) -> bool: + """Whether the stream is the portal's own, and may carry its session. + + The question matters because the answer is a subscriber's credentials. A + create_link URL often points somewhere that is nobody's business but its + own -- a CDN, an operator's edge, another provider entirely -- and it + already carries its own token in the query, so it needs nothing from us. + Sending the MAC and the session token there would hand a third party + everything required to use the subscription. + + Same host, and never a downgrade from https to http. A different *port* is + still the portal: panels routinely serve the stream from :8080 next to the + portal on :80, and those are exactly the ones gated on the mac cookie. + """ + try: + portal, stream = urlparse(portal_url), urlparse(link) + except ValueError: + return False + if stream.scheme not in ("http", "https"): + return False + host = (portal.hostname or "").lower() + if not host or host != (stream.hostname or "").lower(): + return False + return not (portal.scheme == "https" and stream.scheme == "http") + + # Headers a MAG box sends when fetching the stream itself, beyond the # User-Agent and Referer that ffmpeg has dedicated flags for. Providers do # check these: a request that authenticated fine against the portal can still # be refused at the stream if it does not look like the same box. -def stream_headers(model: str = DEFAULT_MODEL) -> Dict[str, str]: - return { - "X-User-Agent": f"Model: {model}; Link: Ethernet", +# +# The session travels with them when the stream is the portal's own. A MAG +# sends the mac cookie and the token to everything it fetches from the portal, +# and panels that gate the stream on that cookie answer a request without it +# with a 403 -- which arrives as a channel that authenticated, resolved, and +# then would not play. What the box would never do is send them anywhere else, +# which is what stream_credential_safe is for. +def stream_headers(cfg: "PortalConfig", link: str, token: str = "") -> Dict[str, str]: + parsed = urlparse(link) + headers = { + "X-User-Agent": f"Model: {cfg.model}; Link: Ethernet", "Accept": "*/*", "Accept-Language": "en-US,en;q=0.9", "Cache-Control": "no-cache", "Pragma": "no-cache", + # Derived from the stream, not the portal: providers serve it from + # another host or port and expect the request to look like it came + # from there. + "Origin": f"{parsed.scheme}://{parsed.netloc}", } + if stream_credential_safe(cfg.url, link): + headers["Cookie"] = ( + f"mac={quote(cfg.mac)}; stb_lang=en; " + f"timezone={quote(cfg.timezone)};" + ) + if token: + headers["Authorization"] = "Bearer " + token + return headers REDIS_PREFIX = "distalker" @@ -217,7 +301,7 @@ class PortalEndpointError(PortalError): A 404, or a body that is not JSON at all. Separated because it is the one failure with a second thing worth trying: the same portal on its other - path -- see :func:`alternate_endpoint`. + path -- see :func:`endpoint_candidates`. """ @@ -232,6 +316,49 @@ class PortalAuthError(PortalError): """ +def auth_refusal(body: Any) -> str: + """What a plain-text answer says about the session, or "" if it says nothing.""" + text = str(body or "").strip() + for pattern, message in AUTH_REFUSALS: + if pattern.match(text): + return message + return "" + + +def envelope_refusal(payload: Any) -> str: + """The same, for portals that refuse inside the envelope rather than instead of it. + + Not Ministra's behaviour, and common in what this plugin actually meets. + Every one of these used to arrive as an ordinary reply: ``get_genres`` + answered ``{'js': {'error': 'Invalid token'}}`` counted as a portal with no + genres, and 'Test portals' reported it as authenticated with 0 groups. + + ``error`` is read with the wide vocabulary. ``msg`` only with the three + exact bodies, and only when nothing else in the reply has already given a + verdict: a reply carrying a ``status`` is one :meth:`Portal.login` reads + for itself, including the sentence beside it -- which is the provider's own + wording, and better than anything this could substitute for it. + """ + js = payload.get("js") if isinstance(payload, dict) else None + # Some panels put the phrase straight in 'js' rather than in a field of it. + if isinstance(js, str): + return auth_refusal(js) + if not isinstance(js, dict): + return "" + + if js.get("status") is None: + exact = auth_refusal(js.get("msg")) + if exact: + return exact + + error = str(js.get("error") or "").strip() + # A field longer than a sentence is a document somebody stuffed in there, + # not a refusal worth quoting back at the user. + if error and len(error) <= 200 and ENVELOPE_REFUSAL.search(error): + return f"portal refused the session: {error}" + return "" + + # --------------------------------------------------------------------------- # Configuration # --------------------------------------------------------------------------- @@ -437,6 +564,140 @@ def stream_id(cmd: str) -> str: return found[0].strip() +# What a real box leaves raw in the command it hands back, and therefore what +# the portal sees before PHP form-decodes the query exactly once. The portal's +# own client concatenates the command into the query string and lets the URL +# layer escape only what a URL cannot carry; anything already percent-escaped +# in the command travels with that escape intact. +# +# quote(cmd, safe="") escaped the percent sign itself, so a command containing +# '%3A' left here as '%253A' and arrived at the portal still encoded -- a +# different string from the one a real box sends, and the stock create_link +# handler matches on it with preg_match. Portals that answer their listing with +# an already-encoded URL are exactly the ones canonical_cmd cannot rewrite, so +# this was worst where it was least recoverable. +# +# Everything outside this set is still percent-encoded. That covers what a URL +# cannot carry -- space, quotes, anything non-ASCII -- and, more to the point, +# the three characters that would restructure the request around it: '&', '#', +# and ';' for a PHP configured with it as an argument separator. A portal +# cannot smuggle extra parameters into our query through a command. +CMD_SAFE = "%/:?=+,@$[]!*()~-_." + + +def encode_cmd(cmd: str) -> str: + """A command as the portal's own client would have put it on the wire.""" + return quote(cmd, safe=CMD_SAFE) + + +# Schemes a stream can actually be handed to ffmpeg on. An allowlist, where +# extract_link deliberately accepts anything with '://': the two answer +# different questions. What a portal *resolves* may be multicast on a scheme +# nobody here has met, and refusing to play it would be worse than not +# recognising it -- but a command the portal never resolved can also be one of +# its own internal pseudo-URLs ('ffrt4://ch/live/1'), which parses like an +# address and plays as nothing. +PLAYABLE_SCHEMES = frozenset( + {"http", "https", "udp", "rtp", "rtsp", "rtmp", "rtmps", "srt", "mms"} +) + + +def _is_portal_local(host: str) -> bool: + """Whether a host can only mean the machine that wrote the address. + + 'ffrt3 http://localhost/ch/1234_' is an instruction to the portal, never an + address a set-top box could open, so a channel carrying one always needs + resolving whatever its flags say. Rather more spellings than the obvious + one: RFC 6761 reserves every name ending in '.localhost' as well, IPv4 + gives the whole of 127.0.0.0/8 to loopback, and a portal that writes its + own address as an IPv4-mapped IPv6 literal has said the same thing again. + + A host that cannot be read at all counts as local, because the question + this answers is "may this be played without asking the portal", and the + only safe answer about an address nobody understands is no. + """ + host = (host or "").strip().strip("[]").rstrip(".").lower() + if not host: + return True + if host == "localhost" or host.endswith(".localhost"): + return True + if host == "localhost.localdomain": + return True + try: + address = ipaddress.ip_address(host) + except ValueError: + # A name, and not one of the reserved loopback ones. + return False + mapped = getattr(address, "ipv4_mapped", None) + if mapped is not None: + address = mapped + return address.is_loopback or address.is_unspecified + + +def portal_flag(value: Any) -> Optional[bool]: + """A portal's 1/0 flag, or None when the portal did not set one. + + None is not False, and the difference is the whole of it: a row carrying + neither flag is a row the portal said nothing about, and silence has to + read as "no evidence" rather than "no". Portals write these as 1/0, as + "1"/"0", and occasionally as real booleans. + """ + if value is None: + return None + if isinstance(value, bool): + return value + text = str(value).strip().lower() + if not text: + return None + return text not in ("0", "false", "no", "off") + + +def plays_without_create_link(cmd: str, needs_link: Optional[bool]) -> bool: + """Whether this channel can be played from its command alone. + + The portal's own player.js asks create_link for a channel only when the row + asks for it -- because the portal proxies it through a per-session link + ('use_http_tmp_link') or picks a storage server per request + ('use_load_balancing'). Every other row plays the command the listing + already handed over. pvr.stalker does the same and cites that line of + player.js for it (ChannelManager::GetStreamURL); iptvnator does the same + again, with guards it added for portals that are not Ministra. + + Those guards are here too, and every one of them can only ever push a row + back onto the create_link path that exists today -- so none of them is able + to break a portal that works now: + + * the portal has to have answered the question at all. Every portal synced + before this was read carries no flags, and taking that silence as a "no" + would move all of them onto the static path at once. + * the command has to contain a URL. 'auto /media/file.mpg' does not, and + only create_link turns that into an address. + * on a scheme ffmpeg can open -- see PLAYABLE_SCHEMES. + * not on a host that can only mean the portal itself. + + The last condition is ours rather than anyone's reference behaviour, and it + is what makes this safe in a resolver rather than in a player: the command + must carry **no query string**. A link that expires keeps its token there, + and the cost of being wrong is not a retry -- by the time a stream fails + the resolver has already become ffmpeg, and Dispatcharr has spent this + channel's failover on a source that resolved perfectly well. A command with + no query has nothing in it that can go stale. + """ + if needs_link is not False: + return False + + link = extract_link(cmd) + if not link: + return False + + parsed = urlparse(link) + if parsed.scheme.lower() not in PLAYABLE_SCHEMES: + return False + if parsed.query: + return False + return not _is_portal_local(parsed.hostname or "") + + def slugify(value: str) -> str: """Reduce a display name to something safe for URLs, keys and filenames.""" slug = re.sub(r"[^a-z0-9]+", "-", value.strip().lower()).strip("-") @@ -478,10 +739,31 @@ def name_from_url(url: str) -> str: def parse_expiry(value: Any) -> Optional[datetime]: - """Read a subscription expiry out of a free-text portal field.""" + """Read a subscription expiry out of whatever field carried it. + + Two shapes, because the two places it is found do not agree. get_main_info + holds free text a reseller typed; the profile's own account_info block + holds a Unix timestamp, which is also how a portal writes "never" -- 0 and + -1 both mean no expiry, and reading either as a date would report every + unlimited account as having run out in 1970. + """ text = str(value or "").strip() if not text or text.startswith("0000-00-00"): return None + + if re.fullmatch(r"-?\d+", text): + seconds = int(text) + if seconds <= 0: + return None + # Past the year 3000 in seconds is a value that was meant as + # milliseconds; portals send both. + if seconds > 32503680000: + seconds //= 1000 + try: + return datetime.fromtimestamp(seconds, timezone.utc) + except (OverflowError, OSError, ValueError): + return None + for fmt in _EXPIRY_FORMATS: try: return datetime.strptime(text, fmt).replace(tzinfo=timezone.utc) @@ -490,6 +772,44 @@ def parse_expiry(value: Any) -> Optional[datetime]: return None +def prehash(mac: str) -> str: + """The SHA1 a client presents itself with, derived from the MAC it uses. + + Neither the stock middleware nor any portal met so far reads this. It is + there for the optional access_filter.php a reseller can install in front of + it, which is also the reason a box that sends nothing at all is the shape + some of those filters reject. + + What to send is written down nowhere. iptvnator computes the SHA1 of the + upper-case MAC; QiTV ships one constant for every install it has ever run. + Only the first says something true about this box, so it is the one copied + here -- and if a portal ever checks it against its own record, a constant + shared by every user of one client is the version that fails. + """ + return hashlib.sha1(mac.upper().encode("utf-8")).hexdigest().upper() + + +# Portals put markup in the sentence they refuse with -- "Your STB is +# damaged.
Call the provider." is a stock one -- and it lands in a +# Dispatcharr notification, where a tag is noise at best. +_MARKUP = re.compile(r"<[^>]*>") + +# Phrasings for the one refusal a user can act on: the portal has this MAC +# bound to a device id that is not the one being sent. Kept to the binding +# itself, because 'device' alone turns up in refusals with no remedy at all +# ("device limit reached"), and labelling one of those would hand the user a +# fix that cannot work. Safe as a phrase set where the raw-body patterns are +# not: this is a field the middleware wrote, not an arbitrary document. +DEVICE_CONFLICT = ( + re.compile(r"device\s*conflict", re.I), + re.compile( + r"device[\s_-]?id[^.!?]{0,40}?" + r"(mismatch|conflict|does\s*not\s*match|not\s*match)", + re.I, + ), +) + + def normalize_mac(mac: str) -> str: """Accept the shapes MAC addresses are quoted in, emit the one we use. @@ -758,53 +1078,303 @@ def normalize_portal_url(url: str) -> str: path = parsed.path lower = path.lower() - if lower.endswith(("/portal.php", "/load.php")): - pass # explicit endpoint: leave exactly as given + if lower.endswith(".php"): + # An explicit endpoint, left exactly as given -- including a path no + # standard install serves. This is the one departure from stalkerhek, + # which swaps any other .php for portal.php in the same directory: + # panels living under a path of their own do exist, that path is the + # address the provider handed out, and replacing it with a guess is how + # the one URL known to work stops being tried at all. + # endpoint_candidates() probes the standard siblings after it anyway, + # so nothing is lost by trusting what was pasted first. + pass elif path in ("", "/"): path = "/portal.php" - elif lower.endswith(".php"): - # Some other .php file: swap in portal.php from the same directory. - directory = path.rsplit("/", 1)[0] - path = f"{directory}/portal.php" else: path = path.rstrip("/") + "/portal.php" return parsed._replace(path=path).geturl() -def alternate_endpoint(url: str) -> str: - """The other place a Stalker API lives, or '' when there isn't one. +# -- redaction -------------------------------------------------------------- +# +# The Portals box was the one place a subscription's credentials were rendered +# back to whoever opened the page, and Dispatcharr serves a plugin's settings +# row to every account on the install. So the row keeps a redacted rendering of +# the list, and the real one lives only in the registry file, which the panel +# cannot reach -- see registry.py. +# +# What this buys is that the credentials are no longer on the screen or in the +# API response. It is not encryption at rest and must not be sold as such: the +# resolver reads the MAC on every tune, in a process with no Django, so +# portals.txt and the state mirrors go on holding it in the clear at 0600. + +# U+2022, because the token has to be something no URL, MAC or key=value could +# be, and something nobody types into the box by accident. An ASCII '****' is a +# perfectly plausible password. +MASK = "•" * 4 + +# Redacted wherever they appear as key=value. The MAC is redacted too, but it +# is positional and handled apart. The rule is what a value *proves* rather +# than what it configures: anything a stranger could authenticate with is +# hidden, while model, timezone, max_streams and the epg switches stay +# readable -- they tune behaviour, and they are what lets a user recognise +# their own line. +SECRET_KEYS = ("username", "password", "device_id", "device_id2", + "serial", "signature") - Ministra answers on two paths and installs differ in which they expose: - ``/c/portal.php``, which is what :func:`normalize_portal_url` builds - and what most providers hand out, and ``/server/load.php``, which is - the older canonical one and the only one pvr.stalker has ever asked for. - A portal serving just one of them used to be unusable if the user had been - given the other, with a 404 and nothing to suggest. - The mapping is pvr.stalker's, read backwards as well as forwards:: +def is_masked(text: Optional[str]) -> bool: + """True if a portal list carries redacted values rather than real ones.""" + return MASK in (text or "") - http://h/c/portal.php -> http://h/server/load.php - http://h/stalker_portal/c/portal.php -> http://h/stalker_portal/server/load.php - http://h/server/load.php -> http://h/c/portal.php + +def _line_body(line: str) -> Tuple[bool, str]: + """Separate a line's comment marker from the portal line inside it. + + A '#' line is how the help text tells users to suspend a portal without + losing its channels, so it holds a real MAC and has to be redacted like any + other -- and put back together the same way. """ - parsed = urlparse(url) - path = parsed.path - directory, _, filename = path.rpartition("/") - filename = filename.lower() - - if filename == "portal.php": - base = directory[:-2] if directory.lower().endswith("/c") else directory - new_path = base + "/server/load.php" - elif filename == "load.php": - base = directory[:-7] if directory.lower().endswith("/server") else directory - new_path = base + "/c/portal.php" - else: - return "" + stripped = line.strip() + if stripped.startswith("#"): + return True, stripped.lstrip("#").strip() + return False, stripped + + +def _mac_index(parts: List[str]) -> int: + """Which '|' field holds the MAC, by split_portal_line's own rule. + + The token counts as a MAC here. Without that, redacting the MAC would move + the fields of an unnamed line: 'url | MAC | extras' reads correctly only + because the second field looks like a MAC, and 'url | | extras' + would otherwise be read as name, url and MAC. + """ + if len(parts) >= 3 and not ( + MAC_RE.match(normalize_mac(parts[1])) or parts[1].strip() == MASK + ): + return 2 + return 1 + - if new_path == path: +def _line_slug(line: str) -> str: + """The slug a portal line is filed under, tolerating a redacted MAC. + + This is what pairs a redacted line back up with the real one it was made + from, so the two have to agree even though one of them no longer has a MAC. + """ + parts = [p.strip() for p in _line_body(line)[1].split("|")] + if len(parts) < 2: return "" - return parsed._replace(path=new_path).geturl() + at = _mac_index(parts) + name = parts[0] if at == 2 else "" + return slugify(name or name_from_url(parts[at - 1])) + + +def _mask_extras(segment: str) -> str: + """Redact the secrets among one segment's key=value pairs.""" + try: + tokens = shlex.split(segment) + except ValueError: + return segment + out = [] + for token in tokens: + key, sep, value = token.partition("=") + if not sep: + out.append(token) + elif value and key.strip().lower() in SECRET_KEYS: + out.append(f"{key}={MASK}") + else: + out.append(f"{key}={_quote_if_needed(value)}") + return " ".join(out) + + +def _unmask_extras(segment: str, extras: Dict[str, str]) -> str: + """Put the secrets back into one segment's key=value pairs.""" + if MASK not in segment: + return segment + try: + tokens = shlex.split(segment) + except ValueError: + return segment + out = [] + for token in tokens: + key, sep, value = token.partition("=") + if sep and value.strip() == MASK: + restored = extras.get(key.strip().lower(), "") + # Nothing to restore leaves the token standing: see unmask_portals. + out.append(f"{key}={_quote_if_needed(restored)}" if restored + else f"{key}={MASK}") + elif sep: + out.append(f"{key}={_quote_if_needed(value)}") + else: + out.append(token) + return " ".join(out) + + +def mask_portals(text: str) -> str: + """Render a portal list with every credential replaced by :data:`MASK`. + + Everything that is not a credential survives -- names, URLs, comments, the + tuning keys -- so the box still reads as the user's own configuration, and + a portal is still deleted by deleting its line. + + The name is written out even where the line derived it from the URL: it is + the identity :func:`unmask_portals` pairs the line back up by, and writing + it also settles where the MAC was once the MAC is gone. + """ + out = [] + for raw in text.splitlines(): + commented, body = _line_body(raw) + if not body: + out.append(raw) + continue + + parsed, _ = split_portal_line(body) + if parsed is None: + # A line that does not parse holds no credential we could find, and + # the user has to go on seeing it to be able to fix it. + out.append(raw) + continue + + parts = [p.strip() for p in body.split("|")] + at = _mac_index(parts) + parts[at] = MASK + parts[at + 1:] = [_mask_extras(part) for part in parts[at + 1:]] + if at == 1: + parts.insert(0, parsed["name"]) + # format_portal_line() always writes an extras field, empty or not. + while len(parts) > 3 and not parts[-1]: + parts.pop() + + line = " | ".join(parts) + out.append(f"# {line}" if commented else line) + + return "\n".join(out) + ("\n" if text.endswith("\n") else "") + + +def unmask_portals(text: str, stored: str) -> str: + """Put the credentials back into a list the panel sent back redacted. + + ``stored`` is the registry's copy, the only place the real values exist. + Lines are paired with it by slug, and failing that by URL: renaming a + portal and moving one are both ordinary edits, and either would otherwise + orphan the line's own MAC. A line typed out in full carries no token and is + returned exactly as typed, so pasting a list back in still works. + + A token nothing matches is left standing rather than resolved to an empty + value: Plugin._portals refuses a list that still holds one and says which + line to retype, which is a great deal easier to act on than a portal + quietly authenticating with nothing. + """ + if not is_masked(text): + return text + + records = [] + for raw in (stored or "").splitlines(): + body = _line_body(raw)[1] + if not body: + continue + parsed, _ = split_portal_line(body) + if parsed is None: + continue + records.append({ + "slug": _line_slug(body), + "url": normalize_portal_url(parsed["url"]), + "parsed": parsed, + "used": False, + }) + + def take(slug: str, url: str) -> Optional[Dict[str, Any]]: + """The stored line this redacted one came from, consumed once. + + Consumed, because a suspended line and its replacement can share both + slug and host -- that is what commenting a line out is for -- and the + second of them must not be handed the first one's credentials. + """ + for field, wanted in (("slug", slug), ("url", url)): + if not wanted: + continue + for record in records: + if not record["used"] and record[field] == wanted: + record["used"] = True + return record["parsed"] + return None + + out = [] + for raw in text.splitlines(): + commented, body = _line_body(raw) + if MASK not in body: + out.append(raw) + continue + + parts = [p.strip() for p in body.split("|")] + at = _mac_index(parts) + source = take(_line_slug(body), normalize_portal_url(parts[at - 1])) + if source is not None: + if parts[at] == MASK: + parts[at] = source["mac"] + parts[at + 1:] = [ + _unmask_extras(part, source["extras"]) for part in parts[at + 1:] + ] + + line = " | ".join(parts) + out.append(f"# {line}" if commented else line) + + return "\n".join(out) + ("\n" if text.endswith("\n") else "") + + +def endpoint_candidates(url: str) -> List[str]: + """Where a Stalker API might answer for this URL, best guess first. + + The endpoint cannot be worked out from what a user pastes. Ministra serves + ``/stalker_portal/server/load.php`` and shows its interface at + ``/stalker_portal/c/``; reseller panels serve ``/c/portal.php`` + or ``/portal.php``, and some serve neither, from a path of their own. + pvr.stalker knows the first pair and stalkerhek the second. No client knows + all of them, which is why this is a list to probe rather than a mapping to + apply. + + The configured URL always comes first, and the second entry is still what + :func:`alternate_endpoint` used to be the whole of -- the pvr.stalker + mapping, read both ways. The rest are added after it, so a portal that was + found on the second try before is found on the second try still. + + The siblings are built from the install root: the configured directory with + a trailing ``/c`` or ``/server`` taken off, because both of those are the + API's own subdirectory rather than part of where the install lives. + """ + parsed = urlparse(url) + directory, _, filename = parsed.path.rpartition("/") + if not filename.lower().endswith(".php"): + # Not an endpoint at all: read the whole path as the directory rather + # than throwing away its last segment. + directory = parsed.path + + base = directory.rstrip("/") + for own in ("/c", "/server"): + if base.lower().endswith(own): + base = base[: -len(own)] + break + + paths = [ + parsed.path, + # The pvr.stalker pair, which is what this list grew out of. + base + "/server/load.php", + base + "/c/portal.php", + base + "/portal.php", + ] + # Already the canonical form when the base ends there -- nesting it again + # would probe a path no server has. + if "/stalker_portal" not in base.lower(): + paths.append(base + "/stalker_portal/server/load.php") + + candidates: List[str] = [] + for path in paths: + candidate = parsed._replace(path=path).geturl() + if candidate not in candidates: + candidates.append(candidate) + return candidates # --------------------------------------------------------------------------- @@ -989,6 +1559,80 @@ def load_portal(slug: str, client=None) -> Optional[PortalConfig]: return cfg +def _static_key(slug: str) -> str: + return f"{REDIS_PREFIX}:static:{slug}" + + +def _static_mirror(slug: str) -> str: + return f"static-{slug}" + + +def static_commands(channels: List["ChannelEntry"]) -> List[str]: + """The commands the resolver may play without asking the portal first.""" + return sorted( + { + channel.cmd + for channel in channels + if plays_without_create_link(channel.cmd, channel.needs_link) + } + ) + + +def save_static_cmds(slug: str, commands: List[str], client=None) -> None: + """Publish the commands that need no create_link, for the resolver to read. + + Written on every sync including when it is empty, which is what nearly + every portal produces -- and what a portal that has *stopped* marking its + channels static has to leave behind, rather than inheriting the last list + that said otherwise. + + Mirrored first, like the portal itself: if Redis refuses, the sync says so + and the resolver still reads the right thing off disk. + """ + payload = list(commands) + _mirror_write(_static_mirror(slug), payload) + client = client or get_redis() + client.set(_static_key(slug), json.dumps(payload)) + + +def load_static_cmds(slug: str, client=None) -> set: + """The same set back, or an empty one. + + Every path out of here that is not a list lands on the empty set, and that + is deliberate rather than lazy: not knowing whether a channel is static has + exactly one safe reading, and it is asking the portal -- which is what this + plugin did before any of this existed. + """ + client = _client_or_none(client) + + raw = None + if client is not None: + try: + raw = client.get(_static_key(slug)) + except Exception: + raw = None + if raw: + try: + payload = json.loads(raw) + except (ValueError, TypeError): + payload = None + if isinstance(payload, list): + return {str(item) for item in payload} + + payload = _mirror_read(_static_mirror(slug)) + if not isinstance(payload, list): + return set() + + # Put it back, so a wiped Redis costs a file read once rather than once + # per tune -- the same bargain load_portal makes. + if client is not None: + try: + client.set(_static_key(slug), json.dumps(payload)) + except Exception: + pass + return {str(item) for item in payload} + + def _sync_lock_key() -> str: return f"{REDIS_PREFIX}:sync-lock" @@ -1127,8 +1771,9 @@ def published_slugs() -> List[str]: def forget_portal(slug: str, client=None) -> None: _mirror_forget(_portal_mirror(slug)) + _mirror_forget(_static_mirror(slug)) client = client or get_redis() - client.delete(_portal_key(slug), _token_key(slug)) + client.delete(_portal_key(slug), _token_key(slug), _static_key(slug)) def _fallback_key() -> str: @@ -1243,6 +1888,10 @@ class ChannelEntry: # canonical_cmd. Counted rather than logged per channel, because on the # portal that prompted it, 647 of them arrived at once. cmd_rewritten: bool = False + # Whether the portal says this channel needs a link minted for it before it + # can be played. None when the row carried neither flag, which is not the + # same as False -- see portal_flag and plays_without_create_link. + needs_link: Optional[bool] = None class Portal: @@ -1284,6 +1933,10 @@ def __init__( # Whether the portal said the token it handed back is already good for # more than the handshake. Reported straight back to it in get_profile. self.valid_token = False + # The nonce the handshake handed back, echoed in get_profile's metrics. + # A portal that issues one and never sees it again is being talked to + # by something that did not read its own handshake. + self.handshake_random = "" # What get_profile answered during login(), kept so nothing has to ask # twice: the expiry report and the blocked flag both read it. self.profile: Dict[str, Any] = {} @@ -1384,30 +2037,45 @@ def _get_json(self, query: str, with_auth: bool = True) -> Any: raise PortalError(message) try: - return resp.json() + payload = resp.json() except ValueError: - snippet = (resp.text or "").strip()[:300] + # Classified on the whole body and displayed truncated: the + # patterns are anchored, so cutting first could only ever hide a + # refusal, never invent one. + body = (resp.text or "").strip() # A dead session is answered in plain text with a 200 attached, so # it arrives here rather than as an HTTP error. Saying so is what # lets the resolver re-authenticate instead of failing the tune. - if snippet.lower() == AUTH_FAILED_BODY: - raise PortalAuthError("portal says the session is no longer authorised") + refusal = auth_refusal(body) + if refusal: + raise PortalAuthError(refusal) # Anything else that is not JSON is an HTML error page, a login # form, or a landing page: something is listening, but it is not a # Stalker API, so the other endpoint is worth a try. - raise PortalEndpointError(f"portal returned non-JSON response: {snippet}") + raise PortalEndpointError( + f"portal returned non-JSON response: {body[:300]}" + ) + + # A refusal can also arrive as perfectly good JSON, and then it is not + # this reply that is unusable but the session behind it. + refusal = envelope_refusal(payload) + if refusal: + raise PortalAuthError(refusal) + return payload # -- authentication --------------------------------------------------- def handshake(self) -> str: """Reserve a token. The portal may hand back a different one.""" data = self._get_json( - f"type=stb&action=handshake&token={self.token}&JsHttpRequest=1-xml", + f"type=stb&action=handshake&token={self.token}" + f"&prehash={prehash(self.cfg.mac)}&JsHttpRequest=1-xml", with_auth=False, ) js = data.get("js") if isinstance(data, dict) else None if isinstance(js, dict) and js.get("token"): self.token = str(js["token"]) + self.handshake_random = str(js.get("random") or "") # 'not_valid' is the portal saying the token still has to be # earned. get_profile is told the same thing back, which is how it # knows whether it is being asked to validate or merely to report. @@ -1466,6 +2134,9 @@ def get_profile(self, auth_second_step: bool = False) -> Dict[str, Any]: f"&device_id={quote(self.cfg.device_id)}" f"&device_id2={quote(self.cfg.device_id2)}" f"&signature={quote(self.cfg.signature)}" + f"&client_type=STB&video_out=hdmi" + f"&metrics={quote(self._metrics())}" + f"&prehash={prehash(self.cfg.mac)}" f"¬_valid_token={0 if self.valid_token else 1}" f"&auth_second_step={1 if auth_second_step else 0}" ) @@ -1473,6 +2144,26 @@ def get_profile(self, auth_second_step: bool = False) -> Dict[str, Any]: js = data.get("js") if isinstance(data, dict) else None return js if isinstance(js, dict) else {} + def _metrics(self) -> str: + """The box describing itself, in the shape get_profile wants it. + + A JSON blob rather than parameters, which is Ministra's choice and not + ours. It is what the admin panel stores and shows the reseller, so a + portal whose operator looks at their device list sees a MAG rather than + a blank row -- and the filters that reject a box reporting nothing read + this too. + """ + return json.dumps( + { + "mac": self.cfg.mac, + "sn": self.cfg.serial_number, + "model": self.cfg.model, + "type": "STB", + "random": self.handshake_random, + }, + separators=(",", ":"), + ) + # What get_profile's 'status' means. The portal decides which authentication # this account needs and says so here, rather than the client guessing from # whether a password happens to be configured. @@ -1508,7 +2199,7 @@ def login(self) -> str: explicit refusal (:class:`PortalAuthError`) is still fatal, because that is the portal answering rather than failing to. """ - self._handshake_on_either_endpoint() + self._handshake_on_any_endpoint() try: self.profile = self.get_profile() @@ -1544,46 +2235,53 @@ def login(self) -> str: return self.token - def _handshake_on_either_endpoint(self) -> None: - """Shake hands, trying the portal's other API path if this one is not it. + def _handshake_on_any_endpoint(self) -> None: + """Shake hands, walking the portal's other API paths if this one is not it. The handshake is every session's first request, so a portal reached at the wrong path fails here and nowhere later -- which makes this the one - place worth spending an extra round-trip on. + place worth spending extra round-trips on. - Only a :class:`PortalEndpointError` earns that second try: a 404, or an - answer that is not JSON. A portal that is unreachable, unwell or - refusing the MAC would answer identically on both paths, and at tune - time a wasted round-trip is time Dispatcharr is not yet spending on the - next source. + Only a :class:`PortalEndpointError` moves to the next candidate: a 404, + or an answer that is not JSON. A portal that is unreachable, unwell or + refusing the MAC would answer identically on every path -- they all + live on the same host -- and at tune time a wasted round-trip is time + Dispatcharr is not yet spending on the next source. That is also why + the list is only walked when the user's URL is wrong, which is a + setup-time mistake rather than something that happens mid-service. The swap lasts for this session only. Nothing is written back, so the cost is one failed request per sync and per token expiry -- small, and the warning tells the user how to stop paying it for good. """ - try: - self.handshake() - return - except PortalEndpointError as exc: - alternate = alternate_endpoint(self.url) - if not alternate: + first_failure: Optional[PortalError] = None + + for candidate in endpoint_candidates(self.url): + self.url = candidate + try: + self.handshake() + except PortalEndpointError as exc: + if first_failure is None: + first_failure = exc + continue + except PortalError: + # Not a statement about the path, so no other path can help. + self.url = self.cfg.url raise - first_failure = exc - self.url = alternate - try: - self.handshake() - except PortalError: - # The other path is no better. Report the original failure: it is - # the one about the URL the user actually configured. - self.url = self.cfg.url - raise first_failure - - self.warnings.append( - f"the portal does not answer at {self.cfg.url} ({first_failure}), " - f"but does at {alternate}; put that on its portal line to save a " - "failed request on every sync" - ) + if candidate != self.cfg.url: + self.warnings.append( + f"the portal does not answer at {self.cfg.url} " + f"({first_failure}), but does at {candidate}; put that on " + "its portal line to save a failed request on every sync" + ) + return + + # Every path was answered by something that was not a Stalker API. + # Report the first failure: it is the one about the URL the user + # actually configured. + self.url = self.cfg.url + raise first_failure @staticmethod def _profile_status(profile: Dict[str, Any]) -> int: @@ -1601,12 +2299,28 @@ def _profile_message(profile: Dict[str, Any]) -> str: """The portal's own explanation, if it gave one. ``block_msg`` first: when both are set it is the specific one, and it - is what the reseller wrote for exactly this situation. + is what the reseller wrote for exactly this situation. Markup comes out + of it, because this ends up in a Dispatcharr notification and portals + write these with ``
`` in them. + + A device conflict gets a sentence added. It is the one refusal here + that the user can do something about, and the portal's own wording for + it names the device rather than the binding -- so the message arrives + describing a problem with the box instead of one with two settings on + the portal line. """ for key in ("block_msg", "msg"): - value = str(profile.get(key) or "").strip() - if value: - return value + value = " ".join(_MARKUP.sub(" ", str(profile.get(key) or "")).split()) + if not value: + continue + if any(pattern.search(value) for pattern in DEVICE_CONFLICT): + value += ( + " -- the portal has this MAC bound to a different device " + "id; put the ones it expects on the portal line with " + "'device_id=' and 'device_id2=', or ask the provider to " + "clear the binding" + ) + return value return "" def account_snapshot(self) -> Dict[str, Any]: @@ -1636,14 +2350,30 @@ def account_snapshot(self) -> Dict[str, Any]: """ snapshot: Dict[str, Any] = {"expires": None, "blocked": False} - try: - js = self._get_json( - "type=account_info&action=get_main_info&JsHttpRequest=1-xml" - ).get("js") - if isinstance(js, dict): - snapshot["expires"] = parse_expiry(js.get("phone")) - except Exception: - pass + # The profile login() already read comes first, and costs nothing: + # 'account_info' is where Ministra itself puts the date, as a Unix + # timestamp. A portal that answered there is not asked again. + info = self.profile.get("account_info") + if isinstance(info, dict): + snapshot["expires"] = parse_expiry(info.get("expire_date")) + + if snapshot["expires"] is None: + try: + js = self._get_json( + "type=account_info&action=get_main_info&JsHttpRequest=1-xml" + ).get("js") + if isinstance(js, dict): + # 'phone' last: it is where the date ends up on the portals + # this plugin actually meets, but it is a free-text field + # and the three named ones mean only this when present. + for field in ("expire_date", "end_date", + "expire_billing_date", "phone"): + found = parse_expiry(js.get(field)) + if found: + snapshot["expires"] = found + break + except Exception: + pass snapshot["blocked"] = str(self.profile.get("blocked") or "0") not in ("0", "") @@ -1688,10 +2418,21 @@ def get_all_channels(self) -> List[ChannelEntry]: ) channels: List[ChannelEntry] = [] + seen = set() for row in rows: channel = self._channel_from_row(row) - if channel is not None: - channels.append(channel) + if channel is None: + continue + # Portals do repeat a channel in this response, and a duplicate is + # not a harmless extra row: Dispatcharr hashes a stream partly on + # its URL, so it becomes a second stream for one channel. Keyed the + # way the paged path keys it -- the id when there is one, the + # command when there is not. + key = channel.channel_id or channel.cmd + if key in seen: + continue + seen.add(key) + channels.append(channel) return channels @staticmethod @@ -1711,6 +2452,16 @@ def _channel_from_row(row: Any) -> Optional[ChannelEntry]: channel_id = str(row.get("id") or "") marker = canonical_cmd(cmd, channel_id) + # Two flags, one answer: either of them set means the portal mints the + # link. A row carrying neither has not answered, and None says so. + tmp_link = portal_flag(row.get("use_http_tmp_link")) + balanced = portal_flag(row.get("use_load_balancing")) + needs_link = ( + None + if tmp_link is None and balanced is None + else bool(tmp_link) or bool(balanced) + ) + return ChannelEntry( channel_id=channel_id, name=name, @@ -1722,6 +2473,7 @@ def _channel_from_row(row: Any) -> Optional[ChannelEntry]: # Portals write these as 1/0, and sometimes as "1"/"0". tv_archive=str(row.get("enable_tv_archive") or "0") not in ("0", ""), tv_archive_duration=str(row.get("tv_archive_duration") or ""), + needs_link=needs_link, ) def get_ordered_list(self, page: int) -> Dict[str, Any]: @@ -1735,7 +2487,14 @@ def get_ordered_list(self, page: int) -> Dict[str, Any]: """ data = self._get_json( "type=itv&action=get_ordered_list&JsHttpRequest=1-xml" - f"&genre=*&fav=0&sortby=number&p={int(page)}" + # 'category' says the same thing as 'genre' to the portals that + # read that one instead, 'force_ch_link_check' is sent empty by + # every client that sends it at all, and 'hd' asks for the whole + # line-up rather than the HD half of it. None of the three changes + # what a Ministra portal answers; each of them is what one of the + # other clients found a portal that wanted it. + f"&genre=*&category=*&fav=0&force_ch_link_check=&hd=0" + f"&sortby=number&p={int(page)}" ) js = data.get("js") if isinstance(data, dict) else None return js if isinstance(js, dict) else {} @@ -1942,7 +2701,14 @@ def create_link(self, cmd: str) -> str: without it -- see :func:`stream_id`. Only ever sent with a value, so a portal that never asked for it sees the request it has always seen. """ - query = f"action=create_link&type=itv&cmd={quote(cmd, safe='')}" + query = ( + "action=create_link&type=itv" + f"&cmd={encode_cmd(cmd)}" + # What the portal's own player.js sends beside the command. Neither + # changes what a live channel answers; both are what a portal + # expecting its own client sees on every request except ours. + "&disable_ad=0&download=0" + ) channel = stream_id(cmd) if channel: query += f"&stream={quote(channel, safe='')}" diff --git a/sync.py b/sync.py index c519cf6..cc1a47d 100644 --- a/sync.py +++ b/sync.py @@ -29,6 +29,8 @@ python_executable, save_fallback, save_portal, + save_static_cmds, + static_commands, ) # Where Dispatcharr's own M3U upload endpoint puts files. Reusing it keeps our @@ -808,9 +810,11 @@ def sync_portal( ) genres = portal.get_genres() - # The resolver reads this at tune time; publish it before the M3U lands so - # a channel can never reference a portal Redis doesn't know about yet. + # The resolver reads these at tune time; publish them before the M3U lands + # so a channel can never reference a portal Redis doesn't know about yet. save_portal(cfg) + static = static_commands(channels) + save_static_cmds(cfg.slug, static) path = write_m3u(cfg.slug, build_m3u(portal, channels, genres)) account, account_created = upsert_account(cfg, path, refresh_hours) @@ -839,6 +843,17 @@ def sync_portal( rewritten, ) + if static: + # Worth saying for the same reason the rewrite count is: it changes + # what happens at tune time, and on the providers concerned it is the + # difference between one request to the portal and none. + logger.info( + "distalker: %s: %d channel(s) are marked as needing no temporary " + "link, and will play from their command without asking the portal", + cfg.name, + len(static), + ) + epg = sync_epg(cfg, portal, channels, logger, trigger_refresh=trigger_refresh) return { @@ -851,6 +866,7 @@ def sync_portal( "file": path, "expires": snapshot["expires"], "blocked": snapshot["blocked"], + "static": len(static), "epg": epg, } diff --git a/tests/test_auth.py b/tests/test_auth.py index 1c41b3a..f8b3c11 100644 --- a/tests/test_auth.py +++ b/tests/test_auth.py @@ -121,6 +121,49 @@ def test_the_portal_gets_the_last_word_on_why(): raise AssertionError("status 1 must refuse the session") +def test_the_refusal_arrives_without_the_markup_it_was_written_in(): + """It lands in a Dispatcharr notification, where a
is noise.""" + p = portal({ + "handshake": HANDSHAKE, + "get_profile": {"js": {"status": 1, + "block_msg": "Subscription expired.
Call us."}}, + }) + try: + p.login() + except s.PortalAuthError as exc: + assert str(exc) == "Subscription expired. Call us.", exc + else: + raise AssertionError("status 1 must refuse the session") + + +def test_a_bound_mac_is_named_as_the_settings_that_fix_it(): + """The one refusal here a user can act on, and the portal misnames it. + + Its own wording points at the box; the problem is two values on the portal + line. Kept to the binding itself -- 'device limit reached' has no remedy, + and offering one would be worse than saying nothing. + """ + p = portal({ + "handshake": HANDSHAKE, + "get_profile": {"js": {"status": 1, "msg": "device id mismatch"}}, + }) + try: + p.login() + except s.PortalAuthError as exc: + assert "device_id2=" in str(exc), exc + else: + raise AssertionError("status 1 must refuse the session") + + plain = portal({ + "handshake": HANDSHAKE, + "get_profile": {"js": {"status": 1, "msg": "device limit reached"}}, + }) + try: + plain.login() + except s.PortalAuthError as exc: + assert str(exc) == "device limit reached", exc + + def test_a_second_step_that_still_fails_is_refused(): p = portal( {"handshake": HANDSHAKE, @@ -184,10 +227,47 @@ def test_the_whole_stb_identity_is_sent(): p.login() query = [q for q in p.queries if "action=get_profile" in q][0] for expected in ("signature=" + "a" * 64, "sn=SN1", "stb_type=MAG322", - "num_banks=1", "image_version=216", "hd=1", "ver=", "hw_version="): + "num_banks=1", "image_version=216", "hd=1", "ver=", "hw_version=", + # The two other clients send these and this one did not. + # Harmless to a portal that ignores them, and the shape a + # reseller's access filter looks for in one that does not. + "client_type=STB", "video_out=hdmi"): + assert expected in query, f"{expected} missing from {query}" + + +def test_the_box_is_described_where_the_admin_panel_reads_it(): + """'metrics' is what a portal stores and shows its operator. + + Echoing the handshake's nonce back inside it is the part a portal could + actually check: it issued that value one request ago, and only something + that read the answer can quote it. + """ + p = portal({"handshake": {"js": {"token": "TOK", "random": "R1"}}, + "get_profile": {"js": {"status": 0}}}, + serial_number="SN1", model="MAG322") + p.login() + query = [q for q in p.queries if "action=get_profile" in q][0] + for expected in ("%22random%22%3A%22R1%22", "%22model%22%3A%22MAG322%22", + "%22sn%22%3A%22SN1%22", "%22type%22%3A%22STB%22"): assert expected in query, f"{expected} missing from {query}" +def test_the_prehash_is_this_box_rather_than_every_box(): + """A constant shared by every user of one client is the version that fails. + + Nothing in Ministra reads it; an access_filter.php in front of it can, and + that is the whole reason to send one at all. + """ + p = portal({"handshake": HANDSHAKE, "get_profile": {"js": {"status": 0}}}) + p.login() + expected = "prehash=" + s.prehash("00:1A:79:AA:BB:CC") + assert any(expected in q for q in p.queries if "action=handshake" in q), p.queries + assert any(expected in q for q in p.queries if "action=get_profile" in q), p.queries + # The MAC, not the box, and not case-sensitive about how it was written. + assert s.prehash("00:1a:79:aa:bb:cc") == s.prehash("00:1A:79:AA:BB:CC") + assert len(s.prehash("00:1A:79:AA:BB:CC")) == 40 + + def test_a_dead_session_in_plain_text_is_an_auth_error(): """Ministra answers 200 with prose, not JSON, once a token has expired. @@ -215,6 +295,57 @@ def json(self): raise AssertionError("'Authorization failed.' must be typed as an auth error") +def test_every_shape_a_refusal_arrives_in(): + """The table, so a shape met next is added here rather than argued about.""" + for body in ("Authorization failed.", + # The counter the stock server appends. The exact comparison + # this replaces did not recognise it, and it is the refusal + # the resolver exists to recover from. + "Authorization failed. 75", + "authorization failed", + "Access denied.", + " Unauthorized request.\n"): + assert s.auth_refusal(body), body + + for body in ("", + # A proxy or WAF page: 38 characters, so only matching the + # whole body keeps it out -- and it must stay out, or a host + # that never answered sends the resolver re-authenticating. + "Access denied", + "Access denied by policy", + "ffmpeg http://host/stream"): + assert not s.auth_refusal(body), body + + +def test_a_refusal_can_arrive_as_perfectly_good_json(): + """Panels that are not Ministra refuse inside the envelope, not instead of it. + + Every one of these used to read as an ordinary reply: get_genres answered + this way counted as a portal with no genres, and 'Test portals' reported it + as authenticated with 0 groups. + """ + assert s.envelope_refusal({"js": "Authorization failed."}) + assert s.envelope_refusal({"js": {"msg": "Access denied."}}) + # The panel's own wording is what gets quoted back. + assert "Invalid token" in s.envelope_refusal({"js": {"error": "Invalid token"}}) + + +def test_a_portal_asking_for_a_password_is_not_a_portal_refusing(): + """The one false positive that would cost a working install. + + 'msg' is where a status-2 reply writes the sentence login() reads to know + it should call do_auth. Reading it here would turn every portal that says + 'Authorization required' into a hard refusal and skip the step it asked + for; a status-1 'msg' is the provider's own wording, which login() quotes + better than a substitute could. + """ + assert not s.envelope_refusal({"js": {"status": 2, "msg": "Authorization required"}}) + assert not s.envelope_refusal({"js": {"status": 1, "msg": "Access denied."}}) + # A reply that worked carries no refusal at all, in any field. + assert not s.envelope_refusal({"js": {"data": [], "total_items": 0}}) + assert not s.envelope_refusal({"js": [{"id": "1", "title": "All"}]}) + + def test_create_link_expiry_is_typed_so_the_resolver_can_recover(): """A dead token is usually a hollow success, not a refusal. @@ -283,17 +414,45 @@ def test_when_neither_endpoint_answers_the_configured_one_is_blamed(): assert p.url == p.cfg.url, p.url -def test_the_two_endpoints_map_onto_each_other(): +def test_the_configured_url_is_always_tried_first(): + """Including one no standard install serves: it is the address handed out.""" + for given in ("http://h/c/portal.php", "http://h/cp/api.php", + "http://h:8080/stalker_portal/server/load.php"): + assert s.endpoint_candidates(given)[0] == given, given + + +def test_the_second_guess_is_still_the_one_it_always_was(): + """The pvr.stalker mapping, read both ways, so nothing found on the second + try before takes any longer now.""" cases = { "http://h/c/portal.php": "http://h/server/load.php", "http://h/stalker_portal/c/portal.php": "http://h/stalker_portal/server/load.php", "http://h/server/load.php": "http://h/c/portal.php", "http://h:8080/c/portal.php": "http://h:8080/server/load.php", - # Nothing sensible to swap to. - "http://h/something.cgi": "", } for given, expected in cases.items(): - assert s.alternate_endpoint(given) == expected, given + assert s.endpoint_candidates(given)[1] == expected, given + + +def test_the_paths_probed_after_that(): + """The spellings pvr.stalker never knew, and no path probed twice.""" + found = s.endpoint_candidates("http://h/c/portal.php") + assert found == [ + "http://h/c/portal.php", + "http://h/server/load.php", + "http://h/portal.php", + "http://h/stalker_portal/server/load.php", + ], found + + # A base that is already a stalker_portal install: nesting it again would + # probe a path no server has. + nested = s.endpoint_candidates("http://h/stalker_portal/c/portal.php") + assert not any(p.count("stalker_portal") > 1 for p in nested), nested + + # A panel under a path of its own keeps that path, and its siblings are + # looked for beside it rather than at the site root. + panel = s.endpoint_candidates("http://h/cp/api.php") + assert all("/cp/" in p for p in panel), panel def test_there_is_no_watchdog(): diff --git a/tests/test_config.py b/tests/test_config.py index 11f625b..15bd6da 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -94,8 +94,11 @@ def test_url_normalisation(): # An explicit load.php must survive: older portals only serve that. "http://a.example/stalker_portal/server/load.php": "http://a.example/stalker_portal/server/load.php", - # Any other .php is swapped for portal.php in the same directory. - "http://a.example/c/other.php": "http://a.example/c/portal.php", + # The one departure from stalkerhek, which swaps any other .php for + # portal.php in the same directory. Panels under a path of their own + # exist, and that path is the address the provider handed out; + # endpoint_candidates() probes the standard siblings after it. + "http://a.example/c/other.php": "http://a.example/c/other.php", # A missing scheme is filled in rather than rejected. "somedomain.com:8080/c/": "http://somedomain.com:8080/c/portal.php", # Ports and deep paths must be preserved. @@ -268,6 +271,20 @@ def test_expiry_is_read_from_the_field_resellers_use(): assert s.parse_expiry(None) is None +def test_expiry_is_also_read_from_the_field_ministra_uses(): + """account_info holds a Unix timestamp, and it is free: login() read it. + + It is also where a portal writes 'never', as 0 or -1. Read as a date, an + unlimited account would be reported as having run out in 1970 -- which is + the one wrong answer worse than no answer at all. + """ + assert s.parse_expiry(1785000000).year == 2026 + # Portals send the same value in milliseconds, and mean the same date. + assert s.parse_expiry("1785000000000") == s.parse_expiry("1785000000") + for unlimited in (0, "0", -1, "-1"): + assert s.parse_expiry(unlimited) is None, unlimited + + def test_the_default_arguments_let_dispatcharr_fail_over(): """ffmpeg must not reconnect on its own: it retries an expired portal link while staying alive, so Dispatcharr sees no failure and never switches to @@ -340,6 +357,111 @@ def test_config_survives_redis_roundtrip(): assert s.PortalConfig.from_dict(portal.to_dict()) == portal +# -- redaction: what the settings panel is allowed to see --------------------- + +STORED = ( + "Salon | http://a.example/c/ | 00:1A:79:00:00:01 | username=joe password=hunter2\n" + "http://b.example/c/ | 00:1A:79:00:00:02 | epg=1 model=MAG270 serial=98765\n" + "# Suspendu | http://c.example/c/ | 00:1A:79:00:00:03\n" +) + + +def test_no_credential_survives_the_redaction(): + """The whole point: nothing in the box lets anyone use the subscription.""" + masked = s.mask_portals(STORED) + for secret in ("00:1A:79:00:00:01", "00:1A:79:00:00:02", "00:1A:79:00:00:03", + "joe", "hunter2", "98765"): + assert secret not in masked, f"{secret} is still readable" + + +def test_what_is_not_a_credential_stays_readable(): + """A box of nothing but bullets is one nobody can recognise their own + portals in, so everything that only tunes behaviour is left alone.""" + masked = s.mask_portals(STORED) + for kept in ("Salon", "http://a.example/c/", "epg=1", "model=MAG270", "# "): + assert kept in masked, f"{kept} should not have been hidden" + + +def test_the_redaction_means_exactly_the_same_thing_once_undone(): + """A round trip has to be invisible to everything downstream, comments and + all -- a suspended line holds a real MAC and is how a portal is paused.""" + restored = s.unmask_portals(s.mask_portals(STORED), STORED) + before, _ = s.parse_portals(STORED) + after, errors = s.parse_portals(restored) + assert not errors + assert [p.to_dict() for p in before] == [p.to_dict() for p in after] + # Not compared as text: a redacted line writes its derived name out. What + # has to survive is what the line means, and that a paused portal is still + # paused rather than quietly back in the line-up. + assert "# Suspendu | http://c.example/c/ | 00:1A:79:00:00:03" in restored + assert [p.slug for p in after] == ["salon", "b"] + + +def test_a_line_typed_out_in_full_is_taken_as_typed(): + """Pasting the list back in from a password manager has to keep working.""" + added = s.mask_portals(STORED) + "New | http://d.example/c/ | 00:1A:79:00:00:04\n" + restored = s.unmask_portals(added, STORED) + assert "New | http://d.example/c/ | 00:1A:79:00:00:04" in restored + assert not s.is_masked(restored) + + +def test_a_renamed_or_moved_portal_keeps_its_own_credentials(): + """Renaming a portal and repointing it are both ordinary edits, and either + one changes the half of the line the other is recognised by. So both are + tried: the slug first, the URL after it.""" + masked = s.mask_portals(STORED) + + renamed = s.unmask_portals(masked.replace("Salon |", "Sejour |"), STORED) + assert "Sejour | http://a.example/c/ | 00:1A:79:00:00:01" in renamed + assert "password=hunter2" in renamed + + moved = s.unmask_portals(masked.replace("http://a.example/c/", "http://z.example/c/"), + STORED) + assert "Salon | http://z.example/c/ | 00:1A:79:00:00:01" in moved + + +def test_a_line_nothing_recognises_stays_hidden_rather_than_emptied(): + """Renaming *and* moving a line in one edit leaves nothing to pair it up + by. Leaving the token standing is what lets the plugin name the line and + ask for it again; an empty MAC would authenticate as nobody and say so in + a message about the portal instead of about the edit.""" + masked = s.mask_portals(STORED) + orphan = masked.replace("Salon | http://a.example/c/", "Sejour | http://z.example/c/") + restored = s.unmask_portals(orphan, STORED) + assert s.is_masked(restored) + assert "00:1A:79:00:00:02" in restored, "the other lines still come back" + + +def test_deleting_a_hidden_line_still_deletes_the_portal(): + """The editing model has to survive the redaction: a line is still a portal + and removing it still removes one.""" + masked = s.mask_portals(STORED) + kept = "\n".join(masked.splitlines()[1:]) + "\n" + portals, errors = s.parse_portals(s.unmask_portals(kept, STORED)) + assert not errors + assert [p.slug for p in portals] == ["b"] + + +def test_the_name_is_written_out_so_the_mac_field_stays_findable(): + """split_portal_line() decides which field is which by where the MAC sits. + Redact the MAC on a line that never named its portal and the fields shift + by one, so the redaction writes the derived name rather than lose them.""" + masked = s.mask_portals("http://b.example/c/ | 00:1A:79:00:00:02 | epg=1\n") + assert masked == f"b | http://b.example/c/ | {s.MASK} | epg=1\n" + + +def test_an_unparseable_line_is_left_where_the_user_can_see_it(): + masked = s.mask_portals("this is not a portal line\n") + assert masked == "this is not a portal line\n" + + +def test_the_token_is_not_something_anyone_types_by_accident(): + """A password of four asterisks is plausible; four bullets are not.""" + assert s.MASK == "\u2022" * 4 + assert not s.is_masked(STORED) + assert s.is_masked(s.mask_portals(STORED)) + + if __name__ == "__main__": failures = 0 for name, fn in sorted(globals().items()): diff --git a/tests/test_fallback.py b/tests/test_fallback.py index e2db3ad..6726c24 100644 --- a/tests/test_fallback.py +++ b/tests/test_fallback.py @@ -134,6 +134,36 @@ def test_a_missing_user_agent_is_not_the_literal_placeholder(): assert "{userAgent}" not in " ".join(cmd) +def test_the_session_never_leaves_the_portal_that_issued_it(): + """A create_link URL frequently points somewhere that is nobody's business. + + It carries its own token in the query, so it needs nothing from us; sending + the MAC and the session token to it would hand a third party everything + required to use the subscription. + """ + cfg = s.PortalConfig(slug="t", name="T", url="http://p.example/c/portal.php", + mac="00:1A:79:AA:BB:CC") + + own = s.stream_headers(cfg, "http://p.example:8080/live/1.ts", "TOK") + # A different port is still the portal: panels serve the stream from :8080 + # next to the portal on :80, and those are the ones gated on the cookie. + assert "Cookie" in own and own["Authorization"] == "Bearer TOK" + + for foreign in ("http://cdn.example/live/1.ts", + "https://other.example/live/1.ts", + "udp://239.0.0.1:1234"): + headers = s.stream_headers(cfg, foreign, "TOK") + assert "Cookie" not in headers, foreign + assert "Authorization" not in headers, foreign + + # An https portal answering with an http stream is a downgrade, and the + # session must not travel in clear because the portal said so. + secure = s.PortalConfig(slug="t", name="T", url="https://p.example/c/portal.php", + mac="00:1A:79:AA:BB:CC") + assert "Cookie" not in s.stream_headers(secure, "http://p.example/live/1.ts", "TOK") + assert "Cookie" in s.stream_headers(secure, "https://p.example/live/1.ts", "TOK") + + if __name__ == "__main__": failures = 0 for name, fn in sorted(globals().items()): diff --git a/tests/test_listing.py b/tests/test_listing.py index 9e9926b..cc82005 100644 --- a/tests/test_listing.py +++ b/tests/test_listing.py @@ -351,6 +351,25 @@ def test_reading_the_channel_out_of_a_command(): assert s.stream_id(cmd) == expected, cmd +def test_a_command_reaches_the_portal_decoded_exactly_once(): + """PHP form-decodes the query once, so what is already escaped must stay so. + + quote(cmd, safe="") escaped the percent sign too, so '%3A' went out as + '%253A' and the portal read back a string no real box would have sent. + """ + assert s.encode_cmd("ffmpeg http://h/a%3Ab") == "ffmpeg%20http://h/a%3Ab" + # The characters a URL keeps raw in a query stay raw, so the bytes we emit + # are the bytes a box emits. + assert s.encode_cmd("http://h/p?a=1&") .startswith("http://h/p?a=1") + # ...except the ones that would restructure our own request around it. + for smuggled in ("&", "#", ";"): + assert smuggled not in s.encode_cmd(f"http://h/1{smuggled}stream=9"), smuggled + # The ordinary case, unchanged: a marker has nothing in it to encode. + assert s.encode_cmd("ffmpeg http://localhost/ch/1_") == ( + "ffmpeg%20http://localhost/ch/1_" + ) + + # -- paging -------------------------------------------------------------- @@ -384,9 +403,124 @@ def page(ids, **extra): return js +def test_the_paged_request_says_the_same_thing_several_ways(): + """Belt and braces, as with the identity in _common_params. + + A Ministra portal reads 'genre' and ignores the rest. The clones do not all + read the same one, and every parameter here is one that some other client + found a portal wanting -- sending them all costs a longer query string. + """ + p = portal() + seen = [] + p._get_json = lambda q, with_auth=True: seen.append(q) or {"js": {"data": []}} + p.get_ordered_list(3) + for expected in ("genre=*", "category=*", "fav=0", "force_ch_link_check=", + "hd=0", "sortby=number", "p=3"): + assert expected in seen[0], f"{expected} missing from {seen[0]}" + + REFUSED = s.PortalError("portal returned an empty channel list (check the MAC address)") +# -- the channels that need no link minted for them ----------------------- + + +def test_silence_is_not_a_no(): + """Every portal synced before these flags were read carries neither. + + Taking that as "needs no link" would move an entire installation onto the + static path at once, on nothing but the absence of evidence. + """ + assert s.Portal._channel_from_row(row()).needs_link is None + assert s.portal_flag(None) is None + assert s.portal_flag("") is None + + +def test_the_flags_are_read_in_every_shape_a_portal_writes_them(): + for value in ("1", 1, True, "true", "yes"): + assert s.portal_flag(value) is True, value + for value in ("0", 0, False, "false", "no", "off"): + assert s.portal_flag(value) is False, value + + # Either flag set means the portal mints the link; both clear means it does + # not; and both must be absent before the row counts as having said nothing. + def needs(**flags): + return s.Portal._channel_from_row(row(**flags)).needs_link + + assert needs(use_http_tmp_link="1", use_load_balancing="0") is True + assert needs(use_http_tmp_link="0", use_load_balancing="1") is True + assert needs(use_http_tmp_link="0", use_load_balancing="0") is False + assert needs(use_load_balancing="0") is False + + +def test_what_may_be_played_without_asking_the_portal(): + """The table. Every entry that is False falls back on today's behaviour, + which is why none of these guards can break a portal that works now.""" + static = "http://prov.example/USER/PASS/1225691" + assert s.plays_without_create_link(static, False) + # Multicast is a perfectly good static address, and ffmpeg opens it. + assert s.plays_without_create_link("udp://239.0.0.1:1234", False) + # 'ffmpeg' is a prefix word, not part of the address. + assert s.plays_without_create_link("ffmpeg " + static, False) + + for cmd, why in ( + ("http://localhost/ch/1_", "the portal talking to itself"), + ("http://127.0.0.1/ch/1_", "the whole of 127.0.0.0/8 is loopback"), + ("http://127.5.5.5/ch/1_", "still loopback"), + ("http://[::1]/ch/1_", "and in IPv6"), + ("http://[::ffff:127.0.0.1]/ch/1_", "and mapped back into IPv6"), + ("http://0.0.0.0/ch/1_", "the unspecified address means the same"), + ("http://stream.localhost/ch/1_", "RFC 6761 reserves the whole suffix"), + ("http:///ch/1", "an address with no host at all"), + ("ffrt4://ch/live/1", "a portal's own pseudo-URL plays as nothing"), + ("auto /media/file.mpg", "not an address until create_link makes one"), + (static + "?play_token=abc", "a token in the query is a link that expires"), + ): + assert not s.plays_without_create_link(cmd, False), why + + # And the flags still decide first. + assert not s.plays_without_create_link(static, True) + assert not s.plays_without_create_link(static, None) + + +def test_the_published_set_is_the_commands_and_nothing_else(): + channels = [ + s.Portal._channel_from_row(row( + id="1", cmd="ffmpeg http://localhost/ch/1_", + use_http_tmp_link="0", use_load_balancing="0")), + s.Portal._channel_from_row(row( + id="", cmd="http://prov.example/USER/PASS/22", + use_http_tmp_link="0", use_load_balancing="0")), + s.Portal._channel_from_row(row( + id="", cmd="http://prov.example/live/33", use_http_tmp_link="1")), + ] + assert s.static_commands(channels) == ["http://prov.example/USER/PASS/22"] + # A portal that marks nothing static publishes an empty list, not nothing. + assert s.static_commands([channels[0]]) == [] + + +def test_a_rewritten_command_is_never_static(): + """canonical_cmd turns a resolved link into a loopback marker, and a marker + is only ever an instruction to the portal -- whatever the flags said about + the row it came from.""" + channel = s.Portal._channel_from_row(row( + id="553690", cmd="http://prov.example/live.php?stream=553690", + use_http_tmp_link="0", use_load_balancing="0")) + assert channel.cmd_rewritten + assert channel.needs_link is False + assert not s.plays_without_create_link(channel.cmd, channel.needs_link) + + +def test_a_channel_repeated_in_one_response_is_still_one_channel(): + """Dispatcharr hashes a stream partly on its URL, so a duplicate row is a + second stream for one channel rather than a harmless extra line.""" + p = portal() + p._get_json = lambda q, with_auth=True: {"js": {"data": [ + row(id="1"), row(id="2", name="Two"), row(id="1"), + ]}} + assert [c.channel_id for c in p.get_all_channels()] == ["1", "2"] + + def test_a_portal_that_answers_in_one_request_is_never_paged(): p = scripted({"js": {"data": [row()]}}, {}) assert len(p.list_channels()) == 1 diff --git a/tests/test_manifest.py b/tests/test_manifest.py index 45480f7..d55e1cc 100644 --- a/tests/test_manifest.py +++ b/tests/test_manifest.py @@ -113,6 +113,11 @@ def publish(line): def run(p, action, settings): + # The panel saves the box before running anything -- PluginCard.jsx says so + # in as many words -- so the stored row and the context always hold the same + # list. Writing it here keeps the two in step; a test handing run() a new + # list is simulating a user who has just typed it into the textarea. + p._write_settings(dict(settings)) return p.run(action, {}, {"settings": dict(settings), "logger": NullLogger()}) @@ -223,6 +228,52 @@ def test_settings_the_code_reads_are_declared(): assert key in ids, f"the code reads '{key}' but the panel never renders it" +def test_settings_the_panel_no_longer_has_a_field_for_are_dropped(): + """Taking a field out of the manifest stops the panel rendering it and + nothing else. The Add-portal form went in 0.4.0 and its values stayed in + the settings row -- a MAC, a password and a URL that Dispatcharr went on + serving to every account on the install, for a portal long since deleted. + """ + reset() + p, store, _ = make_plugin({ + "portals": PORTALS, + "new_url": "http://gone.example:8080/c/", + "new_mac": "00:00:00:00:00:00", + "new_password": "hunter2", + "sync_interval_hours": 12, + }) + try: + with stubbed(): + run(p, "apply_profile", store["settings"]) + finally: + store["restore"]() + + for dead in ("new_url", "new_mac", "new_password", "sync_interval_hours"): + assert dead not in store["settings"], f"{dead} is still being served" + assert "portals" in store["settings"], "the declared ones must stay" + assert store["settings"]["status"], "and so must what the action reported" + + +def test_a_migration_still_gets_the_keys_it_reads(): + """The prune must not run before _migrate_legacy_globals, whose whole input + is settings the manifest stopped declaring three versions ago.""" + reset() + p, store, _ = make_plugin({ + "portals": PORTALS, + "stb_device_id": "a" * 64, + }) + try: + with stubbed(): + run(p, "apply_profile", store["settings"]) + finally: + store["restore"]() + + assert "stb_device_id" not in store["settings"], "the legacy key is gone" + assert "device_id=" + "a" * 64 in registry.load_registry(), ( + "and was folded onto the portal line before being dropped" + ) + + # -- what run() does around a handler ----------------------------------------- def test_an_action_records_what_it_did(): diff --git a/tests/test_mock_portal.py b/tests/test_mock_portal.py index 20b8d69..2a899b0 100644 --- a/tests/test_mock_portal.py +++ b/tests/test_mock_portal.py @@ -209,12 +209,17 @@ def main(): # ffmpeg argv construction, as resolver.py builds it. import resolver - argv = resolver.build_ffmpeg_command(cfg, link) + argv = resolver.build_ffmpeg_command(cfg, link, "TOKEN") print("ffmpeg argv:", argv) assert argv[0] == "ffmpeg" assert link in argv assert s.USER_AGENT in argv assert "pipe:1" in argv + # The mock portal answers with a link on another host, which is the shape + # a real create_link takes most of the time -- so the session stays here. + blob = argv[argv.index("-headers") + 1] + assert "mac=" not in blob and "Bearer" not in blob, blob + assert "X-User-Agent: Model: " in blob, blob server.shutdown() print("\nALL CHECKS PASSED") diff --git a/tests/test_registry.py b/tests/test_registry.py index b6c6b98..59665d4 100644 --- a/tests/test_registry.py +++ b/tests/test_registry.py @@ -93,7 +93,9 @@ def test_absent_key_means_clobbered_and_is_restored(): result = p._reconcile_registry(merged, NullLogger()) assert result["portals"] == PORTAL, "the clobbered list must come back" - assert store["settings"]["portals"] == PORTAL, "and be written back to the DB" + assert store["settings"]["portals"] == s.mask_portals(PORTAL), ( + "and be written back to the DB, redacted -- the row is what the API serves" + ) portals, errors = s.parse_portals(result["portals"]) assert not errors and [x.name for x in portals] == ["livingroom"] @@ -247,6 +249,134 @@ def test_a_marker_left_over_from_an_outside_edit_is_ignored(): assert not registry.is_pending(), "the marker no longer describes the file" +# -- redaction: the row the API serves is not the list the plugin works from -- + +def test_the_settings_row_never_holds_a_credential(): + """Dispatcharr serves the settings row to every account on the install, and + the panel paints it into a textarea. Neither is a place for a MAC.""" + reset() + p, store = make_plugin({}) + p._save_settings({"portals": PORTAL}) + + assert "00:1A:79:AA:BB:CC" not in store["settings"]["portals"] + assert s.is_masked(store["settings"]["portals"]) + assert registry.load_registry() == PORTAL, "the file keeps the real thing" + + +def test_the_panel_sending_the_redaction_back_changes_nothing(): + """The panel can only return what it was shown. That must read as 'no + change', not as the user having replaced every MAC with bullets.""" + reset() + registry.save_registry(PORTAL) + masked = s.mask_portals(PORTAL) + + p, _ = make_plugin({"portals": masked}) + result = p._reconcile_registry({"portals": masked}, NullLogger()) + + assert result["portals"] == PORTAL, "handlers must be given the real list" + assert registry.load_registry() == PORTAL, "and the file must be untouched" + assert not registry.is_pending(), "the panel is in step with the file" + + +def test_reopening_the_panel_does_not_rewrite_the_file(): + """A line that never named its portal comes back from the redaction with + the derived name written out, because that is what makes the MAC's position + findable. That is the rendering's doing, not an edit, and reading it as one + would rewrite the user's file on every click.""" + reset() + unnamed = "http://portal.example/c/ | 00:1A:79:AA:BB:CC | epg=1\n" + registry.save_registry(unnamed) + shown = s.mask_portals(unnamed) + assert shown.startswith("portal | "), "the derived name is written out" + + p, _ = make_plugin({"portals": shown}) + result = p._reconcile_registry({"portals": shown}, NullLogger()) + + assert result["portals"] == unnamed + assert registry.load_registry() == unnamed, "the file must be left alone" + assert not registry.is_pending() + + +def test_an_edit_made_through_the_redaction_keeps_the_hidden_half(): + """Changing a portal's URL while its MAC is hidden is the ordinary case, + and the MAC the user cannot see must survive it.""" + reset() + registry.save_registry(PORTAL) + + edited = s.mask_portals(PORTAL).replace("http://portal.example/c/", + "http://moved.example/c/") + p, store = make_plugin({"portals": edited}) + result = p._reconcile_registry({"portals": edited}, NullLogger()) + + assert "http://moved.example/c/" in result["portals"], "the edit must stick" + assert "00:1A:79:AA:BB:CC" in result["portals"], "and the MAC must come back" + assert registry.load_registry() == result["portals"] + assert s.is_masked(store["settings"]["portals"]), "the row stays redacted" + + +def test_an_upgrade_hides_credentials_the_row_already_holds(): + """Installs coming from an earlier version have their MAC in the row. A + panel that only ever presses Sync never edits the box, so nothing else + would ever rewrite it.""" + reset() + registry.save_registry(PORTAL) + + p, store = make_plugin({"portals": PORTAL}) # as an older version left it + result = p._reconcile_registry({"portals": PORTAL}, NullLogger()) + + assert result["portals"] == PORTAL + assert s.is_masked(store["settings"]["portals"]), "the row must be redacted now" + assert registry.load_registry() == PORTAL + + +def test_nothing_is_hidden_until_the_file_is_known_to_hold_it(): + """Redacting is only safe once there are two copies. A registry that could + not be written leaves the row as the only one there is, and hiding the only + copy of a credential is how a configuration gets lost.""" + reset() + original = plugin_mod.save_registry + plugin_mod.save_registry = lambda text, pending=False: False + try: + p, store = make_plugin({"portals": PORTAL}) + result = p._reconcile_registry({"portals": PORTAL}, NullLogger()) + assert result["portals"] == PORTAL + assert store["settings"]["portals"] == PORTAL, "still the only copy" + finally: + plugin_mod.save_registry = original + + +def test_a_redaction_is_never_written_over_the_real_list(): + """The panel holds bullets and the file that explains them is gone -- a + recreated volume, a partial restore. Adopting what the panel sends would + make the loss permanent by writing the bullets into the file.""" + reset() + masked = s.mask_portals(PORTAL) + + p, store = make_plugin({"portals": masked}) + result = p._reconcile_registry({"portals": masked}, NullLogger()) + + assert s.is_masked(result["portals"]), "nothing can fill these back in" + assert registry.load_registry() is None, "and nothing may be written" + assert store["settings"]["portals"] == masked, "the row is left as it was" + + +def test_a_line_that_could_not_be_filled_back_in_is_named_and_refused(): + """Parsing on would report a MAC address made of bullets, which explains + nothing. The line itself is quoted back instead, because retyping it is + the only thing that fixes this.""" + reset() + masked = s.mask_portals(PORTAL) + p, _ = make_plugin({}) + try: + p._portals({"portals": masked}) + except plugin_mod.PortalError as exc: # plugin.py holds its own import + assert "livingroom" in str(exc), exc + assert registry.REGISTRY_PATH in str(exc), exc + assert "retype" in str(exc), exc + else: + raise AssertionError("a redacted list must not be parsed") + + if __name__ == "__main__": failures = 0 try: diff --git a/tests/test_state.py b/tests/test_state.py index 4aea2aa..fc6b355 100644 --- a/tests/test_state.py +++ b/tests/test_state.py @@ -167,6 +167,106 @@ def test_removing_a_portal_removes_its_mirror(): assert s.load_portal(CFG.slug, client) is None +STATIC_CMD = "http://prov.example/USER/PASS/1225691" + + +def test_the_static_commands_survive_a_restart_too(): + """A tune that read nothing would ask the portal, which is safe but is the + round trip this exists to avoid -- and on the providers concerned it is the + request that answers with a path returning 401.""" + reset() + client = FakeRedis() + s.save_static_cmds(CFG.slug, [STATIC_CMD], client) + assert os.path.exists(s._state_path(f"static-{CFG.slug}")) + + client.store.clear() + assert s.load_static_cmds(CFG.slug, client) == {STATIC_CMD} + assert s._static_key(CFG.slug) in client.store, "and it goes back in Redis" + + +def test_not_knowing_means_asking_the_portal(): + """The one safe reading of every failure here, and the behaviour that was + there before any of this existed.""" + reset() + assert s.load_static_cmds("never-registered", FakeRedis()) == set() + assert s.load_static_cmds(CFG.slug, FakeRedis(broken=True)) == set() + + client = FakeRedis() + s.save_static_cmds(CFG.slug, [STATIC_CMD], client) + client.store[s._static_key(CFG.slug)] = "{not json" + assert s.load_static_cmds(CFG.slug, client) == {STATIC_CMD}, "the mirror wins" + + +def test_a_portal_that_stops_marking_channels_static_is_believed(): + """Inheriting the last list that said otherwise would keep playing a + command the portal has since started minting links for.""" + reset() + client = FakeRedis() + s.save_static_cmds(CFG.slug, [STATIC_CMD], client) + s.save_static_cmds(CFG.slug, [], client) + assert s.load_static_cmds(CFG.slug, client) == set() + + +def test_removing_a_portal_removes_its_static_commands(): + reset() + client = FakeRedis() + s.save_portal(CFG, client) + s.save_static_cmds(CFG.slug, [STATIC_CMD], client) + s.forget_portal(CFG.slug, client) + + assert not os.path.exists(s._state_path(f"static-{CFG.slug}")) + assert s.load_static_cmds(CFG.slug, client) == set() + + +def test_a_static_channel_never_reaches_the_portal(): + """The whole point: no handshake, no create_link, no connection slot.""" + reset() + client = FakeRedis() + s.save_portal(CFG, client) + s.save_static_cmds(CFG.slug, [STATIC_CMD], client) + + class Unreachable(s.Portal): + def login(self): + raise AssertionError("a static channel must not contact the portal") + + def create_link(self, cmd): + raise AssertionError("a static channel must not contact the portal") + + original_portal, original_redis = s.Portal, s.get_redis + s.Portal, s.get_redis = Unreachable, lambda: client + try: + link, cfg, _ = resolver.resolve(CFG.slug, STATIC_CMD) + finally: + s.Portal, s.get_redis = original_portal, original_redis + + assert link == STATIC_CMD + assert cfg.slug == CFG.slug + + +def test_a_channel_the_portal_never_marked_still_goes_through_create_link(): + reset() + client = FakeRedis() + s.save_portal(CFG, client) + s.save_static_cmds(CFG.slug, [STATIC_CMD], client) + + class Answering(s.Portal): + def login(self): + self.token = "fresh" + return self.token + + def create_link(self, cmd): + return "http://prov.example/live/9.ts?token=fresh" + + original_portal, original_redis = s.Portal, s.get_redis + s.Portal, s.get_redis = Answering, lambda: client + try: + link, _, token = resolver.resolve(CFG.slug, "ffmpeg http://localhost/ch/9_") + finally: + s.Portal, s.get_redis = original_portal, original_redis + + assert link.endswith("token=fresh") and token == "fresh" + + def test_the_fallback_command_survives_too(): reset() client = FakeRedis() diff --git a/tests/test_transport.py b/tests/test_transport.py index 59f5fe5..fe38226 100644 --- a/tests/test_transport.py +++ b/tests/test_transport.py @@ -211,6 +211,33 @@ def test_prose_instead_of_json_is_not_a_reason_to_ask_again(): restore() +def test_the_counter_the_stock_server_appends_does_not_hide_the_refusal(): + """'Authorization failed. 75' is what Ministra actually sends. + + The number is a debug counter. Read as prose, this arrived typed as an + endpoint failure -- which the resolver never re-authenticates on, so the + one answer the cached-token path exists to survive failed the tune instead. + """ + p = portal([FakeResponse(200, text="Authorization failed. 75")]) + try: + p._get_json("action=create_link") + except s.PortalAuthError as exc: + assert "no longer authorised" in str(exc), exc + else: + raise AssertionError("the stale-token refusal must reach the resolver") + + +def test_a_refusal_in_json_never_looks_like_an_empty_portal(): + """Nothing here is malformed, so only the wording says the session is gone.""" + p = portal([FakeResponse(200, payload={"js": {"error": "Invalid token"}})]) + try: + p._get_json("action=get_genres") + except s.PortalAuthError as exc: + assert "Invalid token" in str(exc), exc + else: + raise AssertionError("a refused session must not read as a portal with no genres") + + def test_the_sync_asks_for_retries_and_the_test_action_does_not(): """The one asymmetry that matters, pinned so a refactor keeps it.