From ff80478e99f95bf8ede8037f01e8f95075d7d90a Mon Sep 17 00:00:00 2001 From: PiloUnk <198624632+PiloUnk@users.noreply.github.com> Date: Mon, 24 Aug 2026 22:34:12 +0200 Subject: [PATCH 01/10] Read every refusal a portal sends, not the one exact string The stale-token answer was matched by equality against "authorization failed.", and the stock Ministra server appends a debug counter to it. So "Authorization failed. 75" -- what it actually sends -- was not recognised, and neither were the other two refusals in the same family: "Access denied." for a blocked account, "Unauthorized request." when the mac cookie never arrived. All three arrived typed as PortalEndpointError, which has two wrong consequences. The resolver re-authenticates on PortalAuthError alone, so the one answer the cached-token path exists to survive failed the tune instead; and the handshake retries the portal's other path on an endpoint failure, so a portal that was refusing us was asked a second time -- the request the comment there already calls the one that gets a MAC noticed. The bodies are matched against the whole body rather than searched for inside it. They are bare phrases, so a proxy page reading "Access denied" -- 38 characters, under any length a cap would catch -- would otherwise be read as the portal speaking, and send the resolver re-authenticating against a host that never answered at all. Panels that are not Ministra refuse inside the envelope rather than in place of it, and those were not seen anywhere: get_genres answered {'js': {'error': 'Invalid token'}} counted as a portal with no genres, so Test portals reported it as authenticated with 0 groups, and get_all_channels on the same reply told the user to check a MAC address that was never the problem. handshake() had the same hole from the other side -- a reply with no token left the cached one in place and returned it as if it had succeeded. Two vocabularies, because the inputs are not comparable. A raw body is an arbitrary document and gets the three anchored phrases; 'error' is a field a panel filled in deliberately -- a reply that worked has none at all -- so 'Invalid token' and a bare 'unauthorized' are worth reading there. 'msg' gets the exact bodies only, and only when the reply carries no 'status': a reply that has one has a verdict login() reads for itself, and a status-2 portal saying "Authorization required" is asking for a password, not refusing us. Raising here would skip the do_auth it was asking for. Co-Authored-By: Claude Opus 5 --- stalker_api.py | 110 ++++++++++++++++++++++++++++++++++++---- tests/test_auth.py | 51 +++++++++++++++++++ tests/test_transport.py | 27 ++++++++++ 3 files changed, 179 insertions(+), 9 deletions(-) diff --git a/stalker_api.py b/stalker_api.py index abaa5b5..c2138b5 100644 --- a/stalker_api.py +++ b/stalker_api.py @@ -66,10 +66,46 @@ STB_HW_VERSION = "1.7-BD-00" STB_NUM_BANKS = 1 -# What a portal answers with, in plain text and with no JSON around it, once -# the token it was given is no longer good. Matched exactly because it is a -# fixed string in Ministra rather than something a reseller writes. -AUTH_FAILED_BODY = "authorization failed." +# The refusals Ministra answers with in plain text, HTTP 200 attached and no +# JSON around them, and what each of them actually means. Matched against the +# whole body rather than searched for inside it: these are bare phrases, so a +# proxy or a WAF answering 'Access denied' -- 38 +# characters, under any length a cap would catch -- would otherwise be read as +# the portal speaking, and send the resolver off to re-authenticate against +# something that never answered at all. +# +# Only the first carries a trailing number, and it is a debug counter rather +# than anything to read. Matching without it is most of the point of this +# table: 'Authorization failed. 75' is what the stock server actually sends, +# and the exact-string comparison this replaces did not recognise it -- so the +# one refusal the resolver exists to recover from arrived typed as an endpoint +# failure, which is never re-authenticated on and costs a second request to the +# other path of a portal that is already refusing us. +AUTH_REFUSALS = ( + ( + re.compile(r"^authorization\s+failed[.!]*(?:\s+\d+)?$", re.I), + "portal says the session is no longer authorised", + ), + ( + re.compile(r"^access\s+denied[.!]*$", re.I), + "portal says this account is denied access", + ), + ( + re.compile(r"^unauthorized\s+request[.!]*$", re.I), + "portal did not receive the MAC address it authorises on", + ), +) + +# Wording accepted inside a JSON envelope's 'error' field, beyond the three +# above. Wider on purpose, and applied in one narrow place: a panel that fills +# in 'error' has said something went wrong deliberately -- a reply that worked +# carries no 'error' at all -- so 'Invalid token' and a bare 'unauthorized' are +# worth reading there, where the same breadth against an arbitrary body would +# match half the error pages on the internet. +ENVELOPE_REFUSAL = re.compile( + r"authorization|access\s+denied|unauthorized|auth\s+failed|invalid\s+token", + re.I, +) # Answers worth asking again for. Everything else is the portal having made up # its mind: a 404 is not going to become a 200, and a 403 is the subject of @@ -232,6 +268,49 @@ class PortalAuthError(PortalError): """ +def auth_refusal(body: Any) -> str: + """What a plain-text answer says about the session, or "" if it says nothing.""" + text = str(body or "").strip() + for pattern, message in AUTH_REFUSALS: + if pattern.match(text): + return message + return "" + + +def envelope_refusal(payload: Any) -> str: + """The same, for portals that refuse inside the envelope rather than instead of it. + + Not Ministra's behaviour, and common in what this plugin actually meets. + Every one of these used to arrive as an ordinary reply: ``get_genres`` + answered ``{'js': {'error': 'Invalid token'}}`` counted as a portal with no + genres, and 'Test portals' reported it as authenticated with 0 groups. + + ``error`` is read with the wide vocabulary. ``msg`` only with the three + exact bodies, and only when nothing else in the reply has already given a + verdict: a reply carrying a ``status`` is one :meth:`Portal.login` reads + for itself, including the sentence beside it -- which is the provider's own + wording, and better than anything this could substitute for it. + """ + js = payload.get("js") if isinstance(payload, dict) else None + # Some panels put the phrase straight in 'js' rather than in a field of it. + if isinstance(js, str): + return auth_refusal(js) + if not isinstance(js, dict): + return "" + + if js.get("status") is None: + exact = auth_refusal(js.get("msg")) + if exact: + return exact + + error = str(js.get("error") or "").strip() + # A field longer than a sentence is a document somebody stuffed in there, + # not a refusal worth quoting back at the user. + if error and len(error) <= 200 and ENVELOPE_REFUSAL.search(error): + return f"portal refused the session: {error}" + return "" + + # --------------------------------------------------------------------------- # Configuration # --------------------------------------------------------------------------- @@ -1384,18 +1463,31 @@ def _get_json(self, query: str, with_auth: bool = True) -> Any: raise PortalError(message) try: - return resp.json() + payload = resp.json() except ValueError: - snippet = (resp.text or "").strip()[:300] + # Classified on the whole body and displayed truncated: the + # patterns are anchored, so cutting first could only ever hide a + # refusal, never invent one. + body = (resp.text or "").strip() # A dead session is answered in plain text with a 200 attached, so # it arrives here rather than as an HTTP error. Saying so is what # lets the resolver re-authenticate instead of failing the tune. - if snippet.lower() == AUTH_FAILED_BODY: - raise PortalAuthError("portal says the session is no longer authorised") + refusal = auth_refusal(body) + if refusal: + raise PortalAuthError(refusal) # Anything else that is not JSON is an HTML error page, a login # form, or a landing page: something is listening, but it is not a # Stalker API, so the other endpoint is worth a try. - raise PortalEndpointError(f"portal returned non-JSON response: {snippet}") + raise PortalEndpointError( + f"portal returned non-JSON response: {body[:300]}" + ) + + # A refusal can also arrive as perfectly good JSON, and then it is not + # this reply that is unusable but the session behind it. + refusal = envelope_refusal(payload) + if refusal: + raise PortalAuthError(refusal) + return payload # -- authentication --------------------------------------------------- diff --git a/tests/test_auth.py b/tests/test_auth.py index 1c41b3a..19ad45b 100644 --- a/tests/test_auth.py +++ b/tests/test_auth.py @@ -215,6 +215,57 @@ def json(self): raise AssertionError("'Authorization failed.' must be typed as an auth error") +def test_every_shape_a_refusal_arrives_in(): + """The table, so a shape met next is added here rather than argued about.""" + for body in ("Authorization failed.", + # The counter the stock server appends. The exact comparison + # this replaces did not recognise it, and it is the refusal + # the resolver exists to recover from. + "Authorization failed. 75", + "authorization failed", + "Access denied.", + " Unauthorized request.\n"): + assert s.auth_refusal(body), body + + for body in ("", + # A proxy or WAF page: 38 characters, so only matching the + # whole body keeps it out -- and it must stay out, or a host + # that never answered sends the resolver re-authenticating. + "Access denied", + "Access denied by policy", + "ffmpeg http://host/stream"): + assert not s.auth_refusal(body), body + + +def test_a_refusal_can_arrive_as_perfectly_good_json(): + """Panels that are not Ministra refuse inside the envelope, not instead of it. + + Every one of these used to read as an ordinary reply: get_genres answered + this way counted as a portal with no genres, and 'Test portals' reported it + as authenticated with 0 groups. + """ + assert s.envelope_refusal({"js": "Authorization failed."}) + assert s.envelope_refusal({"js": {"msg": "Access denied."}}) + # The panel's own wording is what gets quoted back. + assert "Invalid token" in s.envelope_refusal({"js": {"error": "Invalid token"}}) + + +def test_a_portal_asking_for_a_password_is_not_a_portal_refusing(): + """The one false positive that would cost a working install. + + 'msg' is where a status-2 reply writes the sentence login() reads to know + it should call do_auth. Reading it here would turn every portal that says + 'Authorization required' into a hard refusal and skip the step it asked + for; a status-1 'msg' is the provider's own wording, which login() quotes + better than a substitute could. + """ + assert not s.envelope_refusal({"js": {"status": 2, "msg": "Authorization required"}}) + assert not s.envelope_refusal({"js": {"status": 1, "msg": "Access denied."}}) + # A reply that worked carries no refusal at all, in any field. + assert not s.envelope_refusal({"js": {"data": [], "total_items": 0}}) + assert not s.envelope_refusal({"js": [{"id": "1", "title": "All"}]}) + + def test_create_link_expiry_is_typed_so_the_resolver_can_recover(): """A dead token is usually a hollow success, not a refusal. diff --git a/tests/test_transport.py b/tests/test_transport.py index 59f5fe5..fe38226 100644 --- a/tests/test_transport.py +++ b/tests/test_transport.py @@ -211,6 +211,33 @@ def test_prose_instead_of_json_is_not_a_reason_to_ask_again(): restore() +def test_the_counter_the_stock_server_appends_does_not_hide_the_refusal(): + """'Authorization failed. 75' is what Ministra actually sends. + + The number is a debug counter. Read as prose, this arrived typed as an + endpoint failure -- which the resolver never re-authenticates on, so the + one answer the cached-token path exists to survive failed the tune instead. + """ + p = portal([FakeResponse(200, text="Authorization failed. 75")]) + try: + p._get_json("action=create_link") + except s.PortalAuthError as exc: + assert "no longer authorised" in str(exc), exc + else: + raise AssertionError("the stale-token refusal must reach the resolver") + + +def test_a_refusal_in_json_never_looks_like_an_empty_portal(): + """Nothing here is malformed, so only the wording says the session is gone.""" + p = portal([FakeResponse(200, payload={"js": {"error": "Invalid token"}})]) + try: + p._get_json("action=get_genres") + except s.PortalAuthError as exc: + assert "Invalid token" in str(exc), exc + else: + raise AssertionError("a refused session must not read as a portal with no genres") + + def test_the_sync_asks_for_retries_and_the_test_action_does_not(): """The one asymmetry that matters, pinned so a refactor keeps it. From 18e54f6ebf2e36f4895ee541a1ebb266efe9a485 Mon Sep 17 00:00:00 2001 From: PiloUnk <198624632+PiloUnk@users.noreply.github.com> Date: Mon, 24 Aug 2026 22:40:21 +0200 Subject: [PATCH 02/10] Send what a real box sends, not the subset that happened to work Three clients now agree on more of the STB handshake than this one was sending, and the parts missing here are all read by the same thing: the optional access_filter.php a reseller installs in front of Ministra. The stock middleware ignores every one of them, which is why their absence has never shown up as a failure anyone could trace -- a portal that runs a filter simply refuses, and says nothing about which field made it. - 'prehash' on the handshake and the profile. iptvnator computes the SHA1 of the upper-case MAC, QiTV ships one constant for every install it has ever run; only the first says anything true about this box, and if a portal ever checks the value against its own record, a constant shared by every user of one client is the version that fails. - 'metrics', the JSON blob a portal stores and shows its operator, so a device list has a MAG in it rather than a blank row. It echoes the nonce the handshake handed back, which is the part a portal can actually check: it issued that value one request ago, and only something that read the answer can quote it. - 'client_type' and 'video_out', which both other clients send and this one never did. get_ordered_list gains 'category', 'force_ch_link_check' and 'hd' for the same reason _common_params sends the identity three ways: a Ministra portal reads 'genre' and ignores the rest, the clones do not all read the same one, and the cost of saying it several ways is a longer query string. Co-Authored-By: Claude Opus 5 --- stalker_api.py | 58 +++++++++++++++++++++++++++++++++++++++++-- tests/test_auth.py | 39 ++++++++++++++++++++++++++++- tests/test_listing.py | 16 ++++++++++++ 3 files changed, 110 insertions(+), 3 deletions(-) diff --git a/stalker_api.py b/stalker_api.py index c2138b5..8df933a 100644 --- a/stalker_api.py +++ b/stalker_api.py @@ -19,6 +19,7 @@ from __future__ import annotations +import hashlib import json import os import re @@ -569,6 +570,23 @@ def parse_expiry(value: Any) -> Optional[datetime]: return None +def prehash(mac: str) -> str: + """The SHA1 a client presents itself with, derived from the MAC it uses. + + Neither the stock middleware nor any portal met so far reads this. It is + there for the optional access_filter.php a reseller can install in front of + it, which is also the reason a box that sends nothing at all is the shape + some of those filters reject. + + What to send is written down nowhere. iptvnator computes the SHA1 of the + upper-case MAC; QiTV ships one constant for every install it has ever run. + Only the first says something true about this box, so it is the one copied + here -- and if a portal ever checks it against its own record, a constant + shared by every user of one client is the version that fails. + """ + return hashlib.sha1(mac.upper().encode("utf-8")).hexdigest().upper() + + def normalize_mac(mac: str) -> str: """Accept the shapes MAC addresses are quoted in, emit the one we use. @@ -1363,6 +1381,10 @@ def __init__( # Whether the portal said the token it handed back is already good for # more than the handshake. Reported straight back to it in get_profile. self.valid_token = False + # The nonce the handshake handed back, echoed in get_profile's metrics. + # A portal that issues one and never sees it again is being talked to + # by something that did not read its own handshake. + self.handshake_random = "" # What get_profile answered during login(), kept so nothing has to ask # twice: the expiry report and the blocked flag both read it. self.profile: Dict[str, Any] = {} @@ -1494,12 +1516,14 @@ def _get_json(self, query: str, with_auth: bool = True) -> Any: def handshake(self) -> str: """Reserve a token. The portal may hand back a different one.""" data = self._get_json( - f"type=stb&action=handshake&token={self.token}&JsHttpRequest=1-xml", + f"type=stb&action=handshake&token={self.token}" + f"&prehash={prehash(self.cfg.mac)}&JsHttpRequest=1-xml", with_auth=False, ) js = data.get("js") if isinstance(data, dict) else None if isinstance(js, dict) and js.get("token"): self.token = str(js["token"]) + self.handshake_random = str(js.get("random") or "") # 'not_valid' is the portal saying the token still has to be # earned. get_profile is told the same thing back, which is how it # knows whether it is being asked to validate or merely to report. @@ -1558,6 +1582,9 @@ def get_profile(self, auth_second_step: bool = False) -> Dict[str, Any]: f"&device_id={quote(self.cfg.device_id)}" f"&device_id2={quote(self.cfg.device_id2)}" f"&signature={quote(self.cfg.signature)}" + f"&client_type=STB&video_out=hdmi" + f"&metrics={quote(self._metrics())}" + f"&prehash={prehash(self.cfg.mac)}" f"¬_valid_token={0 if self.valid_token else 1}" f"&auth_second_step={1 if auth_second_step else 0}" ) @@ -1565,6 +1592,26 @@ def get_profile(self, auth_second_step: bool = False) -> Dict[str, Any]: js = data.get("js") if isinstance(data, dict) else None return js if isinstance(js, dict) else {} + def _metrics(self) -> str: + """The box describing itself, in the shape get_profile wants it. + + A JSON blob rather than parameters, which is Ministra's choice and not + ours. It is what the admin panel stores and shows the reseller, so a + portal whose operator looks at their device list sees a MAG rather than + a blank row -- and the filters that reject a box reporting nothing read + this too. + """ + return json.dumps( + { + "mac": self.cfg.mac, + "sn": self.cfg.serial_number, + "model": self.cfg.model, + "type": "STB", + "random": self.handshake_random, + }, + separators=(",", ":"), + ) + # What get_profile's 'status' means. The portal decides which authentication # this account needs and says so here, rather than the client guessing from # whether a password happens to be configured. @@ -1827,7 +1874,14 @@ def get_ordered_list(self, page: int) -> Dict[str, Any]: """ data = self._get_json( "type=itv&action=get_ordered_list&JsHttpRequest=1-xml" - f"&genre=*&fav=0&sortby=number&p={int(page)}" + # 'category' says the same thing as 'genre' to the portals that + # read that one instead, 'force_ch_link_check' is sent empty by + # every client that sends it at all, and 'hd' asks for the whole + # line-up rather than the HD half of it. None of the three changes + # what a Ministra portal answers; each of them is what one of the + # other clients found a portal that wanted it. + f"&genre=*&category=*&fav=0&force_ch_link_check=&hd=0" + f"&sortby=number&p={int(page)}" ) js = data.get("js") if isinstance(data, dict) else None return js if isinstance(js, dict) else {} diff --git a/tests/test_auth.py b/tests/test_auth.py index 19ad45b..af91ec1 100644 --- a/tests/test_auth.py +++ b/tests/test_auth.py @@ -184,10 +184,47 @@ def test_the_whole_stb_identity_is_sent(): p.login() query = [q for q in p.queries if "action=get_profile" in q][0] for expected in ("signature=" + "a" * 64, "sn=SN1", "stb_type=MAG322", - "num_banks=1", "image_version=216", "hd=1", "ver=", "hw_version="): + "num_banks=1", "image_version=216", "hd=1", "ver=", "hw_version=", + # The two other clients send these and this one did not. + # Harmless to a portal that ignores them, and the shape a + # reseller's access filter looks for in one that does not. + "client_type=STB", "video_out=hdmi"): assert expected in query, f"{expected} missing from {query}" +def test_the_box_is_described_where_the_admin_panel_reads_it(): + """'metrics' is what a portal stores and shows its operator. + + Echoing the handshake's nonce back inside it is the part a portal could + actually check: it issued that value one request ago, and only something + that read the answer can quote it. + """ + p = portal({"handshake": {"js": {"token": "TOK", "random": "R1"}}, + "get_profile": {"js": {"status": 0}}}, + serial_number="SN1", model="MAG322") + p.login() + query = [q for q in p.queries if "action=get_profile" in q][0] + for expected in ("%22random%22%3A%22R1%22", "%22model%22%3A%22MAG322%22", + "%22sn%22%3A%22SN1%22", "%22type%22%3A%22STB%22"): + assert expected in query, f"{expected} missing from {query}" + + +def test_the_prehash_is_this_box_rather_than_every_box(): + """A constant shared by every user of one client is the version that fails. + + Nothing in Ministra reads it; an access_filter.php in front of it can, and + that is the whole reason to send one at all. + """ + p = portal({"handshake": HANDSHAKE, "get_profile": {"js": {"status": 0}}}) + p.login() + expected = "prehash=" + s.prehash("00:1A:79:AA:BB:CC") + assert any(expected in q for q in p.queries if "action=handshake" in q), p.queries + assert any(expected in q for q in p.queries if "action=get_profile" in q), p.queries + # The MAC, not the box, and not case-sensitive about how it was written. + assert s.prehash("00:1a:79:aa:bb:cc") == s.prehash("00:1A:79:AA:BB:CC") + assert len(s.prehash("00:1A:79:AA:BB:CC")) == 40 + + def test_a_dead_session_in_plain_text_is_an_auth_error(): """Ministra answers 200 with prose, not JSON, once a token has expired. diff --git a/tests/test_listing.py b/tests/test_listing.py index 9e9926b..c591f40 100644 --- a/tests/test_listing.py +++ b/tests/test_listing.py @@ -384,6 +384,22 @@ def page(ids, **extra): return js +def test_the_paged_request_says_the_same_thing_several_ways(): + """Belt and braces, as with the identity in _common_params. + + A Ministra portal reads 'genre' and ignores the rest. The clones do not all + read the same one, and every parameter here is one that some other client + found a portal wanting -- sending them all costs a longer query string. + """ + p = portal() + seen = [] + p._get_json = lambda q, with_auth=True: seen.append(q) or {"js": {"data": []}} + p.get_ordered_list(3) + for expected in ("genre=*", "category=*", "fav=0", "force_ch_link_check=", + "hd=0", "sortby=number", "p=3"): + assert expected in seen[0], f"{expected} missing from {seen[0]}" + + REFUSED = s.PortalError("portal returned an empty channel list (check the MAC address)") From a72dfe0656877e2f55af869e6ffd69da886f43ce Mon Sep 17 00:00:00 2001 From: PiloUnk <198624632+PiloUnk@users.noreply.github.com> Date: Mon, 24 Aug 2026 22:41:05 +0200 Subject: [PATCH 03/10] Hand a command back the way a box hands it back create_link encoded the command with quote(cmd, safe=""), which escapes the percent sign along with everything else. The portal's own client does not: it concatenates the command into the query string and lets the URL layer escape only what a URL cannot carry, so PHP form-decodes it exactly once and a command that already contains '%3A' arrives with that escape intact. Ours arrived as '%253A' -- still encoded, a different string from the one a real box sends, and the stock create_link handler matches on it with preg_match. The commands this could happen to are the ones a portal answered its listing with as a resolved URL, which are exactly the ones canonical_cmd declines to rewrite when they carry no channel id: worst where there is least left to recover it. The safe set is the one a URL serializer keeps raw in a query, minus nothing that matters: '&', '#' and ';' stay encoded, so a portal cannot restructure our own request by putting them in a command. 'disable_ad' and 'download' go out beside it. Neither changes what a live channel answers; both are what a portal expecting its own client sees on every request except this plugin's. Co-Authored-By: Claude Opus 5 --- stalker_api.py | 35 ++++++++++++++++++++++++++++++++++- tests/test_listing.py | 19 +++++++++++++++++++ 2 files changed, 53 insertions(+), 1 deletion(-) diff --git a/stalker_api.py b/stalker_api.py index 8df933a..e5318f5 100644 --- a/stalker_api.py +++ b/stalker_api.py @@ -517,6 +517,32 @@ def stream_id(cmd: str) -> str: return found[0].strip() +# What a real box leaves raw in the command it hands back, and therefore what +# the portal sees before PHP form-decodes the query exactly once. The portal's +# own client concatenates the command into the query string and lets the URL +# layer escape only what a URL cannot carry; anything already percent-escaped +# in the command travels with that escape intact. +# +# quote(cmd, safe="") escaped the percent sign itself, so a command containing +# '%3A' left here as '%253A' and arrived at the portal still encoded -- a +# different string from the one a real box sends, and the stock create_link +# handler matches on it with preg_match. Portals that answer their listing with +# an already-encoded URL are exactly the ones canonical_cmd cannot rewrite, so +# this was worst where it was least recoverable. +# +# Everything outside this set is still percent-encoded. That covers what a URL +# cannot carry -- space, quotes, anything non-ASCII -- and, more to the point, +# the three characters that would restructure the request around it: '&', '#', +# and ';' for a PHP configured with it as an argument separator. A portal +# cannot smuggle extra parameters into our query through a command. +CMD_SAFE = "%/:?=+,@$[]!*()~-_." + + +def encode_cmd(cmd: str) -> str: + """A command as the portal's own client would have put it on the wire.""" + return quote(cmd, safe=CMD_SAFE) + + def slugify(value: str) -> str: """Reduce a display name to something safe for URLs, keys and filenames.""" slug = re.sub(r"[^a-z0-9]+", "-", value.strip().lower()).strip("-") @@ -2088,7 +2114,14 @@ def create_link(self, cmd: str) -> str: without it -- see :func:`stream_id`. Only ever sent with a value, so a portal that never asked for it sees the request it has always seen. """ - query = f"action=create_link&type=itv&cmd={quote(cmd, safe='')}" + query = ( + "action=create_link&type=itv" + f"&cmd={encode_cmd(cmd)}" + # What the portal's own player.js sends beside the command. Neither + # changes what a live channel answers; both are what a portal + # expecting its own client sees on every request except ours. + "&disable_ad=0&download=0" + ) channel = stream_id(cmd) if channel: query += f"&stream={quote(channel, safe='')}" diff --git a/tests/test_listing.py b/tests/test_listing.py index c591f40..315eb7f 100644 --- a/tests/test_listing.py +++ b/tests/test_listing.py @@ -351,6 +351,25 @@ def test_reading_the_channel_out_of_a_command(): assert s.stream_id(cmd) == expected, cmd +def test_a_command_reaches_the_portal_decoded_exactly_once(): + """PHP form-decodes the query once, so what is already escaped must stay so. + + quote(cmd, safe="") escaped the percent sign too, so '%3A' went out as + '%253A' and the portal read back a string no real box would have sent. + """ + assert s.encode_cmd("ffmpeg http://h/a%3Ab") == "ffmpeg%20http://h/a%3Ab" + # The characters a URL keeps raw in a query stay raw, so the bytes we emit + # are the bytes a box emits. + assert s.encode_cmd("http://h/p?a=1&") .startswith("http://h/p?a=1") + # ...except the ones that would restructure our own request around it. + for smuggled in ("&", "#", ";"): + assert smuggled not in s.encode_cmd(f"http://h/1{smuggled}stream=9"), smuggled + # The ordinary case, unchanged: a marker has nothing in it to encode. + assert s.encode_cmd("ffmpeg http://localhost/ch/1_") == ( + "ffmpeg%20http://localhost/ch/1_" + ) + + # -- paging -------------------------------------------------------------- From 544b4ff30afbd5e6ee6c96ae65041083a3ad3dc4 Mon Sep 17 00:00:00 2001 From: PiloUnk <198624632+PiloUnk@users.noreply.github.com> Date: Mon, 24 Aug 2026 22:43:42 +0200 Subject: [PATCH 04/10] Look for a portal on every path one is served from, not just two The endpoint cannot be worked out from what a user pastes. Ministra serves /stalker_portal/server/load.php; reseller panels serve /c/portal.php or /portal.php, and some serve neither, from a path of their own. pvr.stalker knows the first pair and stalkerhek the second, this plugin knew the second, and a portal on any of the others could not be reached at all. So alternate_endpoint, which was a mapping, becomes endpoint_candidates, which is a list to probe. Its second entry is exactly what the mapping used to return, both ways round, so a portal found on the second try before is found on the second try still; the rest are added after it. The other half is that normalize_portal_url no longer overrides a .php the user pasted. Swapping any other .php for portal.php in the same directory is stalkerhek's rule and the one departure from it here: a panel under a path of its own is not a typo, it is the address the provider handed out, and replacing it with a guess is how the one URL known to work stops being tried. The standard siblings are probed after it, so nothing is lost by trying it first. Walking the list stays bounded by the rule that was already there: only an answer that is not a Stalker API moves to the next candidate. Every path lives on the same host, so a portal that is down, unwell or refusing the MAC fails on the first one -- at tune time a wasted round-trip is time Dispatcharr is not yet spending on the next source. Co-Authored-By: Claude Opus 5 --- README.md | 11 +-- stalker_api.py | 157 +++++++++++++++++++++++++------------------ tests/test_auth.py | 36 ++++++++-- tests/test_config.py | 7 +- 4 files changed, 136 insertions(+), 75 deletions(-) diff --git a/README.md b/README.md index 076b5b9..4ad24fb 100644 --- a/README.md +++ b/README.md @@ -156,13 +156,14 @@ URL: | `http://host:8080/c/` | `http://host:8080/c/portal.php` | | `host:8080/c/` | `http://host:8080/c/portal.php` | | `http://host` | `http://host/portal.php` | -| `http://host/…/load.php` | unchanged — explicit endpoints are preserved | -| `http://host/c/other.php` | `http://host/c/portal.php` | +| `http://host/…/load.php` | unchanged — any explicit `.php` is preserved | +| `http://host/cp/api.php` | unchanged — a panel's own path is an address, not a typo | If that path turns out not to be where the portal answers, Distalker tries the -other one Ministra uses — `…/c/portal.php` and `…/server/load.php` are swapped -for each other — and logs which one worked. Putting the working one on the -portal line saves a failed request on every sync. +others in turn — `…/server/load.php`, `…/c/portal.php`, `…/portal.php` and +`…/stalker_portal/server/load.php`, built from the same install root — and logs +which one worked. Putting the working one on the portal line saves a failed +request on every sync. Anything unusual goes in trailing `key=value` pairs, separated by spaces or further `|` characters, quoted where a value contains spaces diff --git a/stalker_api.py b/stalker_api.py index e5318f5..1aee045 100644 --- a/stalker_api.py +++ b/stalker_api.py @@ -254,7 +254,7 @@ class PortalEndpointError(PortalError): A 404, or a body that is not JSON at all. Separated because it is the one failure with a second thing worth trying: the same portal on its other - path -- see :func:`alternate_endpoint`. + path -- see :func:`endpoint_candidates`. """ @@ -881,53 +881,75 @@ def normalize_portal_url(url: str) -> str: path = parsed.path lower = path.lower() - if lower.endswith(("/portal.php", "/load.php")): - pass # explicit endpoint: leave exactly as given + if lower.endswith(".php"): + # An explicit endpoint, left exactly as given -- including a path no + # standard install serves. This is the one departure from stalkerhek, + # which swaps any other .php for portal.php in the same directory: + # panels living under a path of their own do exist, that path is the + # address the provider handed out, and replacing it with a guess is how + # the one URL known to work stops being tried at all. + # endpoint_candidates() probes the standard siblings after it anyway, + # so nothing is lost by trusting what was pasted first. + pass elif path in ("", "/"): path = "/portal.php" - elif lower.endswith(".php"): - # Some other .php file: swap in portal.php from the same directory. - directory = path.rsplit("/", 1)[0] - path = f"{directory}/portal.php" else: path = path.rstrip("/") + "/portal.php" return parsed._replace(path=path).geturl() -def alternate_endpoint(url: str) -> str: - """The other place a Stalker API lives, or '' when there isn't one. +def endpoint_candidates(url: str) -> List[str]: + """Where a Stalker API might answer for this URL, best guess first. - Ministra answers on two paths and installs differ in which they expose: - ``/c/portal.php``, which is what :func:`normalize_portal_url` builds - and what most providers hand out, and ``/server/load.php``, which is - the older canonical one and the only one pvr.stalker has ever asked for. - A portal serving just one of them used to be unusable if the user had been - given the other, with a 404 and nothing to suggest. + The endpoint cannot be worked out from what a user pastes. Ministra serves + ``/stalker_portal/server/load.php`` and shows its interface at + ``/stalker_portal/c/``; reseller panels serve ``/c/portal.php`` + or ``/portal.php``, and some serve neither, from a path of their own. + pvr.stalker knows the first pair and stalkerhek the second. No client knows + all of them, which is why this is a list to probe rather than a mapping to + apply. - The mapping is pvr.stalker's, read backwards as well as forwards:: + The configured URL always comes first, and the second entry is still what + :func:`alternate_endpoint` used to be the whole of -- the pvr.stalker + mapping, read both ways. The rest are added after it, so a portal that was + found on the second try before is found on the second try still. - http://h/c/portal.php -> http://h/server/load.php - http://h/stalker_portal/c/portal.php -> http://h/stalker_portal/server/load.php - http://h/server/load.php -> http://h/c/portal.php + The siblings are built from the install root: the configured directory with + a trailing ``/c`` or ``/server`` taken off, because both of those are the + API's own subdirectory rather than part of where the install lives. """ parsed = urlparse(url) - path = parsed.path - directory, _, filename = path.rpartition("/") - filename = filename.lower() - - if filename == "portal.php": - base = directory[:-2] if directory.lower().endswith("/c") else directory - new_path = base + "/server/load.php" - elif filename == "load.php": - base = directory[:-7] if directory.lower().endswith("/server") else directory - new_path = base + "/c/portal.php" - else: - return "" + directory, _, filename = parsed.path.rpartition("/") + if not filename.lower().endswith(".php"): + # Not an endpoint at all: read the whole path as the directory rather + # than throwing away its last segment. + directory = parsed.path + + base = directory.rstrip("/") + for own in ("/c", "/server"): + if base.lower().endswith(own): + base = base[: -len(own)] + break - if new_path == path: - return "" - return parsed._replace(path=new_path).geturl() + paths = [ + parsed.path, + # The pvr.stalker pair, which is what this list grew out of. + base + "/server/load.php", + base + "/c/portal.php", + base + "/portal.php", + ] + # Already the canonical form when the base ends there -- nesting it again + # would probe a path no server has. + if "/stalker_portal" not in base.lower(): + paths.append(base + "/stalker_portal/server/load.php") + + candidates: List[str] = [] + for path in paths: + candidate = parsed._replace(path=path).geturl() + if candidate not in candidates: + candidates.append(candidate) + return candidates # --------------------------------------------------------------------------- @@ -1673,7 +1695,7 @@ def login(self) -> str: explicit refusal (:class:`PortalAuthError`) is still fatal, because that is the portal answering rather than failing to. """ - self._handshake_on_either_endpoint() + self._handshake_on_any_endpoint() try: self.profile = self.get_profile() @@ -1709,46 +1731,53 @@ def login(self) -> str: return self.token - def _handshake_on_either_endpoint(self) -> None: - """Shake hands, trying the portal's other API path if this one is not it. + def _handshake_on_any_endpoint(self) -> None: + """Shake hands, walking the portal's other API paths if this one is not it. The handshake is every session's first request, so a portal reached at the wrong path fails here and nowhere later -- which makes this the one - place worth spending an extra round-trip on. + place worth spending extra round-trips on. - Only a :class:`PortalEndpointError` earns that second try: a 404, or an - answer that is not JSON. A portal that is unreachable, unwell or - refusing the MAC would answer identically on both paths, and at tune - time a wasted round-trip is time Dispatcharr is not yet spending on the - next source. + Only a :class:`PortalEndpointError` moves to the next candidate: a 404, + or an answer that is not JSON. A portal that is unreachable, unwell or + refusing the MAC would answer identically on every path -- they all + live on the same host -- and at tune time a wasted round-trip is time + Dispatcharr is not yet spending on the next source. That is also why + the list is only walked when the user's URL is wrong, which is a + setup-time mistake rather than something that happens mid-service. The swap lasts for this session only. Nothing is written back, so the cost is one failed request per sync and per token expiry -- small, and the warning tells the user how to stop paying it for good. """ - try: - self.handshake() - return - except PortalEndpointError as exc: - alternate = alternate_endpoint(self.url) - if not alternate: + first_failure: Optional[PortalError] = None + + for candidate in endpoint_candidates(self.url): + self.url = candidate + try: + self.handshake() + except PortalEndpointError as exc: + if first_failure is None: + first_failure = exc + continue + except PortalError: + # Not a statement about the path, so no other path can help. + self.url = self.cfg.url raise - first_failure = exc - self.url = alternate - try: - self.handshake() - except PortalError: - # The other path is no better. Report the original failure: it is - # the one about the URL the user actually configured. - self.url = self.cfg.url - raise first_failure - - self.warnings.append( - f"the portal does not answer at {self.cfg.url} ({first_failure}), " - f"but does at {alternate}; put that on its portal line to save a " - "failed request on every sync" - ) + if candidate != self.cfg.url: + self.warnings.append( + f"the portal does not answer at {self.cfg.url} " + f"({first_failure}), but does at {candidate}; put that on " + "its portal line to save a failed request on every sync" + ) + return + + # Every path was answered by something that was not a Stalker API. + # Report the first failure: it is the one about the URL the user + # actually configured. + self.url = self.cfg.url + raise first_failure @staticmethod def _profile_status(profile: Dict[str, Any]) -> int: diff --git a/tests/test_auth.py b/tests/test_auth.py index af91ec1..fde3a50 100644 --- a/tests/test_auth.py +++ b/tests/test_auth.py @@ -371,17 +371,45 @@ def test_when_neither_endpoint_answers_the_configured_one_is_blamed(): assert p.url == p.cfg.url, p.url -def test_the_two_endpoints_map_onto_each_other(): +def test_the_configured_url_is_always_tried_first(): + """Including one no standard install serves: it is the address handed out.""" + for given in ("http://h/c/portal.php", "http://h/cp/api.php", + "http://h:8080/stalker_portal/server/load.php"): + assert s.endpoint_candidates(given)[0] == given, given + + +def test_the_second_guess_is_still_the_one_it_always_was(): + """The pvr.stalker mapping, read both ways, so nothing found on the second + try before takes any longer now.""" cases = { "http://h/c/portal.php": "http://h/server/load.php", "http://h/stalker_portal/c/portal.php": "http://h/stalker_portal/server/load.php", "http://h/server/load.php": "http://h/c/portal.php", "http://h:8080/c/portal.php": "http://h:8080/server/load.php", - # Nothing sensible to swap to. - "http://h/something.cgi": "", } for given, expected in cases.items(): - assert s.alternate_endpoint(given) == expected, given + assert s.endpoint_candidates(given)[1] == expected, given + + +def test_the_paths_probed_after_that(): + """The spellings pvr.stalker never knew, and no path probed twice.""" + found = s.endpoint_candidates("http://h/c/portal.php") + assert found == [ + "http://h/c/portal.php", + "http://h/server/load.php", + "http://h/portal.php", + "http://h/stalker_portal/server/load.php", + ], found + + # A base that is already a stalker_portal install: nesting it again would + # probe a path no server has. + nested = s.endpoint_candidates("http://h/stalker_portal/c/portal.php") + assert not any(p.count("stalker_portal") > 1 for p in nested), nested + + # A panel under a path of its own keeps that path, and its siblings are + # looked for beside it rather than at the site root. + panel = s.endpoint_candidates("http://h/cp/api.php") + assert all("/cp/" in p for p in panel), panel def test_there_is_no_watchdog(): diff --git a/tests/test_config.py b/tests/test_config.py index 11f625b..2c29cc4 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -94,8 +94,11 @@ def test_url_normalisation(): # An explicit load.php must survive: older portals only serve that. "http://a.example/stalker_portal/server/load.php": "http://a.example/stalker_portal/server/load.php", - # Any other .php is swapped for portal.php in the same directory. - "http://a.example/c/other.php": "http://a.example/c/portal.php", + # The one departure from stalkerhek, which swaps any other .php for + # portal.php in the same directory. Panels under a path of their own + # exist, and that path is the address the provider handed out; + # endpoint_candidates() probes the standard siblings after it. + "http://a.example/c/other.php": "http://a.example/c/other.php", # A missing scheme is filled in rather than rejected. "somedomain.com:8080/c/": "http://somedomain.com:8080/c/portal.php", # Ports and deep paths must be preserved. From 2592ed604fa1321dcb508e53bb953a385295e7fe Mon Sep 17 00:00:00 2001 From: PiloUnk <198624632+PiloUnk@users.noreply.github.com> Date: Mon, 24 Aug 2026 22:45:56 +0200 Subject: [PATCH 05/10] Read three things the portal was already saying **The expiry.** It was read from get_main_info's 'phone', which is where a reseller types it because Ministra shows that field in the MAG interface. But Ministra's own answer carries it too, as a Unix timestamp in the account_info block of the profile login() has already read -- free, and the authoritative one where it exists. get_main_info is now only asked when that came up empty, and its 'end_date' and 'expire_billing_date' are read alongside 'phone'. Timestamps needed the parser to grow two rules. A portal writes "never" as 0 or -1, and reading either as a date reports every unlimited account as having run out in 1970 -- the one wrong answer worse than no answer at all; and the same value arrives in milliseconds from portals that send it that way. **The refusal.** Portals write these with markup in them ("Subscription expired.
Call us."), and it was going into a Dispatcharr notification with the tags still in it. A device conflict now says what to do about it. It is the only refusal here a user can act on, and the portal's own wording names the box rather than the binding, so the message arrived describing a broken STB instead of two settings on the portal line. Matched on the binding alone: "device limit reached" has no remedy, and offering one would be worse than saying nothing. **A repeated channel.** get_all_channels was deduplicated only on the paged path. Portals do repeat a row there, and Dispatcharr hashes a stream partly on its URL, so a duplicate is a second stream for one channel rather than an extra line nobody sees. Co-Authored-By: Claude Opus 5 --- stalker_api.py | 115 ++++++++++++++++++++++++++++++++++++------ tests/test_auth.py | 43 ++++++++++++++++ tests/test_config.py | 14 +++++ tests/test_listing.py | 10 ++++ 4 files changed, 167 insertions(+), 15 deletions(-) diff --git a/stalker_api.py b/stalker_api.py index 1aee045..a2ce8de 100644 --- a/stalker_api.py +++ b/stalker_api.py @@ -584,10 +584,31 @@ def name_from_url(url: str) -> str: def parse_expiry(value: Any) -> Optional[datetime]: - """Read a subscription expiry out of a free-text portal field.""" + """Read a subscription expiry out of whatever field carried it. + + Two shapes, because the two places it is found do not agree. get_main_info + holds free text a reseller typed; the profile's own account_info block + holds a Unix timestamp, which is also how a portal writes "never" -- 0 and + -1 both mean no expiry, and reading either as a date would report every + unlimited account as having run out in 1970. + """ text = str(value or "").strip() if not text or text.startswith("0000-00-00"): return None + + if re.fullmatch(r"-?\d+", text): + seconds = int(text) + if seconds <= 0: + return None + # Past the year 3000 in seconds is a value that was meant as + # milliseconds; portals send both. + if seconds > 32503680000: + seconds //= 1000 + try: + return datetime.fromtimestamp(seconds, timezone.utc) + except (OverflowError, OSError, ValueError): + return None + for fmt in _EXPIRY_FORMATS: try: return datetime.strptime(text, fmt).replace(tzinfo=timezone.utc) @@ -613,6 +634,27 @@ def prehash(mac: str) -> str: return hashlib.sha1(mac.upper().encode("utf-8")).hexdigest().upper() +# Portals put markup in the sentence they refuse with -- "Your STB is +# damaged.
Call the provider." is a stock one -- and it lands in a +# Dispatcharr notification, where a tag is noise at best. +_MARKUP = re.compile(r"<[^>]*>") + +# Phrasings for the one refusal a user can act on: the portal has this MAC +# bound to a device id that is not the one being sent. Kept to the binding +# itself, because 'device' alone turns up in refusals with no remedy at all +# ("device limit reached"), and labelling one of those would hand the user a +# fix that cannot work. Safe as a phrase set where the raw-body patterns are +# not: this is a field the middleware wrote, not an arbitrary document. +DEVICE_CONFLICT = ( + re.compile(r"device\s*conflict", re.I), + re.compile( + r"device[\s_-]?id[^.!?]{0,40}?" + r"(mismatch|conflict|does\s*not\s*match|not\s*match)", + re.I, + ), +) + + def normalize_mac(mac: str) -> str: """Accept the shapes MAC addresses are quoted in, emit the one we use. @@ -1795,12 +1837,28 @@ def _profile_message(profile: Dict[str, Any]) -> str: """The portal's own explanation, if it gave one. ``block_msg`` first: when both are set it is the specific one, and it - is what the reseller wrote for exactly this situation. + is what the reseller wrote for exactly this situation. Markup comes out + of it, because this ends up in a Dispatcharr notification and portals + write these with ``
`` in them. + + A device conflict gets a sentence added. It is the one refusal here + that the user can do something about, and the portal's own wording for + it names the device rather than the binding -- so the message arrives + describing a problem with the box instead of one with two settings on + the portal line. """ for key in ("block_msg", "msg"): - value = str(profile.get(key) or "").strip() - if value: - return value + value = " ".join(_MARKUP.sub(" ", str(profile.get(key) or "")).split()) + if not value: + continue + if any(pattern.search(value) for pattern in DEVICE_CONFLICT): + value += ( + " -- the portal has this MAC bound to a different device " + "id; put the ones it expects on the portal line with " + "'device_id=' and 'device_id2=', or ask the provider to " + "clear the binding" + ) + return value return "" def account_snapshot(self) -> Dict[str, Any]: @@ -1830,14 +1888,30 @@ def account_snapshot(self) -> Dict[str, Any]: """ snapshot: Dict[str, Any] = {"expires": None, "blocked": False} - try: - js = self._get_json( - "type=account_info&action=get_main_info&JsHttpRequest=1-xml" - ).get("js") - if isinstance(js, dict): - snapshot["expires"] = parse_expiry(js.get("phone")) - except Exception: - pass + # The profile login() already read comes first, and costs nothing: + # 'account_info' is where Ministra itself puts the date, as a Unix + # timestamp. A portal that answered there is not asked again. + info = self.profile.get("account_info") + if isinstance(info, dict): + snapshot["expires"] = parse_expiry(info.get("expire_date")) + + if snapshot["expires"] is None: + try: + js = self._get_json( + "type=account_info&action=get_main_info&JsHttpRequest=1-xml" + ).get("js") + if isinstance(js, dict): + # 'phone' last: it is where the date ends up on the portals + # this plugin actually meets, but it is a free-text field + # and the three named ones mean only this when present. + for field in ("expire_date", "end_date", + "expire_billing_date", "phone"): + found = parse_expiry(js.get(field)) + if found: + snapshot["expires"] = found + break + except Exception: + pass snapshot["blocked"] = str(self.profile.get("blocked") or "0") not in ("0", "") @@ -1882,10 +1956,21 @@ def get_all_channels(self) -> List[ChannelEntry]: ) channels: List[ChannelEntry] = [] + seen = set() for row in rows: channel = self._channel_from_row(row) - if channel is not None: - channels.append(channel) + if channel is None: + continue + # Portals do repeat a channel in this response, and a duplicate is + # not a harmless extra row: Dispatcharr hashes a stream partly on + # its URL, so it becomes a second stream for one channel. Keyed the + # way the paged path keys it -- the id when there is one, the + # command when there is not. + key = channel.channel_id or channel.cmd + if key in seen: + continue + seen.add(key) + channels.append(channel) return channels @staticmethod diff --git a/tests/test_auth.py b/tests/test_auth.py index fde3a50..f8b3c11 100644 --- a/tests/test_auth.py +++ b/tests/test_auth.py @@ -121,6 +121,49 @@ def test_the_portal_gets_the_last_word_on_why(): raise AssertionError("status 1 must refuse the session") +def test_the_refusal_arrives_without_the_markup_it_was_written_in(): + """It lands in a Dispatcharr notification, where a
is noise.""" + p = portal({ + "handshake": HANDSHAKE, + "get_profile": {"js": {"status": 1, + "block_msg": "Subscription expired.
Call us."}}, + }) + try: + p.login() + except s.PortalAuthError as exc: + assert str(exc) == "Subscription expired. Call us.", exc + else: + raise AssertionError("status 1 must refuse the session") + + +def test_a_bound_mac_is_named_as_the_settings_that_fix_it(): + """The one refusal here a user can act on, and the portal misnames it. + + Its own wording points at the box; the problem is two values on the portal + line. Kept to the binding itself -- 'device limit reached' has no remedy, + and offering one would be worse than saying nothing. + """ + p = portal({ + "handshake": HANDSHAKE, + "get_profile": {"js": {"status": 1, "msg": "device id mismatch"}}, + }) + try: + p.login() + except s.PortalAuthError as exc: + assert "device_id2=" in str(exc), exc + else: + raise AssertionError("status 1 must refuse the session") + + plain = portal({ + "handshake": HANDSHAKE, + "get_profile": {"js": {"status": 1, "msg": "device limit reached"}}, + }) + try: + plain.login() + except s.PortalAuthError as exc: + assert str(exc) == "device limit reached", exc + + def test_a_second_step_that_still_fails_is_refused(): p = portal( {"handshake": HANDSHAKE, diff --git a/tests/test_config.py b/tests/test_config.py index 2c29cc4..8eccdde 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -271,6 +271,20 @@ def test_expiry_is_read_from_the_field_resellers_use(): assert s.parse_expiry(None) is None +def test_expiry_is_also_read_from_the_field_ministra_uses(): + """account_info holds a Unix timestamp, and it is free: login() read it. + + It is also where a portal writes 'never', as 0 or -1. Read as a date, an + unlimited account would be reported as having run out in 1970 -- which is + the one wrong answer worse than no answer at all. + """ + assert s.parse_expiry(1785000000).year == 2026 + # Portals send the same value in milliseconds, and mean the same date. + assert s.parse_expiry("1785000000000") == s.parse_expiry("1785000000") + for unlimited in (0, "0", -1, "-1"): + assert s.parse_expiry(unlimited) is None, unlimited + + def test_the_default_arguments_let_dispatcharr_fail_over(): """ffmpeg must not reconnect on its own: it retries an expired portal link while staying alive, so Dispatcharr sees no failure and never switches to diff --git a/tests/test_listing.py b/tests/test_listing.py index 315eb7f..78d0a01 100644 --- a/tests/test_listing.py +++ b/tests/test_listing.py @@ -422,6 +422,16 @@ def test_the_paged_request_says_the_same_thing_several_ways(): REFUSED = s.PortalError("portal returned an empty channel list (check the MAC address)") +def test_a_channel_repeated_in_one_response_is_still_one_channel(): + """Dispatcharr hashes a stream partly on its URL, so a duplicate row is a + second stream for one channel rather than a harmless extra line.""" + p = portal() + p._get_json = lambda q, with_auth=True: {"js": {"data": [ + row(id="1"), row(id="2", name="Two"), row(id="1"), + ]}} + assert [c.channel_id for c in p.get_all_channels()] == ["1", "2"] + + def test_a_portal_that_answers_in_one_request_is_never_paged(): p = scripted({"js": {"data": [row()]}}, {}) assert len(p.list_channels()) == 1 From 0c735db70efb64ec9e3f0cdc16e92fba827e1e7d Mon Sep 17 00:00:00 2001 From: PiloUnk <198624632+PiloUnk@users.noreply.github.com> Date: Mon, 24 Aug 2026 22:48:21 +0200 Subject: [PATCH 06/10] Fetch the portal's own streams as the box that authenticated A MAG sends the mac cookie and the session token to everything it fetches from the portal, and panels that gate the stream on that cookie answer a request without one with a 403. This plugin sent neither, so those channels authenticated, resolved a link, and then would not play -- with nothing in the ffmpeg output to say which of the three steps had actually failed. They go out now, but only where a box would send them. A create_link URL frequently points somewhere that is nobody's business but its own -- a CDN, an operator's edge, another provider entirely -- and it carries its own token in the query, so it needs nothing from us. Sending the MAC and the session token there would hand a third party everything required to use the subscription. Same host, and never a downgrade from https to http. A different port is still the portal: panels routinely serve the stream from :8080 next to the portal on :80, and those are exactly the ones gated on the cookie. resolve() hands the token back with the link because only the session that minted a link is the one the stream will accept -- reading the cache again would race the re-authentication the optimistic path may just have done. --probe reports which of the two shapes it sent, since "resolved but 403" is the failure this exists to explain. Co-Authored-By: Claude Opus 5 --- resolver.py | 35 +++++++++++++++++--------- stalker_api.py | 52 ++++++++++++++++++++++++++++++++++++--- tests/test_fallback.py | 30 ++++++++++++++++++++++ tests/test_mock_portal.py | 7 +++++- 4 files changed, 108 insertions(+), 16 deletions(-) diff --git a/resolver.py b/resolver.py index e10407b..2d3db2e 100644 --- a/resolver.py +++ b/resolver.py @@ -46,8 +46,13 @@ def log(message: str) -> None: print(f"[distalker] {message}", file=sys.stderr, flush=True) -def resolve(slug: str, cmd: str) -> tuple[str, stalker_api.PortalConfig]: - """Return a playable URL for ``cmd``, refreshing the session if needed.""" +def resolve(slug: str, cmd: str) -> tuple[str, stalker_api.PortalConfig, str]: + """Return a playable URL for ``cmd``, refreshing the session if needed. + + The token comes back with it: the stream is fetched with the session when + it is served by the portal itself, and only the session that minted the + link is the one it will accept. + """ try: client = stalker_api.get_redis() except Exception as exc: @@ -83,7 +88,7 @@ def resolve(slug: str, cmd: str) -> tuple[str, stalker_api.PortalConfig]: else: for warning in portal.warnings: log(f"{slug}: {warning}") - return link, cfg + return link, cfg, portal.token portal.login() # Cached before the link is asked for, not after: the token is good either @@ -95,15 +100,21 @@ def resolve(slug: str, cmd: str) -> tuple[str, stalker_api.PortalConfig]: # answer is reported too, and not only what login() found. for warning in portal.warnings: log(f"{slug}: {warning}") - return link, cfg + return link, cfg, portal.token -def build_ffmpeg_command(cfg: stalker_api.PortalConfig, url: str) -> list: +def build_ffmpeg_command( + cfg: stalker_api.PortalConfig, url: str, token: str = "" +) -> list: """Expand the portal's ffmpeg template into an argv list. Referer and Origin are derived from the **stream** URL, not the portal. Providers routinely serve the stream from a different host or port than the portal API, and expect the request to look like it came from there. + + The session goes with them when the stream is the portal's own -- see + :func:`stalker_api.stream_credential_safe`, which is what keeps a + subscriber's MAC out of a request to somebody else's CDN. """ import shlex from urllib.parse import urlparse @@ -111,8 +122,7 @@ def build_ffmpeg_command(cfg: stalker_api.PortalConfig, url: str) -> list: parsed = urlparse(url) origin = f"{parsed.scheme}://{parsed.netloc}" - headers = dict(stalker_api.stream_headers(cfg.model)) - headers["Origin"] = origin + headers = stalker_api.stream_headers(cfg, url, token) header_blob = "".join(f"{k}: {v}\r\n" for k, v in headers.items()) template = cfg.ffmpeg_args or stalker_api.DEFAULT_FFMPEG_ARGS @@ -229,7 +239,7 @@ def probe(pseudo_url: str) -> int: return 2 try: - link, cfg = resolve(slug, cmd) + link, cfg, token = resolve(slug, cmd) except Exception as exc: log(f"resolve failed: {exc}") return 1 @@ -242,10 +252,11 @@ def probe(pseudo_url: str) -> int: parsed = urlparse(link) origin = f"{parsed.scheme}://{parsed.netloc}" - headers = dict(stalker_api.stream_headers(cfg.model)) + headers = stalker_api.stream_headers(cfg, link, token) headers["User-Agent"] = stalker_api.USER_AGENT headers["Referer"] = origin + "/" - headers["Origin"] = origin + print("session : " + ("sent with the stream" + if "Cookie" in headers else "not this host's to send")) try: resp = requests.get(link, headers=headers, stream=True, timeout=20) @@ -293,7 +304,7 @@ def main(argv: list) -> int: return 2 try: - link, cfg = resolve(slug, cmd) + link, cfg, token = resolve(slug, cmd) except PortalError as exc: log(f"{slug}: {exc}") return 1 @@ -303,7 +314,7 @@ def main(argv: list) -> int: log(f"{slug}: resolved -> {link.split('?', 1)[0]}") - command = build_ffmpeg_command(cfg, link) + command = build_ffmpeg_command(cfg, link, token) executable = shutil.which(command[0]) or command[0] try: os.execv(executable, command) diff --git a/stalker_api.py b/stalker_api.py index a2ce8de..bfd4dc4 100644 --- a/stalker_api.py +++ b/stalker_api.py @@ -222,18 +222,64 @@ def is_superseded_ffmpeg_args(value: str) -> bool: current = " ".join((value or "").split()) return any(current == " ".join(old.split()) for old in SUPERSEDED_FFMPEG_ARGS) +def stream_credential_safe(portal_url: str, link: str) -> bool: + """Whether the stream is the portal's own, and may carry its session. + + The question matters because the answer is a subscriber's credentials. A + create_link URL often points somewhere that is nobody's business but its + own -- a CDN, an operator's edge, another provider entirely -- and it + already carries its own token in the query, so it needs nothing from us. + Sending the MAC and the session token there would hand a third party + everything required to use the subscription. + + Same host, and never a downgrade from https to http. A different *port* is + still the portal: panels routinely serve the stream from :8080 next to the + portal on :80, and those are exactly the ones gated on the mac cookie. + """ + try: + portal, stream = urlparse(portal_url), urlparse(link) + except ValueError: + return False + if stream.scheme not in ("http", "https"): + return False + host = (portal.hostname or "").lower() + if not host or host != (stream.hostname or "").lower(): + return False + return not (portal.scheme == "https" and stream.scheme == "http") + + # Headers a MAG box sends when fetching the stream itself, beyond the # User-Agent and Referer that ffmpeg has dedicated flags for. Providers do # check these: a request that authenticated fine against the portal can still # be refused at the stream if it does not look like the same box. -def stream_headers(model: str = DEFAULT_MODEL) -> Dict[str, str]: - return { - "X-User-Agent": f"Model: {model}; Link: Ethernet", +# +# The session travels with them when the stream is the portal's own. A MAG +# sends the mac cookie and the token to everything it fetches from the portal, +# and panels that gate the stream on that cookie answer a request without it +# with a 403 -- which arrives as a channel that authenticated, resolved, and +# then would not play. What the box would never do is send them anywhere else, +# which is what stream_credential_safe is for. +def stream_headers(cfg: "PortalConfig", link: str, token: str = "") -> Dict[str, str]: + parsed = urlparse(link) + headers = { + "X-User-Agent": f"Model: {cfg.model}; Link: Ethernet", "Accept": "*/*", "Accept-Language": "en-US,en;q=0.9", "Cache-Control": "no-cache", "Pragma": "no-cache", + # Derived from the stream, not the portal: providers serve it from + # another host or port and expect the request to look like it came + # from there. + "Origin": f"{parsed.scheme}://{parsed.netloc}", } + if stream_credential_safe(cfg.url, link): + headers["Cookie"] = ( + f"mac={quote(cfg.mac)}; stb_lang=en; " + f"timezone={quote(cfg.timezone)};" + ) + if token: + headers["Authorization"] = "Bearer " + token + return headers REDIS_PREFIX = "distalker" diff --git a/tests/test_fallback.py b/tests/test_fallback.py index e2db3ad..6726c24 100644 --- a/tests/test_fallback.py +++ b/tests/test_fallback.py @@ -134,6 +134,36 @@ def test_a_missing_user_agent_is_not_the_literal_placeholder(): assert "{userAgent}" not in " ".join(cmd) +def test_the_session_never_leaves_the_portal_that_issued_it(): + """A create_link URL frequently points somewhere that is nobody's business. + + It carries its own token in the query, so it needs nothing from us; sending + the MAC and the session token to it would hand a third party everything + required to use the subscription. + """ + cfg = s.PortalConfig(slug="t", name="T", url="http://p.example/c/portal.php", + mac="00:1A:79:AA:BB:CC") + + own = s.stream_headers(cfg, "http://p.example:8080/live/1.ts", "TOK") + # A different port is still the portal: panels serve the stream from :8080 + # next to the portal on :80, and those are the ones gated on the cookie. + assert "Cookie" in own and own["Authorization"] == "Bearer TOK" + + for foreign in ("http://cdn.example/live/1.ts", + "https://other.example/live/1.ts", + "udp://239.0.0.1:1234"): + headers = s.stream_headers(cfg, foreign, "TOK") + assert "Cookie" not in headers, foreign + assert "Authorization" not in headers, foreign + + # An https portal answering with an http stream is a downgrade, and the + # session must not travel in clear because the portal said so. + secure = s.PortalConfig(slug="t", name="T", url="https://p.example/c/portal.php", + mac="00:1A:79:AA:BB:CC") + assert "Cookie" not in s.stream_headers(secure, "http://p.example/live/1.ts", "TOK") + assert "Cookie" in s.stream_headers(secure, "https://p.example/live/1.ts", "TOK") + + if __name__ == "__main__": failures = 0 for name, fn in sorted(globals().items()): diff --git a/tests/test_mock_portal.py b/tests/test_mock_portal.py index 20b8d69..2a899b0 100644 --- a/tests/test_mock_portal.py +++ b/tests/test_mock_portal.py @@ -209,12 +209,17 @@ def main(): # ffmpeg argv construction, as resolver.py builds it. import resolver - argv = resolver.build_ffmpeg_command(cfg, link) + argv = resolver.build_ffmpeg_command(cfg, link, "TOKEN") print("ffmpeg argv:", argv) assert argv[0] == "ffmpeg" assert link in argv assert s.USER_AGENT in argv assert "pipe:1" in argv + # The mock portal answers with a link on another host, which is the shape + # a real create_link takes most of the time -- so the session stays here. + blob = argv[argv.index("-headers") + 1] + assert "mac=" not in blob and "Bearer" not in blob, blob + assert "X-User-Agent: Model: " in blob, blob server.shutdown() print("\nALL CHECKS PASSED") From f7a432de0c625ab5e9d4a75ff920a92139659e9b Mon Sep 17 00:00:00 2001 From: PiloUnk <198624632+PiloUnk@users.noreply.github.com> Date: Mon, 24 Aug 2026 23:07:21 +0200 Subject: [PATCH 07/10] Play the channels the portal says need no link, without asking it A Stalker row carries use_http_tmp_link and use_load_balancing, and the portal's own player.js calls create_link only when one of them is set -- everything else plays the cmd the listing already gave. pvr.stalker does the same and cites that line of player.js for it; iptvnator does it again with guards for portals that are not Ministra. This plugin asked for a link every time. On nearly every portal that changes nothing, and the reason is worth writing down: a stock row is 'ffmpeg http://localhost/ch/123_', which only the portal can resolve whatever its flags say -- and canonical_cmd rewrites the rows that arrive as resolved links into exactly that shape. pvr.stalker has no loopback guard at all, which is itself evidence that Ministra sets the flag on those rows: an ecosystem depends on it. The family this is for is the one undoubled_link was written for. Those providers answer the listing with a resolved link, and answer create_link by gluing their own base in front of it -- producing a path that returns 401, so the reply and its token are thrown away and the command is played anyway. That is one request per tune, spent to arrive back where it started, against providers that count connections. Four guards, each of which falls back on asking the portal, so none of them can break an install that works today. Absent flags are no evidence rather than a no -- every portal synced before this carries neither, and reading silence as "static" would move a whole installation at once. The scheme has to be one ffmpeg opens, which is not the question extract_link answers: a portal's own ffrt4:// pseudo-URL parses like an address and plays as nothing. The host must not be loopback, in rather more spellings than the obvious one. The fourth is ours rather than anybody's reference behaviour, and it is what makes this safe in a resolver rather than in a player: no query string. A link that expires carries its token there, and being wrong does not cost a retry -- by then the resolver has become ffmpeg and Dispatcharr has spent this channel's failover on a source that resolved perfectly well. The set of qualifying commands is published like the portal itself, Redis with a disk mirror, so a restart does not quietly turn the optimisation off. It is rewritten on every sync including when empty: a portal that has stopped marking its channels static must not inherit the list that said otherwise. Co-Authored-By: Claude Opus 5 --- CONTRIBUTING.md | 33 ++++++- resolver.py | 20 +++++ stalker_api.py | 201 +++++++++++++++++++++++++++++++++++++++++- sync.py | 20 ++++- tests/test_listing.py | 89 +++++++++++++++++++ tests/test_state.py | 100 +++++++++++++++++++++ 6 files changed, 456 insertions(+), 7 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ab98fa9..5baf545 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -139,10 +139,10 @@ added or deleted would silently rebind existing channels to the other portal. It runs inside the container with no persistence, so it comes back empty from every restart — which used to kill every channel until someone pressed Sync, -twice observed before it was understood. Everything `save_portal` and -`save_fallback` publish is therefore mirrored to `/data/distalker/state/*.json` -(`0600`: credentials), read only when Redis has nothing, and written back to -Redis on first use. Reads go through `_client_or_none`, so an unreachable Redis +twice observed before it was understood. Everything `save_portal`, +`save_static_cmds` and `save_fallback` publish is therefore mirrored to +`/data/distalker/state/*.json` (`0600`: credentials), read only when Redis has +nothing, and written back to Redis on first use. Reads go through `_client_or_none`, so an unreachable Redis degrades rather than raising. The session token is deliberately *not* mirrored: it expires within the hour, and losing it costs one handshake. @@ -151,6 +151,31 @@ before the first sync, or a lost data volume, and there is again nothing to read. `Plugin._republish` runs on the assign path instead, which the button, every `m3u_refresh` and every `channel_error` all reach. +### Some channels never reach the portal at tune time + +A Stalker row carries `use_http_tmp_link` and `use_load_balancing`, and the +portal's own `player.js` calls `create_link` only when one of them is set — +everything else plays the `cmd` the listing already gave. pvr.stalker does the +same and cites that line for it (`ChannelManager::GetStreamURL`). + +Sync evaluates `plays_without_create_link` per row and publishes the commands +that qualify as a set (`save_static_cmds`); `resolver.resolve` checks it before +doing anything else, and on a hit returns the command's own URL with no +handshake and no `create_link`. On nearly every portal the set is empty — a +stock row is a loopback marker, which only the portal can resolve whatever its +flags say. The family it exists for is the one `undoubled_link` was written +for: providers that answer the listing with a resolved link and then mangle it +when handed it back. + +Four guards narrow it, and every one of them falls back to asking the portal, +so none can break an install that works today. Absent flags mean *no evidence* +rather than "no". The scheme must be one ffmpeg opens (a portal's own +`ffrt4://` pseudo-URL parses like an address and plays as nothing). The host +must not be loopback in any of its spellings. And — this one is ours, not the +reference behaviour — the command must carry **no query string**: a link that +expires keeps its token there, and by the time a stream fails the resolver has +already become ffmpeg and Dispatcharr has spent this channel's failover. + ### The ffmpeg defaults carry no `-reconnect`, and that is load-bearing ffmpeg's own reconnection retries the URL it was handed, which for Stalker is a diff --git a/resolver.py b/resolver.py index 2d3db2e..f5d0c05 100644 --- a/resolver.py +++ b/resolver.py @@ -70,6 +70,26 @@ def resolve(slug: str, cmd: str) -> tuple[str, stalker_api.PortalConfig, str]: ) cached = stalker_api.get_cached_token(slug, client) + + # A channel the portal itself marked as needing no temporary link is played + # from the command the listing gave, with no request to the portal at all. + # That is what the portal's own player does, what pvr.stalker does, and + # here it also skips the one request that is known to go wrong: the + # providers undoubled_link exists for answer create_link by gluing their + # base in front of a command that was already a link, and the reply is + # thrown away again a moment later. + # + # The token is whatever was already cached -- it may be nothing, and that + # is not worth a handshake to fix. It only ever feeds a header, and a + # command with no query string is not a link the portal minted for a + # session in the first place. + if cmd in stalker_api.load_static_cmds(slug, client): + link = stalker_api.extract_link(cmd) + if link: + log(f"{slug}: the portal marks this channel as needing no " + "temporary link; playing its command as it stands") + return link, cfg, cached or "" + portal = stalker_api.Portal(cfg, token=cached or "") if cached: diff --git a/stalker_api.py b/stalker_api.py index bfd4dc4..5302575 100644 --- a/stalker_api.py +++ b/stalker_api.py @@ -20,6 +20,7 @@ from __future__ import annotations import hashlib +import ipaddress import json import os import re @@ -589,6 +590,114 @@ def encode_cmd(cmd: str) -> str: return quote(cmd, safe=CMD_SAFE) +# Schemes a stream can actually be handed to ffmpeg on. An allowlist, where +# extract_link deliberately accepts anything with '://': the two answer +# different questions. What a portal *resolves* may be multicast on a scheme +# nobody here has met, and refusing to play it would be worse than not +# recognising it -- but a command the portal never resolved can also be one of +# its own internal pseudo-URLs ('ffrt4://ch/live/1'), which parses like an +# address and plays as nothing. +PLAYABLE_SCHEMES = frozenset( + {"http", "https", "udp", "rtp", "rtsp", "rtmp", "rtmps", "srt", "mms"} +) + + +def _is_portal_local(host: str) -> bool: + """Whether a host can only mean the machine that wrote the address. + + 'ffrt3 http://localhost/ch/1234_' is an instruction to the portal, never an + address a set-top box could open, so a channel carrying one always needs + resolving whatever its flags say. Rather more spellings than the obvious + one: RFC 6761 reserves every name ending in '.localhost' as well, IPv4 + gives the whole of 127.0.0.0/8 to loopback, and a portal that writes its + own address as an IPv4-mapped IPv6 literal has said the same thing again. + + A host that cannot be read at all counts as local, because the question + this answers is "may this be played without asking the portal", and the + only safe answer about an address nobody understands is no. + """ + host = (host or "").strip().strip("[]").rstrip(".").lower() + if not host: + return True + if host == "localhost" or host.endswith(".localhost"): + return True + if host == "localhost.localdomain": + return True + try: + address = ipaddress.ip_address(host) + except ValueError: + # A name, and not one of the reserved loopback ones. + return False + mapped = getattr(address, "ipv4_mapped", None) + if mapped is not None: + address = mapped + return address.is_loopback or address.is_unspecified + + +def portal_flag(value: Any) -> Optional[bool]: + """A portal's 1/0 flag, or None when the portal did not set one. + + None is not False, and the difference is the whole of it: a row carrying + neither flag is a row the portal said nothing about, and silence has to + read as "no evidence" rather than "no". Portals write these as 1/0, as + "1"/"0", and occasionally as real booleans. + """ + if value is None: + return None + if isinstance(value, bool): + return value + text = str(value).strip().lower() + if not text: + return None + return text not in ("0", "false", "no", "off") + + +def plays_without_create_link(cmd: str, needs_link: Optional[bool]) -> bool: + """Whether this channel can be played from its command alone. + + The portal's own player.js asks create_link for a channel only when the row + asks for it -- because the portal proxies it through a per-session link + ('use_http_tmp_link') or picks a storage server per request + ('use_load_balancing'). Every other row plays the command the listing + already handed over. pvr.stalker does the same and cites that line of + player.js for it (ChannelManager::GetStreamURL); iptvnator does the same + again, with guards it added for portals that are not Ministra. + + Those guards are here too, and every one of them can only ever push a row + back onto the create_link path that exists today -- so none of them is able + to break a portal that works now: + + * the portal has to have answered the question at all. Every portal synced + before this was read carries no flags, and taking that silence as a "no" + would move all of them onto the static path at once. + * the command has to contain a URL. 'auto /media/file.mpg' does not, and + only create_link turns that into an address. + * on a scheme ffmpeg can open -- see PLAYABLE_SCHEMES. + * not on a host that can only mean the portal itself. + + The last condition is ours rather than anyone's reference behaviour, and it + is what makes this safe in a resolver rather than in a player: the command + must carry **no query string**. A link that expires keeps its token there, + and the cost of being wrong is not a retry -- by the time a stream fails + the resolver has already become ffmpeg, and Dispatcharr has spent this + channel's failover on a source that resolved perfectly well. A command with + no query has nothing in it that can go stale. + """ + if needs_link is not False: + return False + + link = extract_link(cmd) + if not link: + return False + + parsed = urlparse(link) + if parsed.scheme.lower() not in PLAYABLE_SCHEMES: + return False + if parsed.query: + return False + return not _is_portal_local(parsed.hostname or "") + + def slugify(value: str) -> str: """Reduce a display name to something safe for URLs, keys and filenames.""" slug = re.sub(r"[^a-z0-9]+", "-", value.strip().lower()).strip("-") @@ -1222,6 +1331,80 @@ def load_portal(slug: str, client=None) -> Optional[PortalConfig]: return cfg +def _static_key(slug: str) -> str: + return f"{REDIS_PREFIX}:static:{slug}" + + +def _static_mirror(slug: str) -> str: + return f"static-{slug}" + + +def static_commands(channels: List["ChannelEntry"]) -> List[str]: + """The commands the resolver may play without asking the portal first.""" + return sorted( + { + channel.cmd + for channel in channels + if plays_without_create_link(channel.cmd, channel.needs_link) + } + ) + + +def save_static_cmds(slug: str, commands: List[str], client=None) -> None: + """Publish the commands that need no create_link, for the resolver to read. + + Written on every sync including when it is empty, which is what nearly + every portal produces -- and what a portal that has *stopped* marking its + channels static has to leave behind, rather than inheriting the last list + that said otherwise. + + Mirrored first, like the portal itself: if Redis refuses, the sync says so + and the resolver still reads the right thing off disk. + """ + payload = list(commands) + _mirror_write(_static_mirror(slug), payload) + client = client or get_redis() + client.set(_static_key(slug), json.dumps(payload)) + + +def load_static_cmds(slug: str, client=None) -> set: + """The same set back, or an empty one. + + Every path out of here that is not a list lands on the empty set, and that + is deliberate rather than lazy: not knowing whether a channel is static has + exactly one safe reading, and it is asking the portal -- which is what this + plugin did before any of this existed. + """ + client = _client_or_none(client) + + raw = None + if client is not None: + try: + raw = client.get(_static_key(slug)) + except Exception: + raw = None + if raw: + try: + payload = json.loads(raw) + except (ValueError, TypeError): + payload = None + if isinstance(payload, list): + return {str(item) for item in payload} + + payload = _mirror_read(_static_mirror(slug)) + if not isinstance(payload, list): + return set() + + # Put it back, so a wiped Redis costs a file read once rather than once + # per tune -- the same bargain load_portal makes. + if client is not None: + try: + client.set(_static_key(slug), json.dumps(payload)) + except Exception: + pass + return {str(item) for item in payload} + + def _sync_lock_key() -> str: return f"{REDIS_PREFIX}:sync-lock" @@ -1360,8 +1543,9 @@ def published_slugs() -> List[str]: def forget_portal(slug: str, client=None) -> None: _mirror_forget(_portal_mirror(slug)) + _mirror_forget(_static_mirror(slug)) client = client or get_redis() - client.delete(_portal_key(slug), _token_key(slug)) + client.delete(_portal_key(slug), _token_key(slug), _static_key(slug)) def _fallback_key() -> str: @@ -1476,6 +1660,10 @@ class ChannelEntry: # canonical_cmd. Counted rather than logged per channel, because on the # portal that prompted it, 647 of them arrived at once. cmd_rewritten: bool = False + # Whether the portal says this channel needs a link minted for it before it + # can be played. None when the row carried neither flag, which is not the + # same as False -- see portal_flag and plays_without_create_link. + needs_link: Optional[bool] = None class Portal: @@ -2036,6 +2224,16 @@ def _channel_from_row(row: Any) -> Optional[ChannelEntry]: channel_id = str(row.get("id") or "") marker = canonical_cmd(cmd, channel_id) + # Two flags, one answer: either of them set means the portal mints the + # link. A row carrying neither has not answered, and None says so. + tmp_link = portal_flag(row.get("use_http_tmp_link")) + balanced = portal_flag(row.get("use_load_balancing")) + needs_link = ( + None + if tmp_link is None and balanced is None + else bool(tmp_link) or bool(balanced) + ) + return ChannelEntry( channel_id=channel_id, name=name, @@ -2047,6 +2245,7 @@ def _channel_from_row(row: Any) -> Optional[ChannelEntry]: # Portals write these as 1/0, and sometimes as "1"/"0". tv_archive=str(row.get("enable_tv_archive") or "0") not in ("0", ""), tv_archive_duration=str(row.get("tv_archive_duration") or ""), + needs_link=needs_link, ) def get_ordered_list(self, page: int) -> Dict[str, Any]: diff --git a/sync.py b/sync.py index c519cf6..cc1a47d 100644 --- a/sync.py +++ b/sync.py @@ -29,6 +29,8 @@ python_executable, save_fallback, save_portal, + save_static_cmds, + static_commands, ) # Where Dispatcharr's own M3U upload endpoint puts files. Reusing it keeps our @@ -808,9 +810,11 @@ def sync_portal( ) genres = portal.get_genres() - # The resolver reads this at tune time; publish it before the M3U lands so - # a channel can never reference a portal Redis doesn't know about yet. + # The resolver reads these at tune time; publish them before the M3U lands + # so a channel can never reference a portal Redis doesn't know about yet. save_portal(cfg) + static = static_commands(channels) + save_static_cmds(cfg.slug, static) path = write_m3u(cfg.slug, build_m3u(portal, channels, genres)) account, account_created = upsert_account(cfg, path, refresh_hours) @@ -839,6 +843,17 @@ def sync_portal( rewritten, ) + if static: + # Worth saying for the same reason the rewrite count is: it changes + # what happens at tune time, and on the providers concerned it is the + # difference between one request to the portal and none. + logger.info( + "distalker: %s: %d channel(s) are marked as needing no temporary " + "link, and will play from their command without asking the portal", + cfg.name, + len(static), + ) + epg = sync_epg(cfg, portal, channels, logger, trigger_refresh=trigger_refresh) return { @@ -851,6 +866,7 @@ def sync_portal( "file": path, "expires": snapshot["expires"], "blocked": snapshot["blocked"], + "static": len(static), "epg": epg, } diff --git a/tests/test_listing.py b/tests/test_listing.py index 78d0a01..cc82005 100644 --- a/tests/test_listing.py +++ b/tests/test_listing.py @@ -422,6 +422,95 @@ def test_the_paged_request_says_the_same_thing_several_ways(): REFUSED = s.PortalError("portal returned an empty channel list (check the MAC address)") +# -- the channels that need no link minted for them ----------------------- + + +def test_silence_is_not_a_no(): + """Every portal synced before these flags were read carries neither. + + Taking that as "needs no link" would move an entire installation onto the + static path at once, on nothing but the absence of evidence. + """ + assert s.Portal._channel_from_row(row()).needs_link is None + assert s.portal_flag(None) is None + assert s.portal_flag("") is None + + +def test_the_flags_are_read_in_every_shape_a_portal_writes_them(): + for value in ("1", 1, True, "true", "yes"): + assert s.portal_flag(value) is True, value + for value in ("0", 0, False, "false", "no", "off"): + assert s.portal_flag(value) is False, value + + # Either flag set means the portal mints the link; both clear means it does + # not; and both must be absent before the row counts as having said nothing. + def needs(**flags): + return s.Portal._channel_from_row(row(**flags)).needs_link + + assert needs(use_http_tmp_link="1", use_load_balancing="0") is True + assert needs(use_http_tmp_link="0", use_load_balancing="1") is True + assert needs(use_http_tmp_link="0", use_load_balancing="0") is False + assert needs(use_load_balancing="0") is False + + +def test_what_may_be_played_without_asking_the_portal(): + """The table. Every entry that is False falls back on today's behaviour, + which is why none of these guards can break a portal that works now.""" + static = "http://prov.example/USER/PASS/1225691" + assert s.plays_without_create_link(static, False) + # Multicast is a perfectly good static address, and ffmpeg opens it. + assert s.plays_without_create_link("udp://239.0.0.1:1234", False) + # 'ffmpeg' is a prefix word, not part of the address. + assert s.plays_without_create_link("ffmpeg " + static, False) + + for cmd, why in ( + ("http://localhost/ch/1_", "the portal talking to itself"), + ("http://127.0.0.1/ch/1_", "the whole of 127.0.0.0/8 is loopback"), + ("http://127.5.5.5/ch/1_", "still loopback"), + ("http://[::1]/ch/1_", "and in IPv6"), + ("http://[::ffff:127.0.0.1]/ch/1_", "and mapped back into IPv6"), + ("http://0.0.0.0/ch/1_", "the unspecified address means the same"), + ("http://stream.localhost/ch/1_", "RFC 6761 reserves the whole suffix"), + ("http:///ch/1", "an address with no host at all"), + ("ffrt4://ch/live/1", "a portal's own pseudo-URL plays as nothing"), + ("auto /media/file.mpg", "not an address until create_link makes one"), + (static + "?play_token=abc", "a token in the query is a link that expires"), + ): + assert not s.plays_without_create_link(cmd, False), why + + # And the flags still decide first. + assert not s.plays_without_create_link(static, True) + assert not s.plays_without_create_link(static, None) + + +def test_the_published_set_is_the_commands_and_nothing_else(): + channels = [ + s.Portal._channel_from_row(row( + id="1", cmd="ffmpeg http://localhost/ch/1_", + use_http_tmp_link="0", use_load_balancing="0")), + s.Portal._channel_from_row(row( + id="", cmd="http://prov.example/USER/PASS/22", + use_http_tmp_link="0", use_load_balancing="0")), + s.Portal._channel_from_row(row( + id="", cmd="http://prov.example/live/33", use_http_tmp_link="1")), + ] + assert s.static_commands(channels) == ["http://prov.example/USER/PASS/22"] + # A portal that marks nothing static publishes an empty list, not nothing. + assert s.static_commands([channels[0]]) == [] + + +def test_a_rewritten_command_is_never_static(): + """canonical_cmd turns a resolved link into a loopback marker, and a marker + is only ever an instruction to the portal -- whatever the flags said about + the row it came from.""" + channel = s.Portal._channel_from_row(row( + id="553690", cmd="http://prov.example/live.php?stream=553690", + use_http_tmp_link="0", use_load_balancing="0")) + assert channel.cmd_rewritten + assert channel.needs_link is False + assert not s.plays_without_create_link(channel.cmd, channel.needs_link) + + def test_a_channel_repeated_in_one_response_is_still_one_channel(): """Dispatcharr hashes a stream partly on its URL, so a duplicate row is a second stream for one channel rather than a harmless extra line.""" diff --git a/tests/test_state.py b/tests/test_state.py index 4aea2aa..fc6b355 100644 --- a/tests/test_state.py +++ b/tests/test_state.py @@ -167,6 +167,106 @@ def test_removing_a_portal_removes_its_mirror(): assert s.load_portal(CFG.slug, client) is None +STATIC_CMD = "http://prov.example/USER/PASS/1225691" + + +def test_the_static_commands_survive_a_restart_too(): + """A tune that read nothing would ask the portal, which is safe but is the + round trip this exists to avoid -- and on the providers concerned it is the + request that answers with a path returning 401.""" + reset() + client = FakeRedis() + s.save_static_cmds(CFG.slug, [STATIC_CMD], client) + assert os.path.exists(s._state_path(f"static-{CFG.slug}")) + + client.store.clear() + assert s.load_static_cmds(CFG.slug, client) == {STATIC_CMD} + assert s._static_key(CFG.slug) in client.store, "and it goes back in Redis" + + +def test_not_knowing_means_asking_the_portal(): + """The one safe reading of every failure here, and the behaviour that was + there before any of this existed.""" + reset() + assert s.load_static_cmds("never-registered", FakeRedis()) == set() + assert s.load_static_cmds(CFG.slug, FakeRedis(broken=True)) == set() + + client = FakeRedis() + s.save_static_cmds(CFG.slug, [STATIC_CMD], client) + client.store[s._static_key(CFG.slug)] = "{not json" + assert s.load_static_cmds(CFG.slug, client) == {STATIC_CMD}, "the mirror wins" + + +def test_a_portal_that_stops_marking_channels_static_is_believed(): + """Inheriting the last list that said otherwise would keep playing a + command the portal has since started minting links for.""" + reset() + client = FakeRedis() + s.save_static_cmds(CFG.slug, [STATIC_CMD], client) + s.save_static_cmds(CFG.slug, [], client) + assert s.load_static_cmds(CFG.slug, client) == set() + + +def test_removing_a_portal_removes_its_static_commands(): + reset() + client = FakeRedis() + s.save_portal(CFG, client) + s.save_static_cmds(CFG.slug, [STATIC_CMD], client) + s.forget_portal(CFG.slug, client) + + assert not os.path.exists(s._state_path(f"static-{CFG.slug}")) + assert s.load_static_cmds(CFG.slug, client) == set() + + +def test_a_static_channel_never_reaches_the_portal(): + """The whole point: no handshake, no create_link, no connection slot.""" + reset() + client = FakeRedis() + s.save_portal(CFG, client) + s.save_static_cmds(CFG.slug, [STATIC_CMD], client) + + class Unreachable(s.Portal): + def login(self): + raise AssertionError("a static channel must not contact the portal") + + def create_link(self, cmd): + raise AssertionError("a static channel must not contact the portal") + + original_portal, original_redis = s.Portal, s.get_redis + s.Portal, s.get_redis = Unreachable, lambda: client + try: + link, cfg, _ = resolver.resolve(CFG.slug, STATIC_CMD) + finally: + s.Portal, s.get_redis = original_portal, original_redis + + assert link == STATIC_CMD + assert cfg.slug == CFG.slug + + +def test_a_channel_the_portal_never_marked_still_goes_through_create_link(): + reset() + client = FakeRedis() + s.save_portal(CFG, client) + s.save_static_cmds(CFG.slug, [STATIC_CMD], client) + + class Answering(s.Portal): + def login(self): + self.token = "fresh" + return self.token + + def create_link(self, cmd): + return "http://prov.example/live/9.ts?token=fresh" + + original_portal, original_redis = s.Portal, s.get_redis + s.Portal, s.get_redis = Answering, lambda: client + try: + link, _, token = resolver.resolve(CFG.slug, "ffmpeg http://localhost/ch/9_") + finally: + s.Portal, s.get_redis = original_portal, original_redis + + assert link.endswith("token=fresh") and token == "fresh" + + def test_the_fallback_command_survives_too(): reset() client = FakeRedis() From c8907978dbd4642f0038751595714e372139ecb2 Mon Sep 17 00:00:00 2001 From: PiloUnk <198624632+PiloUnk@users.noreply.github.com> Date: Mon, 24 Aug 2026 23:54:01 +0200 Subject: [PATCH 08/10] Stop handing a subscription's credentials back to whoever opens the page The Portals box was the one place a MAC, a password and the rest of the account identity were rendered back to the user, and Dispatcharr serves a plugin's settings row to every account on the install -- so they were also in an API response anyone could read. The row now holds a redacted rendering and the real list lives only in portals.txt, which the panel cannot reach. Redaction is a property of the write path alone: _reconcile_registry undoes it on the way in, so every migration, action and sync goes on reading plain lines. Names, URLs and the tuning keys stay readable. A box of nothing but bullets is one nobody can recognise their own portals in, and the rule is what a value proves rather than what it configures. Two guards decide the whole thing. Nothing is redacted until portals.txt is known to hold the same list -- hiding the only copy of a credential is how a configuration gets lost -- and redacted text is never mirrored back over that file, because _failed() can be reached with settings that never passed through the reconciliation. This hides the credentials; it does not encrypt them. The resolver reads the MAC on every tune in a process with no Django, so portals.txt and the state mirrors go on holding it in the clear at 0600, and the README says so. Co-Authored-By: Claude Opus 5 --- CONTRIBUTING.md | 38 +++++++ README.md | 20 +++- plugin.json | 2 +- plugin.py | 114 +++++++++++++++++++-- stalker_api.py | 228 +++++++++++++++++++++++++++++++++++++++++ tests/test_config.py | 105 +++++++++++++++++++ tests/test_manifest.py | 5 + tests/test_registry.py | 132 +++++++++++++++++++++++- 8 files changed, 627 insertions(+), 17 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 5baf545..ab1e92c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -284,6 +284,44 @@ Consequences that shape the code: (`Plugin._report`). Anything more urgent goes to Dispatcharr's notification centre, which does reach an open browser (`sync.announce`). +### The panel is shown a redacted list + +The settings row is served to every account on the install and painted straight +into a textarea, so it is the one copy of the portal list that must not hold a +credential. `Plugin._save_settings` therefore writes two different things: the +whole list to `portals.txt`, and `stalker_api.mask_portals()` of it to the row. +The MAC, `username`, `password`, `device_id`, `device_id2`, `serial` and +`signature` become `••••`; names, URLs and the tuning keys stay, because a box +of nothing but bullets is one nobody can recognise their own portals in. + +Redaction is a property of the **write path and nothing else**. +`_reconcile_registry` runs `unmask_portals()` on whatever the panel sent and +always returns the real list, so every migration, action and sync downstream +goes on reading plain lines and never has to know. + +Four things are easy to break here: + +- **Never mirror redacted text.** `_save_settings` checks `is_masked()` before + writing `portals.txt`, because `_failed()` can be reached with settings that + never passed through `_reconcile_registry` — and mirroring a row of tokens + would write them over the only copy of the credentials. +- **Never redact before the file holds the list.** The guard is + `digest(stored) == digest(text)`. A registry that could not be written leaves + the row as the only copy there is, and hiding the only copy loses it. +- **The token stands where the MAC stands.** `split_portal_line` decides which + field is which by *where the MAC sits*, so `_mac_index()` counts the token as + one; without that, redacting an unnamed line's MAC shifts its fields by one. + For the same reason `mask_portals` writes the derived name out. +- **A redacted line is paired back up by slug, then by URL.** Renaming a portal + and repointing it are both ordinary edits, and each changes the half the other + is recognised by. Change both at once and nothing matches: the token survives + `unmask_portals`, and `Plugin._portals` quotes the line back and asks for it + to be retyped rather than parsing a MAC address made of bullets. + +None of this is encryption at rest, and the README says so: the resolver reads +the MAC on every tune with no Django, so `portals.txt` and the state mirrors go +on holding it in the clear at `0600`. + ### Actions `plugin.json` is the single definition of the UI; `plugin.py` reads it at import diff --git a/README.md b/README.md index 4ad24fb..5d9eec6 100644 --- a/README.md +++ b/README.md @@ -59,6 +59,7 @@ A plugin that writes credentials to disk should say so plainly: | --- | --- | | `/data/uploads/m3us/distalker-.m3u` | The generated playlist | | `/data/distalker/portals.txt` | Your portal list verbatim, **credentials included** | +| Dispatcharr's plugin settings row | The same list with every credential **redacted** — this is the copy the panel renders and the API serves | | `/data/distalker/state/*.json` | What the resolver reads at tune time, **credentials included**, `0600` | | Redis `distalker:*` | The same, plus the session token | @@ -183,9 +184,16 @@ further `|` characters, quoted where a value contains spaces > Raise it only on what your provider told you: exceeding it is the quickest > route to a blocked MAC. -> **Credentials are visible in this box**, and stored unencrypted in the -> Dispatcharr database like every plugin setting. Treat your backups -> accordingly. +> **Credentials are hidden once saved.** The MAC, the password and the rest of +> the account identity come back as `••••` the moment the list is stored: the +> real line lives in `/data/distalker/portals.txt`, which the settings panel +> cannot read. Edit around the bullets and what you leave alone is left alone — +> renaming a portal or repointing its URL both work with the MAC still hidden. +> +> **Keep your own copy of each portal line in a password manager.** The box will +> not give a credential back, and if `/data/distalker` is ever lost the only +> remaining copy goes with it. Hidden is also not encrypted — see +> [What it writes, and where](#what-it-writes-and-where). ### STB identity @@ -392,8 +400,10 @@ deleting it. a stock Dispatcharr can run, so *Refresh every (hours)* drives the M3U accounts' own refresh interval and answers the event that follows. It works, and it is why the interval cannot usefully go below an hour. -- **Credentials are stored unencrypted**, in the Dispatcharr database and on - disk — see [What it writes, and where](#what-it-writes-and-where). +- **Credentials are stored unencrypted** on disk. The settings panel no longer + shows them, but `/data/distalker/portals.txt` and the state mirrors hold them + in the clear at `0600`: the resolver reads them on every tune, in a process + with no database — see [What it writes, and where](#what-it-writes-and-where). - **No session keep-alive.** A cached token is reused and re-issued on demand. Portals that drop idle sessions are untested. - **One `ffmpeg` per tuned channel**, which is normal for any non-proxy stream diff --git a/plugin.json b/plugin.json index 27325d0..fdbe3c8 100644 --- a/plugin.json +++ b/plugin.json @@ -19,7 +19,7 @@ "type": "text", "default": "", "placeholder": "http://portal.example.com:8080/c/ | 00:1A:79:AA:BB:CC", - "help_text": "Portal URL | MAC address, one line each. The name is taken from the host, so put one in front only if you want a different label -- or if two portals share a host, which the sync will then ask you to do. Trailing key=value pairs cover the rest: username, password, max_streams, model, serial, device_id, device_id2, timezone, and epg=1 to fetch this portal's programme guide (epg_hours=48 for more than a day -- a guide is by far the largest thing a sync downloads, which is why it is off unless asked for). A line starting with '#' is ignored, which is how you suspend a portal without losing its channels. Credentials are stored unencrypted and are visible in this box." + "help_text": "Portal URL | MAC address, one line each. The name is taken from the host, so put one in front only if you want a different label -- or if two portals share a host, which the sync will then ask you to do. Trailing key=value pairs cover the rest: username, password, max_streams, model, serial, device_id, device_id2, timezone, and epg=1 to fetch this portal's programme guide (epg_hours=48 for more than a day -- a guide is by far the largest thing a sync downloads, which is why it is off unless asked for). A line starting with '#' is ignored, which is how you suspend a portal without losing its channels. Once saved, the MAC, the password and the rest of the account identity are replaced by bullets here and kept in a file only this plugin reads; edit around the bullets and what you leave alone stays as it was. That hides them from this page, it does not encrypt them -- and the box will not give one back, so keep your own copy of each line in a password manager." }, { "id": "status", diff --git a/plugin.py b/plugin.py index e0e430a..1caeba6 100644 --- a/plugin.py +++ b/plugin.py @@ -42,13 +42,16 @@ claim_auto_sync, forget_portal, format_portal_line, + is_masked, is_superseded_ffmpeg_args, load_portal, + mask_portals, parse_portals, published_slugs, save_portal, split_portal_line, sync_lock_age, + unmask_portals, ) from .sync import ( @@ -329,18 +332,42 @@ def _write_settings(self, settings: Dict[str, Any]) -> None: cfg.save(update_fields=["settings", "updated_at"]) def _save_settings(self, settings: Dict[str, Any]) -> None: - """Persist settings the plugin changed itself.""" - self._write_settings(settings) + """Persist settings the plugin changed itself. + The portal list is stored twice and deliberately not the same way. The + registry file takes it whole, because the next action and the resolver + need the credentials; the settings row takes the redacted rendering, + because Dispatcharr serves that row to every account on the install and + the panel paints it straight into a textarea. + + ``settings`` always carries the real list here -- redaction happens on + the way out and nowhere else, so every caller in between goes on + reading and rewriting plain lines. + """ # Mirror the list somewhere the settings panel cannot reach, and flag it # as a write the open panel has no way of knowing about. Saves that # leave the list alone -- recording a status, clearing the form -- skip # this entirely: rewriting the same text would renew the marker and go # on distrusting a panel that is in fact still in step with the list. + # + # Redacted text is never mirrored. _failed() can be reached with + # settings that never went through _reconcile_registry -- an error + # raised inside it, for one -- and mirroring a row full of tokens would + # write the tokens over the only copy of the credentials. text = settings.get("portals") or "" - if digest(text) != digest(load_registry()): + if not is_masked(text) and digest(text) != digest(load_registry()): save_registry(text, pending=True) + stored = load_registry() + row = dict(settings) + # Redact only once the file is known to hold the same list. A registry + # that could not be written leaves this row as the only copy there is, + # and hiding the only copy is how a configuration gets lost. + if stored is not None and digest(stored) == digest(text): + row["portals"] = mask_portals(text) + + self._write_settings(row) + @staticmethod def _worth_recording(params: Dict[str, Any], result: Dict[str, Any]) -> bool: """Whether this run should overwrite the Last action box. @@ -397,11 +424,21 @@ def _reconcile_registry(self, settings: Dict[str, Any], logger) -> Dict[str, Any what the plugin wrote, so the file wins. Once the panel quotes the current list back -- which happens as soon as it is reopened -- the marker clears and the textarea is authoritative again. + + Returns settings carrying the *real* list in every case, whatever the + row and the panel hold. Everything downstream -- the migrations, the + actions, the sync -- reads plain lines and never has to know that the + stored copy is redacted. """ stored = load_registry() raw = self._raw_settings() panel_value = raw.get("portals") + def adopt(text: str) -> Dict[str, Any]: + updated = dict(settings) + updated["portals"] = text or "" + return updated + if panel_value is None: if stored: logger.info( @@ -409,14 +446,36 @@ def _reconcile_registry(self, settings: Dict[str, Any], logger) -> Dict[str, Any "restoring it from %s", REGISTRY_PATH, ) - settings = dict(settings) - settings["portals"] = stored + settings = adopt(stored) self._save_settings(settings) return settings - if (panel_value or "").strip() == (stored or "").strip(): + # The panel can only send back what it was shown, and what it was shown + # is redacted. Undo that before anything compares or parses it. + plain = unmask_portals(panel_value, stored or "") + + # "Unchanged" is judged against whatever the panel was actually holding. + # A redacted list has to be compared with the redaction: a line that + # never named its portal comes back with the derived name written out, + # and that is the rendering's doing rather than an edit. Comparing the + # plain text instead would read every reopened panel as a hand edit and + # rewrite the file for nothing. The redaction is *not* used to judge a + # list typed in full: it hides the MAC, so a hand-corrected MAC would + # compare equal to the one it replaced. + if is_masked(panel_value): + unchanged = panel_value.strip() == mask_portals(stored or "").strip() + else: + unchanged = plain.strip() == (stored or "").strip() + + if unchanged: # The panel has caught up, so whatever it sends next can be trusted. clear_pending() + settings = adopt(stored) + # An install upgrading into this version still has its credentials + # in the settings row, and a panel that never edits the box would + # never trigger a save. This is where those rows get redacted, once. + if panel_value != mask_portals(stored or ""): + self._save_settings(settings) return settings if is_pending(): @@ -426,19 +485,36 @@ def _reconcile_registry(self, settings: Dict[str, Any], logger) -> Dict[str, Any "panel to edit the list by hand.", REGISTRY_PATH, ) - settings = dict(settings) - settings["portals"] = stored or "" + settings = adopt(stored) self._save_settings(settings) return settings + # A token that survived unmask_portals() belongs to a line nothing in + # the registry answers for -- a lost /data/distalker, or a line renamed + # *and* moved in one edit. Adopting it would write bullets over the + # only copy of the credentials, so the list is left exactly as it is + # and _portals() names the offending lines and refuses. + if is_masked(plain): + logger.error( + "distalker: the settings panel sent back portal lines whose " + "credentials are not in %s; leaving the stored list alone", + REGISTRY_PATH, + ) + return adopt(plain) + # Hand-edited in the textarea, or the very first run. - if not save_registry(panel_value): + settings = adopt(plain) + if not save_registry(plain): logger.warning( "distalker: could not write %s; the portal list is only " "stored in the plugin settings and may be lost", REGISTRY_PATH, ) + # Left in the row as it came, credentials and all: with no file to + # read them back from, redacting them here would erase them. + return settings + self._save_settings(settings) return settings # -- actions ---------------------------------------------------------- @@ -449,7 +525,25 @@ def _portals(self, settings: Dict[str, Any]) -> List[PortalConfig]: Partially applying a mistyped config is worse than doing nothing: it would leave half the accounts pointing at stale files. """ - portals, errors = parse_portals(settings.get("portals") or "") + text = settings.get("portals") or "" + + # A line still holding a redaction token got here without its + # credentials, and _reconcile_registry could not find them: either the + # registry is gone, or the line was renamed *and* moved in one edit, + # which leaves nothing to recognise it by. Parsing on would report a + # MAC address made of bullets, which explains nothing. + if is_masked(text): + lines = [ + line.strip() for line in text.splitlines() if is_masked(line) + ] + raise PortalError( + "these portal lines are still hidden and their credentials " + f"could not be read back from {REGISTRY_PATH}: " + + "; ".join(lines) + + " -- retype each of them in full, URL, MAC and password" + ) + + portals, errors = parse_portals(text) if errors: raise PortalError("invalid portal configuration -- " + "; ".join(errors)) if not portals: diff --git a/stalker_api.py b/stalker_api.py index 5302575..a93d4fa 100644 --- a/stalker_api.py +++ b/stalker_api.py @@ -1096,6 +1096,234 @@ def normalize_portal_url(url: str) -> str: return parsed._replace(path=path).geturl() +# -- redaction -------------------------------------------------------------- +# +# The Portals box was the one place a subscription's credentials were rendered +# back to whoever opened the page, and Dispatcharr serves a plugin's settings +# row to every account on the install. So the row keeps a redacted rendering of +# the list, and the real one lives only in the registry file, which the panel +# cannot reach -- see registry.py. +# +# What this buys is that the credentials are no longer on the screen or in the +# API response. It is not encryption at rest and must not be sold as such: the +# resolver reads the MAC on every tune, in a process with no Django, so +# portals.txt and the state mirrors go on holding it in the clear at 0600. + +# U+2022, because the token has to be something no URL, MAC or key=value could +# be, and something nobody types into the box by accident. An ASCII '****' is a +# perfectly plausible password. +MASK = "•" * 4 + +# Redacted wherever they appear as key=value. The MAC is redacted too, but it +# is positional and handled apart. The rule is what a value *proves* rather +# than what it configures: anything a stranger could authenticate with is +# hidden, while model, timezone, max_streams and the epg switches stay +# readable -- they tune behaviour, and they are what lets a user recognise +# their own line. +SECRET_KEYS = ("username", "password", "device_id", "device_id2", + "serial", "signature") + + +def is_masked(text: Optional[str]) -> bool: + """True if a portal list carries redacted values rather than real ones.""" + return MASK in (text or "") + + +def _line_body(line: str) -> Tuple[bool, str]: + """Separate a line's comment marker from the portal line inside it. + + A '#' line is how the help text tells users to suspend a portal without + losing its channels, so it holds a real MAC and has to be redacted like any + other -- and put back together the same way. + """ + stripped = line.strip() + if stripped.startswith("#"): + return True, stripped.lstrip("#").strip() + return False, stripped + + +def _mac_index(parts: List[str]) -> int: + """Which '|' field holds the MAC, by split_portal_line's own rule. + + The token counts as a MAC here. Without that, redacting the MAC would move + the fields of an unnamed line: 'url | MAC | extras' reads correctly only + because the second field looks like a MAC, and 'url | | extras' + would otherwise be read as name, url and MAC. + """ + if len(parts) >= 3 and not ( + MAC_RE.match(normalize_mac(parts[1])) or parts[1].strip() == MASK + ): + return 2 + return 1 + + +def _line_slug(line: str) -> str: + """The slug a portal line is filed under, tolerating a redacted MAC. + + This is what pairs a redacted line back up with the real one it was made + from, so the two have to agree even though one of them no longer has a MAC. + """ + parts = [p.strip() for p in _line_body(line)[1].split("|")] + if len(parts) < 2: + return "" + at = _mac_index(parts) + name = parts[0] if at == 2 else "" + return slugify(name or name_from_url(parts[at - 1])) + + +def _mask_extras(segment: str) -> str: + """Redact the secrets among one segment's key=value pairs.""" + try: + tokens = shlex.split(segment) + except ValueError: + return segment + out = [] + for token in tokens: + key, sep, value = token.partition("=") + if not sep: + out.append(token) + elif value and key.strip().lower() in SECRET_KEYS: + out.append(f"{key}={MASK}") + else: + out.append(f"{key}={_quote_if_needed(value)}") + return " ".join(out) + + +def _unmask_extras(segment: str, extras: Dict[str, str]) -> str: + """Put the secrets back into one segment's key=value pairs.""" + if MASK not in segment: + return segment + try: + tokens = shlex.split(segment) + except ValueError: + return segment + out = [] + for token in tokens: + key, sep, value = token.partition("=") + if sep and value.strip() == MASK: + restored = extras.get(key.strip().lower(), "") + # Nothing to restore leaves the token standing: see unmask_portals. + out.append(f"{key}={_quote_if_needed(restored)}" if restored + else f"{key}={MASK}") + elif sep: + out.append(f"{key}={_quote_if_needed(value)}") + else: + out.append(token) + return " ".join(out) + + +def mask_portals(text: str) -> str: + """Render a portal list with every credential replaced by :data:`MASK`. + + Everything that is not a credential survives -- names, URLs, comments, the + tuning keys -- so the box still reads as the user's own configuration, and + a portal is still deleted by deleting its line. + + The name is written out even where the line derived it from the URL: it is + the identity :func:`unmask_portals` pairs the line back up by, and writing + it also settles where the MAC was once the MAC is gone. + """ + out = [] + for raw in text.splitlines(): + commented, body = _line_body(raw) + if not body: + out.append(raw) + continue + + parsed, _ = split_portal_line(body) + if parsed is None: + # A line that does not parse holds no credential we could find, and + # the user has to go on seeing it to be able to fix it. + out.append(raw) + continue + + parts = [p.strip() for p in body.split("|")] + at = _mac_index(parts) + parts[at] = MASK + parts[at + 1:] = [_mask_extras(part) for part in parts[at + 1:]] + if at == 1: + parts.insert(0, parsed["name"]) + # format_portal_line() always writes an extras field, empty or not. + while len(parts) > 3 and not parts[-1]: + parts.pop() + + line = " | ".join(parts) + out.append(f"# {line}" if commented else line) + + return "\n".join(out) + ("\n" if text.endswith("\n") else "") + + +def unmask_portals(text: str, stored: str) -> str: + """Put the credentials back into a list the panel sent back redacted. + + ``stored`` is the registry's copy, the only place the real values exist. + Lines are paired with it by slug, and failing that by URL: renaming a + portal and moving one are both ordinary edits, and either would otherwise + orphan the line's own MAC. A line typed out in full carries no token and is + returned exactly as typed, so pasting a list back in still works. + + A token nothing matches is left standing rather than resolved to an empty + value: Plugin._portals refuses a list that still holds one and says which + line to retype, which is a great deal easier to act on than a portal + quietly authenticating with nothing. + """ + if not is_masked(text): + return text + + records = [] + for raw in (stored or "").splitlines(): + body = _line_body(raw)[1] + if not body: + continue + parsed, _ = split_portal_line(body) + if parsed is None: + continue + records.append({ + "slug": _line_slug(body), + "url": normalize_portal_url(parsed["url"]), + "parsed": parsed, + "used": False, + }) + + def take(slug: str, url: str) -> Optional[Dict[str, Any]]: + """The stored line this redacted one came from, consumed once. + + Consumed, because a suspended line and its replacement can share both + slug and host -- that is what commenting a line out is for -- and the + second of them must not be handed the first one's credentials. + """ + for field, wanted in (("slug", slug), ("url", url)): + if not wanted: + continue + for record in records: + if not record["used"] and record[field] == wanted: + record["used"] = True + return record["parsed"] + return None + + out = [] + for raw in text.splitlines(): + commented, body = _line_body(raw) + if MASK not in body: + out.append(raw) + continue + + parts = [p.strip() for p in body.split("|")] + at = _mac_index(parts) + source = take(_line_slug(body), normalize_portal_url(parts[at - 1])) + if source is not None: + if parts[at] == MASK: + parts[at] = source["mac"] + parts[at + 1:] = [ + _unmask_extras(part, source["extras"]) for part in parts[at + 1:] + ] + + line = " | ".join(parts) + out.append(f"# {line}" if commented else line) + + return "\n".join(out) + ("\n" if text.endswith("\n") else "") + + def endpoint_candidates(url: str) -> List[str]: """Where a Stalker API might answer for this URL, best guess first. diff --git a/tests/test_config.py b/tests/test_config.py index 8eccdde..15bd6da 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -357,6 +357,111 @@ def test_config_survives_redis_roundtrip(): assert s.PortalConfig.from_dict(portal.to_dict()) == portal +# -- redaction: what the settings panel is allowed to see --------------------- + +STORED = ( + "Salon | http://a.example/c/ | 00:1A:79:00:00:01 | username=joe password=hunter2\n" + "http://b.example/c/ | 00:1A:79:00:00:02 | epg=1 model=MAG270 serial=98765\n" + "# Suspendu | http://c.example/c/ | 00:1A:79:00:00:03\n" +) + + +def test_no_credential_survives_the_redaction(): + """The whole point: nothing in the box lets anyone use the subscription.""" + masked = s.mask_portals(STORED) + for secret in ("00:1A:79:00:00:01", "00:1A:79:00:00:02", "00:1A:79:00:00:03", + "joe", "hunter2", "98765"): + assert secret not in masked, f"{secret} is still readable" + + +def test_what_is_not_a_credential_stays_readable(): + """A box of nothing but bullets is one nobody can recognise their own + portals in, so everything that only tunes behaviour is left alone.""" + masked = s.mask_portals(STORED) + for kept in ("Salon", "http://a.example/c/", "epg=1", "model=MAG270", "# "): + assert kept in masked, f"{kept} should not have been hidden" + + +def test_the_redaction_means_exactly_the_same_thing_once_undone(): + """A round trip has to be invisible to everything downstream, comments and + all -- a suspended line holds a real MAC and is how a portal is paused.""" + restored = s.unmask_portals(s.mask_portals(STORED), STORED) + before, _ = s.parse_portals(STORED) + after, errors = s.parse_portals(restored) + assert not errors + assert [p.to_dict() for p in before] == [p.to_dict() for p in after] + # Not compared as text: a redacted line writes its derived name out. What + # has to survive is what the line means, and that a paused portal is still + # paused rather than quietly back in the line-up. + assert "# Suspendu | http://c.example/c/ | 00:1A:79:00:00:03" in restored + assert [p.slug for p in after] == ["salon", "b"] + + +def test_a_line_typed_out_in_full_is_taken_as_typed(): + """Pasting the list back in from a password manager has to keep working.""" + added = s.mask_portals(STORED) + "New | http://d.example/c/ | 00:1A:79:00:00:04\n" + restored = s.unmask_portals(added, STORED) + assert "New | http://d.example/c/ | 00:1A:79:00:00:04" in restored + assert not s.is_masked(restored) + + +def test_a_renamed_or_moved_portal_keeps_its_own_credentials(): + """Renaming a portal and repointing it are both ordinary edits, and either + one changes the half of the line the other is recognised by. So both are + tried: the slug first, the URL after it.""" + masked = s.mask_portals(STORED) + + renamed = s.unmask_portals(masked.replace("Salon |", "Sejour |"), STORED) + assert "Sejour | http://a.example/c/ | 00:1A:79:00:00:01" in renamed + assert "password=hunter2" in renamed + + moved = s.unmask_portals(masked.replace("http://a.example/c/", "http://z.example/c/"), + STORED) + assert "Salon | http://z.example/c/ | 00:1A:79:00:00:01" in moved + + +def test_a_line_nothing_recognises_stays_hidden_rather_than_emptied(): + """Renaming *and* moving a line in one edit leaves nothing to pair it up + by. Leaving the token standing is what lets the plugin name the line and + ask for it again; an empty MAC would authenticate as nobody and say so in + a message about the portal instead of about the edit.""" + masked = s.mask_portals(STORED) + orphan = masked.replace("Salon | http://a.example/c/", "Sejour | http://z.example/c/") + restored = s.unmask_portals(orphan, STORED) + assert s.is_masked(restored) + assert "00:1A:79:00:00:02" in restored, "the other lines still come back" + + +def test_deleting_a_hidden_line_still_deletes_the_portal(): + """The editing model has to survive the redaction: a line is still a portal + and removing it still removes one.""" + masked = s.mask_portals(STORED) + kept = "\n".join(masked.splitlines()[1:]) + "\n" + portals, errors = s.parse_portals(s.unmask_portals(kept, STORED)) + assert not errors + assert [p.slug for p in portals] == ["b"] + + +def test_the_name_is_written_out_so_the_mac_field_stays_findable(): + """split_portal_line() decides which field is which by where the MAC sits. + Redact the MAC on a line that never named its portal and the fields shift + by one, so the redaction writes the derived name rather than lose them.""" + masked = s.mask_portals("http://b.example/c/ | 00:1A:79:00:00:02 | epg=1\n") + assert masked == f"b | http://b.example/c/ | {s.MASK} | epg=1\n" + + +def test_an_unparseable_line_is_left_where_the_user_can_see_it(): + masked = s.mask_portals("this is not a portal line\n") + assert masked == "this is not a portal line\n" + + +def test_the_token_is_not_something_anyone_types_by_accident(): + """A password of four asterisks is plausible; four bullets are not.""" + assert s.MASK == "\u2022" * 4 + assert not s.is_masked(STORED) + assert s.is_masked(s.mask_portals(STORED)) + + if __name__ == "__main__": failures = 0 for name, fn in sorted(globals().items()): diff --git a/tests/test_manifest.py b/tests/test_manifest.py index 45480f7..d07b115 100644 --- a/tests/test_manifest.py +++ b/tests/test_manifest.py @@ -113,6 +113,11 @@ def publish(line): def run(p, action, settings): + # The panel saves the box before running anything -- PluginCard.jsx says so + # in as many words -- so the stored row and the context always hold the same + # list. Writing it here keeps the two in step; a test handing run() a new + # list is simulating a user who has just typed it into the textarea. + p._write_settings(dict(settings)) return p.run(action, {}, {"settings": dict(settings), "logger": NullLogger()}) diff --git a/tests/test_registry.py b/tests/test_registry.py index b6c6b98..59665d4 100644 --- a/tests/test_registry.py +++ b/tests/test_registry.py @@ -93,7 +93,9 @@ def test_absent_key_means_clobbered_and_is_restored(): result = p._reconcile_registry(merged, NullLogger()) assert result["portals"] == PORTAL, "the clobbered list must come back" - assert store["settings"]["portals"] == PORTAL, "and be written back to the DB" + assert store["settings"]["portals"] == s.mask_portals(PORTAL), ( + "and be written back to the DB, redacted -- the row is what the API serves" + ) portals, errors = s.parse_portals(result["portals"]) assert not errors and [x.name for x in portals] == ["livingroom"] @@ -247,6 +249,134 @@ def test_a_marker_left_over_from_an_outside_edit_is_ignored(): assert not registry.is_pending(), "the marker no longer describes the file" +# -- redaction: the row the API serves is not the list the plugin works from -- + +def test_the_settings_row_never_holds_a_credential(): + """Dispatcharr serves the settings row to every account on the install, and + the panel paints it into a textarea. Neither is a place for a MAC.""" + reset() + p, store = make_plugin({}) + p._save_settings({"portals": PORTAL}) + + assert "00:1A:79:AA:BB:CC" not in store["settings"]["portals"] + assert s.is_masked(store["settings"]["portals"]) + assert registry.load_registry() == PORTAL, "the file keeps the real thing" + + +def test_the_panel_sending_the_redaction_back_changes_nothing(): + """The panel can only return what it was shown. That must read as 'no + change', not as the user having replaced every MAC with bullets.""" + reset() + registry.save_registry(PORTAL) + masked = s.mask_portals(PORTAL) + + p, _ = make_plugin({"portals": masked}) + result = p._reconcile_registry({"portals": masked}, NullLogger()) + + assert result["portals"] == PORTAL, "handlers must be given the real list" + assert registry.load_registry() == PORTAL, "and the file must be untouched" + assert not registry.is_pending(), "the panel is in step with the file" + + +def test_reopening_the_panel_does_not_rewrite_the_file(): + """A line that never named its portal comes back from the redaction with + the derived name written out, because that is what makes the MAC's position + findable. That is the rendering's doing, not an edit, and reading it as one + would rewrite the user's file on every click.""" + reset() + unnamed = "http://portal.example/c/ | 00:1A:79:AA:BB:CC | epg=1\n" + registry.save_registry(unnamed) + shown = s.mask_portals(unnamed) + assert shown.startswith("portal | "), "the derived name is written out" + + p, _ = make_plugin({"portals": shown}) + result = p._reconcile_registry({"portals": shown}, NullLogger()) + + assert result["portals"] == unnamed + assert registry.load_registry() == unnamed, "the file must be left alone" + assert not registry.is_pending() + + +def test_an_edit_made_through_the_redaction_keeps_the_hidden_half(): + """Changing a portal's URL while its MAC is hidden is the ordinary case, + and the MAC the user cannot see must survive it.""" + reset() + registry.save_registry(PORTAL) + + edited = s.mask_portals(PORTAL).replace("http://portal.example/c/", + "http://moved.example/c/") + p, store = make_plugin({"portals": edited}) + result = p._reconcile_registry({"portals": edited}, NullLogger()) + + assert "http://moved.example/c/" in result["portals"], "the edit must stick" + assert "00:1A:79:AA:BB:CC" in result["portals"], "and the MAC must come back" + assert registry.load_registry() == result["portals"] + assert s.is_masked(store["settings"]["portals"]), "the row stays redacted" + + +def test_an_upgrade_hides_credentials_the_row_already_holds(): + """Installs coming from an earlier version have their MAC in the row. A + panel that only ever presses Sync never edits the box, so nothing else + would ever rewrite it.""" + reset() + registry.save_registry(PORTAL) + + p, store = make_plugin({"portals": PORTAL}) # as an older version left it + result = p._reconcile_registry({"portals": PORTAL}, NullLogger()) + + assert result["portals"] == PORTAL + assert s.is_masked(store["settings"]["portals"]), "the row must be redacted now" + assert registry.load_registry() == PORTAL + + +def test_nothing_is_hidden_until_the_file_is_known_to_hold_it(): + """Redacting is only safe once there are two copies. A registry that could + not be written leaves the row as the only one there is, and hiding the only + copy of a credential is how a configuration gets lost.""" + reset() + original = plugin_mod.save_registry + plugin_mod.save_registry = lambda text, pending=False: False + try: + p, store = make_plugin({"portals": PORTAL}) + result = p._reconcile_registry({"portals": PORTAL}, NullLogger()) + assert result["portals"] == PORTAL + assert store["settings"]["portals"] == PORTAL, "still the only copy" + finally: + plugin_mod.save_registry = original + + +def test_a_redaction_is_never_written_over_the_real_list(): + """The panel holds bullets and the file that explains them is gone -- a + recreated volume, a partial restore. Adopting what the panel sends would + make the loss permanent by writing the bullets into the file.""" + reset() + masked = s.mask_portals(PORTAL) + + p, store = make_plugin({"portals": masked}) + result = p._reconcile_registry({"portals": masked}, NullLogger()) + + assert s.is_masked(result["portals"]), "nothing can fill these back in" + assert registry.load_registry() is None, "and nothing may be written" + assert store["settings"]["portals"] == masked, "the row is left as it was" + + +def test_a_line_that_could_not_be_filled_back_in_is_named_and_refused(): + """Parsing on would report a MAC address made of bullets, which explains + nothing. The line itself is quoted back instead, because retyping it is + the only thing that fixes this.""" + reset() + masked = s.mask_portals(PORTAL) + p, _ = make_plugin({}) + try: + p._portals({"portals": masked}) + except plugin_mod.PortalError as exc: # plugin.py holds its own import + assert "livingroom" in str(exc), exc + assert registry.REGISTRY_PATH in str(exc), exc + assert "retype" in str(exc), exc + else: + raise AssertionError("a redacted list must not be parsed") + + if __name__ == "__main__": failures = 0 try: From 0c81c764896f7ccad7c13a0613fe3bd2be6c897e Mon Sep 17 00:00:00 2001 From: PiloUnk <198624632+PiloUnk@users.noreply.github.com> Date: Tue, 25 Aug 2026 00:42:27 +0200 Subject: [PATCH 09/10] Stop serving settings the panel stopped having a field for Taking a field out of plugin.json only stops the panel rendering it. The value stays in PluginConfig.settings, which Dispatcharr serves to every account on the install -- so the Add-portal form removed in 0.4.0 left a MAC, a password and a portal URL in the API response for a month, belonging to a portal the user had long since deleted. sync_interval_hours went the same way with the schedule. Redacting the portal list and leaving those behind would have been half a job, so anything the manifest does not declare is now dropped from the row. The manifest is the single definition of the panel, and test_manifest.py already pins that every setting the code reads is declared, so the set is safe by construction. Ordering matters: it runs after the migrations, whose whole input is keys the manifest stopped declaring three versions ago and which would otherwise be taken out from under them. And it runs on every action rather than once, because an open panel replays the stale keys until the page is reloaded. Co-Authored-By: Claude Opus 5 --- plugin.py | 37 +++++++++++++++++++++++++++++++++ tests/test_manifest.py | 46 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 83 insertions(+) diff --git a/plugin.py b/plugin.py index 1caeba6..5d27a30 100644 --- a/plugin.py +++ b/plugin.py @@ -131,6 +131,7 @@ def run(self, action: str, params: Dict[str, Any], context: Dict[str, Any]) -> D settings = self._reconcile_registry(settings, logger) settings = self._migrate_legacy_globals(settings, logger) settings = self._migrate_ffmpeg_args(settings, logger) + settings = self._drop_unknown_settings(settings, logger) self._drop_legacy_schedule(logger) result = handler(params or {}, settings, logger) @@ -296,6 +297,41 @@ def _migrate_ffmpeg_args(self, settings: Dict[str, Any], logger) -> Dict[str, An self._save_settings(updated) return updated + def _drop_unknown_settings(self, settings: Dict[str, Any], logger) -> Dict[str, Any]: + """Delete stored settings no field in the manifest declares any more. + + Taking a field out of plugin.json only stops the panel *rendering* it. + The value stays in PluginConfig.settings, which Dispatcharr serves to + every account on the install -- so the Add-portal form that went in + 0.4.0 left a MAC, a password and a portal URL sitting in the API + response for months, belonging to a portal the user had since dropped. + Redacting the portal list and leaving those behind would have been + half a job. + + The manifest is the single definition of the panel, so anything it does + not declare is dead by construction; test_manifest.py pins that every + setting the code reads is declared. Runs after the migrations, which + read keys of their own that this would otherwise take first. + + The open panel goes on replaying the stale keys until the page is + reloaded -- it PUTs the state it fetched, same as with the portal list + -- so this runs on every action rather than once. + """ + known = {field["id"] for field in self.fields} + stale = sorted(key for key in self._raw_settings() if key not in known) + if not stale: + return settings + + updated = {k: v for k, v in settings.items() if k in known} + logger.info( + "distalker: dropped %d stored setting(s) the panel no longer has a " + "field for: %s", + len(stale), + ", ".join(stale), + ) + self._save_settings(updated) + return updated + def _settings_with_defaults(self) -> Dict[str, Any]: """Stored settings over the manifest's defaults. @@ -841,6 +877,7 @@ def run_sync_now(self, full: bool = False, logger=None) -> Dict[str, Any]: settings = self._reconcile_registry(settings, logger) settings = self._migrate_legacy_globals(settings, logger) settings = self._migrate_ffmpeg_args(settings, logger) + settings = self._drop_unknown_settings(settings, logger) try: result = self._sync_portals(settings, logger, full=full) diff --git a/tests/test_manifest.py b/tests/test_manifest.py index d07b115..d55e1cc 100644 --- a/tests/test_manifest.py +++ b/tests/test_manifest.py @@ -228,6 +228,52 @@ def test_settings_the_code_reads_are_declared(): assert key in ids, f"the code reads '{key}' but the panel never renders it" +def test_settings_the_panel_no_longer_has_a_field_for_are_dropped(): + """Taking a field out of the manifest stops the panel rendering it and + nothing else. The Add-portal form went in 0.4.0 and its values stayed in + the settings row -- a MAC, a password and a URL that Dispatcharr went on + serving to every account on the install, for a portal long since deleted. + """ + reset() + p, store, _ = make_plugin({ + "portals": PORTALS, + "new_url": "http://gone.example:8080/c/", + "new_mac": "00:00:00:00:00:00", + "new_password": "hunter2", + "sync_interval_hours": 12, + }) + try: + with stubbed(): + run(p, "apply_profile", store["settings"]) + finally: + store["restore"]() + + for dead in ("new_url", "new_mac", "new_password", "sync_interval_hours"): + assert dead not in store["settings"], f"{dead} is still being served" + assert "portals" in store["settings"], "the declared ones must stay" + assert store["settings"]["status"], "and so must what the action reported" + + +def test_a_migration_still_gets_the_keys_it_reads(): + """The prune must not run before _migrate_legacy_globals, whose whole input + is settings the manifest stopped declaring three versions ago.""" + reset() + p, store, _ = make_plugin({ + "portals": PORTALS, + "stb_device_id": "a" * 64, + }) + try: + with stubbed(): + run(p, "apply_profile", store["settings"]) + finally: + store["restore"]() + + assert "stb_device_id" not in store["settings"], "the legacy key is gone" + assert "device_id=" + "a" * 64 in registry.load_registry(), ( + "and was folded onto the portal line before being dropped" + ) + + # -- what run() does around a handler ----------------------------------------- def test_an_action_records_what_it_did(): From 1e57401330f1d0a73c25d76ae5303b533e6e8f8b Mon Sep 17 00:00:00 2001 From: PiloUnk <198624632+PiloUnk@users.noreply.github.com> Date: Tue, 25 Aug 2026 00:57:22 +0200 Subject: [PATCH 10/10] Cut 0.9.4 Two lines on the release page, because that is what somebody deciding whether to upgrade needs: the portals this now talks to, and the credentials it stops showing. The reasoning behind each sits below the details marker, for whoever comes looking for it in the file. Test and Re-fetch all are named at the top for the same reason as 0.9.3: only the code changed, so Sync compares your settings against what was published, finds nothing moved, and fetches nothing. Co-Authored-By: Claude Opus 5 --- CHANGELOG.md | 75 ++++++++++++++++++++++++++++++++++++++++++++++++++++ plugin.json | 2 +- 2 files changed, 76 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8775743..172bb22 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,80 @@ # Changelog +## 0.9.4 + +**After upgrading, restart Dispatcharr, then press Test portals and Re-fetch +all.** The restart is what puts every worker on the new code, since plugins are +loaded once per process. Sync on its own would report every portal as unchanged +and fetch nothing — it compares your settings against what was last published, +and nothing in them changed, only the code did. + +- **Stalker compatibility improvements.** +- **Security improvements in the Settings panel, which now masks credentials.** + + + +**Talking to more portals** + +- **A refusal is recognised however the portal words it.** Ministra answers a + rejected session with HTTP 200 and a bare line of text, so only one exact + phrase was ever understood; the others arrived as "portal returned non-JSON + response" and cost a channel its failover. `Access denied.`, `Unauthorized + request.`, the stock server's numeric debug counter and the refusals + non-Ministra panels put inside the JSON envelope are all read now, and each + says which of the three things is actually wrong: the session, the account, + or the MAC. +- **A portal is looked for on every path one is served from.** Two were probed; + five are now — the URL as written first, then `/server/load.php`, + `/c/portal.php`, `/portal.php` and `/stalker_portal/server/load.php`. An + explicit `.php` in your own URL is no longer swapped for a guess, since that + path is the address the provider handed out. +- **The box describes itself the way a real one does.** The handshake and + profile now carry `prehash`, `client_type`, `video_out` and the metrics blob + a MAG sends, which is what some panels authenticate on and what the admin + panel reads to show a box as connected. +- **A command reaches the portal decoded exactly once.** A `%` inside a channel + command was being encoded twice, so those channels asked for a link that + never existed. +- **Channels the portal marks as needing no temporary link now play without one** + — the portal's own player skips `create_link` for them, and so does this. It + removes the one request known to go wrong on those providers. +- **The portal's own streams are fetched as the box that authenticated**, with + the session and MAC that minted the link. Never to anyone else's CDN: a + stream on another host gets no credentials, which is what keeps a + subscriber's MAC out of a request that has no business carrying it. +- Three things the portal was already saying are now read: the expiry date + where Ministra puts it, a device-conflict message with the setting that fixes + it named, and a channel listed twice in one response counted once. + +**Credentials are no longer displayed** + +- **The Portals box hides them once saved.** The MAC, the password and the rest + of the account identity come back as `••••`: the real line lives in + `/data/distalker/portals.txt`, which the settings panel cannot read. Names, + URLs and the tuning keys stay, so the box still reads as your own + configuration — and a portal is still deleted by deleting its line. +- This matters beyond the screen. Dispatcharr serves a plugin's settings row to + every account on the install, so the credentials were in an API response + anyone could read. +- **Editing works through the bullets.** Renaming a portal and repointing its + URL both keep the MAC you cannot see; pasting a line back in full still + works. Change a line's name *and* its URL in one edit and there is nothing + left to recognise it by — the plugin then quotes that line back and asks you + to retype it, rather than parsing a MAC address made of bullets. +- **Keep your own copy of each portal line in a password manager.** The box will + not give a credential back, and if `/data/distalker` is ever lost the last + copy goes with it. Nothing is redacted until that file is known to hold the + same list, so a registry that could not be written leaves the credentials + where they were. +- Hiding is not encryption. The resolver reads the MAC on every tune in a + process with no database, so `portals.txt` and the state mirrors go on + holding it in the clear at `0600` — the README says which files those are. +- **Settings the panel no longer has a field for are dropped.** Removing a field + from the manifest stops the panel rendering it and nothing else, so the + Add-portal form retired in 0.4.0 left a MAC, a password and a portal URL in + that same API response — for a portal you may have deleted months ago. They + are cleared on the first action after upgrading. + ## 0.9.3 **After upgrading, restart Dispatcharr, then press Test portals and Re-fetch diff --git a/plugin.json b/plugin.json index fdbe3c8..4e87d10 100644 --- a/plugin.json +++ b/plugin.json @@ -1,6 +1,6 @@ { "name": "Distalker", - "version": "0.9.3", + "version": "0.9.4", "description": "Stalker/MAG portal support for Dispatcharr. Syncs portal channels into a native M3U account and resolves short-lived stream links at tune time -- no extra containers, no extra ports.", "author": "PiloUnk", "license": "AGPL-3.0-only",