[Bug]: M3U-refresh dedup queues the next day's event fixtures as duplicates of yesterday's channels, freezing the group at 0 channels created

[lukeeexd]

ECM Version

Observed on 0.18.1. Source references below re-verified against the current checkout (030c9363, v0.18.2-0037); none of the paths involved have changed.

Priority

Priority 2: ECM is severely degraded

What is the bug?

Event providers reuse a fixed pool of slot names (numbers 01..100) and roll the variable part of each name over daily — the draw, the players, and the date/time. The M3U-refresh dedup path ("BD-A" scorer, ADR-008 §D1) compares the incoming stream name against the existing channel names in the target group with no date/time awareness, so today's fixture in a slot is scored as a duplicate of yesterday's fixture in the same slot. At or above the queue threshold the stream is queued as a pending merge instead of creating a channel, and while a row is pending the auto-creation pipeline defers that stream: no channel created, no EPG assigned.

When the whole slot pool rolls over at once, every stream in the group is deferred, the run reports 0 channels created, and the group sits on yesterday's matches for roughly a third of the day. There is no operator workaround — details below.

Names in the tables below are stand-ins that reproduce the shape of a real provider's slot naming; the scores are measured against the shipped matcher.

The scorer is not date-aware, and a one-day difference is not a close call. find_candidate normalizes both sides with NameCleanMode.CONSERVATIVE — NFC, strip a leading N | prefix, lowercase, strip (backend/services/dedup_matcher.py:522, called at :716 and :730) — and scores with fuzz.token_set_ratio(...) / 100.0 (:744). Nothing in the matcher or in the reused normalization_engine helpers touches month names, HH:MM, AM/PM, GMT/IST, or an @ marker. Measured with the shipped code and the default dedup_threshold = 0.80 (backend/config.py:891):

incoming stream	candidate channel	confidence
EVENT SLOT 04: Championship | Qualifying: Player One - Player Two @ 18 Sep 09:30 AM GMT-1	... Player Three - Player Four @ 17 Sep 01:00 PM GMT-1	0.862 → queued
EVENT SLOT 04: Championship | Qualifying: Player One - Player Two @ 18 Sep 09:30 AM GMT-1	... Player One - Player Two @ 19 Sep 09:30 AM GMT-1	0.988 → queued
(same name)	(same name)	1.000 → queued

That is the whole bug in one line: a name that differs only by the date is scored 0.99. The static slot prefix plus the tournament/round words carry the rest of the score; the slot pool exists precisely to make them identical.

A queued stream is a deferred create, and the deferral is what breaks the run.

• backend/channel_pipeline_executor.py:1490-1529 calls _maybe_enqueue_pending_merge after the exact/normalized name lookups have missed, and returns the result verbatim, skipping the create_channel API call.
• backend/services/m3u_dedup_hook.py:301-416 scores the stream against every channel in the target group — _maybe_enqueue_pending_merge builds the candidate list from self.existing_channels filtered only by channel_group_id (executor.py:6198-6210). There is no exclusion of a channel the stream is already attached to, and no exclusion of a candidate whose dynamic (date/event) part differs.
• On a hit the executor returns ActionResult(skipped=True, ...) with the description Stream 'X' already in pending merges queue; channel creation deferred (executor.py:6270-6278).
• Because the create was skipped, exec_ctx.current_channel_id is never set, so the rule's following assign_epg action fails with No channel context for assign_epg / No channel to update (executor.py:2650-2655). The run then finalises completed_with_errors (engine.py:638-639).

So the operator-visible signature is a run that created nothing, plus a set of assign_epg errors that read like EPG trouble and are actually the dedup queue.

No operator escape hatch exists. Both review actions fail to unblock the pipeline:

1. Accept cannot apply. The rule uses create_channel, so both provider copies of a slot carry the same stream name; POST /{id}/accept cannot disambiguate which copy to attach, and when the candidate channel is gone it returns 404 with the row left pending (backend/routers/channel_merges.py:1351-1366).

2. Dismiss does not stick. POST /{id}/dismiss flips the row to dismissed (channel_merges.py:1648-1652), and the partial unique index that keeps the queue idempotent only covers pending rows:

   Index(
       "uq_pending_merges_active",
       "stream_name",
       "candidate_channel_id",
       unique=True,
       sqlite_where=sa_text("status = 'pending'"),
   ),

   (backend/models.py:3328-3335)

   The next M3U refresh re-scores the same stream against the same candidate, the collision recovery query filters status == "pending" (m3u_dedup_hook.py:234-243), finds nothing, and inserts a fresh pending row. Dismissal is per-row, not sticky per (stream_name, candidate_channel_id) pair.

3. Rows never expire. There is no reaper, no last_seen_at, and nothing that reacts to the candidate channel disappearing. The auto_aged_out action_type is a reserved enum value in migration 0014 (:66, :289-291) that no code ever writes, and idx_pending_merges_candidate is documented as "offline orphan detection (§D4)" (models.py:3320-3322) but is never queried. Pending rows whose candidate channel no longer exists block forever; they are neutralised only by accident, when Pass 4 orphan reconciliation deletes the previous day's channels (engine.py:6021) and the candidate ceases to exist.

Second, simpler bug in the same path. The candidate set includes channels the stream is already attached to, so a stream can be queued as a duplicate of the channel it is on, at confidence 1.0. One of the rows observed in the live incident was exactly that. Nothing in _maybe_enqueue_pending_merge consults the channel's streams list.

Diagnosability makes it invisible. results["pending_merges_added"] exists (engine.py:2898, aggregated at :3422) and reaches the toast at backend/tasks/channel_pipeline.py:755 / :812 / :941 — but only inside the if not has_failed_actions: branch (:805). A run with any failed action skips that notification entirely and emits only the task-layer "Task Completed with Warnings". The returned summary is:

Auto-creation after M3U refresh: 26 streams evaluated, 12 matched, 0 channels created, 0 updated; 24 actions failed

(tasks/channel_pipeline.py:891-893) — which is indistinguishable from "the provider had nothing today", and names neither the deferral nor the blocking row ids. (The engine log line at engine.py:757-762 has the same gap.)

Dry runs cannot see any of this. The hook is bypassed before it reads the queue when exec_ctx.dry_run is true (executor.py:6183-6189), so a preview reports channels as createable that the live run then defers. The asymmetry is itself a diagnosability bug: preview and run disagree about what will happen.

Steps to Reproduce

1. Group G contains a channel EVENT SLOT 04: <Match A> @ <yesterday> ..., created the previous day from slot 04 by an m3u_refresh-triggered create_channel rule ({stream_name} name template, if_exists: merge) with a following assign_epg action.
2. The provider renames slot 04 in place to <Match B> @ <today> ... and the M3U refresh runs.
3. The dedup hook scores the stream against the same-group channels; the slot-04 pair scores 0.86–0.99 (table above) and is queued as a pending merge.
4. The create is skipped, the following assign_epg fails with No channel context for assign_epg, and the run summary reads 0 channels created with N failed actions.
5. Dismissing or accepting the row does not help: dismiss re-queues on the next refresh; accept cannot disambiguate the two provider copies of the slot.
6. The group only heals once Pass 4 deletes the previous day's channels, which dissolves the candidates by accident.

Reproduction of the score itself, with no Dispatcharr needed:

from services.dedup_matcher import find_candidate
find_candidate(
    "EVENT SLOT 04: Championship | Qualifying: Player One - Player Two @ 18 Sep 09:30 AM GMT-1",
    [("c1", "EVENT SLOT 04: Championship | Qualifying: Player One - Player Two @ 19 Sep 09:30 AM GMT-1")],
    0.80,
)
# -> MatchResult(..., confidence=0.988)   # one day apart, queue threshold is 0.80

Screenshots / Logs / Artifacts

Some of it is in the debug bundle I'll attach privately if it helps, but the reasoning above stands on its own.

Additional Context

Dispatcharr 0.31.0.

Proposed fix (happy to implement, but see the scope question).

1. Date/time-aware scoring in the BD-A matcher (the real fix). Strip the date/time token run (@ 18 Sep 09:30 AM GMT-1 and variants) from both sides before scoring, or exclude those tokens from the score. The rule the scorer should encode: a next-day stream in the same slot is a rollover, not a duplicate — when the only strong similarity signal is the static slot prefix and the dynamic part differs, do not queue; create. This is ADR-008 §D2 territory (the 0.60 floor and the operator threshold are untouched), so it likely wants an ADR addendum rather than a silent behaviour change.
2. Never queue a stream against the channel it is already attached to. The candidate list is built with no attachment filter (executor.py:6198-6210); the channel's streams list is available at that point.
3. Pending-row lifecycle. Auto-expire/auto-resolve a pending row when its candidate channel no longer exists (404 on resolve, channel_merges.py:1351-1366) instead of blocking forever; and make dismissal sticky per (stream_name, candidate_channel_id) pair rather than per row (models.py:3328-3335, m3u_dedup_hook.py:234-243). Terminal rows already persist, so a sticky dismissal can be enforced by consulting them at enqueue time without a schema change.
4. Diagnosability. The pipeline summary should count deferred-by-pending-merge streams and name the blocking row ids, instead of reporting only 0 channels created; the deferral reason should be carried into the assign_epg action error so a single deferred slot does not read as an EPG bug; and the dry run should surface pending merges so preview and run agree.

Tests I would add: scorer with date/time tokens stripped (same-slot @ 18 Sep vs @ 17 Sep does not reach the queue threshold; genuinely identical names still do); stream already attached to the candidate → not queued; pending row whose candidate channel 404s → auto-resolved and the pipeline proceeds; dismissed stream+candidate pair not re-queued on the next refresh; integration rollover scenario (day-old channels + refreshed slot names → one run creates today's channels, zero new pending rows).

Scope question. Items 1 and 4 are self-contained and are what actually stops the daily freeze. Item 3 touches queue semantics. I would rather land 1+2+4 first and treat 3 as a follow-up if you agree — say the word and I will open a PR against dev scoped accordingly.

[MotWakorb]

Scope disposition: dry-run pending-merge visibility is tracked separately in #1018. PR #1016 remains scoped to the daily-rollover matcher, candidate filtering, and live-run diagnosability; queue expiry and sticky dismissal also remain outside that PR.