Bug: Galaxy view search returns duplicate results for nearly every match

[uncleJim21]

Summary

When searching from the front-end galaxy view, the result set contains duplicates of almost every match. Nearly all returned items appear two or more times.

Steps to Reproduce

1. Open the galaxy view on the front end.
2. Search for: Elon Musk on Dwarkesh podcast talking about space
3. Observe the returned results.

Expected Behavior

Each matching result (e.g. the Elon Musk / Dwarkesh space segment) appears once in the result set.

Actual Behavior

Almost every result is duplicated — the same item shows up multiple times in the galaxy view.

Notes

• Reproduced via the galaxy view search UI.
• Likely a de-duplication issue in the search/retrieval pipeline (results not being collapsed by unique episode/segment ID before being returned to the front end). Needs investigation to confirm whether duplication originates in the backend query/aggregation or in the front-end rendering layer.

Impact

Degrades search usability — users see cluttered, repeated results making it hard to find distinct matches.

[uncleJim21]

Sample response payload (galaxy view search)

Query: Elon Musk on Dwarkesh podcast talkinga bout space → rewritten to Elon Musk talking about space, total: 20.

Root cause is visible in the data

The same episode is indexed under two different GUID normalization forms and both copies are returned:

• substack:post:186967347 (colon-delimited)
• substack_post_186967347 (underscore-delimited)

Every paragraph from this episode comes back twice — once per GUID form — with near-identical similarity scores and identical timeContext:

sequence	colon shareLink	underscore shareLink	similarity (colon / underscore)
8	substack:post:186967347_p8	substack_post_186967347_p8	0.8204 / 0.8204
24	substack:post:186967347_p24	substack_post_186967347_p24	0.8143 / 0.8143
43	substack:post:186967347_p43	substack_post_186967347_p43	0.8189 / 0.8187
50	substack:post:186967347_p50	substack_post_186967347_p50	0.8218 / 0.8211
80	substack:post:186967347_p80	substack_post_186967347_p80	0.8143 / 0.8143
84	substack:post:186967347_p84	substack_post_186967347_p84	0.8141 / 0.8140
86	substack:post:186967347_p86	substack_post_186967347_p86	0.8258 / 0.8258

The lone exceptions (p416/p417, p367/p370) are slightly different paragraphs, but they too originate from the same colon/underscore-duplicated episode.

This is also reflected in triage.resolvedSignals.guest.guids, which lists both substack:post:186967347 and substack_post_186967347 as separate episodes (episodeCount: 3 when there are really 2 distinct episodes).

Implications

• The duplication is upstream of search — the corpus appears to contain two copies of the substack:post:186967347 episode under inconsistent GUID delimiters (: vs _). Vector search faithfully returns both.
• Fixes to consider:
  1. Ingest/normalization: canonicalize GUIDs to a single delimiter form so the episode is only stored once. Backfill/de-dupe existing docs.
  2. Query-time guard (defensive): de-dupe results by a normalized (guid, sequence) key before returning to the front end, treating substack:post:N and substack_post_N as the same episode.

Full payload

Raw JSON response

{
    "query": "Elon Musk talking about space",
    "total": 20,
    "model": "text-embedding-ada-002",
    "originalQuery": "Elon Musk on Dwarkesh podcast talkinga bout space",
    "triage": {
        "intent": "topical",
        "show_hint": "Dwarkesh",
        "person_hint": "Elon Musk",
        "rewrittenQuery": "Elon Musk talking about space",
        "confidence": 0.9,
        "resolvedSignals": {
            "feed": { "matched": true, "feedId": "1251189", "title": "Dwarkesh Podcast" },
            "guest": {
                "matched": true,
                "episodeCount": 3,
                "guids": [
                    "substack:post:186967347",
                    "substack_post_186967347",
                    "ffd226de-c3cc-4ace-b774-6ab0014a4831"
                ]
            }
        }
    }
}

(Trimmed to the relevant fields — the duplicated substack:post: / substack_post_ paragraph pairs and the two GUID forms in resolvedSignals.guest.guids are the key evidence. Full 20-result payload available on request.)