Skip to content

fix(ranking): sample the rendered feed, not just the ranked window (GEO-2950) - #967

Merged
ohohoreilly merged 1 commit into
mainfrom
ohohoreilly/canary-samples-rendered-feed
Sep 17, 2026
Merged

ohohoreilly merged 1 commit into
mainfrom
ohohoreilly/canary-samples-rendered-feed

Conversation

@ohohoreilly

Copy link
Copy Markdown
Collaborator

The canary I shipped this morning was measuring a feed nobody sees.

What went wrong

0091/0092 sample the ranked window — the rows entities_ranked_for_feed_by_type returns. Explore then reorders that in geogenesis (applyDiversityCap, and with geogenesis#2456 applyTargetMix, then applyPerSpaceQuota) before rendering. Measured on production today, featured spaces, the default three types:

per 10 items window (what the canary recorded) rendered (what readers saw)
Claim 4.1 1.8
Debate 2.0 5.9
News story 3.9 2.3

So its Claim 24 / Debate 8 / News 34 was true and useless. It would not have moved when Preston reported the feed had gone claim-starved, because the window never changed — the reordering did. It took three wrong answers today to find that.

Record both, because the gap is the diagnostic

Not a repoint. A composition change visible in both sources is a ranking change; visible in rendered alone it is read-time reordering; visible in window alone it is something no reader ever saw. That distinction was the entire answer today and the canary could not express it.

  • window samples stay in SQL (sample_feed_composition, unchanged).
  • rendered needs an HTTP call to the app's own feed endpoint, so the binary measures it and record_feed_composition_sample stores it — one table, one shape, config snapshot taken at write time for both.
  • feed_composition_drift now takes a source, so the two histories are never compared against each other. They legitimately differ, so a cross-source comparison would fire on every single run and the alert would be worthless within a day.

FEED_COMPOSITION_FEED_URL is optional: unset, the job still runs and records the window only.

The test harness earned its keep

Adding the source parameter to feed_composition_drift created a sibling overload rather than replacing the function, so every existing one-argument call broke with:

ERROR: function public.feed_composition_drift(unknown) is not unique

The SQL suites caught it immediately — 6 failures across 0091, 0092 and 0093 — before it could reach a deploy. 0077 and 0085 both hit this same trap and document it; the fix is dropping the old signature first, and drizzle runs a migration in one transaction so no caller sees the gap.

Verified

  • tests/0093: the source guard, verbatim storage with a write-time config snapshot, and the assertion this exists for — window and rendered are separate histories, so two rendered samples plus one window sample yield drift for the first and none for the second.
  • 30 tests passing across the SQL harness (15 suites × 2 orders).
  • End to end against the real endpoint: the binary's rendered sample independently reproduced Claim 4 / Debate 13 / News 5, matching what I measured by hand with curl.
  • cargo clippy -- -D warnings and cargo fmt clean.

Deploy

Migration 0093 applies via the api's migrate initContainer; the binary and the new env ship with the next ranking-indexer deploy. Existing rows default to source = 'window', which is what they were.

…EO-2950)

The canary measured entities_ranked_for_feed_by_type's output. Nobody sees
that. Explore applies read-time reordering in geogenesis before rendering, and
on 2026-09-17 the same feed was Claim 24 / Debate 8 / News 34 in the window
while readers were served 1.8 / 5.9 / 2.3 per 10. The canary's numbers were
true and about a feed nobody looks at -- it would not have moved when Preston
reported the claim shortfall.

Records BOTH sources, because the gap is the diagnostic: a change in both is a
ranking change, a change in rendered alone is read-time reordering, a change in
window alone is something no reader saw. That distinction was the whole answer
today and was not available.

window samples stay in SQL. rendered needs an HTTP call to the app's own feed
endpoint, so the binary measures it and record_feed_composition_sample stores
it -- one table, one shape. feed_composition_drift now takes a source so the
two histories are never compared against each other, which would fire on every
run.

Verified end to end: the binary's rendered sample independently reproduces the
4/13/5 measured by hand on production.
@ohohoreilly ohohoreilly self-assigned this Sep 17, 2026
@ohohoreilly
ohohoreilly merged commit 7bd0d6c into main Sep 17, 2026
9 checks passed
@ohohoreilly
ohohoreilly deleted the ohohoreilly/canary-samples-rendered-feed branch September 17, 2026 23:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant