Analytics Phase 1 — make the existing numbers honest - #38
Open
SaurabhPandey9752 wants to merge 3 commits into
Open
Analytics Phase 1 — make the existing numbers honest#38SaurabhPandey9752 wants to merge 3 commits into
SaurabhPandey9752 wants to merge 3 commits into
Conversation
The admin analytics screens were describing our own team and our own redirects. Three fixes, no new surface area. 1. Self-referral attribution. inferSourceMedium() only ever treated the CURRENT request host as internal, so every other Taskly domain scored as an external referrer. We move visitors between our own domains on purpose — the geo router sends a Canadian on .ai to .ca, the gateway hands off, legacy taskly.ca redirects in — and each hop carries a Taskly Referer. New lib/attribution/owned-hosts.ts is the single source of truth, derived from REGIONS so a new market is covered the moment it is added. Migration 187 reclassifies history in place and records every changed row in attribution_reclassified so it reverses with one UPDATE. Measured: 11 of 15 attributed signups, 14 of 15 tasks and 3 of 4 bookings were Taskly referring Taskly. After: 4 signups, 1 task, 1 booking — small, but real. 2. Internal traffic. Nothing excluded us from our own dashboards; isDevHost() only blocks localhost. Migration 188 adds profiles.is_internal and seeds it from the team/test email domains (34 accounts). Migration 189 threads exclude_internal + exclude_bots through funnel_step_stats, and lib/analytics/audience.ts gives every report one vocabulary for "whose traffic is this". Default is the outside world; including the team is an explicit opt-in. A reporting lens only — no rows are deleted and nothing is dropped at ingest. Effect on the last 30 days: 1528 -> 1443 sessions, 7191 -> 6420 events. 3. GMV. finalize-checkout wrote total_price = what Stripe charged, but total_price is the agreed DEAL. Any coupon erased it: 7 production rows sit at total_price = 0 with service_fee equal to minus the whole payout, one at -$20,460. The charge and the deal are different facts — the coupon belongs in discount_amount and the deal stays intact. Fixed on both write paths (marketplace + curated); the existing 7 rows are NOT rewritten here, see the PR for why. Crawler hits are now flagged at ingest (is_bot) rather than dropped, so "how much of this is bots?" becomes answerable instead of invisible. Gates: typecheck clean, 1496 tests pass, lint 0 errors, knip clean, check:migrations OK. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Founder decision: mark, don't repair. The seven rows whose total_price was overwritten with "what Stripe charged" are arithmetically recoverable (discount_amount holds on 6 of 7, provider_payout / 0.8 on all 7) but that is inference — none of them has a stripe_payment_intent_id, because a 100%-off coupon never creates one, so there is nothing authoritative to reconcile against. Migration 190 adds bookings.total_price_suspect, flags the known-broken shape (total_price = 0, or service_fee < 0 which means the deal was clobbered below the payout), and rebuilds admin_booking_analytics to sum revenue over trustworthy rows only while still COUNTING the suspect ones and reporting them as their own figure. The jobs really happened; it is only their money columns we don't believe. Booking counts deliberately still include them — dropping a real booking from the count to hide a bad number would be the same mistake in reverse. Same rule in the UTM report: a suspect booking contributes to the channel's booking count but not its GMV, so a channel is never reported as having produced $0 of value when the truth is that we cannot say. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
190 quarantined these seven rows on the belief that repairing them meant inference. That was wrong. `subtotal` was never touched by the faulty write path and still holds the agreed deal on every affected row — and it agrees exactly with round(provider_payout / 0.8) on all seven, the payout having been derived from the deal at booking time. Two independently-stored columns agreeing is a recovered value, not a guess. Recovers $27,755 of real GMV and restores the correct 20% commission. The worst row goes from -$20,460 "commission" to +$5,115. Guarded on the corroboration, not just the flag: a row that is suspect but whose subtotal does NOT match the implied deal stays flagged and stays out of revenue. A future corruption with a different fingerprint must not be silently restored to a number nobody verified. Prior values captured in booking_money_restored first, so this reverses exactly. total_price_suspect stays on the table for the next case. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SaurabhPandey9752
force-pushed
the
analytics/phase-1-honest-numbers
branch
from
August 1, 2026 12:52
caec0c6 to
611a51b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 1 of the analytics rebuild brief. Backend only — no UI changes. The goal is
that the numbers already on screen stop describing us, so Phase 4 isn't a
better-looking view of the same bad data.
What was wrong
inferSourceMedium()treated only the currentrequest host as internal, so every other Taskly domain scored as an external
referrer. We move visitors between our own domains deliberately (geo routing
.ai → .ca, the.aigateway hand-off, the legacytaskly.caredirect) andeach hop carries a Taskly
Referer.isDevHost()only blocks localhost. Nothingkept the team out of our own dashboards.
finalize-checkoutwrotetotal_price = what Stripe charged, buttotal_priceis the agreed deal.Changes
lib/attribution/owned-hosts.ts(new)REGIONS+ the legacy domain, apex andwww.lib/attribution/index.tsinferSourceMedium()returns{null, null}for any owned host — same as direct. Deliberately not an'internal'source, which would move the pollution rather than remove it.lib/analytics/audience.ts(new)date-range.ts. Default excludes team + bots; including them is an explicit?includeInternal=true.customer_profiles,tasks,bookings. Every changed row is recorded in a newattribution_reclassifiedaudit table first, andcustomer_profilesalso keeps the original inside itsattributionjsonb. Reverses with oneUPDATE … FROM(included as a comment).profiles.is_internal+ partial index + seeds the team/test email domains.funnel_events.is_bot; rebuildsfunnel_step_statswithp_exclude_internal/p_exclude_bots. Both default, so the currently-deployed 4-arg call keeps resolving during the deploy window. Uses a LEFT join toprofiles— an inner join would have silently dropped every anonymous event, which is ~70% of them./api/trackisBotUserAgentinstead of dropping them, so "how much of this is bots?" becomes answerable./api/admin/{funnels,journeys,utm}user_id not in (…)is NULL for anonymous rows and PostgREST would have dropped them all.finalize-checkout+process-eventtotal_pricestays the agreed deal; the coupon lives indiscount_amount;service_feeis Taskly's cut of the deal, not a negative number.lib/analytics/leaks.tsis deliberately unchanged — it's a pure ranker overdata the funnels API has already filtered, so it inherits the exclusion. Adding a
parameter there would have been a no-op.
Acceptance — measured against production, read-only
Attribution, once 187 runs:
customer_profilestasksbookingsInternal exclusion, last 30 days (34 seeded internal accounts):
Verification SQL
Deliberately NOT done
The 7 broken GMV rows are not rewritten. The brief assumed they could be
recovered from the Stripe payment intent — they can't: all 7 have
stripe_payment_intent_id = nullbecause a 100%-off coupon means Stripe nevercreates one. They're arithmetically recoverable (
discount_amountequals the fulldeal on 6 of 7, and
provider_payout / 0.8on all 7), but that's rewriting moneyrows on inference. Needs a founder decision — see the question in the thread.
The write-path fix means no new rows join them.
Gates
typecheckclean · 1,496 tests pass (2 new suites, 20 cases) ·lint0 errors(214 pre-existing warnings) ·
knipclean ·check:migrationsOKBefore merge
is_internal/is_bot, and a missing column throws into acatchthatreturns
[]— a silent empty dashboard.domains, unwanted referrals, internal traffic definition + data filter, and a
hostName = localhostexclusion.🤖 Generated with Claude Code