Skip to content

Analytics Phase 1 — make the existing numbers honest - #38

Open
SaurabhPandey9752 wants to merge 3 commits into
mainfrom
analytics/phase-1-honest-numbers
Open

Analytics Phase 1 — make the existing numbers honest#38
SaurabhPandey9752 wants to merge 3 commits into
mainfrom
analytics/phase-1-honest-numbers

Conversation

@SaurabhPandey9752

Copy link
Copy Markdown
Collaborator

Phase 1 of the analytics rebuild brief. Backend only — no UI changes. The goal is
that the numbers already on screen stop describing us, so Phase 4 isn't a
better-looking view of the same bad data.

What was wrong

  1. Self-referral attribution. inferSourceMedium() treated only the current
    request host as internal, so every other Taskly domain scored as an external
    referrer. We move visitors between our own domains deliberately (geo routing
    .ai → .ca, the .ai gateway hand-off, the legacy taskly.ca redirect) and
    each hop carries a Taskly Referer.
  2. No internal-traffic exclusion. isDevHost() only blocks localhost. Nothing
    kept the team out of our own dashboards.
  3. GMV destroyed by coupons. finalize-checkout wrote total_price = what Stripe charged, but total_price is the agreed deal.

Changes

Area Change
lib/attribution/owned-hosts.ts (new) Single source of truth for every host we own, derived from REGIONS + the legacy domain, apex and www.
lib/attribution/index.ts inferSourceMedium() returns {null, null} for any owned host — same as direct. Deliberately not an 'internal' source, which would move the pollution rather than remove it.
lib/analytics/audience.ts (new) One vocabulary for "whose traffic is this", the audience equivalent of date-range.ts. Default excludes team + bots; including them is an explicit ?includeInternal=true.
Migration 187 Reclassifies historical self-referrals across customer_profiles, tasks, bookings. Every changed row is recorded in a new attribution_reclassified audit table first, and customer_profiles also keeps the original inside its attribution jsonb. Reverses with one UPDATE … FROM (included as a comment).
Migration 188 profiles.is_internal + partial index + seeds the team/test email domains.
Migration 189 funnel_events.is_bot; rebuilds funnel_step_stats with p_exclude_internal / p_exclude_bots. Both default, so the currently-deployed 4-arg call keeps resolving during the deploy window. Uses a LEFT join to profiles — an inner join would have silently dropped every anonymous event, which is ~70% of them.
/api/track Flags crawlers via the existing isBotUserAgent instead of dropping them, so "how much of this is bots?" becomes answerable.
/api/admin/{funnels,journeys,utm} Honour the audience filter and echo it back so the UI can state what it's showing. Journeys filters internal ids in JS on purpose: user_id not in (…) is NULL for anonymous rows and PostgREST would have dropped them all.
finalize-checkout + process-event total_price stays the agreed deal; the coupon lives in discount_amount; service_fee is Taskly's cut of the deal, not a negative number.

lib/analytics/leaks.ts is deliberately unchanged — it's a pure ranker over
data the funnels API has already filtered, so it inherits the exclusion. Adding a
parameter there would have been a no-op.

Acceptance — measured against production, read-only

Attribution, once 187 runs:

Table Attributed now Self-referrals cleared Attributed after
customer_profiles 15 11 4
tasks 15 14 1
bookings 4 3 1

Internal exclusion, last 30 days (34 seeded internal accounts):

Now After
Sessions 1,528 1,443
Events 7,191 6,420
Verification SQL
with owned(host) as (values
  ('tasklyanything.ca'),('www.tasklyanything.ca'),('tasklyanything.in'),('www.tasklyanything.in'),
  ('tasklyanything.ai'),('www.tasklyanything.ai'),('taskly.ca'),('www.taskly.ca'))
select count(*) filter (where utm_source is not null) as attributed_now,
       count(*) filter (where lower(utm_source) in (select host from owned)) as will_clear,
       count(*) filter (where utm_source is not null
                          and lower(utm_source) not in (select host from owned)) as attributed_after
from customer_profiles;

Deliberately NOT done

The 7 broken GMV rows are not rewritten. The brief assumed they could be
recovered from the Stripe payment intent — they can't: all 7 have
stripe_payment_intent_id = null because a 100%-off coupon means Stripe never
creates one. They're arithmetically recoverable (discount_amount equals the full
deal on 6 of 7, and provider_payout / 0.8 on all 7), but that's rewriting money
rows on inference. Needs a founder decision — see the question in the thread.
The write-path fix means no new rows join them.

Gates

typecheck clean · 1,496 tests pass (2 new suites, 20 cases) · lint 0 errors
(214 pre-existing warnings) · knip clean · check:migrations OK

Before merge

  1. Apply 187 → 188 → 189 to production first, then merge. The code reads
    is_internal / is_bot, and a missing column throws into a catch that
    returns [] — a silent empty dashboard.
  2. Phase 0 (GA4 console) is still outstanding and is human-only: configure owned
    domains, unwanted referrals, internal traffic definition + data filter, and a
    hostName = localhost exclusion.

🤖 Generated with Claude Code

SaurabhPandey9752 and others added 3 commits August 1, 2026 17:49
The admin analytics screens were describing our own team and our own
redirects. Three fixes, no new surface area.

1. Self-referral attribution. inferSourceMedium() only ever treated the
   CURRENT request host as internal, so every other Taskly domain scored
   as an external referrer. We move visitors between our own domains on
   purpose — the geo router sends a Canadian on .ai to .ca, the gateway
   hands off, legacy taskly.ca redirects in — and each hop carries a
   Taskly Referer. New lib/attribution/owned-hosts.ts is the single
   source of truth, derived from REGIONS so a new market is covered the
   moment it is added. Migration 187 reclassifies history in place and
   records every changed row in attribution_reclassified so it reverses
   with one UPDATE.

   Measured: 11 of 15 attributed signups, 14 of 15 tasks and 3 of 4
   bookings were Taskly referring Taskly. After: 4 signups, 1 task,
   1 booking — small, but real.

2. Internal traffic. Nothing excluded us from our own dashboards;
   isDevHost() only blocks localhost. Migration 188 adds
   profiles.is_internal and seeds it from the team/test email domains
   (34 accounts). Migration 189 threads exclude_internal + exclude_bots
   through funnel_step_stats, and lib/analytics/audience.ts gives every
   report one vocabulary for "whose traffic is this". Default is the
   outside world; including the team is an explicit opt-in. A reporting
   lens only — no rows are deleted and nothing is dropped at ingest.

   Effect on the last 30 days: 1528 -> 1443 sessions, 7191 -> 6420 events.

3. GMV. finalize-checkout wrote total_price = what Stripe charged, but
   total_price is the agreed DEAL. Any coupon erased it: 7 production
   rows sit at total_price = 0 with service_fee equal to minus the whole
   payout, one at -$20,460. The charge and the deal are different facts —
   the coupon belongs in discount_amount and the deal stays intact. Fixed
   on both write paths (marketplace + curated); the existing 7 rows are
   NOT rewritten here, see the PR for why.

Crawler hits are now flagged at ingest (is_bot) rather than dropped, so
"how much of this is bots?" becomes answerable instead of invisible.

Gates: typecheck clean, 1496 tests pass, lint 0 errors, knip clean,
check:migrations OK.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Founder decision: mark, don't repair. The seven rows whose total_price was
overwritten with "what Stripe charged" are arithmetically recoverable
(discount_amount holds on 6 of 7, provider_payout / 0.8 on all 7) but that
is inference — none of them has a stripe_payment_intent_id, because a
100%-off coupon never creates one, so there is nothing authoritative to
reconcile against.

Migration 190 adds bookings.total_price_suspect, flags the known-broken
shape (total_price = 0, or service_fee < 0 which means the deal was
clobbered below the payout), and rebuilds admin_booking_analytics to sum
revenue over trustworthy rows only while still COUNTING the suspect ones
and reporting them as their own figure. The jobs really happened; it is
only their money columns we don't believe.

Booking counts deliberately still include them — dropping a real booking
from the count to hide a bad number would be the same mistake in reverse.

Same rule in the UTM report: a suspect booking contributes to the channel's
booking count but not its GMV, so a channel is never reported as having
produced $0 of value when the truth is that we cannot say.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
190 quarantined these seven rows on the belief that repairing them meant
inference. That was wrong. `subtotal` was never touched by the faulty write
path and still holds the agreed deal on every affected row — and it agrees
exactly with round(provider_payout / 0.8) on all seven, the payout having
been derived from the deal at booking time. Two independently-stored columns
agreeing is a recovered value, not a guess.

Recovers $27,755 of real GMV and restores the correct 20% commission. The
worst row goes from -$20,460 "commission" to +$5,115.

Guarded on the corroboration, not just the flag: a row that is suspect but
whose subtotal does NOT match the implied deal stays flagged and stays out
of revenue. A future corruption with a different fingerprint must not be
silently restored to a number nobody verified.

Prior values captured in booking_money_restored first, so this reverses
exactly. total_price_suspect stays on the table for the next case.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@SaurabhPandey9752
SaurabhPandey9752 force-pushed the analytics/phase-1-honest-numbers branch from caec0c6 to 611a51b Compare August 1, 2026 12:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant