Measure auto marketing against the frozen holdout: five weekly read-outs

[gumclaw]

Stage

• Scope
• Design
• Build
• Local regression tests
• QA
• Shipped
• Market
• Sell

What

Adds Marketing::MetricsReport, weekly cohort snapshots, and a queryable plain-text read. Draft handoff for Gianfranco; do not merge automatically. Tracks antiwork/gumroad-private#2723.

Why

Compare treatment with the frozen creator holdout without dropping creators who never use a marketing action. Missing telemetry must not masquerade as a safe rollout: complaint counts remain unavailable and blocks_rollout stays true until the safety data is complete.

How

Monday 12:00 UTC on the low queue; a trailing 90-day, end-exclusive UTC window. The existing dashboard is a static generated site, not a marketing POST endpoint, so this writes one Marketing::MetricsSnapshot per window/cohort. No Telegram integration in Rails. Existing assignments and feature gates are untouched.

Read-out	Definition and denominator	Test
First sale	Zero-sale creators assigned during the window with seven complete follow-up days; first retained paid non-renewal purchase within seven days, last eligible UTM click within seven days, one result per creator. Assignment is a publication proxy, not a persisted first-publication event.	Two-proportion
Repeat purchase	Buyers with retained paid purchases before assignment, among creators assigned by window start; one retained non-renewal purchase per buyer in the 90-day window. Unemailed buyers stay in the denominator; email_infos / sent_post_emails provide a separate email-linked count.	Descriptive two-proportion
Net revenue	Canonical USD price_cents minus effective partial refunds; full refunds, unreversed chargebacks and bundle child duplicates excluded through existing scopes. Includes subscription revenue and zero-activity creators assigned by window start. Refund state is current at execution, not historical as-of reconstruction.	Creator-level Welch t
Email safety	Deduplicated purchase-linked deliveries, split by suggested draft origin vs hand-written. Follower deletion after the last delivery is an explicitly labeled unsubscribe proxy. Complaints and follower-only delivered denominators are unavailable, not zero. Absolute rate deterioration is configurable; incomplete telemetry also blocks rollout.	Descriptive two-proportion for observed proxy; complaint p unavailable
Cart recovery	Newly first-published abandoned-cart workflows; unique carts with a complete N-day horizon after first recorded send; retained same-cart/same-seller order purchase inside that horizon. Default N=7. Existing/re-enabled workflows excluded.	Descriptive two-proportion

The buyer/email/cart tests are unadjusted exploratory read-outs, not independent creator-randomized causal estimates. Cohorts are frozen zero versus under_100 + at_least_100; newer assignments are counted separately rather than given partial 90-day outcomes.

Sibling contracts inspected: #7734 uses installments.json_data.marketing_launch_written_copy_digest; #7735 persists Workflow.first_published_at but no auto-enabled origin; #7724 adds no metric columns. Named TODO hooks cover immutable send origin, timestamped complaint events, and auto-enabled workflow provenance; no guessed columns or holdout changes.

Before / after

Before: no durable five-read-out report. After: weekly aggregate JSON and plain text available through Marketing::MetricsSnapshot.order(window_end: :desc).first.report_text. Walkthrough evidence pending.

Test results

• bin/rspec spec/services/marketing/metrics_report_spec.rb spec/services/marketing/metrics_statistics_spec.rb spec/sidekiq/marketing/weekly_metrics_report_job_spec.rb: 26 examples, 0 failures, with isolated test services.
• bin/rubocop -a is absent in this repository; equivalent bundle exec rubocop -a passed on all eight changed Ruby implementation/spec files.
• bin/check-migration-versions: passed; the new-table migration was exercised locally.
• bin/test-confidence: exit 0; seven selected files passed. It skipped a request-spec failure as “likely pre-existing (unverified)”; that skip is not claimed as a verified baseline.
• Freeze proof includes unchanged assignment rows and a pre-window inactive holdout creator in the revenue denominator.

Mutation proofs

Pending execution before final draft handoff.

Review focus

1. Rolling 90-day baseline membership and assignment-as-publication proxy.
2. Incomplete complaint/delivery telemetry and the fail-closed rollout result.
3. Cart recovery definition and the sibling provenance hooks.

Weekly read message contract (folded in 2026-09-17)

The delivered surface is the no_agent cron marketing-holdout-metrics-weekly (script stdout
verbatim), so the message shape lives in the renderer, not in Rails. The whole post is:

• header: Auto-marketing weekly read — <window end>; assigned <treatment>/<holdout> (holdout N%).
• if nothing has an observation yet, one more line and nothing else:
  Assignments are N days old; first read with real comparisons lands <date>.
• otherwise one line per metric that has data, treatment first:
  7-day first sale: 4.2% (n=312) vs 3.4% (n=88), p=0.31.
• one closing line naming the metrics that are not yet readable and why, e.g.
  Not yet readable: cart recovery (need ≥25 creators per arm with a complete 7-day window); 90-day repeat purchase, 90-day net revenue (need ≥25 creators per arm past 90 days); unsubscribe proxy (needs complete delivery telemetry).
• a verdict line only when a decision metric crosses its pre-registered absolute threshold
  (+1 point for 7-day first sale and 90-day repeat purchase, +$1.00/creator for net revenue,
  two-sided p < 0.05) or when complete unsubscribe/complaint telemetry deteriorates — that gate
  blocks rollout.

A metric is readable only when both arms have at least 25 creators with complete follow-up;
below that it is omitted rather than printed as a near-empty rate. With data in both cohorts a
metric gets one labelled line per cohort ((new creators) / (existing sellers)); with data in
one cohort it is reported unlabelled instead of repeating every metric twice. No unavailable-vs-
unavailable rows, no "unavailable is not zero" disclaimers and no methodology footer — the
method is the table above.

Dry run — what the next fire posts (2026-09-17 data, rendered from the audited snapshot, no production query)

Auto-marketing weekly read — Sep 17, 2026; assigned 1,038/284 (holdout 21.5%).
Assignments are 1 day old; first read with real comparisons lands Sep 28, 2026.

Two lines, down from roughly forty. The date is the first Monday 12:00 UTC fire at least seven
complete days after the earliest assignment. --dry-run renders the latest snapshot and writes
nothing; the weekly read itself is an audited, read-only production query. 18 Python tests cover
the message contract, the readability floor, the verdict gate, fail-closed parsing and the UTC
schedule.

[greptile-apps]

Confidence Score: 4/5

The PR should not merge until the suggested-email origin marker is produced in production; the report workload should also be batched or aggregated before operating over the full assignment population.

Findings

1.  Suggested Origin Is Never Set ▶
2.  Report Work Grows Unbounded ▶

Fix with agent prompt

### Issue 1
app/services/marketing/metrics_report.rb:147
This classification depends entirely on `marketing_launch_written_copy_digest`, but this branch has no production writer for that marker; only the test sets it manually. As a result, production deliveries are all classified as `hand_written`, leaving the suggested-email safety denominator empty and making the suggested-versus-hand-written read-out unusable.

### Issue 2
app/services/marketing/metrics_report.rb:29
The report processes every historical holdout assignment and runs several seller-specific queries for each one, including delivery and cart-recovery queries for every assignment. Because assignments accumulate and both cohorts run serially, the weekly job's query count and database load grow by multiple round trips per assigned creator. Please aggregate or batch these metrics so the job remains practical as the assignment table grows.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

This PR adds durable weekly snapshots for five auto-marketing holdout read-outs, statistical comparison helpers, plain-text rendering, and a scheduled low-priority job.

• Persists one JSON snapshot per 90-day window and sales cohort.
• Computes first-sale, repeat-purchase, net-revenue, email-safety, and cart-recovery comparisons.
• Keeps rollout approval fail-closed while complaint telemetry remains unavailable.
• The suggested-email split currently lacks its production marker writer, and the report performs an unbounded per-creator query workload.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  S[Monday 12:00 UTC schedule] --> J[WeeklyMetricsReportJob]
  J --> Z[zero_sale cohort]
  J --> A[already_sells cohort]
  Z --> R[MetricsReport]
  A --> R
  R --> H[Holdout assignments]
  H --> F[First sale]
  H --> P[Repeat purchase]
  H --> N[Net revenue]
  H --> E[Email safety]
  H --> C[Cart recovery]
  F --> M[Aggregate metrics and p-values]
  P --> M
  N --> M
  E --> M
  C --> M
  M --> D[(MarketingMetricsSnapshot)]
  D --> T[Queryable plain-text report]

Loading

_{Reviews (1) · Last reviewed commit: "Measure marketing against frozen creator..."}

[greptile-apps]

Suggested origin is never set

This classification depends entirely on marketing_launch_written_copy_digest, but this branch has no production writer for that marker; only the test sets it manually. As a result, production deliveries are all classified as hand_written, leaving the suggested-email safety denominator empty and making the suggested-versus-hand-written read-out unusable.

Prompt To Fix With AI

This is a comment left during a code review.
Path: app/services/marketing/metrics_report.rb
Line: 147

Comment:
**Suggested origin is never set**

This classification depends entirely on `marketing_launch_written_copy_digest`, but this branch has no production writer for that marker; only the test sets it manually. As a result, production deliveries are all classified as `hand_written`, leaving the suggested-email safety denominator empty and making the suggested-versus-hand-written read-out unusable.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Report work grows unbounded

The report processes every historical holdout assignment and runs several seller-specific queries for each one, including delivery and cart-recovery queries for every assignment. Because assignments accumulate and both cohorts run serially, the weekly job's query count and database load grow by multiple round trips per assigned creator. Please aggregate or batch these metrics so the job remains practical as the assignment table grows.

Prompt To Fix With AI

This is a comment left during a code review.
Path: app/services/marketing/metrics_report.rb
Line: 29

Comment:
**Report work grows unbounded**

The report processes every historical holdout assignment and runs several seller-specific queries for each one, including delivery and cart-recovery queries for every assignment. Because assignments accumulate and both cohorts run serially, the weekly job's query count and database load grow by multiple round trips per assigned creator. Please aggregate or batch these metrics so the job remains practical as the assignment table grows.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[gumclaw]

Moved this to the Gumclaw cron marketing-holdout-metrics-weekly per Sahil, using read-only production-console queries and weekly Telegram delivery instead of Rails changes. Closing this PR; the branch is left intact for reference.