Skip to content

feat(experiments): extend metric-events precomputation to count/sum mean metrics - #75903

Draft
jurajmajerik wants to merge 2 commits into
masterfrom
experiments/mean-metric-events-precompute
Draft

feat(experiments): extend metric-events precomputation to count/sum mean metrics#75903
jurajmajerik wants to merge 2 commits into
masterfrom
experiments/mean-metric-events-precompute

Conversation

@jurajmajerik

@jurajmajerik jurajmajerik commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Problem

Experiment precomputation has two halves: an exposures table (built for every metric type) and a metric-events table (built only for ordered funnel metrics). Mean metrics only get the exposures half, so every uncached load of a mean metric scans the team's raw events. At teams enrolled in precomputation this is the largest remaining source of slow, read-heavy experiment queries, while fully precomputed funnels at the same teams run interactively.

Changes

Extends the metric-events half to mean metrics for the simplest case: events or actions source, count (total) or sum math. No CUPED, no breakdowns, no data warehouse, no session properties. Everything ineligible keeps the existing direct-scan path.

  • _metric_events_precompute_applicable() now admits eligible mean metrics alongside ordered funnels.
  • New MeanQueryBuilder.get_mean_metric_events_query_for_precomputation() writes one row per matching metric event into the existing experiment_metric_events_preaggregated table, using the numeric_value column that was already in the schema. No schema change.
  • When precomputed jobs are ready, the mean metric_events CTE reads the preaggregated table (filtered by job ids, team, and experiment window) instead of scanning events. The rest of the query (conversion-window join, winsorization, threshold) is untouched, so results are identical between the two paths.
  • The precomputed read collapses rows by event identity (GROUP BY entity_id, timestamp, event_uuid). Sum is the first aggregate on this table that is not idempotent under duplicate rows: exposures re-aggregate with argMin/min/max and funnel evaluation tolerates a doubled event, but a replayed build INSERT would double a sum. ReplacingMergeTree only collapses duplicates at merge time and the read does not use FINAL, so the read defends itself the way the exposures read does.
  • get_metric_events_query_for_precomputation() on the query builder dispatches funnel vs mean; the runner's build orchestration, fallback, and experiment_metric_events_path tagging work unchanged.

Design notes:

  • The build query keeps the experiment date bounds as named placeholders with experiment_date_to declared a sentinel, same as the funnel template. Reusing _build_metric_predicate() would bake the resolved dates into the query, so a running experiment's moving window end would change the job hash on every load and defeat cache reuse.
  • Count/sum only for now because those per-event values are already coalesced to non-null floats before storage, making the precomputed and direct paths provably identical. Min/max/avg are equally safe and only need the allowlist widened. Unique session/DAU/group and HogQL math produce non-numeric per-event values and stay direct.
  • A build failure or not-ready build falls back to the direct scan, as before.

Follow-ups (informed by a review of past precompute fix PRs):

  • Widen the math allowlist to min/max/avg.
  • Ratio metrics (two mean-style aggregates, can reuse this build).
  • High-volume mean sources (e.g. a total count of a very dense event) will read more per build than funnel step scans, so large teams may hit the read byte cap (error 307). Those builds fail fast without retry and fall back cleanly, but the metric stays unprecomputed until build windows shrink further. Watch the failed-builds view in query performance after rollout.
  • The uncalculated-cohort build gate walks metric JSON only, so a cohort filter inside an action definition is invisible to it. Pre-existing gap (funnel action steps have it too), but this PR widens the eligible ActionsNode surface. Fix is to resolve action IDs when collecting cohort ids.
  • The result-consistency canary compares precomputed vs direct results in production and will now cover mean metrics. Frozen-window staleness (late events, person merges) is a known divergence class there and winsorization amplifies it for sums; those are staleness, not new bugs.

How did you test this code?

  • Full products/experiments/backend/hogql_queries/test/experiment_query_runner/ suite passes (278 tests). The existing parameterized mean tests' precomputed legs now route through the new build and read path end to end, asserting exact results for sum, count, conversion window, winsorization, ignore-zeros, and threshold. Their 7 ClickHouse query snapshots changed accordingly; ineligible math types kept their snapshots, confirming the gate.
  • New test_experiment_mean_metric_events_preaggregation.py covers only what nothing else catches: a hash-stability test (a refactor that bakes the resolved date bounds into the mean build template would silently kill cache reuse without failing any result assertion), a parameterized gate test (a wrongly widened gate causes swallowed build failures, a wrongly narrowed one silently loses precompute, both invisible to result assertions because of the fallback), and a replay-tolerance test that duplicates every stored build row and asserts results still match the direct scan. I verified the replay test actually fails (sum doubles) when the read-side dedup is removed.
  • Repo-wide mypy --cache-fine-grained . clean, hogli ci:preflight --fix green.

Automatic notifications

  • Publish to changelog?
  • Alert Sales and Marketing teams?

Docs update

Updated LAZY_COMPUTATION.md with a note on the metric-events half.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Implemented with Claude Code from a written brief. Skills invoked: /wt (worktree setup), /writing-tests. The plan considered including min/max/avg in the first PR since they are parity-safe for free, but we kept the allowlist to count/sum to match the brief's scope and keep the review surface small. CUPED support and CUPED usage measurement were explicitly deferred.

After the initial version, I (Claude) reviewed all precompute-related fix PRs merged this year (INSERT date constraints, insert settings parity, ReplacingMergeTree ORDER BY, hash stability, sync distributed inserts, TTL refold, byte-cap no-retry, GROUP BY spill, group-aggregation and cohort gates) and checked which lessons the mean path inherits via the shared executor versus which it must handle itself. That review produced the read-side dedup commit and the follow-ups list above.

@trunk-io

trunk-io Bot commented Jul 31, 2026

Copy link
Copy Markdown

Merging to master in this repository is managed by Trunk.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

@jurajmajerik jurajmajerik self-assigned this Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant