Skip to content

outcomes.md

Latest commit

 

History

History
89 lines (63 loc) · 7.03 KB

outcomes.md

File metadata and controls

89 lines (63 loc) · 7.03 KB
Raw

Outcome events

Adaptif's outcome events let you compare goal rates between users who could see adaptive UI (the shown group, arm treatment in events) and users who get the fallback instead (the held out group, arm holdout). This page describes the stored format, so you can write your own sink, export events from a warehouse, and analyze them.

Which events are sent depends on <AdaptifProvider outcomes>: by default ("measured") only components with a holdout or a goal record exposures, engagement, and dismissals, and goals are recorded once such a component has mounted or been exposed in the session. "all" records every <When>/<Match>, and "off" records nothing.

Stored event format (schema 1)

Every event a sink receives is one JSON object. JSONL files (what jsonlSink writes) hold one object per line. The machine-readable definition is the JSON Schema at @adaptif/core/outcome-schema.json, and TypeScript users can import the StoredOutcomeEvent type from @adaptif/core.

All events share this envelope:

Field Type Meaning
schema 1 Format version. Bumped only for incompatible changes.
receivedAt integer (ms since epoch) Server time the event arrived.
session string, 1–128 chars The browser tab's session ID.
assignmentKey string, 1–128 chars The unit of analysis: the session ID, or the hashed key passed as assignmentKey.
occurredAt integer (ms since epoch) Client time the event happened. Attribution uses this.
type "exposure", "engaged", "dismissed", "goal", "dropped" Which of the shapes below.

Identifiers (questionId, componentId, goal, branch) match /^[a-z][a-z0-9_.-]{0,39}$/i.

exposure

Logged once per assignment key and component (per browser tab), the first time the stability decision said "show", in both arms.

Field Type Meaning
questionId identifier The question's registry id: the explicit id in adaptif.config.ts, or deriveQuestionId when that id is omitted.
componentId identifier The <When>/<Match> id (default: derived from the question).
arm "treatment" | "holdout" Which experience this user gets for this component.
holdout number, 0–0.5 The configured holdout at the time, so an analysis can split by it.
p number, 0–1, optional The situation's probability (<When>).
branch identifier, optional The chosen option ID (<Match>).
goal identifier, optional The component's configured goal.
forced boolean, optional true for QA overrides (?adaptif.arm=, ?adaptif.force=) and mock mode. Leave these out.

engaged and dismissed

questionId and componentId, as above. Only recorded after a treatment exposure (holdout users never see the content).

goal

goal: the goal's name. Recorded in both arms.

dropped

count: how many events a browser discarded before delivery (its queue overflowed, or a batch failed twice). Written by the server, not sent as an event; if any appear, your counts are missing events. occurredAt equals receivedAt.

Example

{"schema":1,"receivedAt":1790000001000,"session":"6f1c…","assignmentKey":"a94a8fe5ccb19ba61c4c0873d391e987","occurredAt":1790000000500,"type":"exposure","questionId":"leavingReason","componentId":"cancellation-offers","arm":"treatment","holdout":0.1,"branch":"too_expensive","goal":"cancellation_reversed"}
{"schema":1,"receivedAt":1790000031000,"session":"6f1c…","assignmentKey":"a94a8fe5ccb19ba61c4c0873d391e987","occurredAt":1790000030200,"type":"goal","goal":"cancellation_reversed"}

What's never stored

Outcome events carry IDs, arms, numbers, and times only. They never include app state, the timeline, question text, IP addresses, or user agents: the server builds each stored event field by field from the validated request, so nothing else about the HTTP request can reach a sink.

Exporting from a warehouse

If outcome events land in your warehouse through webhookSink or a custom sink, you can export them back out as JSONL with the fields above:

-- Postgres: one JSON object per row, keeping only the fields the schema defines.
COPY (
  SELECT json_strip_nulls(json_build_object(
    'schema', 1, 'receivedAt', received_at, 'session', session,
    'assignmentKey', assignment_key, 'occurredAt', occurred_at, 'type', type,
    'questionId', question_id, 'componentId', component_id, 'arm', arm,
    'holdout', holdout, 'p', p, 'branch', branch, 'goal', goal,
    'forced', forced, 'count', count
  ))
  FROM adaptif_outcomes
  WHERE received_at > extract(epoch FROM now() - interval '30 days') * 1000
) TO STDOUT;

parseStoredOutcomeEvent from @adaptif/core validates one line. Keep occurredAt in milliseconds and drop null fields rather than sending them (json_strip_nulls above), since optional fields must be absent, not null.

Attributing outcomes

  • The unit of analysis is the assignmentKey.
  • A unit is exposed to a component if it has an exposure in the period you're analyzing; its first one counts.
  • It converted if it has the goal event at or after that first exposure and within a fixed window (7 days is a reasonable default). Goals before exposure or after the window don't count.
  • Exclude units with any forced exposure, or exposures in both arms (possible when holdout changed), and count how many you excluded.
  • Engagement and dismissal rates are over treatment units.