diff --git a/CLAUDE.md b/CLAUDE.md index 87a1dda..f4e740c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -51,9 +51,9 @@ Analyst/reviewer DB credentials are minted out of band of `stele_api` (design do 1. `app.raw_responses` is append-only and the sole ETL source. dbt reads it only, never the normalized `app.responses` / `app.response_items` read-model. 2. Published surveys are immutable. Edits → new draft → new version → new hash. Every response carries the hash it was answered against. -3. Shown-set is captured at submit time from the SurveyJS engine. Never reconstruct routing by re-evaluating `visibleIf` in SQL. +3. Shown-set AND render order are captured at submit time from the SurveyJS engine. `shown_questions` is in render order; `shown_choice_orders` carries the rendered choice order for any question whose options were randomized. Never reconstruct routing or display order by re-evaluating the definition in SQL. (Design doc §3.5 "Randomization and display-order capture", §4.11.) 4. One JSON parser. API → operational read-model. dbt → warehouse. Both from `raw_responses`. Never chain them. *Enforced by `scripts/check_invariants.py`, which scans dbt SQL for forbidden table references and verifies `sources.yml` declares only `raw_responses` under the `app` schema. Pre-commit runs it; CI verifies.* -5. No cross-version pooling by default. Opt-in via `dim_question.parent_question_id` + `parent_question_rationale`. Never auto-populate either. *Lint checks that writes to `parent_question_id` co-occur with `parent_question_rationale`.* +5. No cross-version pooling by default. Opt-in via `dim_question.parent_question_id` + `parent_question_rationale`. Never auto-populate either. `construct_block` / `construct_item` are **provenance, not a pooling key** — they surface "this is a PHQ-9 item" but never license `GROUP BY construct_block` across surveys; pooling stays the `parent_question_id` opt-in. *Lint checks that writes to `parent_question_id` co-occur with `parent_question_rationale`; `construct_pair_integrity` enforces item⇒block; `construct_tag_stability` rejects within-survey tag drift across versions.* 6. Free-text defaults to `pii_risk = 'high'`. `marts.fact_response_item.value_text` populated only for explicit `low`. *Lint checks that `value_text` writes to `fact_response_item` reference `pii_risk`.* 7. Fact grain: `(respondent, survey_version, question_id, occurrence, selected_option)`. `fact_response` (respondent-question grain) is a separate table. Don't conflate. 8. Exactly one of `{option_key, value_numeric, value_text, value_date}` per fact row. dbt test enforces; don't skip. @@ -86,11 +86,12 @@ Work on branches; never commit directly to `main`. One branch per story/unit of ## Don't add silent defaults for methodological judgments -Three judgments stay explicit, always. Examples of what going-wrong-silently looks like: +Four judgments stay explicit, always. Examples of what going-wrong-silently looks like: -- **Cross-version equivalence.** Don't auto-fill `parent_question_rationale` from prompt similarity between v1 and v2. That's a researcher's call, not a heuristic. +- **Cross-version equivalence.** Don't auto-fill `parent_question_rationale` from prompt similarity between v1 and v2. That's a researcher's call, not a heuristic. Same for `construct_block` ⇒ `parent_question_id`: a shared scale tag is provenance, not equivalence — don't promote it. - **Free-text safety.** Don't downgrade a question from `pii_risk = 'high'` to `'low'` because the prompt "looks innocuous." The default is `'high'`; downgrades are deliberate decisions at definition time. - **Shown vs skipped vs routed-past.** Don't collapse `was_shown = false` and `was_shown = true, value null` into "missing." They mean different things; analyses depend on the distinction. +- **Display order.** Don't render randomized questions/choices without writing the realized order back to `raw_responses` (in `shown_questions` order + `shown_choice_orders`). Don't reconstruct `display_order` in SQL from the published definition — the definition holds the rule, not the per-respondent realization. Treat order capture as part of the submission contract, the same way the shown-set is. If you're writing a default for any of these, that's the bug. @@ -100,6 +101,8 @@ Schema validation → lint (dup names, dangling `visibleIf`, dup option values, New question type = work in three places: runtime, publish test, dbt staging. +Within-page (`Page.questionsOrder`), within-matrix (`Matrix.rowsOrder`), and within-question (`SelectBase.choicesOrder`) randomization are unconstrained — values are stable across re-orderings, and `visibleIf` reads values, not positions. Static-panel and survey-level `pagesOrder` shuffling aren't wired today (static panel is outside the accepted type surface; `pagesOrder` isn't native SurveyJS) — deferred per design doc §5. The page-order extension lands with its own publish-gate lint (reject cross-page `visibleIf` under page randomization). + ## dbt - Staging = views; intermediate + marts = tables; no materialized views. @@ -114,7 +117,8 @@ New question type = work in three places: runtime, publish test, dbt staging. - Give `stele_api` `CREATEROLE`, or put credential role-DDL in the request path. Role-minting lives ONLY in `api.credential_worker` / the `scripts/provision_db_credential.py` CLI over `STELE_PROVISION_DATABASE_URL`; the API just enqueues into `app.provision_requests`. And `stele_analyst`/`stele_pii_reviewer` must never accumulate grants beyond their one schema (`marts`/`pii`) — that boundary is what keeps a leaked analyst credential low-stakes. - Make dbt read normalized `app.*` tables. - `UPDATE`/`DELETE` `raw_responses` outside the tombstone workflow. -- Auto-populate `parent_question_id` from any heuristic. +- Auto-populate `parent_question_id` from any heuristic — including a shared `construct_block` (provenance, not equivalence; design doc §4.10). +- Reconstruct per-respondent display order in SQL from the published definition. The definition holds the rule (`questionsOrder: 'random'`, etc.); the per-respondent realization is captured at submit time and read straight through to `fact_response_item.display_order` (design doc §4.11). - Add a new runtime component (broker, search, separate analytics DB) — check design doc § 5 triggers first. - Postgres-specific SQL in dbt without dispatch or a documented reason. - `git rm` survey definitions or response data. Use the database. diff --git a/patterns/README.md b/patterns/README.md index 74ece36..f2fde72 100644 --- a/patterns/README.md +++ b/patterns/README.md @@ -58,6 +58,7 @@ a panel free-text cell stays `value_text` regardless of `inputType`. | [branching.json](branching.json) | `visibleIf` conditional routing, including complementary branches | | [multi_page.json](multi_page.json) | multiple pages; a `visibleIf` referencing an answer from an earlier page | | [calculated_values.json](calculated_values.json) | a `calculatedValue` as a reusable, named `visibleIf` condition | +| [randomization.json](randomization.json) | `questionsOrder: 'random'` (page + panel) and `choicesOrder: 'random'`, plus the block-aware shape (a `construct_block`-tagged panel shuffling its items while staying contiguous); per-respondent display order is captured at submit time and threads to `fact_response_item.display_order` | Each file annotates itself inline: the survey- and question-level `title` and `description` fields are real SurveyJS properties (they render in the runtime), so diff --git a/patterns/randomization.json b/patterns/randomization.json new file mode 100644 index 0000000..87542b2 --- /dev/null +++ b/patterns/randomization.json @@ -0,0 +1,76 @@ +{ + "title": "Randomization (question order, matrix-row order, choice order)", + "description": "questionsOrder: 'random' on a page shuffles its questions per respondent; rowsOrder: 'random' on a matrix shuffles its row sub-questions; choicesOrder: 'random' on a single/multi-select/ranking shuffles its options. The order each respondent actually saw is captured at submission time — shown_questions in render order, shown_choice_orders for randomized choices — and threads to fact_response_item.display_order (design doc §3.5 'Randomization and display-order capture'; never reconstructed in SQL, §4.11). Block-aware randomization is structural, not a separate flag: a reusable scale fits naturally as a matrix tagged with construct_block (the tag inherits to row sub-questions per §3.5 'Construct membership'), so PHQ-9 items shuffle among themselves and stay contiguous because the matrix is the boundary. visibleIf reads values, not display positions, so within-page / within-matrix / within-question randomization is always safe to combine with branching. Static-panel and survey-level page shuffling aren't wired today (deferred, design doc §5).", + "pages": [ + { + "name": "p_warmup", + "questionsOrder": "random", + "title": "Warm-up questions (randomized)", + "description": "questionsOrder: 'random' on a page shuffles the page's elements per respondent. The two questions here are unrelated, so order is the only thing that changes.", + "elements": [ + { + "type": "radiogroup", + "name": "coffee", + "title": "Coffee or tea?", + "choices": ["coffee", "tea", "neither"] + }, + { + "type": "radiogroup", + "name": "morning_or_night", + "title": "Morning person or night owl?", + "choices": ["morning", "night", "depends"] + } + ] + }, + { + "name": "p_scale", + "elements": [ + { + "type": "matrix", + "name": "phq9", + "title": "Over the last two weeks, how often have you been bothered by…", + "description": "Matrix with rowsOrder: 'random' is the block-aware-randomization shape: each row is its own sub-question (stable_name 'phq9.q1', 'phq9.q2', 'phq9.q3'), the shared response scale lives in columns, and the rows shuffle per respondent. construct_block on the matrix inherits to the row sub-questions in dim_question (design doc §3.5 'Construct membership' inheritance rule), so each row carries construct_block='phq9'. construct_item is leaf-only — set per row, not on the container.", + "construct_block": "phq9", + "rowsOrder": "random", + "rows": [ + { "value": "q1", "text": "Little interest or pleasure in doing things", "construct_item": "phq9_q1" }, + { "value": "q2", "text": "Feeling down, depressed, or hopeless", "construct_item": "phq9_q2" }, + { "value": "q3", "text": "Trouble falling/staying asleep, or sleeping too much", "construct_item": "phq9_q3" } + ], + "columns": [ + { "value": "0", "text": "Not at all" }, + { "value": "1", "text": "Several days" }, + { "value": "2", "text": "More than half the days" }, + { "value": "3", "text": "Nearly every day" } + ] + } + ] + }, + { + "name": "p_choices", + "elements": [ + { + "type": "radiogroup", + "name": "favorite_color", + "title": "Pick a favorite color", + "description": "choicesOrder: 'random' shuffles the options shown to this respondent. The chosen option_key is unchanged — selection is by value, not position — so dim_option.display_order (the authored position) and the per-respondent rendered position are different things. The per-respondent rendered choice order lives in raw_responses.shown_choice_orders; a fact-side choice_display_order column is deferred (design doc §5 'Choice-order analysis surface').", + "choicesOrder": "random", + "choices": [ + { "value": "red", "text": "Red" }, + { "value": "green", "text": "Green" }, + { "value": "blue", "text": "Blue" }, + { "value": "yellow", "text": "Yellow" } + ] + }, + { + "type": "radiogroup", + "name": "color_followup", + "title": "Is that the color of something you own?", + "description": "visibleIf reads the chosen value ({favorite_color}), not its rendered position, so choices-order randomization never affects branching reachability. The publish gate's round-trip oracle treats randomized choices as order-invariant for the same reason (design doc §3.6).", + "visibleIf": "{favorite_color} notempty", + "choices": ["yes", "no", "not_sure"] + } + ] + } + ] +} diff --git a/survey-engine-design-doc.md b/survey-engine-design-doc.md index 4b7164c..f864289 100644 --- a/survey-engine-design-doc.md +++ b/survey-engine-design-doc.md @@ -85,7 +85,8 @@ Out of scope: respondent recruitment, IRB processes, statistical methodology, da **FR-5. Response capture.** On submission, the system records: - The full SurveyJS response payload, unmodified. -- The shown-set as reported by the SurveyJS engine. +- The shown-set as reported by the SurveyJS engine, **in render order** (the top-to-bottom order the respondent actually saw across all visited pages). +- For any question whose options were randomized, the rendered choice order per question. - A reference to the exact published version (by hash). - Server timestamp and client metadata. @@ -102,6 +103,8 @@ Out of scope: respondent recruitment, IRB processes, statistical methodology, da **FR-10. Analytical access.** Analysts can query the warehouse via SQL (Postgres), with predictable joins between fact and dimension tables. R via `DBI`/`dbplyr` and Python notebook environments are first-class. +**FR-13. Randomization with per-respondent display-order capture.** The runtime supports shuffling within the published type surface: question order within a page (`Page.questionsOrder: 'random'`), row order within a matrix (`Matrix.rowsOrder: 'random'`), and choice order within a single/multi-select/ranking question (`choicesOrder: 'random'`). The construct-block-aware case emerges from putting a block's items inside a randomized container (typically a matrix; §3.5 "Construct membership"). For every randomized question, the order the respondent actually saw is captured at submission time, never reconstructed downstream, so order effects are analyzable. Survey-level page randomization and static-panel randomization are deferred (§5). + **FR-11. ETL reproducibility.** The full marts schema can be rebuilt from `raw_responses` with a single command. Every ETL run is logged with row counts, timings, and reproducibility metadata. **FR-12. Edit history.** Response edits create new rows in `raw_responses` rather than updating existing ones. Full submission history is preserved. @@ -227,7 +230,8 @@ Append-only audit log. **Sole source of truth for ETL.** | `survey_id`, `survey_version` | Identifies the exact published definition answered | | `submitted_at` | Server timestamp | | `payload` | Raw SurveyJS response JSON (jsonb) | -| `shown_questions` | jsonb array of `question_id`s actually rendered; written by the API at submission time from the SurveyJS engine's own visibility state | +| `shown_questions` | jsonb array of `question_id`s actually rendered, **in render order** (the top-to-bottom order the respondent saw across all visited pages); written by the API at submission time from the SurveyJS engine's own visibility state. Pre-FR-13 responses, where no randomization existed, are already in render order by construction (definition order = render order), so the spec change is forward-only and needs no backfill. | +| `shown_choice_orders` | jsonb. Optional map `{question_name: [option_value, …]}` recording the rendered choice order *per question whose options were randomized*. Captured at submission time. Sparse: omits questions whose `choicesOrder` was not random, since for them render order equals definition order and `dim_option.display_order` already encodes it. | | `client_metadata` | User agent, locale, etc. (jsonb) | | `definition_snapshot` | jsonb. Frozen copy of the published definition (the SurveyJS JSON plus its `definition_hash` and `published_at`) the response was answered against. Written by the API at submission time. | @@ -316,6 +320,7 @@ erDiagram date value_date boolean was_shown int rank + int display_order } ``` @@ -374,6 +379,33 @@ Inheritance rule for the composite types: a `construct_block` on a matrix or pan The columns live on `dim_question` (the stable abstraction), not `dim_question_version`, because construct membership — like cross-version equivalence above — is about the question as a construct, not about a specific rendering of it. +#### Randomization and display-order capture + +Surveys may shuffle what each respondent sees. The supported mechanisms are SurveyJS-native and constrained to the question-type surface the publish gate accepts; the warehouse contribution is **recording, per respondent, the order they actually saw**, never reconstructing it. + +| Mechanism | Authored as | Scope | +|---|---|---| +| Question order within a page | `Page.questionsOrder: 'random'` | The questions on that page are shuffled per respondent; questions on other pages are unaffected. | +| Row order within a matrix | `Matrix.rowsOrder: 'random'` | The matrix's row sub-questions are shuffled per respondent. The container's column order (the response scale) is unaffected. | +| Choice order within a question | `SelectBase.choicesOrder: 'random'` on `radiogroup` / `dropdown` / `checkbox` / `ranking` | The options shown for that question are shuffled per respondent. | + +Static-panel and survey-level page randomization aren't wired today (§5): static `panel` is outside the accepted question-type surface, and `Survey.pagesOrder` isn't a built-in SurveyJS property — implementing either is a deferred extension, not a default. + +There is no separate "block-aware randomization" mechanism: a construct block expressed as a `matrix` with `rowsOrder: 'random'` is exactly the block-internal shuffle, and the surrounding container (carrying the `construct_block` tag, inherited to its row sub-questions per §3.5 "Construct membership") keeps the block contiguous because shuffling is bounded by the container. The author who wants "shuffle PHQ-9 items within themselves but keep the PHQ-9 block together" gets it by structure rather than by a new flag. + +What we capture and where it lands: + +| At submit time, on `raw_responses` | In the marts | +|---|---| +| `shown_questions` (jsonb array of `question_id`s, in render order) | `fact_response_item.display_order` — integer position of this question within the respondent's rendered sequence, derived from the array index in `shown_questions`. Same value across all option rows for the same `(respondent, question_id, occurrence)` (multi-select fan-out, ranking rows, matrix/panel cells). Null when `was_shown = false` (the question was routed past). | +| `shown_choice_orders` (jsonb map, sparse) | Not surfaced in marts today; see §5 "Choice-order analysis trigger." The raw capture is the backfill hook: when an analyst wants to test for choice-position effects, the column can be added to `fact_response_item` and populated from `shown_choice_orders` without re-collection (A-2). | + +Why this shape: + +- **Render order, not configured order.** Mirrors `was_shown` (invariant 3): the engine is the source of truth for what happened, not the SQL reconstruction of what was supposed to happen. The published definition records the randomization config (and is preserved on every response via `definition_snapshot`), but the per-respondent rendering is captured directly — there is no inference step that could drift. +- **Question display order on `fact_response_item`, not `fact_response`.** The user picks the selection-grain fact as the home so analyses already filtering by `option_key` or `value_*` get `display_order` without an extra join; the value is harmlessly denormalized across the fan-out for multi-select/matrix/panel rows. +- **No `dim_question_version.questions_order_scope` column.** Whether a question was authored as randomized is recoverable from the published JSON (snapshotted on every response); recording it again on the dimension would risk drifting from the snapshot for no benefit at the current scale. Analysts who want to know "was this question randomized" read the definition. Analysts who want to know "what did this respondent see" read `display_order`. + #### Indexes - `(survey_version_id, question_id)` — per-question aggregations within a version. @@ -392,7 +424,8 @@ Gates: - Schema validation against the SurveyJS JSON schema. - Lint checks: duplicate question names, dangling `visibleIf` references, duplicate option values, missing matrix row identifiers. -- Round-trip test (for surveys flagged as going to real respondents): headless run with synthetic respondents covering branches; response payload matches expectations. +- **Randomization × routing safety** (FR-13): within-page (`Page.questionsOrder`), within-matrix (`Matrix.rowsOrder`), and within-question (`choicesOrder`) randomization are unconstrained — per-respondent values are stable across re-orderings and `visibleIf` operates on values, not positions. The deferred page-order extension (§5) lands with a lint that rejects cross-page `visibleIf` / `enableIf` under page randomization, by the same logic. +- Round-trip test (for surveys flagged as going to real respondents): headless run with synthetic respondents covering branches; response payload matches expectations. With `choicesOrder: 'random'`, the oracle's reachability check uses any one rendered order — `visibleIf` reads choice *values*, not display positions, so the analysis is order-invariant. - Hash and freeze. Principle: **immutability matters at publish time, not at author time.** Drafts are throwaway and feel that way. Published surveys are forever and are treated that way. @@ -567,6 +600,14 @@ Analyst and reviewer *data* access is not mediated by the application. Analysts **Rejected.** A `construct_block` tag declares provenance ("this question is part of the PHQ-9 instrument"); it does not declare that two tagged questions are analytically equivalent. Reusable scales are sometimes administered with subtle changes — translated wording, response-scale anchor edits, dropped items — that a researcher may or may not be willing to pool across. Treating shared `construct_block` as automatic pooling would re-introduce the silent-default failure mode invariant 5 was designed to prevent (see §4.8). Pooling stays the `parent_question_id` opt-in; `construct_block` is metadata that helps a researcher *find* the questions that might be candidates for that judgment, not a substitute for making it. +### 4.11 Reconstructing per-respondent display order in SQL from the published definition + +**Rejected** for the same reason as §4.5. The published definition encodes randomization *rules* (`questionsOrder: 'random'`, `choicesOrder: 'random'`, etc.), not the seed or the realized order any individual respondent saw. Without the seed, dbt cannot reproduce the order; with the seed, the engine would still be the authoritative source for edge cases (mid-session re-renders, back-navigation, accessibility re-orderings). Capturing the rendered order at submission time — `shown_questions` for questions, `shown_choice_orders` for choices — is the only path that doesn't drift. Like the shown-set, this is a *capture*, not an *inference*: a richer per-respondent ordering dimension (e.g., choice-order in marts) is backfillable from the raw capture without re-collection. + +### 4.12 Recording randomization config on `dim_question_version` + +**Rejected.** "Was this question randomized?" is recoverable from the published JSON, which is snapshotted on every response (`raw_responses.definition_snapshot`) and lossless. Duplicating the same fact on the dimension would either drift from the snapshot (a silent inconsistency the design refuses, per NFR-3) or require a custom test to pin them together — overhead with no analytical benefit at scale. Analysts who want "was randomization on for this question version" join the snapshot; analysts who want "what did this respondent see" read `fact_response_item.display_order`. The two questions stay separate, which is the right shape. + --- ## 5. Deferred decisions @@ -582,6 +623,9 @@ These are reopened when explicit triggers are met, not on a schedule. | Full routing-trace dimension | Research questions requiring reconstruction of the decision graph beyond what `was_shown` + raw `shown_questions` arrays support. (Backfillable without re-collection.) | | Automated PII detection as first pass on free-text | Free-text volume exceeding reviewer capacity. | | Org/delivery dimension | One survey (instrument) delivered to multiple organizations/cohorts that must be analyzable independently *and* pooled. Likely an attribute on `dim_respondent` (the respondent's org) — orthogonal to question identity, so analyses filter by org for "independently" and omit it for "altogether," while the survey-scoped `question_id` keeps pooling correct. Triggered when a survey is first delivered to more than one org. | +| Choice-order analysis surface | An analyst needs to test for choice-position effects on a randomized-choice question. The raw `shown_choice_orders` is already captured per §3.4; this adds a `choice_display_order` column to `fact_response_item` (integer position of the chosen option within the respondent's rendered choices for that question, null when the question's choices weren't randomized) and the dbt model to populate it from `shown_choice_orders`. Backfillable without re-collection. | +| Static-panel container in the publish gate | An author wants `type: 'panel'` (a non-repeating grouping element) to organize a long survey without using a matrix or paneldynamic. Adds `panel` to the accepted type surface and threads it through dbt staging (no value semantics — a panel is structural — but its `construct_block` would need to inherit to leaves per §3.5). Triggered when grouping ergonomics on a long survey outweigh the "matrix or paneldynamic only" simplicity. | +| Survey-level page-order randomization | An author wants pages shuffled per respondent. Requires a custom shuffler at render time (`Survey.pagesOrder` isn't native) and the publish-gate lint already specified under §3.6 (reject under cross-page `visibleIf`). Triggered when a study design genuinely needs counterbalanced page order. | --- @@ -594,6 +638,8 @@ These are reopened when explicit triggers are met, not on a schedule. | Analyst silently pools questions across versions/surveys that aren't equivalent | `question_id` is survey-scoped `(survey_id, stable_name)`, so it never pools across surveys; `GROUP BY question_version_id` is strict per-version; pooling across a rename requires the explicit `canonical_question_id` opt-in. | | Reworded item silently treated as the same question because its name was kept | Within a survey, keeping `stable_name` across versions *is* the continuity assertion (per-version wording is preserved in `dim_question_version`); a genuine construct change should be a rename, which breaks `question_id` and forces the explicit equivalence opt-in. | | Construct tags treated as a pooling key by an over-eager analyst | `construct_block` / `construct_item` are documented as provenance only; the canonical pooling key remains `canonical_question_id` (built from `parent_question_id`); the warehouse exposes no `GROUP BY construct_block` shortcut for cross-survey rollups. | +| Order effects in a randomized survey go undetected because display order is reconstructed inconsistently or not at all | Per-respondent display order is captured at submission time (`shown_questions` in render order; `shown_choice_orders` for randomized choices) and threaded through to `fact_response_item.display_order`. Reconstruction in SQL is rejected (§4.11). Order-effect analyses use the captured column, not a downstream re-derivation. | +| When page-order randomization (§5) lands, cross-page conditional routing could produce incoherent surveys (a `visibleIf` reading a question not yet asked) | The deferred extension carries its own publish-gate lint: reject page randomization combined with any cross-page `visibleIf` / `enableIf`. Within-page (`Page.questionsOrder`), within-matrix (`Matrix.rowsOrder`), and within-question (`choicesOrder`) randomization — which *are* shipped — don't trip this, because values are stable across re-orderings and `visibleIf` reads values, not positions. | | LLM-generated JSON with subtle logic errors | Round-trip test gate at publish time; pattern library reduces invention surface. | | Two parsers (API and dbt) drift apart over time | dbt reads from `raw_responses` only; normalized tables are a read-model, not an ETL input. | | Analyst confuses "selection count" with "respondent count" | Companion `fact_response` table at respondent-question grain; documented in marts; default examples use `COUNT(DISTINCT respondent_id)`. | diff --git a/survey-engine-for-researcher.md b/survey-engine-for-researcher.md index f9e8725..8bfac11 100644 --- a/survey-engine-for-researcher.md +++ b/survey-engine-for-researcher.md @@ -116,7 +116,7 @@ publish, by design: the system won't accept a question whose answers it can't pl --- -## 4. The three commitments +## 4. The four commitments These are the judgments the system will never make for you. Each one is a place where a "helpful" default would quietly corrupt an analysis, so the system makes the @@ -145,7 +145,15 @@ analyses then opt in by using the derived canonical key (`COALESCE(parent_question_id, question_id)`). What the system refuses: auto-pooling from prompt similarity. That would absorb your -judgment into a default and produce silently inconsistent longitudinal series. +judgment into a default and produce silently inconsistent longitudinal series. The +same refusal extends to **construct tags**: you can tag a question with +`construct_block: "phq9"` and `construct_item: "phq9_q3"` to record that it came from +a named, reusable scale (PHQ-9, GAD-7, eNPS, …), and those tags travel into +`dim_question` so you can *find* every PHQ-9-tagged question across every survey. +But shared tags do **not** license pooling — two surveys' PHQ-9 items are still +distinct `question_id`s, and pooling them is still the explicit `parent_question_id` +opt-in, with rationale. Tags surface candidates for that judgment; they don't make +it for you. > Status: the integrity guard ships today (dbt’s `parent_question_integrity` test), but there isn’t yet a supported workflow to *persist* equivalence decisions — `dim_question` emits these columns as nulls. > The ergonomic tooling + storage around declaring equivalence is the natural next build — @@ -191,6 +199,45 @@ this grid cell" stays distinct from "this respondent's branch never reached the What the system refuses: collapsing `was_shown = false` and `was_shown = true, value null` into one "missing" bucket. +### d. Display order is what the respondent saw, never what the definition says + +Surveys can randomize what each respondent sees — question order within a page, +row order within a matrix, and choice order within a single/multi-select/ranking +question. Whenever shuffling is on, the **order that respondent actually saw is +captured at submission time**, the same way the shown-set is, and lands in +`fact_response_item.display_order` as an integer position within their rendered +sequence (FR-13). + +What you can randomize: + +- **Within a page.** Set `questionsOrder: "random"` on a page. Questions on that + page shuffle per respondent; questions on other pages are unaffected. +- **Within a matrix.** Set `rowsOrder: "random"` on a matrix. The rows (each row + is its own sub-question — see [`patterns/matrix.json`](patterns/matrix.json)) + shuffle per respondent; the column order (the response scale) is unchanged. + This is also how you get **block-aware randomization**: a reusable scale fits + naturally as a matrix (rows = items, columns = response options); tag the + matrix with `construct_block: "phq9"` (the tag inherits to row sub-questions), + set `rowsOrder: "random"`, and PHQ-9 items shuffle among themselves while + staying contiguous because the matrix is the boundary. No separate flag. +- **Within a question.** Set `choicesOrder: "random"` on a `radiogroup`, + `dropdown`, `checkbox`, or `ranking`. The rendered choice order is captured + per-respondent at submit time (in `shown_choice_orders` on the raw response). + Choice-position effects are analyzable; the marts column to surface this is + deferred until first need (design doc §5 "Choice-order analysis surface"). + +Not built today: survey-level page shuffling, and randomization in a static `panel` +container — see "What's intentionally not here" (§8) for the deferred paths and +their triggers. + +What the system refuses: reconstructing the display order in SQL from the published +definition. The definition records the *rule* (`rowsOrder: "random"`), not the +realized order any individual respondent saw. The engine that drew the screen is the +authoritative source; the warehouse reads it through, never recomputes it. +Reading `MIN(display_order)` vs `MAX(display_order)` across respondents tells you +*which* questions were randomized after the fact; comparing answer rates or +distributions across `display_order` quartiles is how you test for order effects. + --- ## 5. PII and the reviewer pass @@ -260,6 +307,15 @@ value_date}`. Which one is determined by the question's `value_kind` `value_numeric` and not among the free text — worth knowing when you filter by "response type." +### Order effects, when randomization is on + +`fact_response_item.display_order` is the integer position of the question within +the respondent's rendered sequence — the same value across all option rows for the +same `(respondent, question_id, occurrence)` (it's a question-grain property +denormalized onto selection-grain rows). Null when the question was routed past +(`was_shown = false`). Use it to test for ordering effects on randomized questions; +ignore it when randomization wasn't on (it just mirrors definition order). + ### Pooling across versions `question_id` is survey-scoped (`(survey_id, stable_name)`): it pools a survey's @@ -287,6 +343,17 @@ So expectations are calibrated (design doc §5): - **Equivalence-pooling tooling** beyond the integrity guard — declarable, but not yet ergonomic (§4a). +- **Choice-position order in the marts.** The rendered choice order is captured per + respondent at submit time (`shown_choice_orders` on the raw response), so it's + there if you need it; a `choice_display_order` column on `fact_response_item` is + the natural next surface and backfillable without re-collection — built when an + analysis first wants it (§4d, design doc §5). +- **Static-panel grouping.** A non-repeating `panel` container for organizing long + surveys is outside today's accepted type surface; use a matrix or a page to + group items for now. Adding `panel` is a deferred extension (design doc §5). +- **Survey-level page shuffling.** `pagesOrder: "random"` isn't a native SurveyJS + property; if a study design genuinely needs counterbalanced page order, that's a + custom shuffler — deferred until a study actually wants it (design doc §5). - **Automated PII detection** — the reviewer pass is human today (§5). - **Scheduled/automatic ETL** — the warehouse rebuilds on demand, when you ask. - **A visual survey designer** — authoring is JSON + live preview + the pattern