Skip to content

Add construct_block / construct_item tags on dim_question - #54

Merged
countercheck merged 2 commits into
mainfrom
chore/construct-tags
Jun 5, 2026
Merged

Add construct_block / construct_item tags on dim_question#54
countercheck merged 2 commits into
mainfrom
chore/construct-tags

Conversation

@countercheck

Copy link
Copy Markdown
Owner

Summary

  • Two optional authored text attributes (construct_block, construct_item) on the survey JSON flow through int_survey_questions to new columns on dim_question, marking questions that belong to a reusable scale (PHQ-9, GAD-7, eNPS, …).
  • Provenance only — not a pooling key. Cross-survey / cross-version pooling stays the explicit parent_question_id opt-in (invariant 5); §4.10 records that decision.
  • Publish-gate validation lints shape and the block/item co-occurrence, and rejects construct_item on a matrix / paneldynamic container (an item identifies a leaf, not the container that groups leaves).
  • New singular dbt test construct_pair_integrity backstops the warehouse-side invariant against backdoor inserts.

Authoring shape

// plain question
{ "type": "radiogroup", "name": "q1", "construct_block": "enps", "construct_item": "enps_q1", "choices": [...] }

// scale-in-a-matrix: block on the matrix, item on each row (block inherits)
{ "type": "matrix", "name": "phq9", "construct_block": "phq9",
  "rows": [{ "value": "q1", "construct_item": "phq9_q1" }, ...],
  "columns": [...] }

Test plan

  • uv run ruff check . && uv run ruff format .
  • uv run mypy api/
  • python3 scripts/check_invariants.py — all 4 invariant checks clean
  • uv run pytest — 435/435 passing (incl. 13 new construct-tag tests)
  • dbt build — 59/59 models + tests passing (incl. the new construct_pair_integrity singular test)
  • Author a survey with construct tags and confirm dim_question.construct_block / construct_item populate as expected end-to-end

🤖 Generated with Claude Code

Two optional text attributes for marking a question as part of a reusable
scale (PHQ-9, GAD-7, eNPS, ...) — authored as custom JSON attributes on the
survey definition, flowing through int_survey_questions to two new columns
on dim_question. Provenance only: cross-survey/cross-version pooling stays
the parent_question_id opt-in (invariant 5).

- Publish-gate validation lints shape + the block/item co-occurrence and
  rejects construct_item on a matrix or paneldynamic container (an item
  belongs to a leaf question, not the container that groups them).
- dbt staging inherits construct_block from a matrix/panel container to its
  leaves; construct_item is leaf-only.
- New singular test construct_pair_integrity backstops the warehouse-side
  invariant against backdoor inserts.
- Design doc §3.5 gains a "Construct membership" subsection; §4.10 records
  the rejected alternative of treating construct_block as a pooling key.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds optional construct provenance tags (construct_block, construct_item) to flow from authored SurveyJS JSON through dbt staging into the dim_question mart, with publish-time validation and a warehouse-side integrity test. This supports identifying questions that belong to reusable scales (e.g., PHQ-9, GAD-7) without changing the explicit cross-survey/version pooling mechanism (parent_question_id).

Changes:

  • Add publish-gate validation + unit tests to enforce construct tag shape, container/leaf placement rules, and block/item co-occurrence.
  • Extend dbt intermediate + mart models to propagate construct tags into dim_question, and document the new columns.
  • Add a dbt singular test to enforce construct_itemconstruct_block integrity.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
survey-engine-design-doc.md Documents new dim_question columns and clarifies “provenance only” (not a pooling key).
dbt/tests/singular/construct_pair_integrity.sql Adds singular test asserting construct_item is never present without construct_block.
dbt/models/marts/dim_question.sql Adds construct_block/construct_item columns to the stable question dimension.
dbt/models/marts/_marts.yml Documents the two new dim_question columns for analysts.
dbt/models/intermediate/int_survey_questions.sql Extracts construct tags from definition JSON, including inheritance for matrix/paneldynamic leaves.
api/tests/test_publish_gate.py Adds publish-gate tests covering construct tag acceptance/rejection cases.
api/survey_engine/validation.py Implements publish-time validation for construct tag type, placement, and inheritance rules.

Comment on lines 42 to +44
min(panel_name) as panel_name,
min(construct_block) as construct_block,
min(construct_item) as construct_item,
Comment on lines +609 to +615
if qtype in MATRIX_TYPES:
if isinstance(owner_item, str):
raise InvalidDefinition(
f"question {name!r}: construct_item belongs to a matrix row, "
"not the matrix itself — move it onto the row"
)
for row in element.get("rows", []) or []:
…ility test

Two silent-failure paths Copilot flagged on #54:

1. construct_block / construct_item on a matrixdropdown column were accepted
   by validation but silently dropped by int_survey_questions (which only
   reads construct_* from rows). Reject loudly at publish — a cell's construct
   identity is its row's, and per-column tagging is a future extension if we
   ever need different items per row dimension.

2. dim_question collapses the (survey_id, stable_name) grain with min(), which
   silently picks one tag when versions disagree (v1 'phq9_q1', v2 'gad7_q1').
   Methodologically a tag change is the same event as a rename and should go
   through parent_question_id — so surface the divergence rather than hide it.
   New construct_tag_stability singular test fails when a question has more
   than one distinct non-null construct_block / construct_item across versions
   (untagged → tagged backfill is still permitted).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@countercheck
countercheck merged commit a14959f into main Jun 5, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants