Implement HFC checks 1,3-6 per DQAP coverage mapping - #2
Open
aleortiz-uchicago wants to merge 1 commit into
Open
Implement HFC checks 1,3-6 per DQAP coverage mapping#2aleortiz-uchicago wants to merge 1 commit into
aleortiz-uchicago wants to merge 1 commit into
Conversation
- 1-import: read .csv (raw is CSV, not DTA); recode -888/-999 sentinels to missing (keep -666 = "Other" gate on hh_watersource/stored_container) - 3-missingness: consent + all always-asked fields + Section D conditional on stored_yn gate, share missing by enumerator (data_02) - 4-outliers: Tukey fences on resp_age, hh_size, hh_children, storage_time (excl. 99), treat_chlorine, treat_boil, duration_min, child_age_* (data_03) - 5-constraints: fixed ranges + storage_time 0-72-or-99 + relational hh_children <= hh_size (data_04) - 6-surveylogic: consent/stored_yn/hh_children skip gates, Other-specify mismatches, diarrhea reverse skip, roster count, head-under-18 (data_05)
There was a problem hiding this comment.
Pull request overview
Implements DQAP coverage mapping checks across the HFC Stata pipeline, producing standardized Excel outputs for missingness (by enumerator), outliers (Tukey fences), constraints (valid ranges + relational), and survey-logic/skip-pattern inconsistencies.
Changes:
- Switched import to read raw
.csvand recoded-888/-999sentinel codes to Stata missing in1-import.do. - Added automated outlier flagging via Tukey fences in
4-outliers.doand expanded constraint validation (including relational checks) in5-constraints.do. - Implemented missingness reporting (conditional on gates) in
3-missingness.doand implemented multiple survey-logic/skip checks in6-surveylogic.dowith a unified “append then export once” pattern.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| code/1-import.do | Imports CSV and standardizes sentinel missing codes for downstream checks. |
| code/3-missingness.do | Computes per-enumerator missingness for always-asked and gated fields and exports to data_02. |
| code/4-outliers.do | Flags outliers via Tukey fences for key numeric fields and exports to data_03. |
| code/5-constraints.do | Flags fixed-range and relational constraint violations and exports to data_04. |
| code/6-surveylogic.do | Flags skip-pattern and logic inconsistencies and exports to data_05. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+41
to
+46
| use `base', clear | ||
| keep if !missing(`v') & (`v' < `lo' | `v' > `hi') | ||
| keep hh_id enumerator `v' | ||
| rename `v' value | ||
| gen variable = "`v'" | ||
| gen valid_range = "[`lo', `hi']" |
Comment on lines
48
to
+52
| gen byte _has_hhid = !missing(hh_id) | ||
| keep if consent == 1 | ||
| collapse (count) n_surveys=_has_hhid (mean) pct_miss_consent=_miss_consent /// | ||
| /* (mean) pct_miss_storage=_miss_storage */ , by(enumerator) | ||
| * gsort -pct_miss_storage | ||
| format pct_miss_* %9.2f | ||
| collapse (count) n_surveys=_has_hhid (mean) _miss_*, by(enumerator) | ||
| rename _miss_* pct_miss_* | ||
| format pct_miss_* %9.3f |
Comment on lines
+75
to
+76
| gen byte _bad = hh_children == 0 & /// | ||
| (!missing(child_name_1) & child_name_1 != "" | !missing(child_age_1)) |
Comment on lines
+154
to
+156
| egen _nchild = rownonmiss(child_age_1 child_age_2 child_age_3) | ||
| gen byte _bad = consent == 1 & !missing(hh_children) & hh_children <= 3 & _nchild != hh_children | ||
| keep if _bad |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the DQAP coverage mapping across the HFC pipeline.
.csv(was.dta); recode-888/-999sentinels to missing, keep-666="Other" gate on hh_watersource/stored_container.stored_yngate; share missing by enumerator.Full
main.doruns clean in Stata 19 — no errors, all 8 sheets written.