Skip to content

[PULL REQUEST] Add Comprehensive PopulationSim Documentation - #113

Open
JiaXu1024 wants to merge 8 commits into
mainfrom
documentation
Open

[PULL REQUEST] Add Comprehensive PopulationSim Documentation#113
JiaXu1024 wants to merge 8 commits into
mainfrom
documentation

Conversation

@JiaXu1024

Copy link
Copy Markdown
Contributor

Describe this pull request. What changes are being made?

This PR adds complete documentation for SANDAG's PopulationSim implementation (prepared by Claude), including technical methodology, user guides, and reference materials.

Main Documentation File
[SANDAG_PopulationSim_Documentation.md] (Comprehensive Technical Guide)

  • Complete end-to-end technical documentation covering methodology, data preparation, execution, and validation
  • Section 1: Introduction & system architecture overview
  • Section 2: PopulationSim algorithm details including updated multi-run GQ processing architecture
  • Section 3: Data preparation (ACS PUMS seed data, UDM controls, SQL transformations)
  • Section 4: Installation, configuration (config.yml with synthesis_runs), and execution workflow
  • Section 5: Validation procedures and quality assurance metrics
  • Section 6: Optional database integration and ETL processes
  • Section 7: Appendices with troubleshooting and references

Wiki Documentation Files

  • [Home.md] - Wiki landing page with quick links, system overview, and navigation guide
  • [Getting-Started.md] - Quick start guide for new users covering installation through first synthesis run in 5 steps
  • [Installation-and-Setup.md] - Detailed installation guide including system requirements, uv package manager setup, ODBC driver configuration, and environment setup
  • [System-Overview.md] - High-level architecture diagrams showing data sources, the 4-run synthesis engine (3 GQ + 1 household), and output processing
  • [Running-PopulationSim.md] - Complete execution guide including:
  • Updated workflow showing 4 separate PopulationSim runs per year (gq_mil, gq_col, gq_oth, household)
  • Step-by-step breakdown of seed file splitting (8 files), control generation (5 files), and output merging
  • Runtime expectations (~40-50 minutes per year)
  • Progress monitoring and error handling
  • [Configuration-Reference.md] - Complete reference for all configuration files including:
  • [config.yml] with new synthesis_runs section defining 4 runs
  • secrets.yml template and security best practices
  • settings.yaml and controls.csv specifications
  • Configuration examples for various scenarios
  • [Data-Preparation.md] - Data sources and preparation including:
  • ACS PUMS seed data extraction split by HH and 3 GQ types (8 output files)
  • UDM control generation for MGRAs, PUMAs, and region
  • SQL query documentation and transformations
  • [Control-Variables.md] - Complete reference of 56 control variables with expressions, importance weights, and validation constraints
  • [Algorithm-Details.md] - Detailed explanation of IPF balancing algorithm, integerization, multiprocessing architecture, and convergence criteria
  • [Output-Files.md] - Specification of all output files including schemas, file formats, ABM integration formats, and validation summaries
  • [Validation-and-QA.md] - Quality assurance procedures including summary file validation, acceptable deviation ranges, and Streamlit dashboard usage
  • [FAQ.md] - Frequently asked questions covering general usage, technical requirements, runtime expectations, and common workflows
  • [Troubleshooting.md] - Common issues and solutions for installation errors, runtime problems, database connectivity, and performance optimization
  • [README.md] - Wiki README explaining documentation organization and contribution guidelines

Key Updates in This Documentation
All documentation has been updated to reflect recent workflow changes:

  • Multi-run GQ architecture: 3 separate GQ runs (military, college, other) plus 1 regular household run per year
  • GQ_TYPES registry: Single source of truth for GQ type definitions in [main.py]
  • Split seed files: 8 seed files (hh + 3 GQ types × households/persons)
  • Split control files: Type-specific MGRA controls for each GQ run
  • Sequential ID assignment: Merged output with collision-free household IDs
  • Updated runtimes: Reflects additional ~5 minutes for 3 GQ runs per year
  • [db.py] centralization: Database engine creation with ODBC Driver 18 compatibility (TrustServerCertificate)

The documentation provides complete guidance for technical staff, ABM users, and QA analysts to install, configure, execute, and validate PopulationSim runs for all 7 forecast years.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant