[PULL REQUEST] Add PopulationSim Datalake Exporter Script - #99
Open
anneku wants to merge 4 commits into
Open
Conversation
Contributor
|
Below are my comments:
|
JiaXu1024
reviewed
May 26, 2026
| # HOW TO RUN | ||
| # ----------------------------------------------------------------------- | ||
| # Step 1 — Activate the uv environment in command prompt: | ||
| # C:\uv_env\asim_140\.venv\Scripts\activate |
Contributor
There was a problem hiding this comment.
We need to update populaitonsim environment to have dependencies required for data lake.
Collaborator
Author
There was a problem hiding this comment.
@JiaXu1024 I updated the datalake_exporter to use the sandag-population-sim venv. Note that the Python version in the venv is quite outdated — you may want to consider upgrading it.
JiaXu1024
reviewed
May 26, 2026
| # C:\uv_env\asim_140\.venv\Scripts\activate | ||
| # | ||
| # Step 2 — Run the script: | ||
| # Usage: python datalake_exporter_popsim.py <output_path> <env> |
Contributor
There was a problem hiding this comment.
Is it okay calling the script in the main.py for different years?
Collaborator
Author
There was a problem hiding this comment.
write_to_datalake is now called from main.py
anneku
force-pushed
the
feature/datalake-exporter
branch
from
July 21, 2026 19:10
105ecb6 to
511038c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe this pull request. What changes are being made?
Adds a new script
datalake_exporter.pythat reads PopulationSim CSV output files from a specified directory, converts them to Parquet format, and uploads them to Azure Data Lake Storage underpopsim/<output_folder>/<timestamp>/. The timestamp is derived from the modification time ofsynthetic_persons.csvso each run is stored in a unique versioned folder.What issues does this pull request address?
This script provides an automated, repeatable way to export PopulationSim outputs to the data lake, replacing any manual upload process.
Additional context
devandprodenvironmentsenvargument is required and must be explicitly passed asdevorprod