Skip to content

WEATHER.md

Latest commit

 

History

History
130 lines (95 loc) · 4 KB

WEATHER.md

File metadata and controls

130 lines (95 loc) · 4 KB
Raw

Plan: Add weather conditions to CMAP attributes

Context

The user wants to enrich the attributes table with daily weather conditions (max temperature, rain indicator) for the day of travel. This is only feasible where exact calendar dates are preserved in the raw survey data.

Date precision per source (summary)

Source Year Month Calendar day Verdict
CMAP ✓ ✓ ✓ (raw travdate is YYYY-MM-DD) Viable
NHTS ✓ ✓ ✗ (YYYYMM only) Not possible
LTDS ✓ ✗ ✗ Not possible
NTS ✓ ✓ ✗ Not possible
QHTS ✓ ✓ ✗ Not possible
VISTA ✓ ✓ ✗ Not possible

CMAP is the only source where the raw household CSV (household.csv) has a full travdate (YYYY-MM-DD) field. The current code parses it but immediately drops it after extracting year/month/weekday. CMAP covers travel years 2017–2019 (Chicago metropolitan area).

Weather data source: Open-Meteo

Open-Meteo historical weather API

  • Free, no API key required
  • REST API returning JSON; callable with urllib.request (no new deps)
  • Global coverage, 9 km grid resolution from 2017 onward (perfect for CMAP 2017–2019)
  • Historical data back to 1940
  • Relevant daily variables:
    • temperature_2m_max → max_temp_c (°C)
    • precipitation_sum → derive rain as precipitation_sum > 0

For all CMAP respondents, use Chicago city-centre coordinates (lat = 41.85, lon = −87.65). The metro area spans ~50 miles but daily max temp and rain are sufficiently uniform at this scale.

Implementation

1. scripts/fetch_weather.py (new)

One-time script to download Chicago daily weather and write a lookup CSV.

scripts/fetch_weather.py
  --start  DATE  (default: 2017-01-01)
  --end    DATE  (default: 2019-12-31)
  --out    PATH  (default: configs/cmap/weather_chicago.csv)

2. configs/cmap/weather_chicago.csv (generated, then committed)

Pre-generated by fetch_weather.py. Keyed on date (YYYY-MM-DD string). Columns: date, max_temp_c, precipitation_mm.

3. foundata/cmap.py

load_households() — preserve the calendar date before dropping it:

hhs = hhs.with_columns(
    survey_date=pl.col("date").dt.strftime("%Y-%m-%d"),   # add
    year=pl.col("date").dt.year().cast(pl.Int32),
    month=pl.col("date").dt.month().cast(pl.Int8),
    day=pl.col("date").dt.weekday().replace_strict(config["day"]),
).drop("date")

New load_weather(configs_root) helper + join in load():

def load_weather(configs_root: Path) -> pl.DataFrame:
    return pl.read_csv(Path(configs_root) / "cmap" / "weather_chicago.csv")

# in load():
weather = load_weather(configs_root)
attributes = attributes.join(weather, left_on="survey_date", right_on="date", how="left")
attributes = attributes.drop("survey_date")

4. configs/core/template.yaml

Add two nullable attributes fields:

max_temp_c:
  dtype: float
  default: True
  description: "Daily maximum temperature (°C) for the travel day. Null if unavailable."

rain:
  dtype: bool
  default: True
  description: "Whether precipitation > 0 mm was recorded on the travel day. Null if unavailable."

Critical files

File Change
scripts/fetch_weather.py create — one-time download script
configs/cmap/weather_chicago.csv create — generated by script, commit to repo
foundata/cmap.py preserve survey_date, add load_weather(), join in load()
configs/core/template.yaml add max_temp_c and rain fields

Verification

# generate the weather CSV (run once)
uv run python scripts/fetch_weather.py

# lint
uv run ruff check foundata/ scripts/

# unit tests
uv run pytest tests/ -v

# dry run (requires CMAP data)
uv run python scripts/run.py --data-root ~/Data/foundata --output ~/Data
# check: all_attributes.csv should have non-null max_temp_c/rain for cmap rows