Add natural gas price regression pipeline

aeo_updates/README.md

@@ -42,22 +42,8 @@ Once you have obtained your api key, create a new environment variable to store
 
 ### Input Data Changes When Updating AEO Data
 #### Natural Gas Prices and Demand
-Natural gas prices and demand can be pulled using the EIA AEO data grabber. The spreadsheet "NG Prices Preprocessing for AEO Inputs.xlsx" is used to calculate the alphas using the preset betas. You need to paste in the NG prices and NG electricity sector demand into the relevant tabs. Historical data needs to be updated to the current dollar year. The deflator to convert the alphas back to 2004$ also need to be updated. The alphas are then put into a csv file to be added to the inputs/fuelprices folder of the ReEDS model repo.
-
-The prices and demand (both for the electricity sector and for all sectors) are also put into the relevant csv files in the inputs/fuelprices folder of the ReEDS model repo. Here are the NG input files that should be updated: 
-
-* ng_tot_demand_AEO_{year}_HOG.csv
-* ng_tot_demand_AEO_{year}_LOG.csv
-* ng_tot_demand_AEO_{year}_reference.csv
-* ng_demand_AEO_{year}_LOG.csv
-* ng_demand_AEO_{year}_HOG.csv
-* ng_demand_AEO_{year}_reference.csv
-* ng_AEO_{year}_LOG.csv
-* ng_AEO_{year}_HOG.csv
-* ng_AEO_{year}_reference.csv
-* alpha_AEO_{year}_LOG.csv
-* alpha_AEO_{year}_HOG.csv
-* alpha_AEO_{year}_reference.csv
+Natural gas prices and demand are updated using the files in natural_gas_price_regression.
+See the README.md in that folder for more information.
 
 #### Coal Prices
 Pulled using the EIA data grabber. Coal data are input into the coal_AEO_{year}_reference.csv.

aeo_updates/natural_gas_price_regression/.gitignore

@@ -0,0 +1,30 @@
+# Python cache
+__pycache__/
+*.pyc
+
+# Visualization outputs (PNGs, PDFs)
+*.png
+*.pdf
+
+# Validation and visualization results (all generated output)
+results validation/
+
+# Alpha regression outputs (all CSVs are generated, reproducible)
+outputs of alpha regression/*.csv
+outputs of alpha regression/raw_aeo_data/
+
+# Beta regression outputs (all generated, reproducible)
+outputs of beta regression/*.csv
+outputs of beta regression/raw_aeo_data/
+outputs of beta regression/bata raw data visualization/
+outputs of beta regression/alpha from beta regression/
+
+# Intermediate beta inputs (copied by sync_beta_to_alpha_inputs.py, not source data)
+inputs for alpha regression/cd_beta0.csv
+inputs for alpha regression/national_beta.csv
+
+# Notebook checkpoints
+.ipynb_checkpoints/
+
+# Stale nested duplicate directory
+AEO_Updates/

aeo_updates/natural_gas_price_regression/README.md

@@ -0,0 +1,88 @@
+# Natural Gas Price Regression
+
+This folder has the natural gas update workflow for ReEDS.
+
+Files to update in the ReEDS repository are described in "outputs of alpha regression/README.md".
+
+## Model Note
+
+`alpha` differs by step:
+
+- Beta step (`alpha1`) is `alpha1(region, year)` and is shared across scenarios.
+- Alpha step (`alpha2`) is `alpha2(region, year, scenario)`, so each output
+  scenario has its own alpha path.
+
+## Prerequisites
+
+### EIA API Key
+
+The pipeline fetches data from the EIA API and requires an API key. Set it as an environment variable before running:
+
+**Windows (persistent — open a new terminal after running):**
+```cmd
+setx EIA_API_KEY your_api_key_here
+```
+
+**Windows (current session only):**
+```cmd
+set EIA_API_KEY=your_api_key_here
+```
+
+**macOS/Linux:**
+```bash
+export EIA_API_KEY=your_api_key_here
+```
+
+> The key is read from the `EIA_API_KEY` environment variable. Do **not** hardcode it in `aeo_pipeline_config.json`.
+
+## Run order
+
+1. Edit `aeo_pipeline_config.json` with your settings.
+
+2. Run the full pipeline:
+
+```bash
+python run_ng_pipeline.py
+```
+
+Optional custom config:
+
+```bash
+python run_ng_pipeline.py --config my_config.json
+```
+
+<details>
+<summary>Advanced: run steps individually</summary>
+
+```bash
+# 1. Beta regression
+python aeo_beta_regression.py --config aeo_pipeline_config.json
+
+# 2. Sync beta outputs into alpha inputs
+python sync_beta_to_alpha_inputs.py --config aeo_pipeline_config.json
+
+# 3. Alpha regression
+python aeo_alpha_regression.py --config aeo_pipeline_config.json
+
+# 4. Visualization and validation
+python visualization.py --config aeo_pipeline_config.json
+```
+
+Skip individual visualization parts with flags:
+```bash
+python visualization.py --skip-raw-scatter --skip-validation
+```
+
+</details>
+
+Default output folder: `results validation` (all plots and validation CSVs).
+
+## Scenario Configuration
+
+Set explicit scenarios in `aeo_pipeline_config.json` under `scenarios`:
+
+- `scenarios.beta_regression.include`: scenarios used to estimate beta.
+- `scenarios.alpha_regression.fetch`: scenarios fetched for alpha preprocessing.
+- `scenarios.alpha_regression.outputs`: mapping from output suffix (`reference`, `HOG`, `LOG`, etc.) to scenario aliases.
+
+Use canonical IDs (for example `ref{aeo_year}`, `highogs`, `lowogs`) for a clear setup.