diff --git a/.gitignore b/.gitignore index 934b0bf..823acba 100644 --- a/.gitignore +++ b/.gitignore @@ -209,6 +209,7 @@ notebooks/404_* /data/emissions/primap-202503/Guetschow_et_al_2025-PRIMAP-hist_v2.6.1_final_no_rounding_13-Mar-2025.nc /data/emissions/cmip7-historical-2025.12.07/country-history.feather /data/emissions/cmip7-historical-2025.12.07/global-workflow-history.csv +/data/emissions/gcb-2025/GCB2025v15_MtCO2_flat.csv /data/gdp/wdi-2025/API_NY.GDP.MKTP.KD_DS2_en_csv_v2_213435.csv /data/gdp/wdi-2025/API_NY.GDP.MKTP.PP.KD_DS2_en_csv_v2_1004.csv /data/gini/unu-wider-2025/WIID-29APR2025.xlsx @@ -216,6 +217,7 @@ notebooks/404_* /data/population/un-owid-2025/population.csv /data/population/un-owid-2025/UN_PPP2024_Output_PopTot.xlsx /data/lulucf/melo-2026/timeseries_NGHGI_v3.1.csv +/data/lulucf/melo-2026-v4/timeseries_NGHGI_gap-filled_4.0.0.csv /data/bunkers/gcb-2024/National_Fossil_Carbon_Emissions_2024v1.0.xlsx /data/scenarios/ipcc_ar6_gidden/ar6_gidden.xlsx /data/scenarios/ipcc_ar6_gidden/ar6_gidden.zip diff --git a/CHANGELOG.md b/CHANGELOG.md index 3984990..6efab16 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,12 @@ Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Versions: [Sem - `fair-shares fetch-data`: downloads every input dataset from source, verifies pinned checksums, records provenance in `data/PROVENANCE.md`. Missing files also fetch automatically on first use. - Python API reference (`docs/api/python-api.md`) for pip-only users. - Root `CONTRIBUTING.md`. This changelog. +- LULUCF Data Hub v4.0.0 (gap-filled NGHGI, 187 countries, 2000-2024) as the opt-in LULUCF source `melo-2026-v4`. `melo-2026` (v3.1.1) stays the default. +- Global Carbon Budget 2025 national fossil CO2 (1850-2024) as the opt-in emissions source `gcb-2025`, `co2-ffi` only. Its world row excludes international bunkers and the 1991 Kuwaiti oil fires, and its runs take bunkers from the same file. The pipeline asks an emissions source only for the categories it declares. `primap-202503` stays the default. +- RCB sources `forster_2026` (IGCC 2025, from 2026) and `ar6_wg1_2021` (AR6 WGI Table SPM.2, from 2020), each with 1.5, 1.7 and 2°C budgets. Their deductions use AR6 scenarios in a peak-warming band (`scenario_selection: peak-warming-band` in `rcbs.yaml`). The band keeps only scenarios that reach net-zero CO2 by 2100, because a remaining carbon budget runs to net-zero CO2 (14, 111 and 37 of 14, 131 and 69 scenarios at 1.5, 1.7 and 2°C). A budget label without a scenario set and an empty band raise an error. A rebase with missing emission years raises an error, and the pipeline skips that source with a warning. The band rule leaves the budgets of the existing sources unchanged; the bunker rebase under "Fixed" changes them. +- Opt-in coverage rule for analysis countries: a `coverage` block on an emissions source (`emissions_recorded_before`, `population_from`). `gcb-2025` sets it to 1990 and 1850, so a country needs an emissions record before 1990 and population from 1850. Eight countries with their first GCB record in 1990 or later (AND, FSM, LSO, MHL, NAM, PLW, TLS, TUV) and Macao (population from 1950) join rest-of-world, which leaves 168 analysis countries in a `co2-ffi` run. `country_data_coverage_summary.csv` names the failed test in `coverage_rule_failed`, and notebook 101 saves `emiss_co2-ffi_first_recorded_year.csv`. Sources without the block, including `primap-202503`, keep their country set and outputs. +- `rebase_fill_max_years` in `rcbs.yaml` (default 1). When a world series of the rebase (fossil CO2, bunkers, LULUCF) ends before the year before the budget baseline, each later year takes the last observed value, up to this number of years, with a warning that names the source, the series, the years and the value. The fill is a placeholder until observed data are published. With `gcb-2025` (observed to 2024) `forster_2026` rebases with 2025 filled. With PRIMAP (observed to 2023) it needs two years and stays skipped. 0 turns the fill off. +- Emission category `co2` with a `co2-ffi`-only emissions source and an active LULUCF source, for example `gcb-2025` with `melo-2026-v4` and target `rcbs`. Notebook 107 derives `co2` as `co2-ffi` plus national-inventory LULUCF and skips the non-CO2 categories that such a source cannot supply. Without a LULUCF source the configuration still raises an error. - World Bank WDI Gini index (`SI.POV.GINI`) as a Gini source, with `analysis/gini_source_comparison.py` reporting the coverage and value differences against WIID. ### Changed @@ -19,6 +25,7 @@ Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Versions: [Sem - **Breaking:** the default Gini source is now World Bank WDI (`wdi-2025`), which is CC-BY-4.0. WIID stays available as `active_gini_source=unu-wider-2025` but is opt-in, and outputs built on it cannot be redistributed under CC BY 4.0. Output directory names change, so existing WIID runs are not overwritten. The two sources give materially different capability-based allocations — WDI/PIP is consumption-based for most low- and middle-income countries. - **Breaking:** analysis-country membership no longer depends on Gini availability. A country with complete emissions, GDP and population is now in the analysis even without a Gini value, and receives the analysis-country mean (`general.gini_missing_policy: fallback-mean`, or `strict` to refuse). The country set grows by 9 on the standard sources; `country_data_coverage_summary.csv` gains a `gini_imputed` column. - Gini source config replaces the unused `world_key` and `gini_year` keys with `selection` and `year_window`, both of which the notebooks read. +- Allocation years start at 1850 (was 1900). With `capability_reference_year` set, the adjusted budget approaches need GDP at the reference year only, so `allocation_year` can precede the GDP series. For `co2` and `all-ghg`, the parameter grid checks `pre_allocation_responsibility_year` against the first year of the responsibility emissions frame it receives. ### Deprecated @@ -30,8 +37,18 @@ Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Versions: [Sem ### Fixed +- `compose_config` did not declare the packaged `data_sources_unified.yaml` as an input, so an existing output folder reused its stale `config.yaml` after a source changed (for example a new `coverage` block) until the run used `--forcerun compose_config`. The rule now re-runs when that file changes. - Global Carbon Budget citation pointed at a DOI that does not resolve; now the correct paper DOI (10.5194/essd-17-965-2025) plus data-product DOI (10.18160/GCP-2024). - Licence statements corrected: WIID is CC BY-NC-SA 3.0 IGO, UN/OWID population is mixed-terms, CMIP7 is CC-BY-SA-4.0 (author-confirmed). - The Python API's preprocessing path wrote `country_gini_stationary.csv` without a Rest-of-World row, unlike the notebook path. Both now use the same code. - Notebook `100_data_preprocess_rcbs` passed the removed `project_root=` argument to `load_and_process_rcbs`, so the RCB pipeline could not run. +- Adjusted budgets of RCB sources with a baseline after 2020 were too low. The rebase to 2020 added world fossil emissions, which exclude international bunkers, and the bunker deduction covered 2020 to net zero, so the bunkers of 2020 to the year before the baseline were deducted and never added. The rebase now adds them (new output column `rebase_bunkers_mt`). Every adjusted budget of `lamboll_2023` (baseline 2023) rises by 2,822 Mt CO2 and every adjusted budget of `forster_2024` (baseline 2024) by 3,959 Mt CO2, for `co2-ffi` and `co2`, with PRIMAP emissions and `gcb-2024` bunkers. `ar6_2020` (baseline 2020) is unchanged. `process_rcb_to_2020_baseline` takes the bunker series as `world_bunker_emissions`. +- Adjusted `co2` budgets of RCB sources with a baseline after 2020 were too low. The rebase to 2020 adds observed LULUCF in the national-inventory (NGHGI) convention for 2020 to the year before the baseline, and the BM-to-NGHGI convention gap also started in 2020, so the conversion for those years was applied twice. The convention gap now runs from the baseline year of the source to net zero (Weber et al. 2026, Eqs. 2-3). Notebook 104 writes one median gap per baseline year (`convention_gap_median_from` in `rcb_scenario_adjustments.yaml`, which replaces `convention_gap_median`), so existing outputs need a re-run of notebook 104. With PRIMAP emissions and `melo-2026` LULUCF, the `1.5p50` budget of `lamboll_2023` rises from 222,434 to 241,477 Mt CO2 and that of `forster_2024` from 206,034 to 230,905 Mt CO2. `ar6_2020` (baseline 2020) and every `co2-ffi` budget are unchanged. The argument `actual_bm_lulucf_emissions` of `load_and_process_rcbs` and `process_rcb_to_2020_baseline` is now `world_nghgi_lulucf_emissions`: the series is the NGHGI world row, and the pipeline holds no observed bookkeeping-model LULUCF series. +- `build_nghgi_world_co2_timeseries` subtracted bunkers from a world fossil series that already excludes them (871 Mt CO2 in 2020; 20,770 Mt over 2000-2019). World `co2` is now fossil plus LULUCF, as in notebook `100_data_preprocess_rcbs`, and the function no longer takes `bunker_ts`. The error reached `run_rcb_preprocessing` and notebook `106_generate_pathways_from_rcbs`; budget files written by notebook 100 did not contain it. +- `compute_bunker_deduction` treated 2023 as the last observed bunker year for every source. It now reads the last year from the bunker data, so `gcb-2025` runs use the observed 2024 value and extrapolate the 2024 rate. Runs with `gcb-2024` bunkers give the same deduction as before. +- `calculate_budget_from_rcb` summed only the years present when the allocation year preceded the first year of the world emissions series (for example `co2`, which starts in 2000, with allocation year 1990). It now raises an error that names the missing years and the first available year. +- `calculate_budget_from_rcb` raised no error when the allocation year was after 2020 and the world emissions ended earlier, and summed only the years present. It now raises an error that names the missing years and the last available year. A year with a missing value counts as missing in this check, in the pre-2020 check and in the rebase to 2020, which earlier summed NaN-padded years as zero. +- For `co2`, the pipeline could run the scenario notebook (104) before the LULUCF notebook (107) and then write a zero convention gap. The scenario rule now depends on notebook 107. `process_rcb_to_2020_baseline` raises an error for a `co2` budget with a baseline after 2020 when it receives no inventory LULUCF series; it used a zero LULUCF rebase term before. A budget with baseline 2020 needs no series. +- `import fair_shares.library.config` and `import fair_shares.library.config.models` failed with a circular import when they were the first import in a process. +- A budget label without a numeric temperature under the peak-warming band rule raised a bare `ValueError`; it now raises a `ConfigurationError` that names the label. `rebase_fill_max_years` must be a non-negative integer. - The Gini-adjusted approaches ignored Gini entirely when `capability_reference_year` named a year before `allocation_year`: the capability snapshot was read from the unfiltered inputs without the adjustment, on both the budget and pathway side. Results on that parameterisation change — on the standard sources, China's share of a `per-capita-adjusted-gini-budget` allocation moves from 10.8% to 5.3% and India's from 23.2% to 28.4%. diff --git a/Snakefile b/Snakefile index 23fd7d1..f7a8eff 100644 --- a/Snakefile +++ b/Snakefile @@ -23,7 +23,9 @@ # Import shared logic from Python modules (single source of truth) from fair_shares.library.utils.data.config import ( build_source_id, + get_bunkers_source, get_emission_preprocessing_categories, + get_emissions_data_parameters, get_final_categories, get_co2_component, is_composite_category, @@ -95,8 +97,13 @@ NOTEBOOK_DIR = "notebooks" # Two category lists drive the pipeline: # EMISSION_CATEGORIES — what PRIMAP extraction (notebook 101) produces # FINAL_CATEGORIES — what the allocation loop iterates over +# The emissions source is asked only for categories it declares. _target = active_target_source or "" -EMISSION_CATEGORIES = get_emission_preprocessing_categories(_target, emission_category) +EMISSION_CATEGORIES = get_emission_preprocessing_categories( + _target, + emission_category, + get_emissions_data_parameters(active_emissions_source).get("available_categories"), +) FINAL_CATEGORIES = get_final_categories(_target, emission_category) is_multi_category = needs_decomposition(_target, emission_category) @@ -143,6 +150,8 @@ _needs_lulucf = emission_category in ("co2", "co2-lulucf", "all-ghg") # Bunker data is needed for all non-pathway targets (RCBs must subtract # international bunker emissions before country allocation). _needs_bunkers = _allocation_mode != "pathway" +# The emissions source names its bunker data; the default is gcb-2024. +_bunkers_source = get_bunkers_source(active_emissions_source) if _needs_lulucf and active_lulucf_source is None: @@ -153,6 +162,13 @@ if _needs_lulucf and active_lulucf_source is None: f"Example: --config ... active_lulucf_source=melo-2026" ) +# Historical series that the scenario rule declares as input. Notebook 107 +# derives co2 when the emissions source declares only co2-ffi. +if emission_category == "co2" and "co2" not in EMISSION_CATEGORIES: + _scenario_emissions_input = f"{OUTPUT_DIR}/intermediate/emissions/emiss_co2_nghgi_timeseries.csv" +else: + _scenario_emissions_input = f"{OUTPUT_DIR}/intermediate/emissions/emiss_{emission_category}_timeseries.csv" + # Resolve scenario source: per-target override → global default _scenario_source_key = ( _target_yaml.get("scenario_source") @@ -274,6 +290,9 @@ rule compose_config: All validation logic is in config/models.py (Pydantic). The Snakefile only does minimal checks — Pydantic does comprehensive validation. """ + input: + # build_data_config composes the output config from this packaged file + sources_yaml=str(packaged_config("data_sources/data_sources_unified.yaml")), output: config=f"{OUTPUT_DIR}/config.yaml", params: @@ -405,6 +424,7 @@ if _needs_lulucf: notebook=f"{OUTPUT_DIR}/notebooks/107_derive_nghgi_categories_{active_lulucf_source}.ipynb", nghgi_world=f"{OUTPUT_DIR}/intermediate/emissions/world_co2-lulucf_timeseries.csv", nghgi_metadata=f"{OUTPUT_DIR}/intermediate/emissions/lulucf_metadata.yaml", + nghgi_co2=f"{OUTPUT_DIR}/intermediate/emissions/emiss_co2_nghgi_timeseries.csv", shell: notebook_cmd("{input.notebook}", "{output.notebook}") @@ -418,10 +438,10 @@ if _needs_bunkers: allocation. Independent of LULUCF — uses GCB fossil emissions data. """ input: - notebook=f"{NOTEBOOK_DIR}/108_data_preprocess_bunkers_gcb-2024.ipynb", + notebook=f"{NOTEBOOK_DIR}/108_data_preprocess_bunkers_{_bunkers_source}.ipynb", config=f"{OUTPUT_DIR}/config.yaml", output: - notebook=f"{OUTPUT_DIR}/notebooks/108_data_preprocess_bunkers_gcb-2024.ipynb", + notebook=f"{OUTPUT_DIR}/notebooks/108_data_preprocess_bunkers_{_bunkers_source}.ipynb", bunker_csv=f"{OUTPUT_DIR}/intermediate/emissions/bunker_timeseries.csv", shell: notebook_cmd("{input.notebook}", "{output.notebook}") @@ -446,7 +466,7 @@ if uses_scenarios: input: notebook=f"{NOTEBOOK_DIR}/{_scenario_nb_stem}.ipynb", config=f"{OUTPUT_DIR}/config.yaml", - emissions_data=f"{OUTPUT_DIR}/intermediate/emissions/emiss_{emission_category}_timeseries.csv", + emissions_data=_scenario_emissions_input, lulucf_notebook=(f"{OUTPUT_DIR}/notebooks/107_derive_nghgi_categories_{active_lulucf_source}.ipynb" if _needs_lulucf else []), output: notebook=f"{OUTPUT_DIR}/notebooks/{_scenario_nb_stem}.ipynb", @@ -459,7 +479,7 @@ if uses_scenarios: input: notebook=scenario_notebook, config=f"{OUTPUT_DIR}/config.yaml", - emissions_data=f"{OUTPUT_DIR}/intermediate/emissions/emiss_{emission_category}_timeseries.csv", + emissions_data=_scenario_emissions_input, lulucf_notebook=(f"{OUTPUT_DIR}/notebooks/107_derive_nghgi_categories_{active_lulucf_source}.ipynb" if _needs_lulucf else []), bunker_csv=(f"{OUTPUT_DIR}/intermediate/emissions/bunker_timeseries.csv" if _needs_bunkers else []), scenario_adjustments=f"{OUTPUT_DIR}/intermediate/scenarios/rcb_scenario_adjustments.yaml", @@ -476,7 +496,9 @@ if uses_scenarios: input: notebook=scenario_notebook, config=f"{OUTPUT_DIR}/config.yaml", - emissions_data=f"{OUTPUT_DIR}/intermediate/emissions/emiss_{emission_category}_timeseries.csv", + emissions_data=_scenario_emissions_input, + # Notebook 107 runs first: it writes the NGHGI series that 104 reads. + lulucf_notebook=(f"{OUTPUT_DIR}/notebooks/107_derive_nghgi_categories_{active_lulucf_source}.ipynb" if _needs_lulucf else []), output: notebook=scenario_nb_out, scenarios=f"{OUTPUT_DIR}/intermediate/scenarios/scenarios_{emission_category}_timeseries.csv", diff --git a/data/emissions/gcb-2025/CITATION.md b/data/emissions/gcb-2025/CITATION.md new file mode 100644 index 0000000..8ca8745 --- /dev/null +++ b/data/emissions/gcb-2025/CITATION.md @@ -0,0 +1,21 @@ +# Citation + +## Global Carbon Budget 2025: National Fossil CO2 Emissions + +**Data product (the file in this directory):** + +Andrew, R. M., & Peters, G. P. (2025). *The Global Carbon Project's fossil CO2 emissions dataset* (2025v15) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.17417124 + +**Paper (methods and results):** + +Friedlingstein, P., et al. (2026). Global Carbon Budget 2025. *Earth System Science Data*, 18, 3211-3288. https://doi.org/10.5194/essd-18-3211-2026 + +Cite both when reporting values derived from this file. + +## Licence + +CC BY 4.0 for both the data product and the paper. + +## Files + +- `GCB2025v15_MtCO2_flat.csv`: territorial fossil CO2 emissions by country, 1750-2024, long format. Units: MtCO2/yr. Also holds international shipping (`XIS`), international aviation (`XIA`), the 1991 Kuwaiti oil fires and the global total (`WLD`). diff --git a/data/gdp/wdi-2025/CITATION.md b/data/gdp/wdi-2025/CITATION.md index 2f68e56..a445372 100644 --- a/data/gdp/wdi-2025/CITATION.md +++ b/data/gdp/wdi-2025/CITATION.md @@ -2,10 +2,10 @@ ## World Bank GDP Data -World Bank. (2025). GDP (constant 2015 US$) and GDP, PPP (constant 2017 international $). World Bank Open Data. https://data.worldbank.org/ +World Bank. (2025). GDP (constant 2015 US$) and GDP, PPP (constant 2021 international $). World Bank Open Data. https://data.worldbank.org/ Files: - `API_NY.GDP.MKTP.KD_DS2_en_csv_v2_213435.csv` — GDP (constant 2015 US$), WDI indicator `NY.GDP.MKTP.KD` -- `API_NY.GDP.MKTP.PP.KD_DS2_en_csv_v2_1004.csv` — GDP, PPP (constant 2017 international $), WDI indicator `NY.GDP.MKTP.PP.KD` +- `API_NY.GDP.MKTP.PP.KD_DS2_en_csv_v2_1004.csv` — GDP, PPP (constant 2021 international $), WDI indicator `NY.GDP.MKTP.PP.KD` -**Why constant-dollar (`.KD`) and not current-dollar (`.CD`):** Constant-dollar series fix prices to a reference year (2015 for MER, 2017 for PPP), so values across years are directly comparable in real terms. Current-dollar series use each year's own prices, which makes long-horizon comparisons silently distorted by price changes. For historical cross-year capability calculations (e.g., allocating a remaining carbon budget from 1990), using current-dollar series understates early-period real GDP by the cumulative inflation between the observation year and the base year — for 1990 values in a 2017-base comparison, this is roughly a factor of two for developed economies. +**Why constant-dollar (`.KD`) and not current-dollar (`.CD`):** Constant-dollar series fix prices to a reference year (2015 for MER, 2021 for PPP), so values across years are directly comparable in real terms. Current-dollar series use each year's own prices, which makes long-horizon comparisons silently distorted by price changes. For historical cross-year capability calculations (e.g., allocating a remaining carbon budget from 1990), using current-dollar series understates early-period real GDP by the cumulative inflation between the observation year and the base year — for 1990 values in a 2021-base comparison, this is roughly a factor of two for developed economies. diff --git a/data/lulucf/melo-2026-v4/CITATION.md b/data/lulucf/melo-2026-v4/CITATION.md new file mode 100644 index 0000000..2e54d63 --- /dev/null +++ b/data/lulucf/melo-2026-v4/CITATION.md @@ -0,0 +1,8 @@ +# Citation + +## NGHGI-Reported LULUCF Emissions (Melo et al. 2026, v4.0.0) + +Melo, J., et al. (2026). The LULUCF Data Hub: translating global land use emissions estimates into the national GHG inventory framework (Version 4.0.0, 2026 NGHGI release). *Zenodo*. https://doi.org/10.5281/zenodo.22828743 + +Files: +- `timeseries_NGHGI_gap-filled_4.0.0.csv` — CO2 fluxes from the National Greenhouse Gas Inventories (NGHGIs) submitted to the UNFCCC, allocated to the classes Forest, Deforestation, Other non-forest land uses, Organic soils and Harvested wood products (HWP), with gaps filled. Units: MtCO2/yr. Sign convention: negative = net sink. Covers 187 countries, 2000-2024. diff --git a/data/rcbs/rcbs.yaml b/data/rcbs/rcbs.yaml index 4f45b07..25ecb2a 100644 --- a/data/rcbs/rcbs.yaml +++ b/data/rcbs/rcbs.yaml @@ -1,16 +1,36 @@ # Remaining Carbon Budgets (RCBs) data # Each source includes: -# - baseline_year: The year from which the RCB is calculated +# - baseline_year: The year from which the RCB is calculated. A budget from +# the start of year X has baseline_year X. # - unit: The unit of the RCB values (typically Gt CO2) # - scenarios: Temperature and probability combinations with their RCB values -# -# Probability labels (e.g. 2p66) map to AR6 climate categories: -# 1.5p50 -> C1, 2p83 -> C2, 2p66 -> C3 +# - scenario_selection (optional): The rule that selects the AR6 scenarios +# behind the LULUCF, convention-gap and bunker deductions. +# ar6-category (default): the AR6 climate category mapped to the label, +# 1.5p50 -> C1, 2p83 -> C2, 2p66 -> C3. Any other label is an error. +# peak-warming-band: every scenario whose median peak warming +# (MAGICCv7.5.3) lies in [T - half_width, T + half_width), where T is +# the temperature in the label. One band serves every likelihood of T. +# A remaining carbon budget runs to net-zero CO2, so the band keeps only +# scenarios whose CO2 emissions reach zero by 2100. +# The two rules differ for scenarios that never reach net-zero CO2: +# peak-warming-band leaves them out, while ar6-category applies no such +# filter and integrates their deductions to 2100. # # Sources report at slightly different probability levels: -# - AR6/Forster: 67% and 83% (mapped to 2p66, 2p83) +# - AR6/Forster 2024: 67% and 83% (mapped to 2p66, 2p83) # - Lamboll: 66% and 90% (mapped to 2p66; no 83% reported) +# Half-width of the peak-warming band, in degrees C. +peak_warming_band_half_width: 0.05 + +# Largest number of years that the rebase to 2020 may fill. When a world series +# (fossil CO2, bunkers, LULUCF) ends before the year before the baseline, each +# later year takes the last observed value, with a warning. This is a +# placeholder until observed data are published. 0 turns the fill off. A source +# that needs more years is skipped. +rebase_fill_max_years: 1 + rcb_data: lamboll_2023: # Lamboll et al (2023), "https://doi.org/10.1038/s41558-023-01848-5" baseline_year: 2023 @@ -35,3 +55,33 @@ rcb_data: "1.5p50": 500 "2p66": 1150 "2p83": 900 + + # Forster et al (2026), IGCC 2025, ESSD 18, 3889-3933, Table 8, + # "https://doi.org/10.5194/essd-18-3889-2026". Budgets from the start of 2026. + forster_2026: + baseline_year: 2026 + unit: "Gt * CO2" + scenario_selection: peak-warming-band + scenarios: + "1.5p50": 130 + "1.5p67": 80 + "1.7p50": 500 + "1.7p67": 390 + "2p50": 1050 + "2p67": 860 + "2p83": 690 + + # IPCC AR6 WGI (2021), Table SPM.2, "https://doi.org/10.1017/9781009157896.001". + # Budgets from the start of 2020. + ar6_wg1_2021: + baseline_year: 2020 + unit: "Gt * CO2" + scenario_selection: peak-warming-band + scenarios: + "1.5p50": 500 + "1.5p67": 400 + "1.7p50": 850 + "1.7p67": 700 + "2p50": 1350 + "2p67": 1150 + "2p83": 900 diff --git a/docs/api/utils/data.md b/docs/api/utils/data.md index 7df675d..6fb9169 100644 --- a/docs/api/utils/data.md +++ b/docs/api/utils/data.md @@ -106,6 +106,22 @@ Functions for parsing RCB scenarios and converting to allocation-ready budgets. heading_level: 4 show_source: false +### select_rcb_scenario_set + +::: fair_shares.library.utils.data.rcb.select_rcb_scenario_set + options: + show_root_heading: true + heading_level: 4 + show_source: false + +### convention_gap_from_baseline + +::: fair_shares.library.utils.data.rcb.convention_gap_from_baseline + options: + show_root_heading: true + heading_level: 4 + show_source: false + ### calculate_budget_from_rcb ::: fair_shares.library.utils.data.rcb.calculate_budget_from_rcb @@ -122,6 +138,14 @@ Functions for parsing RCB scenarios and converting to allocation-ready budgets. heading_level: 4 show_source: false +### fill_rebase_years + +::: fair_shares.library.utils.data.rcb.fill_rebase_years + options: + show_root_heading: true + heading_level: 4 + show_source: false + ## See Also - **[Core Utilities](https://setupelz.github.io/fair-shares/api/utils/core/)**: General data manipulation functions diff --git a/docs/dev-guide/adding-data-sources.md b/docs/dev-guide/adding-data-sources.md index a4ef8c7..824aad2 100644 --- a/docs/dev-guide/adding-data-sources.md +++ b/docs/dev-guide/adding-data-sources.md @@ -18,7 +18,7 @@ Data sources are configured in `src/fair_shares/conf/data_sources/data_sources_u | Type | Purpose | Current Sources | | ------------ | ------------------------------------ | ---------------------- | -| `emissions` | Historical non-LULUCF emissions | PRIMAP-hist | +| `emissions` | Historical non-LULUCF emissions | PRIMAP-hist (default), Global Carbon Budget 2025 | | `gdp` | Economic capability | World Bank WDI | | `population` | Per capita calculations | UN/OWID | | `gini` | Within-country inequality | World Bank WDI (default), UNU-WIDER WIID | @@ -67,7 +67,10 @@ emissions: - co2-ffi - all-ghg world_key: "WORLD" # How the source identifies global totals - scenario: "HISTCR" # Historical scenario identifier + scenario: "HISTCR" # Optional: historical scenario identifier + bunkers_source: "gcb-2024" # Optional: registry name of the bunker data + # Optional: coverage rule for analysis countries + coverage: { emissions_recorded_before: 1990, population_from: 1850 } ``` ### Common Configuration Parameters @@ -77,6 +80,15 @@ emissions: | `path` | Relative path to data file | | `available_categories` | Which emission categories this source provides | | `world_key` | String used to identify global totals in the data | +| `scenario` | Optional. Historical scenario, for sources that have one | +| `bunkers_source` | Optional. Bunker data paired with an emissions source (default `gcb-2024`) | +| `coverage` | Optional. Coverage rule for analysis countries (default: none) | + +The pipeline asks an emissions source only for the categories in `available_categories`. A source that declares `co2-ffi` alone, such as `gcb-2025`, runs one pass of its notebook 101 and supports `co2-ffi` runs. + +By default, an analysis country has complete emissions, GDP and population from 1990 through the last year of each dataset. A `coverage` block adds two tests, and a country that fails either one joins rest-of-world. `emissions_recorded_before` requires a value in the raw emissions file in a year before the threshold. `population_from` requires population for every year from the threshold. A source that zero-fills blank years needs the first test, because a blank and a reported zero look the same after the fill. Its notebook 101 writes `emiss__first_recorded_year.csv` (columns `iso3c`, `first_recorded_year`) for each category it provides. The rule is `coverage_exclusions` in `src/fair_shares/library/preprocessing/coverage.py`. `country_data_coverage_summary.csv` names the failed test for each country in the column `coverage_rule_failed`. A source without the block keeps the default check alone, and its coverage summary has no such column. `gcb-2025` sets both tests. + +Add every new source to `src/fair_shares/conf/data_registry.yaml` with its URL, checksums, licence and citation, and record each DOI in `tests/fixtures/verified_dois.yaml`. The config validates the path of every configured source, so a registered source downloads on first use. --- @@ -166,7 +178,7 @@ df.columns == ["gini"] ## Step 5: Integrate with Pipeline -The Snakemake workflow automatically picks up sources from the configuration. Ensure your preprocessing notebook: +The Snakemake workflow automatically picks up sources from the configuration. It builds the notebook name from the source key: `101_data_preprocess_emiss_{source}.py` for emissions and `108_data_preprocess_bunkers_{bunkers_source}.py` for bunkers. Ensure your preprocessing notebook: 1. Reads from the path specified in the config 2. Outputs to the standard processed data location @@ -228,7 +240,7 @@ Notebook 107 reads the raw LULUCF source and outputs: | Emission category | Uses LULUCF? | Why | |-------------------|-------------|-----| | `co2-ffi` | No | Fossil fuels only | -| `co2` | **Yes** | Total CO2 = fossil − bunkers + NGHGI LULUCF | +| `co2` | **Yes** | Country `co2` = `co2-ffi` + NGHGI `co2-lulucf` on shared years. Bunkers enter only at world and budget level | | `all-ghg` | **Yes** | Decomposes into `co2` (NGHGI) + `non-co2` | | `all-ghg-ex-co2-lulucf` | No | CO2 component is `co2-ffi` | | `co2-lulucf` | Indirect | IS the LULUCF data | @@ -317,6 +329,7 @@ New data sources should: | Notebook | Data Type | Good Example Of | | ------------------------------------------------ | ---------- | -------------------------------------------------- | | `101_data_preprocess_emiss_primap-202503.py` | Emissions | NetCDF processing, category mapping | +| `101_data_preprocess_emiss_gcb-2025.py` | Emissions | Long CSV, single category, logic in one library function | | `102_data_preprocess_gdp_wdi-2025.py` | GDP | CSV processing, country code mapping | | `103_data_preprocess_population_un-owid-2025.py` | Population | Combining historical and projected data | | `105_data_preprocess_gini_wdi-2025.py` | Gini | Latest-available-in-window selection, aggregate filtering | diff --git a/docs/science/allocations.md b/docs/science/allocations.md index 5126cc0..29566c6 100644 --- a/docs/science/allocations.md +++ b/docs/science/allocations.md @@ -135,7 +135,7 @@ Reference year for calculating cumulative per capita allocations. **GDP window beyond the last observed year:** When the allocation cumulative window extends past the last year of the input GDP time series (the default `wdi-2025` ends at 2023), the per-capita budget and pathway primitives forward-fill GDP per capita from the last observed year to cover every subsequent year of the window. The cross-country capability ratios of the last observed year are then held constant for the rest of the window. The cumulative-per-capita-convergence primitives behave differently: their per-country capability scalar is computed only over the intersection of GDP and population years, with no forward-fill at all. In both cases, users who want different post-observation capability dynamics (SSP2 GDP projections, custom growth assumptions, or a future-extended WDI release) should extend the input `gdp_ts` time series before calling the allocation function. The forward-fill is the minimum-disruption default when no projected GDP data is supplied; the library does not prescribe a projection. -**Capability snapshot mode (`capability_reference_year: int | None = None`):** By default, capability is computed year-by-year — `pf(GDP_per_cap(g, t))` is evaluated for each country and year in the allocation window, then combined with that year's population. Setting `capability_reference_year` to an integer freezes capability at a single reference year instead: `pf(GDP_per_cap(g, t_ref))` is computed once per country from that year's data and broadcast across the rest of the window. Available on the adjusted approaches — `per_capita_adjusted_budget`, `per_capita_adjusted_gini_budget`, `per_capita_adjusted`, and `per_capita_adjusted_gini` — with identical semantics on the budget and pathway side; not available on `equal_per_capita*` (no capability term to freeze) or the convergence family. The reference year can be before, at, or after `allocation_year`. Year-by-year and snapshot modes are algebraically equivalent only when capability is constant over the window; in practice they diverge substantially for long windows spanning periods of economic change. See [capability_reference_year](parameter-effects.md#capability_reference_year) for the source-case mechanics (in-window, beyond-last-observed-year, pre-window) and the Gini interaction. +**Capability snapshot mode (`capability_reference_year: int | None = None`):** By default, capability is computed year-by-year — `pf(GDP_per_cap(g, t))` is evaluated for each country and year in the allocation window, then combined with that year's population. Setting `capability_reference_year` to an integer freezes capability at a single reference year instead: `pf(GDP_per_cap(g, t_ref))` is computed once per country from that year's data and broadcast across the rest of the window. Available on the adjusted approaches — `per_capita_adjusted_budget`, `per_capita_adjusted_gini_budget`, `per_capita_adjusted`, and `per_capita_adjusted_gini` — with the same snapshot semantics on the budget and pathway side; not available on `equal_per_capita*` (no capability term to freeze) or the convergence family. The reference year can be before, at, or after `allocation_year`. On the budget approaches a snapshot needs GDP at the reference year only, so `allocation_year` can be earlier than the first GDP year (allocation years start at 1850). The pathway approaches still require GDP at `first_allocation_year`, even when a reference year is set. Year-by-year and snapshot modes are algebraically equivalent only when capability is constant over the window; in practice they diverge substantially for long windows spanning periods of economic change. See [capability_reference_year](parameter-effects.md#capability_reference_year) for the source-case mechanics (in-window, beyond-last-observed-year, pre-window) and the Gini interaction. The canonical example of `ref_year < allocation_year` is the PRR2023 lag-1 convention: PRR2023 defines capability as GDP per capita in the year immediately preceding the allocation decision, so `allocation_year=2015` paired with `capability_reference_year=2014` is the reproduction-faithful configuration. The nb601 notebook (PRR2023 reproduction) uses exactly this. More broadly: use `capability_reference_year=1990` with `allocation_year=1990` for ESABCC 2023 `CPCadjCAP` results. @@ -158,7 +158,7 @@ For complete parameter configurations, worked examples, and use-case-to-approach **Why this exists:** National GDP per capita is a crude measure of capability -- it treats every dollar equally, whether it feeds a family or buys a yacht. Within-country inequality means that two countries with identical GDP per capita can have radically different capacity to mitigate if one concentrates wealth while the other distributes it. The Gini adjustment addresses this by modeling how income is distributed within each country and counting only income above a development threshold toward capability. -Implements an interpretation of the Greenhouse Development Rights (GDR) framework's capability metric for entitlements (note: GDR was originally designed for burden-sharing, not entitlement allocation): only income **above** a development threshold counts as "ability to pay." The income floor works like a tax-free personal allowance — each person's first $7,500/year (2010 PPP, the GDR framework's standard development threshold) of income is exempt from capability calculations. +Implements an interpretation of the Greenhouse Development Rights (GDR) framework's capability metric for entitlements (note: GDR was originally designed for burden-sharing, not entitlement allocation): only income **above** a development threshold counts as "ability to pay." The income floor works like a tax-free personal allowance — each person's first slice of income up to the threshold is exempt from capability calculations. [Baer 2013](https://doi.org/10.1002/wcc.201) sets the GDR threshold at $7,500 per capita per year on a 2005 price base and $8,500 on a 2010 base. The floor must be expressed in the price base of the GDP series passed in. The default GDP source is in constant 2021 international dollars, so a threshold from another price year needs converting first. Income distribution is modelled as log-normal, parameterised by each country's Gini coefficient. When combined with the income floor, higher inequality means more national income sits above the development threshold — increasing measured capability. diff --git a/docs/science/glossary.md b/docs/science/glossary.md index 3226dfe..adb69ad 100644 --- a/docs/science/glossary.md +++ b/docs/science/glossary.md @@ -114,7 +114,7 @@ Only the **ratio** between the two weights matters -- they are normalized by the ### Inequality Parameters **`income_floor`** (type: `float`, default: `0.0`) -: Income below this threshold (USD PPP per capita) is excluded from capability calculations, protecting subsistence needs. The library default is `0.0` (all income counts); the GDR framework's threshold is `7500` ($7,500/year 2010 PPP) and is passed explicitly. Higher floors reduce measured capability for all countries, with the largest effect on middle-income countries where population clusters around the threshold. See [Parameter Effects](parameter-effects.md#income_floor). +: Income below this threshold (USD PPP per capita) is excluded from capability calculations, protecting subsistence needs. The library default is `0.0` (all income counts); the GDR framework's threshold is $7,500 on a 2005 price base ($8,500 on 2010) and must be converted to the GDP series' price base before it is passed. Higher floors reduce measured capability for all countries, with the largest effect on middle-income countries where population clusters around the threshold. See [Parameter Effects](parameter-effects.md#income_floor). **`max_gini_adjustment`** (type: `float`, default: `0.8`) : Maximum proportional reduction from the Gini-based capability correction. Caps the influence of extreme inequality (Gini > 0.6) on measured capability, preventing outsized adjustments from dominating the allocation. At 0.8, the Gini adjustment can reduce a country's measured GDP by at most 80%. Available on `*-gini-*` approaches only. @@ -303,13 +303,13 @@ Brief definitions. For detailed explanations and operationalization, see [Alloca : Megatonnes of CO2 per year. Common unit for annual emissions. **Melo et al. (2026)** -: Country-reported NGHGI LULUCF CO₂ timeseries (v3.1). Covers 187 countries, 2000-2023. Replaces Grassi et al. (2023) with higher coverage and an additional year. See: [NGHGI-Consistent RCB Corrections](other-operations.md#weber-rcb-corrections) +: Country-reported NGHGI LULUCF CO₂ timeseries (v3.1). Covers 185 countries, 2000-2023. Replaces Grassi et al. (2023) with higher coverage and an additional year. See: [NGHGI-Consistent RCB Corrections](other-operations.md#weber-rcb-corrections) **NGHGI** : National Greenhouse Gas Inventory. Country-level emissions reporting under UNFCCC. Includes passive carbon fluxes (CO₂ fertilization, climate feedbacks) in LULUCF estimates, unlike bookkeeping models. See: [NGHGI-Consistent RCB Corrections](other-operations.md#weber-rcb-corrections) **NGHGI-BM convention gap** -: The systematic difference between NGHGI and bookkeeping (BM) LULUCF CO₂ estimates. NGHGI includes indirect effects (CO₂ fertilization of managed forests) that BM excludes, making NGHGI a larger net sink. ~90 GtCO₂ for 1.5°C scenarios [Weber 2026](https://doi.org/10.1038/s41467-026-69078-9). See: [NGHGI-Consistent RCB Corrections](other-operations.md#correction-for-total-co2-budgets-co2) +: The systematic difference between NGHGI and bookkeeping (BM) LULUCF CO₂ estimates. NGHGI includes indirect effects (CO₂ fertilization of managed forests) that BM excludes, making NGHGI a larger net sink. This package computes a cumulative gap of 90.3 GtCO₂ from 2020 to net-zero CO₂ and 65.4 GtCO₂ from 2024 (median of the AR6 C1 scenarios, PRIMAP run with `melo-2026`). [Weber 2026](https://doi.org/10.1038/s41467-026-69078-9) reports that in 2024 the adjustment "reduces the global RCB by 63 (41–121) GtCO2 for 1.5 °C (50%)" (Results, "The global NGHGI-consistent RCB"). See: [NGHGI-Consistent RCB Corrections](other-operations.md#correction-for-total-co2-budgets-co2) **PRIMAP-hist** : Historical emissions dataset from PIK (Potsdam Institute for Climate Impact Research). diff --git a/docs/science/other-operations.md b/docs/science/other-operations.md index 7ab2c68..5cf8097 100644 --- a/docs/science/other-operations.md +++ b/docs/science/other-operations.md @@ -206,9 +206,9 @@ The correction methodology follows Weber et al. (2026). A central design princip | Symbol | Definition | | ------------------------------------- | ----------------------------------------------------------------------------------- | | $\text{RCB}_{\text{BM}}(\text{base})$ | Published Remaining Carbon Budget from baseline year, in BM convention | -| $F_{\text{actual}}(a, b)$ | Cumulative actual fossil CO₂ emissions from year $a$ to year $b$ (e.g. PRIMAP) | +| $F_{\text{actual}}(a, b)$ | Cumulative actual fossil CO₂ emissions without international bunkers, years $a$ to $b$ (e.g. PRIMAP) | | $L_{\text{BM}}(a, b)$ | Cumulative BM LULUCF CO₂ from scenario median (AFOLU\|Direct), years $a$ to $b$ | -| $L_{\text{BM,actual}}(a, b)$ | Cumulative actual observed BM LULUCF CO₂ (e.g. PRIMAP co2-lulucf), years $a$ to $b$ | +| $L_{\text{NGHGI}}(a, b)$ | Cumulative observed NGHGI LULUCF CO₂ (world row of the LULUCF source, Melo et al.), years $a$ to $b$ | | $B(a, b)$ | Cumulative international bunker fuel CO₂ emissions, years $a$ to $b$ | | $\text{gap}(a, b)$ | Cumulative NGHGI--BM convention gap for CO₂, years $a$ to $b$ | | $\text{NZ}$ | Net-zero year (from scenario data) | @@ -221,31 +221,42 @@ The fossil-allocatable budget isolates the portion of the total carbon budget av $$ \mathrm{fossil{\_}budget}(2020) = \text{RCB}_{\text{BM}}(\text{base}) + F_{\text{actual}}(2020,\, \text{base}{-}1) + + B(2020,\, \text{base}{-}1) - L_{\text{BM}}(\text{base},\, \text{NZ}) - B(2020,\, \text{NZ}) $$ -The four terms are: +The five terms are: + +1. **$\text{RCB}_{\text{BM}}(\text{base})$** -- the published carbon budget, which covers _total_ anthropogenic CO₂ (fossil + international bunkers + BM LULUCF) from the baseline year onward. + +2. **$F_{\text{actual}}(2020,\, \text{base}{-}1)$** -- the fossil rebase. When the published baseline is after 2020, actual fossil emissions from 2020 to $\text{base}{-}1$ are added back. This uses only observational data (e.g. PRIMAP), never scenario projections. The world fossil series excludes international bunkers: the PRIMAP world total equals the sum of its countries, and the `gcb-2025` world row is the GCB global total minus bunkers. When $\text{base} = 2020$, this term is zero. + +3. **$B(2020,\, \text{base}{-}1)$** -- the bunker rebase. Observed bunker emissions from 2020 to $\text{base}{-}1$ are added back, because the published budget includes bunkers and the fossil rebase does not. When $\text{base} = 2020$, this term is zero. The output column is `rebase_bunkers_mt`. + +4. **$L_{\text{BM}}(\text{base},\, \text{NZ})$** -- the LULUCF decomposition. Removes the BM LULUCF share of the budget from the baseline year to the scenario net-zero year, using AR6 scenario median pathways (AFOLU|Direct). This is the only way to separate the fossil and land-use portions of the total budget. -1. **$\text{RCB}_{\text{BM}}(\text{base})$** -- the published carbon budget, which covers _total_ anthropogenic CO₂ (fossil + BM LULUCF) from the baseline year onward. +5. **$B(2020,\, \text{NZ})$** -- the bunker deduction. Removes international aviation and shipping emissions that appear in global totals but are excluded from national inventories. Integrated from 2020 to NZ regardless of baseline year. Observed values run to the last year of the bunker data. Later years take the rate of that last year. -2. **$F_{\text{actual}}(2020,\, \text{base}{-}1)$** -- the fossil rebase. When the published baseline is after 2020, actual fossil emissions from 2020 to $\text{base}{-}1$ are added back. This uses only observational data (e.g. PRIMAP), never scenario projections. When $\text{base} = 2020$, this term is zero. +#### Why the rebase adds bunkers -3. **$L_{\text{BM}}(\text{base},\, \text{NZ})$** -- the LULUCF decomposition. Removes the BM LULUCF share of the budget from the baseline year to the scenario net-zero year, using AR6 scenario median pathways (AFOLU|Direct). This is the only way to separate the fossil and land-use portions of the total budget. +The two bunker terms count each bunker year once. The bunker rebase restores the published budget to a total-CO₂ budget from 2020. The deduction then removes every bunker year from 2020 to NZ: -4. **$B(2020,\, \text{NZ})$** -- the bunker deduction. Removes international aviation and shipping emissions that appear in global totals but are excluded from national inventories. Integrated from 2020 to NZ regardless of baseline year. +$$ +B(2020,\, \text{base}{-}1) - B(2020,\, \text{NZ}) = -B(\text{base},\, \text{NZ}) +$$ + +The years 2020 to $\text{base}{-}1$ cancel, and the net deduction covers the years that the published budget holds. The result is the budget from 2020 without any bunker emissions, for every baseline year. A world emissions series without bunkers then extends it to other allocation years. #### Why LULUCF is absent from the co2-ffi rebase -The rebase needs to shift the budget's starting point from $\text{base}$ to 2020. A naive approach would add _all_ actual emissions (fossil + LULUCF) for the rebase period. But the LULUCF decomposition already covers the full range from $\text{base}$ to NZ, so adding actual BM LULUCF from 2020 to $\text{base}{-}1$ alongside decomposing from $\text{base}$ to NZ is equivalent to decomposing from 2020 to NZ: +The rebase shifts the starting point of the budget from $\text{base}$ to 2020. The published budget is in the BM convention, so a full rebase would add BM LULUCF for 2020 to $\text{base}{-}1$. The decomposition would then remove BM LULUCF for 2020 to NZ: $$ -\underbrace{L_{\text{BM,actual}}(2020,\, \text{base}{-}1)}_{\text{rebase LULUCF}} -+ \underbrace{L_{\text{BM}}(\text{base},\, \text{NZ})}_{\text{decomposition}} -\approx L_{\text{BM}}(2020,\, \text{NZ}) +L_{\text{BM}}(2020,\, \text{base}{-}1) - L_{\text{BM}}(2020,\, \text{NZ}) = -L_{\text{BM}}(\text{base},\, \text{NZ}) $$ -Since we would need to subtract $L_{\text{BM}}(2020,\, \text{NZ})$ to isolate the fossil budget anyway, the rebase LULUCF and the decomposition LULUCF from 2020 to $\text{base}{-}1$ cancel algebraically. The formula therefore omits actual LULUCF from the rebase and starts the LULUCF decomposition at $\text{base}$, not 2020. The result is the same, but the formula is simpler and avoids mixing actual and scenario data for overlapping years. +The years 2020 to $\text{base}{-}1$ cancel. The formula therefore omits LULUCF from the rebase and starts the LULUCF decomposition at $\text{base}$. The pipeline holds no observed LULUCF series in the BM convention, and this form needs none. #### Precautionary cap on BM LULUCF @@ -255,38 +266,45 @@ $$ L_{\text{BM}}^{\text{capped}} = \max\!\left(0,\; \underset{i}{\text{median}}\left[\sum_{t=2020}^{t_{\text{nz},i}} L_{\text{BM},i}(t)\right] - \underset{i}{\text{median}}\left[L_{\text{BM},i}(t)\right]_{t=2020}^{\text{base}-1}\right) $$ -The LULUCF decomposition extends from the baseline year to net-zero. Thus we must use AR6 scenario pathways corresponding to the same climate category as the RCB (e.g., 1.5°C 50th percentile scenarios for a 1.5p50 budget). Each scenario's cumulative AFOLU|Direct is integrated from 2020 to its own net-zero year $t_{\text{nz},i}$, then the median across scenarios is taken (integrate per scenario, then median — consistent with the convention gap computation per Weber 2026). The pre-computed median cumulative is stored as `bm_lulucf_cumulative_median` in `rcb_scenario_adjustments.yaml`. When the RCB baseline year $\text{base} > 2020$, the 2020-to-base prefix is subtracted using the per-year median timeseries (accurate because historical BM LULUCF has negligible inter-scenario spread). When the result is negative (net sink), the cap zeros it out because the sink relies on uncertain future reforestation. Configurable via `precautionary_lulucf` in the adjustments config (default: `true`; set to `false` for sensitivity analysis). +The LULUCF decomposition extends from the baseline year to net-zero. Thus we must use AR6 scenario pathways corresponding to the same climate category as the RCB (e.g., 1.5°C 50th percentile scenarios for a 1.5p50 budget). Each scenario's cumulative AFOLU|Direct is integrated from 2020 to its own net-zero year $t_{\text{nz},i}$, then the median across scenarios is taken. The package follows the structure of [Weber 2026](https://doi.org/10.1038/s41467-026-69078-9) (Methods, Eqs. 2--3) and differs in two ways: it takes the median across scenarios, and it uses the scenario Direct flux as the bookkeeping side. The paragraph "Convention choice for the BM side" under [Convention gap decomposition](#convention-gap-decomposition) states both. The pre-computed median cumulative is stored as `bm_lulucf_cumulative_median` in `rcb_scenario_adjustments.yaml`. When the RCB baseline year $\text{base} > 2020$, the 2020-to-base prefix is subtracted using the per-year median timeseries (accurate because historical BM LULUCF has negligible inter-scenario spread). When the result is negative (net sink), the cap zeros it out because the sink relies on uncertain future reforestation. Configurable via `precautionary_lulucf` in the adjustments config (default: `true`; set to `false` for sensitivity analysis). -!!! note "Why `convention_gap_median` is 0 in CO2-FFI output" - The `rcb_scenario_adjustments.yaml` file includes a `convention_gap_median` field for every AR6 category. For **co2-ffi** allocations, this field is **not used** — the LULUCF adjustment comes from `bm_lulucf_cumulative_median` instead (see above). The convention gap only applies to **co2** allocations, where the budget must switch from the bookkeeping model convention to the NGHGI convention. Its presence in the YAML for co2-ffi scenarios is a storage artefact, not a computational input. +!!! note "Why `convention_gap_median_from` is 0 in CO2-FFI output" + The `rcb_scenario_adjustments.yaml` file includes a `convention_gap_median_from` field for every scenario set, with one value per RCB baseline year. For **co2-ffi** allocations, this field is **not used** — the LULUCF adjustment comes from `bm_lulucf_cumulative_median` instead (see above). The convention gap only applies to **co2** allocations, where the budget must switch from the bookkeeping model convention to the NGHGI convention. Its presence in the YAML for co2-ffi scenarios is a storage artefact, not a computational input. ### Correction for total CO₂ budgets (co2) -For budgets covering **total CO₂** including land use, LULUCF stays in the budget but the convention must switch from BM to NGHGI. Unlike the co2-ffi case, there is no LULUCF decomposition -- only a convention gap adjustment: +For budgets covering **total CO₂** including land use, LULUCF stays in the budget but the convention must switch from BM to NGHGI. The co2 case has no LULUCF decomposition. The rebase adds observed NGHGI LULUCF for 2020 to $\text{base}{-}1$, and a convention gap converts the published budget from $\text{base}$ to NZ: $$ \mathrm{nghgi{\_}budget}(2020) = \text{RCB}_{\text{BM}}(\text{base}) + F_{\text{actual}}(2020,\, \text{base}{-}1) - + L_{\text{BM,actual}}(2020,\, \text{base}{-}1) - + \text{gap}(2020,\, \text{NZ}) + + B(2020,\, \text{base}{-}1) + + L_{\text{NGHGI}}(2020,\, \text{base}{-}1) + + \text{gap}(\text{base},\, \text{NZ}) - B(2020,\, \text{NZ}) $$ -The five terms are: +with $\text{gap}(a, b) = \sum_{t=a}^{b} \left[L_{\text{NGHGI}}(t) - L_{\text{BM}}(t)\right]$. + +The six terms are: 1. **$\text{RCB}_{\text{BM}}(\text{base})$** -- the published carbon budget, same as for co2-ffi. 2. **$F_{\text{actual}}(2020,\, \text{base}{-}1)$** -- the fossil rebase, identical to co2-ffi. -3. **$L_{\text{BM,actual}}(2020,\, \text{base}{-}1)$** -- the BM LULUCF rebase. Unlike co2-ffi, actual observed BM LULUCF _is_ included in the rebase. This is because there is no LULUCF decomposition to cancel with -- the budget retains the full land-use component. Source: e.g. PRIMAP co2-lulucf (already in the pipeline). +3. **$B(2020,\, \text{base}{-}1)$** -- the bunker rebase, identical to co2-ffi. -4. **$\text{gap}(2020,\, \text{NZ})$** -- the BM-to-NGHGI convention gap. Covers the full period from 2020 (not from $\text{base}$) because the BM LULUCF rebase is in BM convention — the gap for the rebase years converts it to NGHGI. This quantity is negative (NGHGI reports a larger land sink than BM), so it reduces the allocatable budget. Computed from NGHGI actual LULUCF and scenario BM LULUCF (AFOLU|Direct) data (see [Convention gap decomposition](#convention-gap-decomposition)). +4. **$L_{\text{NGHGI}}(2020,\, \text{base}{-}1)$** -- the NGHGI LULUCF rebase. Observed LULUCF CO₂ in the national-inventory convention is added for 2020 to $\text{base}{-}1$. The source is the world row of the active LULUCF source (Melo et al.), which notebook 107 writes. The pipeline holds no observed LULUCF series in the BM convention. When $\text{base} = 2020$, this term is zero. The output column is `rebase_lulucf_mt`. -5. **$B(2020,\, \text{NZ})$** -- the bunker deduction, same as for co2-ffi. +5. **$\text{gap}(\text{base},\, \text{NZ})$** -- the BM-to-NGHGI convention gap. It covers the years that the published budget covers, from $\text{base}$ to NZ, and replaces BM LULUCF with NGHGI LULUCF for those years. [Weber 2026](https://doi.org/10.1038/s41467-026-69078-9) (Methods, Eqs. 2--3) uses the same split: the correction term starts at the reference year of the budget, and earlier years take observed NGHGI LULUCF. This quantity is negative (NGHGI reports a larger land sink than BM), so it reduces the allocatable budget. It is computed per scenario from observed NGHGI LULUCF and scenario AFOLU data (see [Convention gap decomposition](#convention-gap-decomposition)). Notebook 104 stores one median per baseline year in `convention_gap_median_from`. The output column is `correction_lulucf_nghgi_mt`. -#### Why the co2 rebase includes actual BM LULUCF +6. **$B(2020,\, \text{NZ})$** -- the bunker deduction, same as for co2-ffi. -In the co2-ffi formula, actual BM LULUCF in the rebase period cancels with the decomposition for the same years. In the co2 formula there is no LULUCF decomposition (because land-use emissions stay in the budget), so there is nothing for the rebase LULUCF to cancel with. The rebase must therefore include both fossil and BM LULUCF to correctly shift the total CO₂ budget from $\text{base}$ to 2020. +#### Why the co2 rebase adds NGHGI LULUCF + +The target quantity is cumulative fossil CO₂ plus NGHGI LULUCF from 2020 to NZ. The published budget holds fossil CO₂, bunkers and BM LULUCF from $\text{base}$ to NZ. + +For 2020 to $\text{base}{-}1$ the rebase adds the observed quantities in the target convention: fossil CO₂, bunkers and NGHGI LULUCF. For $\text{base}$ to NZ the convention gap converts the LULUCF share of the published budget from BM to NGHGI. Each year from 2020 to NZ is converted once. The gap starts at $\text{base}$ because the rebase LULUCF of the earlier years is already in the NGHGI convention. ### Design principle: actual data for the rebase, scenario data for the future @@ -295,13 +313,65 @@ The formulas enforce a strict separation: | Quantity | Data type | Rationale | | ------------------------------------------------------- | ------------------------------- | --------------------------------------------------------------------- | | Fossil rebase ($F_{\text{actual}}$) | Actual (e.g. PRIMAP) | Observed emissions -- no projection uncertainty | -| BM LULUCF rebase ($L_{\text{BM,actual}}$, co2 only) | Actual (e.g. PRIMAP co2-lulucf) | Same rationale | +| Bunker rebase ($B$, 2020 to $\text{base}{-}1$) | Actual (GCB bunker series) | Same rationale | +| NGHGI LULUCF rebase ($L_{\text{NGHGI}}$, co2 only) | Actual (Melo et al. world row) | Same rationale | | BM LULUCF decomposition ($L_{\text{BM}}$, co2-ffi only) | Scenario median (AFOLU\|Direct) | Requires future pathway to NZ; no observational data exists | -| Convention gap ($\text{gap}$) | Scenario-based | Forward-looking NGHGI--BM difference requires modeled indirect fluxes | +| Convention gap ($\text{gap}$, co2 only) | Observed NGHGI + scenario data | Forward-looking NGHGI--BM difference requires modeled indirect fluxes | | Net-zero year ($\text{NZ}$) | Scenario data | By definition a future quantity | | Bunker deduction ($B$) | Observational + extrapolation | Historical data extended at last observed rate to NZ | -This means that adding a new RCB source (e.g., Lamboll et al. with baseline 2023) only requires actual emissions data through 2022 for the rebase. The LULUCF decomposition integrates from $\text{base}$ (co2-ffi), while the convention gap and bunker deduction always cover the full 2020--NZ period. +This means that adding a new RCB source (e.g., Lamboll et al. with baseline 2023) only requires actual fossil, bunker and, for co2, NGHGI LULUCF data through 2022 for the rebase. The LULUCF decomposition (co2-ffi) and the convention gap (co2) start at $\text{base}$. The bunker deduction covers 2020 to NZ. + +The rebase needs a value for every year from 2020 to $\text{base}{-}1$ in each series it adds: fossil emissions, bunkers and, for co2, LULUCF. + +#### Placeholder for unobserved rebase years + +A series can end before $\text{base}{-}1$. Each later year then takes the last observed value of that series. This fill is a placeholder until observed data are published. `rebase_fill_max_years` in `rcbs.yaml` sets the largest number of years that one series may take this way (default 1; 0 turns the fill off). Every fill raises a warning that names the source, the series, the filled years and the value. + +A source that needs more years than the limit is left out of the processed budgets. The pipeline prints one warning that lists the sources left out and their missing years. `forster_2026` (baseline 2026) needs values through 2025. With `gcb-2025` emissions and bunkers (observed to 2024) the rebase fills 2025 with the 2024 values. With PRIMAP (observed to 2023) two years are missing and the source is left out. For `co2`, `forster_2026` also needs NGHGI LULUCF through 2025. With `melo-2026` (ends 2023) two years are missing and the source is left out. With `melo-2026-v4` (ends 2024) the rebase fills 2025 with the 2024 value. `lamboll_2023` and `forster_2024` need no fill with either emissions source. + +The bunker deduction already holds the last observed rate for later years. A filled bunker year in the rebase therefore cancels exactly against the same year in the deduction. + +### RCB sources and scenario sets + +`data/rcbs/rcbs.yaml` holds five sources. A budget from the start of year $X$ has `baseline_year: X`. + +| Source key | Publication | Budget from | Scenario selection | +| -------------- | ---------------------------------------------------------------------------------------------------- | ------------- | ------------------ | +| `lamboll_2023` | [Lamboll et al. 2023](https://doi.org/10.1038/s41558-023-01848-5) | Start of 2023 | AR6 category | +| `forster_2024` | [Forster et al. 2024](https://doi.org/10.5194/essd-16-2625-2024), IGCC 2023 | Start of 2024 | AR6 category | +| `ar6_2020` | IPCC AR6 WGI | Start of 2020 | AR6 category | +| `forster_2026` | [Forster et al. 2026](https://doi.org/10.5194/essd-18-3889-2026), IGCC 2025, Table 8 | Start of 2026 | Peak-warming band | +| `ar6_wg1_2021` | [IPCC AR6 WGI, Table SPM.2](https://doi.org/10.1017/9781009157896.001) | Start of 2020 | Peak-warming band | + +The two band sources carry seven budgets each (GtCO₂, total anthropogenic CO₂): + +| Label | Temperature | Likelihood | `forster_2026` | `ar6_wg1_2021` | +| -------- | ----------- | ---------- | -------------- | -------------- | +| `1.5p50` | 1.5°C | 50% | 130 | 500 | +| `1.5p67` | 1.5°C | 67% | 80 | 400 | +| `1.7p50` | 1.7°C | 50% | 500 | 850 | +| `1.7p67` | 1.7°C | 67% | 390 | 700 | +| `2p50` | 2°C | 50% | 1050 | 1350 | +| `2p67` | 2°C | 67% | 860 | 1150 | +| `2p83` | 2°C | 83% | 690 | 900 | + +Every deduction (LULUCF decomposition, convention gap, net-zero year for bunkers) is a median over a set of AR6 scenarios. The `scenario_selection` field of each source in `rcbs.yaml` names the rule that selects the set: + +- **`ar6-category`** (default when the field is absent). The set is one AR6 climate category: `1.5p50` uses C1 (70 scenarios), `2p83` uses C2 (106), `2p66` uses C3 (231). Any other label raises an error that names the source and the label. +- **`peak-warming-band`**. The set is every AR6 scenario whose `Median peak warming (MAGICCv7.5.3)` lies in $[T - 0.05,\; T + 0.05)$, where $T$ is the budget temperature, and that reaches net-zero CO₂ by 2100. A remaining carbon budget runs to net-zero CO₂, so scenarios that never reach it are excluded. The band spans AR6 categories. A band that selects no scenario raises an error that names the temperature and the band. + +One band serves every likelihood of its temperature. The published likelihoods cover TCRE uncertainty only, so the 50%, 67% and 83% budgets of one temperature share the same LULUCF, convention-gap and bunker deductions. + +With the Gidden et al. AR6 metadata, the bands hold 14 scenarios at 1.5°C (all in C1), 131 at 1.7°C (50 in C2, 81 in C3) and 69 at 2°C (34 in C4, 35 in C5). Of these, 14, 111 and 37 scenarios reach net-zero CO₂ by 2100 and form the scenario sets. The 1.5°C deductions rest on 14 scenarios. + +The net-zero test is the one that sets the integration bounds (see [Per-scenario net-zero years](#per-scenario-net-zero-years-as-integration-bounds)): the first year in which the scenario's `Emissions|CO2` time series is at or below zero. The filter and the deduction horizon therefore use one definition. + +The band half-width is the `peak_warming_band_half_width` value in `rcbs.yaml` (default 0.05). Set it to 0.1 for a sensitivity run. + +Notebook 104 stores the results per scenario set in `rcb_scenario_adjustments.yaml`. Category sets are keyed by label (e.g., `1.5p50`), band sets by temperature (e.g., `peak-warming-1.7C`). The selection logic is `select_rcb_scenario_set()` in `src/fair_shares/library/utils/data/rcb.py`. + +The seven-label sources serve the `co2-ffi` and `co2` budget targets. Composite targets (`all-ghg`, `all-ghg-ex-co2-lulucf`) add a non-CO₂ scenario pathway per temperature and likelihood, and those pathways exist for 1.5°C at 50%, 2°C at 66% and 2°C at 83% only. ### Why two LULUCF conventions matter @@ -311,22 +381,22 @@ The global difference is substantial: NGHGI-reported LULUCF is a larger net sink ### Per-scenario net-zero years as integration bounds -Forward-looking quantities (LULUCF decomposition, convention gap, bunker deduction) are integrated from their start year to the **scenario-specific net-zero year** $t_{\text{nz},i}$ -- the first year when that scenario's total CO₂ emissions (`Emissions|CO2`) reach zero. This prevents post-net-zero negative emissions from inflating the corrections. +The LULUCF decomposition and the convention gap are integrated per scenario from their start year to the **scenario-specific net-zero year** $t_{\text{nz},i}$, the first year when that scenario's total CO₂ emissions (`Emissions|CO2`) reach zero. This prevents post-net-zero negative emissions from inflating the corrections. The bunker deduction is integrated to the median net-zero year of the scenario set. Note that `Emissions|CO2` in IAM scenario databases is fossil + BM LULUCF by convention — this is a property of how scenarios report total CO₂, not a methodological choice by fair-shares. -Per-scenario net-zero years are computed from scenario data (e.g. Gidden et al. AR6 reanalysis). Scenario-level summary statistics (median, quartiles) are stored in `rcb_scenario_adjustments.yaml` in the pipeline output directory, keyed by RCB scenario label (e.g., `1.5p50`). The scenario-level median NZ year is used for the bunker integration endpoint (which is observational, not scenario-dependent). +Per-scenario net-zero years are computed from scenario data (e.g. Gidden et al. AR6 reanalysis). Scenario-level summary statistics (median, quartiles) are stored in `rcb_scenario_adjustments.yaml` in the pipeline output directory, keyed by scenario set (e.g., `1.5p50` or `peak-warming-1.7C`; see [RCB sources and scenario sets](#rcb-sources-and-scenario-sets)). The scenario-level median NZ year is used for the bunker integration endpoint (which is observational, not scenario-dependent). -Scenarios that never reach net-zero total CO₂ before 2100 are assigned 2100 as a conservative upper integration bound. +Scenarios that never reach net-zero total CO₂ by 2100 are assigned 2100 as a conservative upper integration bound. This applies to the AR6 category sets (38 of 231 scenarios in C3). The peak-warming band sets hold no such scenario. ### Convention gap decomposition -The per-scenario convention gap $\text{Gap}_i$ decomposes into two temporal segments: +The per-scenario convention gap $\text{Gap}_i$ runs from the baseline year $\text{base}$ of the RCB source to the net-zero year of the scenario. It decomposes into two temporal segments: -**Historical** ($2020 \leq t \leq \mathrm{splice{\_}year}$): NGHGI actual LULUCF (reported values, same for all scenarios) minus BM LULUCF for scenario $i$ (the bookkeeping proxy from scenario data). The splice year is derived dynamically from the data (currently 2023). The gap always covers the full 2020--NZ range because the BM LULUCF rebase years also need convention conversion: +**Observed** ($\text{base} \leq t \leq \mathrm{splice{\_}year}$): NGHGI actual LULUCF (reported values, same for all scenarios) minus BM LULUCF for scenario $i$ (the bookkeeping proxy from scenario data). The splice year is the last year of the NGHGI data (2023 for `melo-2026`, 2024 for `melo-2026-v4`): $$ -\text{Gap}_{i,\text{hist}} = \sum_{t=2020}^{\min(\text{splice},\, t_{\text{nz},i})} \left[\text{NGHGI}_{\text{actual}}(t) - \text{BM}_{\text{Direct},i}(t)\right] +\text{Gap}_{i,\text{obs}} = \sum_{t=\text{base}}^{\min(\text{splice},\, t_{\text{nz},i})} \left[\text{NGHGI}_{\text{actual}}(t) - \text{BM}_{\text{Direct},i}(t)\right] $$ Currently, NGHGI actual LULUCF is sourced from Melo et al. and BM LULUCF from the Gidden et al. AR6 reanalysis (AFOLU|Direct). These are the current data sources for these roles — alternatives providing the same quantities could be substituted. @@ -334,29 +404,35 @@ Currently, NGHGI actual LULUCF is sourced from Melo et al. and BM LULUCF from th **Future** ($t > \mathrm{splice{\_}year}$): Only the NGHGI-consistent indirect component for scenario $i$ (CO₂ fertilization and other passive fluxes), because the direct components cancel in the gap: $$ -\text{Gap}_{i,\text{future}} = \sum_{t=\text{splice}+1}^{t_{\text{nz},i}} \text{Indirect}_{i}(t) +\text{Gap}_{i,\text{future}} = \sum_{t=\max(\text{base},\, \text{splice}+1)}^{t_{\text{nz},i}} \text{Indirect}_{i}(t) $$ Currently sourced from the Gidden et al. AR6 reanalysis (AFOLU|Indirect). -The total per-scenario gap is $\text{Gap}_i = \text{Gap}_{i,\text{hist}} + \text{Gap}_{i,\text{future}}$, and the median is taken across all scenarios $i$ in the corresponding climate category pool. Each scenario's integration ends at its own $t_{\text{nz},i}$. +A baseline year after the splice year leaves the observed segment empty, and the gap is the indirect component from $\text{base}$. A scenario that reaches net zero before $\text{base}$ has a gap of zero. The years 2020 to $\text{base}{-}1$ are outside the gap: the rebase adds observed NGHGI LULUCF for them ([Weber 2026](https://doi.org/10.1038/s41467-026-69078-9), Methods, Eqs. 2--3). + +The total per-scenario gap is $\text{Gap}_i = \text{Gap}_{i,\text{obs}} + \text{Gap}_{i,\text{future}}$, and the median is taken across all scenarios $i$ in the scenario set. Each scenario's integration ends at its own $t_{\text{nz},i}$. Notebook 104 computes one median for each baseline year in `rcbs.yaml` (`convention_gap_median_from` in `rcb_scenario_adjustments.yaml`), and a budget takes the median of its own baseline year. The per-scenario function is `convention_gap_from_baseline()` in `src/fair_shares/library/utils/data/rcb.py`. **Why per-scenario data for the BM side?** The convention gap is a per-scenario quantity — each scenario has its own AFOLU|Direct pathway, NZ year, and Indirect fluxes. Global Carbon Budget multi-model averages cannot be substituted here because they would collapse the per-scenario variation into a single number, breaking the integrate-per-scenario-then-median methodology. For actual historical emissions (the RCB rebase from baseline to 2020), observational data is used — no scenario data is involved in that step. +**Convention choice for the BM side.** The BM side of the convention gap is the scenario `AFOLU|Direct` flux from the Gidden et al. 2023 reanalysis, also for the observed years from $\text{base}$ to the splice year. An observed mean of bookkeeping models is an alternative for those years. [Weber 2026](https://doi.org/10.1038/s41467-026-69078-9) adjusts the Gidden et al. values with the difference between the Global Carbon Budget mean of bookkeeping models and NGHGI-reported values for 2020 to 2023 (Methods, "Calculating a NGHGI-consistent global RCB"). The package holds no observed BM series and makes no such adjustment. Weber et al. also take the mean across scenarios, and the package takes the median. + ### World CO₂ timeseries for backward extension When the allocation year is before 2020, historical emissions must be added back to the RCB (see [RCB Pathway Generation](#rcb-pathway-generation) above). For total CO₂, the per-year world emissions use the NGHGI convention: $$ -E_{\text{world}}(t) = E_{\text{fossil}}(t) - E_{\text{bunkers}}(t) + \text{LULUCF}(t) +E_{\text{world}}(t) = E_{\text{fossil}}(t) + \text{LULUCF}(t) $$ +$E_{\text{fossil}}$ is the world fossil series, which excludes international bunkers. The series therefore matches the adjusted budget, which holds no bunker emissions. `build_nghgi_world_co2_timeseries()` and the preprocessing notebooks build the same series. + Where LULUCF uses: - **2000 onwards**: NGHGI LULUCF (e.g. Melo v3.1, nationally aggregated inventory data) - **Pre-2000**: Not available in NGHGI convention. Categories including LULUCF are limited to the NGHGI data range (2000+). No NGHGI/BM splicing is performed. While earlier NGHGI LULUCF estimates may exist (e.g., via Grassi et al. or historical extensions of Melo), the official NGHGI data begins in 2000. Extending to 1990 would require splicing heterogeneous datasets, which risks introducing artefacts (the BM-to-NGHGI transition around 1990 shows a large jump). We plan to extend coverage when validated pre-2000 NGHGI LULUCF data becomes available. -This ensures the world timeseries passed to `calculate_budget_from_rcb` is NGHGI-consistent, and that function works identically for both `co2-ffi` and `co2` categories. +This ensures the world timeseries passed to `calculate_budget_from_rcb` is NGHGI-consistent, and that function works identically for both `co2-ffi` and `co2` categories. An allocation year before the first year of the world series raises an error that names the first available year. ### Data requirements for new scenario sources @@ -366,31 +442,30 @@ When adding a new RCB source (e.g., a new publication with a different baseline | -------------------------- | -------------------------------------- | --------------------------------------- | | RCB value + baseline year | Starting point | Published literature | | Actual fossil CO₂ | Rebase | e.g. PRIMAP (already in pipeline) | -| Actual BM LULUCF | co2 rebase | e.g. PRIMAP co2-lulucf (already in pipeline) | +| Observed NGHGI LULUCF | co2 rebase | World row of the LULUCF source, Melo et al. (already in pipeline) | | Per-year BM LULUCF pathway | co2-ffi LULUCF decomposition | Scenario data (AFOLU\|Direct median) | | Net-zero year | Integration limit for bunkers + LULUCF | Scenario data | | Convention gap | co2 BM-to-NGHGI adjustment | NGHGI + scenario Indirect AFOLU | -| Bunker fuel timeseries | Bunker deduction | NGHGI (already in pipeline) | +| Bunker fuel timeseries | Bunker rebase and bunker deduction | GCB (already in pipeline) | -The first three rows are observational and already available in the pipeline. The remaining four require scenario data for the new source's mitigation pathway category. +The RCB value comes from the publication. Fossil CO₂, NGHGI LULUCF and the bunker timeseries are observational and already in the pipeline. The bunker deduction extends the last observed bunker rate to the median net-zero year. The BM LULUCF pathway, the net-zero year and the convention gap need scenario data for the scenario set of the new source. ### Data sources | Component | Source | Coverage | | ------------------ | --------------------------------------------- | ---------------------------------------- | | Fossil CO₂ | PRIMAP-hist v2.6.1 | 1750--present | -| BM LULUCF (actual) | PRIMAP co2-lulucf | Country-level, annual | -| NGHGI LULUCF | Melo et al. (2026) v3.1 NGHGI LULUCF | 2000--2023, 187 countries + world | +| NGHGI LULUCF | Melo et al. (2026) NGHGI LULUCF: v3.1.1 (`melo-2026`, default) or v4.0.0 (`melo-2026-v4`) | 2000--2023, 185 countries + world (v3.1.1); 2000--2024, 187 countries + world (v4.0.0) | | BM LULUCF proxy | Gidden et al. AR6 reanalysis, AFOLU\|Direct | 2015--2100, per scenario within category | | Passive flux | Gidden et al. AR6 reanalysis, AFOLU\|Indirect | 2015--2100, per scenario within category | | Net-zero years | Gidden et al. AR6 reanalysis, Emissions\|CO2 | Per scenario (first year total CO₂ ≤ 0) | -| Bunker fuels | GCB2024 historical + rate extrapolation | Historical + extrapolated to median NZ | +| Bunker fuels | GCB historical + rate extrapolation | Observed to the last year of the data, then that rate to median NZ | **[API Reference →](https://setupelz.github.io/fair-shares/api/utils/data/#nghgi-corrections)** | `src/fair_shares/library/utils/data/nghgi.py` ### Worked example: AR6 WG1 1.5C 50% (1.5p50) -Using `ar6_2020` source: 500 GtCO₂ total from 2020, scenario `1.5p50` (70 C1 scenarios, median NZ year ~2050). Values from `make dev-pipeline-rcbs` with PRIMAP v2025.03 emissions and Melo v3.1 LULUCF. +Using `ar6_2020` source: 500 GtCO₂ total from 2020, scenario `1.5p50` (70 C1 scenarios, median NZ year ~2050). Values from two pipeline runs with PRIMAP v2025.03 emissions, `gcb-2024` bunkers and Melo v3.1.1 LULUCF (`melo-2026`): `rcbs_co2-ffi.csv` of `primap-202503_wdi-2025_un-owid-2025_wdi-2025_rcbs_co2-ffi` and `rcbs_co2.csv` of `primap-202503_wdi-2025_un-owid-2025_wdi-2025_melo-2026_rcbs_all-ghg`. Each row is rounded, and the totals use unrounded values. #### Step 1: Weber corrections (RCB to allocatable budget at 2020) @@ -398,31 +473,35 @@ Using `ar6_2020` source: 500 GtCO₂ total from 2020, scenario `1.5p50` (70 C1 s | ----------------------------- | ---------------------- | --------------------- | | Published RCB (total CO₂) | 500 Gt | 500 Gt | | Fossil rebase | 0 (base=2020) | 0 (base=2020) | -| BM LULUCF rebase | -- | 0 (base=2020) | -| LULUCF decomposition / gap | **0** (BM sink capped) | **-90 Gt** (conv gap) | +| Bunker rebase | 0 (base=2020) | 0 (base=2020) | +| NGHGI LULUCF rebase | -- | 0 (base=2020) | +| LULUCF decomposition / gap | **0** (BM sink capped) | **-90 Gt** (conv gap, 2020--NZ) | | Bunker subtraction | -35 Gt | -35 Gt | | **Allocatable budget (2020)** | **465 Gt** | **375 Gt** | **co2-ffi:** The cumulative BM LULUCF is estimated from AR6 scenarios in the corresponding climate category (here 1.5°C 50th percentile). Each scenario's AFOLU|Direct is integrated from 2020 to its own NZ year, then the median across scenarios is taken. The result is a net sink. Under the **precautionary cap** (default), this sink is not credited to the fossil budget (capped to 0). Without the cap (`precautionary_lulucf: false`), the fossil budget would increase. -**co2:** The convention gap is -90 Gt — NGHGI reports a larger land CO₂ sink than bookkeeping models, reducing the allocatable budget. The gap is computed from NGHGI actual LULUCF and scenario BM LULUCF data (see [Convention gap decomposition](#convention-gap-decomposition)). Bunker deduction is ~35 Gt (~870 Mt/yr integrated to median NZ year ~2050). The co2 budget is lower than co2-ffi because the convention gap is a significant negative adjustment. +**co2:** The convention gap from 2020 to NZ is -90 Gt (`1.5p50` median, -90.3 Gt). NGHGI reports a larger land CO₂ sink than bookkeeping models, which reduces the allocatable budget. The gap is computed from NGHGI actual LULUCF and scenario AFOLU data (see [Convention gap decomposition](#convention-gap-decomposition)). Bunker deduction is ~35 Gt (~870 Mt/yr integrated to median NZ year ~2050). The co2 budget is lower than co2-ffi because the convention gap is a significant negative adjustment. -For [Lamboll 2023](https://doi.org/10.1038/s41558-023-01848-5) (`1.5C`, 247 Gt from 2023): +For [Lamboll 2023](https://doi.org/10.1038/s41558-023-01848-5) (`lamboll_2023`, `1.5p50`, 250 Gt from 2023): -| | co2-ffi | co2 | -| ----------------------------- | ---------------------- | --------------------- | -| Published RCB | 247 Gt | 247 Gt | -| Fossil rebase (2020--2022) | +107 Gt | +107 Gt | -| BM LULUCF rebase (2020--2022) | -- | -12 Gt | -| LULUCF decomposition / gap | **0** (BM sink capped) | **-90 Gt** (conv gap) | -| Bunker subtraction | -35 Gt | -35 Gt | -| **Allocatable budget (2020)** | **319 Gt** | **217 Gt** | +| | co2-ffi | co2 | +| ----------------------------- | ---------------------- | ----------------------- | +| Published RCB | 250 Gt | 250 Gt | +| Fossil rebase (2020--2022) | +106.6 Gt | +106.6 Gt | +| Bunker rebase (2020--2022) | +2.8 Gt | +2.8 Gt | +| NGHGI LULUCF rebase (2020--2022) | -- | -12.1 Gt | +| LULUCF decomposition / gap | **0** (BM sink capped) | **-71.2 Gt** (conv gap, 2023--NZ) | +| Bunker subtraction (2020--NZ) | -34.7 Gt | -34.7 Gt | +| **Allocatable budget (2020)** | **324.8 Gt** | **241.5 Gt** | + +The bunker rebase and the bunker subtraction together remove the bunkers of 2023 to NZ (31.8 Gt), the years that the published budget covers. For co2, the rebase adds observed NGHGI LULUCF for 2020 to 2022 (Melo et al. world row, -12.1 Gt), and the convention gap covers 2023 to NZ (`1.5p50` median from 2023, -71.2 Gt). #### Step 2: Allocation year adjustment The allocatable budget is the budget **from 2020 onwards**. The `allocation_year` parameter shifts the starting point by adding historical emissions (before 2020) or subtracting already-used emissions (after 2020). -To regenerate these values, run `make dev-pipeline-rcbs`. +`make dev-pipeline-rcbs` regenerates the co2-ffi values. The co2 values come from a pipeline run with `emission_category=all-ghg` and `active_lulucf_source=melo-2026`. --- diff --git a/docs/science/parameter-effects.md b/docs/science/parameter-effects.md index af7e893..a7a7dd5 100644 --- a/docs/science/parameter-effects.md +++ b/docs/science/parameter-effects.md @@ -55,7 +55,7 @@ _Example uses 1.5°C carbon budget (50% probability). Remaining budget = (cumula `pre_allocation_responsibility_weight` activates per-capita rescaling based on cumulative per-capita emissions in [`pre_allocation_responsibility_year`, `allocation_year`). This is a **separate mechanism** from setting an early `allocation_year` — rescaling is multiplicative and always produces positive allocations if `allocation_year` is the present, whereas cumulative accounting can produce negative allocations (see [comparison table](#if-you-want-x-set-y-guide) below). -`capability_weight` activates capability (ability to pay) adjustments based on GDP per capita from the allocation year onwards. Capability data before the allocation year is not used. When the allocation window extends past the last observed GDP year (default `wdi-2025` ends at 2023), the per-capita budget and pathway primitives forward-fill GDP per capita from the last observed year to cover the rest of the window — holding the cross-country capability ratios of that last year constant for every subsequent year. To use different post-observation capability dynamics (SSP2 projections, custom growth assumptions, or a future-extended WDI release), extend the input `gdp_ts` time series before calling the allocation function. +`capability_weight` activates capability (ability to pay) adjustments based on GDP per capita from the allocation year onwards. Without `capability_reference_year`, capability data before the allocation year is not used; with it, a single snapshot year is used, and that year may precede or follow the allocation year. When the allocation window extends past the last observed GDP year (default `wdi-2025` ends at 2023), the per-capita budget and pathway primitives forward-fill GDP per capita from the last observed year to cover the rest of the window — holding the cross-country capability ratios of that last year constant for every subsequent year. To use different post-observation capability dynamics (SSP2 projections, custom growth assumptions, or a future-extended WDI release), extend the input `gdp_ts` time series before calling the allocation function. **Direction:** @@ -126,6 +126,10 @@ budget and pathway approaches with identical semantics. with a warning; a year outside the data range raises. On the Gini-adjusted approaches the Gini adjustment applies in all three cases — which input the snapshot is read from does not change the capability definition. +- **Start years before the GDP series:** with a reference year set, the budget + approaches need GDP at that year only. `allocation_year` can then be earlier + than the first GDP year, down to 1850. With no reference year, an allocation + year before the GDP series raises an error. **When to use:** replicating studies that fix capability at a reference year (e.g. a single-year GDP-per-capita indicator), or holding the capability @@ -147,8 +151,10 @@ passes through, so both gas components apply the same snapshot. **Direction:** - `floor=0` → All income counts toward capability (no development protection) -- `floor=7500` → $7,500/year (2010 PPP) — the GDR framework's threshold, the level at which basic development indicators (nutrition, infant mortality, education) are broadly met -- `floor=15000` → $15,000/year (2010 PPP) subsistence threshold +- `floor=7500` → 7,500 per person per year, in the units of the GDP series. [Baer 2013](https://doi.org/10.1002/wcc.201) used this value as the GDR development threshold on a 2005 price base and raised it to 8,500 on a 2010 base. +- `floor=15000` → 15,000 per person per year, in the units of the GDP series, a higher subsistence threshold + +The floor must be expressed in the price base of the GDP series passed in. The default GDP source is in constant 2021 international dollars, so a threshold from another price year needs converting first. Higher floors increase the exempt portion, reducing measured capability for all countries and increasing their allocated share of emissions. The effect is largest for middle-income countries where a significant fraction of the population clusters around the threshold. diff --git a/docs/science/principle-to-code.md b/docs/science/principle-to-code.md index 60ff6c8..131029e 100644 --- a/docs/science/principle-to-code.md +++ b/docs/science/principle-to-code.md @@ -308,7 +308,7 @@ allocations = { - `allocation_year=1990` → Cumulative population from 1990 determines each country's share of the total budget - `pre_allocation_responsibility_weight=0.2` + `pre_allocation_responsibility_year=1950` → Per-capita emissions from 1950-1989 rescale allocations (relative mechanism) - `capability_weight=0.8` → GDP-based capability dominates the adjustment (applies from allocation year onwards) -- `income_floor=10000` → Higher than GDR threshold ($7,500/year 2010 PPP) — broader subsistence definition (GDR was designed for burden-sharing; fair-shares adapts its capability metric for entitlement allocation) +- `income_floor=10000` → Higher than the GDR threshold ($7,500/year on a 2005 price base) — broader subsistence definition (GDR was designed for burden-sharing; fair-shares adapts its capability metric for entitlement allocation) - Gini data configured → Within-country inequality affects effective capability **Distributional outcome:** Least developed countries retain most of their remaining allocation. High-income countries have reduced remaining allocations (from cumulative budget accounting since 1990), further reduced by capability_weight=0.8. The pre_allocation_responsibility_weight=0.2 additionally rescales shares based on 1950-1989 per-capita emissions (this rescaling always produces positive allocations if `allocation_year` is the present — it is a multiplicative adjustment, not cumulative accounting). diff --git a/docs/user-guide/data-sources.md b/docs/user-guide/data-sources.md index e7cdc47..c249ba0 100644 --- a/docs/user-guide/data-sources.md +++ b/docs/user-guide/data-sources.md @@ -17,7 +17,9 @@ Bundled sources carry different terms. Check the per-source entry below before r | Data Type | Source | License | Citation Required | | -------------- | ---------------------- | ------------------ | ----------------- | | Emissions | PRIMAP-hist v2.6.1 | **CC-BY-4.0** | Yes | +| Emissions (opt-in) | Global Carbon Budget 2025 (2025v15) | **CC-BY-4.0** | Yes | | LULUCF | Melo et al. 2026 v3.1 | **CC-BY-4.0** (Zenodo) | Yes | +| LULUCF (opt-in) | Melo et al. 2026 v4.0.0 | **CC-BY-4.0** (Zenodo) | Yes | | Population | UN/OWID 2025 | Mixed — see below | Yes | | GDP | World Bank WDI 2025 | **CC-BY-4.0** | Yes | | Gini (default) | World Bank WDI 2025 (SI.POV.GINI) | **CC-BY-4.0** | Yes | @@ -27,6 +29,7 @@ Bundled sources carry different terms. Check the per-source entry below before r | Carbon budgets | Lamboll et al. 2023 | Published values | Yes | | Carbon budgets | Forster et al. 2024 | Published values | Yes | | Bunker fuels | Global Carbon Budget 2024 | paper **CC-BY-4.0**; data product under GCP terms | Yes | +| Bunker fuels (with `gcb-2025` emissions) | Global Carbon Budget 2025 (2025v15) | **CC-BY-4.0** | Yes | --- @@ -44,6 +47,28 @@ Bundled sources carry different terms. Check the per-source entry below before r **What it provides:** National greenhouse gas emissions by country (1750-2023), including CO2 from fossil fuels, land use, and other GHGs. +### Global Carbon Budget 2025 (opt-in) + +**Source (data product):** Andrew, R. M., & Peters, G. P. (2025). The Global Carbon Project's fossil CO2 emissions dataset (2025v15). Zenodo. [doi:10.5281/zenodo.17417124](https://doi.org/10.5281/zenodo.17417124) + +**Source (paper):** Friedlingstein, P., et al. (2026). Global Carbon Budget 2025. *Earth System Science Data*, 18, 3211-3288. [doi:10.5194/essd-18-3211-2026](https://doi.org/10.5194/essd-18-3211-2026) + +**License:** [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/). Cite both DOIs. + +**Location:** `data/emissions/gcb-2025/`. Opt-in: a plain `fair-shares fetch-data` does not download it. Fetch it with `fair-shares fetch-data --source gcb-2025`. + +**What it provides:** Territorial fossil CO2 (`co2-ffi`) by country, 1850 to 2024, in MtCO2 per year. Select it with `active_emissions_source=gcb-2025`. The default stays `primap-202503`. The source declares `co2-ffi` only, and the pipeline asks it for no other category. + +**Processing rules** (`notebooks/101_data_preprocess_emiss_gcb-2025.py`): + +- **Blank years count as zero.** The file leaves a year blank where it reports those emissions under another entity or reports none. Most blanks precede a country's first record. The GCB global total treats every blank as zero. +- **The world row excludes bunkers and the oil fires.** World row = GCB global total minus international shipping, international aviation and the 1991 Kuwaiti oil fires (478 MtCO2). These three belong to no country. The run saves them to `emiss_co2-ffi_excluded_timeseries.csv`, so that world row + excluded rows = GCB global total for every year. +- **Rows outside the region mapping go to rest-of-world.** Kosovo and Antarctica have ISO codes that the region mapping lacks. "Pacific Islands (Palau)" (1955 to 1991, 4.43 MtCO2 in total) and "Ryukyu Islands" (1965 to 1972, 0.82 MtCO2 in total) have no ISO code. The official GCB workbook keeps both in its World column and in no country column, and this source does the same. +- **Bunkers come from the same file.** Runs on `gcb-2025` deduct international shipping plus aviation from this file. Runs on `primap-202503` keep the Global Carbon Budget 2024 workbook. +- **Coverage rule.** An analysis country needs an emissions record before 1990 and population data for every year from 1850 (`coverage` in `data_sources_unified.yaml`). A blank in the raw file is not a record. The run saves each country's first recorded year to `emiss_co2-ffi_first_recorded_year.csv`. Eight countries have their first record in 1990 or later: Andorra, Lesotho and Tuvalu (1990), Namibia (1991), Micronesia, the Marshall Islands and Palau (1992) and Timor-Leste (1994). Their zero-filled values for 1850 to 1989 are all zero, and a responsibility adjustment over those years divides by zero. Macao has population data from 1950. These nine countries join rest-of-world in every allocation, so the country set is the same for every start year from 1850. Rest-of-world is the world row minus the analysis countries, so it takes their emissions, population and GDP. The column `coverage_rule_failed` in `country_data_coverage_summary.csv` names the failed test for each country. Runs on `primap-202503` have no coverage rule and keep their country set. + +With the default GDP and population sources, a `co2-ffi` run holds 168 analysis countries plus rest-of-world: the 177 of a PRIMAP run minus these nine. A `co2` run with `melo-2026-v4` holds 160, because that LULUCF source has no series for eight more countries (Egypt, Hong Kong, Kiribati, Libya, the Maldives, Nauru, Qatar and Sierra Leone). World `co2-ffi` in 2020 is 34,319 MtCO2 (PRIMAP: 34,343). + --- ## LULUCF Data @@ -58,7 +83,19 @@ Bundled sources carry different terms. Check the per-source entry below before r **Location:** `data/lulucf/melo-2026/` -**What it provides:** NGHGI-reported CO2 LULUCF fluxes for 187 countries (2000–2023). Used for all emission categories that include land use (co2, all-ghg). See [Other Operations](../science/other-operations.md) for how NGHGI LULUCF data enters the pipeline. +**What it provides:** NGHGI-reported CO2 LULUCF fluxes for 185 countries (2000–2023). Used for all emission categories that include land use (co2, all-ghg). See [Other Operations](../science/other-operations.md) for how NGHGI LULUCF data enters the pipeline. + +### Melo et al. (NGHGI LULUCF), v4.0.0 (opt-in) + +**Source:** Melo, J., et al. (2026). The LULUCF Data Hub: translating global land use emissions estimates into the national GHG inventory framework (Version 4.0.0, 2026 NGHGI release). Zenodo. + +**DOI:** [10.5281/zenodo.22828743](https://doi.org/10.5281/zenodo.22828743) + +**License:** [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) (Zenodo) + +**Location:** `data/lulucf/melo-2026-v4/` + +**What it provides:** The gap-filled NGHGI CO2 LULUCF file for 187 countries (2000–2024). Select it with `active_lulucf_source=melo-2026-v4`. Fetch it with `fair-shares fetch-data --source melo-2026-v4`. The default stays `melo-2026` (v3.1.1). --- @@ -160,12 +197,20 @@ The global remaining carbon budget (RCB) is a key input for budget-based allocat | Lamboll et al. 2023 | 250 GtCO2 | 1.5°C | 50% | 2023 | Methodology paper for updated RCB estimates | | Forster et al. 2024 | 200 GtCO2 | 1.5°C | 50% | 2024 | IGCC 2023; latest usable with PRIMAP v2.6.1 (through 2023) | | IPCC AR6 WGI | 500 GtCO2 | 1.5°C | 50% | 2020 | Original AR6 estimates from WG1 Chapter 5 | +| Forster et al. 2026 | 130 GtCO2 | 1.5°C | 50% | 2026 | IGCC 2025, Table 8; also 1.7°C and 2°C. Needs emissions through 2025; one missing year takes the last observed value | +| IPCC AR6 WGI (SPM) | 500 GtCO2 | 1.5°C | 50% | 2020 | Table SPM.2; also 1.7°C and 2°C (`ar6_wg1_2021`) | + +The last two sources (`forster_2026`, `ar6_wg1_2021`) carry seven budgets each and select their deduction scenarios by peak-warming band. See [RCB sources and scenario sets](../science/other-operations.md#rcb-sources-and-scenario-sets). **Citations:** > Lamboll, R. D., et al. (2023). Assessing the size and uncertainty of remaining carbon budgets. _Nature Climate Change_, 13, 1360–1367. [doi:10.1038/s41558-023-01848-5](https://doi.org/10.1038/s41558-023-01848-5) -> Forster, P. M., et al. (2024). Indicators of Global Climate Change 2023. _Earth System Science Data_, 16, 2625–2680. [doi:10.5194/essd-16-2625-2024](https://doi.org/10.5194/essd-16-2625-2024) +> Forster, P. M., et al. (2024). Indicators of Global Climate Change 2023. _Earth System Science Data_, 16, 2625–2658. [doi:10.5194/essd-16-2625-2024](https://doi.org/10.5194/essd-16-2625-2024) + +> Forster, P. M., et al. (2026). Indicators of Global Climate Change 2025. _Earth System Science Data_, 18, 3889–3933. [doi:10.5194/essd-18-3889-2026](https://doi.org/10.5194/essd-18-3889-2026) + +> IPCC (2021). Summary for Policymakers. In _Climate Change 2021: The Physical Science Basis_. Cambridge University Press. [doi:10.1017/9781009157896.001](https://doi.org/10.1017/9781009157896.001) !!! note "Budget choice is normatively significant" The choice of carbon budget (source, temperature target, probability level) corresponds to Entry Point 2 of the fair share quantification framework — the allocation quantity [Pelz 2025b](https://doi.org/10.1088/1748-9326/ada45f). Results are sensitive to this choice. Always document the budget source, temperature target, and probability level when reporting allocation results. @@ -202,6 +247,8 @@ The choice of carbon budget (source, temperature target, probability level) corr **What it provides:** International aviation and shipping CO2 emissions, used to deduct bunker fuels from national remaining carbon budgets. See [Other Operations](../science/other-operations.md) for methodology. +Each emissions source names its bunker data (`bunkers_source` in `data_sources_unified.yaml`). The default is this workbook. Runs on `gcb-2025` emissions take bunkers from the [Global Carbon Budget 2025](#global-carbon-budget-2025-opt-in) file. + --- ## Attribution in Your Work diff --git a/docs/user-guide/index.md b/docs/user-guide/index.md index 8a30a97..85107b5 100644 --- a/docs/user-guide/index.md +++ b/docs/user-guide/index.md @@ -93,7 +93,7 @@ fair-shares currently supports three target sources: Two questions: 1. **Budget or pathway?** Do you need a single cumulative target or year-by-year emissions? -2. **Which principles?** Equal entitlements, historical differentiation (via early `allocation_year` for cumulative accounting and/or `pre_allocation_responsibility_weight` for per-capita emissions rescaling), capability (ability to pay, from the allocation year onwards) -- or some combination? Note: "subsistence protection" as a live approach choice has diminished operational value — [Shue 2014](../science/references.md#shue-2014) later acknowledged that what the poor need is energy, not emissions rights. The more relevant operational concept is the GDR development threshold [Baer 2013](https://doi.org/10.1002/wcc.201) (note: GDR was designed for burden-sharing; fair-shares adapts its capability metric for entitlement allocation), which exempts individuals below a development income threshold (~$7,500 PPP) from capability calculations. +2. **Which principles?** Equal entitlements, historical differentiation (via early `allocation_year` for cumulative accounting and/or `pre_allocation_responsibility_weight` for per-capita emissions rescaling), capability (ability to pay, from the allocation year onwards) -- or some combination? Note: "subsistence protection" as a live approach choice has diminished operational value — [Shue 2014](../science/references.md#shue-2014) later acknowledged that what the poor need is energy, not emissions rights. The more relevant operational concept is the GDR development threshold [Baer 2013](https://doi.org/10.1002/wcc.201) (note: GDR was designed for burden-sharing; fair-shares adapts its capability metric for entitlement allocation), which exempts individuals below a development income threshold from capability calculations. Baer 2013 sets it at 7,500 dollars on a 2005 PPP base (8,500 on a 2010 base); give `income_floor` in the price base of the GDP series (default: constant 2021 international dollars). Then: diff --git a/notebooks/100_data_preprocess_pathways.py b/notebooks/100_data_preprocess_pathways.py index d450d52..90946f9 100644 --- a/notebooks/100_data_preprocess_pathways.py +++ b/notebooks/100_data_preprocess_pathways.py @@ -43,6 +43,7 @@ complete_gini, emissions_path, gini_missing_policy, + source_coverage_exclusions, ) from fair_shares.library.utils import ( add_row_timeseries, @@ -404,6 +405,14 @@ # Load region mapping to get the full list of countries all_region_countries = set(region_mapping["iso3c"].unique()) +# Opt-in coverage rule of the emissions source: failing countries join ROW. +coverage_rule_failed = source_coverage_exclusions( + all_region_countries, population, emissions_data_parameters, emiss_intermediate_dir +) +moved_to_row = sorted(analysis_countries & set(coverage_rule_failed)) +analysis_countries = analysis_countries - set(coverage_rule_failed) +print(f"Moved to ROW by the coverage rule: {len(moved_to_row)} {moved_to_row}") + # Create summary dataframe coverage_summary = pd.DataFrame({"iso3c": sorted(all_region_countries)}) @@ -434,6 +443,12 @@ all_region_countries ) & ~coverage_summary["iso3c"].isin(analysis_countries) +# Sources with a coverage rule get one more column: the failed tests. +if emissions_data_parameters.get("coverage"): + coverage_summary["coverage_rule_failed"] = ( + coverage_summary["iso3c"].map(coverage_rule_failed).fillna("") + ) + # Calculate summary statistics total_countries = len(coverage_summary) countries_with_emissions = coverage_summary["has_emissions"].sum() diff --git a/notebooks/100_data_preprocess_rcbs.py b/notebooks/100_data_preprocess_rcbs.py index be556f2..ec78b55 100644 --- a/notebooks/100_data_preprocess_rcbs.py +++ b/notebooks/100_data_preprocess_rcbs.py @@ -47,6 +47,7 @@ complete_gini, emissions_path, gini_missing_policy, + source_coverage_exclusions, ) from fair_shares.library.utils import ( add_row_timeseries, @@ -354,6 +355,14 @@ region_mapping = pd.read_csv(project_root / region_mapping_path) all_region_countries = set(region_mapping["iso3c"].unique()) +# Opt-in coverage rule of the emissions source: failing countries join ROW. +coverage_rule_failed = source_coverage_exclusions( + all_region_countries, population, emissions_data_parameters, emiss_intermediate_dir +) +moved_to_row = sorted(analysis_countries & set(coverage_rule_failed)) +analysis_countries = analysis_countries - set(coverage_rule_failed) +print(f"Moved to ROW by the coverage rule: {len(moved_to_row)} {moved_to_row}") + # Create summary dataframe coverage_summary = pd.DataFrame({"iso3c": sorted(all_region_countries)}) @@ -384,6 +393,12 @@ all_region_countries ) & ~coverage_summary["iso3c"].isin(analysis_countries) +# Sources with a coverage rule get one more column: the failed tests. +if emissions_data_parameters.get("coverage"): + coverage_summary["coverage_rule_failed"] = ( + coverage_summary["iso3c"].map(coverage_rule_failed).fillna("") + ) + # Calculate summary statistics total_countries = len(coverage_summary) countries_with_emissions = coverage_summary["has_emissions"].sum() @@ -558,8 +573,8 @@ # %% [markdown] # ## Verify scenario adjustment constants # -# Per-scenario-category adjustment scalars (NZ years, BM LULUCF, convention gap) -# are generated by the scenario preprocessing notebook and saved to +# Per-scenario-set adjustment scalars (AR6 category or peak-warming band: NZ years, +# BM LULUCF, convention gap) are generated by the scenario preprocessing notebook and saved to # `intermediate/scenarios/rcb_scenario_adjustments.yaml`. # %% @@ -615,10 +630,10 @@ # %% from fair_shares.library.preprocessing.rcbs import load_and_process_rcbs -# Always pass PRIMAP fossil (co2-ffi) as world emissions; -# for total CO2, also pass BM LULUCF for rebase +# Always pass fossil (co2-ffi) as world emissions; +# for total CO2, also pass observed NGHGI LULUCF for the rebase world_fossil_emissions = ensure_string_year_columns(world_emiss["co2-ffi"]) -actual_bm_lulucf = ( +world_nghgi_lulucf = ( ensure_string_year_columns(world_emiss["co2-lulucf"]) if "co2-lulucf" in world_emiss else None @@ -632,7 +647,7 @@ data_dir=project_root / "data", output_dir=project_root / "output", source_id=source_id, - actual_bm_lulucf_emissions=actual_bm_lulucf, + world_nghgi_lulucf_emissions=world_nghgi_lulucf, verbose=True, ) diff --git a/notebooks/101_data_preprocess_emiss_gcb-2025.py b/notebooks/101_data_preprocess_emiss_gcb-2025.py new file mode 100644 index 0000000..021014f --- /dev/null +++ b/notebooks/101_data_preprocess_emiss_gcb-2025.py @@ -0,0 +1,158 @@ +# --- +# jupyter: +# jupytext: +# cell_metadata_filter: tags,-all +# formats: ipynb,py:percent +# text_representation: +# extension: .py +# format_name: percent +# format_version: '1.3' +# jupytext_version: 1.16.6 +# kernelspec: +# display_name: Python 3 (ipykernel) +# language: python +# name: python3 +# --- + +# %% [markdown] +# # Global Carbon Budget 2025 Emissions Data Preprocessing +# +# Territorial fossil CO2 (`co2-ffi`) by country, 1850 to 2024, in MtCO2 per year. +# +# **Source:** Andrew and Peters (2025), Global Carbon Project fossil CO2 +# emissions dataset, version 2025v15, doi:10.5281/zenodo.17417124. Paper: +# Friedlingstein et al. (2026), doi:10.5194/essd-18-3211-2026. +# +# The transformation is `fair_shares.library.utils.data.gcb.gcb_fossil_co2`. +# It applies four rules. +# +# 1. **Years.** 1850 to 2024. The file starts in 1750; earlier years are dropped. +# 2. **Blank years count as zero.** The file leaves a year blank where it +# reports those emissions under another entity or reports none. Most blanks +# precede a country's first record. Some sit inside a series (Ireland 1851 +# to 1923, reported under the United Kingdom). The GCB global total treats +# every blank as zero, and so does this notebook. +# 3. **World row.** GCB global total minus international shipping (`XIS`), +# international aviation (`XIA`) and the Kuwaiti oil fires (478 MtCO2, 1991 +# only). These three belong to no country. They are saved to +# `emiss_co2-ffi_excluded_timeseries.csv`, so that for every year +# world row + excluded rows = GCB global total. +# 4. **Country rows.** A country row is an ISO3 code in the region mapping. +# Every other row stays in the world row and so lands in rest-of-world: +# - Kosovo (`KSV`) and Antarctica (`ATA`), which the region mapping lacks. +# - "Pacific Islands (Palau)" (1955 to 1991, 4.43 MtCO2 in total, at most +# 0.23 per year) and "Ryukyu Islands" (1965 to 1972, 0.82 MtCO2 in total, +# at most 0.13 per year). These rows have no ISO code. The official GCB +# workbook (National_Fossil_Carbon_Emissions_2025) keeps both in its World +# column and in no country column: its Palau column is blank before 1992 +# and its Japan column equals the flat-file Japan row without Ryukyu. +# +# The notebook also saves each country's first recorded year to +# `emiss_co2-ffi_first_recorded_year.csv`. After the zero-fill a blank and a +# reported zero look the same, so the coverage rule of this source reads that +# table: a country with no record before 1990, or without population from +# 1850, joins rest-of-world (`coverage` in `data_sources_unified.yaml`). +# +# **Output:** `intermediate/emissions/emiss_co2-ffi_timeseries.csv` + +# %% +import pandas as pd +import yaml +from pyprojroot import here + +from fair_shares.library.exceptions import ConfigurationError +from fair_shares.library.preprocessing import FIRST_RECORDED_YEAR_FILENAME +from fair_shares.library.utils import build_source_id +from fair_shares.library.utils.data.gcb import gcb_first_recorded_year, gcb_fossil_co2 + +# %% tags=["parameters"] +emission_category = None +active_target_source = None +active_emissions_source = None +active_gdp_source = None +active_population_source = None +active_gini_source = None +active_lulucf_source = None +source_id = None + +# %% +if emission_category is not None: + # Running via Papermill + if source_id is None: + source_id = build_source_id( + emissions=active_emissions_source, + gdp=active_gdp_source, + population=active_population_source, + gini=active_gini_source, + lulucf=active_lulucf_source, + target=active_target_source, + emission_category=emission_category, + ) + with open(here() / f"output/{source_id}/config.yaml") as f: + config = yaml.safe_load(f) +else: + # Running interactively + from fair_shares.library.utils.data.config import build_data_config + + emission_category = "co2-ffi" + active_emissions_source = "gcb-2025" + config, source_id = build_data_config( + emission_category, + { + "emissions": active_emissions_source, + "gdp": "wdi-2025", + "population": "un-owid-2025", + "gini": "wdi-2025", + "target": "rcbs", + }, + ) + config = config.model_dump() + +# %% +project_root = here() +emissions_config = config["emissions"][active_emissions_source] +world_key = emissions_config["data_parameters"]["world_key"] + +if emission_category != "co2-ffi": + raise ConfigurationError( + f"{active_emissions_source} provides co2-ffi only, got '{emission_category}'" + ) + +intermediate_dir = project_root / f"output/{source_id}/intermediate/emissions" +intermediate_dir.mkdir(parents=True, exist_ok=True) + +# %% [markdown] +# ## Load and process data + +# %% +raw = pd.read_csv(project_root / emissions_config["path"]) +region_mapping = pd.read_csv(project_root / config["general"]["region_mapping"]["path"]) + +countries = set(region_mapping["iso3c"]) +emissions, excluded = gcb_fossil_co2(raw, countries, world_key) +first_recorded_year = gcb_first_recorded_year(raw, countries) + +# %% +emissions.reset_index().to_csv( + intermediate_dir / "emiss_co2-ffi_timeseries.csv", index=False +) +excluded.reset_index().to_csv( + intermediate_dir / "emiss_co2-ffi_excluded_timeseries.csv", index=False +) +first_recorded_year.reset_index().to_csv( + intermediate_dir / FIRST_RECORDED_YEAR_FILENAME.format(category="co2-ffi"), + index=False, +) + +# %% [markdown] +# ## Summary + +# %% +world = emissions.xs(world_key, level="iso3c").iloc[0] +print(f"Country rows: {len(emissions) - 1}") +late = first_recorded_year[first_recorded_year >= 1990] +print(f"First record in 1990 or later: {late.to_dict()}") +print(f"Years: {emissions.columns[0]} to {emissions.columns[-1]}") +print(f"World row ({world_key}) in 2024: {world['2024']:.1f} MtCO2") +print("Excluded from the world row in 2024 (MtCO2):") +print(excluded["2024"].round(1).to_string()) diff --git a/notebooks/104_data_preprocess_scenarios_ar6.py b/notebooks/104_data_preprocess_scenarios_ar6.py index b407e69..147b159 100644 --- a/notebooks/104_data_preprocess_scenarios_ar6.py +++ b/notebooks/104_data_preprocess_scenarios_ar6.py @@ -60,6 +60,15 @@ process_iamc_zip, ) from fair_shares.library.preprocessing import emissions_path +from fair_shares.library.utils.data.rcb import ( + AR6_CATEGORY_BY_RCB_LABEL, + NET_ZERO_YEAR_COLUMN, + PEAK_WARMING_COLUMN, + build_rcb_scenario_sets, + convention_gap_from_baseline, + net_zero_year, +) +from fair_shares.library.utils.dataframes import normalize_metadata_column from fair_shares.library.utils.units import _clean_unit_string # %% tags=["parameters"] @@ -273,7 +282,9 @@ # The scenario label encodes temperature target + probability (e.g. "1.5p50"). # Used internally during processing; converted to clean (assessment, quantile) # format at output time via _SCENARIO_TO_ASSESSMENT / _SCENARIO_TO_QUANTILE. -_AR6_TO_SCENARIO = {"C1": "1.5p50", "C2": "2p83", "C3": "2p66"} +_AR6_TO_SCENARIO = { + category: label for label, category in AR6_CATEGORY_BY_RCB_LABEL.items() +} _AR6_CATEGORIES_TO_PROCESS = list(_AR6_TO_SCENARIO.keys()) desired_climate_assessments = list(_AR6_TO_SCENARIO.values()) @@ -298,7 +309,18 @@ # %% # Process the scenario source print(f"Processing {active_target_source} scenarios...") -df = process_iamc_zip(project_root / scenario_path) +df = process_iamc_zip( + project_root / scenario_path, metadata_columns=[PEAK_WARMING_COLUMN] +) + +# Scenario metadata, one row per scenario, for the RCB scenario sets. +_peak_warming_column = normalize_metadata_column(PEAK_WARMING_COLUMN) +scenario_metadata = ( + df[["Model", "Scenario", "Category", _peak_warming_column]] + .drop_duplicates() + .rename(columns={_peak_warming_column: PEAK_WARMING_COLUMN}) +) +df = df.drop(columns=[_peak_warming_column]) # %% [markdown] # ## Load historical emissions for harmonisation @@ -407,6 +429,9 @@ def extract_unit_from_data(var_df): # Rename 'Category' to 'climate-assessment' and relabel to RCB scenario labels df = df.rename(columns={"Category": "climate-assessment"}) +# The RCB scenario sets draw on every category (peak-warming bands span categories). +df_all_categories = df + # Filter to AR6 categories that have a corresponding RCB scenario print(f"Filtering to AR6 categories: {_AR6_CATEGORIES_TO_PROCESS}") df_before_filter = len(df) @@ -798,23 +823,78 @@ def _derive_category(minuend_key, subtrahend_key, new_category): print(f" non-co2 shape: {non_co2.shape}") # %% [markdown] -# ## Pre-compute RCB adjustment scalars per AR6 category +# ## Pre-compute RCB adjustment scalars per scenario set # -# For RCB-based allocations, the library needs per-AR6-category adjustment scalars -# to convert IPCC RCBs (BM convention) to NGHGI-consistent budgets. These are +# For RCB-based allocations, the library needs adjustment scalars to convert +# IPCC RCBs (BM convention) to NGHGI-consistent budgets. These are # pre-computed here from the Gidden scenario data so that the library never needs # to handle the Direct/Indirect AFOLU decomposition directly. # -# Scalars computed per AR6 category: +# Each budget in rcbs.yaml has one scenario set. `build_rcb_scenario_sets` +# selects it by the rule of the RCB source: an AR6 category (key = budget +# label, e.g. `1.5p50`) or a peak-warming band (key = temperature, e.g. +# `peak-warming-1.7C`). +# +# Scalars computed per scenario set: # - `bm_lulucf_cumulative_median`: median of per-scenario cumulative BM LULUCF # (= AFOLU|Direct), each integrated to its own NZ year -# - `convention_gap_median`: median of per-scenario cumulative convention gap -# (historical NGHGI-Direct + future Indirect), each integrated to its own NZ year +# - `convention_gap_median_from`: one entry per RCB baseline year. Each is the +# median of the per-scenario convention gap (observed NGHGI minus Direct up to +# the splice year, Indirect after it) from that baseline year to the +# scenario's own NZ year # - `nz_year_median`, `nz_year_min`, `nz_year_max`: net-zero year statistics -# - `n_scenarios`: number of scenarios in the category +# - `n_scenarios`: number of scenarios in the set # %% -print("\n--- Pre-computing RCB adjustment scalars per scenario category ---") +print("\n--- Pre-computing RCB adjustment scalars per scenario set ---") + +# World rows of the variables behind the adjustments, for every AR6 category. +rcb_scenario_data = { + var_name: df_all_categories[ + (df_all_categories["Variable"] == shorthand_variables[var_name]) + & (df_all_categories["Region"] == world_key) + ] + for var_name in ["CO2", "AFOLU_direct", "AFOLU_indirect"] +} + +# Net-zero year of every scenario: the first year with total CO2 at or below +# zero. The deductions end there, and the band rule keeps scenarios that have one. +_co2_year_cols = sorted( + [col for col in rcb_scenario_data["CO2"].columns if col.isdigit()], key=int +) +_net_zero_years = ( + rcb_scenario_data["CO2"] + .set_index(["Model", "Scenario"])[_co2_year_cols] + .apply(net_zero_year, axis=1) + .rename(NET_ZERO_YEAR_COLUMN) + .reset_index() +) +scenario_metadata = scenario_metadata.merge( + _net_zero_years, on=["Model", "Scenario"], how="left" +) + +# Select the scenario set of every budget in the RCB file of the active target. +# A target without an RCB file (pathway mode) needs no RCB adjustments. +_rcb_target = config.get("active_target_source", active_target_source) +_rcb_path = config["targets"].get(_rcb_target, {}).get("path") +if _rcb_path: + with open(project_root / _rcb_path) as f: + _rcb_yaml = yaml.safe_load(f) + rcb_scenario_sets = build_rcb_scenario_sets(_rcb_yaml, scenario_metadata) + # The convention gap of a budget starts at the baseline year of its source. + rcb_baseline_years = sorted( + {source["baseline_year"] for source in _rcb_yaml["rcb_data"].values()} + ) +else: + rcb_scenario_sets = {} + rcb_baseline_years = [] + print(f" Target '{_rcb_target}' has no RCB file: no RCB adjustments computed.") + + +def rows_of_scenario_set(var_df, scenario_set): + """Return the rows of var_df whose (Model, Scenario) is in the scenario set.""" + return var_df.merge(scenario_set[["Model", "Scenario"]], on=["Model", "Scenario"]) + # Load NGHGI world timeseries for historical convention gap computation. # Only needed for co2/all-ghg (NGHGI corrections); co2-ffi uses BM LULUCF @@ -834,43 +914,29 @@ def _derive_category(minuend_key, subtrahend_key, new_category): f" Convention gap will be zero (fine for co2-ffi; run notebook 107 for co2/all-ghg)." ) -# Compute per-scenario-category adjustment scalars +# Compute adjustment scalars per scenario set rcb_adjustments = {} -for ar6_cat in desired_climate_assessments: - # Filter to scenarios in this AR6 category - cat_direct = scenario_data["AFOLU_direct"][ - scenario_data["AFOLU_direct"]["climate-assessment"] == ar6_cat - ] - cat_total = scenario_data["CO2"][ - scenario_data["CO2"]["climate-assessment"] == ar6_cat - ] +for set_key, scenario_set in rcb_scenario_sets.items(): + cat_direct = rows_of_scenario_set(rcb_scenario_data["AFOLU_direct"], scenario_set) + cat_total = rows_of_scenario_set(rcb_scenario_data["CO2"], scenario_set) if cat_direct.empty or cat_total.empty: - print(f" {ar6_cat}: no data, skipping") - continue + raise DataProcessingError( + f"RCB scenario set '{set_key}': no CO2 or AFOLU|Direct data for its " + f"{len(scenario_set)} scenarios" + ) sample_df = next(iter(scenario_data.values())) year_cols = [col for col in sample_df.columns if col.isdigit()] sorted_year_cols = sorted(year_cols, key=int) - # --- Per-scenario NZ years (from total CO2 BM crossing zero) --- - total_indexed = cat_total.set_index(["Model", "Scenario"])[sorted_year_cols] - nz_years_dict = {} - for scenario_key in total_indexed.index: - row = total_indexed.loc[scenario_key] - for yc in sorted_year_cols: - if row[yc] <= 0: - nz_years_dict[scenario_key] = int(yc) - break - else: - nz_years_dict[scenario_key] = 2100 - - if not nz_years_dict: - print(f" {ar6_cat}: no NZ years computed, skipping") - continue - - nz_series = pd.Series(nz_years_dict, dtype=int) + # --- Per-scenario NZ years, from the metadata column that the band rule reads --- + reached_nz = scenario_set.set_index(["Model", "Scenario"])[ + NET_ZERO_YEAR_COLUMN + ].loc[cat_total.set_index(["Model", "Scenario"]).index] + # A scenario that never reaches net zero is integrated to 2100. + nz_series = reached_nz.fillna(2100).astype(int) # --- Per-scenario cumulative BM LULUCF (= Direct) --- direct_indexed = cat_direct.set_index(["Model", "Scenario"])[sorted_year_cols] @@ -891,67 +957,52 @@ def _derive_category(minuend_key, subtrahend_key, new_category): else 0.0 ) - # --- Per-scenario convention gap (historical NGHGI-Direct + future Indirect) --- - gap_median = 0.0 + # --- Per-scenario convention gap from each baseline year, then the median --- + gap_median_from = {year: 0.0 for year in rcb_baseline_years} if nghgi_world_available: - cat_indirect = scenario_data["AFOLU_indirect"][ - scenario_data["AFOLU_indirect"]["climate-assessment"] == ar6_cat - ] + cat_indirect = rows_of_scenario_set( + rcb_scenario_data["AFOLU_indirect"], scenario_set + ) indirect_indexed = cat_indirect.set_index(["Model", "Scenario"])[ sorted_year_cols ] common_all = common_scenarios.intersection(indirect_indexed.index) - - per_scenario_gaps = [] - for sk in common_all: - nz = nz_series[sk] - hist_end = min(nghgi_splice_year, nz) - - # Historical: NGHGI - Direct - from fair_shares.library.utils.data.nghgi import ( - compute_cumulative_emissions, - ) - - nghgi_hist = compute_cumulative_emissions(nghgi_world_df, 2020, hist_end) - hist_cols = [ - str(y) for y in range(2020, hist_end + 1) if str(y) in sorted_year_cols + nghgi_lulucf = nghgi_world_df.iloc[0] + + for baseline_year in rcb_baseline_years: + per_scenario_gaps = [ + convention_gap_from_baseline( + nghgi_lulucf=nghgi_lulucf, + bm_direct=direct_indexed.loc[sk], + indirect=indirect_indexed.loc[sk], + baseline_year=baseline_year, + nz_year=int(nz_series[sk]), + splice_year=nghgi_splice_year, + ) + for sk in common_all ] - direct_hist = ( - float(direct_indexed.loc[sk, hist_cols].sum()) if hist_cols else 0.0 - ) - hist_gap = nghgi_hist - direct_hist - - # Future: Indirect only (Direct cancels in the gap) - future_gap = 0.0 - if nz > nghgi_splice_year: - future_cols = [ - str(y) - for y in range(nghgi_splice_year + 1, nz + 1) - if str(y) in sorted_year_cols - ] - if future_cols: - future_gap = float(indirect_indexed.loc[sk, future_cols].sum()) - - per_scenario_gaps.append(hist_gap + future_gap) - - gap_median = ( - float(pd.Series(per_scenario_gaps).median()) if per_scenario_gaps else 0.0 - ) + if per_scenario_gaps: + gap_median_from[baseline_year] = float( + pd.Series(per_scenario_gaps).median() + ) # Store results - rcb_adjustments[ar6_cat] = { + rcb_adjustments[set_key] = { "bm_lulucf_cumulative_median": round(bm_median, 1), - "convention_gap_median": round(gap_median, 1), + "convention_gap_median_from": { + int(year): round(gap, 1) for year, gap in gap_median_from.items() + }, "nz_year_median": int(nz_series.median()), "nz_year_min": int(nz_series.min()), "nz_year_max": int(nz_series.max()), "n_scenarios": len(nz_series), - "n_reaching_nz": int((nz_series < 2100).sum()), + "n_reaching_nz": int(reached_nz.notna().sum()), } + gaps_text = ", ".join(f"{y}: {g:.0f}" for y, g in gap_median_from.items()) print( - f" {ar6_cat}: n={len(nz_series)}, NZ_med={int(nz_series.median())}, " - f"BM_LULUCF={bm_median:.0f} Mt, gap={gap_median:.0f} Mt" + f" {set_key}: n={len(nz_series)}, NZ_med={int(nz_series.median())}, " + f"BM_LULUCF={bm_median:.0f} Mt, gap from baseline year (Mt): {gaps_text}" ) # Save RCB adjustments as YAML @@ -963,11 +1014,11 @@ def _derive_category(minuend_key, subtrahend_key, new_category): # %% [markdown] -# ## Pre-compute baseline-shift LULUCF medians per AR6 category +# ## Pre-compute baseline-shift LULUCF medians per scenario set # # For RCB baseline-year shifting, `process_rcb_to_2020_baseline()` needs the -# year-by-year median AFOLU|Direct timeseries (BM LULUCF proxy) per AR6 -# category. Pre-computing these here avoids loading the raw Gidden Excel at +# year-by-year median AFOLU|Direct timeseries (BM LULUCF proxy) per scenario +# set. Pre-computing these here avoids loading the raw Gidden Excel at # runtime in the library. # %% @@ -976,28 +1027,22 @@ def _derive_category(minuend_key, subtrahend_key, new_category): sample_df = next(iter(scenario_data.values())) year_cols = sorted([col for col in sample_df.columns if col.isdigit()], key=int) -for ar6_cat in desired_climate_assessments: - cat_direct = scenario_data["AFOLU_direct"][ - scenario_data["AFOLU_direct"]["climate-assessment"] == ar6_cat - ] - - if cat_direct.empty: - print(f" {ar6_cat}: no AFOLU_direct data, skipping") - continue +for set_key, scenario_set in rcb_scenario_sets.items(): + cat_direct = rows_of_scenario_set(rcb_scenario_data["AFOLU_direct"], scenario_set) # Year-by-year median across scenarios median_vals = cat_direct[year_cols].median(axis=0) - source_label = f"gidden_direct_{ar6_cat}" + source_label = f"gidden_direct_{set_key}" median_df = pd.DataFrame( [median_vals.values], columns=year_cols, index=pd.Index([source_label], name="source"), ) - out_path = intermediate_dir / f"lulucf_shift_median_{ar6_cat}.csv" + out_path = intermediate_dir / f"lulucf_shift_median_{set_key}.csv" median_df.reset_index().to_csv(out_path, index=False) - print(f" {ar6_cat}: saved {len(cat_direct)} scenarios -> {out_path}") + print(f" {set_key}: saved {len(cat_direct)} scenarios -> {out_path}") # %% [markdown] # ## Output diff --git a/notebooks/106_generate_pathways_from_rcbs.py b/notebooks/106_generate_pathways_from_rcbs.py index 37f2224..23d9cdb 100644 --- a/notebooks/106_generate_pathways_from_rcbs.py +++ b/notebooks/106_generate_pathways_from_rcbs.py @@ -273,10 +273,13 @@ def _load_world_emissions( world_keys = ["EARTH", "World", "WLD", "OWID_WRL"] if emission_category == "co2": - # NGHGI-consistent world CO2 = fossil - bunkers + LULUCF(NGHGI) - # This mirrors run_rcb_preprocessing() lines 454-480 in orchestrator.py - from fair_shares.library.preprocessing.rcbs import _load_shared_timeseries - from fair_shares.library.utils.data.nghgi import build_nghgi_world_co2_timeseries + # NGHGI-consistent world CO2 = fossil (excluding bunkers) + LULUCF(NGHGI), + # as in run_rcb_preprocessing() + from fair_shares.library.preprocessing.rcbs import _resolve_template_path + from fair_shares.library.utils.data.nghgi import ( + build_nghgi_world_co2_timeseries, + load_world_co2_lulucf, + ) # Load component timeseries world_co2_ffi_df = _load_world_emissions( @@ -286,23 +289,23 @@ def _load_world_emissions( emissions_intermediate_dir, "co2-lulucf", world_keys, active_lulucf_source ) - # Load NGHGI LULUCF world timeseries and bunker timeseries - nghgi_ts, bunker_ts, splice_year = _load_shared_timeseries( - adjustments_config, project_root, source_id=source_id, verbose=True + # Load NGHGI LULUCF world timeseries + nghgi_ts, splice_year = load_world_co2_lulucf( + _resolve_template_path( + adjustments_config.lulucf_nghgi.path, source_id, project_root + ) ) # Build NGHGI-consistent world CO2 timeseries world_emissions_df = build_nghgi_world_co2_timeseries( fossil_ts=world_co2_ffi_df, nghgi_ts=nghgi_ts, - bunker_ts=bunker_ts, ) print("\nNGHGI-consistent world CO2 timeseries constructed:") - print(" fossil = co2-ffi (PRIMAP)") + print(" fossil = co2-ffi (PRIMAP), excluding international bunkers") print(f" LULUCF = NGHGI actual ({splice_year} end year, no BM splicing)") - print(" bunkers = international bunker fuel") - print(" Formula: total CO2 = fossil - bunkers + LULUCF") + print(" Formula: total CO2 = fossil + LULUCF") else: # co2-ffi: load directly @@ -323,29 +326,28 @@ def _load_world_emissions( # ## Process RCB data to 2020 baseline # # Uses `load_and_process_rcbs` — the same NGHGI-consistent pipeline as -# notebook 100. This ensures per-category net-zero years, BM LULUCF -# shift in the rebase, and the precautionary BM LULUCF cap are all applied. +# notebook 100. This ensures per-category net-zero years, the NGHGI LULUCF +# rebase, and the precautionary BM LULUCF cap are all applied. # %% from fair_shares.library.preprocessing.rcbs import load_and_process_rcbs -# Always pass PRIMAP fossil (co2-ffi) as world emissions; -# for total CO2, also pass BM LULUCF for rebase +# Always pass fossil (co2-ffi) as world emissions; +# for total CO2, also pass observed NGHGI LULUCF for the rebase if emission_category == "co2": _fossil_for_rcb = world_co2_ffi_df - _bm_lulucf_for_rcb = world_co2_lulucf_df + _nghgi_lulucf_for_rcb = world_co2_lulucf_df else: _fossil_for_rcb = world_emissions_df # already co2-ffi - _bm_lulucf_for_rcb = None + _nghgi_lulucf_for_rcb = None rcbs_df = load_and_process_rcbs( rcb_yaml_path=rcb_yaml_path, world_fossil_emissions=_fossil_for_rcb, emission_category=emission_category, adjustments_config=adjustments_config, - project_root=project_root, source_id=source_id, - actual_bm_lulucf_emissions=_bm_lulucf_for_rcb, + world_nghgi_lulucf_emissions=_nghgi_lulucf_for_rcb, verbose=True, ) diff --git a/notebooks/107_derive_nghgi_categories_melo-2026-v4.py b/notebooks/107_derive_nghgi_categories_melo-2026-v4.py new file mode 100644 index 0000000..a8fc2e0 --- /dev/null +++ b/notebooks/107_derive_nghgi_categories_melo-2026-v4.py @@ -0,0 +1,559 @@ +# --- +# jupyter: +# jupytext: +# cell_metadata_filter: tags,-all +# formats: ipynb,py:percent +# text_representation: +# extension: .py +# format_name: percent +# format_version: '1.3' +# jupytext_version: 1.16.6 +# kernelspec: +# display_name: Python 3 (ipykernel) +# language: python +# name: python3 +# --- + +# %% [markdown] +# # Derive NGHGI-Consistent Emission Categories (Melo v4.0.0) +# +# Derive a parallel set of NGHGI-consistent emission categories from the +# active bookkeeping-model (BM) source's FFI and non-LULUCF Kyoto primitives +# (typically PRIMAP, configurable via ``active_emissions_source``) plus the +# Melo et al. (2026) country-reported LULUCF CO2 timeseries. Outputs live +# alongside the BM originals under ``emiss__nghgi_timeseries.csv``. +# 187 country coverage, NGHGI years 2000-2024. +# +# ## Accounting invariant +# +# NGHGI and bookkeeping (BM) LULUCF use different accounting conventions, so +# this notebook does not splice them. Derived categories that include +# ``co2-lulucf`` are bounded to the Melo year range by construction (column +# intersection in ``_align_and_compute``). ``lulucf_metadata.yaml`` exports +# the NGHGI year bounds so downstream code can enforce them. +# +# ## Data flow +# +# **Inputs** (from notebook 101 and raw data): +# - `emiss_co2-ffi_timeseries.csv` — BM-source fossil CO2 (LULUCF-independent) +# - `emiss_all-ghg-ex-co2-lulucf_timeseries.csv` — BM-source all Kyoto +# excluding CO2-LULUCF +# - `timeseries_NGHGI_gap-filled_4.0.0.csv` — Melo v4.0.0 country-level NGHGI LULUCF +# +# **Outputs** (NGHGI-consistent, ``_nghgi`` suffix): +# - `emiss_co2-lulucf_nghgi_timeseries.csv` — Melo NGHGI (Melo years) +# - `emiss_co2_nghgi_timeseries.csv` — co2-ffi + co2-lulucf +# - `emiss_non-co2_nghgi_timeseries.csv` — all-ghg-ex-co2-lulucf − co2-ffi +# - `emiss_all-ghg_nghgi_timeseries.csv` — co2 + non-co2 +# - `emiss_all-ghg-ex-co2-lulucf_nghgi_timeseries.csv` — co2-ffi + non-co2 +# (recomputed; identical to the BM version by construction) +# +# The three non-CO2 outputs above are written only when the emissions source +# declares `all-ghg-ex-co2-lulucf`. A `co2-ffi`-only source gets the two CO2 files. +# +# - `world_co2-lulucf_timeseries.csv` — WRD LULUCF for RCB corrections +# - `lulucf_metadata.yaml` — NGHGI year bounds, affected files + +# %% [markdown] +# ## Set paths and library imports + +# %% +import matplotlib.pyplot as plt +import pandas as pd +import yaml +from pyprojroot import here + +from fair_shares.library.exceptions import ( + DataLoadingError, + DataProcessingError, +) +from fair_shares.library.utils import ( + build_source_id, + ensure_string_year_columns, +) + +BLUE = "#005baa" + +# %% tags=["parameters"] +emission_category = None +active_target_source = None +active_emissions_source = None +active_gdp_source = None +active_population_source = None +active_gini_source = None +active_lulucf_source = None +source_id = None + +# %% +_running_via_papermill = emission_category is not None + +if _running_via_papermill: + print("Running via Papermill") + + if source_id is None: + source_id = build_source_id( + emissions=active_emissions_source, + gdp=active_gdp_source, + population=active_population_source, + gini=active_gini_source, + lulucf=active_lulucf_source, + target=active_target_source, + emission_category=emission_category, + ) + + config_path = here() / f"output/{source_id}/config.yaml" + print(f"Loading config from: {config_path}") + with open(config_path) as f: + config = yaml.safe_load(f) +else: + print("Running interactively - build desired config") + + emission_category = "co2" + active_sources = { + "emissions": "primap-202503", + "gdp": "wdi-2025", + "population": "un-owid-2025", + "gini": "wdi-2025", + "lulucf": "melo-2026-v4", + "target": "rcbs", + } + + from fair_shares.library.utils.data.config import build_data_config + + config, source_id = build_data_config(emission_category, active_sources) + config = config.model_dump() + + active_target_source = active_sources["target"] + active_emissions_source = active_sources["emissions"] + active_gdp_source = active_sources["gdp"] + active_population_source = active_sources["population"] + active_gini_source = active_sources["gini"] + active_lulucf_source = active_sources["lulucf"] + +# %% [markdown] +# ## Prepare parameters + +# %% +project_root = here() +print(f"Project root: {project_root}") + +# Resolve active LULUCF source from config +if active_lulucf_source is None: + # Fallback: pick first available LULUCF source + lulucf_sources = config.get("lulucf", {}) + if lulucf_sources: + active_lulucf_source = next(iter(lulucf_sources)) + else: + raise ValueError("No LULUCF sources configured in data_sources_unified.yaml") + +lulucf_config = config["lulucf"][active_lulucf_source] +lulucf_path = lulucf_config["path"] +lulucf_params = lulucf_config["data_parameters"] + +# Paths +melo_path = project_root / lulucf_path +intermediate_dir_str = f"output/{source_id}/intermediate/emissions" +intermediate_dir = project_root / intermediate_dir_str +intermediate_dir.mkdir(parents=True, exist_ok=True) + +emissions_config = config["emissions"][active_emissions_source] +emissions_world_key = emissions_config["data_parameters"].get("world_key") +# The non-CO2 categories need all-ghg-ex-co2-lulucf from the emissions source. +has_non_co2_input = "all-ghg-ex-co2-lulucf" in emissions_config[ + "data_parameters" +].get("available_categories", []) + +print(f"Active LULUCF source: {active_lulucf_source}") +print(f"LULUCF data path: {melo_path}") +print(f"Intermediate directory: {intermediate_dir_str}") +print(f"Emissions world key: {emissions_world_key}") +if not has_non_co2_input: + print( + f"Emissions source '{active_emissions_source}' declares no " + "all-ghg-ex-co2-lulucf: writing co2-lulucf and co2 only, skipping " + "non-co2, all-ghg and all-ghg-ex-co2-lulucf." + ) + +# %% [markdown] +# ## Step 1: Load Melo v4.0.0 NGHGI LULUCF data +# +# The Melo dataset provides country-reported (NGHGI convention) LULUCF CO2 fluxes +# for 187 countries from 2000-2024. Values are in MtCO2/yr where negative = net sink. + +# %% +print("Loading Melo v4.0.0 NGHGI LULUCF data...") + +if not melo_path.exists(): + raise DataLoadingError(f"Melo LULUCF file not found: {melo_path}") + +melo_raw = pd.read_csv(melo_path) +print(f" Raw data shape: {melo_raw.shape}") +print(f" Columns: {list(melo_raw.columns)}") +print(f" Categories: {melo_raw['Category'].unique()}") +print(f" Year range: {melo_raw['Year'].min()}-{melo_raw['Year'].max()}") +print(f" Countries: {melo_raw['ISO3'].nunique()}") + +# %% +# Filter to LULUCF total, CO2 only, exclude EU27 (avoid double-counting) +melo_lulucf = melo_raw[ + (melo_raw["Category"] == "LULUCF") + & (melo_raw["Gas"] == "CO2") + & (~melo_raw["ISO3"].isin(["EU27"])) +].copy() + +print(f"Filtered LULUCF CO2 rows: {len(melo_lulucf)}") +print(f" Countries (incl WRD): {melo_lulucf['ISO3'].nunique()}") + +# Pivot from long to wide: rows = ISO3, columns = years +melo_wide = melo_lulucf.pivot_table( + index="ISO3", + columns="Year", + values="CFluxes_yr", + aggfunc="sum", +) + +# Rename index to match standard format +melo_wide.index.name = "iso3c" +melo_wide = ensure_string_year_columns(melo_wide) + +# Derive NGHGI year range from data +melo_years = sorted([int(c) for c in melo_wide.columns if c.isdigit()]) +nghgi_start_year = min(melo_years) +nghgi_end_year = max(melo_years) +melo_year_cols = [str(y) for y in melo_years] + +# Separate WRD and country data +wrd_mask = melo_wide.index == "WRD" +melo_countries_raw = melo_wide.loc[~wrd_mask].copy() +melo_wrd_raw = melo_wide.loc[wrd_mask].copy() + +if melo_wrd_raw.empty: + raise DataProcessingError("WRD (world total) not found in Melo data") + +# Map WRD to emissions_world_key for consistency with the active BM source +melo_wrd_raw.index = pd.Index([emissions_world_key], name="iso3c") +melo_all = pd.concat([melo_countries_raw, melo_wrd_raw]) + +# Add unit and emission-category to MultiIndex +melo_all.index = pd.MultiIndex.from_tuples( + [(iso3c, "Mt * CO2e", "co2-lulucf") for iso3c in melo_all.index], + names=["iso3c", "unit", "emission-category"], +) + +print(f" Melo wide shape: {melo_all.shape}") +print(f" NGHGI year range: {nghgi_start_year}-{nghgi_end_year}") +print(f" Countries: {len(melo_countries_raw)}") + +# %% [markdown] +# ## Step 2: Extract WRD world total for RCB corrections +# +# The world-level NGHGI LULUCF timeseries is needed by the RCB correction pipeline +# (rcbs.py) to compute historical LULUCF deductions. + +# %% +# Save WRD as world_co2-lulucf_timeseries.csv +# Format: single-row DataFrame with source index (matches expected loader format) +wrd_data = melo_all.loc[melo_all.index.get_level_values("iso3c") == emissions_world_key] +world_values = wrd_data.reset_index(level=["unit", "emission-category"], drop=True) +nghgi_world_output = pd.DataFrame( + world_values.values, + columns=world_values.columns, + index=pd.Index(["nghgi_lulucf"], name="source"), +) + +world_output_path = intermediate_dir / "world_co2-lulucf_timeseries.csv" +nghgi_world_output.reset_index().to_csv(world_output_path, index=False) +print(f"Saved WRD NGHGI LULUCF to: {world_output_path}") +print(f" NGHGI end year (splice year): {nghgi_end_year}") +print( + f" WRD LULUCF at {nghgi_end_year}: " + f"{nghgi_world_output[str(nghgi_end_year)].iloc[0]:.1f} MtCO2/yr" +) + +# %% [markdown] +# ## Step 3: Save NGHGI-consistent co2-lulucf (Melo years only) +# +# Output is pure Melo NGHGI data across Melo years. Categories derived from +# this file inherit the Melo year range via column intersection in +# ``_align_and_compute``. + +# %% +lulucf_output_path = intermediate_dir / "emiss_co2-lulucf_nghgi_timeseries.csv" +melo_all.reset_index().to_csv(lulucf_output_path, index=False) +print(f"Saved NGHGI co2-lulucf to: {lulucf_output_path}") +print(f" Year range: {nghgi_start_year}-{nghgi_end_year} (Melo NGHGI only)") +print(" NO BM splicing — pure NGHGI convention data") + +# Save NGHGI metadata for downstream use +nghgi_emissions_files = [ + "emiss_co2-lulucf_nghgi_timeseries.csv", + "emiss_co2_nghgi_timeseries.csv", +] +if has_non_co2_input: + nghgi_emissions_files += [ + "emiss_non-co2_nghgi_timeseries.csv", + "emiss_all-ghg_nghgi_timeseries.csv", + "emiss_all-ghg-ex-co2-lulucf_nghgi_timeseries.csv", + ] +nghgi_metadata = { + "nghgi_start_year": nghgi_start_year, + "nghgi_end_year": nghgi_end_year, + "splice_year": nghgi_end_year, + "source": "Melo et al. 2026, v4.0.0", + "n_countries": len(melo_countries_raw), + "nghgi_emissions_files": nghgi_emissions_files, +} +metadata_path = intermediate_dir / "lulucf_metadata.yaml" +with open(metadata_path, "w") as f: + yaml.dump(nghgi_metadata, f, default_flow_style=False) +print(f"Saved NGHGI metadata to: {metadata_path}") + +# %% [markdown] +# ## Step 4: Compute derived categories +# +# From the three primitives (co2-ffi, co2-lulucf, non-co2), we compute: +# - `co2 = co2-ffi + co2-lulucf` (Melo years only) +# - `non-co2 = all-ghg-ex-co2-lulucf - co2-ffi` (full BM-source year range) +# - `all-ghg = co2 + non-co2` (Melo years only, because co2 is Melo-bounded) +# - `all-ghg-ex-co2-lulucf = co2-ffi + non-co2` (full BM-source year range) + +# %% +# Load co2-ffi from notebook 101 output +co2_ffi_path = intermediate_dir / "emiss_co2-ffi_timeseries.csv" +if not co2_ffi_path.exists(): + raise DataLoadingError( + f"co2-ffi not found: {co2_ffi_path}. Run notebook 101 first." + ) + +co2_ffi = pd.read_csv(co2_ffi_path) +co2_ffi = co2_ffi.set_index(["iso3c", "unit", "emission-category"]) +co2_ffi = ensure_string_year_columns(co2_ffi) +print(f"Loaded co2-ffi: {co2_ffi.shape}") + +# Load all-ghg-ex-co2-lulucf from notebook 101 output (for non-co2 derivation) +if has_non_co2_input: + allghg_ex_path = intermediate_dir / "emiss_all-ghg-ex-co2-lulucf_timeseries.csv" + if not allghg_ex_path.exists(): + raise DataLoadingError( + f"all-ghg-ex-co2-lulucf not found: {allghg_ex_path}. " + "Run notebook 101 first." + ) + + allghg_ex = pd.read_csv(allghg_ex_path) + allghg_ex = allghg_ex.set_index(["iso3c", "unit", "emission-category"]) + allghg_ex = ensure_string_year_columns(allghg_ex) + print(f"Loaded all-ghg-ex-co2-lulucf: {allghg_ex.shape}") + + +# %% +# Helper: align two DataFrames on shared countries and years for arithmetic +def _align_and_compute(df1, df2, operation, new_category): + """Align two timeseries DataFrames and perform element-wise operation. + + Parameters + ---------- + df1, df2 : pd.DataFrame + Timeseries with MultiIndex (iso3c, unit, emission-category) + operation : str + "add" or "subtract" (df1 + df2 or df1 - df2) + new_category : str + emission-category label for the result + + Returns + ------- + pd.DataFrame + Result with MultiIndex (iso3c, unit, emission-category) + """ + # Strip emission-category for alignment + a = df1.reset_index(level="emission-category", drop=True) + b = df2.reset_index(level="emission-category", drop=True) + + # Align on shared years (intersection — never extend beyond either input) + shared_years = sorted(set(a.columns) & set(b.columns)) + a = a[shared_years] + b = b[shared_years] + + # Align on shared index (iso3c, unit) + shared_idx = a.index.intersection(b.index) + a = a.loc[shared_idx] + b = b.loc[shared_idx] + + if operation == "add": + result = a.add(b, fill_value=0) + elif operation == "subtract": + result = a.subtract(b, fill_value=0) + else: + raise ValueError(f"Unknown operation: {operation}") + + # Re-attach emission-category + result.index = pd.MultiIndex.from_tuples( + [(iso3c, unit, new_category) for iso3c, unit in result.index], + names=["iso3c", "unit", "emission-category"], + ) + + return result + + +def _save_and_report( + df: pd.DataFrame, path, label: str, *, bounded_by: str | None = None +) -> None: + """Write a derived category CSV and print a one-line summary.""" + df.reset_index().to_csv(path, index=False) + years = sorted(int(c) for c in df.columns if c.isdigit()) + bound_note = f" (bounded by {bounded_by})" if bounded_by else "" + print( + f" {label}: shape={df.shape}, " + f"years={min(years)}-{max(years)}{bound_note}, " + f"saved to {path.name}" + ) + + +# %% +# Compute derived categories. co2 and all-ghg inherit Melo year bounds via the +# column intersection inside _align_and_compute; non-co2 and all-ghg-ex retain +# the full BM-source range (no LULUCF dependency). +print("Computing co2 = co2-ffi + co2-lulucf...") +co2 = ensure_string_year_columns(_align_and_compute(co2_ffi, melo_all, "add", "co2")) +_save_and_report( + co2, + intermediate_dir / "emiss_co2_nghgi_timeseries.csv", + "co2", + bounded_by="Melo", +) + +if has_non_co2_input: + print("Computing non-co2 = all-ghg-ex-co2-lulucf - co2-ffi...") + non_co2 = ensure_string_year_columns( + _align_and_compute(allghg_ex, co2_ffi, "subtract", "non-co2") + ) + _save_and_report( + non_co2, + intermediate_dir / "emiss_non-co2_nghgi_timeseries.csv", + "non-co2", + ) + + print("Computing all-ghg = co2 + non-co2...") + all_ghg = ensure_string_year_columns( + _align_and_compute(co2, non_co2, "add", "all-ghg") + ) + _save_and_report( + all_ghg, + intermediate_dir / "emiss_all-ghg_nghgi_timeseries.csv", + "all-ghg", + bounded_by="Melo", + ) + + # all-ghg-ex-co2-lulucf has no LULUCF dependency; identical to the BM version + # by construction. Written for pipeline symmetry with the other _nghgi files. + print("Computing all-ghg-ex-co2-lulucf = co2-ffi + non-co2...") + allghg_ex_computed = ensure_string_year_columns( + _align_and_compute(co2_ffi, non_co2, "add", "all-ghg-ex-co2-lulucf") + ) + _save_and_report( + allghg_ex_computed, + intermediate_dir / "emiss_all-ghg-ex-co2-lulucf_nghgi_timeseries.csv", + "all-ghg-ex-co2-lulucf", + ) + +# %% [markdown] +# ## Diagnostics + +# %% +# BM-vs-NGHGI LULUCF at world level. BM co2-lulucf is derived implicitly as +# (BM co2 − co2-ffi) from the BM total CO2 file below; not every pipeline +# configuration produces a standalone BM emiss_co2-lulucf_timeseries.csv. +bm_co2_path = intermediate_dir / "emiss_co2_timeseries.csv" +print(f"\n=== BM ({active_emissions_source}) vs NGHGI LULUCF at world level ===") +if bm_co2_path.exists(): + bm_co2 = ensure_string_year_columns( + pd.read_csv(bm_co2_path).set_index(["iso3c", "unit", "emission-category"]) + ) + bm_world_co2 = bm_co2.loc[bm_co2.index.get_level_values("iso3c") == emissions_world_key] + ffi_world = co2_ffi.loc[co2_ffi.index.get_level_values("iso3c") == emissions_world_key] + shared = sorted( + set(bm_world_co2.columns) & set(ffi_world.columns) & set(melo_year_cols) + ) + print(f"{'year':>6} {'NGHGI':>12} {'BM (implied)':>15} {'diff':>10}") + for y in shared[-5:]: + nghgi_val = nghgi_world_output[y].iloc[0] + bm_implied = bm_world_co2[y].iloc[0] - ffi_world[y].iloc[0] + print(f"{y:>6} {nghgi_val:>12.1f} {bm_implied:>15.1f} {nghgi_val - bm_implied:>10.1f}") +else: + print(f"BM co2 file not found at {bm_co2_path}; skipping comparison.") + +# %% +# Coverage and country-sum vs WRD consistency check. +print(f"\n=== Coverage and country-sum vs WRD ===") +print(f"Melo countries: {len(melo_countries_raw)}") +melo_country_data = melo_all.loc[ + melo_all.index.get_level_values("iso3c") != emissions_world_key +] +country_sum = melo_country_data.groupby(level=["unit", "emission-category"]).sum() +print(f"{'year':>6} {'country_sum':>12} {'WRD':>12} {'diff':>10}") +for y in melo_year_cols[-5:]: + if y in country_sum.columns and y in nghgi_world_output.columns: + cs = country_sum[y].iloc[0] + wrd_val = nghgi_world_output[y].iloc[0] + print(f"{y:>6} {cs:>12.1f} {wrd_val:>12.1f} {cs - wrd_val:>10.1f}") + +# %% [markdown] +# ## Plots + +# %% +def _plot_world(ax, df, world_key, *, title, ylabel, label): + world_df = df.loc[df.index.get_level_values("iso3c") == world_key] + if not world_df.empty: + years_int = sorted(int(c) for c in world_df.columns if c.isdigit()) + vals = [world_df[str(y)].iloc[0] for y in years_int] + ax.plot(years_int, vals, label=label, color=BLUE, linewidth=2) + ax.legend() + ax.set_title(title, fontweight="bold") + ax.set_ylabel(ylabel) + ax.grid(True, alpha=0.3) + ax.axhline(y=0, color="black", linestyle="-", alpha=0.3) + + +# The non-CO2 row is drawn only when those categories were derived. +n_rows = 2 if has_non_co2_input else 1 +fig, axes = plt.subplots(n_rows, 2, figsize=(16, 6 * n_rows), squeeze=False) + +_plot_world( + axes[0, 0], melo_all, emissions_world_key, + title="co2-lulucf: World (NGHGI only)", + ylabel="MtCO2/yr", + label="NGHGI (Melo)", +) +_plot_world( + axes[0, 1], co2, emissions_world_key, + title="co2: World (FFI + LULUCF NGHGI)", + ylabel="MtCO2/yr", + label="co2 (NGHGI)", +) +if has_non_co2_input: + _plot_world( + axes[1, 0], non_co2, emissions_world_key, + title="non-co2: World (CH4 + N2O + F-gases)", + ylabel="MtCO2e/yr", + label="non-co2", + ) + _plot_world( + axes[1, 1], all_ghg, emissions_world_key, + title="all-ghg: World (co2 + non-co2, NGHGI)", + ylabel="MtCO2e/yr", + label="all-ghg (NGHGI)", + ) + +plt.tight_layout() +plt.show() + +print("\nLULUCF preprocessing complete.") +print(f"NGHGI year range: {nghgi_start_year}-{nghgi_end_year}") +if has_non_co2_input: + print( + "Categories produced: co2-ffi (unchanged), co2-lulucf (NGHGI), co2, " + "non-co2, all-ghg, all-ghg-ex-co2-lulucf" + ) +else: + print("Categories produced: co2-ffi (unchanged), co2-lulucf (NGHGI), co2") + +# %% diff --git a/notebooks/107_derive_nghgi_categories_melo-2026.py b/notebooks/107_derive_nghgi_categories_melo-2026.py index 0d1ce98..331eafb 100644 --- a/notebooks/107_derive_nghgi_categories_melo-2026.py +++ b/notebooks/107_derive_nghgi_categories_melo-2026.py @@ -22,7 +22,7 @@ # (typically PRIMAP, configurable via ``active_emissions_source``) plus the # Melo et al. (2026) country-reported LULUCF CO2 timeseries. Outputs live # alongside the BM originals under ``emiss__nghgi_timeseries.csv``. -# 187 country coverage, NGHGI years 2000-2023. +# 185 country coverage, NGHGI years 2000-2023. # # ## Accounting invariant # @@ -47,6 +47,10 @@ # - `emiss_all-ghg_nghgi_timeseries.csv` — co2 + non-co2 # - `emiss_all-ghg-ex-co2-lulucf_nghgi_timeseries.csv` — co2-ffi + non-co2 # (recomputed; identical to the BM version by construction) +# +# The three non-CO2 outputs above are written only when the emissions source +# declares `all-ghg-ex-co2-lulucf`. A `co2-ffi`-only source gets the two CO2 files. +# # - `world_co2-lulucf_timeseries.csv` — WRD LULUCF for RCB corrections # - `lulucf_metadata.yaml` — NGHGI year bounds, affected files @@ -154,17 +158,27 @@ emissions_config = config["emissions"][active_emissions_source] emissions_world_key = emissions_config["data_parameters"].get("world_key") +# The non-CO2 categories need all-ghg-ex-co2-lulucf from the emissions source. +has_non_co2_input = "all-ghg-ex-co2-lulucf" in emissions_config[ + "data_parameters" +].get("available_categories", []) print(f"Active LULUCF source: {active_lulucf_source}") print(f"LULUCF data path: {melo_path}") print(f"Intermediate directory: {intermediate_dir_str}") print(f"Emissions world key: {emissions_world_key}") +if not has_non_co2_input: + print( + f"Emissions source '{active_emissions_source}' declares no " + "all-ghg-ex-co2-lulucf: writing co2-lulucf and co2 only, skipping " + "non-co2, all-ghg and all-ghg-ex-co2-lulucf." + ) # %% [markdown] # ## Step 1: Load Melo v3.1 NGHGI LULUCF data # # The Melo dataset provides country-reported (NGHGI convention) LULUCF CO2 fluxes -# for 187 countries from 2000-2023. Values are in MtCO2/yr where negative = net sink. +# for 185 countries from 2000-2023. Values are in MtCO2/yr where negative = net sink. # %% print("Loading Melo v3.1 NGHGI LULUCF data...") @@ -271,19 +285,23 @@ print(" NO BM splicing — pure NGHGI convention data") # Save NGHGI metadata for downstream use +nghgi_emissions_files = [ + "emiss_co2-lulucf_nghgi_timeseries.csv", + "emiss_co2_nghgi_timeseries.csv", +] +if has_non_co2_input: + nghgi_emissions_files += [ + "emiss_non-co2_nghgi_timeseries.csv", + "emiss_all-ghg_nghgi_timeseries.csv", + "emiss_all-ghg-ex-co2-lulucf_nghgi_timeseries.csv", + ] nghgi_metadata = { "nghgi_start_year": nghgi_start_year, "nghgi_end_year": nghgi_end_year, "splice_year": nghgi_end_year, "source": "Melo et al. 2026, v3.1", "n_countries": len(melo_countries_raw), - "nghgi_emissions_files": [ - "emiss_co2-lulucf_nghgi_timeseries.csv", - "emiss_co2_nghgi_timeseries.csv", - "emiss_non-co2_nghgi_timeseries.csv", - "emiss_all-ghg_nghgi_timeseries.csv", - "emiss_all-ghg-ex-co2-lulucf_nghgi_timeseries.csv", - ], + "nghgi_emissions_files": nghgi_emissions_files, } metadata_path = intermediate_dir / "lulucf_metadata.yaml" with open(metadata_path, "w") as f: @@ -313,16 +331,18 @@ print(f"Loaded co2-ffi: {co2_ffi.shape}") # Load all-ghg-ex-co2-lulucf from notebook 101 output (for non-co2 derivation) -allghg_ex_path = intermediate_dir / "emiss_all-ghg-ex-co2-lulucf_timeseries.csv" -if not allghg_ex_path.exists(): - raise DataLoadingError( - f"all-ghg-ex-co2-lulucf not found: {allghg_ex_path}. Run notebook 101 first." - ) +if has_non_co2_input: + allghg_ex_path = intermediate_dir / "emiss_all-ghg-ex-co2-lulucf_timeseries.csv" + if not allghg_ex_path.exists(): + raise DataLoadingError( + f"all-ghg-ex-co2-lulucf not found: {allghg_ex_path}. " + "Run notebook 101 first." + ) -allghg_ex = pd.read_csv(allghg_ex_path) -allghg_ex = allghg_ex.set_index(["iso3c", "unit", "emission-category"]) -allghg_ex = ensure_string_year_columns(allghg_ex) -print(f"Loaded all-ghg-ex-co2-lulucf: {allghg_ex.shape}") + allghg_ex = pd.read_csv(allghg_ex_path) + allghg_ex = allghg_ex.set_index(["iso3c", "unit", "emission-category"]) + allghg_ex = ensure_string_year_columns(allghg_ex) + print(f"Loaded all-ghg-ex-co2-lulucf: {allghg_ex.shape}") # %% @@ -401,36 +421,39 @@ def _save_and_report( bounded_by="Melo", ) -print("Computing non-co2 = all-ghg-ex-co2-lulucf - co2-ffi...") -non_co2 = ensure_string_year_columns( - _align_and_compute(allghg_ex, co2_ffi, "subtract", "non-co2") -) -_save_and_report( - non_co2, - intermediate_dir / "emiss_non-co2_nghgi_timeseries.csv", - "non-co2", -) +if has_non_co2_input: + print("Computing non-co2 = all-ghg-ex-co2-lulucf - co2-ffi...") + non_co2 = ensure_string_year_columns( + _align_and_compute(allghg_ex, co2_ffi, "subtract", "non-co2") + ) + _save_and_report( + non_co2, + intermediate_dir / "emiss_non-co2_nghgi_timeseries.csv", + "non-co2", + ) -print("Computing all-ghg = co2 + non-co2...") -all_ghg = ensure_string_year_columns(_align_and_compute(co2, non_co2, "add", "all-ghg")) -_save_and_report( - all_ghg, - intermediate_dir / "emiss_all-ghg_nghgi_timeseries.csv", - "all-ghg", - bounded_by="Melo", -) + print("Computing all-ghg = co2 + non-co2...") + all_ghg = ensure_string_year_columns( + _align_and_compute(co2, non_co2, "add", "all-ghg") + ) + _save_and_report( + all_ghg, + intermediate_dir / "emiss_all-ghg_nghgi_timeseries.csv", + "all-ghg", + bounded_by="Melo", + ) -# all-ghg-ex-co2-lulucf has no LULUCF dependency; identical to the BM version -# by construction. Written for pipeline symmetry with the other _nghgi files. -print("Computing all-ghg-ex-co2-lulucf = co2-ffi + non-co2...") -allghg_ex_computed = ensure_string_year_columns( - _align_and_compute(co2_ffi, non_co2, "add", "all-ghg-ex-co2-lulucf") -) -_save_and_report( - allghg_ex_computed, - intermediate_dir / "emiss_all-ghg-ex-co2-lulucf_nghgi_timeseries.csv", - "all-ghg-ex-co2-lulucf", -) + # all-ghg-ex-co2-lulucf has no LULUCF dependency; identical to the BM version + # by construction. Written for pipeline symmetry with the other _nghgi files. + print("Computing all-ghg-ex-co2-lulucf = co2-ffi + non-co2...") + allghg_ex_computed = ensure_string_year_columns( + _align_and_compute(co2_ffi, non_co2, "add", "all-ghg-ex-co2-lulucf") + ) + _save_and_report( + allghg_ex_computed, + intermediate_dir / "emiss_all-ghg-ex-co2-lulucf_nghgi_timeseries.csv", + "all-ghg-ex-co2-lulucf", + ) # %% [markdown] # ## Diagnostics @@ -490,7 +513,9 @@ def _plot_world(ax, df, world_key, *, title, ylabel, label): ax.axhline(y=0, color="black", linestyle="-", alpha=0.3) -fig, axes = plt.subplots(2, 2, figsize=(16, 12)) +# The non-CO2 row is drawn only when those categories were derived. +n_rows = 2 if has_non_co2_input else 1 +fig, axes = plt.subplots(n_rows, 2, figsize=(16, 6 * n_rows), squeeze=False) _plot_world( axes[0, 0], melo_all, emissions_world_key, @@ -504,27 +529,31 @@ def _plot_world(ax, df, world_key, *, title, ylabel, label): ylabel="MtCO2/yr", label="co2 (NGHGI)", ) -_plot_world( - axes[1, 0], non_co2, emissions_world_key, - title="non-co2: World (CH4 + N2O + F-gases)", - ylabel="MtCO2e/yr", - label="non-co2", -) -_plot_world( - axes[1, 1], all_ghg, emissions_world_key, - title="all-ghg: World (co2 + non-co2, NGHGI)", - ylabel="MtCO2e/yr", - label="all-ghg (NGHGI)", -) +if has_non_co2_input: + _plot_world( + axes[1, 0], non_co2, emissions_world_key, + title="non-co2: World (CH4 + N2O + F-gases)", + ylabel="MtCO2e/yr", + label="non-co2", + ) + _plot_world( + axes[1, 1], all_ghg, emissions_world_key, + title="all-ghg: World (co2 + non-co2, NGHGI)", + ylabel="MtCO2e/yr", + label="all-ghg (NGHGI)", + ) plt.tight_layout() plt.show() print("\nLULUCF preprocessing complete.") print(f"NGHGI year range: {nghgi_start_year}-{nghgi_end_year}") -print( - "Categories produced: co2-ffi (unchanged), co2-lulucf (NGHGI), co2, " - "non-co2, all-ghg, all-ghg-ex-co2-lulucf" -) +if has_non_co2_input: + print( + "Categories produced: co2-ffi (unchanged), co2-lulucf (NGHGI), co2, " + "non-co2, all-ghg, all-ghg-ex-co2-lulucf" + ) +else: + print("Categories produced: co2-ffi (unchanged), co2-lulucf (NGHGI), co2") # %% diff --git a/notebooks/108_data_preprocess_bunkers_gcb-2025.py b/notebooks/108_data_preprocess_bunkers_gcb-2025.py new file mode 100644 index 0000000..3640264 --- /dev/null +++ b/notebooks/108_data_preprocess_bunkers_gcb-2025.py @@ -0,0 +1,73 @@ +# --- +# jupyter: +# jupytext: +# cell_metadata_filter: tags,-all +# formats: ipynb,py:percent +# text_representation: +# extension: .py +# format_name: percent +# format_version: '1.3' +# jupytext_version: 1.16.6 +# kernelspec: +# display_name: Python 3 (ipykernel) +# language: python +# name: python3 +# --- + +# %% [markdown] +# # International Bunker Fuel CO2 Preprocessing (GCB 2025) +# +# International bunkers are the sum of international shipping (`XIS`) and +# international aviation (`XIA`) in the Global Carbon Budget 2025 flat file, +# 1850 to 2024, in MtCO2 per year. Both series start in 1950; earlier years +# are blank in the file and count as zero. Bunker emissions are subtracted +# from global RCBs before country-level allocation. +# +# **Source:** Andrew and Peters (2025), doi:10.5281/zenodo.17417124 +# **Input:** data/emissions/gcb-2025/GCB2025v15_MtCO2_flat.csv +# **Output:** intermediate/emissions/bunker_timeseries.csv + +# %% +import pandas as pd +from pyprojroot import here + +from fair_shares.library.utils import build_source_id +from fair_shares.library.utils.data.gcb import gcb_bunkers + +# %% tags=["parameters"] +emission_category = None +active_target_source = None +active_emissions_source = None +active_gdp_source = None +active_population_source = None +active_gini_source = None +active_lulucf_source = None +source_id = None + +# %% +if emission_category is not None and source_id is None: + source_id = build_source_id( + emissions=active_emissions_source, + gdp=active_gdp_source, + population=active_population_source, + gini=active_gini_source, + lulucf=active_lulucf_source, + target=active_target_source, + emission_category=emission_category, + ) +elif source_id is None: + source_id = "gcb-2025_wdi-2025_un-owid-2025_wdi-2025_rcbs_co2-ffi" + +project_root = here() +intermediate_dir = project_root / f"output/{source_id}/intermediate/emissions" +intermediate_dir.mkdir(parents=True, exist_ok=True) + +# %% +raw = pd.read_csv(project_root / "data/emissions/gcb-2025/GCB2025v15_MtCO2_flat.csv") +bunkers = gcb_bunkers(raw) + +output_path = intermediate_dir / "bunker_timeseries.csv" +bunkers.reset_index().to_csv(output_path, index=False) +print(f"Saved bunker timeseries to: {output_path}") +print(f" Year range: {bunkers.columns[0]}-{bunkers.columns[-1]}") +print(f" Last year value: {bunkers.iloc[0, -1]:.1f} MtCO2/yr") diff --git a/notebooks/602_reproduce_dekker_2025.py b/notebooks/602_reproduce_dekker_2025.py index 1b5b831..f34f496 100644 --- a/notebooks/602_reproduce_dekker_2025.py +++ b/notebooks/602_reproduce_dekker_2025.py @@ -407,9 +407,7 @@ ecpc_allocations = { "equal-per-capita-budget": [ { - # TODO: Dekker uses 1850 as a historical start year, but fair-shares - # validates allocation_year >= 1900. Relaxing this requires reviewing - # data availability (emissions, population) pre-1900. + # Dekker's historical start year is 1850; this notebook runs 1950 and 1990. "allocation_year": [1950, 1990], "cumulative_end_year": [2050], "preserve_allocation_year_shares": [False], diff --git a/notebooks/604_reproduce_vandenberg_2020.py b/notebooks/604_reproduce_vandenberg_2020.py index 104e5d4..232e13d 100644 --- a/notebooks/604_reproduce_vandenberg_2020.py +++ b/notebooks/604_reproduce_vandenberg_2020.py @@ -432,9 +432,7 @@ ecpc_allocations = { "equal-per-capita-budget": [ { - # TODO: Paper uses 1850 as a historical start year, but fair-shares - # validates allocation_year >= 1900. Relaxing this requires reviewing - # data availability (emissions, population) pre-1900. + # The paper's historical start year is 1850; this notebook runs 1970 and 1990. "allocation_year": [1970, 1990], "cumulative_end_year": [2100], "preserve_allocation_year_shares": [False], diff --git a/notebooks/docs/parameter_effects_examples.py b/notebooks/docs/parameter_effects_examples.py index 007cb9a..de12617 100644 --- a/notebooks/docs/parameter_effects_examples.py +++ b/notebooks/docs/parameter_effects_examples.py @@ -114,13 +114,13 @@ # Calculate carbon budget budget = calculate_budget_from_rcb( rcb_value=rcb_value, - allocation_year=2025, + allocation_year=2020, world_scenario_emissions_ts=world_emissions_df, verbose=False, ) -print(f"RCB value (2020): {rcb_value / 1e9:.1f} GtCO2") -print(f"Global carbon budget (2025): {budget / 1e9:.1f} GtCO2") +print(f"RCB value (2020): {rcb_value / 1e3:.1f} GtCO2") # values are in Mt +print(f"Global carbon budget (2020): {budget / 1e3:.1f} GtCO2") print("Data loaded successfully!") # %% [markdown] diff --git a/src/fair_shares/conf/data_registry.yaml b/src/fair_shares/conf/data_registry.yaml index b334af3..5a75b46 100644 --- a/src/fair_shares/conf/data_registry.yaml +++ b/src/fair_shares/conf/data_registry.yaml @@ -100,6 +100,27 @@ sources: upstream_hash: "md5:19482df604f1dc746fb354ef66ef9047" size_bytes: 161831 + gcb-2025: + tier: optional + version: "2025v15" + doi: "10.5281/zenodo.17417124" + license: "CC-BY-4.0" + redistributable: true + citation: >- + Andrew, R. M., & Peters, G. P. (2025). The Global Carbon Project's fossil + CO2 emissions dataset (2025v15). Zenodo. + https://doi.org/10.5281/zenodo.17417124. Paper: Friedlingstein, P., et + al. (2026). Global Carbon Budget 2025. Earth System Science Data, 18, + 3211-3288. https://doi.org/10.5194/essd-18-3211-2026 + # Opt-in alternative to primap-202503, which stays the default. Fossil CO2 + # only. The same file supplies the bunker series for gcb-2025 runs. + downloads: + - url: "https://zenodo.org/api/records/17417124/files/GCB2025v15_MtCO2_flat.csv/content" + target: "emissions/gcb-2025/GCB2025v15_MtCO2_flat.csv" + sha256: "20650c19b394d91b6b31cddff2fbc9508bcdbbed7c3f93660336f2f655f367ff" + upstream_hash: "md5:3008e30d913af5926a83d0d0775fb72e" + size_bytes: 3147221 + # --- LULUCF ---------------------------------------------------------------- melo-2026: @@ -135,6 +156,29 @@ sources: upstream_hash: "md5:702b46bfe67cca47264fdf272d312005" size_bytes: 1180037 + melo-2026-v4: + tier: optional + version: "4.0.0 (2026 NGHGI release), gap-filled" + doi: "10.5281/zenodo.22828743" + license: "CC-BY-4.0" + redistributable: true + citation: >- + Melo, J., Rossi, S., Achard, F., Alkama, R., Canadell, J. G., Federici, S., + Friedlingstein, P., Gibbs, D., Harris, N., Heinrich, V., O'Sullivan, M., + Peters, G., Pongratz, J., Rose, M., Roman-Cuesta, R., Sanz Sanchez, M. J., + Schwingshackl, C., Sitch, S., & Grassi, G. (2026). The LULUCF Data Hub: + translating global land use emissions estimates into the national GHG + inventory framework (Version 4.0.0, 2026 NGHGI release). Zenodo. + https://doi.org/10.5281/zenodo.22828743 + # Opt-in alternative to melo-2026 (v3.1.1), which stays the default. + # Upstream ships LF line endings, so the pin matches the committed bytes. + downloads: + - url: "https://zenodo.org/api/records/22828743/files/timeseries_NGHGI_gap-filled_4.0.0.csv/content" + target: "lulucf/melo-2026-v4/timeseries_NGHGI_gap-filled_4.0.0.csv" + sha256: "82bb42999e4ed1c60dce3f5e4fc5b194aee0703f79751956fbe89b9beccc1da5" + upstream_hash: "md5:4dc0776f52dfefe1df76538794ab2bef" + size_bytes: 1220236 + # --- GDP ------------------------------------------------------------------- wdi-2025: @@ -144,7 +188,7 @@ sources: redistributable: true citation: >- World Bank. World Development Indicators: GDP (constant 2015 US$, - NY.GDP.MKTP.KD), GDP PPP (constant 2017 international $, + NY.GDP.MKTP.KD), GDP PPP (constant 2021 international $, NY.GDP.MKTP.PP.KD) and Gini index (SI.POV.GINI, sourced by the World Bank from its Poverty and Inequality Platform), 2025 export. https://data.worldbank.org/ @@ -348,12 +392,16 @@ sources: remaining carbon budgets. Nature Climate Change, 13, 1360-1367. https://doi.org/10.1038/s41558-023-01848-5; Forster, P. M., et al. (2024). Indicators of Global Climate Change 2023. Earth System Science Data, 16, - 2625-2658. https://doi.org/10.5194/essd-16-2625-2024; and IPCC AR6 WGI + 2625-2658. https://doi.org/10.5194/essd-16-2625-2024; Forster, P. M., et + al. (2026). Indicators of Global Climate Change 2025. Earth System Science + Data, 18, 3889-3933. https://doi.org/10.5194/essd-18-3889-2026; IPCC + (2021). Summary for Policymakers. In Climate Change 2021: The Physical + Science Basis. https://doi.org/10.1017/9781009157896.001; and IPCC AR6 WGI (https://www.ipcc.ch/report/ar6/wg1/). Net-zero years per AR6 scenario are derived from the IPCC AR6 Scenarios Database. downloads: - target: "rcbs/rcbs.yaml" - sha256: "22f719d8cd18e89639cf7f244caec43b502dd504b33105b33c648f30ad1777b4" + sha256: "ba810a80fd83d18773bd5424e934765788d3462286db695d28e33f33169d9c06" - target: "rcbs/ar6_scenario_nz_years.parquet" sha256: "3df82c7ecfcf7b80782d666093d117e204f75e5e74f0c5ae1f22de80336a8ffc" diff --git a/src/fair_shares/conf/data_sources/data_sources_unified.yaml b/src/fair_shares/conf/data_sources/data_sources_unified.yaml index 4628d0f..bd9b974 100644 --- a/src/fair_shares/conf/data_sources/data_sources_unified.yaml +++ b/src/fair_shares/conf/data_sources/data_sources_unified.yaml @@ -24,6 +24,18 @@ lulucf: iso3_column: "ISO3" year_column: "Year" + melo-2026-v4: + path: "data/lulucf/melo-2026-v4/timeseries_NGHGI_gap-filled_4.0.0.csv" + data_parameters: + world_key: "WRD" + format: "long" + category_filter: "LULUCF" + gas_filter: "CO2" + exclude_regions: ["EU27"] + value_column: "CFluxes_yr" + iso3_column: "ISO3" + year_column: "Year" + # --- Historical emissions --------------------------------------------------- emissions: primap-202503: @@ -34,6 +46,17 @@ emissions: world_key: "EARTH" scenario: "HISTCR" # "HISTCR" for country-reported (can use "HISTTP" for third party) + # Opt-in. Fossil CO2 only. The world row excludes international bunkers and + # the 1991 Kuwaiti oil fires; bunkers come from the same file. + gcb-2025: + path: "data/emissions/gcb-2025/GCB2025v15_MtCO2_flat.csv" + data_parameters: + available_categories: [co2-ffi] + world_key: "WLD" + bunkers_source: "gcb-2025" + # An analysis country needs an emissions record before 1990 and population from 1850; the others join rest-of-world. + coverage: { emissions_recorded_before: 1990, population_from: 1850 } + # --- GDP --------------------------------------------------------------------- gdp: wdi-2025: diff --git a/src/fair_shares/library/allocations/budgets/per_capita.py b/src/fair_shares/library/allocations/budgets/per_capita.py index c991345..6bf7539 100644 --- a/src/fair_shares/library/allocations/budgets/per_capita.py +++ b/src/fair_shares/library/allocations/budgets/per_capita.py @@ -141,7 +141,9 @@ def _per_capita_budget_core( before OR after the allocation year. When ``capability_reference_year < allocation_year``, the snapshot is sourced from the full unfiltered ``gdp_ts`` and ``population_ts`` - inputs; Gini adjustment is NOT applied in that case. + inputs; the Gini adjustment applies to it as in every other case. + With a reference year set, GDP is required at that year only, so + ``allocation_year`` may precede the first GDP year. Ignored if ``capability_weight == 0.0``. Emits a ``UserWarning`` if ``capability_reference_year`` exceeds the last observed GDP year. income_floor @@ -201,6 +203,7 @@ def _per_capita_budget_core( pre_allocation_responsibility_year=pre_allocation_responsibility_year if pre_allocation_responsibility_weight > 0 else None, + capability_reference_year=capability_reference_year, ) # Validate data requirements @@ -366,7 +369,7 @@ def _per_capita_budget_core( ref_label = [c for c in capability_metric_common.columns if int(c) == ref_year][0] snapshot = capability_metric_common[ref_label] - elif ref_year > max(gdp_filtered_years): + elif gdp_filtered_years and ref_year > max(gdp_filtered_years): # User-requested ffill past end-of-series: take the last column. warnings.warn( f"capability_reference_year={ref_year} is beyond the last " @@ -379,8 +382,8 @@ def _per_capita_budget_core( ref_label = capability_metric_common.columns[-1] snapshot = capability_metric_common[ref_label] else: - # ref_year < allocation_year: the reference year was stripped by - # filter_time_columns, so we must source the snapshot from the + # ref_year is outside the window (before allocation_year, or the + # window holds no GDP year), so we must source the snapshot from the # UNFILTERED gdp_ts and population_ts inputs. # Apply the same unit processing used in the main capability path. gdp_full_single_unit = set_single_unit(gdp_ts, unit_level, ur=ur) @@ -853,7 +856,8 @@ def per_capita_adjusted_budget( **Pre-allocation responsibility.** Historical emissions data. Required when ``pre_allocation_responsibility_weight > 0``. gdp_ts - **Capability.** GDP data used from ``allocation_year`` onwards. + **Capability.** GDP data used from ``allocation_year`` onwards, + or at ``capability_reference_year`` only when that is set. Required when ``capability_weight > 0``. pre_allocation_responsibility_weight **Pre-allocation responsibility.** Relative weight (0–1). Only the @@ -892,8 +896,8 @@ def per_capita_adjusted_budget( **Capability.** When ``None`` (default), capability is computed year-by-year. When set to an integer, GDP from that single year is broadcast across the allocation window. May be before or - after ``allocation_year``. - When before, Gini adjustment is NOT applied to the snapshot. + after ``allocation_year``. GDP is then required at that year + only, so ``allocation_year`` may precede the first GDP year. Ignored when ``capability_weight == 0``. max_deviation_sigma **Constraint.** Maximum allowed deviation from equal per capita, @@ -1098,11 +1102,9 @@ def per_capita_adjusted_gini_budget( By default (``capability_reference_year=None``), $C_{\text{Gini}}(g, t)$ is computed year-by-year. Setting ``capability_reference_year`` to an integer freezes capability at that year: $C_{\text{Gini}}(g, t) \equiv - C_{\text{Gini}}(g, t_{\text{ref}})$ for all $t$ in the window. Note that - when ``capability_reference_year < allocation_year``, the Gini adjustment - is NOT applied to the snapshot (see ``capability_reference_year`` in the - Parameters section and ``_per_capita_budget_core`` for the implementation - detail). + C_{\text{Gini}}(g, t_{\text{ref}})$ for all $t$ in the window. The Gini + adjustment applies to the snapshot wherever the reference year lies, + including when ``capability_reference_year < allocation_year``. Two allocation modes are supported based on :code:`preserve_allocation_year_shares`: @@ -1117,7 +1119,8 @@ def per_capita_adjusted_gini_budget( Population time series for each group of interest. gdp_ts **Capability.** GDP time series (required). Used from - ``allocation_year`` onwards for capability calculations. + ``allocation_year`` onwards for capability calculations, or at + ``capability_reference_year`` only when that is set. gini_s **Gini.** Gini coefficients for within-country income inequality (required). Used to adjust GDP before computing the capability @@ -1164,9 +1167,10 @@ def per_capita_adjusted_gini_budget( capability_reference_year **Capability.** When ``None`` (default), capability is computed year-by-year. When set to an integer, GDP from that single year - is broadcast across the allocation window. When before - ``allocation_year``, Gini - adjustment is NOT applied to the snapshot. + is broadcast across the allocation window. May be before or + after ``allocation_year``; the Gini adjustment applies to the + snapshot in both cases. GDP is then required at that year only, + so ``allocation_year`` may precede the first GDP year. Ignored when ``capability_weight == 0``. income_floor **Gini.** Development threshold in USD PPP per capita. Income diff --git a/src/fair_shares/library/allocations/manager.py b/src/fair_shares/library/allocations/manager.py index 99783ae..c2e6861 100644 --- a/src/fair_shares/library/allocations/manager.py +++ b/src/fair_shares/library/allocations/manager.py @@ -53,6 +53,7 @@ from fair_shares.library.utils.dataframes import ( TimeseriesDataFrame, filter_function_parameters, + get_year_columns, ) from fair_shares.library.utils.io import create_param_manifest as _create_param_manifest from fair_shares.library.utils.io import generate_readme as _generate_readme @@ -605,9 +606,20 @@ def run_parameter_grid( allocations_config, harmonisation_year, target_source ) - # Validate allocation year >= 1990 for total CO2 (NGHGI data limit) + # Validate allocation year >= 2000 for total CO2 (NGHGI data limit) if emission_category: - validate_allocation_year_for_co2(allocations_config, emission_category) + # A separate responsibility frame (fossil CO2 for co2 and all-ghg) sets + # the first year a pre-allocation responsibility window may start. + responsibility_min_year = None + if responsibility_emissions_ts is not None: + responsibility_min_year = min( + get_year_columns(responsibility_emissions_ts, return_type="int") + ) + validate_allocation_year_for_co2( + allocations_config, + emission_category, + responsibility_min_year=responsibility_min_year, + ) for approach, params_list in allocations_config.items(): print(f"\nProcessing approach: {approach}") diff --git a/src/fair_shares/library/citations.py b/src/fair_shares/library/citations.py index 4aef682..6a176b5 100644 --- a/src/fair_shares/library/citations.py +++ b/src/fair_shares/library/citations.py @@ -139,7 +139,9 @@ def resolve_source_names( target = active_sources.get("target") if target in BUNKER_TARGETS: - names += ["rcbs", "gcb-2024"] + from fair_shares.library.utils.data.config import get_bunkers_source + + names += ["rcbs", get_bunkers_source(active_sources.get("emissions"))] if _uses_scenarios(target, emission_category): scenario = active_sources.get("scenario", "ar6") diff --git a/src/fair_shares/library/config/models.py b/src/fair_shares/library/config/models.py index c258d42..6c994d5 100644 --- a/src/fair_shares/library/config/models.py +++ b/src/fair_shares/library/config/models.py @@ -14,6 +14,29 @@ from fair_shares.library.utils.dataframes import validate_path_exists +class EmissionsCoverage(BaseModel, extra="forbid"): + """Opt-in rule for which countries are analysis countries. + + A country that fails a test joins rest-of-world. The rule is applied in + ``fair_shares.library.preprocessing.coverage.coverage_exclusions``. + """ + + emissions_recorded_before: int | None = Field( + None, + description=( + "The raw emissions file must hold a value for the country in a " + "year before this one. Zero-filled blanks do not count." + ), + ) + population_from: int | None = Field( + None, + description=( + "The population series must hold a value for every year from this " + "one through its last year." + ), + ) + + class EmissionsDataParameters(BaseModel): """Parameters for emissions data source.""" @@ -21,7 +44,23 @@ class EmissionsDataParameters(BaseModel): ..., description="Available emission categories in this dataset" ) world_key: str = Field(..., description="Key identifier for world/global data") - scenario: str = Field(..., description="Historical scenario identifier") + scenario: str | None = Field( + None, description="Historical scenario identifier, for sources that have one" + ) + bunkers_source: str | None = Field( + None, + description=( + "Registry name of the international bunker data paired with this " + "source. Omit to use the default (gcb-2024)." + ), + ) + coverage: EmissionsCoverage | None = Field( + None, + description=( + "Opt-in coverage rule for analysis countries. Omit to keep the " + "default completeness check alone." + ), + ) class EmissionsSourceConfig(BaseModel): @@ -499,11 +538,20 @@ def validate_emission_category(self) -> DataSourcesConfig: Note: by the time the Pydantic model is constructed, the emission_category has already been resolved to the effective value (e.g., "co2-ffi" for all-ghg runs), so this validator sees the resolved value. + + ``co2`` is also valid when the emissions source declares ``co2-ffi`` and + a LULUCF source is active: the LULUCF notebook derives ``co2`` as + ``co2-ffi`` plus national-inventory LULUCF. """ if self.active_emissions_source: emissions_config = self.emissions[self.active_emissions_source] available = emissions_config.data_parameters.available_categories - if self.emission_category not in available: + co2_from_lulucf = ( + self.emission_category == "co2" + and "co2-ffi" in available + and bool(self.active_lulucf_source) + ) + if self.emission_category not in available and not co2_from_lulucf: suggestion = suggest_similar(self.emission_category, available) msg = format_error( "invalid_emission_category", @@ -515,5 +563,10 @@ def validate_emission_category(self) -> DataSourcesConfig: f"'{self.active_emissions_source}' " f"only provides: {', '.join(available)}" ) + if self.emission_category == "co2" and "co2-ffi" in available: + note += ( + "\n'co2' can be derived from 'co2-ffi' when a LULUCF " + "source is active (for example lulucf='melo-2026-v4')." + ) raise ConfigurationError(msg + note) return self diff --git a/src/fair_shares/library/pipeline/preprocessing.py b/src/fair_shares/library/pipeline/preprocessing.py index ba5869a..543f9ff 100644 --- a/src/fair_shares/library/pipeline/preprocessing.py +++ b/src/fair_shares/library/pipeline/preprocessing.py @@ -15,6 +15,7 @@ from ..paths import data_dir as resolve_data_dir from ..paths import output_dir as resolve_output_dir from ..paths import resolve_source_path +from ..preprocessing.coverage import source_coverage_exclusions from ..preprocessing.gini import complete_gini, gini_missing_policy from ..preprocessing.loaders import ( load_emissions_data as _load_emissions, @@ -241,7 +242,14 @@ def determine_analysis_countries( country_iso3c = complete_sets[0].intersection(*complete_sets[1:]) - return country_iso3c + # Opt-in coverage rule of the emissions source: failing countries join ROW. + excluded = source_coverage_exclusions( + country_iso3c, + population, + self.config["emissions"][self.active_emissions_source]["data_parameters"], + self.emiss_intermediate_dir, + ) + return country_iso3c - set(excluded) def save_processed_data( self, @@ -477,26 +485,29 @@ def run_rcb_preprocessing( if emission_category == "co2": # Build AdjustmentsConfig for loading shared timeseries from ..config.models import AdjustmentsConfig - from ..preprocessing.rcbs import _load_shared_timeseries - from ..utils.data.nghgi import build_nghgi_world_co2_timeseries + from ..preprocessing.rcbs import _resolve_template_path + from ..utils.data.nghgi import ( + build_nghgi_world_co2_timeseries, + load_world_co2_lulucf, + ) rcb_config = config["targets"]["rcbs"] rcb_data_params = rcb_config.get("data_parameters", {}) rcb_adjustments_raw = rcb_data_params.get("adjustments", {}) adjustments_config = AdjustmentsConfig.model_validate(rcb_adjustments_raw) - nghgi_ts, bunker_ts, _splice_year = _load_shared_timeseries( - adjustments_config, - orch.data_dir, - source_id=source_id, - verbose=True, - output_dir=orch.output_dir, + nghgi_ts, _ = load_world_co2_lulucf( + _resolve_template_path( + adjustments_config.lulucf_nghgi.path, + source_id, + orch.data_dir, + orch.output_dir, + ) ) world_emiss["co2"] = build_nghgi_world_co2_timeseries( fossil_ts=world_emiss["co2-ffi"], nghgi_ts=nghgi_ts, - bunker_ts=bunker_ts, ) # Save processed data @@ -509,12 +520,12 @@ def run_rcb_preprocessing( ) # Process and save RCB data — always pass fossil (co2-ffi) emissions; - # for total CO2, also pass BM LULUCF for rebase inside load_and_process_rcbs + # for total CO2, also pass NGHGI LULUCF for rebase inside load_and_process_rcbs _process_and_save_rcbs( orch, config, world_fossil_emissions=world_emiss["co2-ffi"], - actual_bm_lulucf_emissions=world_emiss.get("co2-lulucf"), + world_nghgi_lulucf_emissions=world_emiss.get("co2-lulucf"), ) @@ -565,7 +576,7 @@ def _process_and_save_rcbs( orch: DataPreprocessor, config: dict[str, Any], world_fossil_emissions: pd.DataFrame, - actual_bm_lulucf_emissions: pd.DataFrame | None = None, + world_nghgi_lulucf_emissions: pd.DataFrame | None = None, ) -> None: """Process RCB data and save to CSV. @@ -576,7 +587,8 @@ def _process_and_save_rcbs( orch: Orchestrator instance config: Configuration dict world_fossil_emissions: Fossil world emissions, co2-ffi (e.g. PRIMAP) - actual_bm_lulucf_emissions: BM LULUCF world emissions for co2 rebase + world_nghgi_lulucf_emissions: Observed NGHGI LULUCF world emissions + for the co2 rebase """ from ..config.models import AdjustmentsConfig from ..preprocessing.rcbs import load_and_process_rcbs @@ -599,7 +611,7 @@ def _process_and_save_rcbs( adjustments_config=adjustments_config, data_dir=orch.data_dir, source_id=orch.source_id, - actual_bm_lulucf_emissions=actual_bm_lulucf_emissions, + world_nghgi_lulucf_emissions=world_nghgi_lulucf_emissions, verbose=False, output_dir=orch.output_dir, ) diff --git a/src/fair_shares/library/preprocessing/__init__.py b/src/fair_shares/library/preprocessing/__init__.py index 8832c33..0378407 100644 --- a/src/fair_shares/library/preprocessing/__init__.py +++ b/src/fair_shares/library/preprocessing/__init__.py @@ -6,8 +6,11 @@ from fair_shares.library.preprocessing.config import load_preprocessing_config from fair_shares.library.preprocessing.coverage import ( + FIRST_RECORDED_YEAR_FILENAME, compute_analysis_countries, + coverage_exclusions, create_coverage_summary, + source_coverage_exclusions, ) from fair_shares.library.preprocessing.gini import ( DEFAULT_GINI_MISSING_POLICY, @@ -33,11 +36,13 @@ __all__ = [ "DEFAULT_GINI_MISSING_POLICY", + "FIRST_RECORDED_YEAR_FILENAME", "GINI_MISSING_POLICIES", "NGHGI_CORRECTED_CATEGORIES", "add_row_to_datasets", "complete_gini", "compute_analysis_countries", + "coverage_exclusions", "create_coverage_summary", "emissions_filename", "emissions_path", @@ -50,4 +55,5 @@ "load_preprocessing_config", "load_scenarios_data", "process_complete_scenarios", + "source_coverage_exclusions", ] diff --git a/src/fair_shares/library/preprocessing/coverage.py b/src/fair_shares/library/preprocessing/coverage.py index bbf6186..14e86c1 100644 --- a/src/fair_shares/library/preprocessing/coverage.py +++ b/src/fair_shares/library/preprocessing/coverage.py @@ -4,17 +4,124 @@ import pandas as pd +from fair_shares.library.exceptions import DataLoadingError from fair_shares.library.utils import ( get_complete_iso3c_timeseries, last_year_column, ) +# Written by an emissions notebook (101) next to its timeseries file. +FIRST_RECORDED_YEAR_FILENAME = "emiss_{category}_first_recorded_year.csv" + + +def coverage_exclusions( + countries: set[str], + population: pd.DataFrame, + coverage: dict | None, + first_recorded_year: pd.Series | None = None, +) -> dict[str, str]: + """Return {iso3c: reason} for the countries that fail the coverage rule. + + The rule is the ``coverage`` block of an emissions source in + ``data_sources_unified.yaml``. A source without the block has no rule, and + the result is empty. With the block, a country is an analysis country only + if it passes each configured test: + + - ``emissions_recorded_before``: the raw emissions file holds a value for + the country in a year before this one. A blank that the source fills + with zero is not a record. + - ``population_from``: the population series holds a value for every year + from this one through its last year. + + A country that fails joins rest-of-world, whose rows are the world row + minus the analysis countries. The country set is then the same for every + allocation start year that the two thresholds cover. + + Args: + countries: Countries to test + population: Population DataFrame + coverage: The ``coverage`` block, or None + first_recorded_year: First recorded emissions year by iso3c. Required + when ``emissions_recorded_before`` is set. + + Returns + ------- + Reasons by iso3c, for the failing countries only + """ + if not coverage: + return {} + reasons: dict[str, list[str]] = {} + + recorded_before = coverage.get("emissions_recorded_before") + if recorded_before is not None: + if first_recorded_year is None: + raise DataLoadingError( + "The coverage rule 'emissions_recorded_before' needs the first " + "recorded year of each country, and none was given." + ) + # A country without an entry has no record at all. + first = first_recorded_year.reindex(sorted(countries)) + for iso3c in first.index[~(first < recorded_before)]: + reasons.setdefault(iso3c, []).append( + f"no emissions recorded before {recorded_before}" + ) + + population_from = coverage.get("population_from") + if population_from is not None: + complete = get_complete_iso3c_timeseries( + population, + expected_index_names=["iso3c", "unit"], + start=population_from, + end=last_year_column(population), + ) + for iso3c in sorted(countries - complete): + reasons.setdefault(iso3c, []).append( + f"population incomplete from {population_from}" + ) + + return {iso3c: "; ".join(reasons[iso3c]) for iso3c in sorted(reasons)} + + +def source_coverage_exclusions( + countries: set[str], + population: pd.DataFrame, + emissions_parameters: dict, + emissions_dir: Path, +) -> dict[str, str]: + """Apply the coverage rule of an emissions source to ``countries``. + + Reads the rule from the source's ``data_parameters`` and the first recorded + years from the tables its notebook wrote to ``emissions_dir``. With several + categories a country takes its latest first year. + """ + coverage = emissions_parameters.get("coverage") + if not coverage: + return {} + first_recorded_year = None + if coverage.get("emissions_recorded_before") is not None: + tables = [] + for category in emissions_parameters["available_categories"]: + path = emissions_dir / FIRST_RECORDED_YEAR_FILENAME.format( + category=category + ) + if not path.exists(): + raise DataLoadingError( + f"First recorded year table not found: {path}. The coverage " + "rule 'emissions_recorded_before' needs it from notebook 101." + ) + table = pd.read_csv(path).set_index("iso3c")["first_recorded_year"] + tables.append(table) + first_recorded_year = pd.concat(tables, axis=1).max(axis=1, skipna=False) + return coverage_exclusions(countries, population, coverage, first_recorded_year) + def compute_analysis_countries( emissions_data: dict[str, pd.DataFrame], gdp: pd.DataFrame, population: pd.DataFrame, gini: pd.DataFrame | None = None, + coverage: dict | None = None, + first_recorded_year: pd.Series | None = None, ) -> set[str]: """Compute the set of countries with complete emissions, GDP and population. @@ -28,6 +135,8 @@ def compute_analysis_countries( gdp: GDP DataFrame population: Population DataFrame gini: Accepted for backwards compatibility and ignored. + coverage: Opt-in coverage rule of the emissions source, or None + first_recorded_year: First recorded emissions year by iso3c, for the rule Returns ------- @@ -61,7 +170,10 @@ def compute_analysis_countries( for category_countries in emiss_analysis_countries.values(): analysis_countries = analysis_countries & category_countries - return analysis_countries + excluded = coverage_exclusions( + analysis_countries, population, coverage, first_recorded_year + ) + return analysis_countries - set(excluded) def create_coverage_summary( @@ -73,6 +185,7 @@ def create_coverage_summary( region_mapping: pd.DataFrame, output_dir: Path, gdp_variant: str | None = None, + coverage_rule_failed: dict[str, str] | None = None, ) -> pd.DataFrame: """Create and save data coverage summary. @@ -85,6 +198,8 @@ def create_coverage_summary( region_mapping: Region mapping DataFrame with iso3c column output_dir: Directory to save coverage summary gdp_variant: Optional GDP variant name for reporting + coverage_rule_failed: Reasons from ``coverage_exclusions`` by iso3c. + Adds the ``coverage_rule_failed`` column when given. Returns ------- @@ -148,6 +263,12 @@ def create_coverage_summary( all_region_countries ) & ~coverage_summary["iso3c"].isin(analysis_countries) + # Sources with a coverage rule get one more column: the failed tests. + if coverage_rule_failed is not None: + coverage_summary["coverage_rule_failed"] = ( + coverage_summary["iso3c"].map(coverage_rule_failed).fillna("") + ) + # Calculate summary statistics total_countries = len(coverage_summary) countries_with_emissions = coverage_summary["has_emissions"].sum() diff --git a/src/fair_shares/library/preprocessing/rcbs.py b/src/fair_shares/library/preprocessing/rcbs.py index fba6703..637198d 100644 --- a/src/fair_shares/library/preprocessing/rcbs.py +++ b/src/fair_shares/library/preprocessing/rcbs.py @@ -10,17 +10,22 @@ required because the LULUCF decomposition extends to net-zero (no observational data exists for the future). Adjusted for baseline year by subtracting the 2020-to-base prefix. -- **co2**: uses the pre-computed convention gap (BM -> NGHGI) from - notebook 104, plus actual BM LULUCF for rebase. +- **co2**: the rebase adds observed NGHGI LULUCF for 2020 to the year before + the baseline, and the pre-computed convention gap (BM -> NGHGI) from + notebook 104 runs from the baseline year to net zero. NZ years and convention gap scalars are pre-computed by notebook 104 from the scenario data (currently Gidden et al. AR6 reanalysis) and saved as ``rcb_scenario_adjustments.yaml``. Per-year BM LULUCF medians are stored -as ``lulucf_shift_median_{scenario}.csv``. +as ``lulucf_shift_median_{key}.csv``. Both are keyed by scenario set +(see ``rcb_scenario_set_key``): the budget label for sources under the +AR6 category rule, the temperature band for sources under the peak-warming +band rule. """ from __future__ import annotations +import warnings from pathlib import Path import pandas as pd @@ -40,7 +45,14 @@ from fair_shares.library.utils.data.nghgi import ( compute_bunker_deduction, load_bunker_timeseries, - load_world_co2_lulucf, +) +from fair_shares.library.utils.data.rcb import ( + DEFAULT_REBASE_FILL_MAX_YEARS, + DEFAULT_SCENARIO_SELECTION, + fill_rebase_years, + missing_rebase_years, + rcb_scenario_set_key, + validate_rebase_fill_max_years, ) @@ -60,67 +72,6 @@ def _resolve_template_path( return resolve_source_path(path_str, data_dir=data_dir, output_dir=output_dir) -def _load_shared_timeseries( - adjustments: AdjustmentsConfig, - data_dir: Path | str | None = None, - source_id: str | None = None, - verbose: bool = True, - load_nghgi: bool = True, - output_dir: Path | str | None = None, -) -> tuple[pd.DataFrame | None, pd.DataFrame, int | None]: - """Load scenario-invariant timeseries data (bunkers, and optionally NGHGI LULUCF). - - Called once before the scenario loop to avoid repeated file reads. - - NGHGI LULUCF is only needed for total CO2 (co2) — not for co2-ffi. - Callers should set ``load_nghgi=False`` when processing co2-ffi. - - Parameters - ---------- - adjustments : AdjustmentsConfig - Adjustment configuration with paths. - data_dir : Path or None - Directory holding input data. Defaults to the resolved data directory. - source_id : str or None - Source ID for resolving intermediate paths with {source_id} template. - verbose : bool - Print progress. - load_nghgi : bool - Whether to load NGHGI LULUCF world timeseries. Only needed for - total CO2 emission categories. Default True for backwards compat. - output_dir : Path or None - Directory holding pipeline products. Defaults to the resolved output - directory. - - Returns - ------- - tuple[pd.DataFrame | None, pd.DataFrame, int | None] - (nghgi_ts, bunker_ts, splice_year) — nghgi_ts and splice_year - are None when load_nghgi=False. - """ - nghgi_ts = None - splice_year = None - - if load_nghgi: - nghgi_path = _resolve_template_path( - adjustments.lulucf_nghgi.path, source_id, data_dir, output_dir - ) - if verbose: - print(f" Loading NGHGI LULUCF from: {nghgi_path}") - nghgi_ts, splice_year = load_world_co2_lulucf(nghgi_path) - if verbose: - print(f" NGHGI splice year (from data): {splice_year}") - - bunker_path = _resolve_template_path( - adjustments.bunkers.path, source_id, data_dir, output_dir - ) - if verbose: - print(f" Loading bunker timeseries from: {bunker_path}") - bunker_ts = load_bunker_timeseries(bunker_path) - - return nghgi_ts, bunker_ts, splice_year - - def _load_rcb_scenario_adjustments( intermediate_dir: Path, verbose: bool = True, @@ -138,9 +89,10 @@ def _load_rcb_scenario_adjustments( Returns ------- dict[str, dict] - Mapping of AR6 category (e.g., "C1") to adjustment dict with keys: - ``bm_lulucf_cumulative_median``, ``convention_gap_median``, - ``nz_year_median``, ``n_scenarios`` + Mapping of scenario set key (e.g., "1.5p50") to adjustment dict with keys: + ``bm_lulucf_cumulative_median``, ``convention_gap_median_from`` + (one median gap per RCB baseline year), ``nz_year_median``, + ``n_scenarios`` Raises ------ @@ -163,7 +115,8 @@ def _load_rcb_scenario_adjustments( print( f" {cat}: NZ_med={vals['nz_year_median']}, " f"BM_LULUCF={vals['bm_lulucf_cumulative_median']:.0f} Mt, " - f"gap={vals['convention_gap_median']:.0f} Mt, " + f"gap from baseline year (Mt)=" + f"{vals.get('convention_gap_median_from')}, " f"n={vals['n_scenarios']}" ) @@ -187,7 +140,8 @@ def _resolve_adjustment_scalars( (median of per-scenario cumulatives from 2020 to each scenario's NZ). When ``baseline_year`` > 2020, the 2020-to-base prefix is subtracted from the median timeseries. - For co2: uses the pre-computed convention gap from notebook 104. + For co2: uses the pre-computed convention gap from ``baseline_year`` to + NZ (``convention_gap_median_from[baseline_year]``) from notebook 104. Returns values that can be added directly to the budget: - **bunkers**: always positive (cumulative emissions); caller negates @@ -196,9 +150,10 @@ def _resolve_adjustment_scalars( Parameters ---------- scenario : str - Fair-shares scenario string (e.g. "1.5p50") + Scenario set key (e.g. "1.5p50" or "peak-warming-1.7C") baseline_year : int - RCB source baseline year — LULUCF integration starts here + RCB source baseline year — the LULUCF decomposition (co2-ffi) and + the convention gap (co2) start here net_zero_year : int Category-level NZ year for bunker and LULUCF integration bunker_ts : pd.DataFrame @@ -208,7 +163,7 @@ def _resolve_adjustment_scalars( used for co2-ffi to integrate from baseline_year to NZ rcb_adjustments : dict[str, dict] Pre-computed RCB adjustment scalars from notebook 104, - keyed by scenario (e.g., "1.5p50") + keyed by scenario set (e.g., "1.5p50") emission_category : str Emission category (default: "co2-ffi") precautionary_lulucf : bool @@ -223,21 +178,38 @@ def _resolve_adjustment_scalars( Bunkers is positive; the two LULUCF scalars are sign-ready. Exactly one of the LULUCF scalars is non-zero per category (future_mt for co2-ffi, nghgi_correction_mt for co2). + + Raises + ------ + DataLoadingError + If ``rcb_adjustments`` has no entry for ``scenario``, or, for co2, + no convention gap from ``baseline_year`` """ + if scenario not in rcb_adjustments: + raise DataLoadingError( + f"No RCB scenario adjustments for '{scenario}' " + f"(available: {list(rcb_adjustments)}). " + "Re-run notebook 104 (AR6 scenario preprocessing)." + ) + adj = rcb_adjustments[scenario] + lulucf_future_mt = 0.0 lulucf_nghgi_correction_mt = 0.0 if emission_category == "co2": - # NGHGI-vs-BM convention correction from 2020 to NZ (Weber 2026). - # The rebase uses BM LULUCF; this re-expresses the result against - # national-inventory accounting. Budget still contains FFI + LULUCF. - adj = rcb_adjustments.get(scenario, {}) - lulucf_nghgi_correction_mt = adj.get("convention_gap_median", 0.0) + # NGHGI-minus-BM convention gap from the baseline year to NZ (Weber + # 2026). The rebase adds observed NGHGI LULUCF for the years before it. + gaps_from = adj.get("convention_gap_median_from") or {} + if baseline_year not in gaps_from: + raise DataLoadingError( + f"No convention gap from baseline year {baseline_year} for " + f"'{scenario}' (available baseline years: {sorted(gaps_from)}). " + "Re-run notebook 104 (AR6 scenario preprocessing)." + ) + lulucf_nghgi_correction_mt = gaps_from[baseline_year] if lulucf_nghgi_correction_mt == 0.0: - import warnings - warnings.warn( - f"convention_gap_median is 0.0 for scenario '{scenario}' — " + f"The convention gap is 0.0 for scenario '{scenario}' — " f"this likely means notebook 104 ran before NGHGI data was " f"preprocessed. Re-run the preprocessing pipeline: " f"notebooks 105/107 first, then 104.", @@ -246,8 +218,7 @@ def _resolve_adjustment_scalars( else: # co2-ffi: subtract projected future (base→NZ) BM LULUCF to convert # a published total-CO2 RCB into an FFI-only RCB. - adj = rcb_adjustments.get(scenario, {}) - bm_lulucf_mt = adj.get("bm_lulucf_cumulative_median", 0.0) + bm_lulucf_mt = adj["bm_lulucf_cumulative_median"] # Adjust for baseline year > 2020: subtract the 2020-to-base # prefix using the median timeseries. Historical BM LULUCF has @@ -269,7 +240,8 @@ def _resolve_adjustment_scalars( else: lulucf_future_mt = -bm_lulucf_mt - # --- Bunkers deduction (always positive; caller negates) --- + # --- Bunkers deduction from 2020 to NZ (always positive; caller negates). + # The rebase adds the bunkers of 2020 to baseline_year - 1. --- bunkers_mt = compute_bunker_deduction( bunker_ts=bunker_ts, start_year=2020, @@ -294,7 +266,7 @@ def load_and_process_rcbs( adjustments_config: AdjustmentsConfig, data_dir: Path | str | None = None, source_id: str | None = None, - actual_bm_lulucf_emissions: pd.DataFrame | None = None, + world_nghgi_lulucf_emissions: pd.DataFrame | None = None, verbose: bool = True, output_dir: Path | str | None = None, ) -> pd.DataFrame: @@ -318,9 +290,10 @@ def load_and_process_rcbs( Directory holding input data. Defaults to the resolved data directory. source_id : str or None, optional Source ID for resolving intermediate paths with {source_id} template. - actual_bm_lulucf_emissions : pd.DataFrame or None, optional - Actual BM LULUCF emissions for co2 rebase. Required when - emission_category is "co2". + world_nghgi_lulucf_emissions : pd.DataFrame or None, optional + Observed world LULUCF CO2 emissions in the national-inventory (NGHGI) + convention, for the co2 rebase. Required when emission_category is + "co2". verbose : bool, optional Print processing details output_dir : Path or None, optional @@ -361,15 +334,13 @@ def load_and_process_rcbs( # Ensure world emissions has string year columns world_fossil_emissions = ensure_string_year_columns(world_fossil_emissions) - # Pre-load scenario-invariant timeseries (bunkers always; NGHGI only for total CO2) - _nghgi_ts, bunker_ts, _splice_year = _load_shared_timeseries( - adjustments_config, - data_dir, - source_id=source_id, - verbose=verbose, - load_nghgi=(emission_category == "co2"), - output_dir=output_dir, + # Pre-load the scenario-invariant bunker timeseries + bunker_path = _resolve_template_path( + adjustments_config.bunkers.path, source_id, data_dir, output_dir ) + if verbose: + print(f" Loading bunker timeseries from: {bunker_path}") + bunker_ts = load_bunker_timeseries(bunker_path) # Load pre-computed RCB adjustment scalars from notebook 104 if not source_id: @@ -398,12 +369,17 @@ def load_and_process_rcbs( print(" Adjustment mode: pre-computed (NGHGI-consistent, Weber et al. 2026)") print(" Bunker NZ years: category-level median (from scenario adjustments)") + rebase_fill_max_years = validate_rebase_fill_max_years( + rcb_data.get("rebase_fill_max_years", DEFAULT_REBASE_FILL_MAX_YEARS) + ) + # Pre-load baseline-shift LULUCF median timeseries from notebook 104 output. - # These are year-by-year median AFOLU|Direct CSVs, one per AR6 category. + # These are year-by-year median AFOLU|Direct CSVs, one per scenario set. lulucf_shift_cache: dict[str, pd.DataFrame] = {} # Create a list to store all RCB records rcb_records = [] + skipped_sources = [] # Process each source for source_key, source_data in rcb_data["rcb_data"].items(): @@ -413,6 +389,7 @@ def load_and_process_rcbs( baseline_year = source_data.get("baseline_year") unit = source_data.get("unit", "Gt CO2") scenarios = source_data.get("scenarios", {}) + selection = source_data.get("scenario_selection", DEFAULT_SCENARIO_SELECTION) if baseline_year is None: raise ConfigurationError( @@ -427,34 +404,81 @@ def load_and_process_rcbs( print(f" Baseline year: {baseline_year}") print(f" Unit: {unit}") print(f" Scenarios: {len(scenarios)}") + print(f" Scenario selection: {selection}") + + # A series that ends up to rebase_fill_max_years before the last rebase + # year takes its last observed value for the remaining years. + rebase_fossil = fill_rebase_years( + world_fossil_emissions, + baseline_year, + rebase_fill_max_years, + "world fossil CO2 emissions", + source_key, + ) + rebase_bunkers = fill_rebase_years( + bunker_ts, + baseline_year, + rebase_fill_max_years, + "international bunker emissions", + source_key, + ) + rebase_lulucf = world_nghgi_lulucf_emissions + if emission_category == "co2" and world_nghgi_lulucf_emissions is not None: + rebase_lulucf = fill_rebase_years( + world_nghgi_lulucf_emissions, + baseline_year, + rebase_fill_max_years, + "world NGHGI LULUCF CO2 emissions", + source_key, + ) + + # The rebase to 2020 needs emissions for every year before the baseline. + # A source that the emissions data cannot rebase is left out. + missing_years = missing_rebase_years(baseline_year, rebase_fossil) + missing_years += missing_rebase_years(baseline_year, rebase_bunkers) + if emission_category == "co2" and rebase_lulucf is not None: + missing_years += missing_rebase_years(baseline_year, rebase_lulucf) + if missing_years: + skipped_sources.append( + f"'{source_key}' (baseline {baseline_year}, data lack " + f"{sorted(set(missing_years))})" + ) + continue for scenario, rcb_value in scenarios.items(): climate_assessment, quantile = parse_rcb_scenario(scenario) + set_key = rcb_scenario_set_key(source_key, scenario, selection) + if set_key not in rcb_adjustments: + raise DataLoadingError( + f"RCB source '{source_key}', label '{scenario}': no scenario " + f"adjustments for '{set_key}' in {adj_path}. " + "Re-run notebook 104 (AR6 scenario preprocessing)." + ) # Load pre-computed median LULUCF shift timeseries for baseline shift - if scenario not in lulucf_shift_cache: - shift_csv = scenarios_dir / f"lulucf_shift_median_{scenario}.csv" + if set_key not in lulucf_shift_cache: + shift_csv = scenarios_dir / f"lulucf_shift_median_{set_key}.csv" if not shift_csv.exists(): raise DataLoadingError( f"LULUCF shift median not found: {shift_csv}. " "Run notebook 104 (AR6 scenario preprocessing) first." ) shift_df = pd.read_csv(shift_csv).set_index("source") - lulucf_shift_cache[scenario] = shift_df + lulucf_shift_cache[set_key] = shift_df if verbose: print( - f" Loaded LULUCF shift median for {scenario} " + f" Loaded LULUCF shift median for {set_key} " f"from {shift_csv}" ) - direct_median = lulucf_shift_cache[scenario] + direct_median = lulucf_shift_cache[set_key] # Scenario-level NZ year (for bunker integration) - nz_year = rcb_adjustments[scenario]["nz_year_median"] + nz_year = rcb_adjustments[set_key]["nz_year_median"] # Resolve adjustment scalars from pre-computed values bunkers_mt, lulucf_future_mt, lulucf_nghgi_mt = ( _resolve_adjustment_scalars( - scenario=scenario, + scenario=set_key, baseline_year=baseline_year, net_zero_year=nz_year, bunker_ts=bunker_ts, @@ -471,12 +495,13 @@ def load_and_process_rcbs( rcb_value=rcb_value, rcb_unit=unit, rcb_baseline_year=baseline_year, - world_co2_ffi_emissions=world_fossil_emissions, + world_co2_ffi_emissions=rebase_fossil, emission_category=emission_category, + world_bunker_emissions=rebase_bunkers, bunkers_deduction_mt=bunkers_mt, lulucf_future_deduction_mt=lulucf_future_mt, lulucf_nghgi_correction_mt=lulucf_nghgi_mt, - actual_bm_lulucf_emissions=actual_bm_lulucf_emissions, + world_nghgi_lulucf_emissions=rebase_lulucf, target_baseline_year=2020, source_name=source_key, scenario=scenario, @@ -496,6 +521,7 @@ def load_and_process_rcbs( "net_adjustment_mt": result["net_adjustment_mt"], "rebase_total_mt": result["rebase_total_mt"], "rebase_fossil_mt": result["rebase_fossil_mt"], + "rebase_bunkers_mt": result["rebase_bunkers_mt"], "rebase_lulucf_mt": result["rebase_lulucf_mt"], "deduction_bunkers_mt": result["deduction_bunkers_mt"], "deduction_lulucf_future_mt": result["deduction_lulucf_future_mt"], @@ -506,6 +532,16 @@ def load_and_process_rcbs( rcb_df = pd.DataFrame(rcb_records) + # This step processes every source in rcbs.yaml and does not know which + # one an allocation requests, so it reports the skipped sources once. + if skipped_sources: + warnings.warn( + f"RCB sources left out, no rebase to 2020 with at most " + f"{rebase_fill_max_years} filled year(s) (rebase_fill_max_years): " + f"{'; '.join(skipped_sources)}.", + stacklevel=2, + ) + if verbose: print("\nProcessed RCB data:") print(rcb_df.to_string(index=False)) diff --git a/src/fair_shares/library/utils/data/config.py b/src/fair_shares/library/utils/data/config.py index 0317a5e..215c0bb 100644 --- a/src/fair_shares/library/utils/data/config.py +++ b/src/fair_shares/library/utils/data/config.py @@ -10,17 +10,19 @@ import os from pathlib import Path -from typing import Any, Literal +from typing import TYPE_CHECKING, Any, Literal import yaml -from fair_shares.library.config.models import DataSourcesConfig from fair_shares.library.exceptions import ( ConfigurationError, DataLoadingError, ) from fair_shares.library.paths import packaged_config +if TYPE_CHECKING: + from fair_shares.library.config.models import DataSourcesConfig + # Environment override letting a user supply their own source table without # editing the installed package. CONFIG_ENV = "FAIR_SHARES_CONFIG" @@ -98,7 +100,9 @@ def get_final_categories(target: str, emission_category: str) -> tuple[str, ...] def get_emission_preprocessing_categories( - target: str, emission_category: str + target: str, + emission_category: str, + available_categories: list[str] | None = None, ) -> tuple[str, ...]: """Return emission categories needed for emissions preprocessing. @@ -111,30 +115,45 @@ def get_emission_preprocessing_categories( subtraction (``all-ghg-ex-co2-lulucf − co2-ffi``) in the derive_non_co2 pipeline rule. When ``non-co2`` is the main category, we return its ingredients instead. + + When ``available_categories`` is given, the result keeps only categories + the emissions source declares, so a ``co2-ffi``-only source is asked for + ``co2-ffi`` alone. """ # Base primitives always needed by notebook 107 (LULUCF → derived categories) _LULUCF_PRIMITIVES = {"co2-ffi", "co2-lulucf", "all-ghg-ex-co2-lulucf"} - if not is_composite_category(emission_category): - if emission_category == "co2": - needed = {"co2", "co2-ffi", "co2-lulucf", "all-ghg-ex-co2-lulucf"} - elif emission_category == "non-co2": - # non-co2 is derived by subtraction, not extracted by 101 - needed = _LULUCF_PRIMITIVES.copy() - else: - needed = {emission_category} | _LULUCF_PRIMITIVES - return tuple(sorted(needed)) - - if target == "pathway": - # pathway target has direct data, but notebook 107 still needs primitives + if emission_category == "co2": + needed = {"co2", "co2-ffi", "co2-lulucf", "all-ghg-ex-co2-lulucf"} + elif emission_category == "non-co2": + # non-co2 is derived by subtraction, not extracted by 101 + needed = _LULUCF_PRIMITIVES.copy() + elif emission_category == "all-ghg" and target != "pathway": + # RCB targets take the CO2 part from budgets, so all-ghg itself is not extracted + needed = {"co2"} | _LULUCF_PRIMITIVES + else: needed = {emission_category} | _LULUCF_PRIMITIVES - return tuple(sorted(needed)) - # RCB targets: need source categories for non-co2 derivation + LULUCF primitives - if emission_category == "all-ghg": - return ("all-ghg-ex-co2-lulucf", "co2", "co2-ffi", "co2-lulucf") - else: # all-ghg-ex-co2-lulucf - return ("all-ghg-ex-co2-lulucf", "co2-ffi", "co2-lulucf") + if available_categories is not None: + needed &= set(available_categories) + return tuple(sorted(needed)) + + +# Bunker data used by every emissions source that does not name its own. +DEFAULT_BUNKERS_SOURCE = "gcb-2024" + + +def get_emissions_data_parameters(emissions_source: str | None) -> dict[str, Any]: + """Return ``data_parameters`` of an emissions source in the packaged config.""" + with packaged_config("data_sources/data_sources_unified.yaml").open() as f: + emissions = yaml.safe_load(f).get("emissions", {}) + return emissions.get(emissions_source, {}).get("data_parameters", {}) + + +def get_bunkers_source(emissions_source: str | None) -> str: + """Return the registry name of the bunker data paired with an emissions source.""" + params = get_emissions_data_parameters(emissions_source) + return params.get("bunkers_source") or DEFAULT_BUNKERS_SOURCE def build_source_id( @@ -425,6 +444,9 @@ def build_data_config( "rcb_generator": rcb_generator, # Will be None for non-rcb-pathways targets } + # Imported here because config.models loads utils, which loads this module. + from fair_shares.library.config.models import DataSourcesConfig + # Validate with Pydantic (this will raise ValidationError if invalid) validated_config = DataSourcesConfig(**filtered_config) diff --git a/src/fair_shares/library/utils/data/gcb.py b/src/fair_shares/library/utils/data/gcb.py new file mode 100644 index 0000000..05c7827 --- /dev/null +++ b/src/fair_shares/library/utils/data/gcb.py @@ -0,0 +1,92 @@ +"""Global Carbon Budget fossil CO2: flat CSV to the pipeline's timeseries layout. + +The Andrew and Peters flat file is long format, MtCO2 per year, one row per +entity and year. Besides countries it holds four rows that belong to no country: +the global total (``WLD``), international shipping (``XIS``), international +aviation (``XIA``) and the 1991 Kuwaiti oil fires. The global total equals the +sum of every other row. + +Rules applied here: + +- Every blank year counts as zero. A blank means the file reports those + emissions under another entity (Ireland under the United Kingdom until 1923) + or reports none, and the global total already treats it as zero. +- The world row is the global total minus shipping, aviation and the oil fires. + The three series are returned separately so the split stays auditable. +- A country row is an ISO3 code in ``countries``. Every other row stays in the + world row and so lands in rest-of-world: codes outside the region mapping + (Kosovo, Antarctica) and the two rows without a code ("Pacific Islands + (Palau)", "Ryukyu Islands"), which the official GCB workbook also keeps in + the world total and in no country column. +""" + +from __future__ import annotations + +import pandas as pd + +from fair_shares.library.exceptions import DataLoadingError + +ISO, NAME, YEAR, VALUE = "ISO 3166-1 alpha-3", "Country", "Year", "Total" +GLOBAL = "WLD" +SHIPPING = "XIS" +AVIATION = "XIA" +OIL_FIRES = "Kuwaiti Oil Fires" +UNIT = "Mt * CO2e" + + +def _wide(raw: pd.DataFrame, first_year: int) -> pd.DataFrame: + """Pivot the long file to one row per entity, with blanks as zero.""" + missing = {ISO, NAME, YEAR, VALUE} - set(raw.columns) + if missing: + raise DataLoadingError(f"GCB file lacks expected columns: {sorted(missing)}") + raw = raw[raw[YEAR] >= first_year] + # Rows without an ISO code are keyed by their name. + wide = raw.assign(key=raw[ISO].fillna(raw[NAME])).pivot( + index="key", columns=YEAR, values=VALUE + ) + wide.columns = wide.columns.astype(str).rename(None) + return wide.fillna(0.0) + + +def gcb_fossil_co2( + raw: pd.DataFrame, countries: set[str], world_key: str, first_year: int = 1850 +) -> tuple[pd.DataFrame, pd.DataFrame]: + """Return GCB fossil CO2 as (emissions, excluded), both in MtCO2 per year. + + ``emissions`` holds one row per code in ``countries`` plus the world row, + indexed by (iso3c, unit, emission-category). ``excluded`` holds shipping, + aviation and the oil fires, indexed by source. For every year, + world row + excluded rows = GCB global total. + """ + wide = _wide(raw, first_year) + excluded = wide.loc[[SHIPPING, AVIATION, OIL_FIRES]].rename_axis("source") + world = (wide.loc[GLOBAL] - excluded.sum()).rename(world_key) + + emissions = pd.concat([wide[wide.index.isin(countries)], world.to_frame().T]) + emissions.index = pd.MultiIndex.from_tuples( + [(iso3c, UNIT, "co2-ffi") for iso3c in emissions.index], + names=["iso3c", "unit", "emission-category"], + ) + return emissions, excluded + + +def gcb_first_recorded_year( + raw: pd.DataFrame, countries: set[str], first_year: int = 1850 +) -> pd.Series: + """Return the first year from ``first_year`` with a value in the raw file. + + One entry per code in ``countries`` that has a record, indexed by iso3c. + A blank is not a record, and a reported zero is. After the zero-fill the + two look the same, so the coverage rule reads this table. + """ + recorded = raw[(raw[YEAR] >= first_year) & raw[VALUE].notna()] + recorded = recorded[recorded[ISO].isin(countries)] + first = recorded.groupby(ISO)[YEAR].min() + return first.rename("first_recorded_year").rename_axis("iso3c") + + +def gcb_bunkers(raw: pd.DataFrame, first_year: int = 1850) -> pd.DataFrame: + """Return international shipping plus aviation as the single ``bunkers`` row.""" + wide = _wide(raw, first_year) + bunkers = (wide.loc[SHIPPING] + wide.loc[AVIATION]).rename("bunkers") + return bunkers.to_frame().T.rename_axis("source") diff --git a/src/fair_shares/library/utils/data/nghgi.py b/src/fair_shares/library/utils/data/nghgi.py index e4a962d..4265ead 100644 --- a/src/fair_shares/library/utils/data/nghgi.py +++ b/src/fair_shares/library/utils/data/nghgi.py @@ -11,8 +11,9 @@ - Deductions returned as positive floats (calling code negates as needed) Data structure notes: -- NGHGI LULUCF file: CSV produced by notebook 107 (Melo v3.1) -- Bunker file: CSV produced by notebook 107, sourced from GCB2024 +- NGHGI LULUCF file: CSV produced by notebook 107 from the active LULUCF + source (Melo et al., v3.1.1 or v4.0.0) +- Bunker file: CSV produced by notebook 108 from the active bunker source - AR6 category constants: YAML produced by notebook 104 (scenario preprocessing) """ @@ -29,7 +30,8 @@ def load_world_co2_lulucf(path: str | Path) -> tuple[pd.DataFrame, int]: """Load world-total NGHGI LULUCF CO2 timeseries from notebook-produced CSV. Reads the world-total NGHGI-reported LULUCF CO2 values produced by - notebook 107 (Melo v3.1). The CSV has a single row with a "source" index + notebook 107 from the active LULUCF source (Melo et al., v3.1.1 or + v4.0.0). The CSV has a single row with a "source" index and string year columns. Values are in MtCO2/yr (negative = net sink). The splice year (last year of NGHGI data) is derived dynamically from the @@ -173,12 +175,13 @@ def compute_bunker_deduction( bunker_ts: pd.DataFrame, start_year: int, net_zero_year: int, - historical_end_year: int = 2023, + historical_end_year: int | None = None, ) -> float: """Compute cumulative international bunker fuel CO2 deduction. - Combines historical year-by-year values from GCB2024 with extrapolation - from the last observed annual rate for years beyond the historical record. + Combines historical year-by-year values from the bunker timeseries with + extrapolation from the last observed annual rate for years beyond the + historical record. Parameters ---------- @@ -189,8 +192,8 @@ def compute_bunker_deduction( net_zero_year : int End of integration window (inclusive) historical_end_year : int, optional - Last year covered by the historical timeseries (default: 2023, - matching GCB2024 coverage) + Last year taken from the historical timeseries (default: the last + year with a value in ``bunker_ts``) Returns ------- @@ -202,6 +205,10 @@ def compute_bunker_deduction( DataProcessingError If historical data is insufficient for the start_year """ + if historical_end_year is None: + observed = bunker_ts.iloc[0].dropna() + historical_end_year = max(int(c) for c in observed.index if _is_year(c)) + historical_end = min(historical_end_year, net_zero_year) historical_bunkers = compute_cumulative_emissions( bunker_ts, start_year, historical_end @@ -210,13 +217,7 @@ def compute_bunker_deduction( # Future component: extrapolate last observed rate beyond historical record future_bunkers = 0.0 if net_zero_year > historical_end_year: - last_year_str = str(historical_end_year) - if last_year_str in bunker_ts.columns: - last_rate = float(bunker_ts[last_year_str].iloc[0]) - else: - # Fall back to last available year - avail_years = sorted(int(c) for c in bunker_ts.columns if _is_year(c)) - last_rate = float(bunker_ts[str(avail_years[-1])].iloc[0]) + last_rate = float(bunker_ts[str(historical_end_year)].iloc[0]) future_years = net_zero_year - historical_end_year future_bunkers = last_rate * future_years @@ -226,13 +227,14 @@ def compute_bunker_deduction( def build_nghgi_world_co2_timeseries( fossil_ts: pd.DataFrame, nghgi_ts: pd.DataFrame, - bunker_ts: pd.DataFrame, ) -> pd.DataFrame: """Construct NGHGI-consistent world total CO2 timeseries. For backward extension of allocation years < 2020, Weber Eq. 3 requires - per-year world CO2 = fossil - bunkers + LULUCF, where LULUCF uses: - - 2000 onwards: NGHGI LULUCF (e.g. Melo v3.1) + per-year world CO2 = fossil - bunkers + LULUCF. The world CO2-FFI series + already excludes international bunkers, so the result is fossil + LULUCF, + where LULUCF uses: + - 2000 onwards: NGHGI LULUCF (e.g. Melo et al.) - Pre-2000: NaN (no fallback — NGHGI coverage only) No NGHGI/BM splicing is performed. Years outside NGHGI coverage are NaN. @@ -240,22 +242,19 @@ def build_nghgi_world_co2_timeseries( Parameters ---------- fossil_ts : pd.DataFrame - World CO2-FFI emissions timeseries (e.g. PRIMAP) in Mt CO2/yr. - Must have string year columns and a MultiIndex with - (iso3c, unit, emission-category). + World CO2-FFI emissions timeseries (e.g. PRIMAP) in Mt CO2/yr, + excluding international bunkers. Must have string year columns and a + MultiIndex with (iso3c, unit, emission-category). nghgi_ts : pd.DataFrame NGHGI LULUCF historical timeseries (from load_world_co2_lulucf) in MtCO2/yr. Single-row DataFrame with string year columns. - bunker_ts : pd.DataFrame - Bunker fuel CO2 timeseries (from load_bunker_timeseries) - in MtCO2/yr. Single-row DataFrame with string year columns. Returns ------- pd.DataFrame Single-row DataFrame with same index structure as fossil_ts but emission-category label set to "co2", containing per-year - NGHGI-consistent total CO2 = fossil - bunkers + LULUCF. + NGHGI-consistent total CO2 = fossil + LULUCF. Years outside NGHGI LULUCF coverage will be NaN. """ # Get all year columns from fossil_ts @@ -270,14 +269,8 @@ def build_nghgi_world_co2_timeseries( if y in nghgi_ts.columns: lulucf_vals[y] = nghgi_ts[y].iloc[0] - # Build bunker timeseries (0 for years outside bunker data) - bunker_vals = pd.Series(0.0, index=year_cols, dtype=float) - for y in year_cols: - if y in bunker_ts.columns: - bunker_vals[y] = bunker_ts[y].iloc[0] - - # NGHGI-consistent total CO2 = fossil - bunkers + LULUCF - total_co2 = fossil_vals + lulucf_vals - bunker_vals + # NGHGI-consistent total CO2 = fossil (excluding bunkers) + LULUCF + total_co2 = fossil_vals + lulucf_vals # Build result with same index structure as fossil_ts but co2 label old_idx = fossil_ts.index[0] diff --git a/src/fair_shares/library/utils/data/rcb.py b/src/fair_shares/library/utils/data/rcb.py index 01ed659..8afa189 100644 --- a/src/fair_shares/library/utils/data/rcb.py +++ b/src/fair_shares/library/utils/data/rcb.py @@ -7,16 +7,33 @@ from __future__ import annotations +import warnings from typing import TYPE_CHECKING import pandas as pd -from fair_shares.library.exceptions import DataProcessingError +from fair_shares.library.exceptions import ConfigurationError, DataProcessingError from fair_shares.library.utils.units import get_default_unit_registry if TYPE_CHECKING: from fair_shares.library.utils.dataframes import TimeseriesDataFrame +# AR6 category behind each budget label under the "ar6-category" rule. +AR6_CATEGORY_BY_RCB_LABEL = {"1.5p50": "C1", "2p83": "C2", "2p66": "C3"} + +# AR6 metadata column that the "peak-warming-band" rule reads. +PEAK_WARMING_COLUMN = "Median peak warming (MAGICCv7.5.3)" + +# Scenario metadata column that holds ``net_zero_year`` of each scenario. +NET_ZERO_YEAR_COLUMN = "net_zero_year" + +# A source in rcbs.yaml without a ``scenario_selection`` field uses this rule. +DEFAULT_SCENARIO_SELECTION = "ar6-category" +DEFAULT_PEAK_WARMING_BAND_HALF_WIDTH = 0.05 + +# Largest number of trailing rebase years that take the last observed value. +DEFAULT_REBASE_FILL_MAX_YEARS = 1 + def parse_rcb_scenario(scenario_string: str) -> tuple[str, str]: """ @@ -63,6 +80,384 @@ def parse_rcb_scenario(scenario_string: str) -> tuple[str, str]: return climate_assessment, quantile +def _ar6_category(source: str, label: str) -> str: + """Return the AR6 category mapped to a budget label, or raise.""" + if label not in AR6_CATEGORY_BY_RCB_LABEL: + raise ConfigurationError( + f"RCB source '{source}', label '{label}': no AR6 category is mapped " + f"to this label (mapped labels: {list(AR6_CATEGORY_BY_RCB_LABEL)}) " + f"and the source does not use the peak-warming band rule. Set " + f"'scenario_selection: peak-warming-band' for this source in " + f"rcbs.yaml, or use a mapped label." + ) + return AR6_CATEGORY_BY_RCB_LABEL[label] + + +def net_zero_year(co2_emissions: pd.Series) -> int | None: + """ + Return the first year in which total CO2 emissions are at or below zero. + + The deductions of a budget end in this year, and the peak-warming band + keeps only scenarios that have one. + + Parameters + ---------- + co2_emissions : pd.Series + Total CO2 emissions of one scenario, indexed by year in ascending order + + Returns + ------- + int or None + The net-zero year, or None when emissions stay above zero + """ + for year, value in co2_emissions.items(): + if value <= 0: + return int(year) + return None + + +def _sum_years(series: pd.Series, first_year: int, last_year: int) -> float: + """Sum the years from ``first_year`` to ``last_year`` that ``series`` holds.""" + years = [str(y) for y in range(first_year, last_year + 1) if str(y) in series.index] + return float(series[years].sum()) if years else 0.0 + + +def convention_gap_from_baseline( + nghgi_lulucf: pd.Series, + bm_direct: pd.Series, + indirect: pd.Series, + baseline_year: int, + nz_year: int, + splice_year: int, +) -> float: + """ + Return the NGHGI-minus-BM LULUCF gap of one scenario from the budget baseline. + + The gap converts a budget in the bookkeeping-model (BM) convention to the + national-inventory (NGHGI) convention. It covers the years that the + published budget covers: the baseline year to the net-zero year of the + scenario (Weber et al. 2026, Eqs. 2-3). The years before the baseline are + not part of it; the rebase adds observed NGHGI LULUCF for those years. + + - Years up to ``splice_year``: observed NGHGI LULUCF minus the scenario + BM LULUCF (AFOLU|Direct). + - Years after ``splice_year``: the scenario indirect flux (AFOLU|Indirect), + because the direct flux cancels in the difference. + + Parameters + ---------- + nghgi_lulucf : pd.Series + Observed world NGHGI LULUCF CO2, indexed by year as a string + bm_direct : pd.Series + AFOLU|Direct of the scenario, indexed by year as a string + indirect : pd.Series + AFOLU|Indirect of the scenario, indexed by year as a string + baseline_year : int + First year of the published budget + nz_year : int + Net-zero year of the scenario (last year of the gap) + splice_year : int + Last observed year of ``nghgi_lulucf`` + + Returns + ------- + float + The cumulative gap from ``baseline_year`` to ``nz_year``. Zero when + the net-zero year is before the baseline year. A year that a series + does not hold adds nothing. + """ + if nz_year < baseline_year: + return 0.0 + observed_end = min(splice_year, nz_year) + observed_gap = _sum_years(nghgi_lulucf, baseline_year, observed_end) - _sum_years( + bm_direct, baseline_year, observed_end + ) + scenario_gap = _sum_years(indirect, max(baseline_year, splice_year + 1), nz_year) + return observed_gap + scenario_gap + + +def rcb_scenario_set_key(source: str, label: str, selection: str) -> str: + """ + Name the scenario set behind the deductions of one budget. + + The key identifies the entry in ``rcb_scenario_adjustments.yaml`` and the + ``lulucf_shift_median_{key}.csv`` file that notebook 104 writes. + + Parameters + ---------- + source : str + RCB source key in rcbs.yaml (e.g., "forster_2026") + label : str + Budget label (e.g., "1.7p50") + selection : str + Scenario selection rule of the source: "ar6-category" or + "peak-warming-band" + + Returns + ------- + str + - "ar6-category": the label itself (e.g., "1.5p50") + - "peak-warming-band": the temperature only (e.g., "peak-warming-1.7C"), + because one band serves every likelihood of its temperature + + Raises + ------ + ConfigurationError + If the rule is unknown, or the label has no AR6 category under the + "ar6-category" rule + """ + climate_assessment, _ = parse_rcb_scenario(label) + if selection == "ar6-category": + _ar6_category(source, label) + return label + if selection == "peak-warming-band": + return f"peak-warming-{climate_assessment}" + raise ConfigurationError( + f"RCB source '{source}': unknown scenario_selection '{selection}'. " + f"Expected 'ar6-category' or 'peak-warming-band'." + ) + + +def select_rcb_scenario_set( + metadata: pd.DataFrame, + source: str, + label: str, + selection: str, + band_half_width: float = DEFAULT_PEAK_WARMING_BAND_HALF_WIDTH, +) -> pd.DataFrame: + """ + Select the AR6 scenarios behind the deductions of one budget. + + Parameters + ---------- + metadata : pd.DataFrame + AR6 scenario metadata, one row per scenario, with the columns + "Category" and "Median peak warming (MAGICCv7.5.3)". The + "peak-warming-band" rule also needs "net_zero_year" (see + ``net_zero_year``), empty for a scenario that never reaches net zero. + source : str + RCB source key in rcbs.yaml (e.g., "forster_2026") + label : str + Budget label (e.g., "1.7p50") + selection : str + Scenario selection rule of the source: + + - "ar6-category": every scenario in the AR6 category mapped to the + label (1.5p50 -> C1, 2p83 -> C2, 2p66 -> C3) + - "peak-warming-band": every scenario whose median peak warming lies + in [T - band_half_width, T + band_half_width), where T is the + budget temperature, and that reaches net-zero CO2. A remaining + carbon budget runs to net-zero CO2, so a scenario that never reaches + it is excluded. The likelihood in the label plays no role. + band_half_width : float, optional + Half-width of the peak-warming band in degrees C (default: 0.05) + + Returns + ------- + pd.DataFrame + The selected rows of ``metadata`` + + Raises + ------ + ConfigurationError + If the rule is unknown, the label has no AR6 category under the + "ar6-category" rule, the label has no numeric temperature under the + "peak-warming-band" rule, or the metadata lack "net_zero_year" under it + DataProcessingError + If the rule selects no scenario + """ + climate_assessment, _ = parse_rcb_scenario(label) + if selection == "ar6-category": + category = _ar6_category(source, label) + selected = metadata[metadata["Category"] == category] + criterion = f"AR6 category {category}" + elif selection == "peak-warming-band": + try: + temperature = float(climate_assessment.removesuffix("C")) + except ValueError: + raise ConfigurationError( + f"RCB source '{source}', label '{label}': the peak-warming band " + f"rule needs a numeric temperature before 'p' (e.g. '1.5p50')." + ) from None + # Rounding keeps the bounds exact decimals (1.7 - 0.05 gives 1.65). + lower = round(temperature - band_half_width, 6) + upper = round(temperature + band_half_width, 6) + if NET_ZERO_YEAR_COLUMN not in metadata.columns: + raise ConfigurationError( + f"RCB source '{source}': the peak-warming band rule needs the " + f"scenario metadata column '{NET_ZERO_YEAR_COLUMN}'." + ) + peak_warming = metadata[PEAK_WARMING_COLUMN] + # A budget runs to net-zero CO2, so the band keeps scenarios that reach it. + selected = metadata[ + (peak_warming >= lower) + & (peak_warming < upper) + & metadata[NET_ZERO_YEAR_COLUMN].notna() + ] + criterion = ( + f"median peak warming in the {climate_assessment} band " + f"[{lower}, {upper}) and a net-zero CO2 year" + ) + else: + raise ConfigurationError( + f"RCB source '{source}': unknown scenario_selection '{selection}'. " + f"Expected 'ar6-category' or 'peak-warming-band'." + ) + + if selected.empty: + raise DataProcessingError( + f"RCB source '{source}', label '{label}': no scenario has {criterion}." + ) + return selected + + +def build_rcb_scenario_sets( + rcb_yaml: dict, metadata: pd.DataFrame +) -> dict[str, pd.DataFrame]: + """ + Select the scenario set for every budget in a loaded rcbs.yaml. + + Parameters + ---------- + rcb_yaml : dict + Content of rcbs.yaml. Each source under ``rcb_data`` may carry a + ``scenario_selection`` field (default: "ar6-category"). The top-level + ``peak_warming_band_half_width`` sets the band half-width (default: 0.05). + metadata : pd.DataFrame + AR6 scenario metadata (see ``select_rcb_scenario_set``) + + Returns + ------- + dict[str, pd.DataFrame] + Selected metadata rows, keyed by ``rcb_scenario_set_key`` + """ + band_half_width = rcb_yaml.get( + "peak_warming_band_half_width", DEFAULT_PEAK_WARMING_BAND_HALF_WIDTH + ) + scenario_sets = {} + for source, source_data in rcb_yaml["rcb_data"].items(): + selection = source_data.get("scenario_selection", DEFAULT_SCENARIO_SELECTION) + for label in source_data["scenarios"]: + key = rcb_scenario_set_key(source, label, selection) + if key not in scenario_sets: + scenario_sets[key] = select_rcb_scenario_set( + metadata, source, label, selection, band_half_width + ) + return scenario_sets + + +def validate_rebase_fill_max_years(value: object) -> int: + """Return ``rebase_fill_max_years`` from rcbs.yaml, or raise if it is not a count.""" + if isinstance(value, bool) or not isinstance(value, int) or value < 0: + raise ConfigurationError( + f"rcbs.yaml: rebase_fill_max_years must be a non-negative integer, " + f"got {value!r}." + ) + return value + + +def _available_years(emissions: pd.DataFrame) -> list[int]: + """List the years whose column exists and holds a value in every row.""" + return sorted( + int(c) + for c in emissions.columns + if str(c).isdigit() and emissions[c].notna().all() + ) + + +def _years_lacking(emissions: pd.DataFrame, years: range | list[int]) -> list[int]: + """List the years in ``years`` that ``emissions`` has no value for.""" + available = set(_available_years(emissions)) + return [year for year in years if year not in available] + + +def missing_rebase_years( + rcb_baseline_year: int, + emissions: pd.DataFrame, + target_baseline_year: int = 2020, +) -> list[int]: + """ + List the rebase years that an emissions timeseries lacks. + + Rebasing a budget from ``rcb_baseline_year`` to ``target_baseline_year`` + adds the emissions of every year from the target baseline to the year + before the RCB baseline. A year with a missing value counts as lacking. + """ + return _years_lacking(emissions, range(target_baseline_year, rcb_baseline_year)) + + +def fill_rebase_years( + emissions: pd.DataFrame, + rcb_baseline_year: int, + max_fill_years: int = DEFAULT_REBASE_FILL_MAX_YEARS, + series_name: str = "emissions", + source_name: str = "", +) -> pd.DataFrame: + """ + Hold the last observed value of a world series over the rebase years after it. + + The rebase of a budget to 2020 needs every year up to the year before the + RCB baseline. When the series ends earlier, each year after its last + observed year takes the last observed value. The fill is a placeholder + until observed data are published, and every use raises a warning. + + Parameters + ---------- + emissions : pd.DataFrame + Single-row world timeseries with string year columns + rcb_baseline_year : int + The year from which the RCB is calculated + max_fill_years : int, optional + Largest number of years to fill (default: 1). 0 turns the fill off. + series_name : str, optional + Name of the series for the warning (e.g., "world fossil CO2 emissions") + source_name : str, optional + Name of the RCB source for the warning + + Returns + ------- + pd.DataFrame + The series with the filled years, or the series unchanged when no + year is missing or more than ``max_fill_years`` years are missing. + Years before the last observed year are never filled. + """ + observed = emissions.iloc[0].dropna() + observed_years = [int(c) for c in observed.index if str(c).isdigit()] + if not observed_years: + return emissions + last_observed_year = max(observed_years) + + fill_years = list(range(last_observed_year + 1, rcb_baseline_year)) + if not fill_years or len(fill_years) > max_fill_years: + return emissions + + last_value = float(observed[str(last_observed_year)]) + filled = emissions.copy() + for year in fill_years: + filled[str(year)] = last_value + warnings.warn( + f"RCB source '{source_name}' (baseline year {rcb_baseline_year}): " + f"{series_name} end in {last_observed_year}. The rebase to 2020 uses the " + f"{last_observed_year} value ({last_value:,.1f} Mt CO2) for {fill_years}. " + f"This is a placeholder until observed data are published.", + stacklevel=2, + ) + return filled + + +def _year_ranges(years: list[int]) -> str: + """Format sorted years as ranges, e.g. [1990, 1991, 1995] -> '1990-1991, 1995'.""" + ranges: list[list[int]] = [] + for year in years: + if ranges and year == ranges[-1][1] + 1: + ranges[-1][1] = year + else: + ranges.append([year, year]) + return ", ".join( + str(first) if first == last else f"{first}-{last}" for first, last in ranges + ) + + def calculate_budget_from_rcb( rcb_value: float, allocation_year: int, @@ -102,19 +497,30 @@ def calculate_budget_from_rcb( ------- float Total budget to allocate in Mt CO2 + + Raises + ------ + DataProcessingError + If the world emissions lack any year from allocation_year to 2019 + (allocation_year before 2020) or from 2020 to allocation_year - 1 + (allocation_year after 2020) """ if allocation_year < 2020: # Add historical emissions before RCB period - year_cols = [ - str(y) - for y in range(allocation_year, 2020) - if str(y) in world_scenario_emissions_ts.columns - ] + year_cols = [str(y) for y in range(allocation_year, 2020)] - if not year_cols: + # A partial sum understates the budget, so every year must hold a value. + missing_years = _years_lacking( + world_scenario_emissions_ts, range(allocation_year, 2020) + ) + if missing_years: + years_with_data = _available_years(world_scenario_emissions_ts) + first_year = years_with_data[0] if years_with_data else None raise DataProcessingError( - f"No emission data found for years {allocation_year}-2019. " - f"Cannot calculate historical component of budget." + f"Allocation year {allocation_year} needs world emissions for " + f"{allocation_year}-2019, but the world emissions data lack " + f"{_year_ranges(missing_years)}. The first available year is " + f"{first_year}." ) historical_emissions = ( @@ -141,16 +547,20 @@ def calculate_budget_from_rcb( else: # allocation_year > 2020 # Subtract emissions already used from RCB - year_cols = [ - str(y) - for y in range(2020, allocation_year) - if str(y) in world_scenario_emissions_ts.columns - ] + year_cols = [str(y) for y in range(2020, allocation_year)] - if not year_cols: + # A partial sum overstates the budget, so every year must hold a value. + missing_years = _years_lacking( + world_scenario_emissions_ts, range(2020, allocation_year) + ) + if missing_years: + years_with_data = _available_years(world_scenario_emissions_ts) + last_year = years_with_data[-1] if years_with_data else None raise DataProcessingError( - f"No emission data found for years 2020-{allocation_year - 1}. " - f"Cannot calculate emissions already used from RCB." + f"Allocation year {allocation_year} needs world emissions for " + f"2020-{allocation_year - 1}, but the world emissions data lack " + f"{_year_ranges(missing_years)}. The last available year is " + f"{last_year}." ) emissions_used = world_scenario_emissions_ts[year_cols].sum(axis=1).iloc[0] @@ -172,7 +582,8 @@ def process_rcb_to_2020_baseline( rcb_baseline_year: int, emission_category: str, world_co2_ffi_emissions: pd.DataFrame, - actual_bm_lulucf_emissions: pd.DataFrame | None = None, + world_nghgi_lulucf_emissions: pd.DataFrame | None = None, + world_bunker_emissions: pd.DataFrame | None = None, bunkers_deduction_mt: float = 0.0, lulucf_future_deduction_mt: float = 0.0, lulucf_nghgi_correction_mt: float = 0.0, @@ -188,6 +599,12 @@ def process_rcb_to_2020_baseline( standardized 2020 baseline. It also applies adjustments for international bunkers and LULUCF following Weber et al. (2026). + The published RCB covers total anthropogenic CO2 from its baseline year and + includes international bunkers. The world CO2-FFI series excludes them, and + the bunker deduction covers 2020 to net zero. The rebase therefore adds + fossil emissions and bunker emissions for 2020 to (baseline_year - 1), so + each bunker year from 2020 is deducted exactly once. + The rebase always uses actual observational data (e.g. PRIMAP), never scenario projections. What enters the rebase depends on the emission category: @@ -196,18 +613,19 @@ def process_rcb_to_2020_baseline( ``lulucf_future_deduction_mt`` subtracts expected future (base→NZ) BM LULUCF, converting the published total-CO2 RCB to an FFI-only RCB. ``lulucf_nghgi_correction_mt`` is not used for this category. - - **co2**: Rebase uses fossil CO2 + actual bookkeeping-model LULUCF. - ``lulucf_nghgi_correction_mt`` applies the NGHGI-vs-BM convention gap - from Weber et al. (2026) to re-express the budget against national- - inventory accounting. ``lulucf_future_deduction_mt`` is not used for - this category; the budget retains both FFI and LULUCF. + - **co2**: Rebase uses fossil CO2 + observed LULUCF in the national- + inventory (NGHGI) convention. ``lulucf_nghgi_correction_mt`` applies + the NGHGI-vs-BM convention gap from the baseline year to net zero + (Weber et al. 2026), which re-expresses the published budget against + national-inventory accounting. ``lulucf_future_deduction_mt`` is not + used for this category; the budget retains both FFI and LULUCF. The calculation follows these steps: 1. Convert RCB from source unit to Mt * CO2e 2. If baseline_year > 2020: Add actual emissions from 2020 to - (baseline_year - 1) — fossil only for co2-ffi, fossil + BM LULUCF - for co2 - 3. Subtract bunkers deduction (always reduces budget) + (baseline_year - 1) — fossil + bunkers for co2-ffi, fossil + bunkers + + NGHGI LULUCF for co2 + 3. Subtract bunkers deduction from 2020 to net zero (always reduces budget) 4. Apply LULUCF deduction (sign-ready from caller) Sign convention for deduction parameters: @@ -217,8 +635,8 @@ def process_rcb_to_2020_baseline( (increases fossil budget), capped at 0 if caller applies a precautionary rule. Zero for co2. - lulucf_nghgi_correction_mt: sign-ready from caller (added directly). - For co2: the NGHGI-vs-BM convention gap from Weber et al. (2026), - typically negative. Zero for co2-ffi. + For co2: the NGHGI-vs-BM convention gap from the baseline year to + net zero (Weber et al. 2026), typically negative. Zero for co2-ffi. Parameters ---------- @@ -229,24 +647,30 @@ def process_rcb_to_2020_baseline( rcb_baseline_year : int The year from which the RCB is calculated (must be >= 2020) emission_category : str - Emission category: "co2-ffi" or "co2". Controls whether BM LULUCF - is included in the rebase. + Emission category: "co2-ffi" or "co2". Controls whether NGHGI + LULUCF is included in the rebase. world_co2_ffi_emissions : pd.DataFrame - World-level CO2-FFI emissions timeseries with year columns (in Mt * CO2e) - actual_bm_lulucf_emissions : pd.DataFrame or None, optional - Actual bookkeeping-model LULUCF CO2 emissions (e.g. PRIMAP), with year - columns (in Mt * CO2e). Used ONLY for the co2 rebase (default: None). + World-level CO2-FFI emissions timeseries with year columns (in Mt * CO2e). + Excludes international bunkers. + world_nghgi_lulucf_emissions : pd.DataFrame or None, optional + Observed world LULUCF CO2 emissions in the national-inventory (NGHGI) + convention (e.g. the Melo et al. world row), with year columns + (in Mt * CO2e). Used ONLY for the co2 rebase (default: None). + world_bunker_emissions : pd.DataFrame or None, optional + International bunker CO2 emissions timeseries with year columns + (in Mt * CO2e). Required when baseline_year > 2020 and + bunkers_deduction_mt is non-zero (default: None). bunkers_deduction_mt : float, optional - Total bunker CO2 emissions from 2020-2100 in Mt * CO2e (default: 0.0). - Always positive; subtracted from budget. + Total bunker CO2 emissions from 2020 to net zero in Mt * CO2e + (default: 0.0). Always positive; subtracted from budget. lulucf_future_deduction_mt : float, optional Projected future (2020/base → NZ) BM LULUCF adjustment in Mt * CO2e, sign-ready (default: 0.0). Non-zero for co2-ffi only, where it subtracts LULUCF to convert a total-CO2 RCB to FFI-only. lulucf_nghgi_correction_mt : float, optional - NGHGI-vs-BM convention correction in Mt * CO2e, sign-ready - (default: 0.0). Non-zero for co2 only, where it re-expresses the - budget from bookkeeping-model to NGHGI accounting. + NGHGI-vs-BM convention gap from the baseline year to net zero in + Mt * CO2e, sign-ready (default: 0.0). Non-zero for co2 only, where it + re-expresses the budget from bookkeeping-model to NGHGI accounting. target_baseline_year : int, optional Target baseline year for standardization (default: 2020) source_name : str, optional @@ -265,11 +689,13 @@ def process_rcb_to_2020_baseline( - 'rcb_original_unit': Original RCB unit - 'baseline_year': Original baseline year - 'rebase_total_mt': Emissions added to rebase from source year to 2020 - (positive, Mt * CO2e); fossil only for co2-ffi, fossil + actual BM - LULUCF for co2 + (positive, Mt * CO2e); fossil + bunkers for co2-ffi, fossil + bunkers + + observed NGHGI LULUCF for co2 - 'rebase_fossil_mt': Fossil-only component of rebase (Mt * CO2e) - - 'rebase_lulucf_mt': Actual BM LULUCF component of rebase (Mt * CO2e); - only non-zero for co2 + - 'rebase_bunkers_mt': International bunker component of rebase + (Mt * CO2e) + - 'rebase_lulucf_mt': Observed NGHGI LULUCF component of rebase + (Mt * CO2e); only non-zero for co2 - 'deduction_bunkers_mt': Bunker fuel deduction (negative, Mt * CO2e) - 'deduction_lulucf_future_mt': projected-LULUCF deduction applied to convert total-CO2 → FFI-only. Non-zero for co2-ffi, zero for co2. @@ -277,6 +703,12 @@ def process_rcb_to_2020_baseline( Non-zero for co2, zero for co2-ffi. - 'net_adjustment_mt': Total change from original to 2020 baseline (rebase + deductions + correction, Mt * CO2e) + + Raises + ------ + DataProcessingError + If a timeseries lacks a year of the rebase, or if baseline_year > 2020 + and bunkers are deducted without ``world_bunker_emissions`` """ # Get unit registry ureg = get_default_unit_registry() @@ -293,41 +725,82 @@ def process_rcb_to_2020_baseline( # Initialize rebase (baseline year shift) values rebase_total_mt = 0.0 rebase_fossil_mt = 0.0 + rebase_bunkers_mt = 0.0 rebase_lulucf_mt = 0.0 # Calculate emissions adjustment based on baseline year if rcb_baseline_year > target_baseline_year: # Need to add emissions from 2020 to (baseline_year - 1) - year_cols = [ - str(y) - for y in range(target_baseline_year, rcb_baseline_year) - if str(y) in world_co2_ffi_emissions.columns - ] + year_cols = [str(y) for y in range(target_baseline_year, rcb_baseline_year)] - if not year_cols: + # A partial rebase understates the budget, so every year must be present. + missing_years = missing_rebase_years( + rcb_baseline_year, world_co2_ffi_emissions, target_baseline_year + ) + if missing_years: raise DataProcessingError( - f"No CO2-FFI emission data found for years " - f"{target_baseline_year}-{rcb_baseline_year - 1}. " - f"Cannot calculate emissions adjustment for RCB conversion." + f"RCB source '{source_name}' has baseline year {rcb_baseline_year}. " + f"Rebasing to {target_baseline_year} needs CO2-FFI emissions for " + f"{target_baseline_year}-{rcb_baseline_year - 1}, but the " + f"emissions data lack {missing_years}." ) # Fossil rebase — always from actual PRIMAP data rebase_fossil_mt = world_co2_ffi_emissions[year_cols].sum(axis=1).iloc[0] - # BM LULUCF rebase — only for co2 (total CO2 needs LULUCF in rebase) + # Bunker rebase. The fossil series excludes bunkers and the deduction + # starts in 2020, so the rebase adds the bunkers of these years. + if world_bunker_emissions is not None: + missing_years = missing_rebase_years( + rcb_baseline_year, world_bunker_emissions, target_baseline_year + ) + if missing_years: + raise DataProcessingError( + f"RCB source '{source_name}' has baseline year " + f"{rcb_baseline_year}. Rebasing to {target_baseline_year} " + f"needs bunker emissions for " + f"{target_baseline_year}-{rcb_baseline_year - 1}, but the " + f"bunker data lack {missing_years}." + ) + rebase_bunkers_mt = world_bunker_emissions[year_cols].sum(axis=1).iloc[0] + elif bunkers_deduction_mt != 0: + raise DataProcessingError( + f"RCB source '{source_name}' has baseline year {rcb_baseline_year} " + f"and a bunker deduction from {target_baseline_year}. Rebasing " + f"needs the bunker emissions for " + f"{target_baseline_year}-{rcb_baseline_year - 1}: pass " + f"world_bunker_emissions." + ) + + # NGHGI LULUCF rebase — only for co2 (total CO2 needs LULUCF in rebase) # For co2-ffi, LULUCF is omitted because it cancels with the LULUCF # decomposition rebase_lulucf_mt = 0.0 - if emission_category == "co2" and actual_bm_lulucf_emissions is not None: - lulucf_year_cols = [ - y for y in year_cols if y in actual_bm_lulucf_emissions.columns - ] - if lulucf_year_cols: - rebase_lulucf_mt = ( - actual_bm_lulucf_emissions[lulucf_year_cols].sum(axis=1).iloc[0] + if emission_category == "co2" and world_nghgi_lulucf_emissions is None: + raise DataProcessingError( + f"RCB source '{source_name}' has baseline year {rcb_baseline_year}. " + f"Rebasing a co2 budget to {target_baseline_year} needs the " + f"observed inventory (NGHGI) LULUCF emissions for " + f"{target_baseline_year}-{rcb_baseline_year - 1}: pass " + f"world_nghgi_lulucf_emissions." + ) + if emission_category == "co2": + missing_years = missing_rebase_years( + rcb_baseline_year, world_nghgi_lulucf_emissions, target_baseline_year + ) + if missing_years: + raise DataProcessingError( + f"RCB source '{source_name}' has baseline year " + f"{rcb_baseline_year}. Rebasing to {target_baseline_year} " + f"needs LULUCF emissions for " + f"{target_baseline_year}-{rcb_baseline_year - 1}, but the " + f"emissions data lack {missing_years}." ) + rebase_lulucf_mt = ( + world_nghgi_lulucf_emissions[year_cols].sum(axis=1).iloc[0] + ) - rebase_total_mt = rebase_fossil_mt + rebase_lulucf_mt + rebase_total_mt = rebase_fossil_mt + rebase_bunkers_mt + rebase_lulucf_mt if verbose: print( @@ -339,9 +812,14 @@ def process_rcb_to_2020_baseline( f"({target_baseline_year}-{rcb_baseline_year - 1}): " f"+{rebase_fossil_mt:.1f} Mt * CO2e" ) + print( + f" Adding bunker emissions " + f"({target_baseline_year}-{rcb_baseline_year - 1}): " + f"+{rebase_bunkers_mt:.1f} Mt * CO2e" + ) if emission_category == "co2": print( - f" Adding actual BM LULUCF emissions " + f" Adding observed NGHGI LULUCF emissions " f"({target_baseline_year}-{rcb_baseline_year - 1}): " f"+{rebase_lulucf_mt:.1f} Mt * CO2e" ) @@ -386,7 +864,7 @@ def process_rcb_to_2020_baseline( if verbose: if bunkers_deduction_mt > 0: print( - f" Bunkers deduction (2020-2100): " + f" Bunkers deduction ({target_baseline_year}-NZ): " f"{deduction_bunkers_mt:.1f} Mt * CO2e" ) if deduction_lulucf_future_mt != 0: @@ -396,7 +874,7 @@ def process_rcb_to_2020_baseline( ) if correction_lulucf_nghgi_mt != 0: print( - f" LULUCF NGHGI correction: " + f" LULUCF NGHGI correction ({rcb_baseline_year}-NZ): " f"{correction_lulucf_nghgi_mt:.1f} Mt * CO2e" ) print( @@ -411,6 +889,7 @@ def process_rcb_to_2020_baseline( "baseline_year": rcb_baseline_year, "rebase_total_mt": round(rebase_total_mt), "rebase_fossil_mt": round(rebase_fossil_mt), + "rebase_bunkers_mt": round(rebase_bunkers_mt), "rebase_lulucf_mt": round(rebase_lulucf_mt), "deduction_bunkers_mt": round(deduction_bunkers_mt), "deduction_lulucf_future_mt": round(deduction_lulucf_future_mt), diff --git a/src/fair_shares/library/utils/math/allocation.py b/src/fair_shares/library/utils/math/allocation.py index dec05f5..f759c13 100644 --- a/src/fair_shares/library/utils/math/allocation.py +++ b/src/fair_shares/library/utils/math/allocation.py @@ -325,7 +325,8 @@ def calculate_gini_adjusted_gdp( Array of Gini coefficients (0-1) for each country. income_floor Development threshold in currency units per capita per year. - GDR default: $7,500/year (2010 PPP). + Must use the price base of ``total_gdps``. Baer (2013) gives $7,500 + on a 2005 base and $8,500 on a 2010 base. total_populations Array of total population values for each country. max_adjustment diff --git a/src/fair_shares/library/validation/config.py b/src/fair_shares/library/validation/config.py index 5c76648..ca81784 100644 --- a/src/fair_shares/library/validation/config.py +++ b/src/fair_shares/library/validation/config.py @@ -116,6 +116,7 @@ def validate_allocation_year_for_co2( allocations_config: dict[str, list[dict[str, Any]]], emission_category: str, nghgi_min_year: int | None = None, + responsibility_min_year: int | None = None, ) -> None: """ Enforce year parameters >= nghgi_min_year for LULUCF-containing categories. @@ -129,6 +130,11 @@ def validate_allocation_year_for_co2( approaches (``pre_allocation_responsibility_year``) to ensure methodological consistency. + The pre-allocation responsibility window reads its own emissions frame, + which for these categories can be the fossil CO2 series. When the caller + passes the first year of that frame as ``responsibility_min_year``, + ``pre_allocation_responsibility_year`` is checked against it instead. + Parameters ---------- allocations_config : dict[str, list[dict[str, Any]]] @@ -138,12 +144,18 @@ def validate_allocation_year_for_co2( nghgi_min_year : int or None Minimum year for NGHGI-consistent allocations (derived from data). If None, uses the default (currently 2000). + responsibility_min_year : int or None + First year of the emissions frame used for pre-allocation + responsibility. If None, ``pre_allocation_responsibility_year`` is + checked against the NGHGI minimum year. Raises ------ AllocationError - If allocation start year or pre_allocation_responsibility_year < nghgi_min_year - for LULUCF-containing emission categories ("co2", "all-ghg") + If allocation start year < nghgi_min_year, or + pre_allocation_responsibility_year is before the first year of its + emissions frame (nghgi_min_year when ``responsibility_min_year`` is + None), for LULUCF-containing emission categories ("co2", "all-ghg") """ # Resolve NGHGI min year min_year = nghgi_min_year if nghgi_min_year is not None else _NGHGI_MIN_YEAR_DEFAULT @@ -177,7 +189,16 @@ def validate_allocation_year_for_co2( hist_year if isinstance(hist_year, (list, tuple)) else [hist_year] ) for hy in hist_years: - if hy < min_year: + if responsibility_min_year is not None: + if hy < responsibility_min_year: + raise AllocationError( + f"Configuration error for approach '{approach}':\n" + f" pre_allocation_responsibility_year = {hy} is " + f"before {responsibility_min_year}, the first year " + f"of the emissions data used for pre-allocation " + f"responsibility." + ) + elif hy < min_year: _raise_nghgi_year_error( approach, "pre_allocation_responsibility_year", diff --git a/src/fair_shares/library/validation/models.py b/src/fair_shares/library/validation/models.py index 8144b2f..5ab8bec 100644 --- a/src/fair_shares/library/validation/models.py +++ b/src/fair_shares/library/validation/models.py @@ -54,13 +54,13 @@ class AllocationInputs(BaseModel): ) first_allocation_year: int = Field( ..., - ge=1900, + ge=1850, le=2100, description="First year to allocate (must exist in data)", ) last_allocation_year: int = Field( ..., - ge=1900, + ge=1850, le=2100, description="Last year to allocate (must exist in data)", ) @@ -68,7 +68,10 @@ class AllocationInputs(BaseModel): # Optional inputs for adjustments gdp_ts: TimeseriesDataFrame | None = Field( None, - description="GDP time series data (required for capability adjustments, used from allocation year onwards)", + description=( + "GDP time series data (required for capability adjustments, used from " + "allocation year onwards, or at capability_reference_year when set)" + ), ) gini_s: pd.DataFrame | None = Field( None, @@ -91,6 +94,12 @@ class AllocationInputs(BaseModel): description="Base year for pre-allocation responsibility calculation", ) + # Capability snapshot year (when applicable) + capability_reference_year: int | None = Field( + None, + description="Capability snapshot year (GDP is read at this year only)", + ) + model_config = {"arbitrary_types_allowed": True} @field_validator("population_ts") @@ -182,8 +191,9 @@ def validate_year_coverage(self) -> AllocationInputs: "Population", ) - # Check GDP if provided - only validate first year to match decorator behavior - if self.gdp_ts is not None: + # Check GDP at the first year only. A capability snapshot reads GDP at its + # reference year instead, which the allocation function checks. + if self.gdp_ts is not None and self.capability_reference_year is None: validate_year_in_data( self.first_allocation_year, self.gdp_ts, diff --git a/tests/fixtures/python_api/distributed_all-ghg-ex-co2-lulucf.parquet b/tests/fixtures/python_api/distributed_all-ghg-ex-co2-lulucf.parquet index dfa875a..e2534bf 100644 Binary files a/tests/fixtures/python_api/distributed_all-ghg-ex-co2-lulucf.parquet and b/tests/fixtures/python_api/distributed_all-ghg-ex-co2-lulucf.parquet differ diff --git a/tests/fixtures/python_api/distributed_all-ghg.parquet b/tests/fixtures/python_api/distributed_all-ghg.parquet index b7c34ca..9a20ad7 100644 Binary files a/tests/fixtures/python_api/distributed_all-ghg.parquet and b/tests/fixtures/python_api/distributed_all-ghg.parquet differ diff --git a/tests/fixtures/python_api/distributed_co2-ffi.parquet b/tests/fixtures/python_api/distributed_co2-ffi.parquet index 178e4c4..39d91f3 100644 Binary files a/tests/fixtures/python_api/distributed_co2-ffi.parquet and b/tests/fixtures/python_api/distributed_co2-ffi.parquet differ diff --git a/tests/fixtures/python_api_sources.py b/tests/fixtures/python_api_sources.py new file mode 100644 index 0000000..fa25c53 --- /dev/null +++ b/tests/fixtures/python_api_sources.py @@ -0,0 +1,27 @@ +"""Data sources and source-id derivation shared by the 601 fixture script and its test.""" + +from __future__ import annotations + +from fair_shares.library.utils.data.config import build_data_config + +# Must match the ``active_sources`` defaults in notebooks/601_reproduce_esabcc_2023.py. +SOURCES = { + "emissions_source": "primap-202503", + "gdp_source": "wdi-2025", + "population_source": "un-owid-2025", + "gini_source": "wdi-2025", + "lulucf_source": "melo-2026", +} + + +def source_id(category: str) -> str: + """Return the output folder id notebook 601 writes to for ``category``.""" + active_sources = { + "target": "rcbs", + "emissions": SOURCES["emissions_source"], + "gdp": SOURCES["gdp_source"], + "population": SOURCES["population_source"], + "gini": SOURCES["gini_source"], + "lulucf": SOURCES["lulucf_source"], + } + return build_data_config(category, active_sources)[1] diff --git a/tests/fixtures/rcb_golden_master.yaml b/tests/fixtures/rcb_golden_master.yaml new file mode 100644 index 0000000..bf29219 --- /dev/null +++ b/tests/fixtures/rcb_golden_master.yaml @@ -0,0 +1,122 @@ +# Golden master for the adjusted budgets of lamboll_2023, forster_2024 and ar6_2020. +# Inputs come from the pipeline outputs of primap-202503 with melo-2026 and cover +# 2020 to 2023 in Mt CO2. The test pins these values; do not regenerate. +# `expected` holds the budgets with the bunker rebase: the rebase adds the bunkers of +# 2020 to the year before the baseline (rebase_bunkers_mt). +# For co2 the rebase adds observed NGHGI LULUCF for the same years (rebase_lulucf_mt) +# and the convention gap runs from the baseline year of the source to net zero +# (`convention_gap_median_from`, keyed by baseline year). +# `rcb_2020_without_bunker_rebase_mt` holds the budgets captured 2026-10-01 from +# rcbs_co2-ffi.csv and rcbs_co2.csv, when the rebase added fossil emissions only. +# `rcb_2020_with_gap_from_2020_mt` holds the co2 budgets captured 2026-10-01 from +# rcbs_co2.csv, when every source took the convention gap from 2020 +# (`convention_gap_median_from_2020_captured_mt`). +inputs: + world_fossil_co2_mt: + 2020: 34343.1362 + 2021: 36068.3054 + 2022: 36228.508400000006 + 2023: 36737.4233 + world_co2_lulucf_mt: + 2020: -4121.601528 + 2021: -3756.03238 + 2022: -4209.201465 + 2023: -4274.392925 + bunkers_mt: + 2020: 870.9263603658384 + 2021: 911.739887821367 + 2022: 1039.5388075214562 + 2023: 1137.0650964616757 + scenario_adjustments: + 1.5p50: + bm_lulucf_cumulative_median: -53642.7 + convention_gap_median_from: {2020: -90280.9, 2023: -71238.1, 2024: -65410.0} + nz_year_median: 2050 + nz_year_min: 2034 + nz_year_max: 2075 + n_scenarios: 70 + n_reaching_nz: 70 + 2p83: + bm_lulucf_cumulative_median: -58710.7 + convention_gap_median_from: {2020: -119745.7, 2023: -100236.4, 2024: -94415.2} + nz_year_median: 2057 + nz_year_min: 2040 + nz_year_max: 2078 + n_scenarios: 106 + n_reaching_nz: 106 + 2p66: + bm_lulucf_cumulative_median: -87539.1 + convention_gap_median_from: {2020: -145477.8, 2023: -125793.3, 2024: -119566.0} + nz_year_median: 2070 + nz_year_min: 2044 + nz_year_max: 2100 + n_scenarios: 231 + n_reaching_nz: 191 + lulucf_shift_median_mt: + 1.5p50: + 2020: 2835.98316 + 2021: 2422.07858775 + 2022: 1649.3290775 + 2023: 1180.521057 + 2p83: + 2020: 2853.994815 + 2021: 2522.731611 + 2022: 1829.732873 + 2023: 1462.5375215 + 2p66: + 2020: 2901.91244 + 2021: 2524.902436 + 2022: 1961.33544 + 2023: 1621.06979 +expected: + co2-ffi: + - {source: lamboll_2023, scenario: 1.5p50, climate-assessment: 1.5C, quantile: 0.5, rcb_2020_nghgi_mt: 324802, rebase_fossil_mt: 106640, rebase_bunkers_mt: 2822, rebase_lulucf_mt: 0, deduction_bunkers_mt: -34660, deduction_lulucf_future_mt: 0, correction_lulucf_nghgi_mt: 0} + - {source: lamboll_2023, scenario: 2p66, climate-assessment: 2C, quantile: 0.66, rcb_2020_nghgi_mt: 992061, rebase_fossil_mt: 106640, rebase_bunkers_mt: 2822, rebase_lulucf_mt: 0, deduction_bunkers_mt: -57401, deduction_lulucf_future_mt: 0, correction_lulucf_nghgi_mt: 0} + - {source: forster_2024, scenario: 1.5p50, climate-assessment: 1.5C, quantile: 0.5, rcb_2020_nghgi_mt: 312677, rebase_fossil_mt: 143377, rebase_bunkers_mt: 3959, rebase_lulucf_mt: 0, deduction_bunkers_mt: -34660, deduction_lulucf_future_mt: 0, correction_lulucf_nghgi_mt: 0} + - {source: forster_2024, scenario: 2p66, climate-assessment: 2C, quantile: 0.66, rcb_2020_nghgi_mt: 989935, rebase_fossil_mt: 143377, rebase_bunkers_mt: 3959, rebase_lulucf_mt: 0, deduction_bunkers_mt: -57401, deduction_lulucf_future_mt: 0, correction_lulucf_nghgi_mt: 0} + - {source: forster_2024, scenario: 2p83, climate-assessment: 2C, quantile: 0.83, rcb_2020_nghgi_mt: 854717, rebase_fossil_mt: 143377, rebase_bunkers_mt: 3959, rebase_lulucf_mt: 0, deduction_bunkers_mt: -42619, deduction_lulucf_future_mt: 0, correction_lulucf_nghgi_mt: 0} + - {source: ar6_2020, scenario: 1.5p50, climate-assessment: 1.5C, quantile: 0.5, rcb_2020_nghgi_mt: 465340, rebase_fossil_mt: 0, rebase_bunkers_mt: 0, rebase_lulucf_mt: 0, deduction_bunkers_mt: -34660, deduction_lulucf_future_mt: 0, correction_lulucf_nghgi_mt: 0} + - {source: ar6_2020, scenario: 2p66, climate-assessment: 2C, quantile: 0.66, rcb_2020_nghgi_mt: 1092599, rebase_fossil_mt: 0, rebase_bunkers_mt: 0, rebase_lulucf_mt: 0, deduction_bunkers_mt: -57401, deduction_lulucf_future_mt: 0, correction_lulucf_nghgi_mt: 0} + - {source: ar6_2020, scenario: 2p83, climate-assessment: 2C, quantile: 0.83, rcb_2020_nghgi_mt: 857381, rebase_fossil_mt: 0, rebase_bunkers_mt: 0, rebase_lulucf_mt: 0, deduction_bunkers_mt: -42619, deduction_lulucf_future_mt: 0, correction_lulucf_nghgi_mt: 0} + co2: + - {source: lamboll_2023, scenario: 1.5p50, climate-assessment: 1.5C, quantile: 0.5, rcb_2020_nghgi_mt: 241477, rebase_fossil_mt: 106640, rebase_bunkers_mt: 2822, rebase_lulucf_mt: -12087, deduction_bunkers_mt: -34660, deduction_lulucf_future_mt: 0, correction_lulucf_nghgi_mt: -71238} + - {source: lamboll_2023, scenario: 2p66, climate-assessment: 2C, quantile: 0.66, rcb_2020_nghgi_mt: 854181, rebase_fossil_mt: 106640, rebase_bunkers_mt: 2822, rebase_lulucf_mt: -12087, deduction_bunkers_mt: -57401, deduction_lulucf_future_mt: 0, correction_lulucf_nghgi_mt: -125793} + - {source: forster_2024, scenario: 1.5p50, climate-assessment: 1.5C, quantile: 0.5, rcb_2020_nghgi_mt: 230905, rebase_fossil_mt: 143377, rebase_bunkers_mt: 3959, rebase_lulucf_mt: -16361, deduction_bunkers_mt: -34660, deduction_lulucf_future_mt: 0, correction_lulucf_nghgi_mt: -65410} + - {source: forster_2024, scenario: 2p66, climate-assessment: 2C, quantile: 0.66, rcb_2020_nghgi_mt: 854008, rebase_fossil_mt: 143377, rebase_bunkers_mt: 3959, rebase_lulucf_mt: -16361, deduction_bunkers_mt: -57401, deduction_lulucf_future_mt: 0, correction_lulucf_nghgi_mt: -119566} + - {source: forster_2024, scenario: 2p83, climate-assessment: 2C, quantile: 0.83, rcb_2020_nghgi_mt: 743941, rebase_fossil_mt: 143377, rebase_bunkers_mt: 3959, rebase_lulucf_mt: -16361, deduction_bunkers_mt: -42619, deduction_lulucf_future_mt: 0, correction_lulucf_nghgi_mt: -94415} + - {source: ar6_2020, scenario: 1.5p50, climate-assessment: 1.5C, quantile: 0.5, rcb_2020_nghgi_mt: 375059, rebase_fossil_mt: 0, rebase_bunkers_mt: 0, rebase_lulucf_mt: 0, deduction_bunkers_mt: -34660, deduction_lulucf_future_mt: 0, correction_lulucf_nghgi_mt: -90281} + - {source: ar6_2020, scenario: 2p66, climate-assessment: 2C, quantile: 0.66, rcb_2020_nghgi_mt: 947121, rebase_fossil_mt: 0, rebase_bunkers_mt: 0, rebase_lulucf_mt: 0, deduction_bunkers_mt: -57401, deduction_lulucf_future_mt: 0, correction_lulucf_nghgi_mt: -145478} + - {source: ar6_2020, scenario: 2p83, climate-assessment: 2C, quantile: 0.83, rcb_2020_nghgi_mt: 737635, rebase_fossil_mt: 0, rebase_bunkers_mt: 0, rebase_lulucf_mt: 0, deduction_bunkers_mt: -42619, deduction_lulucf_future_mt: 0, correction_lulucf_nghgi_mt: -119746} +rcb_2020_without_bunker_rebase_mt: + co2-ffi: + - {source: lamboll_2023, scenario: 1.5p50, rcb_2020_nghgi_mt: 321980} + - {source: lamboll_2023, scenario: 2p66, rcb_2020_nghgi_mt: 989239} + - {source: forster_2024, scenario: 1.5p50, rcb_2020_nghgi_mt: 308717} + - {source: forster_2024, scenario: 2p66, rcb_2020_nghgi_mt: 985976} + - {source: forster_2024, scenario: 2p83, rcb_2020_nghgi_mt: 850758} + - {source: ar6_2020, scenario: 1.5p50, rcb_2020_nghgi_mt: 465340} + - {source: ar6_2020, scenario: 2p66, rcb_2020_nghgi_mt: 1092599} + - {source: ar6_2020, scenario: 2p83, rcb_2020_nghgi_mt: 857381} + co2: + - {source: lamboll_2023, scenario: 1.5p50, rcb_2020_nghgi_mt: 219612} + - {source: lamboll_2023, scenario: 2p66, rcb_2020_nghgi_mt: 831674} + - {source: forster_2024, scenario: 1.5p50, rcb_2020_nghgi_mt: 202075} + - {source: forster_2024, scenario: 2p66, rcb_2020_nghgi_mt: 824137} + - {source: forster_2024, scenario: 2p83, rcb_2020_nghgi_mt: 714651} + - {source: ar6_2020, scenario: 1.5p50, rcb_2020_nghgi_mt: 375059} + - {source: ar6_2020, scenario: 2p66, rcb_2020_nghgi_mt: 947121} + - {source: ar6_2020, scenario: 2p83, rcb_2020_nghgi_mt: 737635} +rcb_2020_with_gap_from_2020_mt: + co2: + - {source: lamboll_2023, scenario: 1.5p50, rcb_2020_nghgi_mt: 222434} + - {source: lamboll_2023, scenario: 2p66, rcb_2020_nghgi_mt: 834496} + - {source: forster_2024, scenario: 1.5p50, rcb_2020_nghgi_mt: 206034} + - {source: forster_2024, scenario: 2p66, rcb_2020_nghgi_mt: 828096} + - {source: forster_2024, scenario: 2p83, rcb_2020_nghgi_mt: 718610} + - {source: ar6_2020, scenario: 1.5p50, rcb_2020_nghgi_mt: 375059} + - {source: ar6_2020, scenario: 2p66, rcb_2020_nghgi_mt: 947121} + - {source: ar6_2020, scenario: 2p83, rcb_2020_nghgi_mt: 737635} +convention_gap_median_from_2020_captured_mt: + 1.5p50: -90280.9 + 2p83: -119745.7 + 2p66: -145477.8 diff --git a/tests/fixtures/save_python_api_fixture.py b/tests/fixtures/save_python_api_fixture.py index af33782..a88e461 100644 --- a/tests/fixtures/save_python_api_fixture.py +++ b/tests/fixtures/save_python_api_fixture.py @@ -25,6 +25,7 @@ import sys import pandas as pd +from python_api_sources import source_id from fair_shares.library.paths import find_repo_root @@ -76,11 +77,16 @@ def main() -> None: for parquet in parquets: df = pd.read_parquet(parquet) category = str(df["category"].iloc[0]) + # Only the folder notebook 601 wrote for this category; others are stale runs. + if parquet.parents[2].name != source_id(category): + continue sliced = df[df["iso3c"].isin(EXAMPLE_COUNTRIES)].reset_index(drop=True) out = fixture_dir / f"distributed_{category}.parquet" sliced.to_parquet(out, index=False) print(f" {category}: {len(sliced)} rows -> {out.relative_to(project_root)}") saved.append(category) + if not saved: + raise SystemExit("no distributed parquet matched the notebook-601 source ids") (fixture_dir / "metadata.json").write_text( json.dumps( diff --git a/tests/fixtures/verified_dois.yaml b/tests/fixtures/verified_dois.yaml index 313c9a9..0202ca8 100644 --- a/tests/fixtures/verified_dois.yaml +++ b/tests/fixtures/verified_dois.yaml @@ -49,6 +49,14 @@ verified: publicationYear 2026, rights cc-by-4.0, access_right open. Confirms the record is the 3.1.1 release even though the file inside is named v3.1. + - doi: "10.5281/zenodo.22828743" + used_by: melo-2026-v4 + verified: 2026-10-01 + registrar: DataCite + evidence: >- + The LULUCF Data Hub, Melo et al. Zenodo record reports version 4.0.0, + published 2026-09-18, licence cc-by-4.0, access_right open. + - doi: "10.5281/zenodo.17845154" used_by: cmip7-historical-2025.12.07 verified: 2026-08-03 @@ -106,6 +114,25 @@ verified: Data 17, 965-1039, 2025, CC BY 4.0. This is the correct paper DOI; see the known_bad entry below for the one it replaces. + - doi: "10.5281/zenodo.17417124" + used_by: gcb-2025 + verified: 2026-10-01 + registrar: DataCite + evidence: >- + "The Global Carbon Project's fossil CO2 emissions dataset", Andrew and + Peters, Zenodo, version 2025v15, published 2025-10-22, rights cc-by-4.0, + access_right open. The record lists GCB2025v15_MtCO2_flat.csv at + 3,147,221 bytes with the md5 pinned in the registry. + + - doi: "10.5194/essd-18-3211-2026" + used_by: gcb-2025 + verified: 2026-10-01 + registrar: Crossref + evidence: >- + Friedlingstein et al., "Global Carbon Budget 2025", Earth System Science + Data 18, 3211-3288, issued 2026-05-13, CC BY 4.0. The paper behind the + dataset above. + - doi: "10.1038/s41558-023-01848-5" used_by: rcbs verified: 2026-08-03 @@ -124,6 +151,25 @@ verified: Science Data 16, 2625-2658, CC BY 4.0. The other published source behind the budget table. + - doi: "10.5194/essd-18-3889-2026" + used_by: rcbs + verified: 2026-10-01 + registrar: Crossref + evidence: >- + Forster et al., "Indicators of Global Climate Change 2025: annual update + of key indicators of the state of the climate system and human + influence", Earth System Science Data 18, 3889-3933, published + 2026-06-11. Source of the `forster_2026` budgets. + + - doi: "10.1017/9781009157896.001" + used_by: rcbs + verified: 2026-10-01 + registrar: Crossref + evidence: >- + "Summary for Policymakers", book chapter in "Climate Change 2021 – The + Physical Science Basis", Cambridge University Press, pages 3-32. Source + of the `ar6_wg1_2021` budgets (Table SPM.2). + # DOIs known to be wrong. The registry must never carry one of these. known_bad: - doi: "10.5194/essd-16-5567-2024" diff --git a/tests/integration/test_gcb_2025_coverage_rule.py b/tests/integration/test_gcb_2025_coverage_rule.py new file mode 100644 index 0000000..8930f37 --- /dev/null +++ b/tests/integration/test_gcb_2025_coverage_rule.py @@ -0,0 +1,116 @@ +"""The ``gcb-2025`` coverage rule on a real pipeline run, with start years from 1850. + +``gcb-2025`` keeps a country in the analysis only if the raw file records +emissions before 1990 and population is complete from 1850. Reads the tree of +a ``gcb-2025`` + ``co2-ffi`` + ``rcbs`` pipeline run and skips when it has not +been built (command in ``test_gcb_2025_smoke.py``). +""" + +from __future__ import annotations + +import pandas as pd +import pytest +import yaml +from pyprojroot import here + +from fair_shares.library.allocations.budgets.per_capita import ( + equal_per_capita_budget, + per_capita_adjusted_budget, +) +from fair_shares.library.notebook_helpers import load_allocation_data +from fair_shares.library.utils import ensure_string_year_columns + +RUN = here() / "output/gcb-2025_wdi-2025_un-owid-2025_wdi-2025_rcbs_co2-ffi" +PROCESSED = RUN / "intermediate/processed" +NO_RECORD_BEFORE_1990 = {"AND", "FSM", "LSO", "MHL", "NAM", "PLW", "TLS", "TUV"} +NO_POPULATION_FROM_1850 = {"MAC"} + + +@pytest.fixture(scope="module") +def data() -> dict: + if not (PROCESSED / "rcbs_co2-ffi.csv").is_file(): + pytest.skip("gcb-2025 processed data not built") + return load_allocation_data(PROCESSED, "rcbs", ["co2-ffi"], "co2-ffi") + + +def _world_row(folder: str, name: str, world_key: str) -> pd.Series: + frame = pd.read_csv(RUN / "intermediate" / folder / name).set_index("iso3c") + frame = ensure_string_year_columns(frame) + return frame.loc[world_key] + + +def test_the_nine_countries_are_in_rest_of_world_and_totals_close(data): + summary = pd.read_csv(PROCESSED / "country_data_coverage_summary.csv") + summary = summary.set_index("iso3c").fillna({"coverage_rule_failed": ""}) + # A country that the rule alone moves has complete data under the default check. + complete = summary["has_emissions"] & summary["has_gdp"] & summary["has_population"] + moved = summary[complete & (summary["coverage_rule_failed"] != "")] + assert set(moved.index) == NO_RECORD_BEFORE_1990 | NO_POPULATION_FROM_1850 + assert moved["in_row"].all() + reasons = moved["coverage_rule_failed"] + assert set(reasons[sorted(NO_RECORD_BEFORE_1990)]) == { + "no emissions recorded before 1990" + } + assert reasons["MAC"] == "population incomplete from 1850" + assert summary["in_analysis"].sum() == 168 + + emissions = data["emissions_data"]["co2-ffi"] + population = data["country_population_df"] + gdp = data["country_gdp_df"] + analysis = set(summary.index[summary["in_analysis"]]) | {"ROW"} + for frame in (emissions, population, gdp): + assert set(frame.index.get_level_values("iso3c")) == analysis + assert not frame.isna().any().any() + + # Countries + rest-of-world = world row, for every year of each dataset. + with open(RUN / "config.yaml") as f: + config = yaml.safe_load(f) + gdp_key = config["gdp"]["wdi-2025"]["data_parameters"]["world_key"] + population_key = config["population"]["un-owid-2025"]["data_parameters"][ + "historical_world_key" + ] + world_emissions = data["world_emissions_data"]["co2-ffi"].iloc[0] + world_population = _world_row( + "population", "population_timeseries.csv", population_key + ) + world_gdp = _world_row("gdp", "gdp_timeseries.csv", gdp_key) + for frame, world in ( + (emissions, world_emissions), + (population, world_population), + (gdp, world_gdp), + ): + total = frame.sum() + assert total.to_numpy() == pytest.approx( + world[total.index].to_numpy(dtype=float), rel=1e-9 + ) + + +@pytest.mark.parametrize("per_capita", [False, True]) +def test_responsibility_from_1850_gives_complete_shares(data, per_capita): + result = per_capita_adjusted_budget( + population_ts=data["country_population_df"], + country_actual_emissions_ts=data["emissions_data"]["co2-ffi"], + allocation_year=1990, + emission_category="co2-ffi", + pre_allocation_responsibility_weight=1.0, + capability_weight=0.0, + pre_allocation_responsibility_year=1850, + pre_allocation_responsibility_per_capita=per_capita, + ) + shares = result.relative_shares_cumulative_emission["1990"] + assert len(shares) == 169 + assert not shares.isna().any() + assert shares.sum() == pytest.approx(1.0, abs=1e-9) + + +def test_fixed_1850_population_shares_are_complete(data): + result = equal_per_capita_budget( + population_ts=data["country_population_df"], + allocation_year=1850, + emission_category="co2-ffi", + preserve_allocation_year_shares=True, + ) + shares = result.relative_shares_cumulative_emission["1850"] + assert len(shares) == 169 + assert not shares.isna().any() + assert shares.sum() == pytest.approx(1.0, abs=1e-9) diff --git a/tests/integration/test_gcb_2025_melo_v4_co2_smoke.py b/tests/integration/test_gcb_2025_melo_v4_co2_smoke.py new file mode 100644 index 0000000..09cca95 --- /dev/null +++ b/tests/integration/test_gcb_2025_melo_v4_co2_smoke.py @@ -0,0 +1,78 @@ +"""End-to-end smoke test: equal per capita ``co2`` budget on ``gcb-2025`` + ``melo-2026-v4``. + +``co2`` is ``gcb-2025`` fossil CO2 plus national-inventory LULUCF from the LULUCF +Data Hub v4.0.0. Reads the processed tree of that pipeline run and skips when it +has not been built:: + + uv run snakemake --cores 1 --config emission_category=co2 \ + active_emissions_source=gcb-2025 active_gdp_source=wdi-2025 \ + active_population_source=un-owid-2025 active_gini_source=wdi-2025 \ + active_lulucf_source=melo-2026-v4 active_target_source=rcbs +""" + +from __future__ import annotations + +import pandas as pd +import pytest +from pyprojroot import here + +from fair_shares.library.allocations.manager import run_allocation +from fair_shares.library.utils import ensure_string_year_columns +from fair_shares.library.utils.data.rcb import calculate_budget_from_rcb + +PROCESSED = ( + here() + / "output/gcb-2025_wdi-2025_un-owid-2025_wdi-2025_melo-2026-v4_rcbs_co2" + / "intermediate/processed" +) +# The inventory LULUCF series starts in 2000, and so does world co2. +ALLOCATION_YEAR = 2000 +EMISSIONS_INDEX = ["iso3c", "unit", "emission-category"] + + +def _read(name: str, index: list[str]) -> pd.DataFrame: + return ensure_string_year_columns(pd.read_csv(PROCESSED / name).set_index(index)) + + +@pytest.mark.parametrize("rcb_source", ["forster_2026", "ar6_wg1_2021"]) +def test_shares_sum_to_one_and_budgets_sum_to_the_adjusted_total(rcb_source): + if not (PROCESSED / "rcbs_co2.csv").is_file(): + pytest.skip("gcb-2025 + melo-2026-v4 co2 processed data not built") + rcbs = pd.read_csv(PROCESSED / "rcbs_co2.csv") + row = rcbs[(rcbs["source"] == rcb_source) & (rcbs["scenario"] == "1.7p50")] + if row.empty: + pytest.skip(f"{rcb_source} 1.7p50 is not in the processed budgets") + + population = _read("country_population_timeseries.csv", ["iso3c", "unit"]) + world = _read("world_emissions_co2_timeseries.csv", EMISSIONS_INDEX) + fossil = _read("world_emissions_co2-ffi_timeseries.csv", EMISSIONS_INDEX) + lulucf = _read("world_emissions_co2-lulucf_timeseries.csv", EMISSIONS_INDEX) + + # World co2 is fossil CO2 plus inventory LULUCF in every year it covers. + years = list(world.columns) + assert years[0] == str(ALLOCATION_YEAR) + assert world[years].to_numpy() == pytest.approx( + fossil[years].to_numpy() + lulucf[years].to_numpy() + ) + + result = run_allocation( + "equal-per-capita-budget", + population_ts=population, + allocation_year=ALLOCATION_YEAR, + emission_category="co2", + ) + shares = result.relative_shares_cumulative_emission[str(ALLOCATION_YEAR)] + assert shares.sum() == pytest.approx(1.0, abs=1e-9) + + # Adjusted total = world co2 2000-2019 + the budget rebased to 2020. + budget_2020 = row["rcb_2020_nghgi_mt"].iloc[0] + total = calculate_budget_from_rcb( + budget_2020, ALLOCATION_YEAR, world, verbose=False + ) + budget = pd.DataFrame({str(ALLOCATION_YEAR): [total]}, index=world.index) + absolute = result.get_absolute_budgets(budget)[str(ALLOCATION_YEAR)] + assert absolute.sum() == pytest.approx(total, rel=1e-9) + assert total > budget_2020 > 0 + # A post-2020 baseline carries the observed inventory LULUCF of the rebase years. + if row["baseline_year"].iloc[0] > 2020: + assert row["rebase_lulucf_mt"].iloc[0] != 0 diff --git a/tests/integration/test_gcb_2025_smoke.py b/tests/integration/test_gcb_2025_smoke.py new file mode 100644 index 0000000..a524f68 --- /dev/null +++ b/tests/integration/test_gcb_2025_smoke.py @@ -0,0 +1,64 @@ +"""End-to-end smoke test: equal per capita budget on ``gcb-2025`` emissions. + +Reads the processed tree of a ``gcb-2025`` + ``co2-ffi`` + ``rcbs`` pipeline run +and skips when it has not been built:: + + uv run snakemake --cores 1 --config emission_category=co2-ffi \ + active_emissions_source=gcb-2025 active_gdp_source=wdi-2025 \ + active_population_source=un-owid-2025 active_gini_source=wdi-2025 \ + active_target_source=rcbs +""" + +from __future__ import annotations + +import pandas as pd +import pytest +from pyprojroot import here + +from fair_shares.library.allocations.manager import run_allocation +from fair_shares.library.utils import ensure_string_year_columns +from fair_shares.library.utils.data.rcb import calculate_budget_from_rcb + +PROCESSED = ( + here() + / "output/gcb-2025_wdi-2025_un-owid-2025_wdi-2025_rcbs_co2-ffi" + / "intermediate/processed" +) +ALLOCATION_YEAR = 1990 + + +def _read(name: str, index: list[str]) -> pd.DataFrame: + return ensure_string_year_columns(pd.read_csv(PROCESSED / name).set_index(index)) + + +@pytest.mark.parametrize("rcb_source", ["forster_2026", "ar6_wg1_2021"]) +def test_shares_sum_to_one_and_budgets_sum_to_the_adjusted_total(rcb_source): + if not (PROCESSED / "rcbs_co2-ffi.csv").is_file(): + pytest.skip("gcb-2025 processed data not built") + rcbs = pd.read_csv(PROCESSED / "rcbs_co2-ffi.csv") + row = rcbs[(rcbs["source"] == rcb_source) & (rcbs["scenario"] == "1.7p50")] + if row.empty: + pytest.skip(f"{rcb_source} 1.7p50 is not in the processed budgets") + + population = _read("country_population_timeseries.csv", ["iso3c", "unit"]) + world = _read( + "world_emissions_co2-ffi_timeseries.csv", + ["iso3c", "unit", "emission-category"], + ) + result = run_allocation( + "equal-per-capita-budget", + population_ts=population, + allocation_year=ALLOCATION_YEAR, + emission_category="co2-ffi", + ) + shares = result.relative_shares_cumulative_emission[str(ALLOCATION_YEAR)] + assert shares.sum() == pytest.approx(1.0, abs=1e-9) + + # Adjusted total = world emissions 1990-2019 + the budget rebased to 2020. + total = calculate_budget_from_rcb( + row["rcb_2020_nghgi_mt"].iloc[0], ALLOCATION_YEAR, world, verbose=False + ) + budget = pd.DataFrame({str(ALLOCATION_YEAR): [total]}, index=world.index) + absolute = result.get_absolute_budgets(budget)[str(ALLOCATION_YEAR)] + assert absolute.sum() == pytest.approx(total, rel=1e-9) + assert total > row["rcb_2020_nghgi_mt"].iloc[0] > 0 diff --git a/tests/integration/test_python_api_reproduces_notebook.py b/tests/integration/test_python_api_reproduces_notebook.py index a232445..58a42f9 100644 --- a/tests/integration/test_python_api_reproduces_notebook.py +++ b/tests/integration/test_python_api_reproduces_notebook.py @@ -21,26 +21,28 @@ from __future__ import annotations +import importlib.util + import numpy as np import pandas as pd import pytest from pyprojroot import here from fair_shares.library.python_api import calculate_allocation_timeseries -from fair_shares.library.utils.data.config import build_data_config FIXTURE_DIR = here() / "tests/fixtures/python_api" EXAMPLE_COUNTRIES = ["DEU", "FRA", "USA", "CHN", "IND", "BRA", "POL"] +_spec = importlib.util.spec_from_file_location( + "python_api_sources", FIXTURE_DIR.parent / "python_api_sources.py" +) +_sources = importlib.util.module_from_spec(_spec) +_spec.loader.exec_module(_sources) +SOURCES, source_id = _sources.SOURCES, _sources.source_id + # The anchor + data sources the fixtures were saved for (notebook 601 defaults). +# The sources and source-id derivation are shared with the fixture script. ANCHOR = {"rcb_source": "ar6_2020", "climate_assessment": "1.5C", "quantile": 0.5} -SOURCES = { - "emissions_source": "primap-202503", - "gdp_source": "wdi-2025", - "population_source": "un-owid-2025", - "gini_source": "unu-wider-2025", - "lulucf_source": "melo-2026", -} # Columns that identify one series (exclude provenance + per-country value # columns, which the year-value comparison covers or which vary in null repr). @@ -165,16 +167,7 @@ def _require_processed_data(category: str) -> None: """Skip unless the processed input tree for this category has been built.""" - active_sources = { - "target": "rcbs", - "emissions": SOURCES["emissions_source"], - "gdp": SOURCES["gdp_source"], - "population": SOURCES["population_source"], - "gini": SOURCES["gini_source"], - "lulucf": SOURCES["lulucf_source"], - } - _, source_id = build_data_config(category, active_sources) - processed = here() / "output" / source_id / "intermediate" / "processed" + processed = here() / "output" / source_id(category) / "intermediate" / "processed" if not processed.is_dir(): pytest.skip( f"processed input data not built ({processed.relative_to(here())}); " diff --git a/tests/unit/test_coverage_rule.py b/tests/unit/test_coverage_rule.py new file mode 100644 index 0000000..a55e30e --- /dev/null +++ b/tests/unit/test_coverage_rule.py @@ -0,0 +1,106 @@ +"""The opt-in coverage rule of an emissions source. + +A source with a ``coverage`` block keeps a country in the analysis only if the +raw file records emissions before one year and population is complete from +another. A source without the block keeps the default membership. +""" + +from __future__ import annotations + +import pandas as pd +import pytest + +from fair_shares.library.config.models import EmissionsDataParameters +from fair_shares.library.exceptions import DataLoadingError +from fair_shares.library.preprocessing import ( + FIRST_RECORDED_YEAR_FILENAME, + compute_analysis_countries, + coverage_exclusions, + create_coverage_summary, + source_coverage_exclusions, +) + +YEARS = [str(y) for y in range(1850, 2001)] +COUNTRIES = ["AAA", "BBB", "CCC"] +ALL = set(COUNTRIES) +RULE = {"emissions_recorded_before": 1990, "population_from": 1850} +# AAA passes both tests. BBB is first recorded in 1992. CCC has population from 1950. +FIRST_RECORDED = pd.Series({"AAA": 1900, "BBB": 1992, "CCC": 1850}) + + +def _frame(index_names: list[str], extra: tuple[str, ...]) -> pd.DataFrame: + return pd.DataFrame( + 1.0, + index=pd.MultiIndex.from_tuples( + [(c, *extra) for c in COUNTRIES], names=index_names + ), + columns=YEARS, + ) + + +def _population() -> pd.DataFrame: + population = _frame(["iso3c", "unit"], ("million",)) + population.loc["CCC", [str(y) for y in range(1850, 1950)]] = None + return population + + +def _inputs(): + emissions = _frame(["iso3c", "unit", "emission-category"], ("Mt", "co2-ffi")) + gdp = _frame(["iso3c", "unit"], ("billion",)) + return {"co2-ffi": emissions}, gdp, _population() + + +def test_a_source_without_the_block_keeps_the_default_membership(): + assert coverage_exclusions(ALL, _population(), None, FIRST_RECORDED) == {} + assert compute_analysis_countries(*_inputs()) == ALL + + +def test_each_failing_country_is_excluded_with_its_reason(): + excluded = coverage_exclusions(ALL, _population(), RULE, FIRST_RECORDED) + assert excluded == { + "BBB": "no emissions recorded before 1990", + "CCC": "population incomplete from 1850", + } + countries = compute_analysis_countries( + *_inputs(), coverage=RULE, first_recorded_year=FIRST_RECORDED + ) + assert countries == {"AAA"} + + +def test_a_record_in_the_threshold_year_or_no_record_fails(): + first = pd.Series({"AAA": 1989, "BBB": 1990}) + rule = {"emissions_recorded_before": 1990} + excluded = coverage_exclusions(ALL, _population(), rule, first) + assert set(excluded) == {"BBB", "CCC"} + with pytest.raises(DataLoadingError, match="first recorded year"): + coverage_exclusions(ALL, _population(), rule) + + +def test_the_source_rule_reads_the_table_and_the_summary_names_the_reason(tmp_path): + parameters = {"available_categories": ["co2-ffi"], "world_key": "WLD"} + population = _population() + assert source_coverage_exclusions(ALL, population, parameters, tmp_path) == {} + + # The rule passes through the config model that the pipeline dumps. + parameters = EmissionsDataParameters(**parameters, coverage=RULE).model_dump() + with pytest.raises(DataLoadingError, match="table not found"): + source_coverage_exclusions(ALL, population, parameters, tmp_path) + table = FIRST_RECORDED.rename("first_recorded_year").rename_axis("iso3c") + table.reset_index().to_csv( + tmp_path / FIRST_RECORDED_YEAR_FILENAME.format(category="co2-ffi"), index=False + ) + excluded = source_coverage_exclusions(ALL, population, parameters, tmp_path) + assert set(excluded) == {"BBB", "CCC"} + + emissions, gdp, population = _inputs() + gini = pd.DataFrame( + {"gini": [0.3]}, + index=pd.MultiIndex.from_tuples([("AAA", "unitless")], names=["iso3c", "unit"]), + ) + mapping = pd.DataFrame({"iso3c": COUNTRIES}) + args = ({"AAA"}, emissions, gdp, population, gini, mapping, tmp_path) + assert "coverage_rule_failed" not in create_coverage_summary(*args).columns + summary = create_coverage_summary(*args, coverage_rule_failed=excluded) + summary = summary.set_index("iso3c") + assert summary["coverage_rule_failed"].to_dict() == {"AAA": "", **excluded} + assert summary["in_row"].to_dict() == {"AAA": False, "BBB": True, "CCC": True} diff --git a/tests/unit/test_early_start_years.py b/tests/unit/test_early_start_years.py new file mode 100644 index 0000000..29f94a6 --- /dev/null +++ b/tests/unit/test_early_start_years.py @@ -0,0 +1,327 @@ +"""Budget allocations from start years before the GDP series. + +Population and fossil CO2 start in 1850; GDP starts in 1990. A capability +snapshot (``capability_reference_year``) needs GDP at the reference year only, +so the adjusted budget approaches run from 1850. Year-by-year capability still +needs GDP at the allocation year. +""" + +from __future__ import annotations + +import numpy as np +import pandas as pd +import pytest +from pydantic import ValidationError + +from fair_shares.library.allocations import run_parameter_grid +from fair_shares.library.allocations.budgets import ( + equal_per_capita_budget, + per_capita_adjusted_budget, + per_capita_adjusted_gini_budget, +) +from fair_shares.library.exceptions import AllocationError, DataProcessingError + +COUNTRIES = ["AAA", "BBB", "CCC", "DDD"] +POP_YEARS = [1850, 1900, 1950, 1990, 2000, 2015, 2020, 2023, 2050] +GDP_YEARS = [1990, 2000, 2015, 2020, 2023] +EMISSION_YEARS = [1850, 1900, 1950, 1990, 2000, 2015, 2020, 2023] +# GDP per capita in 2020 (USD): two countries below a 7500 floor, two above. +GDP_PER_CAPITA_2020 = [2000.0, 6000.0, 25000.0, 60000.0] + + +def _frames() -> dict[str, pd.DataFrame]: + """Population from 1850, GDP from 1990, fossil emissions from 1850.""" + population = pd.DataFrame( + { + str(y): [(10.0 + 5.0 * i) * (1.0 + 0.004 * (y - 1850)) for i in range(4)] + for y in POP_YEARS + }, + index=pd.MultiIndex.from_product( + [COUNTRIES, ["million"]], names=["iso3c", "unit"] + ), + ) + # GDP in million USD so that GDP / population (million) is USD per capita. + gdp = pd.DataFrame( + { + str(y): [ + GDP_PER_CAPITA_2020[i] + * (1.0 + 0.02 * (i + 1) * (y - 2020) / 10.0) + * population[str(y)].iloc[i] + for i in range(4) + ] + for y in GDP_YEARS + }, + index=pd.MultiIndex.from_product( + [COUNTRIES, ["million"]], names=["iso3c", "unit"] + ), + ) + emissions = pd.DataFrame( + { + str(y): [(1.0 + 3.0 * i) * (1.0 + 0.01 * (y - 1850)) for i in range(4)] + for y in EMISSION_YEARS + }, + index=pd.MultiIndex.from_product( + [COUNTRIES, ["Mt * CO2e"], ["co2-ffi"]], + names=["iso3c", "unit", "emission-category"], + ), + ) + gini = pd.DataFrame({"iso3c": COUNTRIES, "gini": [0.30, 0.45, 0.35, 0.40]}) + return {"population": population, "gdp": gdp, "emissions": emissions, "gini": gini} + + +def _shares(result) -> pd.Series: + df = result.relative_shares_cumulative_emission + return df.iloc[:, 0].droplevel(["unit", "emission-category"]) + + +def _adjusted(f, **kwargs) -> pd.Series: + return _shares( + per_capita_adjusted_budget( + population_ts=f["population"], + gdp_ts=f["gdp"], + emission_category="co2-ffi", + **kwargs, + ) + ) + + +def _gini(f, **kwargs) -> pd.Series: + return _shares( + per_capita_adjusted_gini_budget( + population_ts=f["population"], + gdp_ts=f["gdp"], + gini_s=f["gini"], + emission_category="co2-ffi", + income_floor=7500.0, + **kwargs, + ) + ) + + +def test_equal_per_capita_budget_runs_from_1850(): + f = _frames() + shares = _shares( + equal_per_capita_budget( + population_ts=f["population"], + allocation_year=1850, + emission_category="co2-ffi", + ) + ) + cumulative_population = f["population"].sum(axis=1).droplevel("unit") + expected = cumulative_population / cumulative_population.sum() + np.testing.assert_allclose(shares.to_numpy(), expected.to_numpy(), rtol=1e-12) + + +def test_allocation_year_before_1850_raises(): + f = _frames() + with pytest.raises(ValidationError, match="greater than or equal to 1850"): + equal_per_capita_budget( + population_ts=f["population"], + allocation_year=1849, + emission_category="co2-ffi", + ) + + +@pytest.mark.parametrize("allocation_year", [1850, 1990]) +@pytest.mark.parametrize("run", [_adjusted, _gini]) +def test_capability_snapshot_runs_from_early_start_years(run, allocation_year): + shares = run( + _frames(), + allocation_year=allocation_year, + capability_weight=1.0, + capability_reference_year=2020, + ) + assert shares.notna().all() + assert shares.sum() == pytest.approx(1.0, abs=1e-12) + # The poorest country gains over its equal per capita share. + assert shares["AAA"] > 10.0 / 70.0 + + +def test_snapshot_with_a_window_that_ends_before_the_gdp_series(): + shares = _adjusted( + _frames(), + allocation_year=1850, + cumulative_end_year=1950, + capability_weight=1.0, + capability_reference_year=2020, + ) + assert shares.sum() == pytest.approx(1.0, abs=1e-12) + + +def test_year_by_year_capability_before_the_gdp_series_raises(): + with pytest.raises(DataProcessingError, match="Required years not found in GDP"): + _adjusted(_frames(), allocation_year=1850, capability_weight=1.0) + + +def test_reference_year_outside_the_gdp_series_raises(): + with pytest.raises(AllocationError, match="outside the GDP data range"): + _adjusted( + _frames(), + allocation_year=1850, + capability_weight=1.0, + capability_reference_year=1950, + ) + + +def test_capability_only_shares_do_not_depend_on_the_weight(): + """Weights are normalised to their sum, so a lone capability weight acts as 1.""" + kwargs = dict(allocation_year=1850, capability_reference_year=2020) + pd.testing.assert_series_equal( + _adjusted(_frames(), capability_weight=0.3, **kwargs), + _adjusted(_frames(), capability_weight=1.0, **kwargs), + ) + + +def test_responsibility_only_run_with_a_reference_year_does_not_read_gdp(): + """The grid passes GDP to every adjusted run, also when capability is off.""" + f = _frames() + kwargs = dict( + allocation_year=1950, + country_actual_emissions_ts=f["emissions"], + pre_allocation_responsibility_weight=1.0, + pre_allocation_responsibility_year=1850, + capability_reference_year=2020, + ) + without_gdp = _shares( + per_capita_adjusted_budget( + population_ts=f["population"], emission_category="co2-ffi", **kwargs + ) + ) + pd.testing.assert_series_equal(_adjusted(f, **kwargs), without_gdp) + + +# Shares from the code before the snapshot change (allocation years inside the +# GDP series), in COUNTRIES order. +PINNED = { + "adjusted, 1990, year-by-year": ( + dict(allocation_year=1990, capability_weight=1.0), + [ + 0.5282854161360591, + 0.27415035675052307, + 0.11178139292044748, + 0.08578283419297027, + ], + ), + "adjusted, 1990, snapshot 2020": ( + dict( + allocation_year=1990, capability_weight=1.0, capability_reference_year=2020 + ), + [ + 0.5343436171274648, + 0.2728272964878631, + 0.1095039618388284, + 0.08332512454584375, + ], + ), + "adjusted, 2015, snapshot 2000, responsibility from 1850": ( + dict( + allocation_year=2015, + capability_weight=0.5, + capability_reference_year=2000, + pre_allocation_responsibility_weight=0.5, + pre_allocation_responsibility_year=1850, + ), + [ + 0.5054500871798192, + 0.23403826224142912, + 0.1392553692665111, + 0.12125628131224056, + ], + ), + "gini, 1990, year-by-year": ( + dict(allocation_year=1990), + [ + 0.6557159057156675, + 0.2719315511156184, + 0.04220255322242418, + 0.030149989946289923, + ], + ), + "gini, 1990, snapshot 2020": ( + dict(allocation_year=1990, capability_reference_year=2020), + [ + 0.6677068995500234, + 0.2625082675903675, + 0.04075970083585096, + 0.029025132023758147, + ], + ), + "gini, 2015, snapshot 2000": ( + dict(allocation_year=2015, capability_reference_year=2000), + [ + 0.6346141754907716, + 0.288667408408946, + 0.04466689143303458, + 0.032051524667247644, + ], + ), +} + + +@pytest.mark.parametrize("case", PINNED) +def test_results_inside_the_gdp_series_are_unchanged(case): + kwargs, expected = PINNED[case] + f = _frames() + if "pre_allocation_responsibility_year" in kwargs: + kwargs = {**kwargs, "country_actual_emissions_ts": f["emissions"]} + run = _gini if case.startswith("gini") else _adjusted + shares = run(f, **kwargs) + np.testing.assert_allclose(shares.loc[COUNTRIES].to_numpy(), expected, rtol=1e-12) + + +RESPONSIBILITY_FROM_1850 = { + "per-capita-adjusted-budget": [ + { + "allocation-year": 2015, + "pre-allocation-responsibility-weight": 1.0, + "capability-weight": 0.0, + "pre-allocation-responsibility-year": 1850, + } + ] +} + + +class TestResponsibilityWindowForCo2: + """For ``co2`` the responsibility frame is fossil CO2, which starts in 1850.""" + + @staticmethod + def _co2_frame(f) -> pd.DataFrame: + """Composite ``co2`` emissions, available from 2000 only.""" + return ( + f["emissions"] + .loc[:, "2000":] + .rename(index={"co2-ffi": "co2"}, level="emission-category") + ) + + def _grid_shares(self, f, responsibility_emissions_ts) -> pd.Series: + (result,) = run_parameter_grid( + allocations_config=RESPONSIBILITY_FROM_1850, + population_ts=f["population"], + country_actual_emissions_ts=self._co2_frame(f), + responsibility_emissions_ts=responsibility_emissions_ts, + emission_category="co2", + ) + return _shares(result) + + def test_window_from_1850_passes_with_the_fossil_frame(self): + f = _frames() + base = self._grid_shares(f, f["emissions"]) + assert base.sum() == pytest.approx(1.0, abs=1e-12) + + # The window is [1850, 2015): 1850 counts, 2015 does not. + changed_1850 = f["emissions"].copy() + changed_1850.loc["AAA", "1850"] = 500.0 + assert (self._grid_shares(f, changed_1850) - base).abs().max() > 1e-6 + + changed_2015 = f["emissions"].copy() + changed_2015.loc["AAA", "2015"] = 500.0 + pd.testing.assert_series_equal(self._grid_shares(f, changed_2015), base) + + def test_window_before_the_frame_raises(self): + f = _frames() + with pytest.raises(AllocationError, match="is before 1990, the first year"): + self._grid_shares(f, f["emissions"].loc[:, "1990":]) + + def test_without_a_separate_frame_the_nghgi_floor_stands(self): + with pytest.raises(AllocationError, match="is before 2000"): + self._grid_shares(_frames(), None) diff --git a/tests/unit/test_emissions_gcb_2025.py b/tests/unit/test_emissions_gcb_2025.py new file mode 100644 index 0000000..0dface3 --- /dev/null +++ b/tests/unit/test_emissions_gcb_2025.py @@ -0,0 +1,252 @@ +"""The opt-in Global Carbon Budget 2025 emissions source (``gcb-2025``).""" + +from __future__ import annotations + +from types import SimpleNamespace + +import pandas as pd +import pytest +import yaml + +from fair_shares.library import citations, paths +from fair_shares.library.config.models import ( + DataSourcesConfig, + EmissionsDataParameters, +) +from fair_shares.library.data_registry import load_registry, normalise_target +from fair_shares.library.exceptions import ConfigurationError +from fair_shares.library.preprocessing import coverage_exclusions +from fair_shares.library.utils import add_row_timeseries +from fair_shares.library.utils.data.config import ( + get_bunkers_source, + get_emission_preprocessing_categories, +) +from fair_shares.library.utils.data.gcb import ( + AVIATION, + GLOBAL, + ISO, + NAME, + OIL_FIRES, + SHIPPING, + VALUE, + YEAR, + gcb_bunkers, + gcb_first_recorded_year, + gcb_fossil_co2, +) + +KEY = "gcb-2025" +# Analysis countries under the default rule with no GCB record before 1990. +NO_RECORD_BEFORE_1990 = {"AND", "FSM", "LSO", "MHL", "NAM", "PLW", "TLS", "TUV"} +ALL_CATEGORIES = [ + "co2-ffi", + "co2", + "co2-lulucf", + "non-co2", + "all-ghg", + "all-ghg-ex-co2-lulucf", +] + + +def _config() -> dict: + raw = yaml.safe_load( + paths.packaged_config("data_sources/data_sources_unified.yaml").read_text( + encoding="utf-8" + ) + ) + return raw["emissions"][KEY] + + +def _long(rows: dict[tuple[str, str | None], dict[int, float | None]]) -> pd.DataFrame: + """Build a GCB-shaped long frame from {(name, iso): {year: value}}.""" + return pd.DataFrame( + [ + {NAME: name, ISO: iso, YEAR: year, VALUE: value} + for (name, iso), series in rows.items() + for year, value in series.items() + ] + ) + + +def test_registry_entry_is_opt_in_and_matches_config(): + source = load_registry()[KEY] + assert source.tier == "optional" + assert source.doi == "10.5281/zenodo.17417124" + assert source.license == "CC-BY-4.0" + (download,) = source.downloads + assert normalise_target(_config()["path"]) == download.target + + +def test_a_co2_ffi_only_source_is_asked_for_no_other_category(): + available = _config()["data_parameters"]["available_categories"] + assert available == ["co2-ffi"] + for target in ("rcbs", "rcb-pathways"): + requested = get_emission_preprocessing_categories(target, "co2-ffi", available) + assert requested == ("co2-ffi",) + # A source that declares every category is asked for the LULUCF primitives too. + assert get_emission_preprocessing_categories("rcbs", "co2-ffi", ALL_CATEGORIES) == ( + "all-ghg-ex-co2-lulucf", + "co2-ffi", + "co2-lulucf", + ) + # The config entry validates without a historical scenario. + assert EmissionsDataParameters(**_config()["data_parameters"]).scenario is None + + +def test_co2_is_valid_for_a_co2_ffi_only_source_with_a_lulucf_source(): + """``co2`` = ``co2-ffi`` + inventory LULUCF, so it needs an active LULUCF source.""" + available = _config()["data_parameters"]["available_categories"] + + def config(category: str, lulucf: str | None) -> SimpleNamespace: + parameters = SimpleNamespace(available_categories=available) + return SimpleNamespace( + active_emissions_source=KEY, + emission_category=category, + active_lulucf_source=lulucf, + emissions={KEY: SimpleNamespace(data_parameters=parameters)}, + ) + + validate = DataSourcesConfig.validate_emission_category + validate(config("co2", "melo-2026-v4")) + with pytest.raises(ConfigurationError, match="when a LULUCF source is active"): + validate(config("co2", None)) + with pytest.raises(ConfigurationError, match="only provides: co2-ffi"): + validate(config("all-ghg", "melo-2026-v4")) + # The pipeline still asks the emissions source for co2-ffi alone. + assert get_emission_preprocessing_categories("rcbs", "co2", available) == ( + "co2-ffi", + ) + + +def test_bunkers_follow_the_emissions_source(): + assert get_bunkers_source(KEY) == KEY + assert get_bunkers_source("primap-202503") == "gcb-2024" + run = {"target": "rcbs", "emissions": KEY, "gdp": "wdi-2025"} + names = citations.resolve_source_names(run, emission_category="co2-ffi") + assert KEY in names + assert "gcb-2024" not in names + + +def test_blank_years_special_rows_and_world_row(): + """Synthetic file: two countries, one unmapped code, one row without a code.""" + raw = _long( + { + ("Aland", "AAA"): {2000: None, 2001: None, 2002: 5.0, 2003: 6.0}, + # 1999 precedes first_year and is dropped. + ("Beland", "BBB"): { + 1999: 9.0, + 2000: 10.0, + 2001: None, + 2002: 12.0, + 2003: 13.0, + }, + ("Unmapped", "UUU"): {2000: 1.0, 2001: 1.0, 2002: 1.0, 2003: 1.0}, + ("Some Islands", None): {2000: 0.5, 2001: 0.5, 2002: None, 2003: None}, + ("International Shipping", SHIPPING): {2000: 3, 2001: 3, 2002: 3, 2003: 3}, + ("International Aviation", AVIATION): {2000: 2, 2001: 2, 2002: 2, 2003: 2}, + (OIL_FIRES, None): {2000: None, 2001: 7.0, 2002: 0.0, 2003: 0.0}, + ("Global", GLOBAL): {2000: 16.5, 2001: 13.5, 2002: 23.0, 2003: 25.0}, + } + ) + emissions, excluded = gcb_fossil_co2(raw, {"AAA", "BBB"}, "WLD", first_year=2000) + + values = emissions.droplevel(["unit", "emission-category"]) + assert list(values.index) == ["AAA", "BBB", "WLD"] + assert list(values.columns) == ["2000", "2001", "2002", "2003"] + assert emissions.index.get_level_values("unit").unique().tolist() == ["Mt * CO2e"] + # Blanks count as zero, before the first record and inside a series. + assert values.loc["AAA"].tolist() == [0.0, 0.0, 5.0, 6.0] + assert values.loc["BBB"].tolist() == [10.0, 0.0, 12.0, 13.0] + # World row = Global - shipping - aviation - oil fires. + assert values.loc["WLD"].tolist() == [11.5, 1.5, 18.0, 20.0] + # The unmapped code and the row without a code stay in the world row only. + rest = values.loc["WLD"] - values.loc[["AAA", "BBB"]].sum() + assert rest.tolist() == [1.5, 1.5, 1.0, 1.0] + + assert excluded.loc[OIL_FIRES].tolist() == [0.0, 7.0, 0.0, 0.0] + assert gcb_bunkers(raw, first_year=2000).loc["bunkers"].tolist() == [5.0] * 4 + + +def test_world_closure_on_the_real_file(): + """Countries + other rows + bunkers + oil fires = GCB Global, 1850-2024.""" + config = _config() + # Another test seeds the cached data directory with a temp path. + paths.reset_path_cache() + path = paths.resolve_source_path(config["path"]) + if not path.exists(): + pytest.skip("GCB 2025 file not fetched") + raw = pd.read_csv(path) + world_key = config["data_parameters"]["world_key"] + every_code = set(raw[ISO].dropna()) - {GLOBAL, SHIPPING, AVIATION} + + emissions, excluded = gcb_fossil_co2(raw, every_code, world_key) + values = emissions.droplevel(["unit", "emission-category"]) + world = values.loc[world_key] + assert list(values.columns) == [str(y) for y in range(1850, 2025)] + assert not values.isna().any().any() + + raw = raw[raw[YEAR] >= 1850] + global_total = raw[raw[ISO] == GLOBAL].set_index(YEAR)[VALUE] + global_total.index = global_total.index.astype(str) + no_code = raw[raw[ISO].isna() & (raw[NAME] != OIL_FIRES)] + no_code = no_code.groupby(YEAR)[VALUE].sum().reindex(range(1850, 2025)).fillna(0) + no_code.index = no_code.index.astype(str) + + # The file rounds to six decimals, so sums over 220 rows agree to about 1e-5. + countries = values.drop(index=world_key).sum() + assert (countries + no_code - world).abs().max() < 1e-4 + assert (world + excluded.sum() - global_total).abs().max() < 1e-9 + bunkers = gcb_bunkers(raw).loc["bunkers"] + assert (bunkers - excluded.loc[[SHIPPING, AVIATION]].sum()).abs().max() == 0 + assert excluded.loc[OIL_FIRES, "1991"] == pytest.approx(477.924832) + + +def test_coverage_rule_on_the_real_file_moves_late_records_to_rest_of_world(): + """Countries without a record before 1990 join rest-of-world; the world closes.""" + config = _config() + assert config["data_parameters"]["coverage"] == { + "emissions_recorded_before": 1990, + "population_from": 1850, + } + paths.reset_path_cache() + path = paths.resolve_source_path(config["path"]) + if not path.exists(): + pytest.skip("GCB 2025 file not fetched") + raw = pd.read_csv(path) + world_key = config["data_parameters"]["world_key"] + every_code = set(raw[ISO].dropna()) - {GLOBAL, SHIPPING, AVIATION} + emissions, _ = gcb_fossil_co2(raw, every_code, world_key) + + first = gcb_first_recorded_year(raw, every_code) + assert first[sorted(NO_RECORD_BEFORE_1990)].to_dict() == { + "AND": 1990, + "FSM": 1992, + "LSO": 1990, + "MHL": 1992, + "NAM": 1991, + "PLW": 1992, + "TLS": 1994, + "TUV": 1990, + } + # Their zero-filled 1850-1989 series is all zero, which the rule keeps out. + early = [str(y) for y in range(1850, 1990)] + values = emissions.droplevel(["unit", "emission-category"]) + assert (values.loc[sorted(NO_RECORD_BEFORE_1990), early] == 0).all().all() + + rule = {"emissions_recorded_before": 1990} + excluded = coverage_exclusions(every_code, pd.DataFrame(), rule, first) + assert NO_RECORD_BEFORE_1990 <= set(excluded) + analysis = every_code - set(excluded) + assert (values.loc[sorted(analysis), early].sum(axis=1) > 0).all() + + # Rest-of-world absorbs the moved countries: countries + ROW = world row. + index = ["iso3c", "unit", "emission-category"] + world = emissions[emissions.index.get_level_values("iso3c") == world_key] + complete = add_row_timeseries(emissions, analysis, world, index, verbose=False) + codes = complete.index.get_level_values("iso3c") + assert set(codes) == analysis | {"ROW"} + assert (complete.sum() - world.iloc[0]).abs().max() < 1e-6 + # Its increase equals the emissions of the excluded countries. + before = add_row_timeseries(emissions, every_code, world, index, verbose=False) + gain = complete[codes == "ROW"].iloc[0] - before.xs("ROW", level="iso3c").iloc[0] + assert (gain - values.loc[sorted(excluded)].sum()).abs().max() < 1e-9 diff --git a/tests/unit/test_import_order.py b/tests/unit/test_import_order.py new file mode 100644 index 0000000..48f9259 --- /dev/null +++ b/tests/unit/test_import_order.py @@ -0,0 +1,26 @@ +"""Each package module imports cleanly as the first import of a fresh interpreter.""" + +import subprocess +import sys + +import pytest + +MODULES = [ + "fair_shares", + "fair_shares.library.config", + "fair_shares.library.config.models", + "fair_shares.library.utils", + "fair_shares.library.utils.data.config", + "fair_shares.library.validation", +] + + +@pytest.mark.parametrize("module", MODULES) +def test_module_imports_first(module): + """A circular import shows up only when the module loads first.""" + result = subprocess.run( + [sys.executable, "-c", f"import {module}"], + capture_output=True, + text=True, + ) + assert result.returncode == 0, result.stderr diff --git a/tests/unit/test_lulucf_melo_v4.py b/tests/unit/test_lulucf_melo_v4.py new file mode 100644 index 0000000..529b456 --- /dev/null +++ b/tests/unit/test_lulucf_melo_v4.py @@ -0,0 +1,62 @@ +"""The opt-in LULUCF Data Hub v4.0.0 source (``melo-2026-v4``).""" + +from __future__ import annotations + +import pandas as pd +import pytest +import yaml + +from fair_shares.library import paths +from fair_shares.library.data_registry import load_registry, normalise_target + +KEY = "melo-2026-v4" + + +def _config() -> dict: + raw = yaml.safe_load( + paths.packaged_config("data_sources/data_sources_unified.yaml").read_text( + encoding="utf-8" + ) + ) + return raw["lulucf"][KEY] + + +def test_registry_entry_is_opt_in_and_matches_config(): + source = load_registry()[KEY] + assert source.tier == "optional" + assert source.doi == "10.5281/zenodo.22828743" + assert source.license == "CC-BY-4.0" + (download,) = source.downloads + assert len(download.sha256) == 64 + assert normalise_target(_config()["path"]) == download.target + + +def test_world_row_and_years_come_from_the_gap_filled_file(): + config = _config() + params = config["data_parameters"] + # Another test seeds the cached data directory with a temp path. + paths.reset_path_cache() + path = paths.resolve_source_path(config["path"]) + if not path.exists(): + pytest.skip("v4.0.0 LULUCF file not fetched") + + raw = pd.read_csv(path) + lulucf = raw[ + (raw[params["iso3_column"]] != "EU27") + & (raw["Category"] == params["category_filter"]) + & (raw["Gas"] == params["gas_filter"]) + ] + wide = lulucf.pivot_table( + index=params["iso3_column"], + columns=params["year_column"], + values=params["value_column"], + aggfunc="sum", + ) + assert list(wide.columns) == list(range(2000, 2025)) + expected = raw[ + (raw[params["iso3_column"]] == params["world_key"]) + & (raw["Category"] == "LULUCF") + ].set_index(params["year_column"])[params["value_column"]] + pd.testing.assert_series_equal( + wide.loc[params["world_key"]], expected, check_names=False + ) diff --git a/tests/unit/test_nghgi_corrections.py b/tests/unit/test_nghgi_corrections.py index 7ce1673..859c145 100644 --- a/tests/unit/test_nghgi_corrections.py +++ b/tests/unit/test_nghgi_corrections.py @@ -284,6 +284,19 @@ def test_single_year(self, bunker_ts): ) assert result == pytest.approx(1000.0) # 2020 only + def test_end_year_comes_from_the_data(self): + """The last observed year is used as observed; later years take its rate.""" + observed_to_2024 = _make_timeseries( + {2020: 800.0, 2021: 900.0, 2022: 1000.0, 2023: 1100.0, 2024: 1200.0}, + row_label="bunkers", + ) + result = compute_bunker_deduction( + bunker_ts=observed_to_2024, start_year=2020, net_zero_year=2030 + ) + observed = 800.0 + 900.0 + 1000.0 + 1100.0 + 1200.0 + extrapolated = 1200.0 * (2030 - 2024) + assert result == pytest.approx(observed + extrapolated) + def test_shorter_nz_year_smaller_deduction(self, bunker_ts): """Using a shorter net-zero year reduces the total deduction.""" result_short = compute_bunker_deduction( @@ -388,10 +401,15 @@ def test_net_adjustment_matches_components(self, ffi_emissions): rcb_baseline_year=2023, emission_category="co2-ffi", world_co2_ffi_emissions=ffi_emissions, + world_bunker_emissions=_make_timeseries( + {2020: 800.0, 2021: 900.0, 2022: 1000.0} + ), bunkers_deduction_mt=47_000.0, lulucf_future_deduction_mt=20_000.0, verbose=False, ) + assert result["rebase_bunkers_mt"] == 2_700 + assert result["rebase_total_mt"] == 27_300 + 2_700 expected_total = ( result["rebase_total_mt"] + result["deduction_bunkers_mt"] @@ -417,6 +435,7 @@ def test_output_dict_has_required_keys(self, ffi_emissions): "baseline_year", "rebase_total_mt", "rebase_fossil_mt", + "rebase_bunkers_mt", "rebase_lulucf_mt", "deduction_bunkers_mt", "deduction_lulucf_future_mt", @@ -427,20 +446,20 @@ def test_output_dict_has_required_keys(self, ffi_emissions): # --------------------------------------------------------------------------- -# process_rcb_to_2020_baseline — co2 category with actual BM LULUCF rebase +# process_rcb_to_2020_baseline — co2 category with NGHGI LULUCF rebase # --------------------------------------------------------------------------- class TestProcessRcbTo2020BaselineWithCo2Rebase: - """Tests for co2 emission category including actual BM LULUCF in rebase.""" + """Tests for co2 emission category including observed NGHGI LULUCF in rebase.""" @pytest.fixture def ffi_emissions(self) -> pd.DataFrame: return _make_ffi_emissions({2020: 9000.0, 2021: 9100.0, 2022: 9200.0}) @pytest.fixture - def actual_bm_lulucf(self) -> pd.DataFrame: - """Actual BM LULUCF emissions: 3000 MtCO2/yr for 2020-2022.""" + def nghgi_lulucf(self) -> pd.DataFrame: + """Observed NGHGI LULUCF emissions: 3000 to 3200 MtCO2/yr for 2020-2022.""" index = pd.MultiIndex.from_tuples( [("World", "Mt * CO2e", "co2-lulucf")], names=["iso3c", "unit", "emission-category"], @@ -448,15 +467,15 @@ def actual_bm_lulucf(self) -> pd.DataFrame: data = {"2020": [3000.0], "2021": [3100.0], "2022": [3200.0]} return pd.DataFrame(data, index=index) - def test_co2_rebase_includes_bm_lulucf(self, ffi_emissions, actual_bm_lulucf): - """co2 rebase includes fossil + actual BM LULUCF.""" + def test_co2_rebase_includes_nghgi_lulucf(self, ffi_emissions, nghgi_lulucf): + """co2 rebase includes fossil + observed NGHGI LULUCF.""" result = process_rcb_to_2020_baseline( rcb_value=400.0, rcb_unit="Gt * CO2", rcb_baseline_year=2023, emission_category="co2", world_co2_ffi_emissions=ffi_emissions, - actual_bm_lulucf_emissions=actual_bm_lulucf, + world_nghgi_lulucf_emissions=nghgi_lulucf, bunkers_deduction_mt=0.0, lulucf_future_deduction_mt=0.0, verbose=False, @@ -469,15 +488,15 @@ def test_co2_rebase_includes_bm_lulucf(self, ffi_emissions, actual_bm_lulucf): assert result["rebase_lulucf_mt"] == round(expected_lulucf) assert result["rebase_total_mt"] == round(expected_total_shift) - def test_co2ffi_rebase_excludes_bm_lulucf(self, ffi_emissions, actual_bm_lulucf): - """co2-ffi rebase uses fossil only, even when actual BM LULUCF is provided.""" + def test_co2ffi_rebase_excludes_nghgi_lulucf(self, ffi_emissions, nghgi_lulucf): + """co2-ffi rebase uses fossil only, even when NGHGI LULUCF is provided.""" result_ffi = process_rcb_to_2020_baseline( rcb_value=400.0, rcb_unit="Gt * CO2", rcb_baseline_year=2023, emission_category="co2-ffi", world_co2_ffi_emissions=ffi_emissions, - actual_bm_lulucf_emissions=actual_bm_lulucf, + world_nghgi_lulucf_emissions=nghgi_lulucf, verbose=False, ) assert result_ffi["rebase_lulucf_mt"] == 0 @@ -485,7 +504,7 @@ def test_co2ffi_rebase_excludes_bm_lulucf(self, ffi_emissions, actual_bm_lulucf) assert result_ffi["rebase_fossil_mt"] == round(expected_fossil) assert result_ffi["rebase_total_mt"] == round(expected_fossil) - def test_co2_rebase_larger_than_ffi_only(self, ffi_emissions, actual_bm_lulucf): + def test_co2_rebase_larger_than_ffi_only(self, ffi_emissions, nghgi_lulucf): """co2 rebase (fossil + LULUCF) produces larger rcb_2020_nghgi_mt than co2-ffi.""" result_ffi = process_rcb_to_2020_baseline( rcb_value=400.0, @@ -501,13 +520,13 @@ def test_co2_rebase_larger_than_ffi_only(self, ffi_emissions, actual_bm_lulucf): rcb_baseline_year=2023, emission_category="co2", world_co2_ffi_emissions=ffi_emissions, - actual_bm_lulucf_emissions=actual_bm_lulucf, + world_nghgi_lulucf_emissions=nghgi_lulucf, verbose=False, ) assert result_co2["rcb_2020_nghgi_mt"] > result_ffi["rcb_2020_nghgi_mt"] def test_baseline_2020_no_rebase_regardless_of_category( - self, ffi_emissions, actual_bm_lulucf + self, ffi_emissions, nghgi_lulucf ): """When baseline == 2020, no rebase needed regardless of emission category.""" result = process_rcb_to_2020_baseline( @@ -516,14 +535,14 @@ def test_baseline_2020_no_rebase_regardless_of_category( rcb_baseline_year=2020, emission_category="co2", world_co2_ffi_emissions=ffi_emissions, - actual_bm_lulucf_emissions=actual_bm_lulucf, + world_nghgi_lulucf_emissions=nghgi_lulucf, verbose=False, ) assert result["rebase_fossil_mt"] == 0 assert result["rebase_lulucf_mt"] == 0 assert result["rebase_total_mt"] == 0 - def test_provenance_fields_present(self, ffi_emissions, actual_bm_lulucf): + def test_provenance_fields_present(self, ffi_emissions, nghgi_lulucf): """All provenance fields must be present in output.""" result = process_rcb_to_2020_baseline( rcb_value=400.0, @@ -531,7 +550,7 @@ def test_provenance_fields_present(self, ffi_emissions, actual_bm_lulucf): rcb_baseline_year=2023, emission_category="co2", world_co2_ffi_emissions=ffi_emissions, - actual_bm_lulucf_emissions=actual_bm_lulucf, + world_nghgi_lulucf_emissions=nghgi_lulucf, verbose=False, ) assert "rebase_fossil_mt" in result @@ -570,60 +589,41 @@ def nghgi_ts(self) -> pd.DataFrame: {1990: -400.0, 2000: -500.0, 2020: -600.0}, row_label="nghgi_lulucf" ) - @pytest.fixture - def bunker_ts(self) -> pd.DataFrame: - return _make_timeseries( - {1980: 100.0, 1990: 200.0, 2000: 300.0, 2020: 400.0}, row_label="bunkers" - ) - - def test_pre_nghgi_years_are_nan(self, fossil_ts, nghgi_ts, bunker_ts): + def test_pre_nghgi_years_are_nan(self, fossil_ts, nghgi_ts): """Years outside NGHGI coverage should be NaN (no BM fallback).""" result = build_nghgi_world_co2_timeseries( fossil_ts=fossil_ts, nghgi_ts=nghgi_ts, - bunker_ts=bunker_ts, ) # 1980: no NGHGI data -> NaN assert pd.isna(result["1980"].iloc[0]) - def test_nghgi_years_use_nghgi(self, fossil_ts, nghgi_ts, bunker_ts): - """Years with NGHGI data should use NGHGI LULUCF.""" + def test_world_co2_is_fossil_plus_nghgi_lulucf(self, fossil_ts, nghgi_ts): + """World CO2 = fossil + NGHGI LULUCF. The fossil series excludes bunkers.""" result = build_nghgi_world_co2_timeseries( fossil_ts=fossil_ts, nghgi_ts=nghgi_ts, - bunker_ts=bunker_ts, ) - # 1990: fossil(6000) + NGHGI(-400) - bunkers(200) = 5400 - assert result["1990"].iloc[0] == pytest.approx(5400.0) - # 2000: fossil(7000) + NGHGI(-500) - bunkers(300) = 6200 - assert result["2000"].iloc[0] == pytest.approx(6200.0) + # 1990: fossil(6000) + NGHGI(-400) = 5600 + assert result["1990"].iloc[0] == pytest.approx(5600.0) + # 2000: fossil(7000) + NGHGI(-500) = 6500 + assert result["2000"].iloc[0] == pytest.approx(6500.0) + # 2020: fossil(9000) + NGHGI(-600) = 8400 + assert result["2020"].iloc[0] == pytest.approx(8400.0) - def test_bunkers_subtracted(self, fossil_ts, nghgi_ts, bunker_ts): - """Bunker emissions are subtracted in years with NGHGI data.""" - result = build_nghgi_world_co2_timeseries( - fossil_ts=fossil_ts, - nghgi_ts=nghgi_ts, - bunker_ts=bunker_ts, - ) - # Years with NGHGI data should have bunkers subtracted - for y in ["1990", "2000", "2020"]: - assert result[y].iloc[0] < fossil_ts[y].iloc[0] - - def test_emission_category_label_is_co2(self, fossil_ts, nghgi_ts, bunker_ts): + def test_emission_category_label_is_co2(self, fossil_ts, nghgi_ts): """Output emission-category label should be 'co2'.""" result = build_nghgi_world_co2_timeseries( fossil_ts=fossil_ts, nghgi_ts=nghgi_ts, - bunker_ts=bunker_ts, ) assert result.index.get_level_values("emission-category")[0] == "co2" - def test_index_structure_matches_fossil_ts(self, fossil_ts, nghgi_ts, bunker_ts): + def test_index_structure_matches_fossil_ts(self, fossil_ts, nghgi_ts): """Output index structure matches fossil_ts (same names, same iso3c/unit).""" result = build_nghgi_world_co2_timeseries( fossil_ts=fossil_ts, nghgi_ts=nghgi_ts, - bunker_ts=bunker_ts, ) assert result.index.names == fossil_ts.index.names assert result.index.get_level_values("iso3c")[0] == "World" @@ -672,7 +672,7 @@ def rcb_adj_sink(self) -> dict[str, dict]: return { "1.5p50": { "bm_lulucf_cumulative_median": -3100.0, - "convention_gap_median": -5000.0, + "convention_gap_median_from": {2020: -5000.0}, "nz_year_median": 2050, "n_scenarios": 10, } @@ -684,7 +684,7 @@ def rcb_adj_source(self) -> dict[str, dict]: return { "1.5p50": { "bm_lulucf_cumulative_median": 3100.0, - "convention_gap_median": -5000.0, + "convention_gap_median_from": {2020: -5000.0}, "nz_year_median": 2050, "n_scenarios": 10, } @@ -842,7 +842,7 @@ def rcb_adj(self) -> dict[str, dict]: return { "1.5p50": { "bm_lulucf_cumulative_median": 7750.0, - "convention_gap_median": -2000.0, + "convention_gap_median_from": {2020: -2000.0}, "nz_year_median": 2050, "n_scenarios": 10, } diff --git a/tests/unit/test_notebook_106_nghgi_co2.py b/tests/unit/test_notebook_106_nghgi_co2.py index 5b5d99a..9bd12c2 100644 --- a/tests/unit/test_notebook_106_nghgi_co2.py +++ b/tests/unit/test_notebook_106_nghgi_co2.py @@ -1,10 +1,9 @@ """ -Tests for BUG 7 fix: NGHGI-consistent world CO2 in notebook 106. +Tests for the NGHGI-consistent world CO2 in notebook 106. Verifies that when emission_category=co2, notebook 106 constructs the world -emissions timeseries using the 3-parameter build_nghgi_world_co2_timeseries() -from nghgi.py (fossil, nghgi, bunker), NOT a simplified co2_ffi + -co2_lulucf sum. +emissions timeseries with build_nghgi_world_co2_timeseries() from nghgi.py: +fossil CO2 (which excludes international bunkers) plus NGHGI LULUCF. These tests use synthetic data to validate the notebook's NGHGI loading logic in isolation, without requiring actual data files. @@ -41,28 +40,16 @@ def _make_nghgi_ts(year_values: dict[int, float]) -> pd.DataFrame: ) -def _make_bunker_ts(year_values: dict[int, float]) -> pd.DataFrame: - """Build a single-row bunker timeseries (GCB format).""" - return pd.DataFrame( - [[year_values[y] for y in sorted(year_values)]], - columns=[str(y) for y in sorted(year_values)], - index=pd.Index(["bunkers"], name="source"), - ) - - # --------------------------------------------------------------------------- # Test: build_nghgi_world_co2_timeseries is used for co2 category # --------------------------------------------------------------------------- class TestNotebook106NghgiCo2: - """Verify notebook 106 uses the 3-parameter build_nghgi_world_co2_timeseries.""" + """Verify the world CO2 that notebook 106 builds for the co2 category.""" - def test_co2_uses_build_nghgi_world_co2_timeseries(self): - """When emission_category=co2, the notebook must call - build_nghgi_world_co2_timeseries with 3 parameters (fossil, nghgi, - bunker), not a simplified 2-parameter version. - """ + def test_co2_is_fossil_plus_nghgi_lulucf(self): + """World CO2 = fossil + LULUCF(NGHGI). The fossil series excludes bunkers.""" from fair_shares.library.utils.data.nghgi import ( build_nghgi_world_co2_timeseries, ) @@ -72,68 +59,18 @@ def test_co2_uses_build_nghgi_world_co2_timeseries(self): {1990: 6000.0, 2000: 7000.0, 2020: 9000.0}, "co2-ffi" ) nghgi_ts = _make_nghgi_ts({1990: -400.0, 2000: -500.0, 2020: -600.0}) - bunker_ts = _make_bunker_ts({1990: 200.0, 2000: 300.0, 2020: 400.0}) - - result = build_nghgi_world_co2_timeseries( - fossil_ts=fossil_ts, - nghgi_ts=nghgi_ts, - bunker_ts=bunker_ts, - ) - - # Verify the formula: total CO2 = fossil - bunkers + LULUCF(NGHGI) - # 1990: 6000 - 200 + (-400) = 5400 - assert result["1990"].iloc[0] == pytest.approx(5400.0) - # 2000: 7000 - 300 + (-500) = 6200 - assert result["2000"].iloc[0] == pytest.approx(6200.0) - # 2020: 9000 - 400 + (-600) = 8000 - assert result["2020"].iloc[0] == pytest.approx(8000.0) - - def test_co2_result_includes_bunker_deduction(self): - """The NGHGI world CO2 must include bunker deduction. - - A simplified co2_ffi + co2_lulucf would miss the bunker deduction - entirely, producing higher emissions than correct. - """ - from fair_shares.library.utils.data.nghgi import ( - build_nghgi_world_co2_timeseries, - ) - - fossil_ts = _make_world_emissions({2020: 10000.0}, "co2-ffi") - nghgi_ts = _make_nghgi_ts({2020: -500.0}) - bunker_ts = _make_bunker_ts({2020: 600.0}) - - result = build_nghgi_world_co2_timeseries( - fossil_ts=fossil_ts, - nghgi_ts=nghgi_ts, - bunker_ts=bunker_ts, - ) - - # With bunkers: 10000 - 600 + (-500) = 8900 - expected_with_bunkers = 8900.0 - # Without bunkers (the bug): 10000 + (-500) = 9500 - wrong_without_bunkers = 9500.0 - - assert result["2020"].iloc[0] == pytest.approx(expected_with_bunkers) - assert result["2020"].iloc[0] != pytest.approx(wrong_without_bunkers) - - def test_co2_result_uses_nghgi_lulucf(self): - """The NGHGI world CO2 must use Melo NGHGI LULUCF where available.""" - from fair_shares.library.utils.data.nghgi import ( - build_nghgi_world_co2_timeseries, - ) - - fossil_ts = _make_world_emissions({2000: 7000.0}, "co2-ffi") - nghgi_ts = _make_nghgi_ts({2000: -800.0}) - bunker_ts = _make_bunker_ts({2000: 200.0}) result = build_nghgi_world_co2_timeseries( fossil_ts=fossil_ts, nghgi_ts=nghgi_ts, - bunker_ts=bunker_ts, ) - # With NGHGI: 7000 - 200 + (-800) = 6000 - assert result["2000"].iloc[0] == pytest.approx(6000.0) + # 1990: 6000 + (-400) = 5600 + assert result["1990"].iloc[0] == pytest.approx(5600.0) + # 2000: 7000 + (-500) = 6500 + assert result["2000"].iloc[0] == pytest.approx(6500.0) + # 2020: 9000 + (-600) = 8400 + assert result["2020"].iloc[0] == pytest.approx(8400.0) def test_co2_result_category_label_is_co2(self): """The result emission-category label should be 'co2'.""" @@ -143,12 +80,10 @@ def test_co2_result_category_label_is_co2(self): fossil_ts = _make_world_emissions({2020: 9000.0}, "co2-ffi") nghgi_ts = _make_nghgi_ts({2020: -600.0}) - bunker_ts = _make_bunker_ts({2020: 400.0}) result = build_nghgi_world_co2_timeseries( fossil_ts=fossil_ts, nghgi_ts=nghgi_ts, - bunker_ts=bunker_ts, ) # Result should have co2 as emission-category diff --git a/tests/unit/test_rcb_calculations.py b/tests/unit/test_rcb_calculations.py index 213ee12..881e601 100644 --- a/tests/unit/test_rcb_calculations.py +++ b/tests/unit/test_rcb_calculations.py @@ -159,6 +159,53 @@ def test_verbose_output(self, world_scenario_emissions_ts, capsys): assert "Allocation year 2025 > 2020" in captured.out assert "Used" in captured.out + def test_allocation_year_before_the_world_series_raises(self): + """A start year before the first year of the world series is an error.""" + index = pd.MultiIndex.from_tuples( + [("World", "Mt * CO2e", "co2")], + names=["iso3c", "unit", "emission-category"], + ) + from_2000 = pd.DataFrame( + {str(y): [10.0] for y in range(2000, 2024)}, index=index + ) + with pytest.raises(DataProcessingError) as error: + calculate_budget_from_rcb(500.0, 1990, from_2000, verbose=False) + assert "lack 1990-1999" in str(error.value) + assert "first available year is 2000" in str(error.value) + + # A column that holds no value counts as missing. + with_nan = from_2000.copy() + with_nan.insert(0, "1999", float("nan")) + with pytest.raises(DataProcessingError, match="lack 1999"): + calculate_budget_from_rcb(500.0, 1999, with_nan, verbose=False) + + assert calculate_budget_from_rcb(500.0, 2000, from_2000, verbose=False) == 700 + + def test_allocation_year_after_the_world_series_raises(self): + """A start year beyond the world series would drop years from the sum.""" + index = pd.MultiIndex.from_tuples( + [("World", "Mt * CO2e", "co2")], + names=["iso3c", "unit", "emission-category"], + ) + to_2023 = pd.DataFrame({str(y): [10.0] for y in range(2000, 2024)}, index=index) + with pytest.raises(DataProcessingError) as error: + calculate_budget_from_rcb(500.0, 2025, to_2023, verbose=False) + assert "lack 2024" in str(error.value) + assert "last available year is 2023" in str(error.value) + + assert calculate_budget_from_rcb(500.0, 2024, to_2023, verbose=False) == 460 + + def test_a_nan_year_counts_as_missing_after_2020(self): + """A year column without a value is lacking, not skipped by the sum.""" + index = pd.MultiIndex.from_tuples( + [("World", "Mt * CO2e", "co2")], + names=["iso3c", "unit", "emission-category"], + ) + padded = pd.DataFrame({str(y): [10.0] for y in range(2000, 2026)}, index=index) + padded[["2023", "2024"]] = float("nan") + with pytest.raises(DataProcessingError, match="lack 2023-2024"): + calculate_budget_from_rcb(500.0, 2025, padded, verbose=False) + def test_with_variable_emissions(self): """Test with variable emissions to ensure correct summation.""" # Create variable emissions data diff --git a/tests/unit/test_rcb_sources.py b/tests/unit/test_rcb_sources.py new file mode 100644 index 0000000..b539bfe --- /dev/null +++ b/tests/unit/test_rcb_sources.py @@ -0,0 +1,611 @@ +"""Tests for the RCB sources in rcbs.yaml and the scenario sets behind their deductions.""" + +from __future__ import annotations + +import warnings +from pathlib import Path + +import pandas as pd +import pytest +import yaml + +from fair_shares.library.config.models import AdjustmentsConfig +from fair_shares.library.exceptions import ( + ConfigurationError, + DataLoadingError, + DataProcessingError, +) +from fair_shares.library.preprocessing.rcbs import load_and_process_rcbs +from fair_shares.library.utils.data.nghgi import compute_bunker_deduction +from fair_shares.library.utils.data.rcb import ( + NET_ZERO_YEAR_COLUMN, + PEAK_WARMING_COLUMN, + build_rcb_scenario_sets, + convention_gap_from_baseline, + fill_rebase_years, + missing_rebase_years, + net_zero_year, + parse_rcb_scenario, + process_rcb_to_2020_baseline, + rcb_scenario_set_key, + select_rcb_scenario_set, + validate_rebase_fill_max_years, +) + +REPO_ROOT = Path(__file__).resolve().parents[2] +RCB_YAML = REPO_ROOT / "data" / "rcbs" / "rcbs.yaml" +GOLDEN_MASTER = REPO_ROOT / "tests" / "fixtures" / "rcb_golden_master.yaml" +AR6_METADATA = ( + REPO_ROOT / "data" / "scenarios" / "ipcc_ar6_gidden" / "metadata_ar6_gidden.xlsx" +) + +# Scenario sets written by notebook 104 in a gcb-2025 pipeline run. +PIPELINE_ADJUSTMENTS = ( + REPO_ROOT + / "output" + / "gcb-2025_wdi-2025_un-owid-2025_wdi-2025_rcbs_co2-ffi" + / "intermediate" + / "scenarios" + / "rcb_scenario_adjustments.yaml" +) + +# Scenario sets written by notebook 104 in the PRIMAP run with melo-2026 LULUCF. +PIPELINE_CO2_ADJUSTMENTS = ( + REPO_ROOT + / "output" + / "primap-202503_wdi-2025_un-owid-2025_wdi-2025_melo-2026_rcbs_all-ghg" + / "intermediate" + / "scenarios" + / "rcb_scenario_adjustments.yaml" +) + +EXISTING_SOURCES = ["lamboll_2023", "forster_2024", "ar6_2020"] +LABELS = ["1.5p50", "1.5p67", "1.7p50", "1.7p67", "2p50", "2p67", "2p83"] +BAND = "peak-warming-band" + + +def test_new_sources_hold_the_published_budgets(): + """forster_2026 (IGCC 2025, Table 8) and ar6_wg1_2021 (AR6 WGI, Table SPM.2).""" + sources = yaml.safe_load(RCB_YAML.read_text())["rcb_data"] + + forster = sources["forster_2026"] + assert forster["baseline_year"] == 2026 + assert forster["unit"] == "Gt * CO2" + assert forster["scenario_selection"] == BAND + assert forster["scenarios"] == dict( + zip(LABELS, [130, 80, 500, 390, 1050, 860, 690]) + ) + + ar6 = sources["ar6_wg1_2021"] + assert ar6["baseline_year"] == 2020 + assert ar6["unit"] == "Gt * CO2" + assert ar6["scenario_selection"] == BAND + assert ar6["scenarios"] == dict(zip(LABELS, [500, 400, 850, 700, 1350, 1150, 900])) + + +def test_new_labels_parse_and_share_one_band_per_temperature(): + """Each label keeps its own quantile; likelihoods of one temperature share a band.""" + assert parse_rcb_scenario("1.7p50") == ("1.7C", "0.5") + assert parse_rcb_scenario("2p67") == ("2C", "0.67") + assert parse_rcb_scenario("2p66") == ("2C", "0.66") + + assert rcb_scenario_set_key("forster_2026", "1.7p50", BAND) == "peak-warming-1.7C" + assert rcb_scenario_set_key("forster_2026", "1.7p67", BAND) == "peak-warming-1.7C" + assert rcb_scenario_set_key("forster_2026", "2p83", BAND) == "peak-warming-2C" + assert rcb_scenario_set_key("forster_2024", "2p83", "ar6-category") == "2p83" + + +def test_band_is_closed_below_and_open_above(): + """The band is [T - 0.05, T + 0.05); the half-width is configurable.""" + metadata = pd.DataFrame( + { + "scenario": ["below", "lower_bound", "centre", "upper_bound"], + "Category": ["C1", "C1", "C3", "C3"], + PEAK_WARMING_COLUMN: [1.6499, 1.65, 1.7, 1.75], + NET_ZERO_YEAR_COLUMN: [2050, 2055, 2060, 2070], + } + ) + + selected = select_rcb_scenario_set(metadata, "forster_2026", "1.7p50", BAND) + assert list(selected["scenario"]) == ["lower_bound", "centre"] + + # One band serves every likelihood of its temperature. + same = select_rcb_scenario_set(metadata, "forster_2026", "1.7p67", BAND) + assert list(same["scenario"]) == ["lower_bound", "centre"] + + wide = select_rcb_scenario_set(metadata, "forster_2026", "1.7p50", BAND, 0.1) + assert list(wide["scenario"]) == ["below", "lower_bound", "centre", "upper_bound"] + + +def test_band_keeps_only_scenarios_that_reach_net_zero(): + """A budget runs to net-zero CO2, so the band drops scenarios that never reach it.""" + assert net_zero_year(pd.Series({"2040": 5.0, "2050": 0.0, "2060": -1.0})) == 2050 + assert net_zero_year(pd.Series({"2040": 5.0, "2100": 0.1})) is None + + metadata = pd.DataFrame( + { + "scenario": ["reaches", "never", "reaches_in_2100"], + "Category": ["C3", "C3", "C3"], + PEAK_WARMING_COLUMN: [2.0, 2.0, 2.0], + NET_ZERO_YEAR_COLUMN: [2070, None, 2100], + } + ) + selected = select_rcb_scenario_set(metadata, "forster_2026", "2p50", BAND) + assert list(selected["scenario"]) == ["reaches", "reaches_in_2100"] + + # The AR6 category rule keeps every scenario of the category. + category_set = select_rcb_scenario_set(metadata, "ar6_2020", "2p66", "ar6-category") + assert len(category_set) == 3 + + never = metadata[metadata["scenario"] == "never"] + with pytest.raises(DataProcessingError, match="net-zero CO2 year"): + select_rcb_scenario_set(never, "forster_2026", "2p50", BAND) + + without_column = metadata.drop(columns=NET_ZERO_YEAR_COLUMN) + with pytest.raises(ConfigurationError, match=NET_ZERO_YEAR_COLUMN): + select_rcb_scenario_set(without_column, "forster_2026", "2p50", BAND) + + +def test_empty_band_raises_naming_temperature_and_band(): + metadata = pd.DataFrame( + { + "Category": ["C1"], + PEAK_WARMING_COLUMN: [1.5], + NET_ZERO_YEAR_COLUMN: [2050], + } + ) + with pytest.raises(DataProcessingError) as error: + select_rcb_scenario_set(metadata, "forster_2026", "2p50", BAND) + assert "2C band [1.95, 2.05)" in str(error.value) + + +def test_unmapped_label_raises_naming_source_and_label(): + """A label with no AR6 category and no band rule is an error, never a zero.""" + with pytest.raises(ConfigurationError, match="'my_source', label '1.7p50'"): + rcb_scenario_set_key("my_source", "1.7p50", "ar6-category") + with pytest.raises(ConfigurationError, match="unknown scenario_selection"): + rcb_scenario_set_key("my_source", "1.7p50", "nearest-category") + + +def test_bands_hold_14_131_69_scenarios_and_14_111_37_reach_net_zero(): + """Band sizes at 1.5, 1.7 and 2 degrees C, before and after the net-zero filter. + + The band sizes come from the AR6 metadata file. The net-zero years come + from the scenario time series, so the sizes after the filter are read + from the scenario sets that notebook 104 writes. + """ + if not AR6_METADATA.exists(): + pytest.skip("AR6 scenario metadata is not redistributable and is absent") + metadata = pd.read_excel(AR6_METADATA) + rcb_yaml = yaml.safe_load(RCB_YAML.read_text()) + + # With a net-zero year for every scenario, each set is its whole band. + metadata[NET_ZERO_YEAR_COLUMN] = 2100 + bands = build_rcb_scenario_sets(rcb_yaml, metadata) + assert len(bands["peak-warming-1.5C"]) == 14 + assert len(bands["peak-warming-1.7C"]) == 131 + assert len(bands["peak-warming-2C"]) == 69 + + if not PIPELINE_ADJUSTMENTS.is_file(): + pytest.skip("gcb-2025 scenario adjustments not built") + adjustments = yaml.safe_load(PIPELINE_ADJUSTMENTS.read_text()) + assert adjustments["peak-warming-1.5C"]["n_scenarios"] == 14 + assert adjustments["peak-warming-1.7C"]["n_scenarios"] == 111 + assert adjustments["peak-warming-2C"]["n_scenarios"] == 37 + + +def test_rebase_raises_when_emissions_end_before_the_baseline_year(): + """A partial rebase understates the budget, so a missing year is an error.""" + emissions = pd.DataFrame({str(y): [36000.0] for y in range(2020, 2024)}) + with pytest.raises(DataProcessingError) as error: + process_rcb_to_2020_baseline( + rcb_value=130, + rcb_unit="Gt * CO2", + rcb_baseline_year=2026, + emission_category="co2-ffi", + world_co2_ffi_emissions=emissions, + source_name="forster_2026", + verbose=False, + ) + assert "lack [2024, 2025]" in str(error.value) + + +def _one_row_csv(path: Path, values_by_year: dict[int, float], label: str) -> None: + """Write a single-row timeseries CSV with a ``source`` column.""" + path.parent.mkdir(parents=True, exist_ok=True) + row = {"source": label, **{str(y): v for y, v in values_by_year.items()}} + pd.DataFrame([row]).to_csv(path, index=False) + + +def _world_row(values_by_year: dict[int, float], category: str) -> pd.DataFrame: + """Build a world emissions row in the format the pipeline passes to the RCB step.""" + index = pd.MultiIndex.from_tuples( + [("World", "Mt * CO2e", category)], + names=["iso3c", "unit", "emission-category"], + ) + return pd.DataFrame({str(y): [v] for y, v in values_by_year.items()}, index=index) + + +@pytest.mark.parametrize("emission_category", ["co2-ffi", "co2"]) +def test_existing_sources_match_golden_master(tmp_path, emission_category): + """Adjusted budgets of lamboll_2023, forster_2024 and ar6_2020. + + The rebase adds the bunkers of 2020 to the year before the baseline, so + ar6_2020 equals its budget without that term, and lamboll_2023 and + forster_2024 exceed theirs by exactly those bunker emissions. + + For co2 the convention gap runs from the baseline year of the source, so + a budget exceeds the one with the gap from 2020 by the gap of 2020 to the + year before the baseline. ar6_2020 and every co2-ffi budget stay the same. + """ + golden = yaml.safe_load(GOLDEN_MASTER.read_text()) + inputs = golden["inputs"] + + emissions_dir = tmp_path / "golden" / "intermediate" / "emissions" + scenarios_dir = tmp_path / "golden" / "intermediate" / "scenarios" + _one_row_csv(emissions_dir / "bunker_timeseries.csv", inputs["bunkers_mt"], "gcb") + _one_row_csv( + emissions_dir / "world_co2-lulucf_timeseries.csv", + inputs["world_co2_lulucf_mt"], + "nghgi", + ) + scenarios_dir.mkdir(parents=True) + (scenarios_dir / "rcb_scenario_adjustments.yaml").write_text( + yaml.safe_dump(inputs["scenario_adjustments"], sort_keys=False) + ) + for key, values_by_year in inputs["lulucf_shift_median_mt"].items(): + _one_row_csv( + scenarios_dir / f"lulucf_shift_median_{key}.csv", values_by_year, key + ) + + all_sources = yaml.safe_load(RCB_YAML.read_text())["rcb_data"] + rcb_yaml = tmp_path / "rcbs.yaml" + existing = {"rcb_data": {s: all_sources[s] for s in EXISTING_SOURCES}} + rcb_yaml.write_text(yaml.safe_dump(existing, sort_keys=False)) + + adjustments_config = AdjustmentsConfig.model_validate( + { + "lulucf_nghgi": { + "path": "output/{source_id}/intermediate/emissions/" + "world_co2-lulucf_timeseries.csv" + }, + "bunkers": { + "path": "output/{source_id}/intermediate/emissions/bunker_timeseries.csv" + }, + } + ) + # The inputs cover every rebase year, so a placeholder fill is an error. + with warnings.catch_warnings(): + warnings.simplefilter("error") + result = load_and_process_rcbs( + rcb_yaml_path=rcb_yaml, + world_fossil_emissions=_world_row(inputs["world_fossil_co2_mt"], "co2-ffi"), + emission_category=emission_category, + adjustments_config=adjustments_config, + data_dir=tmp_path, + source_id="golden", + world_nghgi_lulucf_emissions=_world_row( + inputs["world_co2_lulucf_mt"], "co2-lulucf" + ), + verbose=False, + output_dir=tmp_path, + ) + + expected = pd.DataFrame(golden["expected"][emission_category]) + actual = result[expected.columns].reset_index(drop=True) + actual["quantile"] = actual["quantile"].astype(float) + pd.testing.assert_frame_equal(actual, expected, check_dtype=False) + + without_rebase = pd.DataFrame( + golden["rcb_2020_without_bunker_rebase_mt"][emission_category] + ) + assert list(without_rebase["source"]) == list(actual["source"]) + assert list(without_rebase["scenario"]) == list(actual["scenario"]) + increase = actual["rcb_2020_nghgi_mt"] - without_rebase["rcb_2020_nghgi_mt"] + # The co2 budgets with the convention gap from 2020 for every source. + if emission_category == "co2": + gap_from_2020 = pd.DataFrame(golden["rcb_2020_with_gap_from_2020_mt"]["co2"]) + assert list(gap_from_2020["source"]) == list(actual["source"]) + assert list(gap_from_2020["scenario"]) == list(actual["scenario"]) + gap_change = actual["rcb_2020_nghgi_mt"] - gap_from_2020["rcb_2020_nghgi_mt"] + for row, (source, label) in enumerate(zip(actual["source"], actual["scenario"])): + baseline_year = all_sources[source]["baseline_year"] + rebase_years = range(2020, baseline_year) + bunkers = sum(inputs["bunkers_mt"][y] for y in rebase_years) + assert actual["rebase_bunkers_mt"][row] == round(bunkers) + + # The gap of 2020 to the year before the baseline (negative). + gaps = inputs["scenario_adjustments"][label]["convention_gap_median_from"] + gap_before_baseline = gaps[2020] - gaps[baseline_year] + if emission_category == "co2": + assert actual["correction_lulucf_nghgi_mt"][row] == round( + gaps[baseline_year] + ) + assert gap_change[row] == pytest.approx(-gap_before_baseline, abs=1) + # It is the observed NGHGI LULUCF minus the scenario BM LULUCF of + # those years. The median of per-scenario sums differs from the sum + # of per-year medians by up to 3%. + nghgi_minus_bm = sum( + inputs["world_co2_lulucf_mt"][y] + - inputs["lulucf_shift_median_mt"][label][y] + for y in rebase_years + ) + assert gap_before_baseline == pytest.approx(nghgi_minus_bm, rel=0.03) + bunkers -= gap_before_baseline + # Each budget is rounded to 1 Mt, so the difference holds within 1 Mt. + assert increase[row] == pytest.approx(bunkers, abs=1) + assert (increase[actual["source"] == "ar6_2020"] == 0).all() + + +def test_convention_gap_starts_at_the_baseline_year(): + """Hand calculation: gap per scenario from the baseline year, then the median. + + Observed NGHGI LULUCF is -4 Mt in each year to the splice year 2023. + Scenario A reaches net zero in 2030 with Direct 1 and Indirect -5 each year. + Scenario B reaches net zero in 2022 with Direct 2 and Indirect -3 each year. + """ + years = [str(y) for y in range(2020, 2031)] + nghgi = pd.Series(-4.0, index=years[:4]) + scenarios = { + "A": (pd.Series(1.0, index=years), pd.Series(-5.0, index=years), 2030), + "B": (pd.Series(2.0, index=years), pd.Series(-3.0, index=years), 2022), + } + + def gaps(baseline_year: int) -> dict[str, float]: + return { + name: convention_gap_from_baseline( + nghgi, direct, indirect, baseline_year, nz_year, splice_year=2023 + ) + for name, (direct, indirect, nz_year) in scenarios.items() + } + + # From 2020. A: 4 x (-4 - 1) + 7 x -5 = -55. B: 3 x (-4 - 2) = -18. + assert gaps(2020) == {"A": -55.0, "B": -18.0} + assert pd.Series(gaps(2020)).median() == -36.5 + # From 2023. A: (-4 - 1) + 7 x -5 = -40. B reaches net zero before 2023. + assert gaps(2023) == {"A": -40.0, "B": 0.0} + assert pd.Series(gaps(2023)).median() == -20.0 + # From 2026, after the splice year. A: Indirect only, 5 x -5 = -25. + assert gaps(2026) == {"A": -25.0, "B": 0.0} + # A baseline in the net-zero year keeps that one year: -4 - 2 = -6. + assert gaps(2022)["B"] == -6.0 + + +def test_co2_budget_needs_a_convention_gap_from_its_baseline_year(tmp_path): + """Adjustments without a gap from the baseline year name notebook 104.""" + emissions_dir = tmp_path / "stale" / "intermediate" / "emissions" + scenarios_dir = tmp_path / "stale" / "intermediate" / "scenarios" + years = {y: 100.0 for y in range(2020, 2024)} + _one_row_csv(emissions_dir / "bunker_timeseries.csv", years, "gcb") + scenarios_dir.mkdir(parents=True) + adjustments = { + "1.5p50": { + "bm_lulucf_cumulative_median": 0.0, + "convention_gap_median_from": {2020: -50.0}, + "nz_year_median": 2030, + "n_scenarios": 1, + } + } + (scenarios_dir / "rcb_scenario_adjustments.yaml").write_text( + yaml.safe_dump(adjustments) + ) + _one_row_csv(scenarios_dir / "lulucf_shift_median_1.5p50.csv", years, "1.5p50") + rcb_yaml = tmp_path / "rcbs.yaml" + source = {"baseline_year": 2023, "unit": "Gt * CO2", "scenarios": {"1.5p50": 1}} + rcb_yaml.write_text(yaml.safe_dump({"rcb_data": {"budget_2023": source}})) + adjustments_config = AdjustmentsConfig.model_validate( + { + "lulucf_nghgi": {"path": "output/{source_id}/unused.csv"}, + "bunkers": { + "path": "output/{source_id}/intermediate/emissions/bunker_timeseries.csv" + }, + } + ) + with pytest.raises(DataLoadingError) as error: + load_and_process_rcbs( + rcb_yaml_path=rcb_yaml, + world_fossil_emissions=_world_row(years, "co2-ffi"), + emission_category="co2", + adjustments_config=adjustments_config, + data_dir=tmp_path, + source_id="stale", + world_nghgi_lulucf_emissions=_world_row(years, "co2-lulucf"), + verbose=False, + output_dir=tmp_path, + ) + message = str(error.value) + assert "No convention gap from baseline year 2023 for '1.5p50'" in message + assert "Re-run notebook 104" in message + + +def test_gap_from_2020_equals_the_captured_gap(): + """On the real scenario data the 2020 entry equals the captured gap from 2020.""" + if not PIPELINE_CO2_ADJUSTMENTS.is_file(): + pytest.skip("AR6 scenario adjustments with melo-2026 LULUCF not built") + adjustments = yaml.safe_load(PIPELINE_CO2_ADJUSTMENTS.read_text()) + golden = yaml.safe_load(GOLDEN_MASTER.read_text()) + captured = golden["convention_gap_median_from_2020_captured_mt"] + for set_key, gap in captured.items(): + assert adjustments[set_key]["convention_gap_median_from"][2020] == gap + fixture_set = golden["inputs"]["scenario_adjustments"][set_key] + assert fixture_set["convention_gap_median_from"][2020] == gap + + +@pytest.mark.parametrize("baseline_year", [2020, 2024]) +def test_rebase_counts_each_bunker_year_once(baseline_year): + """Hand calculation: two baselines of one world give one allocatable budget. + + Each year from 2020 to net zero in 2029 emits 100 Mt fossil CO2 without + bunkers and 10 Mt bunker CO2. A published budget covers both, so it is + 1100 Mt from 2020 and 660 Mt from 2024. The budget without bunkers from + 2020 is 10 x 100 = 1000 Mt. + """ + fossil = pd.DataFrame({str(y): [100.0] for y in range(2020, 2024)}) + bunkers = pd.DataFrame({str(y): [10.0] for y in range(2020, 2024)}) + published = {2020: 1100.0, 2024: 660.0}[baseline_year] + + # Observed 2020-2023 (40 Mt) plus the 2023 rate for 2024-2029 (60 Mt). + deduction = compute_bunker_deduction(bunkers, start_year=2020, net_zero_year=2029) + assert deduction == pytest.approx(100.0) + + result = process_rcb_to_2020_baseline( + rcb_value=published, + rcb_unit="Mt * CO2", + rcb_baseline_year=baseline_year, + emission_category="co2-ffi", + world_co2_ffi_emissions=fossil, + world_bunker_emissions=bunkers, + bunkers_deduction_mt=deduction, + verbose=False, + ) + + assert result["rebase_fossil_mt"] == {2020: 0, 2024: 400}[baseline_year] + assert result["rebase_bunkers_mt"] == {2020: 0, 2024: 40}[baseline_year] + assert result["deduction_bunkers_mt"] == -100 + assert result["rcb_2020_nghgi_mt"] == 1000 + + +def test_rebase_with_a_bunker_deduction_needs_the_bunker_series(): + """A deduction from 2020 without the bunkers of the rebase years is an error.""" + fossil = pd.DataFrame({str(y): [100.0] for y in range(2020, 2024)}) + with pytest.raises(DataProcessingError, match="pass world_bunker_emissions"): + process_rcb_to_2020_baseline( + rcb_value=660.0, + rcb_unit="Mt * CO2", + rcb_baseline_year=2024, + emission_category="co2-ffi", + world_co2_ffi_emissions=fossil, + bunkers_deduction_mt=100.0, + verbose=False, + ) + + +def test_co2_rebase_needs_the_inventory_lulucf_series(): + """A co2 budget from after 2020 raises without LULUCF; a 2020 budget runs.""" + fossil = pd.DataFrame({str(y): [100.0] for y in range(2020, 2024)}) + kwargs = { + "rcb_unit": "Mt * CO2", + "emission_category": "co2", + "world_co2_ffi_emissions": fossil, + "verbose": False, + } + with pytest.raises(DataProcessingError, match="pass world_nghgi_lulucf_emissions"): + process_rcb_to_2020_baseline(rcb_value=660.0, rcb_baseline_year=2024, **kwargs) + + result = process_rcb_to_2020_baseline( + rcb_value=1000.0, rcb_baseline_year=2020, **kwargs + ) + assert result["rebase_lulucf_mt"] == 0 + assert result["rcb_2020_nghgi_mt"] == 1000 + + +def test_rebase_fill_holds_the_last_observed_value(): + """Observed to 2024, baseline 2026: 2025 takes the 2024 value, with a warning. + + Hand calculation: 100 Mt fossil CO2 and 10 Mt bunker CO2 each year to net + zero in 2029. The published budget from 2026 is 4 x 110 = 440 Mt and the + budget without bunkers from 2020 is 1000 Mt. + """ + fossil = pd.DataFrame({str(y): [100.0] for y in range(2020, 2025)}) + bunkers = pd.DataFrame({str(y): [10.0] for y in range(2020, 2025)}) + + with pytest.warns(UserWarning) as record: + filled_fossil = fill_rebase_years( + fossil, 2026, 1, "world fossil CO2 emissions", "forster_2026" + ) + message = str(record[0].message) + assert "'forster_2026'" in message + assert "world fossil CO2 emissions end in 2024" in message + assert "2024 value (100.0 Mt CO2) for [2025]" in message + assert "placeholder until observed data are published" in message + assert filled_fossil["2025"].iloc[0] == fossil["2024"].iloc[0] + assert "2025" not in fossil.columns + + with pytest.warns(UserWarning): + filled_bunkers = fill_rebase_years(bunkers, 2026, 1, "bunkers", "forster_2026") + result = process_rcb_to_2020_baseline( + rcb_value=440.0, + rcb_unit="Mt * CO2", + rcb_baseline_year=2026, + emission_category="co2-ffi", + world_co2_ffi_emissions=filled_fossil, + world_bunker_emissions=filled_bunkers, + bunkers_deduction_mt=compute_bunker_deduction(bunkers, 2020, 2029), + verbose=False, + ) + assert result["rebase_fossil_mt"] == 600 + assert result["rebase_bunkers_mt"] == 60 + assert result["deduction_bunkers_mt"] == -100 + assert result["rcb_2020_nghgi_mt"] == 1000 + + +def test_rebase_fill_stops_at_the_limit(): + """More missing years than the limit, or a limit of 0, leave the series as is.""" + to_2023 = pd.DataFrame({str(y): [100.0] for y in range(2020, 2024)}) + to_2024 = pd.DataFrame({str(y): [100.0] for y in range(2020, 2025)}) + + with warnings.catch_warnings(): + warnings.simplefilter("error") + assert fill_rebase_years(to_2023, 2026, 1).equals(to_2023) + assert fill_rebase_years(to_2024, 2026, 0).equals(to_2024) + # A series that covers every rebase year needs no fill. + assert fill_rebase_years(to_2023, 2024, 1).equals(to_2023) + + with pytest.raises(DataProcessingError, match=r"lack \[2024, 2025\]"): + process_rcb_to_2020_baseline( + rcb_value=440.0, + rcb_unit="Mt * CO2", + rcb_baseline_year=2026, + emission_category="co2-ffi", + world_co2_ffi_emissions=fill_rebase_years(to_2023, 2026, 1), + verbose=False, + ) + + +def test_rebase_counts_a_year_without_a_value_as_missing(): + """NaN-padded trailing years are lacking, so the rebase raises instead of summing less.""" + padded = pd.DataFrame({str(y): [100.0] for y in range(2020, 2026)}) + padded[["2023", "2024", "2025"]] = float("nan") + assert missing_rebase_years(2026, padded) == [2023, 2024, 2025] + + with pytest.raises(DataProcessingError, match=r"lack \[2023, 2024, 2025\]"): + process_rcb_to_2020_baseline( + rcb_value=440.0, + rcb_unit="Mt * CO2", + rcb_baseline_year=2026, + emission_category="co2-ffi", + world_co2_ffi_emissions=padded, + verbose=False, + ) + + +def test_rebase_nan_gap_beyond_the_fill_limit_is_missing(): + """A NaN gap longer than rebase_fill_max_years stays unfilled and counts as lacking.""" + gap = pd.DataFrame({str(y): [100.0] for y in range(2020, 2025)}) + gap[["2022", "2023", "2024"]] = float("nan") + unfilled = fill_rebase_years(gap, 2026, 1) + assert unfilled.equals(gap) + assert missing_rebase_years(2026, unfilled) == [2022, 2023, 2024, 2025] + + +def test_peak_warming_label_without_a_temperature_raises_configuration_error(): + """A label such as 'abcp50' names itself in a ConfigurationError.""" + metadata = pd.DataFrame( + { + "scenario": ["a"], + "Category": ["C1"], + PEAK_WARMING_COLUMN: [1.5], + NET_ZERO_YEAR_COLUMN: [2050], + } + ) + with pytest.raises(ConfigurationError, match="label 'abcp50'"): + select_rcb_scenario_set(metadata, "forster_2026", "abcp50", BAND) + + +@pytest.mark.parametrize("value", [-1, 1.5, "1", None, True]) +def test_rebase_fill_max_years_must_be_a_non_negative_integer(value): + """Anything other than a count of zero or more is a configuration error.""" + with pytest.raises(ConfigurationError, match="rebase_fill_max_years"): + validate_rebase_fill_max_years(value) + assert validate_rebase_fill_max_years(0) == 0 + assert validate_rebase_fill_max_years(2) == 2