Offline MAgPIE lookup-table generator for the MAgPIE-based land-use emulator in OPEN-PROM.
This repository has one responsibility: run MAgPIE over a grid of second-generation bioenergy demand and greenhouse-gas price scenarios, extract the resulting biomass and AFOLU variables, and publish two checked lookup workbooks. It does not choose regression forms, estimate OPEN-PROM coefficients, modify the GAMS equations, or validate final emulator runs against soft-link references.
Downstream responsibilities are deliberately separate:
mrpromreadslookup_table_openprom.xlsx, fits the selected price and emission functions, and writes OPEN-PROM input CSV files.- OPEN-PROM reads those CSV files when
landUseEmulator = magpieand evaluates the emulator in its recursive solve. postpromruns and reports the OPEN-PROM–MAgPIE soft-link reference.- Design history and validation plots live in
e3m-notes/land-use-emulators/MAgPIE/.
Rscript start.R runs the following pipeline:
start.R
scripts/run_grid.R
one MAgPIE solve per carbon-price × BioDem point
-> logs/run_index.csv and per-point completion records
scripts/build_lookup.R
requested-demand MIF + report.mif + fulldata.gdx
-> scripts/disaggregate.R (H12 EUR to OPEN-PROM EU28)
-> lookup_table_h12.xlsx
-> lookup_table_openprom.xlsx
Each workbook has one row per (Region, Variable, BioDem, GHGPrice) and a year time series through 2100:
lookup_table_h12.xlsx: 12 native MAgPIE regions plus World.lookup_table_openprom.xlsx: 39 OPEN-PROM regions.
The tables retain a broad set of quantities, prices and emissions for downstream fitting and auditing. They also include the exact requested 2G demand and MAgPIE's native Prices|Bioenergy. The presence of a row in the lookup does not mean OPEN-PROM independently emulates that row.
The production pipeline ends when both workbooks have been built and validated. scripts/visualize_lookup.R is an optional lookup-only QA utility; it does not fit or assess the deployed OPEN-PROM emulator.
The demand axis uses BioDemXXX, where XXX = multiplier × 100 and BioDem100 is the 1× path. The repository ships a 0–8× grid at 0.25× intervals. A run is the Cartesian product of the selected demand levels and carbon-price MIFs.
One combined coupling MIF passes both the requested 2G energy-crop demand and CO₂/CH₄/N₂O price paths to MAgPIE. Values are held constant from 2100 to MAgPIE's internal 2150 horizon; published lookup tables stop at 2100.
The H12 table preserves native MAgPIE regions. For the OPEN-PROM table:
- the 11 non-EUR H12 regions map one-to-one;
- EUR emission flows are split to EU28 using run-specific cell/cluster weights and checked for conservation;
- EUR bioenergy quantities are split using cropland-area shares and checked for conservation;
- EUR prices are broadcast to EU28 because prices are not additive flows.
input/disagg/magpie-afolu-emission-variables.csv defines 200 MAgPIE emission series: 64 parents and 136 leaves covering 11 species. The following tree keeps only the aggregation levels needed to understand the coverage; [lookup] marks the three aggregates published in the lookup workbook.
MAgPIE AFOLU emissions (200 series)
├── CO₂ (58)
│ ├── raw net Land CO₂ = Indirect + Land-use Change
│ │ ├── Indirect: managed-forest sink [construction QA only]
│ │ └── Land-use Change [lookup]
│ │ ├── forest change: deforestation, degradation and regrowth
│ │ ├── land and soil: other conversion, peatland, soil and residual
│ │ └── wood: harvested-wood products and wood harvest
│ ├── Land fires: forest, grassland and peat burning
│ └── crop-residue burning and an AFOLU-agriculture inventory
├── CH₄ (16)
│ ├── Land | Agriculture [lookup]: enteric fermentation, manure and rice
│ ├── crop-residue burning and managed peatland
│ └── Land fires: forest, grassland and peat burning
├── N₂O (22)
│ ├── Land | Agriculture [lookup]
│ │ ├── agricultural soils: residues, fertilizer, manure, pasture and soil-organic-matter loss
│ │ └── animal-waste management
│ ├── crop-residue burning and managed peatland
│ └── Land fires: forest, grassland and peat burning
├── NH₃ and NO₂ (22 each)
│ └── the same soil, animal-waste, crop-burning, peatland and fire source structure as N₂O
├── NO₃⁻ (15)
│ └── agricultural soils, animal waste, crop-residue burning and managed peatland; no Fires subtree
└── BC, CO, OC, SO₂ and VOC (9 each)
├── Land fires: forest, grassland and peat burning
├── crop-residue burning
└── AFOLU-agriculture inventory
The tree is an accounting hierarchy, not a list of independent quantities. A parent equals the sum of its direct |+| children, so a parent and its descendants must never be added together. The curated 200-series set already removes cumulative/GWP series, alternate |++| slices and a duplicated managed-forest-sink alias. Different species also cannot be summed without an explicit conversion such as GWP.
The disaggregation code reads this full tree so that EUR-to-EU28 splitting can rebuild and conservation-check parent totals. During every lookup build it verifies raw Land CO₂ = Land-use Change + Indirect at native H12 and disaggregated OP39 resolution. Raw Land and Indirect are diagnostic inputs only and are not published. The lookup retains the three marked aggregates used for emulator fitting: land-use-change CO₂ emissions/removals excluding indirect land CO₂ and fire emissions, agricultural CH₄ and agricultural N₂O. Fires, Indirect, source-level components, gross removals and the other eight pollutants remain available in the underlying MAgPIE/soft-link reporting tree but are not independently emulated by this repository.
Machine-specific settings live in the git-ignored config.json. Start from config.template.json:
{
"magpie_path": "/path/to/a/dedicated/magpie/checkout",
"magpie_scenario": "2C",
"bioenergy_demand": ["BioDem025", "BioDem275", "BioDem575"],
"carbon_price": ["NDC_LTT"]
}magpie_pathis the only external path required by this repository. Use a dedicated MAgPIE clone or worktree: MAgPIE's normalstart_run()rewrites configuration-dependent source files in its checkout.magpie_scenarioselects onetitlerow ininput/magpie_config/scenarios.csv.bioenergy_demandselects demand MIF basenames frominput/2nd_bioenergy_demand/; use"all"for the full grid.carbon_priceselects MIF basenames frominput/carbon_price/.
All land-scenario tables and coupling inputs are shipped in this repository, so producing a lookup does not require an OPEN-PROM run.
Rscript start.R
Rscript start.R resume=run_<timestamp>A fresh run creates output/run_<timestamp>/. Progress is written after every validated grid point. Resume uses that run's own config.json, solver policy and input checksums, reruns only missing points, and rebuilds the workbooks only when the expected key set is complete.
To add new demand or carbon-price inputs to an unfinished run, add the files, update that run folder's config.json, and resume. Do not change magpie_scenario inside an existing run.
The runner accepts only complete MAgPIE results that:
- contain non-empty
report.mif,fulldata.gdxandfull.log; - reach 2100 in both report and GDX;
- show normal completion;
- contain only allowed model status 1 or 2;
- match the expected carbon-price and BioDem key.
Failed points are recorded in logs/failed_runs.csv and never enter run_index.csv. Indexes and workbooks are published atomically, and a run lock prevents concurrent writers.
output/run_<timestamp>/
config.json
inputs.md5
lookup_table_h12.xlsx
lookup_table_openprom.xlsx
logs/
run_plan.json
solver_policy.json
run_index.csv
failed_runs.csv # only when a point fails
log_run_grid_<carbon>.log
log_build_lookup.log
The entire output/ tree is git-ignored because every lookup run is reproducible from its input snapshot and indexed MAgPIE results.
Plotting is optional and is not run by start.R. After a lookup workbook has been built, run the script from the repository root with one workbook path:
# Native MAgPIE H12 regions plus World
Rscript scripts/visualize_lookup.R output/run_<timestamp>/lookup_table_h12.xlsx
# Or the 39 OPEN-PROM regions
Rscript scripts/visualize_lookup.R output/run_<timestamp>/lookup_table_openprom.xlsxThe figures are written to output/run_<timestamp>/figs/. Choose the H12 or OPEN-PROM workbook according to the spatial resolution needed; both commands use the same figs/ directory, so running the second can overwrite files for region names shared by the two tables.
For every available region, the script plots four bioenergy-price responses and the three lookup emission responses. The x-axis is total second-generation bioenergy demand, each coloured line is one year from 2010 onward, and the points along a line are the BioDem demand levels. If the workbook contains multiple GHG-price scenarios, the scenario label is added to each filename. Missing or all-NA combinations are skipped and reported in the final console summary.
figs/
<region>_bioenergy_<product>.png
<region>_emission_<variable>.png
These plots are lookup-table QA only: they show the MAgPIE response surface and do not fit regression equations or compare an OPEN-PROM emulator run with a soft-link reference.
- A MAgPIE checkout, GAMS 50.1 or newer, and CONOPT for
run_grid.R. - R packages
magclass,magpie4,gdx2,madrat,jsonlite,openxlsx, andfilelock. Missing packages are installed on first launch from the PIK r-universe and CRAN.
No postprom installation is required. The necessary OP39 disaggregation code is self-contained in scripts/disaggregate.R.
config.template.json tracked configuration template
config.json machine-local configuration, git-ignored
start.R driver, validation and resume logic
scripts/
run_grid.R MAgPIE grid execution
build_lookup.R extraction and workbook publication
disaggregate.R H12 EUR to EU28 disaggregation
visualize_lookup.R optional lookup QA figures
input/
2nd_bioenergy_demand/ BioDemXXX demand MIFs
carbon_price/ greenhouse-gas price MIFs
magpie_config/ land-scenario tables
disagg/ region and emission mappings
output/run_<timestamp>/ the two lookup workbooks and run records