catalmat_jae

ALIGNN training pipelines for catalysis-relevant adsorption energies and surface properties, applied across twelve datasets. Each sub-project is a self-contained training run: one script loads a dataset zip, prepares POSCAR + id_prop.csv inputs, and trains an ALIGNN graph neural network on the held-out adsorption/surface energy target.

The project goal is adsorption-energy prediction for catalyst design. v01 is the pipeline-validation arm (surface energy on JARVIS surface_db); v02-v09 are single-adsorbate catalyst datasets from AGRA and TinNet; v10-v12 are mixed-adsorbate Open Catalyst (OC20) subsets at three scales.

Repository layout

catalmat/
  alignn_v01_surfacedb/     JARVIS surface_db slabs    (N = 607, target surf_en in J/m^2)
  alignn_v02_AGRA_O/        AGRA O catalyst dataset    (N = 1000,  target ead in eV)
  alignn_v03_AGRA_OH/       AGRA OH                    (N = 877,   target ead)
  alignn_v04_AGRA_COOH/     AGRA COOH                  (N = 280,   target ead)
  alignn_v05_AGRA_CHO/      AGRA CHO                   (N = 216,   target ead)
  alignn_v06_AGRA_CO/       AGRA CO                    (N = 194,   target ead)
  alignn_v07_tinnet_N/      TinNet N                   (N = 329,   target ead)
  alignn_v08_tinnet_O/      TinNet O                   (N = 747,   target ead)
  alignn_v09_tinnet_OH/     TinNet OH                  (N = 748,   target ead)
  alignn_v10_ocp10k/        Open Catalyst (OC20) 10k subset    (N = 59,886,  target relaxed_energy in eV)
  alignn_v11_ocp100k/       Open Catalyst (OC20) 100k subset   (N = 149,886, target relaxed_energy)
  alignn_v12_ocp_all/       Open Catalyst (OC20) full subset   (N = 510,214, target relaxed_energy)

Each sub-directory contains:

• prepare_data.py — reads the dataset zip in the same directory, filters for valid atoms + target, writes data/POSCAR-*.vasp plus data/id_prop.csv (one row per structure: <filename>,<target>).
• config.json — ALIGNN training config (4 alignn-layers + 4 GCN-layers, 256 hidden features, MSE loss, AdamW + OneCycle scheduler, 80/10/10 train/val/test split, 300 epochs).
• data/ (generated) — POSCAR files + id_prop.csv produced by prepare_data.py.
• trained/ (generated) — model checkpoints, history JSONs, per-batch prediction artifacts.
• analysis/ — parity plots, loss curve, per-split prediction JSONs (Train_results.json, Val_results.json, Test_results.json).

Dataset zips, cluster job scripts, training outputs, and one-off analysis utilities are not tracked in this repo (see "Reproducing" below for how to obtain them).

Setup

Dependencies (see atomgptlab/alignn and atomgptlab/jarvis-tools):

conda create -n alignn python=3.9
conda activate alignn
pip install alignn jarvis-tools tqdm
pip install dgl==1.1.3 -f https://data.dgl.ai/wheels/repo.html

dgl is not a transitive dep of pip install alignn and must be installed explicitly from the dgl wheel index, otherwise from alignn.pretrained import get_figshare_model fails with ModuleNotFoundError: No module named 'dgl'.

Reproducing one run

# 1. Download the dataset zip
#    (URL is the corresponding figshare DOI for the dataset name; see "Datasets" below)
cd catalmat/alignn_v02_AGRA_O
wget -O AGRA_O.json.zip https://figshare.com/ndownloader/files/41923284

# 2. Build POSCAR + id_prop.csv from the zip
python prepare_data.py

# 3. Train (alignn CLI is installed on $PATH after `pip install alignn`)
train_alignn.py \
    --root_dir data \
    --config_name config.json \
    --file_format poscar \
    --output_dir trained \
    --device cpu

The training run writes trained/history_{train,val}.json (per-epoch loss), trained/{best,current}_model.pt (checkpoints), and trained/{Train,Val}_results.json (per-batch predictions, refreshed only when a new best-val MSE is achieved). The held-out test split is NOT automatically evaluated by ALIGNN 2026.4.2; see "Test-set inference" below.

Datasets

All datasets are available from the JARVIS figshare hub via jarvis.db.figshare.data(<name>). The local prepare_data.py reads from a pre-downloaded .json.zip in the same directory.

Sub-project	Dataset name	JARVIS reference	Upstream source
v01	surfacedb	JARVIS surface_db	Choudhary et al., Digital Discovery (RSC) 2024 — InterMat
v02-v06	AGRA_O, AGRA_OH, AGRA_COOH, AGRA_CHO, AGRA_CO	JARVIS catalyst databases	Feugmo-Group/AGRA
v07-v09	tinnet_N, tinnet_O, tinnet_OH	JARVIS catalyst databases	hlxin/tinnet
v10-v12	ocp10k, ocp100k, ocp_all	JARVIS open catalyst databases	Open Catalyst Project (OC20)

Target property per run

Each run predicts a single scalar per structure. Per-atom forces, stresses, and atomwise quantities are not predicted (those head weights are zeroed in config.json).

Run	Target field	What it physically represents	Units
v01 surface_db	surf_en	Surface energy — energy to create a slab surface from bulk	J/m²
v02 AGRA_O	ead	*O adsorption energy (ORR intermediate; CHE H₂O+H₂ reference)	eV
v03 AGRA_OH	ead	*OH adsorption energy (ORR intermediate; CHE H₂O+H₂ reference)	eV
v04 AGRA_COOH	ead	*COOH adsorption energy (CO₂RR intermediate; CO+H₂ reference)	eV
v05 AGRA_CHO	ead	*CHO adsorption energy (CO₂RR intermediate; CO+H₂ reference)	eV
v06 AGRA_CO	ead	*CO adsorption energy (CO₂RR intermediate; gas-phase CO reference)	eV
v07 tinnet_N	ead	*N chemisorption energy (atomic-N reference)	eV
v08 tinnet_O	ead	*O chemisorption energy (atomic-O reference)	eV
v09 tinnet_OH	ead	*OH chemisorption energy (gas-phase OH-radical reference)	eV
v10 ocp10k	relaxed_energy	OC20 IS2RE adsorption energy: predict relaxed-system ΔE from the initial unrelaxed structure	eV
v11 ocp100k	relaxed_energy	Same as v10, larger dataset (149,886 entries)	eV
v12 ocp_all	relaxed_energy	Same as v10, full OC20-IS2RE training pool (510,214 entries)	eV

Quick mental model:

• v01 is the odd one out — predicts how much energy is needed to create the surface itself (J/m²). All other runs predict how much energy is released or absorbed when a molecule binds to that surface (eV).
• v02-v09 are "given a relaxed adsorbate-on-slab structure, predict its adsorption energy" — standard supervised ML for catalysis.
• v10-v12 are the harder OC20 IS2RE task — "given the initial (unrelaxed) adsorbate-on-slab structure, predict the energy of the relaxed system." The model has to implicitly approximate the relaxation as part of the prediction.

Test-set inference

ALIGNN 2026.4.2 writes Train_results.json and Val_results.json from the best-val checkpoint but does NOT auto-evaluate the held-out test split. To produce test predictions, load trained/best_model.pt and re-infer on the test IDs from trained/ids_train_val_test.json. The result format mirrors the train/val JSONs (per-batch dicts with target_out / pred_out arrays of length batch_size).

alignn_v01_surfacedb/analysis/Test_results.json in this repo was produced this way. Reading the JSON and computing metrics:

import json, numpy as np
with open("alignn_v01_surfacedb/analysis/Test_results.json") as f:
    batches = json.load(f)
y = np.array([t for b in batches for t in b["target_out"]])
p = np.array([t for b in batches for t in b["pred_out"]])
print(f"MAE = {np.mean(np.abs(p - y)):.3f}")

Reading the headline tables

Every cohort table below uses the same format: n_eval / n_total for train and test. n_total is the size of the originally-allocated split (after 80/10/10 ratios + ALIGNN's split-ratio floor rounding, which can drop 0–2 entries across the three folds). n_eval is the number ALIGNN actually evaluated, after two more sources of attrition:

1. drop_last=True on the data loader (ALIGNN default). The last partial batch of each fold is discarded, so each fold loses up to (batch_size − 1) entries. v01–v09 used batch_size=16 (up to 15 lost per fold); v10–v12 use batch_size=64 (up to 63 lost per fold). This is why v10's train shows 47,872 / 47,908 — 47,908 mod 64 = 36 train entries dropped.
2. Analysis-time outlier filters, applied only where flagged. v01 has filter rules (y > −10 train, y < 30 test) covering DFT-artifact slabs; v06 has a single-point train filter (|residual| > 0.5 eV) for the severe DFT-convergence outlier. Filtered rows are marked with ¹ or (filtered). v02-v05 and v07-v12 are unfiltered.

The test inference convention for v02–v10 uses batch_size=1, so drop_last=True drops zero test entries (their test rows read 100/100, 87/87, etc.). v01 was re-run at batch_size=1 to bring it to 60/60 evaluated (down from 48/60 at the original batch_size=16 inference).

Test relative MAE reduction

The cohort tables report a normalized metric in their last column, relative MAE reduction (abbreviated Test rel. MAE reduction in the headline tables):

$$\text{Relative MAE reduction} = 1 - \frac{\text{MAE}_{\text{model}}}{\text{MAE}_{\text{baseline}}}$$

where the baseline MAE is $\text{mean}(|y - \bar{y}|)$ — the MAE you'd get by predicting the test-set mean $\bar{y}$ for every entry (the trivial "constant prediction" baseline) — and the model MAE is the model's actual test MAE. The metric expresses the fraction of the baseline's error the model has eliminated.

• Relative MAE reduction = 1 → perfect prediction (zero model error).
• Relative MAE reduction = 0.8 → model's typical error is 20% of the baseline MAE.
• Relative MAE reduction = 0 → model is no better than predicting the mean.
• Relative MAE reduction < 0 → model is worse than the baseline (failure mode).

Because the metric divides out the dataset's intrinsic spread, it's invariant to reference convention — directly comparable across all eleven ead-style runs even though they use four different reference-state conventions (detailed in Cross-cohort comparability below). A model with MAE = 0.1 eV on TinNet (whose targets span ~7 eV) and the same MAE = 0.1 eV on AGRA ORR (whose targets span ~2 eV) would look "equally accurate" by absolute MAE, but relative MAE reduction would correctly show that the TinNet result reflects more learning relative to that dataset's harder spread.

v02-v09 values come from ead_comprehensive_analysis/metrics_summary.csv; v10-v12 from rele_comprehensive_analysis/metrics_summary.csv; v01 is computed locally from its Test_results.json.

Headline result — v01 surface_db

JARVIS surface_db training run — predicts per-slab surface energy surf_en (J/m²). Different physical quantity than v02-v12's adsorption energies, so not directly comparable in absolute terms. 300 epochs on the CPU cluster, 16 cores.

Sub-project	n_train eval / total	n_test eval / total	Best val MSE	Best epoch	Test MAE (J/m²)	Test R²	Test rel. MAE reduction
v01 surface_db¹	478 / 485	58 / 60	0.921	130	0.233	0.823	0.704

¹ Filter rules applied by analyze_with_test.py at analysis time only (the model is trained against the full unfiltered data, so it learns to ignore the bad labels — the filter just keeps DFT-artifact slabs from dominating the reported metrics):

• Train: y > -10 J/m² — drops two slabs with surf_en = -20.1 and -26.7 J/m² (negative surface energy violates bulk-stability thermodynamics).
• Test: y < 10 J/m² — drops two slabs with surf_en = +38.0 J/m² (POSCAR-00211) and +17.6 J/m² (POSCAR-00575). Both are well above the typical clean-surface-energy ceiling (~5 J/m² for most metals); the model predicts ~0.5 J/m² for the latter (residual 17.1), indicating it learned to ignore the DFT-artifact value.
• Val: unfiltered — no extreme outliers landed in this split.

A fifth unphysical slab (surf_en = +130 J/m²) was silently dropped before any of the per-split JSONs were written, by ALIGNN's split-ratio floor rounding + drop_last=True batching, so no filter rule is needed for it. POSCAR-00575 (+17.6) surfaced only after the test-set re-inference at batch_size=1; the original batch_size=16 inference dropped it via drop_last, hiding the fifth outlier from earlier metrics.

One additional borderline value (+12.7 J/m²) is also above the typical clean-surface-energy ceiling but appears in the train fold, where the model fits it cleanly (residual 0.74) — likely physical for a complex surface and kept in the reported metrics.

n_train_eval / total and n_test_eval / total show post-drop_last post-filter counts over the originally-allocated split sizes (485 train, 60 test; v01 lost 5/12/0 to drop_last (train at batch=16, val at batch=16, test at batch=1), then 2/0/2 to the filter rules above).

Parity figures: alignn_v01_surfacedb/analysis/parity_{train,val,test}.png.

Headline result — AGRA ORR (v02-v03)

Two oxygen-reduction-reaction intermediate adsorption energies — *O on v02 and *OH on v03. Both share the same agra_ORR_* reference convention — the Nørskov computational hydrogen electrode (CHE / RHE) framework, confirmed by Batchelor et al. Joule 3, 834 (2019), eqs 3 & 4:

$$\Delta E_{\mathrm{OH}} = E_{\mathrm{slab+OH}} + \tfrac{1}{2} E_{\mathrm{H_2}} - (E_{\mathrm{slab}} + E_{\mathrm{H_2O}})$$

$$\Delta E_{\mathrm{O}} = E_{\mathrm{slab+O}} + E_{\mathrm{H_2}} - (E_{\mathrm{slab}} + E_{\mathrm{H_2O}})$$

Absolute MAE is directly comparable within this cohort. 300 epochs each on the CPU cluster.

Sub-project	n_train eval / total	n_test eval / total	Best val MSE	Best epoch	Test MAE (eV)	Test R²	Test rel. MAE reduction
v02 AGRA_O	800 / 800	100 / 100	0.182	281	0.037	0.980	0.892
v03 AGRA_OH	688 / 701	87 / 87	0.126	257	0.029	0.986	0.892

• Target range is ~2.5 eV — narrowest in the project. Three constraints: (1) one adsorbate per run (*O or *OH); (2) one HEA (IrPdPtRhRu), so the chemistry pool is permutations of those five platinum-group metals; (3) the CHE $\mathrm{H_2O} + \mathrm{H_2}$ reference puts *O and *OH binding energies near the ORR Sabatier-volcano peak (~1 eV), which is naturally positive on these noble metals. Practical implication: small baseline MAE (~0.3 eV) — so the project's lowest absolute test MAE (0.03 eV) translates to a relative MAE reduction of 0.89, strong but below TinNet v08's 0.913 (the relative metric is harder to maximize when the range is already narrow).
• Lowest absolute test MAE in the project (≈0.03 eV). Relative MAE reduction 0.89 on both. The two largest training folds in the catalyst series (n_train = 800 / 701) likely help; both runs trained for 257-281 epochs before hitting best val, suggesting they could absorb the full 300-epoch budget.
• Train fold accounting: v02 divides evenly into batch_size 16 (zero entries dropped); v03's 701 mod 16 = 5 + split-rounding losses give 13 entries dropped. See "Reading the headline tables" above for the general mechanism.

Headline result — AGRA CO₂RR (v04-v06)

Three CO₂-reduction intermediate adsorption energies — *COOH (v04), *CHO (v05), *CO (v06). Same dataset family as AGRA ORR; entry IDs agra_CO2RR_* distinguish the reaction cycle. Source is Chen et al. ACS Catalysis 12, 14864 (2022); reference convention confirmed from Chen 2022 SI eq 1:

$$\Delta E_{X^{\ast}} = E_{X^{\ast}} - E_{\mathrm{cat}} - E_X$$

The SI states explicitly that "the energies of CHO and COOH refer to the energies of H₂ and CO" — so the gas-phase reference is a CO + H₂ basis (CHE-style but not identical to AGRA ORR's H₂O + H₂ basis). Stoichiometric reading:

$$E_X(\mathrm{CO}) = E_{\mathrm{CO}(g)}, \qquad E_X(\mathrm{CHO}) = E_{\mathrm{CO}} + \tfrac{1}{2} E_{\mathrm{H_2}}$$

with $E_X(\mathrm{COOH})$ a linear combination of $E_{\mathrm{H_2}}$ and $E_{\mathrm{CO}}$ (the exact O-reference decomposition isn't fully spelled out in the SI). Within-cohort MAE comparison is safe (all three runs share the same convention).

Sub-project	n_train eval / total	n_test eval / total	Best val MSE	Best epoch	Test MAE (eV)	Test R²	Test rel. MAE reduction
v04 AGRA_COOH	224 / 224	28 / 28	0.048	139	0.048	0.959	0.768
v05 AGRA_CHO	160 / 172	21 / 21	0.060	142	0.073	0.735	0.450
v06 AGRA_CO¹	143 / 155	19 / 19	0.053	186	0.063	0.941	0.735

• Target range is ~2.6 eV across the cohort, split into per-adsorbate sub-ranges (*COOH −1.5 to +0.1, *CHO −2.5 to −0.4, *CO −2.3 to −0.8 eV — each ~1.5 eV wide). Two factors push binding deeper than AGRA ORR: (1) the FeCoNiCuMo HEA includes more reactive 3d metals (Fe, Co, Mo) than ORR's IrPdPtRhRu; (2) the $\mathrm{CO} + \mathrm{H_2}$ reference makes the carbon-metal bond contribution naturally negative on transition-metal surfaces, where CO-on-metal binding is intrinsically strong.
• Smaller datasets, weaker fits than AGRA ORR. Cohort-mean n_train is ~175 vs ORR's ~750. Test MAE roughly doubles (0.05-0.07 vs 0.03 eV); relative MAE reduction drops from ~0.89 to 0.45-0.77.
• Small test folds make R² noisy. v05's R² 0.735 looks weak but its test MAE (0.073) is in line with v04/v06. n_test is only 19-21 and v05's test fold covers only 40% of the dataset's y range (test_coverage_pct = 40.1 in ead_comprehensive_analysis/metrics_summary.csv) — so SS_tot is tiny and R² is hypersensitive to a few mispredictions. Relative MAE reduction (0.450) is the more honest metric and is also lower, consistent with the genuinely harder fit on this small + narrow-range subset.
• v06 had one severe train outlier, now filtered. A single train entry (y = −1.01 eV, predicted −2.09 eV, |residual| = 1.08 eV — 9× larger than the next-worst residual in v06 train) was driving v06's train fit anomaly. After dropping that one point (|residual| > 0.5 eV filter, applied at analysis time only — the model still trained on it), v06 train R² rises from 0.887 → 0.979, MAE 0.040 → 0.033, RMSE 0.099 → 0.041, putting v06 in line with v04 and v05. The most likely cause is a DFT convergence artifact on that one slab (same "model gives up and predicts the mean" signature v01's unphysical slabs showed). See footnote ¹ below.

¹ v06 AGRA_CO train-only filter applied at analysis time (the model still trained on the outlier; only the reported train MAE / RMSE / R² in ead_comprehensive_analysis/metrics_summary.csv need to be updated to reflect the filtered values). Filter rule: drop train entries with |p − y| > 0.5 eV. This catches exactly one entry in v06 (1 of 144 train); zero in v04 or v05 train; zero in v06 test. v06's headline row above shows the filtered count (143 / 155) for the n_train_eval column. Test metrics are unfiltered.

Headline result — TinNet (v07-v09)

Three small-adsorbate chemisorption energies — *N (v07), *O (v08), *OH (v09). Reference state confirmed from Wang, Pillai, Xin Nat. Commun. 11, 6132 (2020) (the Bayesian-chemisorption parent paper the TinNet authors cite): "Gas phase species of O and OH were used as the reference for adsorption energies of *O and *OH, respectively." So *v08 O is gas-phase atomic O, *v09 OH is gas-phase OH radical, and *v07 N is gas-phase atomic N by analogy (the TinNet paper extends the same framework to N but I haven't found an explicit quote — treat v07 as inferred). All three give deeply negative values (−8 to −1 eV). 300 epochs each on the CPU cluster.

Sub-project	n_train eval / total	n_test eval / total	Best val MSE	Best epoch	Test MAE (eV)	Test R²	Test rel. MAE reduction
v07 tinnet_N	256 / 263	32 / 32	0.142	215	0.106	0.928	0.757
v08 tinnet_O	592 / 597	74 / 74	0.512	43	0.114	0.986	0.913
v09 tinnet_OH	592 / 598	74 / 74	0.328	53	0.075	0.975	0.867

• Target range is ~7 eV (−8 to −1 eV) — driven by the atomic-reference convention. Free gas-phase O and N atoms are the highest-energy reference points possible (no bonds in vacuum); any binding to a metal surface is strongly favorable, pushing all ΔE values deeply negative. The wide spread comes from the 26 metals × multiple host families covered (intermetallics, near-surface alloys, single-atom alloys, Pt-bimetallics), so d-band centers and coordination geometries vary much more than in AGRA's fixed-composition HEAs. v08 *O has the widest within-cohort range (~6.5 eV, from −7.9 to −1.4) because oxygen is the most chemically reactive of the three TinNet adsorbates.
• v08 has the highest relative MAE reduction in the entire project (0.913). The atomic-reference convention gives TinNet a 5-7 eV target spread, so baseline MAE is large (1.31 eV for v08) and even a "modest" model MAE of 0.114 eV looks excellent in relative terms. The metric rewards wide dynamic range when the model can track it.
• TinNet converges fast. v08 and v09 hit best val at epochs 43 and 53 — much earlier than AGRA (139-281) or v10 OCP (296). This is the empirical evidence that motivated n_early_stopping: 20 for the v10-v12 GPU configs; without it, the runs would have spent ~80% of the 300-epoch budget post-plateau.
• Wider dynamic range keeps test R² high despite small n_test. Test R² 0.93-0.99 on n_test = 32-74. Compare to v05/v06's noisy R² at similar size: wider y span makes SS_tot larger and R² more robust.

Headline result — OCP (v10-v12)

OC20 IS2RE task: predict the relaxed-system adsorption energy from the initial (unrelaxed) structure. Reference convention confirmed from Chanussot et al. ACS Catal. 11, 6059 (2021):

$$E_{\mathrm{ad}} = E_{\mathrm{sys}} - E_{\mathrm{slab}} - E_{\mathrm{gas}}$$

where $E_{\mathrm{gas}}$ is a linear combination of $\mathrm{N_2}$, $\mathrm{H_2O}$, $\mathrm{CO}$, and $\mathrm{H_2}$ chosen to match each adsorbate's composition (4-reference scheme; atomic-energy coefficients tabulated in the OC20 SI). The relaxed_energy field in the JARVIS mirror is already this reference-corrected adsorption energy, not a raw DFT total. Absolute MAE is directly comparable within this cohort. All three runs finished on a single A100 on DSAI; v11 and v12 hit the SLURM 72-hour cap before completing their 300-epoch budget.

Sub-project	n_train eval / total	n_test eval / total	Best val MSE	Best epoch	Test MAE (eV)	Test R²	Test rel. MAE reduction
v10 ocp10k	47,872 / 47,908	5,988 / 5,988	24.80	296 / 300	0.293	0.916	0.832
v11 ocp100k²	119,872 / 119,908	14,988 / 14,988	120.84	156 / 156	0.538	0.842	0.695
v12 ocp_all²	408,128 / 408,171	51,021 / 51,021	442.27	39 / 58	0.556	0.843	0.682

² Wall-clock-truncated: both v11 (156 epochs) and v12 (58 epochs) hit the cluster's 72-hour SLURM limit before reaching the configured 300 epochs. v11's best val was achieved at the very last completed epoch (156/156) — the model was still improving when killed, so its reported metrics represent an under-trained checkpoint, not the architecture's ceiling at this scale. v12 is different — its best val landed at epoch 39 of 58 completed, with the next 19 epochs showing no improvement; v12 had genuinely plateaued well before the wall-clock cancellation and would have triggered n_early_stopping=20 had the job continued one more epoch. Full 300-epoch projections (train-only, never realised): v11 ≈ 5.2 days, v12 ≈ 11.6 days — both fundamentally incompatible with a 3-day cluster cap.

v10/v11/v12 observations:

• Target range is ~19 eV wide — ~3× any other cohort (next-widest is TinNet at ~6.5 eV). Four compounding reasons: (1) OC20 spans 82 adsorbates × 55 elements vs. v02-v09's single-adsorbate, constrained-alloy slices; (2) the OC20 sampler intentionally covers both very-oxophilic early transition metals (→ −9 eV tail) and inert / detached-adsorbate configurations (Au, Ag, mispositioned → near zero or positive); (3) the 4-reference $E_{\mathrm{gas}}$ scheme references multi-atom adsorbates against ~6 gas-phase species — small DFT errors in any one shift $\Delta E$ noticeably; (4) the IS2RE task starts from unrelaxed initial structures, and the relaxation trajectories vary widely. Practical implication: v10's absolute MAE (0.293 eV) looks worse than v02-v09 (0.03-0.11 eV) because the baseline MAE is also much larger (1.74 eV vs 0.13-1.31 eV) — exactly why the relative MAE reduction metric (0.832 here) is the meaningful cross-cohort comparison.
• Did not plateau by epoch 300. Best val landed at epoch 296 of 300 — the loss was still drifting down slowly. The n_early_stopping: 20 watchdog never triggered (best val kept improving within 20-epoch windows; the run hit the epoch-budget ceiling instead of the patience floor). Unlike v01 (plateaued by epoch 130) or TinNet v08/v09 (epochs 43-53), v10 could plausibly absorb 400-500 epochs.
• Train-to-test gap is wider than v02-v09. Train R² 0.998 vs Test R² 0.916 (≈8 pp drop) vs ~0-6 pp for the v02-v09 cohorts. Expected: OCP10k mixes adsorbates, surfaces, and metals across the full OC20 sampler, so the held-out test fold can contain configurations the train fold underrepresents. Relative MAE reduction 0.832 is still competitive against the project range (v02-v09 span 0.45-0.91).
• Tail underfit on extreme-y entries. Largest test residuals (~8 eV in v10, ~10–12 eV in v11/v12) sit on the most negative relaxed_energy configurations — the model damps predictions toward the mean for tail extremes. Same behavior v01 showed for unphysical slabs, but here the underfit entries are legitimate OC20 data.
• More data did not improve generalisation in this training budget. v11 (150k entries) and v12 (510k entries) both report worse test metrics than v10 (60k): test rel. MAE reduction 0.695, 0.682, vs. v10's 0.832. v11 is under-trained — still improving at termination, so 0.695 is a lower bound rather than the architecture's ceiling at 150k entries. v12 is more diagnostic: it plateaued at epoch 39 with rel. MAE reduction 0.682 and would not have improved at this learning-rate schedule with more time. This could be a capacity ceiling at the 510k scale, or it could be that the OneCycle LR schedule (configured for 300 epochs) hadn't yet entered the fine-tuning phase by epoch 39 of an aborted run. Distinguishing the two requires either multi-GPU parallelism, a higher walltime allowance, or a re-tuned LR schedule for shorter training — all beyond the current setup.

Test predictions for all three OCP runs are from infer_test.py at batch_size=1 for full coverage (alignn 2026.4.2 does not evaluate the held-out test split automatically). Parity figures: alignn_v1?_*/analysis/parity_{train,test}.png. Composition-stats and per-dominant-metal target distributions are in each run's analysis/composition_* artifacts.

Parity figures + loss curves per run live in each alignn_v0?_*/analysis/.

Cross-dataset comparison (metrics CSV + 6 summary figures): ead_comprehensive_analysis/ covers the v02–v09 ead runs; rele_comprehensive_analysis/ covers the v10–v12 OC20 relaxed_energy runs. The two cohorts predict different quantities (ead ranges ±2 to ±8 eV with family-specific references; relaxed_energy is OC20's 4-reference target on the same eV scale but a different zero), so they are tracked separately.

Cross-cohort comparability

The eleven adsorption-energy runs (v02-v12) use four different reference-state conventions, so absolute MAE / RMSE are only directly comparable within a cohort. The cohorts:

Cohort	Runs	Reference convention	Source paper
AGRA ORR	v02 AGRA_O, v03 AGRA_OH	Nørskov CHE / RHE with $\mathrm{H_2O} + \mathrm{H_2}$ basis — see headline section above for the explicit eqs	Batchelor et al. Joule 3, 834 (2019), eqs 3 & 4
AGRA CO₂RR	v04 AGRA_COOH, v05 AGRA_CHO, v06 AGRA_CO	$\Delta E_{X^{\ast}} = E_{X^{\ast}} - E_{\mathrm{cat}} - E_X$ with $\mathrm{CO} + \mathrm{H_2}$ gas-phase basis — see headline section above for the per-adsorbate $E_X$ decomposition	Chen et al. ACS Catal. 12, 14864 (2022), SI eq 1
TinNet	v07 tinnet_N, v08 tinnet_O, v09 tinnet_OH	Gas-phase species reference: *O ← atomic O(g), *OH ← OH radical(g), *N ← atomic N(g) by extension	Wang, Pillai, Xin Nat. Commun. 11, 6132 (2020) for *O / *OH; *N inferred from same framework
OC20 IS2RE	v10 ocp10k, v11 ocp100k, v12 ocp_all	$E_{\mathrm{ad}} = E_{\mathrm{sys}} - E_{\mathrm{slab}} - E_{\mathrm{gas}}$ with 4-reference $(\mathrm{N_2}, \mathrm{H_2O}, \mathrm{CO}, \mathrm{H_2})$ linear combination	Chanussot et al. ACS Catal. 11, 6059 (2021), arXiv:2010.09990v5 page 6

The two AGRA cohorts (ORR v02-v03 vs CO₂RR v04-v06) aren't directly comparable: although both are CHE-flavored, they use different gas-phase reference bases — AGRA ORR uses $\mathrm{H_2O} + \mathrm{H_2}$ (per Batchelor 2019), AGRA CO₂RR uses $\mathrm{CO} + \mathrm{H_2}$ (per Chen 2022 SI eq 1). The absolute $\Delta E$ values from the two AGRA cohorts therefore sit on different zeros. Use the cohort-neutral relative MAE reduction metric (defined in Reading the headline tables above; values are in the last column of every cohort table).

v01 stands alone. surf_en (J/m²) is a different physical quantity entirely — energy to create a surface from bulk, not adsorption — so it isn't comparable to any of the ead-style runs.

Compute scale (v01-v12)

v01-v09 ran end-to-end on the CPU cluster. v10-v12 were attempted there too but proved infeasible — v10 managed only 2 epochs in 7 hours, v11 didn't finish a single epoch, v12 was still in pre-training graph construction at 3.5 hours. All three were cancelled and migrated to the DSAI cluster's a100 GPU partition with a GPU-flavored config (batch_size: 64, learning_rate: 0.002, n_early_stopping: 20, --device cuda). v10 finished on a single A100-SXM4-80GB in 29 h 19 m of total job wall-clock; v11 and v12 are still running.

Mean atoms per entry, measured from the local zips:

Family	n_atoms range	mean atoms
v01 surface_db	8-112	31
v02 AGRA_O, v03 AGRA_OH, v07-v09 (TinNet N/O/OH)	17-19 (single fixed slab per dataset)	17-19
v04 AGRA_COOH, v05 AGRA_CHO, v06 AGRA_CO	66-68 (single fixed slab per dataset, larger base)	66-68
v10 OCP (v11/v12 to be measured)	7-225	78

Sub-project	n	Settings	prepare_data.py	Per-epoch	300-epoch wall-clock
v01 surface_db	607	batch=16, 16 cores	<1 min	48 s	4 h 0 m
v02 AGRA_O	1000	batch=16, 16 cores	<1 min	48 s	4 h 1 m
v03 AGRA_OH	877	batch=16, 16 cores	<1 min	45 s	3 h 46 m
v04 AGRA_COOH	280	batch=16, 16 cores	<1 min	58 s	4 h 51 m
v05 AGRA_CHO	216	batch=16, 16 cores	<1 min	44 s	3 h 38 m
v06 AGRA_CO	194	batch=16, 16 cores	<1 min	39 s	3 h 17 m
v07 tinnet_N	329	batch=16, 16 cores	<1 min	20 s	1 h 40 m
v08 tinnet_O	747	batch=16, 16 cores	<1 min	41 s	3 h 24 m
v09 tinnet_OH	748	batch=16, 16 cores	<1 min	29 s	2 h 24 m
v10 ocp10k	59,886	batch=64, 1× A100 GPU	1 m 5 s	5 m 9 s²	25 h 45 m²
v11 ocp100k	149,886	batch=64, 1× A100 GPU	~3 min	24 m 49 s³	64 h 30 m (156 epochs only)³
v12 ocp_all	510,214	batch=64, 1× A100 GPU	~10 min	55 m 35 s³	53 h 44 m (58 epochs only)³

A few observations from the measured runs:

• Per-epoch wall-clock scales roughly with n × mean_atoms (using dataset n, not n_train). On the CPU cluster the ratio runs 0.0026-0.0032 s per (entry × atom) across v01-v08 (mean ~0.0030). v02 happens to match v01's 48 s/epoch despite 2× the entries because v02's slabs are ~half the atom count, so the products are similar.
• The AGRA family splits into two slab-size groups. v02 AGRA_O and v03 AGRA_OH use a small base slab (17-18 atoms); v04 AGRA_COOH, v05 AGRA_CHO, and v06 AGRA_CO use a larger base slab (66-68 atoms). Within the large-slab group, v04 is the slowest simply because it has the most entries (n=280 vs 216/194), not because COOH is a larger adsorbate.
• v07 is fast because it's the smallest workload, not because the structures are smaller. n=329 × 19 atoms = 6,251 entry-atoms is the lowest in the catalyst set; the per-epoch rate (20 s) matches the same ~0.003 s/(entry-atom) constant.
• The A100 per-(entry × atom) rate is ~45× faster than the CPU cluster. v10's 309 s/epoch over 59,886 × 77.8 = 4.66 M entry-atoms works out to 6.6 × 10⁻⁵ s per (entry × atom) on a single A100-SXM4-80GB, versus ~0.003 s on the CPU cluster. v11/v12 per-epoch times will be reported once those runs finish.

² v10 measured per-epoch (mean over 300 epochs) = 309 s on A100; 300-epoch training time only. Add ~1 h 36 m of one-time line-graph pre-compute (47,908 train + 5,988 val + 5,988 test graphs at ~10 it/s, single-threaded CPU) for total job wall-clock of 29 h 19 m. The training-only number is what's tabulated for cross-row comparability with v01-v09 (which had negligible pre-compute on smaller datasets).

³ v11 and v12 did not reach the 300-epoch budget: both jobs hit the SLURM 72-hour cluster cap and were cancelled mid-training. v11 completed 156 epochs at 24 m 49 s/epoch (best val at epoch 156/156 — still improving when killed); v12 completed 58 epochs at 55 m 35 s/epoch (best val at epoch 39/58 — had genuinely plateaued; n_early_stopping=20 would have triggered one epoch later). The "300-epoch wall-clock" column reports the actual training time completed, not a projection. Full-300-epoch projections (train-only, never realised): v11 ≈ 5.2 days, v12 ≈ 11.6 days.

All three OCP runs now use the GPU config — batch_size: 64, learning_rate: 0.002, n_early_stopping: 20, pin_memory: true, num_workers: 8, --device cuda — and a single A100 per job on DSAI's a100 partition (3-day wall-clock max). Early stopping at 20 epochs is essential because v01 plateaued by epoch 130 of 300 and v08/v09 plateaued at 43/53; without it, 300 epochs would burn most of the 3-day budget for no metric gain.

For v12, run prepare_data.py in a detachable tmux or screen session — the multi-hour wall-clock makes ssh disconnects costly.

v12 at full 300 epochs is likely infeasible on a CPU-only cluster even at the bumped settings. GPU training (the original OC20 recipe) is the right venue for full-scale OCP work.

Findings

What v01–v10 demonstrate, synthesized:

Pipeline reproducibility. ALIGNN 2026.4.2 — applied through a uniform JARVIS-backed pipeline with no per-dataset architecture tuning — produces test MAE that beats every upstream-paper baseline on the catalyst runs:

• v02-v06 (AGRA datasets): v02 *O 0.037 eV, v03 *OH 0.029, v04 *COOH 0.048, v05 *CHO 0.073, v06 *CO 0.063 — all lower than the AGRA paper's Table II ALIGNN-on-AGRA-graphing results (Gariepy et al. APL Machine Learning 2023; 0.047 / 0.034 / 0.065 / 0.095 / 0.095 eV respectively). Most likely cause of the gap: different train/val/test split seeds (the AGRA paper averages 5 retrains; ours is a single fixed split).
• v07-v09 (TinNet datasets): v07 *N 0.106, v08 *O 0.114, v09 *OH 0.075 — all lower than the TinNet paper's reported TinNet-model MAEs (Wang, Pillai, Wang, Achenie, Xin Nat. Commun. 12, 2021; 0.116 / 0.147 / 0.118 eV respectively, from Fig. 2b, 5a, 5b).
• v10 OC20 IS2RE: 0.293 eV beats the OC20 paper's reported CGCNN / SchNet / DimeNet++ baselines (0.62 / 0.64 / 0.56 eV; Chanussot et al. 2020, Table 6, In-Domain test). Direct comparison is approximate — v10 trained on ~48k entries vs. the OC20 paper's 460k, on a different held-out test split, with a stronger model architecture (ALIGNN vs the three earlier graph baselines).

Cohort-neutral score. Test relative MAE reduction spans 0.45–0.913 across the eleven ead-style runs. v08 TinNet *O is the best-fit run (0.913), v05 AGRA *CHO is the worst (0.450) — the latter driven by small train fold (n=160) + narrow test-fold coverage (40% of the y-range), not by any genuine model failure. Absolute MAE varies 10× across cohorts (0.029–0.293 eV) entirely because of reference-convention-driven differences in baseline MAE; the relative metric corrects for this.

Scale-up to OC20 reveals a wall-clock floor. v10 (60k entries × 82 adsorbates × 55 elements) achieves relative MAE reduction 0.832 after 296/300 epochs in ~26 h. Larger OC20 subsets are wall-clock-bound: v11 (150k) trained for 156/300 epochs and was still improving when killed at the 72-h cluster cap (rel. MAE reduction 0.695 at termination — a lower bound, not a converged value); v12 (510k) trained for 58/300 epochs but plateaued at epoch 39 (rel. MAE reduction 0.682, genuinely converged at this scale; n_early_stopping=20 would have fired one epoch after the SLURM cancellation). The 300-epoch projection for v12 is ~11.6 days on a single A100 — fundamentally incompatible with the cluster's 3-day cap. So ALIGNN's IS2RE quality on broad-chemistry OC20 plateaus at 0.68 rel. MAE reduction at the 510k scale under this training budget; whether the result is architecture-limited or capacity-limited is a follow-up question.

CPU→GPU scaling. A single A100-SXM4-80GB is ~45× faster per (entry × atom) than the CPU cluster for ALIGNN training. Concretely: v10's 60k-entry run finished 300 epochs in ~26 h on A100 vs. failing to complete 2 epochs in 7 h on CPU. This sets the threshold above which GPU is mandatory for ALIGNN catalysis-ML at this scale.

Model-robustness signature. Across v01, v10, v11, and v12, the model damps predictions toward the cohort mean for extreme-y outliers. This is a feature on noisy DFT data (v01's unphysical surf_en slabs) but a limitation for legitimate-but-rare extreme binding (v10's tail underfit at −8.7 eV; v11/v12 show even larger ~10–12 eV tail residuals on extreme configurations).

Chemistry of the datasets. Top-site metal extraction and per-element ΔE statistics confirm well-known oxophilicity trends inside each cohort and quantify the reference-convention gap across them:

• *v02 AGRA O within the IrPdPtRhRu HEA: Ru (mean +0.46 eV) < Ir (+0.92) < Rh (+1.18) < Pt (+1.75) < Pd (+1.88) — Pd is the most oxophobic site, matching the Sabatier-volcano position for ORR.
• *v08 TinNet O spans 35 unique metal-like elements: Nb (mean −6.96 eV), W (−6.77), V (−6.72) are the strongest *O binders; Ag (−1.87) and Au (−2.13) the weakest — early-vs-late transition-metal d-band trend, atomic-reference scale.
• Same metal, two cohorts: top-site Pt averages +1.75 eV in v02 (AGRA HEA, CHE H₂O+H₂ reference) but −3.02 eV in v08 (TinNet SAA, atomic-O reference). The 4.8 eV gap is entirely the reference convention — a concrete demonstration of why absolute ΔE comparison across cohorts is meaningless (and why the relative MAE reduction metric is necessary). See ead_comprehensive_analysis/fig8_per_metal_binding_cross_cohort.png.
• Per-dataset composition_stats.csv + composition_analysis.png in each alignn_v?X/analysis/ directory (v01-v12). Cross-cohort chemistry_summary.csv (one row per dataset, 12 total) lives in ead_comprehensive_analysis/. Cross-cohort figures are split: ead_comprehensive_analysis/fig7-fig9 compare v02-v09 under different reference conventions (showing the 4.8 eV Pt gap), while rele_comprehensive_analysis/fig7-fig9 compare v10/v11/v12 as OC20 subsamples (showing the smaller cohorts faithfully reproduce v12's per-element and per-metal distributions).

License

MIT, see LICENSE.