Dev/v2026.05

.github/workflows/tests.yml

@@ -2,25 +2,23 @@ name: Tests
 
 on:
   push:
-    branches: [ main, master, dev/v2026.01 ]
+    branches: [ main, master, dev/v2026.01, dev/v2026.05 ]
   pull_request:
-    branches: [ main, master, dev/v2026.01 ]
+    branches: [ main, master, dev/v2026.01, dev/v2026.05 ]
 
 jobs:
   test:
     runs-on: ubuntu-latest
-    env:
-      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true
     strategy:
       fail-fast: false
       matrix:
         python-version: ["3.9", "3.10", "3.11", "3.12", "3.13", "3.14"]
 
     steps:
-    - uses: actions/checkout@v4
-    
+    - uses: actions/checkout@v5
+
     - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v5
+      uses: actions/setup-python@v6
       with:
         python-version: ${{ matrix.python-version }}
         allow-prereleases: true

.junie/instructions/data.md

@@ -1,19 +1 @@
-# Reference Data & Metadata Management
-
-EM-TEST relies on reference data to validate EM-DAT content.
-
-## 1. Reference Files
-Reference data is stored in `emtest\validation_data\`.
-- **`classification_tree.toml`**: The source of truth for disaster types and subgroups.
-- **`UNSD_M49_standards.csv`**: Reference for country and region names.
-- **`gaul_adm1_code.txt`**: GAUL administrative level 1 codes.
-
-## 2. Loading Data
-Reference data is loaded by `emtest\validation_data\data_loader.py`.
-When adding new reference files, update the `data_loader.py` and ensure they are included in `pyproject.toml` under `tool.setuptools.package-data`.
-
-## 3. Metadata and Citations
-- **`CITATION.cff`**: Project citation info for scientific researchers.
-- **`LICENSE`**: MIT License.
-- **Versioning**: Follow semantic versioning, which is reflected in `pyproject.toml`.
-- **README Updates**: If you add new validation checks, update the corresponding tables in `README.md`.
+Moved to [`instructions/data.md`](../../instructions/data.md).

.junie/instructions/domain.md

@@ -1,22 +1 @@
-# EM-DAT Domain Knowledge
-
-EM-TEST validates data related to international disasters.
-
-## 1. Geospatial Consistency
-- **ISO3 Codes**: Country codes should be valid ISO-3166-1 alpha-3 codes.
-- **Geography**: Valid latitude (-90 to 90) and longitude (-180 to 180).
-- **Standards**: The project follows UNSD M49 standards for country and area codes.
-
-## 2. Temporal Logic
-Disaster numbers (`DisNo.`) contain the year the disaster was recorded.
-- **Consistency**: `Start Year` should generally match the year in `DisNo.`.
-- **Duration**: `Start Year/Month/Day` must be before or equal to `End Year/Month/Day`.
-
-## 3. Classification
-Disasters are categorized hierarchically:
-- **Disaster Group** (e.g., Natural)
-- **Disaster Subgroup** (e.g., Meteorological)
-- **Disaster Type** (e.g., Storm)
-- **Disaster Subtype** (e.g., Tropical cyclone)
-
-Validation must ensure that combinations of these fields follow the official `classification_tree.toml`.
+Moved to [`instructions/domain.md`](../../instructions/domain.md).

.junie/instructions/validation.md

@@ -1,23 +1 @@
-# Data Validation with Pandera and Pandas
-
-The core of EM-TEST is built on `pandera` and `pandas`.
-
-## 1. Schema Definition
-Schemas are defined in `emtest\validation_schemas.py`.
-- **`DataFrameSchema`**: Use this to define per-column constraints.
-- **Nullability**: In Pandas, the standard `int` type is not nullable. Use `float` for columns that contain integers but may have null values (e.g., `Start Month`, `Start Day`).
-
-## 2. Custom Checks
-Complex validation logic should be implemented in `emtest\custom_checks.py`.
-- **Single-Column Checks**: Implement logic for a specific field (e.g., `check_disno`, `check_month`).
-- **Multi-Column ("Wide") Checks**: Use these for consistency between fields (e.g., `check_start_end_year_consistency`).
-
-## 3. Regular Expression Engineering
-Identifiers are often validated via Regex:
-- **DisNo format**: `^\d{4}-\d{4}-[A-Z]{3}$` (Year-Sequence-CountryISO).
-- **External IDs**: GLIDE numbers and other identifiers rely on specific patterns.
-
-## 4. Testing Pattern
-Since there are no traditional automated tests, verify changes using the example script:
-- `python examples\validation_script.py`
-This script uses the schema to validate a sample dataset.
+Moved to [`instructions/validation.md`](../../instructions/validation.md).

AGENTS.md

@@ -0,0 +1,81 @@
+# EM-TEST Agent Instructions
+
+EM-TEST is a Python data validation framework for [EM-DAT](https://www.emdat.be/) disaster datasets. It validates disaster event DataFrames against a comprehensive schema of type constraints, business rules, and multi-column consistency checks using `pandera` and `pandas`.
+
+Versioning follows CalVer: `YYYY.MM.N` (e.g., `2026.04.0`). Version is defined in `emtest/__init__.py` and must also be updated in `CITATION.cff` on release.
+
+## Commands
+
+```bash
+# Install with all dev dependencies (preferred)
+uv sync --all-extras --dev
+
+# Run all tests
+uv run pytest
+
+# Run a single test file
+uv run pytest tests/test_custom_checks.py
+
+# Run a specific test
+uv run pytest tests/test_custom_checks.py::test_check_disno
+```
+
+No Makefile or linting config exists — the project relies on `pytest` for correctness checks and manual review.
+
+## Architecture
+
+```
+emtest/
+├── __init__.py              # Public API: exports emdat_schema and __version__
+├── validation_schemas.py    # DataFrameSchema: 50+ columns + multi-column checks
+├── custom_checks.py         # Check functions used inside validation_schemas.py
+├── utils.py                 # get_validation_report(), update_column_checks(), etc.
+└── validation_data/         # Reference data package
+    ├── data_loader.py       # Loads TOML/CSV/TXT files via Path(__file__).parent
+    ├── classification.py    # Disaster classification tree (loaded from .toml)
+    ├── areas.py             # ISO3 codes, country names, regions, GAUL codes
+    └── magnitude.py         # Magnitude scale units per disaster type
+```
+
+**Data flow**: user loads an Excel/CSV file → passes a DataFrame with `DisNo.` as index to `emdat_schema.validate(df, lazy=True)` → pandera raises `SchemaErrors` → `get_validation_report()` converts errors to a flat DataFrame for export.
+
+## Domain Knowledge
+
+See [`instructions/domain.md`](instructions/domain.md) for the full reference. Key points:
+
+- **ISO3 / geography**: country codes follow ISO-3166-1 alpha-3; latitude −90–90, longitude −180–180; regions follow UNSD M49.
+- **DisNo. year**: `Start Year` should generally match the four-digit year embedded in `DisNo.` (a warning, not an error).
+- **Date chronology**: `Start Year/Month/Day` ≤ `End Year/Month/Day` at every available resolution.
+- **Classification**: Group → Subgroup → Type → Subtype combinations are validated against `classification_tree.toml`.
+
+## Non-Obvious Design Decisions
+
+**Nullable integer columns use `float`**: Fields like `Start Month`, `Start Day`, `End Month`, `End Day` can be null. Pandas `int` dtype is non-nullable, so these are defined as `float` with `NaN` for missing values throughout the schema and checks.
+
+**Wide (multi-column) checks and deduplication**: Multi-column check functions in pandera fail every row they touch, causing duplicated error rows in reports. `deduplicate_errors()` in `utils.py` filters these so each logical failure appears once. When adding new wide checks, define which column should "own" the error in `WIDE_CHECKS_TO_KEEP`.
+
+**Index access inside column checks**: The `check_disno_vs_start_year` check compares the year embedded in `DisNo.` (the DataFrame index) against the `Start Year` column. Since the index isn't a regular column in pandera, this check is attached to the `Start Year` column and accesses `series.index` internally.
+
+**Parameterized date consistency via `partial()`**: `check_start_end_consistency()` in `custom_checks.py` is called three times (year/month/day resolution) using `functools.partial()`. The helper `_convert_to_date` assembles a date string and calls `pd.to_datetime(..., errors='coerce')`. Two subtleties: (1) month/day NaN values map to `'00'`, producing an invalid date string that coerces to `NaT` — the `pd.isna()` guard in the caller then skips the comparison, which is the intended behaviour for incomplete dates; (2) year columns must be converted via `int()` before `str()` because when any row has a null `End Year`, pandas reads the entire column as `float64`, making `.astype(str)` produce `'2024.0'` instead of `'2024'`.
+
+**GAUL code validation is expensive**: `has_valid_GAUL_codes()` parses JSON strings in the `Admin Units` column, then validates codes against two large reference lists (ADM1: ~1k codes, ADM2: ~50k codes). It is only applied to non-null values. If performance is an issue on large datasets, this check is the likely bottleneck.
+
+**Disaster type-specific magnitude rules**: Magnitude validation branches on `Classification Key` (a semicolon-delimited hierarchy like `nat;geo;earthquake`). Earthquake expects 3–10, cold wave ≤10°C, heat wave ≥25°C, everything else >0. The logic lives in `custom_checks.py` functions like `check_earthquake_magnitude`.
+
+**Warnings vs errors**: Some checks (ISO code, country name, start year vs DisNo. year, CPI range) are configured as `raise_warning=True` in `validation_schemas.py`. `get_validation_report(add_warnings=True)` includes them; `set_warnings_to_errors()` in `utils.py` promotes all warnings to hard errors.
+
+## Tests
+
+`tests/conftest.py` provides two fixtures reused across test files:
+- `valid_emdat_row` — dict with all 50 columns set to valid values
+- `valid_df` — single-row DataFrame with `DisNo.` as index
+
+The typical test pattern mutates one field of `valid_emdat_row`, constructs a DataFrame, and asserts that `emdat_schema.validate()` raises `SchemaError`.
+
+## Detailed Instructions
+
+For specific tasks, refer to the following guides in `instructions/`:
+
+- [**Data Validation**](instructions/validation.md): Writing schemas, custom checks, and handling nullable types.
+- [**EM-DAT Domain**](instructions/domain.md): Geospatial, temporal, and classification logic for disaster data.
+- [**Reference Data & Metadata**](instructions/data.md): Managing reference files and project metadata (`CITATION.cff`).

CITATION.cff

@@ -19,6 +19,6 @@ keywords:
   - testing-framework
   - python
 license: MIT
-version: 2026.04.0
-date-released: '2026-04-08'
+version: 2026.05.0
+date-released: '2026-04-30'
 

README.md

@@ -3,14 +3,14 @@
 [![Tests](https://github.com/em-dat/em-test/actions/workflows/tests.yml/badge.svg)](https://github.com/em-dat/em-test/actions/workflows/tests.yml)
 ![Python Versions](https://img.shields.io/badge/python-3.9%20%7C%203.10%20%7C%203.11%20%7C%203.12%20%7C%203.13%20%7C%203.14-blue)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
-[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.14275790.svg)](https://doi.org/10.5281/zenodo.14275790)
+[![DOI](https://img.shields.io/badge/DOI-10.5281%2Fzenodo.14275790-blue)](https://doi.org/10.5281/zenodo.14275790)
 
 EM-TEST is a testing framework for [EM-DAT](https://www.emdat.be/) public
 data, built on the [`pandas`](https://pandas.pydata.org/) and
 [`pandera`](https://pandera.readthedocs.io/en/stable/) Python packages.
 
 > [!IMPORTANT]
-> This version of EM-TEST has been built for EM-DAT public data on 2026/04/08.
+> This version of EM-TEST has been built for EM-DAT public data on 2026/04/30.
 > Some tests might fail for versions prior to this date. EM-TEST is not
 > suitable for EM-DAT versions prior to September 26, 2023.
 
@@ -245,6 +245,7 @@ See [EM-DAT Documentation](https://doc.emdat.be/docs/protocols/economic-adjustme
 | Total Damage, Adjusted ('000 US$)         | greater_than(0.)                 | Test whether value is greater than 0                   | Error     |
 | CPI                                       | in_range(0., 110.)               | Test whether value is within range 0-110.              | Warning   |
 | Admin Units                               | is_valid_json                    | Test whether value is a json string                    | Error     |
+|                                           | has_valid_GAUL_codes             | Test whether value contains valid GAUL codes           | Error     |
 | GADM Admin Units                          | is_valid_json                    | Test whether value is a json string                    | Error     |
 | Entry Date                                | in_range(1988/1/1, CURRENT_DATE) | Test whether value is within valid date range          | Error     |
 | Last Update                               | in_range(1988/1/1, CURRENT_DATE) | Test whether value is within valid date range          | Error     |
@@ -293,15 +294,15 @@ citation below or the citation metadata file `CITATION.cff`.
   month        = dec,
   year         = 2026,
   publisher    = {Zenodo},
-  version      = {2026.04.0},
+  version      = {2026.05.0},
   doi          = {10.5281/zenodo.14275790},
   url          = {https://doi.org/10.5281/zenodo.14275790}
 }
 ```
 Or
 
 >Delforge, D., & Wathelet, V. (2026). EM-TEST: A Testing Framework for the 
-> EM-DAT Data (2026.04.0). Zenodo. https://doi.org/10.5281/zenodo.14275790
+> EM-DAT Data (2026.05.0). Zenodo. https://doi.org/10.5281/zenodo.14275790
 
 ## Useful Links
 

checklist.md

@@ -28,6 +28,7 @@ This checklist outlines the items to check, modify, and do before finalizing the
 
 ## 5. Testing and CI/CD
 - [ ] Run the full test suite using `pytest`.
+- [ ] Expand CI workflow to include version branch testing.
 - [ ] Ensure all GitHub Actions workflows are passing.
 - [ ] Add regression tests for any bugs fixed in this release cycle.
 - [ ] Verify that the package can be built successfully (`python -m build`).

emtest/__init__.py

@@ -1,4 +1,4 @@
-__version__ = '2026.04.0'
+__version__ = '2026.05.0'
 __author__ = 'Damien Delforge, Valentin Wathelet'
 
 from emtest.validation_schemas import (

emtest/custom_checks.py

@@ -242,23 +242,27 @@ def float_to_string(x):
         else:
             return str(int(x)).zfill(2)
 
+    year = df[f'{start_or_end} Year'].apply(
+        lambda x: None if pd.isna(x) else str(int(x))
+    )
+
     if resolution == 'day':
         return pd.to_datetime(
-            df[f'{start_or_end} Year'].astype(str) +
+            year +
             df[f'{start_or_end} Month'].apply(float_to_string) +
             df[f'{start_or_end} Day'].apply(float_to_string),
             format='%Y%m%d',
             errors='coerce'
         )
     elif resolution == 'month':
         return pd.to_datetime(
-            df[f'{start_or_end} Year'].astype(str) +
+            year +
             df[f'{start_or_end} Month'].apply(float_to_string),
             format='%Y%m',
             errors='coerce'
         )
     elif resolution == 'year':
-        return pd.to_datetime(df[f'{start_or_end} Year'], format='%Y')
+        return pd.to_datetime(year, format='%Y', errors='coerce')
 
 
 def _extract_GAUL_code(d: dict) -> tuple[int, int]:

emtest/utils.py

@@ -7,7 +7,6 @@
 
 WIDE_CHECKS_TO_KEEP: dict[str, list[str]] = {
     'Missing latitude or longitude coordinates': ['Latitude', 'Longitude'],
-    'Start date and end date inconsistency': ['Start Year'],
     'Invalid coldwave magnitude': ['Magnitude'],
     'Invalid earthquake magnitude': ['Magnitude'],
     'Invalid heatwave magnitude': ['Magnitude'],
@@ -20,7 +19,6 @@
 }
 
 
-
 def get_validation_report(
         df: pd.DataFrame,
         schema: DataFrameSchema,

instructions/data.md

@@ -0,0 +1,19 @@
+# Reference Data & Metadata Management
+
+EM-TEST relies on reference data to validate EM-DAT content.
+
+## 1. Reference Files
+Reference data is stored in `emtest/validation_data/`.
+- **`classification_tree.toml`**: The source of truth for disaster types and subgroups.
+- **`UNSD_M49_standards.csv`**: Reference for country and region names.
+- **`gaul_adm1_code.txt`**: GAUL administrative level 1 codes.
+
+## 2. Loading Data
+Reference data is loaded by `emtest/validation_data/data_loader.py`.
+When adding new reference files, update `data_loader.py` and ensure they are included in `pyproject.toml` under `tool.setuptools.package-data`.
+
+## 3. Metadata and Citations
+- **`CITATION.cff`**: Project citation info for scientific researchers.
+- **`LICENSE`**: MIT License.
+- **Versioning**: Follow CalVer (`YYYY.MM.N`), reflected in `emtest/__init__.py`, `pyproject.toml`, and `CITATION.cff`.
+- **README Updates**: If you add new validation checks, update the corresponding tables in `README.md`.

instructions/domain.md

@@ -0,0 +1,22 @@
+# EM-DAT Domain Knowledge
+
+EM-TEST validates data related to international disasters.
+
+## 1. Geospatial Consistency
+- **ISO3 Codes**: Country codes should be valid ISO-3166-1 alpha-3 codes.
+- **Geography**: Valid latitude (-90 to 90) and longitude (-180 to 180).
+- **Standards**: The project follows UNSD M49 standards for country and area codes.
+
+## 2. Temporal Logic
+Disaster numbers (`DisNo.`) contain the year the disaster was recorded.
+- **Consistency**: `Start Year` should generally match the year in `DisNo.`.
+- **Duration**: `Start Year/Month/Day` must be before or equal to `End Year/Month/Day`.
+
+## 3. Classification
+Disasters are categorized hierarchically:
+- **Disaster Group** (e.g., Natural)
+- **Disaster Subgroup** (e.g., Meteorological)
+- **Disaster Type** (e.g., Storm)
+- **Disaster Subtype** (e.g., Tropical cyclone)
+
+Validation must ensure that combinations of these fields follow the official `classification_tree.toml`.

instructions/validation.md

@@ -0,0 +1,27 @@
+# Data Validation with Pandera and Pandas
+
+The core of EM-TEST is built on `pandera` and `pandas`.
+
+## 1. Schema Definition
+Schemas are defined in `emtest/validation_schemas.py`.
+- **`DataFrameSchema`**: Use this to define per-column constraints.
+- **Nullability**: In Pandas, the standard `int` type is not nullable. Use `float` for columns that contain integers but may have null values (e.g., `Start Month`, `Start Day`).
+
+## 2. Custom Checks
+Complex validation logic should be implemented in `emtest/custom_checks.py`.
+- **Single-Column Checks**: Implement logic for a specific field (e.g., `check_disno`, `check_month`).
+- **Multi-Column ("Wide") Checks**: Use these for consistency between fields (e.g., `check_start_end_year_consistency`).
+
+## 3. Regular Expression Engineering
+Identifiers are often validated via Regex:
+- **DisNo format**: `^\d{4}-\d{4}-[A-Z]{3}$` (Year-Sequence-CountryISO).
+- **External IDs**: GLIDE numbers and other identifiers rely on specific patterns.
+
+## 4. Testing Pattern
+Changes must be verified with the pytest test suite:
+- `uv run pytest` — runs all tests in `tests/`
+- `uv run pytest tests/test_validation.py::test_date_consistency` — run a single test
+
+The suite tests both individual check functions (`tests/test_custom_checks.py`) and the full schema (`tests/test_validation.py`). The fixture in `tests/conftest.py` provides a valid single-row DataFrame; tests mutate one field at a time and assert the expected `SchemaError`.
+
+The example scripts in `examples/` validate real datasets and can be used for end-to-end verification after schema changes.