Summary
Frictionless data validation breaks on OEPS data packages because of foreign key relationships to shapefiles. As a workaround, we use --skip-validation when creating packages (e.g. in the create data packages workflow).
Current behavior
- Data packages define foreign keys from CSV resources to shapefile geometry resources
- Frictionless validation fails when validating these relationships
flask create-data-package is run with --skip-validation to bypass this- For DSuite2023, we also use
--skip-foreign-keys to avoid the problem entirely (at the cost of losing FK metadata)
Desired outcome
Either:
- Fix the package structure – Define foreign keys or references in a way that Frictionless can validate, or
- Address shapefile support – Investigate Frictionless support for shapefiles in foreign keys and adjust our approach accordingly
The goal is to remove the need for --skip-validation and, if possible, use --skip-foreign-keys only where truly needed, so packages can be validated before upload.
References
flask create-data-package command: backend/oeps/commands/create_data_package.py- Data package creation:
backend/oeps/clients/frictionless.py
- Related workflow: create_data_packages.yml
- Upstream: Frictionless Data specifications
Summary
Frictionless data validation breaks on OEPS data packages because of foreign key relationships to shapefiles. As a workaround, we use
--skip-validationwhen creating packages (e.g. in the create data packages workflow).Current behavior
flask create-data-packageis run with--skip-validationto bypass this--skip-foreign-keysto avoid the problem entirely (at the cost of losing FK metadata)Desired outcome
Either:
The goal is to remove the need for
--skip-validationand, if possible, use--skip-foreign-keysonly where truly needed, so packages can be validated before upload.References
flask create-data-packagecommand:backend/oeps/commands/create_data_package.pybackend/oeps/clients/frictionless.py