Develop dataset level validations and review tools

[bdeadman]

Overview

Currently, schema validations in ORD focus on the reaction level—checking how the schema is used for individual reactions. However, when reviewing submitted datasets, additional checks need to be performed at the dataset level to ensure data quality and consistency across the entire submission.

Goals

Develop a suite of dataset-level validation and review tools to:

1. Automate dataset checking and reporting

   • Generate comprehensive dataset checking reports highlighting potential issues, inconsistencies, and quality metrics
   • Create automated visualizations of dataset contents and structure
   • Identify common errors and data quality problems across the dataset

2. Streamline the dataset submission workflow

   • Integrate validation tools into the submission process to catch issues early
   • Reduce manual review burden by automating routine checks
   • Provide reviewers with clear, actionable reports to expedite peer review
   • Note: Human peer review remains the final authority and takes precedence

3. Support data preparation and quality incentives

   • Provide experimenters with validation and visualization tools they can use locally
   • Enable researchers to check their data for common errors before submission
   • Create a quality incentive structure that encourages researchers to prepare data to ORD standards

Key Outputs and Visualizations

Compound Analysis

• Compound Inventory: Generate a list of all unique compounds in the dataset with:
  • Compound name/identifier
  • Assigned roles (reactant, product, solvent, catalyst, etc.)
  • All chemical identifiers (SMILES, InChI, CAS, etc.)
  • RDKit-generated visualizations of compound structures from SMILES
  • Detection of SMILES errors through structure rendering
  • This list is designed to be human readable to facilitate peer review of the compounds
• Identifier Validation: Cross-check identifiers against PubChem where applicable (e.g., validate that provided name and SMILES correspond to the same compound)

Data Distribution Analysis

• Numeric Field Charts: Histograms for continuous variables (yield, temperature, time, etc.) to identify outliers and data distribution patterns
• Categorical Field Charts: Bar charts for categorical data to visualize distribution and identify unusual categories

Reaction Pattern Analysis

• Reaction Template Extraction: Identify and extract repeating reaction templates/patterns
• Template Sampling: Provide representative samples of each distinct reaction type for detailed inspection

Validation Checks

The tool should flag the following issues during review:

1. Duplicate Detection: Identify suspicious duplicate reactions that may be data entry errors or unintended copies

2. Reaction Identifier Consistency: Validate that reaction identifiers are either:

   • Unique identifiers for each reaction, OR
   • Categorical groupings (e.g., batches, experiment series)
   • Flag cases where an identifier appears to be both unique and used for grouping (inconsistent usage)

3. Transcription Error Detection: Flag variable field values that appear anomalous relative to the dataset (e.g., an outlier temperature in a series of constant-temperature reactions)

4. Blank Reaction Handling: Check consistency in how blank/control reactions are recorded (e.g., all blanks should follow the same null/empty convention)

5. Message Component Ordering: Validate that components within messages follow consistent ordering conventions (e.g., for a mixed solution/neat base input, base always comes before solvent, ensuring predictable data access patterns like input['KEY'].component[0])

Scope

This effort should include:

• Dataset-level validation rules and checks (beyond individual reaction validation)
• Report generation tools for dataset submissions
• Data visualization tools for dataset exploration and quality assessment
• Integration with the dataset submission workflow
• Tools suitable for both internal review and external use by experimenters

Implementation Considerations

• Tools should be designed as modular, reusable components where appropriate
• Should complement (not replace) human peer review in the submission workflow
• Design with accessibility in mind for experimenters preparing datasets locally
• Consider both automated checks and templated reporting for common issues
• Leverage RDKit for chemical structure visualization and validation
• Consider PubChem integration for identifier cross-validation

Related

• Current reaction-level validations in the schema
• Dataset submission workflow and review process