Skip to content

test: add usage of jsonschema to validate the testme.json file and get more understandable errors. - #405

Open
svalat wants to merge 2 commits into
idefix-code:developfrom
svalat:feature/testme-json-schema
Open

test: add usage of jsonschema to validate the testme.json file and get more understandable errors.#405
svalat wants to merge 2 commits into
idefix-code:developfrom
svalat:feature/testme-json-schema

Conversation

@svalat

@svalat svalat commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

This PR adds format validation for the testme.json files about the test suite to get more understandable errors in case of invalid format, wrong values....

It is based on jsonschema a standard way to validate JSON / YAML files : https://json-schema.org/.

Example of error : invalid value

--------------- testme.json format error ---------------
Invalid format for : /home/........./idefix/test/MHD/AmbipolarCshock3D/testme.json
'no' is not of type 'boolean'

Failed validating 'type' in schema['properties']['default']['properties']['single']:
    {'type': 'boolean'}

On instance['default']['single']:
    'no'
--------------------------------------------------------

Example of error : wrong property name

--------------- testme.json format error ---------------
Invalid format for : /home/................/idefix/test/MHD/AmbipolarCshock3D/testme.json
Additional properties are not allowed ('tlrance' was unexpected)

Failed validating 'additionalProperties' in schema['properties']['default']:
    {'type': 'object',
     'additionalProperties': False,
     'properties': {'ini': {'type': 'string'},
                    'definitionFile': {'type': 'string'},
                    'dumpname': {'type': 'string'},
                    'tolerance': {'type': 'number'},
                    'nonRegressionTest': {'type': 'boolean'},
                    'standardTest': {'type': 'boolean'},
                    'noplot': {'type': 'boolean'},
                    'reconstruction': {'type': 'integer'},
                    'dec': {'type': 'array', 'items': {'type': 'number'}},
                    'single': {'type': 'boolean'},
                    'vectPot': {'type': 'boolean'},
                    'mpi': {'type': 'boolean'},
                    'restart': {'type': 'boolean'},
                    'callPyFunctionBefore': {'type': 'string'},
                    'callPyFunctionAfter': {'type': 'string'},
                    'check_file_produced': {'type': 'array',
                                            'items': {'type': 'string'}},
                    'restart_no_overwrite': {'type': 'array',
                                             'items': {'type': 'string'}}}}

On instance['default']:
    {'dumpname': 'dump.0001.dmp',
     'noplot': True,
     'single': False,
     'reconstruction': 2,
     'tlrance': 3e-14}
--------------------------------------------------------

@svalat svalat added the enhancement New feature or request label Aug 28, 2026
@svalat
svalat force-pushed the feature/testme-json-schema branch from 80b947e to 21b5358 Compare August 28, 2026 08:59

@glesur glesur left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be possible to validate the json file as a pre-commit action? at the moment, no externally-triggered action can launch _validateNaming, while I believe this could be useful on a systematic basis.

…he error messages of test.py + make it faster to start.
@svalat

svalat commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator Author

Done.
There is now two checks in the pre-commit :

  1. Validation via the jsonschema
  2. Call ./test.py --validate-testme-json which also pass by _validateNaming.

Note: it is triggered only when there are changes in testme.json files.

Note: there is also a change in test.py implementation to make it faster (in precommit call as for execution).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants