Automated Testing
Testing, in particular automated testing, is a cornerstone of effective community modeling, and is discussed in many places. For instance, a three-level test approach including unit, integration and end-to-end (E2E) testing is presented at the Understanding The Test Pyramid by Software Tester web site or at The Practical Test Pyramid web site. A three-level system with slightly different nomenclature can be found at the BSSw web site. Other descriptions of testing practices identify more levels of testing including, for instance, system and regression tests (e.g., Kim et al. 2021, page 37), and address testing in the context of a continuous learning process.
WW4 will use a four-level software testing strategy, starting with the smallest components and expanding to the entire system.
-
Level 1: Unit Testing
- This is the foundational level of testing, focusing on the smallest testable pieces of code, typically individual functions, in isolation. These tests verify that each "unit" behaves as expected, ensuring the basic building blocks are correct before they are combined.
-
Level 2: Integration Testing
- Individual units are combined into larger groups or modules to test how they interact. The goal is to uncover faults in the "interfaces" between components—checking that data is passed correctly and that modules work together as intended.
-
Level 3: Functional Testing
- This level treats the software as a "black box," ignoring the internal code structure. It focuses on testing the application against expected results. Examples of such tests are cases with refraction or shoaling only, or source term integration in time- or fetch-limited conditions.
-
Level 4: Regression Testing
- This is a validation process focused on the full model application. After major updates -- such as adding new model physics, changing data inputs, or refining algorithms – a comprehensive suite of existing applications is re-run. The goal is to ensure these new changes haven't negatively impacted the model's established, validated behaviors. This goes beyond checking the basic software stack or idealized cases; it verifies that the complete, integrated model still performs correctly and that a fix in one component hasn't inadvertently broken established functionality elsewhere.
Generation of Unit and Integration tests can be automated up to a level by using Agentic AI for code development (e.g. Jules from Google, using the ctest environment adopted by WW4. Once Unit (L1 )and Integration (L2) tests are present, they are automatically integrated in the GitHub workflow through tests.yaml in the .github/workflow directory.
Functional (L3) and Regression (L4) tests have not yet been added to WW4.
Unit and Integration tests for C++ code are gathered in the tests directory of the repository. This directory includes a README file discussing how tests can be run interactively, and how they have been automated.
Kim, G, J. Humble, P. Dubois, and J. Willis, 2021: The Devops Handbook: How to create world-class agility, reliability, & security in technological organizations, Second Edition. IT Revolution, Portland Oregon, 483 pp.
