E2E tests for ONNX and RVC4 NNArchive metric consistency with LuxonisTrain

[dtronmans]

Purpose

For the semantic segmentation, instance segmentation, keypoint detection and object detection tasks, run an NNArchive to evaluate the metric scores and compare to baseline checkpoint metrics. Tests fail if the difference for any metric between NNArchive and checkpoint is > 2%.

Some notes:

• Running RVC4 tests: can be done locally through pytest tests/test_rvc4_nnarchive_regression.py -q --device-ip <ip-or-mxid> or through the testbed: pytest tests/test_rvc4_nnarchive_regression.py -q --testbed-name <testbed-name>
• branched off from Parser, metrics and pre-processing consistency #17
• Started implementing E2E tests instead of unit tests because we have a current state that is consistent across metrics.
• Tests pass locally
• Metric difference threshold for passing tests is 2% now but this can be tightened, this PR is mostly about the test structure itself.
• _get_dataset_identifier: this addition was maybe needed because my LuxonisML was installed on the latest version (0.9.0) and the identifier for the dataset name changed (will check this myself).
• RegressionCase as a dataclass with only a str looks redundant but this might be needed for the future when extra parameters get attached to each test case, like with the parse-export equivalence tests in LuxonisML.
• NormalizedTestData is on Google Cloud buckets, and for each task contains:
  • depthai_eval.yaml
  • onnx_eval.yaml
  • metrics_summary.json
  • onnx_model.onnx.tar.xz
  • rvc4_model.rvc4.tar.xz
  • training_config.yaml

Specification

None / not applicable

Dependencies & Potential Impact

None / not applicable

Deployment Plan

None / not applicable

Testing & Validation

None / not applicable

AI Usage

Assisted-by: AGENT_NAME:MODEL_VERSION [TOOL1] [TOOL2]

Submitted code was reviewed by a human: YES/NO

The author is taking the responsibility for the contribution: YES/NO

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 9ab9cf78-7f18-4c59-83ad-17801300b7bc

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/e2e-tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}