Summary
Evaluate integrating Airspeed Velocity (ASV) to provide fine-grained performance benchmarking for MC/DC.
While mcdc-vvp is the primary repository for verification, validation, and larger-scale performance studies, ASV could complement it by tracking the performance of core transport routines and other stable components over time. This would help detect performance regressions during routine development.
Motivation
Currently, performance evaluation is primarily conducted through mcdc-vvp, which focuses on end-to-end benchmark problems. These benchmarks are valuable for measuring overall application performance but are less suited for identifying regressions in individual algorithms or kernels.
ASV is designed for micro- and meso-scale benchmarks that can be executed automatically across commits, making it useful for continuous performance tracking.
Potential scope
- Identify stable functions or interfaces suitable for benchmarking.
- Develop representative ASV benchmark cases for these components.
- Integrate ASV into the development workflow (optionally via GitHub Actions).
- Document the benchmarking strategy and guidelines.
Challenges
The primary challenge is selecting benchmarks that remain meaningful as the codebase evolves. Benchmarks should target interfaces that are expected to remain relatively stable, minimizing maintenance as internal implementations change.
Summary
Evaluate integrating Airspeed Velocity (ASV) to provide fine-grained performance benchmarking for MC/DC.
While
mcdc-vvpis the primary repository for verification, validation, and larger-scale performance studies, ASV could complement it by tracking the performance of core transport routines and other stable components over time. This would help detect performance regressions during routine development.Motivation
Currently, performance evaluation is primarily conducted through
mcdc-vvp, which focuses on end-to-end benchmark problems. These benchmarks are valuable for measuring overall application performance but are less suited for identifying regressions in individual algorithms or kernels.ASV is designed for micro- and meso-scale benchmarks that can be executed automatically across commits, making it useful for continuous performance tracking.
Potential scope
Challenges
The primary challenge is selecting benchmarks that remain meaningful as the codebase evolves. Benchmarks should target interfaces that are expected to remain relatively stable, minimizing maintenance as internal implementations change.