Skip to content

Add verifier tests#762

Draft
NoraHagmeyer wants to merge 4 commits into
mainfrom
add-integration-tests
Draft

Add verifier tests#762
NoraHagmeyer wants to merge 4 commits into
mainfrom
add-integration-tests

Conversation

@NoraHagmeyer

@NoraHagmeyer NoraHagmeyer commented Jun 13, 2026

Copy link
Copy Markdown
Contributor
  • Adds the verifier tests
  • Includes benchmark code via submodule
  • Outputs the log/trace for xfail Tests

This is not ideal. In the future, we would probably like to

  • include each external benchmark repository separately
  • include NPBench the same way
  • unify this over repos (i.e., for the Benchmark repo)
  • Run the verifier nightly once we have enough unit test coverage

@daisytuner

daisytuner Bot commented Jun 13, 2026

Copy link
Copy Markdown

Daisytuner Report - mlir_torch_models (chamomile)

@@                                   Benchmarks                                   @@
=====================================================================================
  Benchmark              Time        ΔTime       Thr         Energy      ΔEnergy     
=====================================================================================
# resnet18_torch         19.89 s     -0.08%      N/A         3939.17 J   -0.00%      
# resnet18_docc_none     17.96 s     -0.10%      N/A         4753.04 J   -0.07%      
# resnet18_docc_sequential17.67 s     +0.48%      N/A         4652.17 J   +0.45%      
# resnet18_docc_openmp   23.84 s     +1.25%      N/A         6998.42 J   +1.27%      
# resnet18_docc_cuda     5.45 s      +1.05%      N/A         1064.24 J   +0.94%      

@NoraHagmeyer NoraHagmeyer force-pushed the add-integration-tests branch from 10259e3 to 3c8f5de Compare June 13, 2026 22:06
@daisytuner

daisytuner Bot commented Jun 13, 2026

Copy link
Copy Markdown

Daisytuner Report - python_npbench (zinnia)

@@                                   Benchmarks                                   @@
=====================================================================================
  Benchmark              Time        ΔTime       Thr         Energy      ΔEnergy     
=====================================================================================
# adi_numpy              1.30 s      -0.27%      N/A         132.71 J    -0.91%      
# adi_omp                2.58 s      -2.99%      N/A         325.66 J    -4.73%      
# adi_cuda               3.64 s      -1.09%      N/A         365.96 J    -1.80%      
# adi_seq_tuning         2.50 s      -1.93%      N/A         236.78 J    -2.64%      
# atax_numpy             2.15 s      -0.05%      N/A         227.30 J    -0.65%      
# atax_omp               3.13 s      +1.05%      N/A         409.73 J    +2.60%      
# atax_cuda              4.27 s      -0.19%      N/A         448.31 J    -0.76%      
# atax_seq_tuning        2.80 s      -2.90%      N/A         287.56 J    -1.38%      
# gemm_numpy             1.20 s      -0.95%      N/A         194.29 J    -1.23%      
# gemm_omp               1.15 s      -0.36%      N/A         168.72 J    -0.58%      
# gemm_cuda              10.71 s     -0.10%      N/A         1043.25 J   -0.69%      
# gemm_seq_tuning        1.15 s      +0.12%      N/A         168.73 J    -0.39%      
# gesummv_numpy          1.74 s      -0.69%      N/A         252.65 J    -1.41%      
# gesummv_omp            2.12 s      -1.01%      N/A         346.36 J    -0.90%      
# gesummv_cuda           5.26 s      -0.90%      N/A         733.31 J    -1.02%      
# gesummv_seq_tuning     4.60 s      -1.44%      N/A         610.18 J    -0.91%      
# gemver_numpy           1.08 s      -0.73%      N/A         168.01 J    -1.56%      
# gemver_omp             937.97 ms   -1.64%      N/A         126.05 J    -3.05%      
# gemver_cuda            2.54 s      +0.57%      N/A         274.09 J    +0.20%      
# gemver_seq_tuning      1.66 s      -2.10%      N/A         145.00 J    -0.67%      
# k2mm_numpy             1.19 s      -0.73%      N/A         197.41 J    -1.24%      
# k2mm_omp               3.58 s      -0.01%      N/A         675.52 J    -0.88%      
# k2mm_cuda              12.75 s     +0.05%      N/A         1238.20 J   -0.58%      
# k2mm_seq_tuning        3.06 s      +2.46%      N/A         406.47 J    -0.51%      
# k3mm_numpy             1.02 s      +0.01%      N/A         183.12 J    -1.08%      
# k3mm_omp               5.58 s      -0.65%      N/A         960.63 J    -0.69%      
# k3mm_cuda              18.40 s     -0.20%      N/A         1774.93 J   -0.91%      
# k3mm_seq_tuning        4.88 s      -1.25%      N/A         697.49 J    -1.09%      
# mvt_numpy              2.42 s      -0.70%      N/A         252.06 J    -1.62%      
# mvt_omp                2.77 s      -0.34%      N/A         294.57 J    -0.85%      
# mvt_cuda               3.44 s      -0.03%      N/A         358.71 J    -0.70%      
# mvt_seq_tuning         2.78 s      -0.22%      N/A         294.37 J    -0.82%      
# symm_numpy             776.01 ms   -0.22%      N/A         81.60 J     -0.79%      
# symm_omp               989.05 ms   -1.13%      N/A         121.17 J    -0.97%      
# symm_seq_tuning        1.79 s      -5.48%      N/A         152.55 J    -3.07%      
# syr2k_numpy            875.99 ms   +0.70%      N/A         91.15 J     +0.11%      
# syr2k_omp              931.98 ms   -2.20%      N/A         106.47 J    -2.24%      
# syr2k_cuda             1.57 s      -1.07%      N/A         168.18 J    -1.35%      
# syr2k_seq_tuning       936.58 ms   -1.14%      N/A         107.16 J    -1.17%      
# syrk_numpy             765.28 ms   +0.10%      N/A         80.52 J     -0.69%      
# syrk_omp               783.28 ms   -0.70%      N/A         92.17 J     -1.07%      
# syrk_cuda              1.43 s      -1.19%      N/A         154.39 J    -1.30%      
# syrk_seq_tuning        773.12 ms   -0.66%      N/A         91.37 J     -0.76%      
# trmm_numpy             870.61 ms   +0.01%      N/A         90.67 J     -0.73%      
# trmm_omp               750.02 ms   -0.14%      N/A         95.67 J     -0.72%      
+ trmm_seq_tuning        1.60 s      -13.35%     N/A         133.61 J    -7.54%      

@NoraHagmeyer NoraHagmeyer marked this pull request as ready for review June 13, 2026 22:21
@NoraHagmeyer NoraHagmeyer marked this pull request as draft June 13, 2026 22:21
@NoraHagmeyer NoraHagmeyer marked this pull request as ready for review June 13, 2026 22:32
@NoraHagmeyer NoraHagmeyer marked this pull request as draft June 13, 2026 22:33
@NoraHagmeyer NoraHagmeyer force-pushed the add-integration-tests branch from 3c8f5de to 6c07256 Compare June 13, 2026 22:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant