Skip to content

Test JSOBenchmarks with SolverBenchmarks: try non-forked branches#309

Open
MaxenceGollier wants to merge 3 commits intojso-bmarksfrom
jso-bmarks-pr
Open

Test JSOBenchmarks with SolverBenchmarks: try non-forked branches#309
MaxenceGollier wants to merge 3 commits intojso-bmarksfrom
jso-bmarks-pr

Conversation

@MaxenceGollier
Copy link
Copy Markdown
Collaborator

@MaxenceGollier MaxenceGollier commented Mar 7, 2026

DO NOT MERGE

@MaxenceGollier MaxenceGollier mentioned this pull request Mar 7, 2026
Closed
@MaxenceGollier MaxenceGollier added the run benchmarks Used to trigger benchmarks in a PR label Mar 7, 2026
@MaxenceGollier MaxenceGollier reopened this Mar 9, 2026
@MaxenceGollier MaxenceGollier reopened this Mar 9, 2026
@MaxenceGollier MaxenceGollier reopened this Mar 9, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 9, 2026

Gist: https://gist.github.com/dpo/7a44ce5a8bd668d39e452de951d037bd

Full results stored as artifacts

Overview

this_commit_vs_reference_R2_precise.svg

this_commit_vs_reference_R2_imprecise.svg

@MaxenceGollier MaxenceGollier reopened this Mar 9, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 9, 2026

Gist: https://gist.github.com/dpo/bfe8576c5f8e93940a57007513aebb0f

Full results stored as artifacts

Overview

this_commit_vs_reference_R2_precise.svg

this_commit_vs_reference_R2_imprecise.svg

@MaxenceGollier
Copy link
Copy Markdown
Collaborator Author

MaxenceGollier commented Mar 9, 2026

@dpo, this is where I am now, please see the gist.

Apart from performance profiles, what else do you want to see in the gist ?
Is it really necessary to upload .jld2 artifacts if we already show all the results in the gist ?

@MaxenceGollier MaxenceGollier mentioned this pull request Mar 9, 2026
Merged
@MaxenceGollier MaxenceGollier force-pushed the jso-bmarks-pr branch 3 times, most recently from f6f6463 to f73c6dc Compare March 9, 2026 20:27
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 9, 2026

Gist: https://gist.github.com/dpo/2b0037224b7b5ef213703ec7071aa14b

Full results stored as artifacts

Overview

this_commit_vs_reference_R2_precise.svg

this_commit_vs_reference_R2_imprecise.svg

@MaxenceGollier MaxenceGollier reopened this Mar 9, 2026
@MaxenceGollier MaxenceGollier reopened this Mar 9, 2026
@MaxenceGollier MaxenceGollier reopened this Mar 9, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 9, 2026

Gist: https://gist.github.com/dpo/fb3dab228a9ce0dc01fe4b364a037064

Full results stored as artifacts

Overview

this_commit_vs_reference_R2_precise.svg

this_commit_vs_reference_R2_imprecise.svg

@MaxenceGollier MaxenceGollier reopened this Mar 9, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 9, 2026

Gist: https://gist.github.com/dpo/84b41bbc32e8cb8718791e742c626deb

Full results stored as artifacts

Overview

[## This commit vs reference: R2_precise

.svg](https://gist.github.com/dpo/84b41bbc32e8cb8718791e742c626deb/raw/## This commit vs reference: R2_precise

.svg?sanitize=true)

[## This commit vs reference: R2_imprecise

.svg](https://gist.github.com/dpo/84b41bbc32e8cb8718791e742c626deb/raw/## This commit vs reference: R2_imprecise

.svg?sanitize=true)

@MaxenceGollier MaxenceGollier reopened this Mar 9, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 12, 2026

Gist: https://gist.github.com/dpo/d7bcb00bb0a1d5496149479252f11cd7

Full results stored as artifacts

Overview

this_commit_vs_reference_R2_precise.svg

this_commit_vs_reference_R2_imprecise.svg

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 12, 2026

Gist: https://gist.github.com/dpo/16da40d3078fb6f212260c12cc96e2ea

Full results stored as artifacts

Overview

this_commit_vs_reference_R2_precise.svg

this_commit_vs_reference_R2_imprecise.svg

@MaxenceGollier MaxenceGollier force-pushed the jso-bmarks-pr branch 2 times, most recently from c85df6c to 5f332b9 Compare March 12, 2026 23:42
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 13, 2026

Gist: https://gist.github.com/dpo/945f48c4159f6f8e7094351fb918f211

Full results stored as artifacts

Overview

this_commit_vs_reference_R2_precise.svg

this_commit_vs_reference_R2_imprecise.svg

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 13, 2026

Gist: https://gist.github.com/dpo/9c2701b731f97c34e95bfb1d00f79a97

Full results stored as artifacts

Overview

this_commit_vs_reference_R2_precise.svg

this_commit_vs_reference_R2_imprecise.svg

@MaxenceGollier MaxenceGollier force-pushed the jso-bmarks-pr branch 3 times, most recently from af71c0c to abfd6cd Compare March 13, 2026 19:38
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 13, 2026

Gist: https://gist.github.com/dpo/999e6af22d5321f9257549f6ccb605c8

Full results stored as artifacts

Overview

this_commit_vs_reference_R2_precise.svg

this_commit_vs_reference_R2_imprecise.svg

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 13, 2026

Gist: https://gist.github.com/dpo/6c9ec1cb1fc3d4f754b047a0ab0f3c56

Full results stored as artifacts

Overview

this_commit_vs_reference_R2_precise.svg

this_commit_vs_reference_R2_imprecise.svg

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 16, 2026

SolverBenchmark Profiles

this_commit_vs_reference_R2_precise.svg

this_commit_vs_reference_R2_imprecise.svg

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 16, 2026

SolverBenchmark Profiles

this_commit_vs_reference_R2_precise.svg

this_commit_vs_reference_R2_imprecise.svg

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 16, 2026

Gist: https://gist.github.com/dpo/b7a23a011aa9ca88229c021da425b11a

Full results stored as artifacts

Overview

this_commit_vs_reference_R2_time.svg

this_commit_vs_reference_R2_memory.svg

this_commit_vs_reference_R2_gctime.svg

this_commit_vs_reference_R2_allocations.svg

profiles_this_commit_vs_reference_R2.svg

Judgement

Benchmark Report for RegularizedOptimization

Job Properties

  • Time of benchmarks:
    • Target: 16 Mar 2026 - 17:06
    • Baseline: 16 Mar 2026 - 17:07
  • Package commits:
  • Julia commits:
    • Target: 5fe89b8
    • Baseline: 5fe89b8
  • Julia command flags:
    • Target: None
    • Baseline: None
  • Environment variables:
    • Target: None
    • Baseline: None

Results

A ratio greater than 1.0 denotes a possible regression (marked with ❌), while a ratio less
than 1.0 denotes a possible improvement (marked with ✅). Brackets display tolerances for the benchmark estimates. Only significant results - results
that indicate possible regressions or improvements - are shown below (thus, an empty table means that all
benchmark results remained invariant between builds).

ID time ratio memory ratio
["R2", "bpdn_B0"] 3.87 (5%) ❌ 2.89 (1%) ❌
["R2", "bpdn_l0"] 0.67 (5%) ✅ 0.99 (1%)
["R2", "bpdn_l1"] 3.17 (5%) ❌ 1.10 (1%) ❌

Benchmark Group List

Here's a list of all the benchmark groups executed by this job:

  • ["R2"]

Julia versioninfo

Target

Julia Version 1.12.5
Commit 5fe89b8ddc1 (2026-02-09 16:05 UTC)
Build Info:
  Official https://julialang.org release
Platform Info:
  OS: macOS (arm64-apple-darwin24.0.0)
  uname: Darwin 24.6.0 Darwin Kernel Version 24.6.0: Mon Jan 19 22:02:01 PST 2026; root:xnu-11417.140.69.708.3~1/RELEASE_ARM64_VMAPPLE arm64 arm
  CPU: Apple M1 (Virtual): 
              speed         user         nice          sys         idle          irq
       #1  2400 MHz        104 s          0 s         77 s         68 s          0 s  
       #2  2400 MHz        109 s          0 s         64 s         75 s          0 s  
       #3  2400 MHz        110 s          0 s         60 s         78 s          0 s  
  Memory: 7.0 GB (171.578125 MB free)
  Uptime: 248.0 sec
  Load Avg:  11.32568359375  24.6806640625  13.5771484375
  WORD_SIZE: 64
  LLVM: libLLVM-18.1.7 (ORCJIT, apple-m1)
  GC: Built with stock GC
Threads: 1 default, 1 interactive, 1 GC (on 3 virtual cores)

Baseline

Julia Version 1.12.5
Commit 5fe89b8ddc1 (2026-02-09 16:05 UTC)
Build Info:
  Official https://julialang.org release
Platform Info:
  OS: macOS (arm64-apple-darwin24.0.0)
  uname: Darwin 24.6.0 Darwin Kernel Version 24.6.0: Mon Jan 19 22:02:01 PST 2026; root:xnu-11417.140.69.708.3~1/RELEASE_ARM64_VMAPPLE arm64 arm
  CPU: Apple M1 (Virtual): 
              speed         user         nice          sys         idle          irq
       #1  2400 MHz        129 s          0 s         85 s         79 s          0 s  
       #2  2400 MHz        136 s          0 s         71 s         86 s          0 s  
       #3  2400 MHz        136 s          0 s         67 s         90 s          0 s  
  Memory: 7.0 GB (224.25 MB free)
  Uptime: 292.0 sec
  Load Avg:  9.35498046875  22.31103515625  13.275390625
  WORD_SIZE: 64
  LLVM: libLLVM-18.1.7 (ORCJIT, apple-m1)
  GC: Built with stock GC
Threads: 1 default, 1 interactive, 1 GC (on 3 virtual cores)

this_commit

Benchmark Report for RegularizedOptimization

Job Properties

  • Time of benchmark: 16 Mar 2026 - 17:06
  • Package commit: ae7d95c
  • Julia commit: 5fe89b8
  • Julia command flags: None
  • Environment variables: None

Results

Below is a table of this job's results, obtained by running the benchmarks.
The values listed in the ID column have the structure [parent_group, child_group, ..., key], and can be used to
index into the BaseBenchmarks suite to retrieve the corresponding benchmarks.
The percentages accompanying time and memory values in the below table are noise tolerances. The "true"
time/memory value for a given benchmark is expected to fall within this percentage of the reported value.
An empty cell means that the value was zero.

ID time GC time memory allocations
["R2", "bpdn_B0"] 11.163 ms (5%) 1.81 MiB (1%) 4319
["R2", "bpdn_l0"] 489.090 ms (5%) 3.27 MiB (1%) 175079
["R2", "bpdn_l1"] 6.395 ms (5%) 369.67 KiB (1%) 2604

Benchmark Group List

Here's a list of all the benchmark groups executed by this job:

  • ["R2"]

Julia versioninfo

Julia Version 1.12.5
Commit 5fe89b8ddc1 (2026-02-09 16:05 UTC)
Build Info:
  Official https://julialang.org release
Platform Info:
  OS: macOS (arm64-apple-darwin24.0.0)
  uname: Darwin 24.6.0 Darwin Kernel Version 24.6.0: Mon Jan 19 22:02:01 PST 2026; root:xnu-11417.140.69.708.3~1/RELEASE_ARM64_VMAPPLE arm64 arm
  CPU: Apple M1 (Virtual): 
              speed         user         nice          sys         idle          irq
       #1  2400 MHz        104 s          0 s         77 s         68 s          0 s  
       #2  2400 MHz        109 s          0 s         64 s         75 s          0 s  
       #3  2400 MHz        110 s          0 s         60 s         78 s          0 s  
  Memory: 7.0 GB (171.578125 MB free)
  Uptime: 248.0 sec
  Load Avg:  11.32568359375  24.6806640625  13.5771484375
  WORD_SIZE: 64
  LLVM: libLLVM-18.1.7 (ORCJIT, apple-m1)
  GC: Built with stock GC
Threads: 1 default, 1 interactive, 1 GC (on 3 virtual cores)

Reference

Benchmark Report for RegularizedOptimization

Job Properties

  • Time of benchmark: 16 Mar 2026 - 17:07
  • Package commit: c5606a2
  • Julia commit: 5fe89b8
  • Julia command flags: None
  • Environment variables: None

Results

Below is a table of this job's results, obtained by running the benchmarks.
The values listed in the ID column have the structure [parent_group, child_group, ..., key], and can be used to
index into the BaseBenchmarks suite to retrieve the corresponding benchmarks.
The percentages accompanying time and memory values in the below table are noise tolerances. The "true"
time/memory value for a given benchmark is expected to fall within this percentage of the reported value.
An empty cell means that the value was zero.

ID time GC time memory allocations
["R2", "bpdn_B0"] 2.884 ms (5%) 642.27 KiB (1%) 946
["R2", "bpdn_l0"] 733.902 ms (5%) 3.29 MiB (1%) 176301
["R2", "bpdn_l1"] 2.020 ms (5%) 336.62 KiB (1%) 666

Benchmark Group List

Here's a list of all the benchmark groups executed by this job:

  • ["R2"]

Julia versioninfo

Julia Version 1.12.5
Commit 5fe89b8ddc1 (2026-02-09 16:05 UTC)
Build Info:
  Official https://julialang.org release
Platform Info:
  OS: macOS (arm64-apple-darwin24.0.0)
  uname: Darwin 24.6.0 Darwin Kernel Version 24.6.0: Mon Jan 19 22:02:01 PST 2026; root:xnu-11417.140.69.708.3~1/RELEASE_ARM64_VMAPPLE arm64 arm
  CPU: Apple M1 (Virtual): 
              speed         user         nice          sys         idle          irq
       #1  2400 MHz        129 s          0 s         85 s         79 s          0 s  
       #2  2400 MHz        136 s          0 s         71 s         86 s          0 s  
       #3  2400 MHz        136 s          0 s         67 s         90 s          0 s  
  Memory: 7.0 GB (224.25 MB free)
  Uptime: 292.0 sec
  Load Avg:  9.35498046875  22.31103515625  13.275390625
  WORD_SIZE: 64
  LLVM: libLLVM-18.1.7 (ORCJIT, apple-m1)
  GC: Built with stock GC
Threads: 1 default, 1 interactive, 1 GC (on 3 virtual cores)
SolverBenchmark Profiles

this_commit_vs_reference_R2_precise.svg

this_commit_vs_reference_R2_imprecise.svg

@MaxenceGollier MaxenceGollier force-pushed the jso-bmarks-pr branch 2 times, most recently from 6b481ac to ac0ae03 Compare March 23, 2026 16:01
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 23, 2026

Gist: https://gist.github.com/dpo/e5f8f8226af5d940466575e96cb9ec4a

Full results stored as artifacts

Overview

this_commit_vs_reference_R2_time.svg

this_commit_vs_reference_R2_memory.svg

this_commit_vs_reference_R2_gctime.svg

this_commit_vs_reference_R2_allocations.svg

profiles_this_commit_vs_reference_R2.svg

Judgement

Benchmark Report for RegularizedOptimization

Job Properties

  • Time of benchmarks:
    • Target: 23 Mar 2026 - 16:03
    • Baseline: 23 Mar 2026 - 16:04
  • Package commits:
  • Julia commits:
    • Target: 5fe89b8
    • Baseline: 5fe89b8
  • Julia command flags:
    • Target: None
    • Baseline: None
  • Environment variables:
    • Target: None
    • Baseline: None

Results

A ratio greater than 1.0 denotes a possible regression (marked with ❌), while a ratio less
than 1.0 denotes a possible improvement (marked with ✅). Brackets display tolerances for the benchmark estimates. Only significant results - results
that indicate possible regressions or improvements - are shown below (thus, an empty table means that all
benchmark results remained invariant between builds).

ID time ratio memory ratio
["R2", "bpdn_B0"] 3.74 (5%) ❌ 2.89 (1%) ❌
["R2", "bpdn_l0"] 0.89 (5%) ✅ 0.99 (1%)
["R2", "bpdn_l1"] 3.16 (5%) ❌ 1.10 (1%) ❌

Benchmark Group List

Here's a list of all the benchmark groups executed by this job:

  • ["R2"]

Julia versioninfo

Target

Julia Version 1.12.5
Commit 5fe89b8ddc1 (2026-02-09 16:05 UTC)
Build Info:
  Official https://julialang.org release
Platform Info:
  OS: macOS (arm64-apple-darwin24.0.0)
  uname: Darwin 24.6.0 Darwin Kernel Version 24.6.0: Mon Jan 19 22:02:01 PST 2026; root:xnu-11417.140.69.708.3~1/RELEASE_ARM64_VMAPPLE arm64 arm
  CPU: Apple M1 (Virtual): 
              speed         user         nice          sys         idle          irq
       #1  2400 MHz         87 s          0 s         41 s        104 s          0 s  
       #2  2400 MHz         85 s          0 s         32 s        115 s          0 s  
       #3  2400 MHz         80 s          0 s         29 s        124 s          0 s  
  Memory: 7.0 GB (67.5 MB free)
  Uptime: 232.0 sec
  Load Avg:  7.3291015625  10.0185546875  5.2978515625
  WORD_SIZE: 64
  LLVM: libLLVM-18.1.7 (ORCJIT, apple-m1)
  GC: Built with stock GC
Threads: 1 default, 1 interactive, 1 GC (on 3 virtual cores)

Baseline

Julia Version 1.12.5
Commit 5fe89b8ddc1 (2026-02-09 16:05 UTC)
Build Info:
  Official https://julialang.org release
Platform Info:
  OS: macOS (arm64-apple-darwin24.0.0)
  uname: Darwin 24.6.0 Darwin Kernel Version 24.6.0: Mon Jan 19 22:02:01 PST 2026; root:xnu-11417.140.69.708.3~1/RELEASE_ARM64_VMAPPLE arm64 arm
  CPU: Apple M1 (Virtual): 
              speed         user         nice          sys         idle          irq
       #1  2400 MHz        102 s          0 s         44 s        118 s          0 s  
       #2  2400 MHz        100 s          0 s         34 s        130 s          0 s  
       #3  2400 MHz         95 s          0 s         31 s        138 s          0 s  
  Memory: 7.0 GB (75.078125 MB free)
  Uptime: 264.0 sec
  Load Avg:  6.2880859375  9.4365234375  5.2734375
  WORD_SIZE: 64
  LLVM: libLLVM-18.1.7 (ORCJIT, apple-m1)
  GC: Built with stock GC
Threads: 1 default, 1 interactive, 1 GC (on 3 virtual cores)

this_commit

Benchmark Report for RegularizedOptimization

Job Properties

  • Time of benchmark: 23 Mar 2026 - 16:03
  • Package commit: 2f5e96b
  • Julia commit: 5fe89b8
  • Julia command flags: None
  • Environment variables: None

Results

Below is a table of this job's results, obtained by running the benchmarks.
The values listed in the ID column have the structure [parent_group, child_group, ..., key], and can be used to
index into the BaseBenchmarks suite to retrieve the corresponding benchmarks.
The percentages accompanying time and memory values in the below table are noise tolerances. The "true"
time/memory value for a given benchmark is expected to fall within this percentage of the reported value.
An empty cell means that the value was zero.

ID time GC time memory allocations
["R2", "bpdn_B0"] 10.263 ms (5%) 1.81 MiB (1%) 4319
["R2", "bpdn_l0"] 405.620 ms (5%) 3.27 MiB (1%) 175079
["R2", "bpdn_l1"] 6.052 ms (5%) 369.67 KiB (1%) 2604

Benchmark Group List

Here's a list of all the benchmark groups executed by this job:

  • ["R2"]

Julia versioninfo

Julia Version 1.12.5
Commit 5fe89b8ddc1 (2026-02-09 16:05 UTC)
Build Info:
  Official https://julialang.org release
Platform Info:
  OS: macOS (arm64-apple-darwin24.0.0)
  uname: Darwin 24.6.0 Darwin Kernel Version 24.6.0: Mon Jan 19 22:02:01 PST 2026; root:xnu-11417.140.69.708.3~1/RELEASE_ARM64_VMAPPLE arm64 arm
  CPU: Apple M1 (Virtual): 
              speed         user         nice          sys         idle          irq
       #1  2400 MHz         87 s          0 s         41 s        104 s          0 s  
       #2  2400 MHz         85 s          0 s         32 s        115 s          0 s  
       #3  2400 MHz         80 s          0 s         29 s        124 s          0 s  
  Memory: 7.0 GB (67.5 MB free)
  Uptime: 232.0 sec
  Load Avg:  7.3291015625  10.0185546875  5.2978515625
  WORD_SIZE: 64
  LLVM: libLLVM-18.1.7 (ORCJIT, apple-m1)
  GC: Built with stock GC
Threads: 1 default, 1 interactive, 1 GC (on 3 virtual cores)

Reference

Benchmark Report for RegularizedOptimization

Job Properties

  • Time of benchmark: 23 Mar 2026 - 16:04
  • Package commit: 60a9a7c
  • Julia commit: 5fe89b8
  • Julia command flags: None
  • Environment variables: None

Results

Below is a table of this job's results, obtained by running the benchmarks.
The values listed in the ID column have the structure [parent_group, child_group, ..., key], and can be used to
index into the BaseBenchmarks suite to retrieve the corresponding benchmarks.
The percentages accompanying time and memory values in the below table are noise tolerances. The "true"
time/memory value for a given benchmark is expected to fall within this percentage of the reported value.
An empty cell means that the value was zero.

ID time GC time memory allocations
["R2", "bpdn_B0"] 2.747 ms (5%) 642.27 KiB (1%) 946
["R2", "bpdn_l0"] 454.664 ms (5%) 3.29 MiB (1%) 176301
["R2", "bpdn_l1"] 1.916 ms (5%) 336.62 KiB (1%) 666

Benchmark Group List

Here's a list of all the benchmark groups executed by this job:

  • ["R2"]

Julia versioninfo

Julia Version 1.12.5
Commit 5fe89b8ddc1 (2026-02-09 16:05 UTC)
Build Info:
  Official https://julialang.org release
Platform Info:
  OS: macOS (arm64-apple-darwin24.0.0)
  uname: Darwin 24.6.0 Darwin Kernel Version 24.6.0: Mon Jan 19 22:02:01 PST 2026; root:xnu-11417.140.69.708.3~1/RELEASE_ARM64_VMAPPLE arm64 arm
  CPU: Apple M1 (Virtual): 
              speed         user         nice          sys         idle          irq
       #1  2400 MHz        102 s          0 s         44 s        118 s          0 s  
       #2  2400 MHz        100 s          0 s         34 s        130 s          0 s  
       #3  2400 MHz         95 s          0 s         31 s        138 s          0 s  
  Memory: 7.0 GB (75.078125 MB free)
  Uptime: 264.0 sec
  Load Avg:  6.2880859375  9.4365234375  5.2734375
  WORD_SIZE: 64
  LLVM: libLLVM-18.1.7 (ORCJIT, apple-m1)
  GC: Built with stock GC
Threads: 1 default, 1 interactive, 1 GC (on 3 virtual cores)
SolverBenchmark Profiles

this_commit_vs_reference_R2DH_precise.svg

this_commit_vs_reference_TR_sr1_default.svg

this_commit_vs_reference_R2N_sr1_default.svg

this_commit_vs_reference_TRDH_precise.svg

this_commit_vs_reference_R2N_bfgs_imprecise.svg

this_commit_vs_reference_R2_imprecise.svg

this_commit_vs_reference_TR_bfgs_default.svg

this_commit_vs_reference_R2N_bfgs_precise.svg

this_commit_vs_reference_TR_sr1_precise.svg

this_commit_vs_reference_R2DH_default.svg

this_commit_vs_reference_R2_default.svg

this_commit_vs_reference_TRDH_default.svg

this_commit_vs_reference_R2DH_imprecise.svg

this_commit_vs_reference_TRDH_imprecise.svg

this_commit_vs_reference_R2N_sr1_imprecise.svg

this_commit_vs_reference_TR_bfgs_imprecise.svg

this_commit_vs_reference_R2N_sr1_precise.svg

this_commit_vs_reference_TR_sr1_imprecise.svg

this_commit_vs_reference_TR_bfgs_precise.svg

this_commit_vs_reference_R2_precise.svg

this_commit_vs_reference_R2N_bfgs_default.svg

@MaxenceGollier MaxenceGollier force-pushed the jso-bmarks-pr branch 3 times, most recently from 54a6dc4 to 1798d3d Compare March 24, 2026 21:52
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 25, 2026

Gist: https://gist.github.com/dpo/2612aaea3a80c28bdc70bee4cafe3a18

Full results stored as artifacts

Overview

this_commit_vs_reference_R2_time.svg

this_commit_vs_reference_R2_memory.svg

this_commit_vs_reference_R2_gctime.svg

this_commit_vs_reference_R2_allocations.svg

profiles_this_commit_vs_reference_R2.svg

Judgement

Benchmark Report for RegularizedOptimization

Job Properties

  • Time of benchmarks:
    • Target: 24 Mar 2026 - 22:34
    • Baseline: 24 Mar 2026 - 22:34
  • Package commits:
  • Julia commits:
    • Target: 5fe89b8
    • Baseline: 5fe89b8
  • Julia command flags:
    • Target: None
    • Baseline: None
  • Environment variables:
    • Target: None
    • Baseline: None

Results

A ratio greater than 1.0 denotes a possible regression (marked with ❌), while a ratio less
than 1.0 denotes a possible improvement (marked with ✅). Brackets display tolerances for the benchmark estimates. Only significant results - results
that indicate possible regressions or improvements - are shown below (thus, an empty table means that all
benchmark results remained invariant between builds).

ID time ratio memory ratio
["R2", "bpdn_B0"] 4.03 (5%) ❌ 2.89 (1%) ❌
["R2", "bpdn_l0"] 1.05 (5%) ❌ 0.99 (1%)
["R2", "bpdn_l1"] 3.35 (5%) ❌ 1.10 (1%) ❌

Benchmark Group List

Here's a list of all the benchmark groups executed by this job:

  • ["R2"]

Julia versioninfo

Target

Julia Version 1.12.5
Commit 5fe89b8ddc1 (2026-02-09 16:05 UTC)
Build Info:
  Official https://julialang.org release
Platform Info:
  OS: macOS (arm64-apple-darwin24.0.0)
  uname: Darwin 24.6.0 Darwin Kernel Version 24.6.0: Mon Jan 19 22:02:01 PST 2026; root:xnu-11417.140.69.708.3~1/RELEASE_ARM64_VMAPPLE arm64 arm
  CPU: Apple M1 (Virtual): 
              speed         user         nice          sys         idle          irq
       #1  2400 MHz        223 s          0 s        105 s        192 s          0 s  
       #2  2400 MHz        223 s          0 s         89 s        207 s          0 s  
       #3  2400 MHz        218 s          0 s         82 s        220 s          0 s  
  Memory: 7.0 GB (119.3125 MB free)
  Uptime: 520.0 sec
  Load Avg:  14.85302734375  13.00146484375  8.37060546875
  WORD_SIZE: 64
  LLVM: libLLVM-18.1.7 (ORCJIT, apple-m1)
  GC: Built with stock GC
Threads: 1 default, 1 interactive, 1 GC (on 3 virtual cores)

Baseline

Julia Version 1.12.5
Commit 5fe89b8ddc1 (2026-02-09 16:05 UTC)
Build Info:
  Official https://julialang.org release
Platform Info:
  OS: macOS (arm64-apple-darwin24.0.0)
  uname: Darwin 24.6.0 Darwin Kernel Version 24.6.0: Mon Jan 19 22:02:01 PST 2026; root:xnu-11417.140.69.708.3~1/RELEASE_ARM64_VMAPPLE arm64 arm
  CPU: Apple M1 (Virtual): 
              speed         user         nice          sys         idle          irq
       #1  2400 MHz        252 s          0 s        113 s        194 s          0 s  
       #2  2400 MHz        253 s          0 s         96 s        209 s          0 s  
       #3  2400 MHz        247 s          0 s         90 s        222 s          0 s  
  Memory: 7.0 GB (111.796875 MB free)
  Uptime: 559.0 sec
  Load Avg:  15.93017578125  13.43505859375  8.7119140625
  WORD_SIZE: 64
  LLVM: libLLVM-18.1.7 (ORCJIT, apple-m1)
  GC: Built with stock GC
Threads: 1 default, 1 interactive, 1 GC (on 3 virtual cores)

this_commit

Benchmark Report for RegularizedOptimization

Job Properties

  • Time of benchmark: 24 Mar 2026 - 22:34
  • Package commit: 663d8c0
  • Julia commit: 5fe89b8
  • Julia command flags: None
  • Environment variables: None

Results

Below is a table of this job's results, obtained by running the benchmarks.
The values listed in the ID column have the structure [parent_group, child_group, ..., key], and can be used to
index into the BaseBenchmarks suite to retrieve the corresponding benchmarks.
The percentages accompanying time and memory values in the below table are noise tolerances. The "true"
time/memory value for a given benchmark is expected to fall within this percentage of the reported value.
An empty cell means that the value was zero.

ID time GC time memory allocations
["R2", "bpdn_B0"] 10.737 ms (5%) 1.81 MiB (1%) 4319
["R2", "bpdn_l0"] 492.965 ms (5%) 3.27 MiB (1%) 175079
["R2", "bpdn_l1"] 6.329 ms (5%) 369.67 KiB (1%) 2604

Benchmark Group List

Here's a list of all the benchmark groups executed by this job:

  • ["R2"]

Julia versioninfo

Julia Version 1.12.5
Commit 5fe89b8ddc1 (2026-02-09 16:05 UTC)
Build Info:
  Official https://julialang.org release
Platform Info:
  OS: macOS (arm64-apple-darwin24.0.0)
  uname: Darwin 24.6.0 Darwin Kernel Version 24.6.0: Mon Jan 19 22:02:01 PST 2026; root:xnu-11417.140.69.708.3~1/RELEASE_ARM64_VMAPPLE arm64 arm
  CPU: Apple M1 (Virtual): 
              speed         user         nice          sys         idle          irq
       #1  2400 MHz        223 s          0 s        105 s        192 s          0 s  
       #2  2400 MHz        223 s          0 s         89 s        207 s          0 s  
       #3  2400 MHz        218 s          0 s         82 s        220 s          0 s  
  Memory: 7.0 GB (119.3125 MB free)
  Uptime: 520.0 sec
  Load Avg:  14.85302734375  13.00146484375  8.37060546875
  WORD_SIZE: 64
  LLVM: libLLVM-18.1.7 (ORCJIT, apple-m1)
  GC: Built with stock GC
Threads: 1 default, 1 interactive, 1 GC (on 3 virtual cores)

Reference

Benchmark Report for RegularizedOptimization

Job Properties

  • Time of benchmark: 24 Mar 2026 - 22:34
  • Package commit: e18466a
  • Julia commit: 5fe89b8
  • Julia command flags: None
  • Environment variables: None

Results

Below is a table of this job's results, obtained by running the benchmarks.
The values listed in the ID column have the structure [parent_group, child_group, ..., key], and can be used to
index into the BaseBenchmarks suite to retrieve the corresponding benchmarks.
The percentages accompanying time and memory values in the below table are noise tolerances. The "true"
time/memory value for a given benchmark is expected to fall within this percentage of the reported value.
An empty cell means that the value was zero.

ID time GC time memory allocations
["R2", "bpdn_B0"] 2.662 ms (5%) 642.27 KiB (1%) 946
["R2", "bpdn_l0"] 468.507 ms (5%) 3.29 MiB (1%) 176301
["R2", "bpdn_l1"] 1.886 ms (5%) 336.62 KiB (1%) 666

Benchmark Group List

Here's a list of all the benchmark groups executed by this job:

  • ["R2"]

Julia versioninfo

Julia Version 1.12.5
Commit 5fe89b8ddc1 (2026-02-09 16:05 UTC)
Build Info:
  Official https://julialang.org release
Platform Info:
  OS: macOS (arm64-apple-darwin24.0.0)
  uname: Darwin 24.6.0 Darwin Kernel Version 24.6.0: Mon Jan 19 22:02:01 PST 2026; root:xnu-11417.140.69.708.3~1/RELEASE_ARM64_VMAPPLE arm64 arm
  CPU: Apple M1 (Virtual): 
              speed         user         nice          sys         idle          irq
       #1  2400 MHz        252 s          0 s        113 s        194 s          0 s  
       #2  2400 MHz        253 s          0 s         96 s        209 s          0 s  
       #3  2400 MHz        247 s          0 s         90 s        222 s          0 s  
  Memory: 7.0 GB (111.796875 MB free)
  Uptime: 559.0 sec
  Load Avg:  15.93017578125  13.43505859375  8.7119140625
  WORD_SIZE: 64
  LLVM: libLLVM-18.1.7 (ORCJIT, apple-m1)
  GC: Built with stock GC
Threads: 1 default, 1 interactive, 1 GC (on 3 virtual cores)
SolverBenchmark Profiles

this_commit_vs_reference_TR_sr1_default.svg

this_commit_vs_reference_R2N_sr1_default.svg

this_commit_vs_reference_R2N_bfgs_imprecise.svg

this_commit_vs_reference_R2_imprecise.svg

this_commit_vs_reference_TR_bfgs_default.svg

this_commit_vs_reference_R2DH_default.svg

this_commit_vs_reference_R2_default.svg

this_commit_vs_reference_TRDH_default.svg

this_commit_vs_reference_R2DH_imprecise.svg

this_commit_vs_reference_TRDH_imprecise.svg

this_commit_vs_reference_R2N_sr1_imprecise.svg

this_commit_vs_reference_TR_bfgs_imprecise.svg

this_commit_vs_reference_TR_sr1_imprecise.svg

this_commit_vs_reference_R2N_bfgs_default.svg

@MaxenceGollier
Copy link
Copy Markdown
Collaborator Author

MaxenceGollier commented Mar 25, 2026 •
edited
Loading

@dpo, I added some problems and support for all NLPModels solvers. Benchmarks take about 3 hours which i think is fine. What are the next steps here ?

Should we keep PkgBenchmark ? It adds almost 0 information and adding a benchmark suite for it would double the implementation work as well as the benchmark time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

run benchmarks Used to trigger benchmarks in a PR

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@MaxenceGollier