JOSS Review: Functionality (paper example)

[pierre-borie]

• Functionality: the example given in the paper did not work for me with the packages listed. In particular, it required to install
  • NLPModelsModifiers.jl to define the LSR1Model
  • ShiftedProximalOperators.jl to define the RootNormLhalf regularizer

I tried running the example with Julia 1.12.2, in case the example works on a earlier version.

Another minor question / remark: is the package planed to be registered? Installation via the github repostory is not an issue per se but most of the JSO packages seem to be on JuliaRegisteries.

openjournals/joss-reviews#9344

[rafaelbailo]

Adding to @pierre-borie's comment, I find that I cannot run the example in the JOSS paper. The line

julia> model, nls_model, _ = RegularizedProblems.svm_train_model() # Build SVM model

errors:

ERROR: MethodError: objects of type Module are not callable
Stacktrace:
 [1] tan_data_train()
   @ RegularizedProblems ~/.julia/packages/RegularizedProblems/nbg07/src/nonlin_svm_model.jl:5
 [2] svm_train_model()
   @ RegularizedProblems ~/.julia/packages/RegularizedProblems/nbg07/src/nonlin_svm_model.jl:101
 [3] top-level scope
   @ REPL[4]:1

[MohamedLaghdafHABIBOULLAH]

Hello @rafaelbailo @pierre-borie, thanks for your review. We have registered the package, and we would like to emphasize that all benchmarks and examples were run using Julia 1.11.7.

Here are the steps to run and reproduce the experiments on macOS using Julia 1.11.7.

Running the experiments (branch paper) — RegularizedOptimization.jl

1) Clone the repository on the paper branch

git clone --branch paper --single-branch https://github.com/JuliaSmoothOptimizers/RegularizedOptimization.jl.git
cd RegularizedOptimization.jl

Alternative (if you already cloned the repository):

git fetch origin
git checkout paper

---

2) Start/Install Julia-1.11.7 and activate the examples environment

From the RegularizedOptimization.jl folder, start Julia:

juliaup add 1.11.7
julia +1.11.7

Then in the Julia REPL:

cd("paper/examples/")
using Pkg
Pkg.activate(".")
Pkg.instantiate()

---

3) Run the example

Still in the Julia REPL:

include("example.jl")

With MacOs you should be able to recover the example of the paper

[ Info:  outer  inner     f(x)     h(x)  √(ξ1/ν)        ρ        σ      ‖x‖      ‖s‖      ‖B‖ R2N 
[ Info:      0      1  7.6e+03  7.8e+02  1.5e+02  4.5e-01  7.4e-04  2.8e+01  1.5e+02  1.0e+00 =
...
[ Info:    179    200  1.9e+02  2.0e+02  2.1e-02 -8.6e-01  6.3e-05  1.0e+03  2.3e+02  1.5e-01 ↗
[ Info:    180    200  1.9e+02  2.0e+02  2.1e-02 -8.6e-01  1.9e-04  1.0e+03  1.9e+02  1.5e-01 ↗
[ Info:    181    200  1.9e+02  2.0e+02  2.1e-02 -6.8e-01  5.6e-04  1.0e+03  8.1e+01  1.5e-01 ↗
[ Info:    182    200  1.9e+02  2.0e+02  2.1e-02  4.9e-01  1.7e-03  1.0e+03  1.2e+01  1.5e-01 =
[ Info:    183     24  1.9e+02  2.1e+02  1.9e-02  1.1e+00  1.7e-03  1.0e+03  2.6e+00  1.9e-01 ↘
[ Info:    184      0  1.9e+02  2.1e+02  7.5e-03  1.1e+00  5.6e-04  1.0e+03  2.6e+00  1.3e-01                
[ Info: R2N: terminating with √(ξ1/ν) = 0.0074767797420856276
"Execution stats: first-order stationary"

4) Run Benchmarks

julia> include("Benchmark.jl")
main (generic function with 1 method)

julia> main()

=== SVM: solver comparison ===

→ TR
  status          = first_order
  time (s)        = 4.3677
  # f eval        = 347
  # ∇f eval       = 291
  # prox calls    = 4037
  final objective = 179.837

→ R2N
  status          = first_order
  time (s)        = 1.8588
  # f eval        = 185
  # ∇f eval       = 101
  # prox calls    = 27932
  final objective = 192.4927

→ LM
  status          = first_order
  time (s)        = 16.9891
  # f eval        = 6
  # ∇f eval       = 2876
  # prox calls    = 1001
  final objective = 201.1862

→ LMTR
  status          = first_order
  time (s)        = 10.8845
  # f eval        = 11
  # ∇f eval       = 1614
  # prox calls    = 432
  final objective = 188.2737

=== Full Benchmark Table ===
(name = "TR", status = "first_order", time = 4.3677, fe = 347, ge = 291, prox = 4037, obj = 179.837)
(name = "R2N", status = "first_order", time = 1.8588, fe = 185, ge = 101, prox = 27932, obj = 192.4927)
(name = "LM", status = "first_order", time = 16.9891, fe = 6, ge = 2876, prox = 1001, obj = 201.1862)
(name = "LMTR", status = "first_order", time = 10.8845, fe = 11, ge = 1614, prox = 432, obj = 188.2737)

You can check the behavior across different operating systems here:
https://github.com/MaxenceGollier/RegularizedOptimization.jl/actions/runs/21079257881/job/60628720557?pr=3

As shown in the logs, the results vary across operating systems and Julia versions. This kind of variability is expected in numerical experiments (differences in floating-point behavior and underlying linear algebra libraries), and it is not specific to Julia.
Note: On macOS with Julia 1.12, we observed a specific issue, a fix is proposed in this PR.

[pierre-borie]

@MohamedLaghdafHABIBOULLAH thanks for your feedback and the detailed tutorial. I followed the steps and managed to reproduce the results shown in the paper. The submission was made before the release of Julia 1.12 so I understand you could not predict how the paper example would be impacted. What is important in the long term is that the examples in the documentation work, which they do, so this is fine for me.

I will wait for @rafaelbailo feedback before closing the issue.

[rafaelbailo]

Agreed with @pierre-borie , we can close the issue.