Optimization: Third-party package or self-implemented?

[bonh]

Possible third-party packages (order without ranking):

cpp

• OptimLib
• GNU GSL multimin
• Boost minimization
• NLopt (good start, relatively easy to use, does not come with a lot of baggage, and has many different algorithms)
• ROL
• TAO
• IPopt

python

• scipy.optmize

Advantages of self-implementation (are more or less disadvantages of third-party packages and vice versa):

• Tailored cpp algorithms for specific maps might be faster (see LevenbergMarquadtSolver in MonotoneLeastSquares.cpp.
• All usecases will use the same cpp optimization framework (MparT's) and not rely on specific packages for each language.

Disadvantages of self-implementation:

• Needs work and maintenance (is it worth the effort?).

[dannys4]

My two cents are that, if we use an external dependency, it should be tailored towards optimization. I'm not thrilled about the idea of using Boost. That's the only stake I have in this conversation though.

[bonh]

I was going to pull out the LevenbergMarquadtSolver function in MonotoneLeastSquares.cpp to resuse it in other examples. But then I figured that this could also be realized by a low-level optimizer class with LevenbergMarquadtSolver as a method.

[bonh]

My two cents are that, if we use an external dependency, it should be tailored towards optimization. I'm not thrilled about the idea of using Boost.

I wasn't even aware that boost provides something like that...

[bonh]

Right now I am more in favor of self-implementation. Does anyone know some research about maps and optimization?

[mparno]

I’m also in support of self-implementation. I have some trust region and line search methods we can adapt from MUQ.

Just for completeness, here are a couple of other good c++ optimization packages. NLOpt is probably where I’d try to start if we decide to use an external optimization package because it’s relatively easy to use, does not come with a lot of baggage, and has many different algorithms.

• NLopt
• ROL
• TAO
• IPopt

[bonh]

I added them to the list above.

[dannys4]

some more thoughts-- GSL multimin is GPL licensed, which is undesirable for gaining traction with others. For TAO, you'd have to install the entirety of PETSc from my understanding (I could be wrong, but I can't find an individual TAO library), so that goes against my previously shared opinion on having something that's exclusively optimization. Similarly, ROL is a package within the entire Trilinios library. I've used NLopt and IPopt several times in the past in Julia, and I'm happy with them, and the fact that NLopt doesn't require LAPACK or anything makes it nice as a dependency, so I'd probably agree with Matt about starting with NLopt if we sourced our optimization externally.

[bonh]

@dannys4 are you in favor of external package or self-implemented?

[dannys4]

I could go either way. I don't know the ins and outs of where we could gain from using our own transport map-oriented code, but theoretically, it would have to be pretty large to justify the person cost of building out an optimization suite on that reason alone. Also, it would be great to have something that's rigorously tested out of the box for this.

On the other hand, (and @mparno can correct me if I'm wrong), using an external suite would prevent an efficient GPU-based implementation, unless I'm mistaken. Also, using std::vector for NLopt makes me a little uneasy. Further, a lot of these are general black-box nonlinear optimizers, but (again, matt can correct me on this) many of the transport map optimization problems are convex by construction, which could let us really dig deep and get good results either in terms of accuracy or efficiency.

After writing all this out, I think I also might just barely lean toward self-implementation, if only because of the GPU factor, but maybe using NLopt as a guide and a placeholder would be a good idea? i.e. we can spend a few hours now to let it link with NLopt and use that until we have a robust and tested suite of optimization tools, then we can let users pick and choose (admitting that our optimizers might be less desirable on the CPU than a battle-hardened and widely-known suite of tools).

[dannys4]

Consensus- Create framework so we can add self-implementation of optimization later, but currently use an "NLOpt" backend interfacing with this new framework

[dannys4]

Problem with self-implementation- Levenberg-Marquardt (and most optimization methods) use linear algebra, so our options are to include CuSOLVER (and have pretty gnarly stuff) or pull in a new dependency (e.g. MAGMA) that makes it easier. Leaving this problem open for the future