Multinomial Logistic regression

[dsweber2]

I'm needing to run logistic regression, ideally with the ability to be multinomial. I noticed that you ( @simonster ) also built a wrapper for GLMnet; have you had any plans on adding this feature to either? If not, how difficult an extension do you think it would be? I'm willing to contribute, but not entirely sure where to start.

[youngjaewoo]

@dsweber2 I'm also in need of multinomial methods. I explored GLMnet.jl package, but was unsuccessful. Did you figure out how to implement or resolved your usage?

[dsweber2]

I have not yet, other projects took precedence. I may reexamine the issue sometime this summer.

-- cheers, David Weber

-----Original Message----- Re: [simonster/Lasso.jl] Multinomial Logistic regression (#8) From: youngjaewoo <notifications@github.com> To: simonster/Lasso.jl <Lasso.jl@noreply.github.com> Cc: David Weber <david.weber2@gmail.com>,Mention <mention@noreply.github.com

Monday, May 15, 2017 at 08:59 @dsweber2 <https://github.com/dsweber2> I'm also in need of multinomial methods. I explored GLMnet.jl package, but was unsuccessful. Did you figure out how to implement or resolved your usage? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGxFYTvNAZtxmvLt-JI0iJPlYpK1H6nwks5r6HZIgaJpZM4KqvD8> .

[sourish-cmi]

I also need multinomial logistic regression with the lasso. Do you have a developer document? I want to develop. Please let me know if it is possible !

[AsafManela]

There is currently no dev doc, but feel free to ask question and I can try to answer. In any case, the contribution is welcome.

[anurag-mds]

Hi @AsafManela, I’m interested in working on this.

I’ve been looking through the current Lasso.jl code and the coordinate descent implementation, and I’m trying to understand what the cleanest path would be for adding multinomial logistic regression without forcing awkward abstractions. I’ve also been reading the Friedman et al. (2010) GLMnet paper to get a sense of how the multinomial case is handled there.

Before starting, I wanted to sanity-check one design point: do you expect this to extend the existing LassoPath machinery, or would it make more sense to treat multinomial as a separate path type with matrix-valued coefficients and a different update loop? From a quick scan, the per-coordinate updates and loss evaluation seem like the parts that would need the most restructuring.

If you have any guidance on which parts of the codebase are most fragile or which constraints you’d like preserved, that would help me avoid going in the wrong direction. Otherwise, I’ll start by prototyping a minimal multinomial path and see how it fits.

Thanks.

[AsafManela]

I think LassoPath is generic enough to support it, unless you hit a major roadblock that I don't foresee. The coordinate_descent machinery would have to be changed of course, but in keeping with good julia practice, you may be able to simply overload some functions for matrix / mulitinomial types.

Thanks for taking a stab at this!

[anurag-mds]

That said, I'll proceed assuming LassoPath remains the core abstraction and prototype a minimal multinomial extension on top of it.

My plan is to:

1. First introduce a matrix-valued coefficient type for the multinomial case while reusing the existing lambda machinery.
2. Then, I will overload the loss, gradient and coordinate update steps for multinomial logistic loss following Friedman et al. 2010
3. Finally I will keep the public API unchanged where possible and isolate multinomial-specific logic behind dispatch so existing behaviour is unaffected.

So, I'll start with a minimal working path with no optimizations then will iterate once the structure is valid
I'll open a draft PR early so we can discuss design choices before things harden.

Let me know if there are any invariants or performance constraints you would like preserved. Otherwise, I'll optimize for clarity and correctness first.

Thanks for the guidance