collaboration with Apache Arrow org

[ExpandingMan]

I'm opening this issue in case anything should be done to ensure we have collaboration with the rest of the Apache Arrow organization. I'll join the mailing list today and reach out, but I'm not sure what if anything else should be done.

I suspect we'll meet with some skepticism because I didn't spend very much time looking at the C++ arrow code when I wrote this, so ensuring that this package has identical behaviors to those wrapping C++ is potentially difficult.

I think it's pretty safe to say that those of us primarily using Julia consider it very important to have a pure Julia implementation, in my view it would be very unfortunate if we wound up replacing parts of this with a wrapper. It's just much easier and nicer to work on if it's in pure Julia, and one of the goals of the language is to be able to replace C++ in this sort of use case. At the core of the Julia implementation should probably always be a "maximally efficient" AbstractVector for referencing Arrow data.

Otherwise, suggested changes to be more compliant with arrow standards are certainly welcome.

Will post back here if there's any relevant news.

@quinnj @sglyon @davidanthoff @wesm

[wesm]

The next steps would be to collect any Arrow-relevant IP and make a code donation to the Apache Arrow project (like we did with Go http://incubator.apache.org/ip-clearance/arrow-go-library.html and Ruby), then continue development there. Let me know if you want to proceed with that, and I'll be ready to help.

[ExpandingMan]

Thanks for your help!

In general I find all the legalistic tomfoolery horrifying, but I won't let that stop us unless there actually is some problem here.

Would this involve abandoning the MIT license in favor of some form of Apache license? Is the apache license in any sense more restrictive? Any thoughts from the other Julia people on this?

Any feeling for whether the rest of the arrow community will be upset that this is not a wrapper? I'm hoping they might not care if they don't use Julia, but admittedly I won't be particularly happy if I clone this repo a year from now and find that someone completely rewrote it as a wrapper.

I'm also pretty concerned that making this part of the arrow repository will make it a 100 times harder to work on than if it were e.g. in https://github.com/JuliaIO. (Again, feedback is welcome here, I wasn't necessarily planning on doing much more work on this in the near future, so I'm definitely open to input from anyone who feels they may be a potential contributor.)

[ExpandingMan]

Tagging some more Julia people that may have interest in this, apologies if you don't

@ararslan @ihnorton @nalimilan @tanmaykm

(feel free to add others)

[xhochy]

(Arrow developer/PMC, hoping to also answer some questions, haven't used Julia in 5 years, so I rather count as non-knowledgable in that space)

In general I find all the legalistic tomfoolery horrifying, but I won't let that stop us unless there actually is some problem here.

Given that most commits are from you and the other two contributors only contributed small changes, this will be very easy at the current stage. It will involve some "paper work" emails but they should be straight-forward.

Would this involve abandoning the MIT license in favor of some form of Apache license? Is the apache license in any sense more restrictive? Any thoughts from the other Julia people on this?

Yes this would involve changing the license to Apache. The Apache license is much longer in terms of text but normally as accepted as the MIT license. If you are concerned, there are plentyful of short description of what the Apache license is about, read one of them.

Any feeling for whether the rest of the arrow community will be upset that this is not a wrapper?

No, in the Arrow repository we already have distinct implementations in JavaScript, Java, Go, C++ and Rust. They all share no code but we do have integration tests between them to ensure that they are all compatible. That is one of the main advantages of having them all in single repository: To ensure that they all work together eventhough that they don't share anything.

I'm also pretty concerned that making this part of the arrow repository will make it a 100 times harder to work on than if it were e.g. in https://github.com/JuliaIO.

What are the benefits of doing it in JuliaIO?

[wesm]

In general I find all the legalistic tomfoolery horrifying, but I won't let that stop us unless there actually is some problem here.

When you're talking about software redistribution, being careful about IP lineage and third party code makes everyone's lives (especially the lawyers') a lot easier in the long run. The process itself is not onerous despite any appearances.

Would this involve abandoning the MIT license in favor of some form of Apache license? Is the apache license in any sense more restrictive? Any thoughts from the other Julia people on this?

The main difference between Apache 2.0 and MIT license is that the Apache license provides a patent grant to users, which provides additional security / peace of mind for developers who may produce commercial software that depends on a project. MIT and Apache 2.0 are compatible from a code reuse perspective as permissive licenses.

I'm also pretty concerned that making this part of the arrow repository will make it a 100 times harder to work on than if it were e.g. in https://github.com/JuliaIO.

I'm curious what might be perceived as "harder". No one (out of > 150 contributors) has been having a hard time contributing to the project as far as I can tell -- I think that "use pull requests and do not break the build" is a reasonable bar of professionalism for an open source contributor. We also ask that contributors help maintain an intelligible change log for the project and write commit messages / PR descriptions that explain the content of their work.

[ExpandingMan]

Thanks guys. I think that's good enough for me.

I suppose my consternation about the project being harder to work on in the arrow repo just comes from the fact that the Julia community is relatively small, and this is something I've grown pretty comfortable with. No worries.

It would probably be convenient if we could keep a mirror in an Arrow.jl repo somewhere, as that would probably simplify its installation with the Julia package manager, not sure what that would look like but I've seen such things done elsewhere.

Ok, let's give people some time to respond with comments and then move ahead. Thanks again!

[wesm]

It would probably be convenient if we could keep a mirror in an Arrow.jl repo somewhere, as that would probably simplify its installation with the Julia package manager, not sure what that would look like but I've seen such things done elsewhere.

Seems like what we'd want to do after voting on a Julia release is to push the changes and tag the new release in a repo that's connected to the Julia package management system (it seems like GitHub -- or git repositories at least -- and Julia packages are intertwined, is that right?).

[davidanthoff]

I'm not really convinced that moving the code into the main arrow repo is a good idea. Here are some issues I see:

• The julia package manager right now is really built around the notion of one repo per package. Deviating from that will make things a lot more complicated (if it is possible at all). Aside from just the package manager, there are lots and lots of other tools in the julia ecosystem that make the same assumption, and it would be really, really painful to move away from that (things like editors, the whole CI infrastructure etc.)
• The julia package presumably would have to version in sync with all the other arrow implementations. I don't think that is desirable at this point, I think the julia package at this point needs to be able to cut a release on its own schedule.
• I think realistically we probably will have more contributors to this package from regular julia contributors. Those are very familiar with the current setup (one repo per package, everything conforms to the normal julia package patterns), and I think we would make it less likely to get PRs from folks if the code was moved into a different repo with lots of other stuff. There is of course a flip side to this point, namely that we would probably be more likely to get PRs from regular arrow contributors if the code lived in the main arrow repo. In my unscientific, subjective judgment we are more likely to get PRs from julia folks, though, so I think we should make it easy for them.

I do think it would be fantastic to make sure we sync the testing story somehow. But I don't think we need to go to one repo for that, there seem to be lots of options to make sure stuff gets tested in sync while maintaining the current repo structure.

On the license: that to me seems mostly extra work that at least I don't want to deal with ;) If folks want to change it, go ahead... But if we keep the repos distinct, I don't see a need to change the license. There is also again a situation that within the julia ecosystem the MIT license is by far the most used, and it might be easier to just stick with that. But, mostly, I'm agnostic about this point.

[wesm]

The julia package manager right now is really built around the notion of one repo per package. Deviating from that will make things a lot more complicated (if it is possible at all).

I don't see why packaging and development process need to be tightly coupled. You only need to update the package manager when a release actually happens, and this update to the package manager should take < 1 minute to do.

I think the julia package at this point needs to be able to cut a release on its own schedule.

We are cutting separate JavaScript releases already, so I don't think this is an issue

I think we would make it less likely to get PRs from folks if the code was moved into a different repo with lots of other stuff.

This seems like FUD to me. I really question how valuable the contributions are from people who are put off by a "who moved my cheese" type of issue. This is already a pretty difficult project to contribute to on the spectrum of open source projects just by its low-level systems nature (data structures, binary protocols, file formats, serialization, etc.)

One could possibly create an apache/Arrow.jl repo, but this would make integration testing and other things a lot harder; the monorepo brings a lot of benefits when you are dealing with binary interoperability

[nalimilan]

I agree it would make sense to move the Julia package to the same repo as all other implementations. The advantages of that approach may not be striking right now, but I imagine it will make it much easier in the long run to keep all implementations in sync while the Arrow format evolves. It would make sense to have some kind of mirror repo to work with the Julia package manager, though.

[ExpandingMan]

The concept of Julia packages are indeed closely tied to git repositories (but not necessarily github). In principle it's possible to make a package out of any type of repository, but indeed it does greatly simplify things if the repo is in the standard format like you see for Arrow.jl right now.

I feel like this problem is probably solvable with git and github though, isn't it possible to make a mirror of a directory or something? I seem to remember seeing some setups like that but can't find any good examples at the moment.

I think at this point it is worth asking though: why is there such eagerness to have everything in a single repository? In my experience this actually makes things more difficult, not easier. I don't understand the thinking behind why it is so important to have so many languages in that one repo. We work on things in separate repos all the time and it's never an issue in the slightest, even when it involves IO and binary stuff.

Regardless, I'm willing to donate it to that repo as long as we have some sort of "Arrow.jl", however that would work.

[davidanthoff]

I don't see why packaging and development process need to be tightly coupled.

Well, it is on julia, for all practical purposes.

You only need to update the package manager when a release actually happens

That is not how it works on julia. The whole workflow of downloading a package, working on a dev version of the package etc. is tightly integrated with the package manager. We also would no longer have access to most of the tool integration we have if the code moved into the arrow repo (things like attobot, femtocleaner, integrated testing & build in VS Code, Documenter.jl, Coverage.jl and probably more stuff).

We are cutting separate JavaScript releases already, so I don't think this is an issue

Ah, so you are using tags that include things like apache-arrow-js in the name, right? The tooling on the julia side expects tags to be of the form v0.1.0, so that is another thing we would have to manually sort out...

This seems like FUD to me. I really question how valuable the contributions are from people who are put off by a "who moved my cheese" type of issue. This is already a pretty difficult project to contribute to on the spectrum of open source projects just by its low-level systems nature (data structures, binary protocols, file formats, serialization, etc.)

I think in reality we would constantly get PRs against the repo that the package manger knows about because that is the workflow that every julia documentation recommends. I think that would be cumbersome. I also think it is quite vital to make it easy to open PRs even for contributors that are not low level super devs. There are lots of little things in the Arrow.jl codebase where even less experienced devs can help, and I would not want to lose them. A good example is the one commit I contributed to this repo. I think pretty much anyone could have done that, certainly no low level coding experience required.

One could possibly create an apache/Arrow.jl repo, but this would make integration testing and other things a lot harder; the monorepo brings a lot of benefits when you are dealing with binary interoperability

I think a productive conversation at this point would be what kind of testing we would like to see. Once we have figured that out, we can see whether that would be easier with two repos or a monorepo.

I don't really know what type of integration testing there is right now in the arrow repo between the different implementations, I think it would be really helpful to understand that better.

[wesm]

I don't understand the thinking behind why it is so important to have so many languages in that one repo.

This project is held together by its inter-language binary integration tests. A single pull request may affect multiple implementations -- if we split the project up into multiple git repos, effectively we would have a network of circular dependencies in the CI. So if you needed to make a change that affected both Java and C++, you need a way to integration tests two PRs jointly. If you merge a breaking change in one repo, all the builds in the other repo break. By working in a monorepo, we maintain harmonious CI builds across all the subprojects.

As an example, the JavaScript developers recently merged support for emitting binary streams from JS to consume in other implementations: apache/arrow@fc7a382. In this patch, the CI verifies that the JavaScript data emitted can be consumed by Java and C++. As time goes on, the matrix of implementations producing and consuming data will grow larger. It's much easier to stay in sync this way

[ExpandingMan]

This project is held together by its inter-language binary integration tests. A single pull request may affect multiple implementations -- if we split the project up into multiple git repos, effectively we would have a network of circular dependencies in the CI.

That sounds a little weird to me, as projects depend critically on other projects in different repos all the time, I don't quite see why this should be a special case. My understanding was that proper versioning was supposed to take care of this sort of thing. Right now there are (more than one) Julia Feather repos that depend on this package, if I break one of them I consider myself to have screwed up, just like if a numpy tag breaks pandas those guys would have screwed up. (Though @wesm undoubtedly has about a billion times more experience than I do with this sort of thing, so perhaps I don't know what I'm talking about.)

Anyway, I really don't want to get us all into an argument here about the relative merits of how this gets set up, my interest is just in making sure this gets maintained and letting people know that they can integrate their Arrow stuff with Julia. I think having some sort of mirror repo or something might solve that. I'll ask around on Julia discourse about whether anyone has done something like this. I'm imagining an "arrow/Arrow.jl" that somehow links to the appropriate part of the arrow repo.

[wesm]

The relationship between the different Arrow implementations is different from a normal package dependency. We have made deliberate changes to the binary format over the last 2.5 years, and it's a lot of work to keep everything in sync -- it sounds like the burden will be on the Julia-Arrow developers to create bespoke tooling to assist with integration testing against the other implementations. There will likely be cases where the Julia implementation will "slip" and become incompatible with the other implementations, as it will not be possible in all cases for CI to force the incompatibility to be resolved with each patch contributed to the project. We could set up some nightly builds to at least complain to the mailing list if something is broken in any 24 hour time span.

This is to say, all of this discussion is moot until the Julia implementation is able to consume and produce valid Arrow binary messages, and validate them against a source of truth (i.e. the JSON integration test format developed).