Documentation: best way to access numpy arrays from Java without copying

[kno10]

Sorry if this is a FAQ, but I could not find this in the documentation, hence a request to improve the documentation.

Does JArray.of(numpy_array) map the data, or copy? The documentation is not clear, but I assume it copies.

The documentation states:

NumPy arrays integrate seamlessly with JPype and allow fast, memory-efficient data transfers to Java. For example, a NumPy array can be mapped directly to a Java primitive array, enabling high-speed operations without unnecessary copying.

but there is no code example given that maps the data. Maybe this refers to the JArray.of function, but from the documentation this seems to return, e.g., double[][]; which given Java's memory layout is likely only possible with copying.
(well, if the data is row-major and has the right endianess, you could point java primitive arrays to the backing buffer, but this sounds horribly fragile with GC to not cause invalid frees?)

So assume we have some large data in the Python ecosystem, which often means in an existing numpy array.

I'd like to access this data from Java without copying (because of memory). It's fine if I need an adapter on the Java side, my code hopefully has the typing for that; even if that means I may need to implement slicing in Java. Say, I am fine with supporting dense and contiguous ndarrays only, so this will basically just boil down to offset computations into a direct buffer and potentially endianness conversions, I assume. (I understand that we will not get Java arrays from it, I am talking about buffers.)

How can we do this?

The documentation I found https://jpype.readthedocs.io/en/latest/userguide.html#buffer-backed-numpy-arrays
always creates a new buffer. Is there a way to expose an existing numpy buffer to Java without copying?

Can we simply convert the .data (and other fields of the ndarray)?

jb = jpype.nio.convertToDirectBuffer(numpyarray.data)

– I could not find instructions on how to zero-copy map an existing numpy array to Java; but I'd imagine that a lot of users will be looking for this.

Thank you.

[marscher]

Hi Erich,
so are you asking how to re-implement a Numpy like memory access in Java for contiguous Python Numpy arrays? For 1d arrays you can directly map the data with zero copy. For multiple dimensions you have to do the index math.

The approach is to use direct buffers to wrap the np.data attribute (similar to what is shown in the documentation). I agree, that we probably should extend the docs with a Numpy buffer example as it is a very common use-case. I guess we also have such codes already it in our test suite.

We are happy to accept a PR showing such an example.

[kno10]

It was not clear to me from the documentation whether zero-copy works for 1d contiguous, or not. So judging from your reply, jb = jpype.nio.convertToDirectBuffer(numpyarray.data) should give me a zero-copy access from Java to the backing array, right?
I'm not scared of doing the index math myself, we already have own code for reading .npy files.

[marscher]

A 1d contiguous array is just a linear buffer, right?

[kno10]

A 1d contiguous array is just a linear buffer, right?

linear depends on the indexing. If I am not mistaken, numpy 2d arrays are also stored in a contiguous backing array, but the stride controls the 2d indexing. But if you have a slice or a masked array, it may no longer have contiguous backing storage.

For example,

a=numpy.array([[0,1],[2,3]])
print(a[0,:].data.contiguous, a[:,0].data.contiguous)

the row is contiguous, the column is not, if the input matrix was in C order. (We probably can still handle this particular case easily with strides on the backing array. I'm not sure when numpy starts copying the data and loses write-through.)