defer_cube: Investigate replacing cube saving with data payload saving

[arjclark]

✨ Feature Request

In the cube.defer_cube() routine we save cubes to disk and then load them back in so that the data payloads are lazy. We should only save the data and load that back into the cube.

Requirements

No significant changes in resource overheads for swapping from saving/loading cubes to cube.data payload saving either via the dask or numpy method as appropriate.

Context

Iris does not guarantee round tripping of save > load loops - items can change/get reprocessed on load meaning what was saved is not necessarily what you get back. In the decomposition this is unhelpful as what you get out the end should be the same as what you would get if you hadn't used it at all. In practice, given cube.defer_cube() is used in the decomposition with splitting up and recombining cubes, we should replace the iris cube operations with ones that only work with the data. In practice this should be a minor change in terms of number of lines changed. The change should be tested in anger by one of the workflow owners to make sure there are no unexpected resource changes.

[arjclark]

Mental note for the ticket: it may be that with swapping the data out rather than constructing a whole new cube that the Python garbage collection keeps the old realised data around for longer than necessary. If memory footprints look off with the implementation then we should explicitly kick off the python garbage collector as part of the defer_cube routine.

[arjclark]

Additional - if the data is already lazy there's no point in taking on the IO cost in defer_cube, so potentially use isinstance(cube.core_data(), dask.array) to check if the data is not worth dumping out to disk and back.

[arjclark]

An attempt at a proof of concept shows that numpy arrays that are masked cannot currently be saved to disk:

NotImplementedError: MaskedArray.tofile() not implemented yet.

In theory this can be worked around with a bit of care - possibly relevant discussion over at: https://stackoverflow.com/questions/13877063/how-to-save-numpy-masked-array-to-file