Some functions create too many roundtrips

[cowlicks]

With 4 engines, using the count_round_trip context manager, with a 100x100 block distributed distarray:

• 9 roundtrips with DistArray.sum, it is strange that this is an odd number and not divisible by the number of engines.
• 8 roundtrips with DistArray.fill this should be 4
• stuff in distarray.dist.functions has 10 roundtrips. Also not divisible by the number of engines.

I'll add new things to this issue as I find them.

[cowlicks]

• DistArray.mean and DistArray.var also have 9 roundtrips like sum.
• DistArray.tondarray has 8 roundtrips, this should be 4.

[cowlicks]

• Context.ones, zeros, and empty all have 12 roundtrips.

[cowlicks]

The odd 9 round trips from sum, mean, and var are from pushing arguments (4), calling LocalArray.sum (4), then pulling the result from engine 0 (1).

@kwmsmith will this be fixed in your recent reduction work?

[kwmsmith]

@cowlicks -- yes, this will be addressed in the reduction improvements. After those improvements, reductions will take 1 roundtrip per engine.

[kwmsmith]

@cowlicks can you check that the reduction methods take one roundtrip per engine? If that is resolved, then please either close or update this issue with the remaining methods to fix.

[kwmsmith]

@cowlicks moving this to 0.4 for the remaining methods.

[cowlicks]

A DistArray.sum() on 4 engines with a square dist=('b', 'n') distarray takes 17 roundtrips the first time, and 13 the following times. Same thing summing along the distributed axis.

Summing along the undistributed axis takes 16 roundtrips.

[kwmsmith]

This will be improved when .get_localshapes() can be refactored so it doesn't require any communication.

[cowlicks]

With PR #423:
DistArray.sum() on 4 engines with a square dist=('b', 'n') distarray takes 9 roundtrips the first time, and 5 the following times. Same thing summing along the distributed axis.

Summing along the undistributed axis takes 8 roundtrips.