Reduce the number of roundtrips for DistArray.get_localshapes to zero.

[cowlicks]

This also adds functions to metadata utils that could be useful for local map construction. But that can wait for another PR.

[kwmsmith]

What if stop or start is None? Is stop - start supposed to raise an exception in that case?

[cowlicks]

TypeError: unsupported operand type(s) for -: `int` and `NoneType`

[kwmsmith]

It seems a bit more straightforward to let it error out earlier, giving a KeyError rather than a TypeError, which to my mind is more informative:

stop, start = dim_data['stop'], dim_data['start']
return stop - start

[cowlicks]

That does make more sense. Which things in the dim_data should have default values incase they are not provided. Did I get it right here and here?

[kwmsmith]

On Tue, Jun 10, 2014 at 11:35 AM, Blake Griffith notifications@github.com
wrote:

In distarray/metadata_utils.py:

@@ -229,3 +229,87 @@ def normalize_reduction_axes(axes, ndim):
else:
axes = tuple(positivify(a, ndim) for a in axes)
return axes
+
+
+# functions for getting a size from a dim_data for each dist_type
+# n
+def non_dist_size(dim_data):

• return dim_data['size']

+# b
+def block_size(dim_data):

• stop = dim_data.get('stop', None)
• start = dim_data.get('start', None)
• return stop - start

That does make more sense. Which things in the dim_data should have
default values incase they are not provided. Did I get it right here
https://github.com/enthought/distarray/pull/423/files#diff-4eed00415103bcf79667fada17612dbaR271
and here
https://github.com/enthought/distarray/pull/423/files#diff-4eed00415103bcf79667fada17612dbaR282
?

The distributed array protocol goes into the details:

http://distributed-array-protocol.readthedocs.org/en/rel-0.10.0/

—
Reply to this email directly or view it on GitHub
https://github.com/enthought/distarray/pull/423/files#r13604231.

Kurt W. Smith, Ph.D. ksmith@enthought.com
Enthought, Inc. http://www.enthought.com | 512.536.1057

[cowlicks]

@kwmsmith @bgrant what is the standard way to make a cyclic distribution with block_size > 1?

[bgrant]

Use a global_dim_data:

For example,

In [3]: from distarray.dist import DistArray, Context, Distribution
In [4]: c = Context()
In [5]: bc_dim = dict(dist_type='c', proc_grid_size=4, size=50, block_size=3)
In [6]: global_dim_data = (bc_dim,)
In [8]: d = Distribution(c, global_dim_data)
In [11]: d.maps[0]
Out[11]: <distarray.dist.maps.BlockCyclicMap at 0x104ca8750>
In [12]: d.maps[0].block_size
Out[12]: 3

[kwmsmith]

@cowlicks I'd prefer if there was a .get_localshapes() method on client-side Distribution objects, and DistArray.get_localshapes() just returns self.distribution.get_localshapes(). The Distribution.get_localshapes() would just do:

return get_shape_from_dim_data_per_rank(self.get_dim_data_per_rank())

Having a Distribution.get_localshapes() method will be useful in other places as well.

[cowlicks]

On that note, since these no longer require communication, should we drop the get_ prefixes?

[kwmsmith]

For now, keeping it Distribution.get_localshapes() is fine, the get_ prefix doesn't imply communication, just that it's a getter. We might make it into a localshapes property down the line, but since there's some non-trivial logic going on here, I'd prefer the get_localshapes() name.

[cowlicks]

I think this is ready, just waiting on Travis.

[coveralls]

Coverage increased (+0.1%) when pulling 548fc40 on cowlicks:get_localshapes into e4acdac on enthought:master.

[kwmsmith]

Restarting Travis.

[cowlicks]

Travis is having a bad day.