Fixes regressions when running on a cluster with more or fewer than 4 engines by kwmsmith · Pull Request #489 · enthought/distarray

kwmsmith · 2014-07-03T17:58:40Z

This is based on @cowlicks PR #441. Rebased on master.

coveralls · 2014-07-03T18:17:15Z

Coverage increased (+0.04%) when pulling 9b34b3a on bugfix/cluster-size-regression into 02d25eb on master.

bgrant · 2014-07-03T20:54:20Z

Hey @kwmsmith : I get a failure with 11 engines under Python3:

$ make test
python -m unittest discover
..........................s........................................................................................................ssss.................................................................................................................................E....................................................
======================================================================
ERROR: test_dist_sizes (distarray.tests.test_metadata_utils.TestGridSizes)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./distarray/tests/test_metadata_utils.py", line 225, in test_dist_sizes
    dist = Distribution(self.context, (2, 3, 4), dist=('n', 'b', 'c'))
  File "./distarray/dist/maps.py", line 594, in __new__
    grid_shape = grid_shape or make_grid_shape(shape, dist, len(targets))
  File "./distarray/metadata_utils.py", line 180, in make_grid_shape
    check_grid_shape_postconditions(out_grid_shape, shape, dist, comm_size)
  File "./distarray/metadata_utils.py", line 69, in check_grid_shape_postconditions
    "= %s" % (shape, grid_shape))
ValueError: all(gs <= s for (s, gs) in zip(shape, grid_shape) if s > 0) not satisfied, shape = (2, 3, 4) and grid_shape = (1, 1, 11)

----------------------------------------------------------------------
Ran 316 tests in 21.922s

FAILED (errors=1, skipped=5)
make: *** [test_client] Error 1

bgrant · 2014-07-03T21:00:38Z

Also lots of HubTimeoutErrors for n=1...

bgrant · 2014-07-03T21:02:54Z

Here's the first error:

$ python -m unittest discover -cf
......s..ss.s...s......s........ssssssE
======================================================================
ERROR: setUpClass (distarray.dist.tests.test_distarray.TestSetItemSlicing)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./distarray/testing.py", line 180, in setUpClass
  File "/Users/robertgrant/development/venvs/py33/lib/python3.3/site-packages/IPython/parallel/client/client.py", line 496, in __init__
    self._connect(sshserver, ssh_kwargs, timeout)
  File "/Users/robertgrant/development/venvs/py33/lib/python3.3/site-packages/IPython/parallel/client/client.py", line 615, in _connect
    raise error.TimeoutError("Hub connection request timed out")
IPython.parallel.error.TimeoutError: Hub connection request timed out

----------------------------------------------------------------------
Ran 30 tests in 11.289s

FAILED (errors=1, skipped=12)
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "./distarray/dist/cleanup.py", line 27, in cleanup_all
  File "/Users/robertgrant/development/venvs/py33/lib/python3.3/site-packages/IPython/parallel/client/client.py", line 496, in __init__
    self._connect(sshserver, ssh_kwargs, timeout)
  File "/Users/robertgrant/development/venvs/py33/lib/python3.3/site-packages/IPython/parallel/client/client.py", line 615, in _connect
    raise error.TimeoutError("Hub connection request timed out")
IPython.parallel.error.TimeoutError: Hub connection request timed out
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "./distarray/dist/cleanup.py", line 69, in clear_all
  File "/Users/robertgrant/development/venvs/py33/lib/python3.3/site-packages/IPython/parallel/client/client.py", line 496, in __init__
  File "/Users/robertgrant/development/venvs/py33/lib/python3.3/site-packages/IPython/parallel/client/client.py", line 615, in _connect
    raise error.TimeoutError("Hub connection request timed out")
IPython.parallel.error.TimeoutError: Hub connection request timed out

It seems to be the same under Python 2.

kwmsmith · 2014-07-03T21:16:39Z

@bgrant the all(gs <= s for (s, gs) in zip(shape, grid_shape) if s > 0) not satisfied, shape = (2, 3, 4) and grid_shape = (1, 1, 11) error is a separate issue, to be addressed in a different PR.

Don't know about the hub timeout stuff, looking into it.

kwmsmith · 2014-07-03T21:24:43Z

@bgrant I can't reproduce the hub timeout error, but I did fix something in test_metadata_utils.py. Please try again.

bgrant · 2014-07-03T21:31:49Z

$ dacluster restart -n1                                                                                                          [10/9626]
2014-07-03 16:29:56.978 [IPClusterStop] Using existing profile dir: '/Users/robertgrant/development/venvs/py33/ipython-config/profile_default'
2014-07-03 16:29:57.018 [IPClusterStop] Stopping cluster [pid=24265] with [signal=2]
2014-07-03 16:29:57.565 [IPClusterStart] Using existing profile dir: '/Users/robertgrant/development/venvs/py33/ipython-config/profile_default'
2014-07-03 16:29:57.573 [IPClusterStart] Starting ipcluster with [daemon=False]
2014-07-03 16:29:57.574 [IPClusterStart] Creating pid file: /Users/robertgrant/development/venvs/py33/ipython-config/profile_default/pid/ipcluster.pid
2014-07-03 16:29:57.574 [IPClusterStart] Starting Controller with LocalControllerLauncher
2014-07-03 16:29:58.575 [IPClusterStart] Starting 1 Engines with MPIEngineSetLauncher
2014-07-03 16:30:28.581 [IPClusterStart] Engines appear to have started successfully

$ python -m unittest discover -cf
......s..ss.s...s......s........ssssssE
======================================================================
ERROR: setUpClass (distarray.dist.tests.test_distarray.TestSetItemSlicing)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./distarray/testing.py", line 180, in setUpClass
  File "/Users/robertgrant/development/venvs/py33/lib/python3.3/site-packages/IPython/parallel/client/client.py", line 496, in __init__
  File "/Users/robertgrant/development/venvs/py33/lib/python3.3/site-packages/IPython/parallel/client/client.py", line 615, in _connect
IPython.parallel.error.TimeoutError: Hub connection request timed out

----------------------------------------------------------------------
Ran 30 tests in 11.282s

FAILED (errors=1, skipped=12)
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "./distarray/dist/cleanup.py", line 27, in cleanup_all
  File "/Users/robertgrant/development/venvs/py33/lib/python3.3/site-packages/IPython/parallel/client/client.py", line 496, in __init__
    self._connect(sshserver, ssh_kwargs, timeout)
  File "/Users/robertgrant/development/venvs/py33/lib/python3.3/site-packages/IPython/parallel/client/client.py", line 615, in _connect
    raise error.TimeoutError("Hub connection request timed out")
IPython.parallel.error.TimeoutError: Hub connection request timed out
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "./distarray/dist/cleanup.py", line 69, in clear_all
  File "/Users/robertgrant/development/venvs/py33/lib/python3.3/site-packages/IPython/parallel/client/client.py", line 496, in __init__
  File "/Users/robertgrant/development/venvs/py33/lib/python3.3/site-packages/IPython/parallel/client/client.py", line 615, in _connect
    raise error.TimeoutError("Hub connection request timed out")
IPython.parallel.error.TimeoutError: Hub connection request timed out

bgrant · 2014-07-03T21:34:57Z

Or with -n3:

$ python -m unittest discover -cf
......s...s.......s......s........sssssssE
======================================================================
ERROR: setUpClass (distarray.dist.tests.test_distributed_io.TestDnpyFileIO)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./distarray/dist/tests/test_distributed_io.py", line 43, in setUpClass
  File "./distarray/testing.py", line 180, in setUpClass
  File "/Users/robertgrant/development/venvs/py33/lib/python3.3/site-packages/IPython/parallel/client/client.py", line 496, in __init__
  File "/Users/robertgrant/development/venvs/py33/lib/python3.3/site-packages/IPython/parallel/client/client.py", line 615, in _connect
IPython.parallel.error.TimeoutError: Hub connection request timed out

----------------------------------------------------------------------
Ran 33 tests in 12.985s

FAILED (errors=1, skipped=11)
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "./distarray/dist/cleanup.py", line 27, in cleanup_all
  File "/Users/robertgrant/development/venvs/py33/lib/python3.3/site-packages/IPython/parallel/client/client.py", line 496, in __init__
    self._connect(sshserver, ssh_kwargs, timeout)
  File "/Users/robertgrant/development/venvs/py33/lib/python3.3/site-packages/IPython/parallel/client/client.py", line 615, in _connect
    raise error.TimeoutError("Hub connection request timed out")
IPython.parallel.error.TimeoutError: Hub connection request timed out
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "./distarray/dist/cleanup.py", line 69, in clear_all
  File "/Users/robertgrant/development/venvs/py33/lib/python3.3/site-packages/IPython/parallel/client/client.py", line 496, in __init__
  File "/Users/robertgrant/development/venvs/py33/lib/python3.3/site-packages/IPython/parallel/client/client.py", line 615, in _connect
IPython.parallel.error.TimeoutError: Hub connection request timed out

bgrant · 2014-07-04T18:31:23Z

Okay- this works for me with n >= 4 and apparently for everyone else with 0 < n < 4, so I'm merging.

Fixes regressions when running on a cluster with more or fewer than 4 engines

cowlicks and others added 4 commits July 3, 2014 12:24

Use distribution.targets instead of context.targets in decorators.py

accc7e1

Change ContextError to DistributionError in decorators.

40a2233

Reflect changes in decorator tests

ed09d8d

Fix regression when running on more than 4 engines.

9b34b3a

kwmsmith added the bug label Jul 3, 2014

kwmsmith added this to the 0.4 milestone Jul 3, 2014

This was referenced Jul 3, 2014

Fixes issue #403. #490

Merged

Fix decorators to use the distribution.targets instead of context.targets where appropriate. #441

Closed

Use ContextTestCase as base class.

aa2126b

bgrant added a commit that referenced this pull request Jul 4, 2014

Merge pull request #489 from enthought/bugfix/cluster-size-regression

e493295

Fixes regressions when running on a cluster with more or fewer than 4 engines

bgrant merged commit e493295 into master Jul 4, 2014

bgrant deleted the bugfix/cluster-size-regression branch July 4, 2014 18:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes regressions when running on a cluster with more or fewer than 4 engines#489

Fixes regressions when running on a cluster with more or fewer than 4 engines#489
bgrant merged 5 commits into
masterfrom
bugfix/cluster-size-regression

kwmsmith commented Jul 3, 2014

Uh oh!

coveralls commented Jul 3, 2014

Uh oh!

bgrant commented Jul 3, 2014

Uh oh!

bgrant commented Jul 3, 2014

Uh oh!

bgrant commented Jul 3, 2014

Uh oh!

kwmsmith commented Jul 3, 2014

Uh oh!

kwmsmith commented Jul 3, 2014

Uh oh!

bgrant commented Jul 3, 2014

Uh oh!

bgrant commented Jul 3, 2014

Uh oh!

bgrant commented Jul 4, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

kwmsmith commented Jul 3, 2014

Uh oh!

coveralls commented Jul 3, 2014

Uh oh!

bgrant commented Jul 3, 2014

Uh oh!

bgrant commented Jul 3, 2014

Uh oh!

bgrant commented Jul 3, 2014

Uh oh!

kwmsmith commented Jul 3, 2014

Uh oh!

kwmsmith commented Jul 3, 2014

Uh oh!

bgrant commented Jul 3, 2014

Uh oh!

bgrant commented Jul 3, 2014

Uh oh!

bgrant commented Jul 4, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants