Default CDH and CM version not working

[d-becker]

Running

clusterdock start topology_cdh

without further arguments fails with the following:

$ clusterdock start topology_cdh
2018-07-05 11:40:06,283 [INFO] [clusterdock.cli] clusterdock_config_directory_path = /Users/danielbecker/.clusterdock
2018-07-05 11:40:06,306 [INFO] [clusterdock.models] self.clusterdock_config_host_dir = /Users/danielbecker/.clusterdock
2018-07-05 11:40:06,306 [INFO] [clusterdock.models] self.clusterdock_config_host_dir = /Users/danielbecker/.clusterdock
2018-07-05 11:40:06,306 [INFO] [clusterdock.models] Starting cluster (fissile_bullet) on network (cluster) ...
2018-07-05 11:40:06,429 [INFO] [clusterdock.models] Starting node node-1.cluster ...
2018-07-05 11:40:06,435 [INFO] [clusterdock.models] Could not find docker.io/cloudera/clusterdock:cdh5.12.0_cm5.12.0_primary-node locally. Attempting to pull ...
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 229, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 939, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.35/containers/create

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/clusterdock/models.py", line 357, in start
    **create_container_kwargs)['Id']
  File "/usr/local/lib/python3.7/site-packages/docker/api/container.py", line 411, in create_container
    return self.create_container_from_config(config, name)
  File "/usr/local/lib/python3.7/site-packages/docker/api/container.py", line 422, in create_container_from_config
    return self._result(res, True)
  File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 235, in _result
    self._raise_for_status(response)
  File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 231, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/local/lib/python3.7/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error: Not Found ("No such image: cloudera/clusterdock:cdh5.12.0_cm5.12.0_primary-node")

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 229, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 939, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.35/images/create?tag=cdh5.12.0_cm5.12.0_primary-node&fromImage=docker.io%2Fcloudera%2Fclusterdock

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/clusterdock", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/clusterdock/cli.py", line 239, in main
    action.main(args)
  File "/usr/local/lib/python3.7/site-packages/clusterdock/actions/start.py", line 31, in main
    action.main(args)
  File "/Users/danielbecker/clusterdock/topology_cdh/start.py", line 137, in main
    cluster.start(args.network)
  File "/usr/local/lib/python3.7/site-packages/clusterdock/models.py", line 121, in start
    node.start(self.network, cluster_name=self.name)
  File "/usr/local/lib/python3.7/site-packages/clusterdock/models.py", line 360, in start
    client.images.pull(self.image)
  File "/usr/local/lib/python3.7/site-packages/docker/models/images.py", line 412, in pull
    self.client.api.pull(repository, tag=tag, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/docker/api/image.py", line 399, in pull
    self._raise_for_status(response)
  File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 231, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/local/lib/python3.7/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.NotFound: 404 Client Error: Not Found ("manifest for cloudera/clusterdock:cdh5.12.0_cm5.12.0_primary-node not found")

It seems the docker images are missing, the only one I could find is for CDH version 5.8.0 and CM version 5.8.2 (https://hub.docker.com/r/cloudera/clusterdock/tags/). With those versions the topology does work (or, more precisely, gets past this point and fails elsewhere...).

[dimaspivak]

This is happening because Cloudera’s license doesn’t allow us to distribute CM, so these images would have to be created manually by the user first. I’ll update the docs and likely remove the default arguments for —cdh-version and —cm-version to make it more clear.

[umbrant]

Hi Dima,

I actually hit this same problem today too, thanks for the quick response! Is there a script or instructions on how to build a CM+CDH image yourself? Without images, it's hard to get going with clusterdock.

[av8orbynight]

@dimaspivak I'm a bit confused. I'm simply following the two instructions:

$ git clone https://github.com/clusterdock/topology_cdh.git
$ clusterdock start topology_cdh

and am receiving the same error.

running:

clusterdock start topology_cdh -cdh-version=5.8.0 -cm-version=5.8.2

still gives the same error.

How is this topology useful if it won't create the cluster?

Is there something I'm missing?

@umbrant @d-becker Did you find out how to create cdh primary and secondary nodes using clusterdock?

Thanks,

K