Skip to content

503 from compose: Crash during parallel execution #7

@mcochner

Description

@mcochner

Pachelbel crashes during parallel execution.

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x78439e]

goroutine 231 [running]:
github.com/benjdewan/pachelbel/vendor/github.com/benjdewan/gocomposeapi.(*Client).getJSON(0xc4200709b0, 0x892147, 0xb, 0xae5f882ea153170d, 0xf6a0465b0e40a0cf, 0xee34cb14f73d306d, 0xf6a0465b0e40a0cf, 0xe934cb14f73d306d)
	/go/src/github.com/benjdewan/pachelbel/vendor/github.com/benjdewan/gocomposeapi/composeapi.go:127 +0xce
github.com/benjdewan/pachelbel/vendor/github.com/benjdewan/gocomposeapi.(*Client).GetDeploymentsJSON(0xc4200709b0, 0xef34cb14f73d306d, 0xf6a0465b0e40a0cf, 0xec34cb14f73d306d, 0xf6a0465b0e40a0cf, 0xed34cb14f73d306d)
	/go/src/github.com/benjdewan/pachelbel/vendor/github.com/benjdewan/gocomposeapi/deployment.go:168 +0x40
github.com/benjdewan/pachelbel/vendor/github.com/benjdewan/gocomposeapi.(*Client).GetDeployments(0xc4200709b0, 0xc42057ea38, 0x43274b, 0x12e, 0xc42057ea48)
	/go/src/github.com/benjdewan/pachelbel/vendor/github.com/benjdewan/gocomposeapi/deployment.go:172 +0x2f
github.com/benjdewan/pachelbel/vendor/github.com/benjdewan/gocomposeapi.(*Client).GetDeploymentByName(0xc4200709b0, 0xc4202e62d0, 0x2d, 0xc4202e62d0, 0x2d, 0x0, 0x0)
	/go/src/github.com/benjdewan/pachelbel/vendor/github.com/benjdewan/gocomposeapi/deployment.go:206 +0x43
github.com/benjdewan/pachelbel/connection.(*Connection).GetAndAdd(0xc42006dc80, 0xc4202e62d0, 0x2d, 0xc4201cdc10, 0xf)
	/go/src/github.com/benjdewan/pachelbel/connection/connection.go:137 +0x5d
github.com/benjdewan/pachelbel/runner.dryRunLookup(0xc42006dc80, 0x8ee840, 0xc4201c00d0, 0x0, 0xc4203795a0)
	/go/src/github.com/benjdewan/pachelbel/runner/runner.go:88 +0x43
github.com/benjdewan/pachelbel/runner.(*Controller).Run.func1(0xc4202401e0, 0xc4201a8008, 0xc4203f23d0, 0x8ee840, 0xc4201c00d0, 0xc420358480, 0x13, 0x8b6010)
	/go/src/github.com/benjdewan/pachelbel/runner/controller.go:53 +0x4e
created by github.com/benjdewan/pachelbel/runner.(*Controller).Run
	/go/src/github.com/benjdewan/pachelbel/runner/controller.go:52 +0x255

The reason is that Compose.io sometimes returns 503 instead of valid json when running pachabel in parallel multiple times.

<!DOCTYPE html>
<html class="error-layout" lang="en"><head>
    <title>Error 503: Service Unavailable - Compose</title>
    <meta name="viewport" content="width=device-width,initial-scale=1">
    <meta charset="utf-8">
    <link rel="shortcut icon" type="image/x-icon" href="https://d11f55tj5eo9e5.cloudfront.net/assets/megatron/favicon.ico">
    <link rel="stylesheet" media="screen" href="https://d11f55tj5eo9e5.cloudfront.net/assets/megatron/megatron-error-pages-0.1.81.css.gz">
  </head>
  <body>
    <div class="error-page error-503 ">
      <svg xmlns="http://www.w3.org/2000/svg" width="312"><g id="group-1-cube-1"><path fill="#1D2125" d="M210.6 0l-20 11.2v22.4l20 11.2 20-11.2V11.2L210.6 0z"></path><path fill="#DADEDE" d="M210.6 5.3l-15.3 8.5 15.3 8.6 15.3-8.6-15.3-8.5"></path></g><g id="group-1-cube-2"><path fill="#1D2125" d="M190.6 11.2l-20 11.2v22.4l20 11.2 20-11.2V22.4l-20-11.2z"></path><path fill="#DADEDE" d="M190.6 16.5L175.3 25l15.3 8.6 15.3-8.6-15.3-8.5"></path></g><g id="group-1-cube-3"><path fill="#1D2125" d="M170.6 22.4l-20 11.2V56l20 11.2 20-11.2V33.6l-20-11.2z"></path><path fill="#DADEDE" d="M170.6 27.7l-15.3 8.5 15.3 8.6 15.3-8.6-15.3-8.5"></path></g><g id="group-1-cube-4"><path fill="#1D2125" d="M230.6 11.2l-20 11.2v22.4l20 11.2 20-11.2V22.4l-20-11.2z"></path><path fill="#DADEDE" d="M230.6 16.5L215.3 25l15.3 8.6 15.3-8.6-15.3-8.5"></path></g><g id="group-1-cube-5"><path fill="#1D2125" d="M250.6 22.4l-20 11.2V56l20 11.2 20-11.2V33.6l-20-11.2z"></path><path fill="#DADEDE" d="M250.6 27.7l-15.3 8.5 15.3 8.6 15.3-8.6-15.3-8.5"></path></g><g id="group-1-cube-6"><path fill="#1D2125" d="M230.6 33.6l-20 11.2v22.4l20 11.2 20-11.2V44.8l-20-11.2z"></path><path fill="#DADEDE" d="M230.6 38.9l-15.3 8.5 15.3 8.6 15.3-8.6-15.3-8.5"></path></g><g id="group-1-cube-7"><path fill="#1D2125" d="M210.6 44.8l-20 11.2v22.4l20 11.2 20-11.2V56l-20-11.2z"></path><path fill="#DADEDE" d="M210.6 50.1l-15.3 8.5 15.3 8.6 15.3-8.6-15.3-8.5"></path></g><g id="group-1-cube-8"><path fill="#1D2125" d="M270.6 33.6l-20 11.2v22.4l20 11.2 20-11.2V44.8l-20-11.2z"></path><path fill="#DADEDE" d="M270.6 38.9l-15.3 8.5 15.3 8.6 15.3-8.6-15.3-8.5"></path></g><g id="group-1-cube-9"><path fill="#1D2125" d="M290.6 44.8l-20 11.2v22.4l20 11.2 20-11.2V56l-20-11.2z"></path><path fill="#DADEDE" d="M290.6 50.1l-15.3 8.5 15.3 8.6 15.3-8.6-15.3-8.5"></path></g><g id="group-1-cube-10"><path fill="#1D2125" d="M270.6 56l-20 11.2v22.4l20 11.2 20-11.2V67.2l-20-11.2z"></path><path fill="#DADEDE" d="M270.6 61.3l-15.3 8.5 15.3 8.6 15.3-8.6-15.3-8.5"></path></g><g id="group-1-cube-11"><path fill="#1D2125" d="M250.6 67.2l-20 11.2v22.4l20 11.2 20-11.2V78.4l-20-11.2z"></path><path fill="#DADEDE" d="M250.6 72.5L235.3 81l15.3 8.6 15.3-8.6-15.3-8.5"></path></g><g id="group-2-cube-1"><path fill="#1D2125" d="M135.3 42.2l-20 11.2v22.4l20 11.2 20-11.2V53.4l-20-11.2z"></path><path fill="#DADEDE" d="M135.3 47.5L120 56l15.3 8.6 15.3-8.6-15.3-8.5"></path></g><g id="group-2-cube-2"><path fill="#1D2125" d="M115.3 53.4l-20 11.2V87l20 11.2 20-11.2V64.6l-20-11.2z"></path><path fill="#DADEDE" d="M115.3 58.7L100 67.2l15.3 8.6 15.3-8.6-15.3-8.5"></path></g><g id="group-2-cube-3"><path fill="#1D2125" d="M95.3 64.6l-20 11.2v22.4l20 11.2 20-11.2V75.8l-20-11.2z"></path><path fill="#DADEDE" d="M95.3 69.9L80 78.4 95.3 87l15.3-8.6-15.3-8.5"></path></g><g id="group-2-cube-4"><path fill="#1D2125" d="M155.3 53.4l-20 11.2V87l20 11.2 20-11.2V64.6l-20-11.2z"></path><path fill="#DADEDE" d="M155.3 58.7L140 67.2l15.3 8.6 15.3-8.6-15.3-8.5"></path></g><g id="group-2-cube-5"><path fill="#1D2125" d="M175.3 64.6l-20 11.2v22.4l20 11.2 20-11.2V75.8l-20-11.2z"></path><path fill="#DADEDE" d="M175.3 69.9L160 78.4l15.3 8.6 15.3-8.6-15.3-8.5"></path></g><g id="group-2-cube-6"><path fill="#1D2125" d="M115.3 75.8L95.3 87v22.4l20 11.2 20-11.2V87l-20-11.2z"></path><path fill="#DADEDE" d="M115.3 81.1L100 89.6l15.3 8.6 15.3-8.6-15.3-8.5"></path></g><g id="group-2-cube-7"><path fill="#1D2125" d="M135.3 87l-20 11.2v22.4l20 11.2 20-11.2V98.2l-20-11.2z"></path><path fill="#DADEDE" d="M135.3 92.3l-15.3 8.5 15.3 8.6 15.3-8.6-15.3-8.5"></path></g><g id="group-2-cube-8"><path fill="#1D2125" d="M155.3 98.2l-20 11.2v22.4l20 11.2 20-11.2v-22.4l-20-11.2z"></path><path fill="#DADEDE" d="M155.3 103.5L140 112l15.3 8.6 15.3-8.6-15.3-8.5"></path></g><g id="group-2-cube-9"><path fill="#1D2125" d="M195.3 75.8l-20 11.2v22.4l20 11.2 20-11.2V87l-20-11.2z"></path><path fill="#DADEDE" d="M195.3 81.1L180 89.6l15.3 8.6 15.3-8.6-15.3-8.5"></path></g><g id="group-2-cube-10"><path fill="#1D2125" d="M215.3 87l-20 11.2v22.4l20 11.2 20-11.2V98.2l-20-11.2z"></path><path fill="#DADEDE" d="M2

    <div class="error-message">

          <h1><span>Service Unavailable.</span></h1>
          <p><span>But don't worry, this shouldn't affect your databases.</span></p>

      </div>
      <footer class="support-links">
        <nav>
          <a href="mailto:support@compose.io">Contact Support</a><a href="https://status.compose.io">Compose Status</a>
        </nav>
        <a href="https://compose.io" class="compose-logo"><span class="hidden_label">Compose</span></a>
      </footer>
    </div>

</body></html>

One can simulate it by GNU parallel:

$ seq 25 | parallel -j 25 bash -c "echo 'begin{#}'; ./pachelbel --dry-run -l multi_compose.log_{#} provision example_multi_dev.yaml; echo 'end{#}'"
 $ cat example_multi_dev.yaml
config_version: 1
name: postgres-XXXXXXXXX-YYYYYYYYYY
type: postgresql
cluster: some-cluster
---
config_version: 1
name: postgres-XXXXXXXXX-1
type: elasticsearch
cluster: some-cluster
---
config_version: 1
name: elasticsearch-XXXXXXXXX-1
type: elasticsearch
cluster: some-cluster
---
config_version: 1
name: elasticsearch-XXXXXXXXX-2
type: elasticsearch
cluster: some-cluster
---
config_version: 1
name: elasticsearch-XXXXXXXXX-3
type: elasticsearch
cluster: some-cluster

Metadata

Metadata

Assignees

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions