Skip to content

release-process.md

Latest commit

 

History

History
74 lines (54 loc) · 3.53 KB

release-process.md

File metadata and controls

74 lines (54 loc) · 3.53 KB

Release Process

Environments

There are three sets of LeadOut environments:

  1. Sandbox - Each developer has their own sandbox environment, which they have full ownership over and can update at will.
  2. Dev - There is one development environment, whose purpose is for the LeadOut team and tests to see live code changes. This environment is updated in the releae pipeline, below.
  3. Prod - There is one Production environment, whose purpose is for users to use the product. This environment is updated in the releae pipeline, below.

Additionally, there is an auxiliary environment, which doesn't run the LeadOut application:

  1. Deployment - This environment runs the release pipeline.

Pipeline

The pipeline is triggered whenever code changes in the source repository, defined via a CodeStar Connection. Once triggered:

  1. Codebuild builds the client and CDK code.
  2. The deployments env updates itself.
  3. The dev environment updates.
  4. The pipeline waits 1 hour. During this time, devs can stop the pipeline if they see any problems in dev.
  5. The prod environment updates.

Next Steps / TODOs

These are a few things not-yet-implemented, largely from the prod design doc.

Stop the prod release on alarm

If any dev alarms are firing, that is a signal there is a problem with the dev release and it should not reach prod. Currently, devs have to manually stop the release to prevent it from updating prod.

This may be done by adding a CodeBuild step to the pipeline between dev and prod, which executes a command that looks for any alarms in dev. We tried this but ran into (not-insurmountable) timeout and permissions issues.

Automatically roll back on failed release

When a release fails, it may leave the application in a partially-updated state. If the pipeline fails for any reason, it should re-deploy to the last-known-good state.

This may be done by splitting the (frequently-changing) application deployment out from the (relatively-static) infrastructure deployment:

  1. Infrastructure, such as CloudFront, is deployed via the pipeline, as-is. CodePipeline does not natively support (automated) rollbacks, so rollbacks would have to be done manually. Since this changes infrequently, this is acceptable.
  2. Application code, such as the API, is deployed via CodeDeploy with a canary step. CodeDeploy supports automated rollbacks, but can only deploy to lambdas, ECS, or EC2.

Restart the tile server on release

The tile server does not automatically pick up schema changes and requires a human to restart the jobs. The release pipeline should restart the tile server jobs so that it can use schema changes on startup.

This may be done by adding a CodeDeploy step that updates the tile server in ECS.