Terraform, Lambda, and Frontend Changes

[gt1074]

Full implementation of the Terraform AWS suite. Transfers data pipeline from scripts to Terraform configuration. Initializes all necessary S3 buckets, Lambda functions, IAM privileges, ECS and EC2 modules, SNS modules, and private VPC configurations. All required attributes are configurable through terraform variables.

Changed Geojson metadata configuration. Geojson metadata is now configured in the validation lambda, fixing a bug where non validated files were left hanging in the staging bucket with no way to delete them.

Added AWS SSM parameter to handle the provider authentication string for submitting to DCDB using the submission lambda.

Submission and validation lambda now correctly update metadata to include upload and validation status, while showing whether a file is online, archived, or deleted. Metadata is now kept even if a file is deleted from S3.

Changed python consumers and socket manager to include payload and reconnection support.

Configured the frontend Geojson map to render tiles from AWS Location Services.

Configured frontend to now display status (upload/validation/conversion) and state (online/archived/deleted) of files.

Use AWS RDS instances instead of ECS container database instances for the frontend and manager.

Switched to AWS Elasticache for Redis implementation.

[brian-r-calder]

I can't claim to have fully understood every part of the PR (because I haven't internalised all of the structure of the code, only examined the diffs), but except as indicated it appears logical to me. I agree with Brian that we should do a dry install of the system using this setup to check before fully merging.

[selimnairb]

@gt1074 As part of wrapping up this PR, please implement the S3 backend for storing Terraform state as is done for the UploadServer, as described here. You can use the same default bucket name ("unhjhc-wibl-tf-state"), but pick a different key for now, maybe something like "terraform/state/wibl-processing-server-deploy.tfstate". Eventually, we can use the same key so that the UploadServer can be deployed into the same stack, but for now we'll keep them separate.

[gt1074]

Do you want me to have the user run a script nearly identical to the Terraform-bootstrap.sh script, meaning the upload server and the wibl-python section both use different buckets. Or do you want to just port the configurations, leading the Terraform module in wibl-python to save a copy of its state file in the same bucket as the upload server, just a different location.

[selimnairb]

Do you want me to have the user run a script nearly identical to the Terraform-bootstrap.sh script, meaning the upload server and the wibl-python section both use different buckets. Or do you want to just port the configurations, leading the Terraform module in wibl-python to save a copy of its state file in the same bucket as the upload server, just a different location.

Just port the configuration over for now, but use a different default key name for the state file for wibl-python. We can merge them later (ideally, the upload server will be fully integrated into the wibl-python cloud stack). I just want to make them as amenable to merging as we can right now.