Vault Presentation

Introduction

Walking through creating a highly available Wordpress blog using AWS and Terraform. We’ll then ensure high security by leveraging the power of Hashicorp Vault to centralize control and auditing of secrets and access rights.

Why use Vault?

• Dynamic generation of AWS access credentials based on IAM policies defined and maintained via Terraform (Ifrastructure as Code practices)
• Smaller attack surface - all secret are managed and distributed as needed with a defined TTL to the various components of the application
• Microservices friendly ‘break glass’ policies - respond to security breaches in a targeted fashion that locks down only the affected parts of the system, leading to higher durability and shorter recovery time
• Full audit trail of all access and secret usage combined with easy to define TTL to ensure correct access for all users
• Simple encryption API for sensitive user/customer data

WP Deployment

Architecture Diagram

Vault Demo

General Credential Storage

Simple key/value pair storage and access in Vault

export VAULT_ADDR=http://127.0.0.1:8200
vault kv put secret/wprds user=dbuser pass=dbpass
vault kv get secret/wprds

==== Data ==== Key Value — ----- pass dbpass user dbuser

AWS IAM Credentials

Create the role for Vault to use and write the credentials:

export VAULT_ADDR=http://127.0.0.1:8200
vault write aws/config/root \
    access_key=ACCESS_KEY_ID \
    secret_key=SECRET_ACCESS_KEY

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:AttachUserPolicy",
                "iam:CreateAccessKey",
                "iam:CreateUser",
                "iam:DeleteAccessKey",
                "iam:DeleteUser",
                "iam:DeleteUserPolicy",
                "iam:DetachUserPolicy",
                "iam:ListAccessKeys",
                "iam:ListAttachedUserPolicies",
                "iam:ListGroupsForUser",
                "iam:ListUserPolicies",
                "iam:PutUserPolicy",
                "iam:RemoveUserFromGroup"
            ],
            "Resource": [
                "arn:aws:iam::ACCOUNT_ID:user/vault-*"
            ]
        }
    ]
}

Create a role in Vault for the new IAM users. In this instance, S3 CRUD rights.

export VAULT_ADDR=http://127.0.0.1:8200
vault write aws/roles/s3-all-crud-role policy=-<<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:*"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}
EOF

vault write aws/config/lease lease=1m lease_max=5m

Then read from the secrets engine to get the credentials:

export VAULT_ADDR=http://127.0.0.1:8200
vault read -format=json aws/creds/s3-all-crud-role

MySQL DB Credentials

Docker Compose for Linked Containers

version: '3.6'

services:

  vault:
    build:
      context: ./vault
      dockerfile: Dockerfile
    ports:
      - 8200:8200
    volumes:
      - ./vault/config:/vault/config
      - ./vault/policies:/vault/policies
      - ./vault/data:/vault/data
      - ./vault/logs:/vault/logs
    environment:
      - VAULT_ADDR=http://127.0.0.1:8200
    command: server -config=/vault/config/vault-config.json
    cap_add:
      - IPC_LOCK
  db:
    image: mysql:5.7
    restart: always
    environment:
      MYSQL_DATABASE: 'db'
      # So you don't have to use root, but you can if you like
      MYSQL_USER: 'vault'
      # You can use whatever password you like
      MYSQL_PASSWORD: 'vaultpass'
      # Password for root access
      MYSQL_ROOT_PASSWORD: 'rootpass'
    ports:
      # <Port exposed> : < MySQL Port running inside container>
      - '3306:3306'
    expose:
      # Opens port 3306 on the container
      - '3306'
      # Where our data will be persisted
    volumes:
      - my-db:/var/lib/mysql
# Names our volume
volumes:
  my-db:

Vault database engine

# Enable the plugin/engine
vault secrets enable database

# Configure the endpoint with credentials with appropriate permissions. Root used only for example purposes.
# For additional security! Role can be tied to the IP of the specific db instance
vault write database/config/vdb plugin_name=mysql-database-plugin connection_url="root:rootpass@tcp(172.18.0.3:3306)/" allowed_roles="my-role"
vault write database/roles/my-role \
   db_name=vdb \
   creation_statements="CREATE USER '{{name}}'@'%' IDENTIFIED BY '{{password}}';GRANT SELECT ON *.* TO '{{name}}'@'%';" \
   default_ttl="1h" \
   max_ttl="24h"

# Read creates new credentials with a defined TTL and the permissions granted in 'my-role' above (GRANT SELECT * ON *.* TO '{{name}}'@'%';)
vault read database/creds/my-role

# login to the DB to confirm lease
mysql -u -p
SHOW GRANTS;
#+---------------------------------------------------------------+
#| Grants for v-root-my-role-25lpj8pMP3Gy24OFm@%                 |
#+---------------------------------------------------------------+
#| GRANT SELECT ON *.* TO 'v-root-my-role-25lpj8pMP3Gy24OFm'@'%' |
#+---------------------------------------------------------------+

# revoke lease early
vault lease revoke database/creds/my-role/b70f4581-55de-d7e5-11d2-f543f3f120ee

Dev Server SSH - OTP

Install vault-ssh-helper on target server: https://github.com/hashicorp/vault-ssh-helper

# Enable the secrets engine
vault secrets enable ssh

# Create key role
vault write ssh/roles/otp_key_role \
    key_type=otp \
    default_user=vaulter \
    cidr_list=134.209.125.174/32

vault write ssh/creds/otp_key_role ip=134.209.125.174

ssh vaulter@134.209.125.174

Config

Development Machine Setup

Tools

• Python 3.7.2
• Terraform v0.11.11
• ansible 2.7.7

AWS Setup

• New User: cloudbase

aws configure --profile cb

Configure Route53

atcloudbase.net

• Get Route53 re-usable delegation set

  aws route53 create-reusable-delegation-set --caller-reference 1224 --profile cb
      

• Update nameservers for atcloudbase.net

Terraforming

Keeping secrets from git

**/.terraform/*
*.tfstate
*.tfstate.*
*.tfvars
.terraform
*.plan
credentials.csv
img/
*lock*

Region and profile setup

aws_profile = "cb"
aws_region  = "us-east-1"

provider "aws" {
  region  = "${var.aws_region}"
  profile = "${var.aws_profile}"
}

variable "aws_region"  {}
variable "aws_profile" {}

Initialize Terraform

terraform init

IAM Access Roles (s3)

#----- IAM -----

#S3_access
resource "aws_iam_instance_profile" "s3_access_profile" {
  name = "s3_access"
  role = "${aws_iam_role.s3_access_role.name}"
}

resource "aws_iam_role_policy" "s3_access_policy" {
  name = "s3_access_policy"
  role = "${aws_iam_role.s3_access_role.id}"

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": "*"
      }
    ]
  }
EOF
}


resource "aws_iam_role" "s3_access_role" {
  name = "s3_access_role"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
      {
        "Action": "sts:AssumeRole",
        "Principal": {
          "Service": "ec2.amazonaws.com"
          },
        "Effect": "Allow",
        "Sid": ""
      }
    ]
  }
EOF
}

Create the VPC

VPC Setup

Define the VPC resource, references CIDR block variable

#----- VPC ------

resource "aws_vpc" "wp_vpc" {
  cidr_block           = "${var.vpc_cidr}"
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags {
    Name = "wp_vpc"
  }
}

Define the CIDR block variable in terraform.tfvars and variables.tf

vpc_cidr = "10.0.0.0/16"

variable "vpc_cidr" {}

Internet Gateway

# Internet Gateway
resource "aws_internet_gateway" "wp_internet_gateway" {
  vpc_id = "${aws_vpc.wp_vpc.id}"

  tags {
    Name = "wp_igw"
  }
}

Route Tables

# Route Tables
resource "aws_route_table" "wp_public_rt" {
  vpc_id = "${aws_vpc.wp_vpc.id}"

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = "${aws_internet_gateway.wp_internet_gateway.id}"
  }

  tags {
    Name = "wp_public"
  }
}

resource "aws_default_route_table" "wp_private_rt" {
  default_route_table_id = "${aws_vpc.wp_vpc.default_route_table_id}"

  tags {
    Name = "wp_private"
  }
}

Subnets

Gather the availability zone information and create cidr blocks array

cidrs = {
  public1  = "10.0.1.0/24"
  public2  = "10.0.2.0/24"
  private1 = "10.0.3.0/24"
  private2 = "10.0.4.0/24"
  rds1     = "10.0.5.0/24"
  rds2     = "10.0.6.0/24"
  rds3     = "10.0.7.0/24"
}

data "aws_availability_zones" "available" {}
variable "cidrs" {
  type = "map"
}

# Subnets
# Public subnets
resource "aws_subnet" "wp_public1_subnet" {
  vpc_id = "${aws_vpc.wp_vpc.id}"
  cidr_block = "${var.cidrs["public1"]}"
  map_public_ip_on_launch = true
  availability_zone = "${data.aws_availability_zones.available.names[0]}"

  tags {
    Name = "wp_public1"
  }
}

resource "aws_subnet" "wp_public2_subnet" {
  vpc_id = "${aws_vpc.wp_vpc.id}"
  cidr_block = "${var.cidrs["public2"]}"
  map_public_ip_on_launch = true
  availability_zone = "${data.aws_availability_zones.available.names[1]}"

  tags {
    Name = "wp_public2"
  }
}

# Private Subnets
resource "aws_subnet" "wp_private1_subnet" {
  vpc_id = "${aws_vpc.wp_vpc.id}"
  cidr_block = "${var.cidrs["private1"]}"
  map_public_ip_on_launch = false
  availability_zone = "${data.aws_availability_zones.available.names[0]}"

  tags {
    Name = "wp_private1"
  }
}

resource "aws_subnet" "wp_private2_subnet" {
  vpc_id = "${aws_vpc.wp_vpc.id}"
  cidr_block = "${var.cidrs["private2"]}"
  map_public_ip_on_launch = false
  availability_zone = "${data.aws_availability_zones.available.names[1]}"

  tags {
    Name = "wp_private2"
  }
}

# RDS Subnets
resource "aws_subnet" "wp_rds1_subnet" {
  vpc_id = "${aws_vpc.wp_vpc.id}"
  cidr_block = "${var.cidrs["rds1"]}"
  map_public_ip_on_launch = false
  availability_zone = "${data.aws_availability_zones.available.names[0]}"

  tags {
    Name = "wp_rds1"
  }
}

resource "aws_subnet" "wp_rds2_subnet" {
  vpc_id = "${aws_vpc.wp_vpc.id}"
  cidr_block = "${var.cidrs["rds2"]}"
  map_public_ip_on_launch = false
  availability_zone = "${data.aws_availability_zones.available.names[1]}"

  tags {
    Name = "wp_rds2"
  }
}

resource "aws_subnet" "wp_rds3_subnet" {
  vpc_id = "${aws_vpc.wp_vpc.id}"
  cidr_block = "${var.cidrs["rds3"]}"
  map_public_ip_on_launch = false
  availability_zone = "${data.aws_availability_zones.available.names[2]}"

  tags {
    Name = "wp_rds3"
  }
}

Subnet Groups

RDS Groups

# RDS Subnet Group

resource "aws_db_subnet_group" "wp_rds_subnetgroup" {
  name = "wp_rds_subnetgroup"

  subnet_ids = [
    "${aws_subnet.wp_rds1_subnet.id}",
    "${aws_subnet.wp_rds2_subnet.id}",
    "${aws_subnet.wp_rds3_subnet.id}"
  ]

  tags {
    Name = "wp_rds_sng"
  }
}

Public Subnet Associations

# Public Subnet Associations

resource "aws_route_table_association" "wp_public_assoc1" {
  subnet_id = "${aws_subnet.wp_public1_subnet.id}"
  route_table_id = "${aws_route_table.wp_public_rt.id}"
}

resource "aws_route_table_association" "wp_public_assoc2" {
  subnet_id = "${aws_subnet.wp_public2_subnet.id}"
  route_table_id = "${aws_route_table.wp_public_rt.id}"
}

Clean up - terraform the terraforming

terraform fmt

Security Groups

ELB

Port 80 open

#----- Security Groups -----

# Public Sec Group
resource "aws_security_group" "wp_public_sg" {
  name = "wp_public_sg"
  description = "ELB public access"
  vpc_id = "${aws_vpc.wp_vpc.id}"

  # HTTP
  ingress {
    from_port = 80
    to_port = 80
    protocol = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port = 0
    to_port = 0
    protocol = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Dev Instance

HTTP, SSH access from local IP

localip = "0.0.0.0/0"

variable "localip" {}

# Dev access from local IP

resource "aws_security_group" "wp_dev_sg" {
  name = "wp_dev_sg"
  description = "Used for access to the dev instance"
  vpc_id = "${aws_vpc.wp_vpc.id}"

  # SSH Rules

  ingress {
    from_port = 22
    to_port = 22
    protocol = "tcp"
    cidr_blocks = ["${var.localip}"]
  }

  # HTTP

  ingress {
    from_port = 80
    to_port = 80
    protocol = "tcp"
    cidr_blocks = ["${var.localip}"]
  }

  egress {
  from_port = 0
  to_port = 0
  protocol = "-1"
  cidr_blocks = ["0.0.0.0/0"]
  }
}

Private Instances (Auto-scaling Group)

Access only within VPC

# Access to entire VPC CIDR

resource "aws_security_group" "wp_private_sg" {
  name = "wp_private_sg"
  description = "Private network access to from VPC"
  vpc_id = "${aws_vpc.wp_vpc.id}"

  ingress {
    from_port = 0
    to_port = 0
    protocol = "-1"
    cidr_blocks = ["${var.vpc_cidr}"]
  }

  egress {
    from_port = 0
    to_port = 0
    protocol = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Database

Only VPC, port 3306 (MYSQL)

# RDS Security Group

resource "aws_security_group" "wp_rds_sg" {
  name = "wp_rds_sg"
  description = "Restricted access for RDS instances"
  vpc_id = "${aws_vpc.wp_vpc.id}"

  ingress {
    to_port = 3306
    from_port = 3306
    protocol = "tcp"

    security_groups = ["${aws_security_group.wp_dev_sg.id}",
      "${aws_security_group.wp_public_sg.id}",
      "${aws_security_group.wp_private_sg.id}"
    ]
  }
}

S3 Bucket and VPC Endpoint

VPC Endpoint

# ----- S3 VPC Endpoint -----

resource "aws_vpc_endpoint" "wp_private-s3_endpoint" {
  service_name = "com.amazonaws.${var.aws_region}.s3"
  vpc_id = "${aws_vpc.wp_vpc.id}"

  route_table_ids = ["${aws_vpc.wp_vpc.main_route_table_id}",
                     "${aws_route_table.wp_public_rt.id}"
                    ]
  policy = <<POLICY
{
    "Statement": [
      {
        "Action": "*",
        "Effect": "Allow",
        "Resource": "*",
        "Principal": "*"
      }
    ]
}
POLICY
}

S3 Bucket

variable "domain_name" {}

domain_name = "atcloudbase"

Getting a random bucket name

#----- S3 Code Bucket -----

resource "random_id" "wp_code_bucket" {
  byte_length = 2
}

resource "aws_s3_bucket" "code" {
  bucket = "${var.domain_name}-${random_id.wp_code_bucket.dec}"
  acl = "private"
  force_destroy = true

  tags {
    Name = "code bucket"
  }
}

NOTE: Must re-run terraform init to initialize the ‘random’ plugin

RDS

Before there was Vault and secure creds creation…

#----- RDS ------

resource "aws_db_instance" "wp_db" {
  allocated_storage = 10
  engine = "mysql"
  engine_version = "5.7"
  instance_class = "${var.db_instance_class}"
  name = "${var.dbname}"
  username = "${var.dbuser}"
  password = "${var.dbpass}"
  db_subnet_group_name = "${aws_db_subnet_group.wp_rds_subnetgroup.name}"
  vpc_security_group_ids = ["${aws_security_group.wp_rds_sg.id}"]
  skip_final_snapshot = true
}

variable "db_instance_class" {}
variable "dbname" {}
variable "dbuser" {}
variable "dbpass" {}

db_instance_class = "db.t2.micro"
dbname = "cbdb"
dbuser = "cloudbase"
dbpass = "cbdbpassing"

NOTE: Plain text pass used here for example purposes. See below for implementation of Vault to create dynamic credentials.

ELB

TODO :: Update to Application Load Balancer

#----- ELB -----

resource "aws_elb" "wp_elb" {
  name = "${var.domain_name}-elb"

  subnets = ["${aws_subnet.wp_public1_subnet.id}",
            "${aws_subnet.wp_public2_subnet.id}"]

  security_groups = ["${aws_security_group.wp_public_sg.id}"]

  listener {
    instance_port = 80
    instance_protocol = "http"
    lb_port = 80
    lb_protocol = "http"
  }

  health_check {
    healthy_threshold = "${var.elb_healthy_threshold}"
    unhealthy_threshold = "${var.elb_unhealthy_threshold}"
    timeout = "${var.elb_timeout}"
    target = "TCP:80"
    interval = "${var.elb_interval}"
  }

  cross_zone_load_balancing = true
  idle_timeout = 400
  connection_draining = true
  connection_draining_timeout = 400

  tags {
    Name = "wp_${var.domain_name}-elb"
  }
}

variable "elb_healthy_threshold" {}
variable "elb_unhealthy_threshold" {}
variable "elb_timeout" {}
variable "elb_interval" {}

elb_healthy_threshold = "2"
elb_unhealthy_threshold = "2"
elb_timeout = "3"
elb_interval = "30"

Creating the Dev Instance

#----- Dev -----

# Key Pair

resource "aws_key_pair" "wp_auth" {
  key_name = "${var.key_name}"
  public_key = "${file(var.public_key_path)}"
}

# Dev Server

resource "aws_instance" "wp_dev" {
  instance_type = "${var.dev_instance_type}"
  ami = "${var.dev_ami}"

  tags {
    Name = "wp_dev"
  }

  key_name = "${aws_key_pair.wp_auth.id}"
  vpc_security_group_ids = ["${aws_security_group.wp_dev_sg.id}"]
  iam_instance_profile = "${aws_iam_instance_profile.s3_access_profile.id}"
  subnet_id = "${aws_subnet.wp_public1_subnet.id}"

  # user_data = "${data.template_file.user_data.rendered}"
  provisioner "local-exec" {
    command = <<EOD
cat <<EOF > aws_hosts
[dev]
${aws_instance.wp_dev.public_ip}
[dev:vars]
s3code=${aws_s3_bucket.code.bucket}
domain=${var.domain_name}
EOF
EOD
  }
    provisioner "local-exec" {
        command = "aws ec2 wait instance-status-ok --instance-ids ${aws_instance.wp_dev.id} --profile ${var.aws_profile} && ansible-playbook -i aws_hosts --extra-vars 'dbname=${var.dbname} dbuser=${random_string.username.result} dbpass=${random_string.password.result} dbaddr=db.${var.domain_name}.net' wordpress.yml"
   }
}

variable "dev_instance_type" {}
variable "dev_ami" {}
variable "public_key_path" {}
variable "key_name" {}

dev_instance_type = "t2.micro"
dev_ami = "ami-b73b63a0"
public_key_path = "/home/alexs/.ssh/cloudbase.pub"
key_name = "cloudbase"

#----- Golden AMI ------

# random AMI ID

resource "random_id" "golden_ami" {
  byte_length = 3
}

# AMI

resource "aws_ami_from_instance" "wp_golden" {
  name = "wp_ami-${random_id.golden_ami.b64}"
  source_instance_id = "${aws_instance.wp_dev.id}"

  provisioner "local-exec" {
    command = <<EOT
cat <<EOF > userdata
#!/bin/bash
/usr/bin/aws s3 sync s3://${aws_s3_bucket.code.bucket} /var/www/html/
/bin/touch /var/spool/cron/root
sudo /bin/echo '*/5 * * * * aws s3 sync s3://${aws_s3_bucket.code.bucket} /var/www/html' >> /var/spool/cron/root
EOF
EOT
  }
}

Auto-scaling Group and Launch Configuration

#----- Launch Config -----

resource "aws_launch_configuration" "wp_lc" {
  name_prefix = "wp_lc-"
  image_id = "${aws_ami_from_instance.wp_golden.id}"
  instance_type = "${var.lc_instance_type}"
  security_groups = ["${aws_security_group.wp_private_sg.id}"]
  iam_instance_profile = "${aws_iam_instance_profile.s3_access_profile.id}"
  key_name = "${aws_key_pair.wp_auth.id}"
  user_data = "${file("userdata")}"

  lifecycle {
    create_before_destroy = true
  }
}

#----- ASG -----

resource "aws_autoscaling_group" "wp_asg" {
  name = "asg-${aws_launch_configuration.wp_lc.id}"
  max_size = "${var.asg_max}"
  min_size = "${var.asg_min}"
  health_check_grace_period = "${var.asg_grace}"
  health_check_type = "${var.asg_hct}"
  desired_capacity = "${var.asg_cap}"
  force_delete = true
  load_balancers = ["${aws_elb.wp_elb.id}"]

  vpc_zone_identifier = ["${aws_subnet.wp_private1_subnet.id}",
                         "${aws_subnet.wp_private2_subnet.id}"
                        ]
  launch_configuration = "${aws_launch_configuration.wp_lc.name}"

  tag {
    key = "Name"
    value = "wp_asg-instance"
    propagate_at_launch = true
  }

  lifecycle {
    create_before_destroy = true
  }
}

variable "lc_instance_type" {}
variable "asg_max" {}
variable "asg_min" {}
variable "asg_grace" {}
variable "asg_hct" {}
variable "asg_cap" {}

lc_instance_type = "t2.micro"
asg_max = "2"
asg_min = "1"
asg_grace = "300"
asg_hct = "EC2"
asg_cap = "2"

Route 53 Records

#----- Route 53 -----

# Primary Zone

resource "aws_route53_zone" "primary" {
  name = "${var.domain_name}.net"
  delegation_set_id = "${var.delegation_set}"
}

# WWW Record

resource "aws_route53_record" "www" {
  zone_id = "${aws_route53_zone.primary.zone_id}"
  name = "www.${var.domain_name}.net"
  type = "A"

  alias {
    name = "${aws_elb.wp_elb.dns_name}"
    zone_id = "${aws_elb.wp_elb.zone_id}"
    evaluate_target_health = false
  }
}

# Dev Record

resource "aws_route53_record" "dev" {
  zone_id = "${aws_route53_zone.primary.zone_id}"
  name = "dev.${var.domain_name}.net"
  type = "A"
  ttl = "300"
  records = ["${aws_instance.wp_dev.public_ip}"]
}

# Private Zone

resource "aws_route53_zone" "secondary" {
  name = "${var.domain_name}.net"
  vpc {
    vpc_id = "${aws_vpc.wp_vpc.id}"
  }
}

# DB Record

resource "aws_route53_record" "db" {
  zone_id = "${aws_route53_zone.secondary.zone_id}"
  name = "db.${var.domain_name}.net"
  type = "CNAME"
  ttl = "300"
  records = ["${aws_db_instance.wp_db.address}"]
}

variable "delegation_set" {}

delegation_set = "N2WOUDW0QCOUSM"

Ansible

Using the right tool for the job - Terraform is great for infrastructure management, while Ansible handles configuration of instances.

Config Note: host_key_checking = false in /etc/ansible/ansible.cfg

Install Wordpress

---
- hosts: dev
  become: yes
  remote_user: ec2-user
  tasks:
    - name: Install Apache
      yum: name={{ item }} state=present
      with_items:
      - httpd
      - php
      - php-mysql
    - name: Make Dir Tree Readable
      file:
        path: /var/www/html
        mode: u=rwX,g=rX,o=rX
        recurse: yes
        owner: apache
        group: apache
    - name: Download Wordpress
      get_url: url=http://wordpress.org/wordpress-latest.tar.gz dest=/var/www/html/wordpress.tar.gz force=yes
    - name: Extract Wordpress
      command: "tar xzf /var/www/html/wordpress.tar.gz -C /var/www/html --strip-components 1"
    - name: Copy wp-config-sample.php to wp-config.php
      command: cp "/var/www/html/wp-config-sample.php" "/var/www/html/wp-config.php"
    - name: Update database credentials in the file
      replace:
        path: "/var/www/html/wp-config.php"
        regexp: "password_here"
        replace: "{{ dbpass }}"

    - name: Update database name in the file
      replace:
        path: "/var/www/html/wp-config.php"
        regexp: "database_name_here"
        replace: "{{ dbname }}"

    - name: Update database user in the file
      replace:
        path: "/var/www/html/wp-config.php"
        regexp: "username_here"
        replace: "{{ dbuser }}"

    - name: Update database address in the file
      replace:
        path: "/var/www/html/wp-config.php"
        regexp: "localhost"
        replace: "{{ dbaddr }}"


    - name: Start and enable Apache
      service: name=httpd state=started enabled=yes

S3 Update

---
- hosts: dev
  become: yes
  remote_user: ec2-user
  tasks:
  - name: Update s3 code bucket
    command: aws s3 sync /var/www/html s3://{{ s3code }}/ --delete
  - shell: echo "define('WP_SITEURL','http://dev."{{ domain }}".net');" >> wp-config.php
    args:
      chdir: /var/www/html
  - shell: echo "define('WP_HOME,'http://dev."{{ domain }}".net');" >> wp-config.php
    args:
      chdir: /var/www/html

Time to Apply!

ssh-agent bash
ssh-add ~/.ssh/cloudbase
terraform plan --out terraform.plan
terraform

Next Steps

1. Visit dev.atcloudbase.net and perform WP initial setup and config
2. Change settings to visit www. instead of dev.
3. Run ‘ansible-playbook -i aws_hosts s3update.yml’ (after all config changes on dev)
4. Install an s3 fileshare plugin (change to cloudfront for future state)

Adding the Vault

Deploy Vault in the VPC

Deploying vault in the same VPC as WP using existing private subnets.

Once complete this does require SSH in, vault operator init, unseal

Set local environment variable VAULT_ADDR and VAULT_SKIP_VERIFY (look into getting around this for production deploys)

vault login with token to get it all together

# :tangle variables.tf
variable vc_ami_id {}
variable vault_cluster_name {}
variable consul_cluster_name {}
variable vault_cluster_size {}
variable consul_cluster_size {}
variable vault_instance_type {}
variable consul_instance_type {}
variable consul_cluster_tag_key {}

#  :tangle terraform.tfvars
vc_ami_id = "ami-0d3b2cf862bc2d41b"
vault_cluster_name = "vault-s3"
consul_cluster_name = "consul-s3"
vault_cluster_size = "3"
consul_cluster_size = "3"
vault_instance_type = "t2.micro"
consul_instance_type = "t2.micro"
consul_cluster_tag_key = "consul-vault-s3-servers"

# ---------------------------------------------------------------------------------------------------------------------
# DEPLOY THE VAULT SERVER CLUSTER
# ---------------------------------------------------------------------------------------------------------------------

module "vault_cluster" {
  # When using these modules in your own templates, you will need to use a Git URL with a ref attribute that pins you
  # to a specific version of the modules, such as the following example:
  source = "github.com/hashicorp/terraform-aws-vault//modules/vault-cluster?ref=v0.0.1"
  cluster_name  = "${var.vault_cluster_name}"
  cluster_size  = "${var.vault_cluster_size}"
  instance_type = "${var.vault_instance_type}"

  ami_id    = "${var.vc_ami_id}"
  user_data = "${data.template_file.user_data_vault_cluster.rendered}"

  vpc_id     = "${aws_vpc.wp_vpc.id}"
  subnet_ids = ["${aws_subnet.wp_public1_subnet.id}",
    "${aws_subnet.wp_public2_subnet.id}"]

  # To make testing easier, we allow requests from any IP address here but in a production deployment, we *strongly*
  # recommend you limit this to the IP address ranges of known, trusted servers inside your VPC.

  allowed_ssh_cidr_blocks              = ["0.0.0.0/0"]
  allowed_inbound_cidr_blocks          = ["0.0.0.0/0"]
  allowed_inbound_security_group_ids   = []
  ssh_key_name                         = "${var.key_name}"
  s3_bucket_name = "hello-big-time12335425432"
}

# ---------------------------------------------------------------------------------------------------------------------
# ATTACH IAM POLICIES FOR CONSUL
# To allow our Vault servers to automatically discover the Consul servers, we need to give them the IAM permissions from
# the Consul AWS Module's consul-iam-policies module.
# ---------------------------------------------------------------------------------------------------------------------

module "consul_iam_policies_servers" {
  source = "github.com/hashicorp/terraform-aws-consul.git//modules/consul-iam-policies?ref=v0.4.0"

  iam_role_id = "${module.vault_cluster.iam_role_id}"
}

# ---------------------------------------------------------------------------------------------------------------------
# THE USER DATA SCRIPT THAT WILL RUN ON EACH VAULT SERVER WHEN IT'S BOOTING
# This script will configure and start Vault
# ---------------------------------------------------------------------------------------------------------------------

data "template_file" "user_data_vault_cluster" {
  template = "${file("${path.module}/user-data-vault.sh")}"

  vars {
    aws_region               = "${var.aws_region}"
    consul_cluster_tag_key   = "${var.consul_cluster_tag_key}"
    consul_cluster_tag_value = "${var.consul_cluster_name}"
  }
}

# ---------------------------------------------------------------------------------------------------------------------
# PERMIT CONSUL SPECIFIC TRAFFIC IN VAULT CLUSTER
# To allow our Vault servers consul agents to communicate with other consul agents and participate in the LAN gossip,
# we open up the consul specific protocols and ports for consul traffic
# ---------------------------------------------------------------------------------------------------------------------

module "security_group_rules" {
  source = "github.com/hashicorp/terraform-aws-consul.git//modules/consul-client-security-group-rules?ref=v0.4.0"

  security_group_id = "${module.vault_cluster.security_group_id}"

  # To make testing easier, we allow requests from any IP address here but in a production deployment, we *strongly*
  # recommend you limit this to the IP address ranges of known, trusted servers inside your VPC.

  allowed_inbound_cidr_blocks = ["0.0.0.0/0"]
}

# ---------------------------------------------------------------------------------------------------------------------
# DEPLOY THE CONSUL SERVER CLUSTER
# ---------------------------------------------------------------------------------------------------------------------

module "consul_cluster" {
  source = "github.com/hashicorp/terraform-aws-consul.git//modules/consul-cluster?ref=v0.4.0"

  cluster_name  = "${var.consul_cluster_name}"
  cluster_size  = "${var.consul_cluster_size}"
  instance_type = "${var.consul_instance_type}"

  # The EC2 Instances will use these tags to automatically discover each other and form a cluster
  cluster_tag_key   = "${var.consul_cluster_tag_key}"
  cluster_tag_value = "${var.consul_cluster_name}"

  ami_id    = "${var.vc_ami_id}"
  user_data = "${data.template_file.user_data_consul.rendered}"

  vpc_id     = "${aws_vpc.wp_vpc.id}"
  subnet_ids = ["${aws_subnet.wp_public1_subnet.id}",
    "${aws_subnet.wp_public2_subnet.id}"]

  # To make testing easier, we allow Consul and SSH requests from any IP address here but in a production
  # deployment, we strongly recommend you limit this to the IP address ranges of known, trusted servers inside your VPC.

  allowed_ssh_cidr_blocks     = ["0.0.0.0/0"]
  allowed_inbound_cidr_blocks = ["0.0.0.0/0"]
  ssh_key_name                = "${var.key_name}"
}

# ---------------------------------------------------------------------------------------------------------------------
# THE USER DATA SCRIPT THAT WILL RUN ON EACH CONSUL SERVER WHEN IT'S BOOTING
# This script will configure and start Consul
# ---------------------------------------------------------------------------------------------------------------------

data "template_file" "user_data_consul" {
  template = "${file("${path.module}/user-data-consul.sh")}"

  vars {
    consul_cluster_tag_key   = "${var.consul_cluster_tag_key}"
    consul_cluster_tag_value = "${var.consul_cluster_name}"
  }
}

#!/bin/bash
# This script is meant to be run in the User Data of each EC2 Instance while it's booting. The script uses the
# run-consul script to configure and start Consul in client mode and then the run-vault script to configure and start
# Vault in server mode. Note that this script assumes it's running in an AMI built from the Packer template in
# examples/vault-consul-ami/vault-consul.json.

set -e

# Send the log output from this script to user-data.log, syslog, and the console
# From: https://alestic.com/2010/12/ec2-user-data-output/
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1

# The Packer template puts the TLS certs in these file paths
readonly VAULT_TLS_CERT_FILE="/opt/vault/tls/vault.crt.pem"
readonly VAULT_TLS_KEY_FILE="/opt/vault/tls/vault.key.pem"

# The variables below are filled in via Terraform interpolation
/opt/consul/bin/run-consul --client --cluster-tag-key "${consul_cluster_tag_key}" --cluster-tag-value "${consul_cluster_tag_value}"
/opt/vault/bin/run-vault --tls-cert-file "$VAULT_TLS_CERT_FILE"  --tls-key-file "$VAULT_TLS_KEY_FILE"

#!/bin/bash
# This script is meant to be run in the User Data of each EC2 Instance while it's booting. The script uses the
# run-consul script to configure and start Consul in server mode. Note that this script assumes it's running in an AMI
# built from the Packer template in examples/vault-consul-ami/vault-consul.json.

set -e

# Send the log output from this script to user-data.log, syslog, and the console
# From: https://alestic.com/2010/12/ec2-user-data-output/
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1

# These variables are passed in via Terraform template interpolation
/opt/consul/bin/run-consul --server --cluster-tag-key "${consul_cluster_tag_key}" --cluster-tag-value "${consul_cluster_tag_value}"

Security in depth with terraform and vault

basic principles for securing terraform state

terraform state can contain sensitive data in the first place protect state files as secrets

use an encrypted backend (encrypted s3)

secrets generation and management in terraform

terraform can generate random username/passwords use resource “random_string”, assign and use as username/pass upload and store these credentials in vault resource “vault_dynamic_secret” “credentials”

showcasae dynamic secrets engine with vault

better option! dynamic secrets backend ex. create mysql users, use the hvac python library in application code to securly interact with the database from the application

example terraform/vault integrations

Using dynamic secrets to generate access credentials for secure terraform use

Admin setup of backend role

resource "vault_aws_secret_backend" "aws" {
  region = "${var.aws_region}"
  default_lease_ttl_seconds = "120"
  max_lease_ttl_seconds     = "240"
}
resource "vault_aws_secret_backend_role" "ec2-admin" {
  backend = "${vault_aws_secret_backend.aws.path}"
  name    = "ec2-admin-role"
policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "iam:*", "ec2:*"
      ],
      "Resource": "*"
    }
  ]
}
EOF
}

data "vault_aws_access_credentials" "creds" {
  backend = "aws"
  role    = "ec2-admin-role"
}
provider "aws" {
  access_key = "${data.vault_aws_access_credentials.creds.access_key}"
  secret_key = "${data.vault_aws_access_credentials.creds.secret_key}"
  region  = "${var.aws_region}"
}

Generating random DB credentials and storing them in vault

#----- RDS ------

# Credential generation

resource "random_string" "username" {
  length = 16
  special = false
  number = false
  upper = false
}

resource "random_string" "password" {
  length = 16
  special = false
}

resource "aws_db_instance" "wp_db" {
  allocated_storage = 10
  engine = "mysql"
  engine_version = "5.7"
  instance_class = "${var.db_instance_class}"
  name = "${var.dbname}"
  username = "${random_string.username.result}"
  password = "${random_string.password.result}"
  db_subnet_group_name = "${aws_db_subnet_group.wp_rds_subnetgroup.name}"
  vpc_security_group_ids = ["${aws_security_group.wp_rds_sg.id}"]
  skip_final_snapshot = true
}

variable "service_name" {}
variable "blog_name" {}
variable "vault_token" {}

service_name = "secret"
blog_name = "At Cloudbase"
vault_token = "s.Q08DkKLVIkDJy2vQpcUliLYE"

New user_data script for the dev instance using the credentials and pre-configuring Wordpress

# Auto-install Wordpress With Creds and wp-config.php already sorted

data "template_file" "user_data" {
  template = "${file("user_data.tpl")}"

  vars {
   dbuser = "${random_string.username.result}",
   dbpass = "${random_string.password.result}",
   dbname = "${var.dbname}"
  }
}

Auto-config user_data script

#!/usr/bin/env bash

#download wordpress
curl -O https://wordpress.org/latest.tar.gz
#unzip wordpress
tar -zxvf latest.tar.gz
#change dir to wordpress
cd wordpress
#copy file to parent dir
cp -rf . ..
#move back to parent dir
cd ..
#remove files from wordpress folder
rm -R wordpress
#create wp config
cp wp-config-sample.php wp-config.php
#set database details with perl find and replace
perl -pi -e "s/database_name_here/${dbname}/g" wp-config.php
perl -pi -e "s/username_here/${dbuser}/g" wp-config.php
perl -pi -e "s/password_here/${dbpass}/g" wp-config.php

#set WP salts
perl -i -pe'
  BEGIN {
    @chars = ("a" .. "z", "A" .. "Z", 0 .. 9);
    push @chars, split //, "!@#$%^&*()-_ []{}<>~\`+=,.;:/?|";
    sub salt { join "", map $chars[ rand @chars ], 1 .. 64 }
  }
  s/put your unique phrase here/salt()/ge
' wp-config.php

#create uploads folder and set permissions
mkdir wp-content/uploads
chmod 775 wp-content/uploads
echo "Cleaning..."
#remove zip file
rm latest.tar.gz

TODOS [0/2]

Add a bastion host to get into production via SSH

Secure key access to dev host with Vault

https://learn.hashicorp.com/vault/secrets-management/sm-ssh-otp