Resilient Vault with ASG, KMS Auto-Unseal and S3 Storage

Keeping Vault available but easy to manage can be achieved with AWS toolings

I am a big fan of Hashicorp Vault (I even run it on my dev box). But for bigger installations more thought needs to be put into availability and recovery. Using AWS tooling we can achieve this goal easily even on a single node cluster.

A major piece of modern architecture is a cattle over pets model. If our Vault node becomes unreachable, restarts unexpectedly or is terminated we don’t want to have to manually intervene to get the service back online. We want to rely on AWS tooling to do this for us. So the idea here is to have an immutable instance of Vault which if it is killed for any reason will bring itself back up and online in a fully working state but with minimal maintenance overheads.

Note: this setup will involve downtime if there is an issue with Vault or the EC2 instance. For super high uptime installations please look at clustering Vault with a high availability backend.

1. The Auto Scaling Group

As the plan is to only run a single node cluster (in most companies I have worked a few minutes of MTTR is acceptable), using an auto scaling group allows the Vault to be “self-healing”, in that if it goes down for any reason a new instance will be spun up and the User Data scripts run again.

This keeps the availability high for unexpected issues, and also allows upgrades to take place at the operators command by either minting a new AMI (see Packer) and updating the Launch configuration/template or updating the User Data to pull in different versions. A prebuilt AMI will likely load faster than configuring via User Data so bear this in mind when decided how to setup Vault.

Using an ASG also allows Vault to run on SPOT so that if there are budget restrictions, or running a dev Vault system this can be done in the cheapest way and a new instance will be spun up again after a SPOT termination.

Of course if you have a Kubernetes cluster available then it might be more appropriate to run an immutable Vault on there …. Just don’t store the secrets you need to build the cluster on the Vault ;) I tend to keep Vault outside of the Kube cluster purely from a separation of concerns POV.

2. S3 Storage Backend

The common backend Vault uses is the filesystem (or Raft) for storing data. This wont give us the simple immutability that we are looking for as we would lose the data for our Vault if the instance was terminated.

We could use DynamoDB, and this would be the choice for clustered scenarios, but for a single node, S3 is much easier to configure with practically zero maintenance required.

To setup the S3 backend you will need to add the storage “s3” stanza to your HCL config file.

In its simplest form that could look like this.

storage "s3" {
  access_key = "abcd1234"
  secret_key = "defg5678"
  bucket     = "my-bucket"
}

If you are using an IAM on your Vault node you can omit the access_key and secret_key e.g.

storage "s3" {
  bucket     = "my-bucket"
  region     = "eu-west-1"
  path       = "vault/data/"
}

For a full range of options for the backend see the docs

3. KMS Auto-Unseal

A headache of a default setup Vault is that on reboot you need to enter in a set number of unseal keys to be able to bring Vault back online. While this is super secure, in most environments this is overkill and doesn’t fit the cattle model we are looking for.

Vault allows itself to be automatically unsealed using AWS KMS. Once you have created the KMS key you will need to give your Vault instance IAM permissions on that key.

An example Terraform policy would be

data "aws_iam_policy_document" "vault-kms-unseal" {
  statement {
    sid       = "VaultKMSUnseal"
    effect    = "Allow"
    resources = [aws_kms_key.vault.arn]

    actions = [
      "kms:Encrypt",
      "kms:Decrypt",
      "kms:DescribeKey",
    ]
  }
}

and then you will need to set the seal "awskms" stanza in your config file.

seal "awskms" {
    region = "eu-west-1"
    kms_key_id = "1a111b22-a123-b123-c123-333e9ca99999"
}

4. Cloudwatch

As we are using AWS services, might as well log to AWS too.

Using the AWS Cloudwatch Agent logs can be sent from the Vault node to Cloudwatch.

A file audit device will need to be configured to allow Vault to log to a path that the Cloudwatch agent will collect from.

vault audit enable file file_path=/var/log/vault_audit.log

Conclusion

Combining these three AWS tools we can run a single node Vault cluster in a way that it can be treated like cattle and will withstand termination but still remain secure and usable.

Detailed Configuration: I haven’t gone into any details about how things should be setup as there are too many variables to consider e.g. Public or Private subnets, run in a container or install via yum, Amazon Linux or Debian, Packer or Userdata etc etc. The idea was to give an overview architecture and let you choose how to implement it.

Security: for the same reasons as above I have not given any security configurations as your company might have totally different policies to how I would do it. So please ensure you secure S3 buckets, Security Groups etc inline with best practice and your policies.

References:

• https://developer.hashicorp.com/vault/tutorials/auto-unseal/autounseal-aws-kms#step-1-provision-the-cloud-resources
• https://developer.hashicorp.com/vault/docs/configuration/storage/s3
• https://developer.hashicorp.com/vault/docs/audit/file
• https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html
• https://docs.aws.amazon.com/autoscaling/ec2/userguide/auto-scaling-groups.html
• https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html
• https://www.packer.io/