One more terraform best practice

In this article, I will discuss some terraform approaches that are often overlooked or underutilized. This article is not specific to any cloud provider like AWS or GCP, nor does it involve any wrappers like Terragrunt. Instead, it provides general recommendations to improve the quality of your infrastructure code.

Improving Idempotency

I often see terraform code separated by repository or branches for each environment. From my point of view, this is not the best way to maintain the idempotency of your code. No one is immune to typos or forgetting to add changes to another copy of the repository, especially when changes are made in a lower environment and then transferred to the production repository or branch.

Workspaces

Workspaces are a straightforward terraform feature that allows you to maintain separate state files with different names. This functionality makes it easier to support the idempotency of your code across different environments.

Here’s how it works in practice:

• First, create a workspace for each environment (dev, staging, production).
• To minimize future errors, I recommend incorporating the workspace name into the names of each terraform resource.

resource "google_service_account" "prometheus" {
  account id = "prometheus-${terraform.workspace}"
  display_name = "Prometheus service account in ${terraform.workspace}"
}

In this example, we will create a “service account” with a unique name for each environment (e.g., prometheus-dev, prometheus-staging, prometheus-production). Additionally, each environment’s state will be stored in separate files within our terraform remote storage.

External variables

No matter how much we strive for identical environments, there will always be resources or parameters unique to each environment (e.g., project names, cloud provider IDs, or network ranges). To address this, we can use another terraform feature: external variables.

We save all unique environment variables to different files (e.g., envs/dev.tfvars, envs/staging.tfvars, envs/production.tfvars), while the rest of the variables are stored in variables.tf as usual. As a result, we have a setup where all default variables are saved in variables.tf, and the unique variables for each environment are used as overrides.

Example of using described approach

terraform workspace select staging
terraform plan -var-file=./envs/staging.tfvars

The described approach has at least two benefits compared to having multiple branches or repositories per environment:

1. The code will execute in all environments with minimal differences. This ensures that the code in the production environment runs exactly the same way as it does in lower environments (dev, staging).
2. Since there is no need to maintain an additional branch or repository for each environment, the risk of forgetting to transfer changes between environments is eliminated.

Work with tags

This point may seem trivial, especially since it is covered in the official terraform documentation for using “locals.” However, few people actually use it. By doing so, we have the opportunity to define the common set of tags used in one place.

locals {
  labels = {
    env = terraform.workspace,
    team = "sre",
    backup = false
  }
}

and then we can use the array of labels by replacing what we need

labels = merge(local.labels, {team = "gui"})

Hierarchy of states

Regardless of the size of your infrastructure, I recommend always separating stateful resources into distinct states (e.g., DB, S3). Here are some tips on how to separate your states by tiers:

1. Network state: This forms the foundation of any infrastructure and includes VPC, subnets, network ACLs, etc.
2. Peering state: This should be in a separate state because peering connections require the approval of the destination VPC.
3. Independent resources: This state includes resources that can be deployed at any time, such as SSH keys, global IAM policies and users, and global variables that can’t be acquired from other states (e.g., repository URLs, IP addresses).
4. Data layer: Always separate this from any state because mistakes or typos can permanently remove unique data.

It’s important to ensure that the directory hierarchy of terraform corresponds with the directory hierarchy in the “remote state.” Otherwise, supporting the code becomes quite challenging, as you’ll need to locate the path to the remote state each time.

Idempotency support

Data Source using

When building infrastructure, we almost always need to use resources from other states (e.g., vpc_id, subnet ranges, service accounts when deploying a new VM). Since our infrastructure components should be divided into different states, here is a short list of rules to follow to completely reproduce infrastructure on an empty account or project:

• terraform_remote_state: Use data resources from providers first, and only then use outputs from other states.
• Unidirectional graph for building infrastructure: Use resources only from states situated lower in the hierarchy (states deployed before the current one).
• Always keep the data layer separated.
• If any manual action is needed, always separate the state into two or more parts (e.g., peering approval from another account or project).

More advice

Minimize the Number of Resources in Each Root Module

It is essential to prevent a single root configuration from growing too large, with too many resources stored in the same directory and state. All resources in a particular root configuration are refreshed every time terraform is run, which can cause slow execution if too many resources are included in a single state.

Deletion Protection

For stateful resources, such as databases, ensure that deletion protection is enabled.

Use Separate Directories for Each Application

To manage applications and projects independently, place the resources for each application and project in their own terraform directories. A service might represent a particular application or a typical service, such as shared networking. Organize all terraform code for a specific service under one directory, including sub directories.

Security

KMS Encryption

When working with cloud providers such as AWS or GCP, we have access to the “Key Management Service” (KMS). terraform can use KMS to encrypt sensitive data within the code.

This approach has its pros and cons:

Advantages:

• Granular Access Control: You can precisely control who has access to sensitive data. If an employee leaves or moves to another department, their access can be easily revoked. All access policies are managed via the cloud provider’s IAM.

Disadvantages:

• Vendor Lock-in: There are challenges when changing providers. For instance, with AWS, you can’t move KMS keys to a new AWS account. You must re-encrypt all sensitive data with a new KMS key from the new AWS account, or grant the new account access to the old KMS key (more information here: AWS Knowledge Center).
• Provider Change Issues: If you switch providers, you will need to re-encrypt all sensitive data with new KMS keys or change your approach to managing data encryption.

Mozilla SOPS encryption

https://github.com/mozilla/sops

SOPS is an acronym for “Secrets OPerationS”, built by Mozilla, and an easy way to encrypt your sensitive files.

Using SOPS with GPG

Using SOPS with GPG is a solid alternative to KMS encryption. It allows you to manage sensitive data locally with your GPG keys, eliminating the need for external providers. Additionally, it supports multiple users with different permissions.

Pros:

• No Vendor Lock-in: You maintain access regardless of the cloud provider.
• Local Management: All actions are performed locally with your GPG keys.

Cons:

• Historical Access: If data was encrypted with a GPG key that was later revoked, you could roll back in the Git history to access this data, which might pose a security risk.

Terraform State Encryption

Even if your data is encrypted in the code, remember that terraform state files store this data as plaintext. You should check if your terraform provider supports state encryption to secure this information. This is particularly important if your state files are stored locally.