Account Factory for Terraform: Lessons from the Trenches

During the last couple of years I have managed the AWS environment for one of my customers using AFT, and now its time to share my experience.

TL;DR

• Fork AFT if you deploy it with custom providers, need multi-region support, or want to implement enhanced security controls
• Watch out for the privileged access in management account — the default AWSAFTExecution and AWSAFTService roles need tightening
• AFT’s account request feature is great for GitOps-style account provisioning
• Keep AFT focused on account bootstrapping — it’s not ideal for frequently changing infrastructure
• Consider the propagation time (couple of hours for 100 accounts) when planning changes
• Global customizations work best with a modular approach and clear component separation
• Staging of changes is possible through OU-based exclusions in global customizations — test in one OU before rolling out widely
• Monitor pipeline status separately — step function success doesn’t guarantee pipeline completion

Going back about two years ago, as a Terraform workshop and consumers of AWS Control Tower, AFT was our goto for AWS accounts bootstrapping and platform infrastructure distribution. As we don’t use Terraform Cloud or Enterprise we used the publicly free option.

The Deployment

Custom Providers

The AFT Terraform module specifies [providers.tf](https://github.com/aws-ia/terraform-aws-control_tower_account_factory/blob/main/providers.tf) file which prevents you from calling the module with your own providers, because of how terraform module and providers work:

Error: Cannot override provider configuration
│
│ on main.tf line 4, in module "aft":
│ 4: aws.ct_management = aws.management
│
│ The configuration of module.aft has its own local configuration for aws.ct_management, and so it cannot accept an overridden configuration provided by the root module.

The reason to use our own providers is twofold. First, applying our own tagging strategy on the AFT deployed resources leveraging the provider’s default_tags attribute. And second, our secure terraform deployment pipeline blueprint, requires us to provision anything Terraform with it’s own IAM target roles. And thus we have to define the provider for each target account. This is why, as mentioned in this HashiCorp discussion:

Shared modules should not have provider blocks

However, since AFT module does, we had to fork it.

Required VPC and GitLab support

Back then, deploying Lambda function within a VPC was mandatory (AFT team fixed that last February 2024), and GitLab was not supported as VCS (although CodeStar connection introduced GitLab support on August 2023, AFT added it just about year later on October 2024). With those two limitations it was a no-brainer → fork was a must.

Security Scanning (PaC)

Another reason you might end up forking AFT is modifications to the global/account customizations CodeBuild projects’ buildspec. This was required for us in order to embed some verification and security scanning mechanism in the accounts pipelines.

A privileged access concern

I’m ok with cross account roles that has privileged access for such extensive automation frameworks like AFT. However, the management account is definitely an exception to that approach.

Now, if I have access to aft-management account and capable to assume the AWSAFTAdmin role (trusts any local principal) I have now got Admin on the organizations management account, by simply assuming the AWSAFTExecution or AWSAFTService role, which is an Administrator!!! WHAT??? see in the repo.

module "ct_management_exec_role" {
  source = "./admin-role"
  providers = {
    aws = aws.ct_management
  }
  trusted_entity        = aws_iam_role.aft_admin_role.arn
  aft_admin_session_arn = local.aft_admin_assumed_role_arn
}

module "ct_management_service_role" {
  source = "./service-role"
  providers = {
    aws = aws.ct_management
  }
  trusted_entity        = aws_iam_role.aft_admin_role.arn
  aft_admin_session_arn = local.aft_admin_assumed_role_arn
}

This is where you want to fork and set this role to be LEAST PRIVILEGED. In our case we reduced the AWSAFTExecution and AWSAFTService role permission in the management account, and on the side of AWSAFTAdmin we limited the trust policy to the different principals that should use it (CodeBuild projects, StepFunction, etc.). I guess that the AFT team requirement to have a dedicated aft-management account eases the severity here, but then you have to treat this account as very sensitive one, where usually Platform Eng. team will have a wide open access.

The following policies works for me:

(!) WARNING: Please test those policies thoroughly, as AFT is updated from time to time, and your environment might have different settings.

AWSAFTService permissions for management account:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AFTServicePermissions",
      "Effect": "Allow",
      "Action": [
        "cloudformation:GetTemplateSummary",
        "iam:GetRole",
        "organizations:DescribeAccount",
        "organizations:ListAccounts",
        "organizations:ListParents",
        "organizations:TagResource",
        "servicecatalog:AssociatePrincipalWithPortfolio",
        "servicecatalog:DescribeProductAsAdmin",
        "servicecatalog:DescribeProvisioningArtifact",
        "servicecatalog:GetProvisionedProductOutputs",
        "servicecatalog:ListPortfolios",
        "servicecatalog:ProvisionProduct",
        "servicecatalog:ScanProvisionedProducts",
        "servicecatalog:SearchProvisionedProducts",
        "servicecatalog:UpdateProvisionedProduct",
        "sts:AssumeRole",
        "sts:GetCallerIdentity"
      ],
      "Resource": "*"
    }
  ]
}

AWSAFTExecution permissions for management account:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "SupportedServiceSid0",
      "Effect": "Allow",
      "Action": [
        "organizations:DescribeAccount",
        "organizations:ListAccounts",
        "organizations:ListParents"
      ],
      "Resource": "*"
    }
  ]
}

Account Request is powerful!

AFT account request allows account creation as part of our git process, which allows us to leverage our native, controlled change process through git. Security wise, there is no need for Admin console access on the Control Tower (management) account when creating new accounts. And AFT handles queueing to prevent throttling of Control Tower Account Factory.

Another great option is the custom_fields which allows us to implant general metadata in form of SSM parameters in the created accounts. We are mainly use those for organization information which is not availble on the account level and might be needed by local business logic. An example is our local tagging corrective control which need to know if this is dev, uat, or prod account. Since it doesn’t have access to read the account tagging from organizations it can simply read the SSM parameter /aft/account-request/custom-fields/stage.

module "account_corp1-app-myapp-dev-01" {
  source = "./modules/aft-account-request"

  control_tower_parameters = {
    AccountEmail              = "aws-corp1+app-myapp-dev-01@corp.com"
    AccountName               = "corp1-app-myapp-dev-01"
    ManagedOrganizationalUnit = "Develop (ou-t3d6-34fi53)"
    SSOUserEmail              = "johndoe@corp.com"
    SSOUserFirstName          = "John"
    SSOUserLastName           = "Doe"
  }

  account_tags = {
    "corp:account:owner"    = "myappteam@corp.com"
    "corp:account:type"     = "app"
    "corp:account:stage"    = "dev"
    "corp:account:function" = "MyApp development account"
  }

  change_management_parameters = {
    change_requested_by = "John Doe"
    change_reason       = "Account creation"
  }

  custom_fields = {
    stage = "dev"
    ou    = "Root/Workloads/Dev"
  }

  account_customizations_name = "corp1-app-myapp-dev-01"
}

Note about control_tower_parameters: For our case I chose to disable the IAM Identity Center integration. Thus, SSOUser details are not required. However, the Service Catalog product “AWS Control Tower Account Factory” requires those parameters to be none empty strings. The workaround in this case is to provide either placeholders or the details of the person that actually pushes the request.

Thanks for these options

We found AFT options, straight-forward, simple and useful. We leverage both aft_feature_cloudtrail_data_events and aft_feature_delete_default_vpcs_enabled. It would be nice to have more granularity in terms of the CloudTrail configuration, like S3 buckets prefix name and additional data event types.

Constructing Global Customizations

The general structure of our terraform/ folder includes a modules/ folder and a configuration file per applied component.

.
├── aft-providers.jinja
├── backend.jinja
...    
---------------------------COMPONENTS
├── notifications.tf
├── alerts.tf
├── contacts.tf
├── iam-password-policy.tf
├── ebs-default-encryption.tf
├── imds-default-settings.tf
├── prevent-public-ami.tf
├── prevent-snapshot-sharing.tf
├── service-catalog-role.tf
├── custom-control-vpc.tf
├── custom-control-sg.tf
├── custom-control-instance.tf
...
---------------------------MODULES
├── modules
│   ├── thirdparty-instance-control ...
│   ├── thirdparty-vpc-control ...
│   ├── datatrail ...
│   ├── solution1-role ...
│   ├── notifications ...
│   └── service-catalog-role ...
├── data.tf
├── locals.tf
└── variables.tf

The modules and the dedicated terraform file per component make it easy to navigate the repository and incorporate changes. Each component includes calls to one or more modules/resources, and a local exclude_<component-name>boolean. The exclude_ determines whether the component is deployed or not to the account, based on list of excluded accounts and/or OUs.

Staging changes

Other than allowing exceptions to the global configuration, excluding OUs also supports staging of changes. New component can be applied on single OU, and after validation takes place, it can be promoted to higher environments. For example imds-default-settings.tf configuration:

# data.tf

data "aws_ssm_parameter" "ou_path" {
  name = "/aft/account-request/custom-fields/ou"
}

locals {
  current_ou = data.aws_ssm_parameter.ou_path.value
  current_id = data.aws_caller_identity.current.account_id
}

# imds-default-settings.tf

locals {
  exclude_ec2_imds_default_settings = contains(var.exclude_imds_default_settings.ous, local.current_ou) || contains(var.exclude_imds_default_settings.accounts, local.current_id)
}

resource "aws_ec2_instance_metadata_defaults" "this" {
  count                       = local.exclude_ec2_imds_default_settings ? 0 : 1
  http_endpoint               = "no-preference"
  http_tokens                 = "required"
  http_put_response_hop_limit = 2
  instance_metadata_tags      = "no-preference"
}

The configuration in this example aws_ec2_instance_metadata_defaults will be deployed only if the current account is not part of excluded OUs/Accounts.

Note: Considering the AWS declarative policies announcement configuring IMDS default settings from AFT doesn’t necessary make sense anymore, but that besides the point.

Account Customizations Configurations

Each account request includes an account_customizations_name field. This name corresponds to a folder in your customizations repository, and AFT will apply any Terraform configurations found in that folder to the account. In our case we were able to group the customizations according to our SDLC lifecycle OUs. A dedicated folder per: Production, UAT, Dev, and Sandbox. If there is an exception configuration which need to be deployed to a subset of accounts, a red indication will turned on and we will rethink if the right place for this configuration is AFT. If an account should be excluded from these customization we remove the account_customizations_name field from the account request.

A Quick Look at API Helpers

Think of API helpers as your Swiss Army knife in AFT — they let you run Python or Bash scripts before and after your Terraform deployments. While this capability gives you tons of flexibility, we’ve found that keeping things simple and declarative with regular Terraform configurations usually works best. Too many options isn’t always helpful.

Use Case for Account Provisioning Customizations

This very open ended mechanism embedded to AFT is yet to address any need for our platform. We did considered it few times, but ended up find an easier, simple and more consistent solution with either global or account customizations.

Rolling out

Source code versus Actual state

When rolling out changes, after testing, you want to make sure it is applied to all accounts. This prevents gaps between the source code and the applied configuration. You don’t want to discover that a pipeline for specific account fails when the last time it run successfully was 20 commits ago, which leads us to the next point.

AFT rushing no where

Propagating a change in AFT to the whole fleet of accounts, in our case ~100 AWS accounts, takes couple of hours. This is mainly because it invokes a batch of 5 pipelines for 5 accounts, and only when those complete it invokes the next batch. This is actually configurable, however, when we tried to increase the batch size we exhausted other quotas, for instance our SVC clone requests. Thus, I would consider the turn around of AFT changes to be a day or half day. My recommendation is to stay away from AFT for configurations that tend to change frequently. Also, when testing your change make sure to target a single account, and rather than starting the step function execution, just release the account pipeline directly.

Monitoring

After waiting for AFT couple of hours to complete, the step function execution seems to be GREEN, however, if you check the accounts’ pipelines themselves you might discover a failed RED pipeline for one or more accounts. The step function doesn’t wait for the pipeline to complete successfully, but rather reporting success on invocation of the pipeline. And when you have more than ~100 account to will have to put a complementary mechanism out there to indicate failed change propagation.

AWSCC provider

For some cases, especially in legacy environments, your terraform need to query previously created infrastructure in order to decide on how to deploy a resource or if to deploy it at all. [awscc](https://registry.terraform.io/providers/hashicorp/awscc/latest/docs)provider is very useful for quarrying the environment, and so, I added it to aft-providers.jinja:

provider "awscc" {
  region = "{{ provider_region }}"
  assume_role = {
    role_arn    = "{{ target_admin_role_arn }}"
  }
}

Avoid making AFT a jack-of-all-trades tool

Due to the many options AFT provides and due to the fact that it is an open source, there might be a tendency to abuse it and make it our go to for any infrastructure deployment. This approach undermine the primary aim of AFT to manage account bootstrapping. AFT propagates changes to all environment stages and the risk of deploying unverified change is there. In addition, due to the slow and vast scope of propagating changes, frequent changing infrastructure is not a good fit for AFT.

Multi Region support

AFT was not designed for multi region support! However, in reality multi-region will mostly be there, especially for high regulated environments where resiliency is a strict requirement. To support additional regions to the primary one (CT and AFT region) AFT suggests custom AWS “regional” providers, one per additional region. A good implementation example can be seen in this post by Josh Pattison.

In conclusion

After two years of using AWS Account Factory for Terraform (AFT) for AWS account bootstrapping, I’ve found it to be a powerful tool that shines when used for its primary purpose — account provisioning and baseline configuration. However, like any tool, it comes with its quirks and limitations that require careful consideration, especially around security, scalability, and change management.