The key to the problem — journey of Azure OpenAI with Customer Managed Key

Hi there,

If you’re reading this, chances are you’ve been googling and debugging for a while, trying to figure out “how to deploy Azure OpenAI with Customer Managed Key (CMK) using Terraform”. Maybe you’ve even managed to deploy it before, only to find it failing now. Little did you know, there were things happening behind the scenes at Microsoft that were causing these issues. I was fortunate enough to get in touch with an amazing Microsoft support engineer who guided me through this journey over the past few weeks. Now, I’m excited to share my story with you. But if you’re only interested in the solution, feel free to jump to the Solution section.

Journey

Let me start at the beginning…

After extensive exchanges with the Enterprise Solution Architect, Security team and even an external review, the team got the green light: We can start the pilot testing of the developed GenAI solution built around the Azure OpenAI resource. But then, disaster struck. The deployment failed. Impossible! I had successfully managed it only a few weeks ago without any issues. Now, the same Terraform code / deployment is timing out due to the requested addition of a CMK to the Azure OpenAI resource.

I started running in circles, debugging furiously. I vividly remember battling with the organization’s Azure Policies, especially the one that stated:

• Cognitive Services accounts should enable data encryption with a customer-managed key — denying the creation of an Azure OpenAI resource without a CMK.

Back then, this led me to the solution of using ‘azurerm_cognitive_account’ resource with a ‘customer_managed_key’ and ‘identity’ based on the Terraform Azurerm Documentation. [The user assigned identity needs to have the right permissions prior to the provisioning of the Azure OpenAI resource]

resource "azurerm_cognitive_account" "example" {
  ...

  customer_managed_key {
    key_vault_key_id = azurerm_key_vault_key.example.id
    identity_client_id = azurerm_user_assigned_identity.example.id
  }

  identity {
    type         = "UserAssigned"
    identity_ids = [azurerm_user_assigned_identity.example.id]
  }
}

Everything looked the same, nothing changed. With no other options, I had to open a Microsoft Support ticket while admitting the defeat.

After a few exchanges with various support members from Microsoft, my ticket landed on the desk of, if not the best, then certainly one of the most knowledgeable support engineers. He understood my situation perfectly and shared the following insights:

1. To comply with the policies, the user assigned identity was a good approach, but Microsoft does not support it with Azure OpenAI. This feature was intermittently available, which explained why I could deploy the resource initially.
2. System assigned identity with the right permissions to the CMK is required (and only supported) to add CMK to the Azure OpenAI resource. This meant that the Azurerm Documentation and example were (still are) incorrect — as it uses of user assigned identity for adding the CMK.

Okay, but — “ What about the policy of Cognitive Services accounts should enable data encryption with a customer-managed key — with Deny action” — I asked.

“The system assigned identity is created along the Azure OpenAI resource and thus it won’t have yet permissions to CMK. Consequently the CMK can not be added to the resource and thus the policy denies its creation. That’s why the user assigned identity was the only solution to compile with the policy”

Not much could be done. At the end, the organization had to change the policy’s specific Deny action to Audit, hoping this would resolve the issue.

We were wrong. It turned out there was a general problem with the CMK assignment to the Azure OpenAI resource, which took almost two weeks to resolve on Microsoft’s side.

Nevertheless, we now have a working solution, and the pilot study is running within the organization.

Solution

1. Adding CMK to the Azure OpenAI service: This is only possible through a Microsoft-managed identity with the right permissions to the key.
2. Policy Adjustment: The built-in policy “Cognitive Services accounts should enable data encryption with a customer-managed key” can only be used with the action set to Audit. There is no way to provision a resource with the system assigned identity with the right permissions within one Terraform resource (nor via the portal).
3. Terraform configuration: Terraform cannot resolve the correct provisioning sequence. The ‘depends_on’ meta-argument is required to ensure that the system assigned identity has the right permissions before adding the CMK to the Azure OpenAI resource. In subsequent deployments, Terraform recognizes the added CMK as a ‘custom_managed_key’ block within the ‘azurerm_cognitive_account’ and thus it will try to remove it (unsuccessfully). Add the block of ‘lifecycle’ with ‘ignore_changes = [custom_managed_key]’ to suppress the unwanted changes.

resource "azurerm_cognitive_account" "example" {
  ...

  identity {
    type         = "SystemAssigned"
  }

  lifecycle {
    ignore_changes = [custom_managed_key]  
  }
}

resource "azurerm_cognitive_account_customer_managed_key" "example" {
  cognitive_account_id = azurerm_cognitive_account.example.id
  key_vault_key_id     = azurerm_key_vault_key.example.id
  identity_client_id   = azurerm_cognitive_account.example.identity[0].tenant_id

  depends_on = [
    azurerm_cognitive_account.example,
    azurerm_key_vault_key.example
    azurerm_role_assignment.openai_system_managed_identity
  ]
}

Remarks

1. The Terraform example does work even if Terraform times out, but not what you think. In the example, both the system and user assigned identity (SID/UID) get the same permissions to the key vault but the CMK added to the Azure OpenAI resource using the UID. When only the UID gets permission to the key vault (but not the SID), the CMK can not be added using the UID to the Azure OpenAI resource. It looks like that the CMK assignment internally checks for the required permissions and somehow resolves it even if the assignment is made using the UID (by implicitly taking the SID as it has the same permissions).
2. Use built-in policies controlling resource provisioning with considerations. Built-in policies allowing for Deny action can result in prohibiting resource creation, since certain Terraform resources do not support the creation of the resource the way how the policy would enforce it. As a rule of thumb, the sequence of resource creation and configuration within the Azure Portal should be followed using the Terraform resources, and policies should be only set to Audit if they block resource creation when initial conditions cannot be met.
3. The Azure documentation should be considered the source of truth, as other parties may fall behind in updating their documentation and still refer to legacy solutions.
4. Do not remove the permissions of the Microsoft-managed identity before removing the CMK from the resource. You will end up with a resource in a pending state and won’t be able to delete it until the action times out.