How to solve error with Values for account properties are incompatible: containerDeleteRetentionPolicy for Azure Data Lake Gen 2

What is Azure Data Lake Gen 2?

Azure Data Lake Storage Gen2 is an advanced data storage solution provided by Microsoft Azure, built specifically for big data analytics. It combines the scalability and cost-efficiency of Azure Blob Storage with the hierarchical file system features traditionally found in Hadoop Distributed File System (HDFS). This makes it suitable for modern big data and analytics workloads.

Common Use Cases

• Data Lake Architecture: For collecting, storing, and processing petabyte-scale datasets.
• Big Data Analytics: Enables batch and real-time analytics using frameworks like Spark or Hadoop.
• Machine Learning: Stores large training datasets and serves as a data source for model development.
• Data Warehousing: Works with Azure Synapse to create integrated data lake-house solutions.

Objective of this article

This article aims to offer a comprehensive, step-by-step guide to troubleshoot and resolve issues related to the containerDeleteRetentionPolicy in Azure Data Lake Storage Gen2. Readers will gain insights into the root causes of these errors, explore various solutions, and learn best practices to prevent similar issues in the future

Lets start first by upgrading our normal storage account into a Data Lake. To do this, visit the settings section and click the on Data Lake Gen2 Uprade button.

Fig 1. Upgrade button

The process consists of three essential steps:

1. Review Account Changes: Thoroughly assess the planned changes to ensure they are compatible and to prevent potential issues during the upgrade.
2. Validate the Account: Conduct a comprehensive validation to confirm that the account meets all prerequisites and is prepared for the upgrade.
3. Upgrade the Account: Carry out the upgrade process, ensuring all configurations are correctly applied and verified after the upgrade.

The error containerDeleteRetentionPolicy for Azure Data Lake Gen 2 usually occur in stage 2

Fig 2. Stage 2 error

Understand the problem

The containerDeleteRetentionPolicy property, which manages soft-delete retention for containers in Blob Storage accounts, may encounter compatibility issues when enabling Azure Data Lake Storage Gen2 features. This is because Gen2 is built on top of Blob Storage and has certain limitations regarding property support.

The solution

Option 1: Disable the containerDeleteRetentionPolicy

If the soft-delete retention policy for containers is not a critical requirement for your scenario:

1. Navigate to the Storage Account in Azure Portal:

• Go to your storage account in the Azure Portal.
• Open the Data Management section. Click on the Data Protection tab. This can also be done from the properties tab of the overview page

2. Update the Soft-Delete Policy:

• Disable the Container Soft Delete feature:
• Select Container Soft Delete.
• Turn off the policy by unchecking the box.
• Save your changes.

Option 2: Reconfigure the Storage Account for Compatibility

1. If the storage account must use Data Lake Storage Gen2, ensure the account properties align with Gen2 requirements.
2. Check all settings and ensure no conflicting properties (e.g., lifecycle management rules or retention policies incompatible with Gen2).

In this example, we will go with option 1, let’s disable soft delete either of the options shown can be used to disable soft delete

Fig 3. 1. From Data management, click data protect

Fig 3.2. From overview, click properties and click on enabled

Lets focus Fig 3.2, we can clearly see that when the soft delete box is unchecked, soft delete gets disabled. Click save and thats it

Fig 3.3 After saving

We are now ready to upgrade our storage account without any errors. Just follow all the steps as before

Fig 4. There is no more error

If we move to the final step, we can see that we successfully upgraded

Fig 5. Data Lake Gen2 upgrade successeded.

Conclusion

By resolving the containerDeleteRetentionPolicy error, you can achieve seamless integration and functionality with Azure Data Lake Storage Gen2. This process underscores the importance of understanding the compatibility nuances between Blob Storage and Data Lake Gen2 features. By following the outlined steps, you can prevent similar issues in the future and confidently harness the full potential of Azure’s advanced storage solutions for your data analytics and storage needs.