Secure Generative AI RAG solutions

Today, clients are seeking assistance in designing, architecting, and developing Retrieval-Augmented Generation (RAG) solutions utilizing Azure OpenAI models, Azure AI Search, Azure Document Intelligence, and other Azure AI Services. In addition to inquiries about the differences between Copilot Studio and Azure AI Foundry, we frequently receive questions on how to secure a RAG solution for scenarios requiring high confidentiality. In this article, I aim to guide you through a layered security architecture for a hypothetical generative AI project, inspired by a real-life scenario.

Project introduction

This project will use data including Word and PDF documents containing previous research results, including tables, images, and text, are stored in Azure Storage. The chatbot, leveraging this data, must function as an active participant in meetings and offline communications, offering informed opinions on the next steps in specific research projects. The chat interactions need to be personalized and secure, with each user having access to different sets of research documents. Ensuring security and precision are the top priorities for validation.

High-level architecture

In Azure Data Lake Storage Gen2, containers are designated for each research project, with folders named by the date the files are stored. These files will first be preprocessed using Azure Document Intelligence to extract different data types from the documents, such as text, images, or tables. Then, Azure AI Search uses the data that is preprocessed by Azure Document Intelligence, and indexes them to enable the RAG solution to retrieve information effortlessly and precisely. To enhance the search relevance and performance, an effective indexing strategy, with well-defined schemas and scoring profiles will be implemented. Using Semantic Search (Azure AI Search with Semantic Kernel) will enable the solution to retrieve the exact document needed. The combination of these services can handle diverse data types, even provide images and tables with text as the generated response.

Utilizing the capabilities of Azure AI Studio, Azure OpenAI models are going to be developed, trained, and fine-tuned for the RAG solution, using the indexed data to generate accurate responses. The evaluated and validated model is then deployed, after which it is advisable to use the model endpoint in Copilot Studio to build a chatbot that can integrate seamlessly with Microsoft Teams. Chat history will be stored for each user, ensuring that users can only access the history of their own conversations with the bot. Developers will have a dedicated environment with dummy data. Each developer has access to logs, code, and other relevant information that are made available and generated during their own development session.

Base architecture for a RAG solution

How to secure this solution?

To ensure the utmost security for the project, the **Zero Trust** strategy is implemented. The approach operates on the principle that threats can originate both inside and outside the organization. Therefore, every access request, regardless of its source, must undergo through authentication, authorization, and encryption before access is granted. The implementation focuses on security by design, encompasses several aspects: data, network, access, application, and governance security.

Source: Microsoft Build 2024

AI Platform security

AI platform security focuses on safeguarding the training data used by models and AI services, while also preventing data poisoning and theft.

By leveraging Azure AI Services, we ensure that all data remains within the subscription, never used for self-training or shared with anyone else. Data is always encrypted at rest and transit, enhanced with double encryption, using Microsoft’s AES-256 encryption, and optionally, a customer-managed key. Importantly, the data processed through the pipeline is never used to train OpenAI’s foundation models or Azure OpenAI models. This includes prompts, generated responses, embeddings, or any training data used to fine-tune the solution.

Moreover, data is safeguarded by comprehensive enterprise compliance and security controls. During both inference and development, the model accesses data exclusively available to the interacting user. Additionally, the project adheres to the EU AI Act and implements security measure to align with the client’s values and policies.

Beyond data encryption, we often see the need to provide access to specific data to a limited number of users.

Utilizing Azure Data Lake Storage Gen2 enables us to manage access using access control list (ACL) for blob folders and child folders. Depending on the requirements, permissions can be granted to security principals which may represent a user, group, service, or managed identity that are defined in Microsoft Entra ID.

Azure AI Search enables us to implement document level access control, which can be set up either to determine what information should be available while generating responses using security filters, or tailor responses based on the Microsoft Entra ID user or group access permissions to certain information. Furthermore, full double encryption using customer managed keys (CMK) is highly recommended as it applies additional safety layer for sensitive content such as indexes, indexers, skillsets, and data sources.

Any data that is generated by Azure AI Search and Azure AI Studio, stored in a structure that allows additional data safety settings. For example, users have access to their own chat history, developers have access to their own service logs. Finally, it is also a good idea to maintain separate indexes considering different sensitivity levels.

AI application security

AI application security encompasses the entire lifecycle of generative AI-based applications, addressing issues such as prompt injection attacks, data leaks, and insecure plugin design.

Communication between the RAG solution, Azure AI Search, Document Intelligence, and the storages is protected using Transport Layer Security (TLS) 1.2 or 1.3. The interacting services are organized within an Azure Virtual Network (VNet), with Azure OpenAI models and the RAG solution located in the AI Subnet, and data processing with AI Search and Document Intelligence situated in the Data Subnet.

The VNet with AI and Data Subnets

To ensure that only authorized users and applications could interact with the services of the solution, several key security measures are implemented.

First, service principals should be created for Azure Document Intelligence and Azure AI Search, assigning them the necessary roles to ensure they had the minimum required permissions.

The use of private endpoints restricts access to Azure AI Search and storage resources, ensures that all traffic flowed only within the VNet. This adds an extra layer of security by keeping the data isolated from the public internet.

Role-Based Access Control (RBAC) is also a critical component to control access to Azure resources, assigning roles at the resource group or resource level. This grants users, groups, and applications the appropriate permissions to access only what they needed. Moreover, by using Microsoft Entra ID for authentication and authorization, sensitive data and services are only accessible by selected users.

Monitoring and governance is crucial to track and analyze performance and security of the solution.

• Integrate Application Insights to monitor the performance and usage of the solution.
• Purview AI Hub helps understanding data lineage and ensures compliance with data governance policies. Sensitivity labels can be applied to classify and protect data based on its label. Policies enforce organizational standards and compliance.
• Microsoft Defender with Copilot for Security detects and responses to threats across the environment, moreover, assists in analyzing security alerts and providing actionable insights. With Copilot for Security security incidents can be investigated and it helps identifying the root cause of issues.

[embed]

Microsoft Defender with Copilot for Security (Source: Microsoft Build 2024)

AI usage security

Securing user interactions with generative AI-based applications involves addressing several key risks, including sensitive information disclosure, harmful third-party LLM-based apps or plugins, and jailbreak attempts.

Users may maliciously input sensitive information into the system which could potentially lead to data leaks, or they may attempt to bypass safety protocols to elicit unauthorized behavior from the solution. Third-party applications or plugins may also introduce vulnerabilities or harmful content. Next to the use of tools such as Purview AI Hub and Microsoft Defender with Copilot for Security, additionally, content safety and abuse monitoring are crucial to implement for the solution.

Azure OpenAI service’s content safety with prompt shields are designed to safeguard generative AI systems from generating harmful or inappropriate content. Risks of user prompt and document attacks are detected and mitigated. For example, prompt shields can block prompts that attempt to provoke inappropriate responses or extract sensitive information.

Source: Microsoft Build 2024

Prompts and responses are processed within the customer-specified geography and evaluated in real time for harmful input and generated content based on a configured threshold using content filtering. Furthermore, configuring abuse monitoring enables the detection and mitigation of instances of recurring content and behaviors that may violate product terms.

Conclusion

The performance and security measure of the model must be continuously evaluated with the use of metrics such as groundedness, similarity, fluency, coherence, relevance, and whatever metrics implemented by developers. Furthermore, such evaluation highlights harmful actions of users or the solution during inferencing and development.

The security measures discussed in the article ensures an enhanced safety configuration for any RAG solution used in highly confidential use cases.

Resources

Data, privacy, and security for Azure OpenAI Service — Azure AI services | Microsoft Learn

Security Best Practices for LLM Applications in Azure

Azure OpenAI chat baseline architecture in an Azure landing zone — Azure Architecture Center | Microsoft Learn

Security Best Practices for LLM Applications in Azure (microsoft.com)

Encrypt data using customer-managed keys — Azure AI Search | Microsoft Learn

Use the Azure portal to manage ACLs in Azure Data Lake Storage — Azure Storage | Microsoft Learn

Security filters to trim results using Azure Active Directory — Azure AI Search | Microsoft Learn

Using your data with Azure OpenAI securely — Azure OpenAI | Microsoft Learn

Encrypt data using customer-managed keys — Azure AI Search | Microsoft Learn

Microsoft Copilot in Microsoft Defender — Microsoft Defender XDR | Microsoft Learn

Security overview — Azure AI Search | Microsoft Learn

Data, privacy, and security for Document Intelligence — Azure AI services | Microsoft Learn

Considerations for deploying Microsoft Purview AI Hub and data security and compliance protections for Microsoft 365 Copilot and Microsoft Copilot | Microsoft Learn

Prompt Shields in Azure AI Content Safety — Azure AI services | Microsoft Learn

Azure OpenAI Service content filtering — Azure OpenAI | Microsoft Learn

Azure OpenAI Service abuse monitoring — Azure OpenAI | Microsoft Learn