Threat Modelling GenAI Applications: The Questions Nobody Is Asking

There’s an alarming problem that goes beyond credentials and identity. As organisations deploy business-critical GenAI applications in the cloud, they often overlook an entire threat model.

The questions every team building these applications should be asking:

Question 1: Who has access to the prompts and responses?

This is the data flow question that gets overlooked because everyone’s excited about the AI capabilities. When a user prompts your GenAI application, that prompt, and the model’s response, passes through multiple systems. Who can see it?

The model provider. If you’re calling an external API (OpenAI, Anthropic, Google), your prompts and responses traverse their infrastructure. Are you allowing model providers access to sensitive data? Most enterprise API agreements include data processing terms, but the default settings vary. Some providers use API data for model training unless you opt out. Others don’t. Do you know which your provider does?

Your own infrastructure. Prompts and responses are logged, cached, and stored across your cloud environment, in API gateway logs, application logs, observability platforms, and potentially in training data pipelines. Who has read access to those logs? A developer with CloudWatch access can read every prompt your users sent to your AI application.

Third-party integrations. If your GenAI application uses RAG (retrieval-augmented generation), MCP tools, or plugin architectures, prompts may be forwarded to third-party services for context enrichment. Each hop is another point where sensitive data can be observed, logged, or retained.

Question 2: Does the model have access to data the prompter shouldn’t see?

This is the privilege escalation question hiding in plain sight, and it breaks down into two critical sub-questions:

2a: Can the LLM access data that the prompter does not have authorisation to see?

This is the “confused deputy” problem applied to data retrieval. Consider: your customer support agent has RAG access to the entire knowledge base, including internal engineering documents, HR policies, and financial reports. When a customer asks a question, the LLM searches across all of that data to formulate a response. If the retrieval system doesn’t enforce the user’s access level, only the agent’s access level, the LLM can surface information the customer should never see.

This isn’t hypothetical. In the 2024 Slack AI incident, researchers demonstrated that indirect prompt injection in private channels could cause the AI to summarise sensitive conversations and expose them to users who didn’t have access to those channels. The AI had access; the user didn’t. The AI bridged that gap.

2b: Can the model return information that the prompter should not know?

Even when the model doesn’t directly expose raw data, it can infer and synthesise information that reveals sensitive patterns. A financial analyst asks: “What were the main themes in executive emails this quarter?” If the model has access to executive communications, it might summarise strategic plans, M&A discussions, or personnel decisions that the analyst has no authorisation to know about.

The model doesn’t think of this as a security violation. It’s doing exactly what it was asked to do, summarising information it has access to. The violation is architectural: the model’s read permissions exceed the user’s read permissions, and no authorisation layer mediates the gap.

The threat model canvas for GenAI in cloud environments

Every GenAI application your organisation builds should be evaluated against this framework:

Data flow mapping. Trace every prompt from user input to model response and back, including every system that touches, logs, or caches the data. Document which entities (human, machine, model provider) can observe each hop.

Permission inheritance audit. For every data source the model can access (RAG indices, databases, APIs, file systems), verify that the user’s authorisation level, not just the agent’s, is enforced at retrieval time. This requires row-level, document-level, or attribute-level access control in your retrieval layer.

Output classification. Before model responses reach the user, classify the output for sensitivity. Does it contain PII, financial data, intellectual property, or information from restricted data sources? If yes, does the user have authorisation to see it?

Provider trust boundaries. For each model provider in your stack, document: data retention policies, training data usage, access controls on their side, geographic data residency, and incident notification commitments. These are your trust boundaries, enforce them.

Side-channel analysis. Even when direct data exposure is controlled, consider what an attacker could infer from patterns of model behaviour, response times, refusal patterns, differential outputs that reveal whether certain data exists.

GenAI Threat Model Canvas