Privacy Law Still Applies to Generative AI

Generative AI interfaces feel personal and conversational, which can create a sense of privacy that does not reflect how data is actually handled behind the scenes. While many providers have introduced safeguards such as retention limits, training restrictions, and enterprise environments, most data practices remain outside the public view, including training sources, internal access controls, and model evaluation processes. In practice, users and organizations often rely on contractual terms, policy statements, and regulatory oversight rather than independent verification. This creates an accountability gap in which trust depends largely on the provider’s internal governance rather than on transparency that users can directly assess. There remains a blind spot between what users see and what they can verify, but what is being done about it?

Innovation First, Regulation Later

Even though we navigated this privacy-for-tech inadequacy in the 1990s when the internet went mainstream, AI stakeholders are working hard to convince anyone who will listen that regulating AI is a ridiculous idea that will harm innovation and, what? Ruin humanity, maybe? When it comes to our privacy, today’s generative AI systems are advancing faster than the legal frameworks designed to govern the use of personal data, but those frameworks still apply to AI and new regulations will add to them.

Generative AI isn’t new as a concept; researchers have been building content-generating systems for decades, but large-scale transformer models capable of producing human-quality text and images only emerged around 2017, with mass public adoption beginning in 2022. On the regulation side, Europe’s General Data Protection Regulation (GDPR) became applicable in 2018, and California’s Privacy Rights Act (CPRA) was approved in 2020, with most of its obligations taking effect in 2023. These laws did not anticipate AI systems capable of generating content and interacting with people in highly persuasive ways. Nevertheless, privacy laws regulate what happens to people’s data, so they still apply to the entirety of AI systems, from training and inference to logging and monitoring.

Artificial intelligence is inseparable from data. Models are trained on data, refined through data, prompted with data, and evaluated using data. Even systems that do not store traditional records can generate inferences, embeddings, and outputs that reveal personal information indirectly. Under most privacy frameworks, activities such as collecting training data, reusing historical datasets, processing prompts that contain personal information, storing logs, and generating outputs about identifiable individuals constitute “processing” when personal data is involved. If personal data is processed at any stage of an AI system’s lifecycle, privacy obligations are triggered, whether the system is experimental, internal, or customer-facing.

Organizations cannot treat AI as a neutral layer sitting outside their privacy obligations. Once AI is introduced, privacy risk expands beyond databases and into model behavior itself. “Where data is stored?” is now followed by, “And how is it transformed, inferred, and reproduced by automated systems?”

How GDPR Interacts With AI Systems

GDPR’s principles map closely to the realities of AI, which is why GDPR has become one of the most influential regulatory frameworks shaping AI governance worldwide.

Lawful basis remains foundational. Before personal data can be used in an AI system, an organization must identify a valid legal basis for processing. AI initiatives often create risk when data collected for one purpose is repurposed for training or fine-tuning without reassessing whether that use is permitted.

Transparency creates another pressure point. Where automated decision-making produces legal or similarly significant effects, GDPR requires organizations to provide meaningful information about the logic involved and the potential consequences. Not all AI use triggers these requirements, but systems that make or materially influence important decisions must meet higher disclosure and safeguard standards.

Purpose limitation is frequently overlooked. Data collected for customer support, analytics, or internal operations cannot automatically be reused for AI training unless the new use is compatible with the original purpose or another lawful basis applies.

Data minimization also becomes more demanding in an AI context. Collecting large datasets solely to improve performance can be difficult to justify unless the organization can demonstrate that the personal data is necessary and proportionate to the system’s purpose.

Data subject rights introduce operational challenges. Individuals retain the right to access, correct, restrict, or erase their data. When personal data has been used to train or influence a model, organizations must still provide a meaningful response, even where deletion or isolation of model influence is technically challenging.

Finally, Data Protection Impact Assessments (DPIAs) become central. Under GDPR Article 35, systems involving profiling, large-scale processing, sensitive data, or decisions with significant effects typically require a DPIA. Many AI systems meet these thresholds. Treating DPIAs as optional or delaying them until after deployment increases enforcement risk.

The goal for technological innovation should always be to ensure that new capabilities are introduced within a framework that leadership can understand, audit, and defend.

How CPRA Applies to AI in Practice

California’s CPRA approaches privacy through a consumer protection framework, but its implications for AI are equally significant. The law places particular emphasis on sensitive personal information, including precise geolocation, financial information, biometric data, and certain inferred characteristics commonly implicated by AI systems. CPRA also authorizes new rights related to certain automated decision-making uses, with specific requirements being implemented through California Privacy Protection Agency (CPPA) regulations and phased compliance timelines. Organizations using AI for employment, credit, housing, eligibility, or similar high-impact decisions should expect increasing disclosure, access, and control obligations.

CPRA’s notice and transparency requirements apply when personal information is used in AI systems, including clear disclosure of what information is collected, how it is used, and how individuals can exercise their rights. Proposed CPPA regulations would also introduce risk assessment obligations for high-risk automated decision-making uses, signaling a shift toward more formal oversight of algorithmic risk.

The Unique Privacy Risks AI Introduces

AI introduces privacy risks that traditional software rarely did. Training data may surface unexpectedly through generated outputs, even when models are not designed to retrieve records. Seemingly harmless prompts can trigger memorized fragments or inferred personal information.

Inference risk is particularly complex. AI systems can infer sensitive traits such as health status, political beliefs, or protected characteristics from indirect signals. These inferred attributes may themselves qualify as personal data and, in some cases, sensitive data under privacy law.

Prompt leakage is another common issue. Employees often paste customer data, internal documents, or confidential information into AI tools without understanding how that data may be stored, processed, or reused. Outputs can also expose private or proprietary information without malicious intent. Privacy leakage often emerges from normal model behavior rather than deliberate misuse.

Finally, explainability creates a structural tension. Privacy law demands meaningful information about automated decisions, while many AI systems operate as probabilistic models. Bridging that gap is one of the central governance challenges of modern AI.

How Organizations Can Reduce AI Privacy Risk

Reducing privacy risk in AI systems begins with restraint rather than sophistication. Sensitive data should generally not be entered into public or consumer-grade AI tools unless appropriate contractual and technical safeguards are in place. For regulated or high-risk data, organizations should use private or enterprise environments with appropriate security and governance controls.

High-impact AI systems should undergo privacy impact assessments before deployment. Privacy notices should clearly disclose AI use and explain individual rights in plain language. Access controls should extend beyond traditional applications to include prompts, outputs, and model configuration. Vendor assessments should evaluate training practices, retention policies, subprocessors, and safeguards with the same rigor applied to other high-risk processors.

Employee training remains one of the most effective controls, as many real-world AI privacy incidents result from misunderstanding rather than malicious intent. Privacy controls should be applied using a risk-based approach, with governance, documentation, and safeguards scaled to the sensitivity of the data and the potential impact on individuals. Organizations should maintain documentation demonstrating lawful basis, data sources, risk assessments, vendor due diligence, and decision-making rationale. In practice, enforcement risk often depends less on whether problems occur and more on whether the organization can show that risks were evaluated and managed.

Effective AI privacy management also requires defined ownership across legal, privacy, security, and product teams, with clear escalation paths for high-risk use cases.

Understand, Audit, Defend

Privacy law did not anticipate generative AI, but it did anticipate systems that process personal data at scale, influence people’s lives, and operate in ways that are difficult to see or challenge. This is why GDPR and CPRA apply so directly to AI systems today. Regulators are increasingly focusing enforcement on data governance failures rather than the use of AI itself. In most cases, legal risk arises from unclear data sourcing, secondary use without reassessment, inadequate transparency, or weak vendor controls.

Organizations that manage AI risk successfully treat privacy as part of their governance architecture, not a standalone compliance task. That means inventorying AI use cases, classifying them by data sensitivity and impact, documenting data sources and lawful basis, conducting risk assessments before deployment, and establishing ongoing monitoring as systems evolve.

Better-defined AI regulations are on the digital horizon, and it just makes sense to follow current law and regulations in the meantime. Keeping in mind the recent court rulings against LLM builders for copyright violations as well as the hefty settlements they now have to pay, the goal for technological innovation should always be to ensure that new capabilities are introduced within a framework that leadership can understand, audit, and defend.

[embed]Who Gets to Write the Rules for AI? The power struggle behind the future of AI regulationmedium.com

[embed]AI Governance Is Where the Internet Was in 1998 What Organizations Can Learn from the Pastmedium.com