The Governance Problem Hiding Inside Generative Search

A decade ago, search was simple: you typed a query, you got a list of links, you chose where to go. Today, search is everywhere — and increasingly, the decision of what you see is made before you arrive.

Generative search engines now retrieve sources, select what’s relevant, discard the rest, and produce a single synthesized answer. According to Semrush research, informational queries dominated AI Overviews in 2024, accounting for nearly 90% of triggered searches — though that share has since declined significantly as platforms expand generative search to commercial and transactional intent.

That makes generative search engines something new: not a directory, but an editorial actor. They decide which sources count, which voices are heard, and how information is framed. Yet the accountability for these decisions remains almost entirely undefined. Who is responsible when the answer misleads? When does curation become manipulation? Who audits the systems shaping what millions of people believe?

Not the Model — the System

EU AI Act Recital 97 draws a distinction that matters: a general-purpose AI model is not the same thing as an AI system. The model — GPT, Claude, Gemini, Llama — is the underlying capability layer. It becomes a system only when combined with an interface, a search tool, an API integration, or a platform feature that puts it in front of users.

In simple terms: the model is the engine. The AI system is the car built around it.

It’s tempting to focus accountability debates on the model itself — is GPT safe? Is Gemini biased? But what users actually interact with is the system built around the model: the interface that decides which sources to retrieve, how to rank them, how to summarize them, and what to leave out.

This distinction matters because the most consequential decisions happen at the system level, not the model level. A model can be audited for bias — but who audits the retrieval logic? Who is responsible for the choice to cite one source over another? These are system-level questions, and current governance frameworks are only beginning to catch up.

Editorial Power, Editorial Responsibility

Recital 119 of the EU AI Act marks a conceptual shift. AI systems, it acknowledges, are no longer just software tools — some now function as information gateways. They retrieve, select, combine, and deliver a single answer to the user. In doing so, they may simultaneously fall under multiple regulatory frameworks: the AI Act and the Digital Services Act.

But regulation follows slowly behind reality. The deeper question Recital 119 surfaces is this: when an AI system shapes what people see, believe, or decide — who is responsible for that?

A generative search engine is not neutral in practice. Every answer it produces reflects a series of decisions: which sources to retrieve, which to ignore, how to frame a summary, whether to signal uncertainty, whether to warn the user. In high-stakes contexts — medical information, financial advice, electoral content — these decisions carry real consequences.

Where Does Curation End and Manipulation Begin?

EU AI Act Article 5 prohibits AI systems that deploy subliminal techniques or purposefully manipulative and deceptive methods, where the objective or effect is to materially distort a person’s behaviour by impairing their ability to make an informed decision — causing them to take a decision they would not have otherwise taken, in a way that causes significant harm.

On paper, this seems clear. In practice, applied to generative search, the line becomes remarkably difficult to draw.

When a generative search engine summarizes a health query and surfaces one treatment approach over another — without disclosing which sources were selected or why — is that curation or distortion? When a financial query returns an AI-generated answer that happens to favor products with higher affiliate value to the platform, does that cross into manipulation? The Act’s threshold requires demonstrating intent or measurable harm, neither of which is easy to prove inside a black-box retrieval system.

Article 5(1)(b) extends the prohibition further: AI systems that exploit vulnerabilities related to age, disability, or specific social and economic situations — where the effect materially distorts behavior in a harmful way — are also prohibited. This is where generative search becomes particularly concerning. Elderly users searching for medical information, financially vulnerable users seeking advice on debt or savings, or users in crisis searching for mental health resources — all of these are high-stakes interactions where a miscalibrated summary carries real consequence. The system doesn’t need to intend harm. Under Article 5(b), the effect is sufficient.

Yet, to date, there appears to be no publicly available enforcement decision testing whether a generative search engine falls within the AI Act’s prohibitions on subliminal, manipulative, or deceptive techniques. The Act entered into force in August 2024, while its prohibitions began applying in February 2025. The European Commission has published guidelines on prohibited AI practices, including concepts such as “subliminal techniques,” “material distortion,” and “significant harm” — yet their application to AI-powered information retrieval and generative search systems remains largely untested in enforcement practice or court interpretation. This is not a minor gap — it is one of the governance questions that may shape how AI-powered information systems are regulated in the coming decade.

Transparency Obligations — and the Gap They Reveal

Recitals 136 and 120 of the EU AI Act establish a clear principle: AI-generated content must be detectable and disclosed, and very large platforms with AI-driven retrieval systems have active duties to identify and mitigate systemic risks. In plain terms — when an AI system shapes what a user sees and believes, that process cannot remain invisible.

Applied to generative search, this raises an immediate question: how much disclosure is actually required, and how much already exists?

There is a reasonable objection here, and it deserves a direct answer. SEO and GEO practitioners have spent years learning — through testing, documentation, and Google’s own public guidance — that visibility in AI-powered search depends on topical authority, brand recognition, crawlability, and demonstrated expertise. Platforms have published guidelines, documentation, and best practices. Doesn’t this already count as transparency?

It counts as something — but not transparency in the sense that EU law is beginning to require.

What platforms have disclosed is an optimisation framework: the inputs that make a source more likely to be retrieved. Think of it as a rulebook for getting in the door. What remains entirely opaque is what happens once the door closes.

Because transparency here is not one question. It is at least three.

The first is the one most users already understand: was this answer generated by AI? This layer is largely resolved and sits at the baseline of what Recital 136 requires.

The second is source selection: which sources were retrieved for this query, which were discarded, and on what basis? A generative search engine may surface citations, but it does not explain the selection logic. The difference between “here are three sources” and “here is why these three and not the other eight hundred” is precisely the difference between the appearance of transparency and transparency itself.

The third is representational fidelity: does the generated summary accurately reflect what its sources actually say? Summarisation involves choices — which details to retain, what tone to adopt, how to handle conflicting sources. These are editorial decisions, currently made invisibly. A source can be cited and still be misrepresented.

A newspaper that tells journalists how to write a good article has not thereby explained why a specific story ran on the front page. Platforms have disclosed enough to allow content producers to compete for visibility. They have not disclosed enough to allow users, regulators, or civil society to evaluate whether the information those users receive is accurate, balanced, or free from subtle distortion. Under the EU AI Act, that second category of disclosure is what the Act is now beginning to demand.

Who Is Responsible?

When a generative search engine returns a misleading answer — one that misrepresents its sources, omits critical context, or frames information in a way that distorts a user’s decision — the question of accountability becomes surprisingly difficult to answer.

Is it the model provider? The company that trained the underlying AI system made choices about what data to include, how to handle conflicting information, and what outputs to reward. Those choices shape every answer the system produces.

Is it the platform deploying it? Google, Microsoft, or Perplexity built the search system, designed the retrieval logic, chose which sources to index, and decided how summaries are presented to users. The editorial architecture is theirs.

Is it the content source? The third-party website, review platform, or forum thread that was retrieved and summarised did not consent to being reinterpreted. Its original meaning may not survive the summarisation intact.

Under current frameworks, the honest answer is: all of them, and therefore effectively none of them. Responsibility is distributed across a chain of actors — model providers, platform deployers, content sources — in a way that makes it nearly impossible to assign accountability for any specific harmful output.

Recital 119 of the EU AI Act begins to address this by recognising that AI systems functioning as information gateways may simultaneously fall under multiple regulatory frameworks, including both the AI Act and the Digital Services Act. This layered jurisdiction is not a bug — it is a deliberate acknowledgment that no single framework can capture the full accountability chain alone.

But acknowledging the problem is not the same as solving it. Who investigates when a generative search answer causes harm? Which authority has jurisdiction — the AI Office, national data protection authorities, or DSA enforcement bodies? How is harm even measured when it manifests as a gradual distortion of what millions of users believe to be true?

These are the questions governance is now trying to answer. The lines are being drawn — but they are not drawn yet.

Why Governance Matters Here

The questions raised in this article are not primarily technical ones. They do not require a better algorithm or a more powerful model. They require decisions about accountability, transparency, and the distribution of power over information.

Generative search systems are now among the most consequential editorial actors in the world — shaping what millions of people understand about health, finance, politics, and the decisions they make every day. That scale of influence has always, in other contexts, come with oversight. Broadcasters are regulated. Publishers are liable. Platforms are increasingly subject to the Digital Services Act.

The EU AI Act does not solve this problem. But it begins to name it — and naming it is where governance always starts. The prohibited practices of Article 5, the accountability questions of Recital 119, the transparency obligations of Recitals 136 and 120 together form the outline of a framework that does not yet exist in full.

This is not a reason for pessimism. It is a reason for the kind of work that sits at the intersection of law, philosophy, and technology — the work of drawing the lines before the harm becomes irreversible.

References

EU AI Act, Article 5 — artificialintelligenceact.eu/article/5/

EU AI Act, Recitals 97, 119, 120, 136 — artificialintelligenceact.eu

Semrush AI Overviews Study — semrush.com/blog/semrush-ai-overviews-study/

European Commission — Guidelines on Prohibited AI Practices (AI Act) — digital-strategy.ec.europa.eu/en/library/commission-publishes-guidelines-prohibited-artificial-intelligence-ai-practices-defined-ai-act