Photo by NEOM on Unsplash

We keep auditing what AI does. We should be auditing how it came to be.

“Knowledge is not made for understanding; it is made for cutting.” — Michel Foucault, Nietzsche, Genealogy, History (1971)

There is a growing movement to make AI systems transparent. Governments are building registers. Researchers are designing trust frameworks. Advocacy groups are demanding that the public be told what algorithmic systems are doing in their name, with their data, on their lives.

This is good work. It matters. And it is not enough. Even here in Canada there are now government agencies collating AI registers towards the goal of transparency in government use of technology.

[embed]Government of Canada AI Register (Minimum Viable Product) - Open Government Portal The Government of Canada (GC) Artificial Intelligence (AI) Register collects information about the Government of…open.canada.ca

But here is the problem. Most of the transparency tools we have, registers, impact assessments, fairness audits, operate at the surface. They ask: What does this system do? Who is accountable? What kind of decision is it making? What data was used to train it?

These are the right questions to start with. But they are the starting line, not the finish. They describe the building. They do not examine the foundation.

The missing layer: genealogy

A training dataset is not a neutral input. It is a historical artifact. It carries the conditions of its own production: who collected it, from whom, when, for what purpose, under what structures of consent or coercion, and through what power relations. Every dataset has a backstory. Most audits never ask for it.

[embed]The man and his typewriter in Bonn that shaped how the world talks Whose hand is on the things we call "standards"kemlaurin.substack.com

Think about what happens when an institution digitizes its records. A child welfare agency converts decades of case files into structured data. A police force feeds years of arrest records into an analytics platform. A health system migrates patient histories into an AI-powered triage tool. In each case, the biases embedded in the analogue system, who got flagged, who got overlooked, which communities were over-policed, which patients were undertreated, do not disappear in the conversion. They are encoded. And then they are scaled.

What was once one caseworker’s discretion becomes a province-wide standard automation. What was once a local pattern of over-policing becomes a national predictive model.

The isms of the analogue world, racism, sexism, ableism, colonialism, become the default logic of the digital one, operating at a speed, reach, and institutional authority that their originators never imagined.

[embed]They dug through us: the case for an autopsy of big tech and the data they took An urgent call for excavation, accountability, and new lawkemlaurin.substack.com

This is what I mean by genealogy: the full chain of custody from the moment data was first collected to the moment it shapes a decision about someone’s life. Without it, transparency is a façade. You are reading the label on a building, not inspecting what it is made of.

Who bears the weight

This is not an abstract concern. It lands on specific people.

Racialized communities whose historical over-representation in criminal justice data becomes the training set for predictive policing tools. Indigenous peoples whose information is extracted without adherence to data sovereignty principles. Women whose career trajectories, health outcomes, and creditworthiness are scored by systems trained on decades of patriarchal institutional practice. Disabled people whose lives are measured against normative baselines they were never consulted about. Migrants whose movements are tracked by platforms built on enforcement data they never consented to provide.

[embed]

These are not edge cases. These are the populations most exposed to the consequences of AI, and least represented in its design. When we audit an AI system and find that it performs well “on average,” we are telling a story that centres the majority and erases the margins. Aggregate accuracy is not justice. A system that works beautifully for most people and catastrophically for the most vulnerable is not a fair system with a few bugs. It is an unfair system with good marketing.

Any audit worthy of the name has to ask not just “does this system work?” but “for whom does it work, and at whose expense?”

Values are not an add-on

There is a deeper issue that the technology industry has been remarkably slow to confront. Values are not something you layer on top of a technical system after it has been built. They are in the architecture from the start. Every design decision, from the choice of training data to the selection of an optimization target to the granularity of the categories used to classify a human being, embeds a set of assumptions about what matters, what counts as relevant, and what can be safely ignored.

When a system is optimized for efficiency, that is a value. When it scores people by risk, the definition of risk is a value. When it classifies individuals into categories, the taxonomy is a value. When it decides what to measure and what to leave out, the boundary between included and excluded is a value. These choices are not neutral. They are ideological in the precise sense: they reflect a particular way of seeing the world, one that is no less partial and contestable for being expressed in code rather than in prose.

And yet, in most procurement processes, these choices are treated as technical specifications. They are made by engineers and product managers, reviewed (if at all) by internal ethics boards with advisory rather than binding authority, and deployed on populations who had no voice in the design. The affected communities, the people whose lives the system will characterize, score, and sort, are typically the last to know and the least empowered to contest.

This is the conversation I have been trying to surface for six years. Not “how do we make AI more ethical?” as though ethics were a feature to be bolted on. But “how do we think about the value systems we are already embedding?” Because by the time anyone writes a register entry, the values are already there. They were encoded in the first line of data collection. They were sealed in the moment someone chose what to optimize for. The question is whether we surface them, name them, and create the conditions under which they can be contested, or whether we let them operate in silence.

The orientation is right; the depth is not yet sufficient

I want to be clear here: the people building AI transparency tools, registers, trust frameworks, public-facing accountability structures, are doing important and often thankless work. The direction is right. We are moving, slowly, toward a world in which the public can ask meaningful questions about the algorithmic systems that shape their lives.

But we are still operating mostly at the surface. We are asking what systems do. We are not yet asking, with the same rigour and the same public expectation, how they came to be. We are not tracing the genealogy of the data. We are not examining whether the analogue biases of the last century have been digitized and amplified into the infrastructure of this one. We are not centering the communities who bear the greatest risk. And we are not confronting the philosophical question that sits underneath all of it: by what right does any system characterize a person at all?

That is the next step. Not a replacement for what exists, but the layer beneath it, the archaeology of the thing rather than its surface.

Archaeology is not only a metaphor

And here the metaphor stops being only a metaphor. There is already a discipline that does this work. Digital archaeology, the field that brings 3D modelling, LiDAR, photogrammetry, and geographic information systems to the study of the human past, exists to read a site without destroying it and to reconstruct what time and damage have left in fragments. Its practitioners do not treat the artifact as a neutral object. They treat it as something deposited at a particular moment, by particular people, under particular conditions, and they record where it came from as carefully as the thing itself.

Provenance is not metadata. It is part of what the object is.

That is the orientation I am asking us to bring to data. A training dataset is a site. It has layers, each one laid down by an institution at a moment in its own history. It has a chain of custody that can be traced or lost. And like any excavation, the way you handle it can either preserve what it has to tell you or quietly destroy it. The archaeologists know this better than the engineers do: their own field warns that as the instruments grow more powerful, data piles up faster than anyone’s ability to say what it means or where it belongs, and that more sophisticated technique can widen the margin of error rather than close it. That caution is almost entirely absent from how we build and audit AI.

[embed]What did the AI know and when: interrogating the AI model 5 questions & 5 reason — dissecting AI’s knowledge basekemlaurin.medium.com

The parallel runs deeper than method. Archaeology has had to reckon, late and imperfectly, with whose heritage it was excavating and on whose authority, with the difference between studying a people and extracting from them. Those are the same questions raised by Indigenous data sovereignty, by communities whose records were taken without consent and now sit inside a training set. The discipline that finally learned to ask “by what right do we dig here, and to whom does this belong?” has something to teach the one that has not yet learned to ask it of data.

Toward the framework

I have been working on a framework that attempts to operationalize these questions into something a practitioner can actually use in the field. I will share more about it soon. But the argument does not depend on any particular tool. It depends on a shift in orientation: from describing what AI systems do to excavating how they came to be, and for whom.

The seeing stones are already here. The question is whether we demand to see what they are made of.

Kem-Laurin Lubin, PhD, is a researcher in computational rhetoric and AI in Design at the University of Waterloo, founder of the Human Futures AI Training Institute, and author of Design Heuristics for Emerging Technologies: AI, Data, & Human-Centered Futures; Considerations for the Rights of Women. Her work focuses on how AI characterizes humans and the material consequences for marginalized communities.