AI for Everybody — Lesson 4

What Is This Thing? The Three Ingredients: Data, Compute, Algorithms

Image by Grace Ann Hansen using NANO BANANA 2

Last week we walked the seventy-year history. Each winter happened in part since one ingredient was missing. Each renaissance happened in part since all three lined up at once. This week we name the ingredients.

There are three of them. Data. Compute. Algorithms. Every AI system you have ever used was built from those three things. Every wave you read about in Lesson 3, including the wave we are currently in, came from a moment when all three were available at the same time at the right scale. Holding onto the recipe gives you a clean way to evaluate any AI claim for the rest of the course.

I will walk through them in the order they tend to appear in a typical project: first the data, then the compute that trains a model on the data, then the algorithm that says what kind of model gets built. Most popular coverage gets the order backward. Most working researchers do not.

Data

Data is the examples a model is shown during training. For a chatbot, that means text: trillions of words pulled from books, websites, forum posts, code repositories, and licensed corpora. For an image generator, it means pictures paired with captions. For a voice model, it means audio paired with transcripts. The shape changes; the principle does not. The model is whatever the data was.

Three things to notice about data, all of which you will encounter again later in this course.

First, the amount of data matters more than people who have not built models tend to expect. There is a rough working rule, well documented in Goodfellow, Bengio, and Courville’s Deep Learning textbook (Goodfellow et al., 2016), that a deep neural network needs roughly one to ten million labeled examples to match the accuracy of a careful classical model on the same task. A simpler model trained on a thousand examples will often outperform a deep network trained on a thousand examples, and the deep network will only start winning when the dataset gets several orders of magnitude larger. The phrase “deep learning is data-hungry” is doing real work; it is not slang.

Second, the kind of data matters. The three classical learning settings the field works in are supervised (each example comes with a correct answer attached), unsupervised (examples come without answers, the system has to find structure on its own), and reinforcement (no examples; the system learns from trial and error against a reward signal). Sutton and Barto’s textbook (Sutton & Barto, 2018) is the canonical reference for the third one. We will not work through these settings in depth this week, but the vocabulary will recur. A chatbot’s main training pass is unsupervised in a specific narrow sense, and its later “make it actually helpful” pass is reinforcement-flavored. We will open that distinction up in Part 3.

Third, the source of data is the entire ethics and copyright story of the last two years. Where the training text came from, whether the people whose words got used were asked, what counts as fair use, what counts as theft, and how labs handle these questions: that is most of the AI legal news you have seen, and the cases are not settled. We will spend a week on it in Lesson 49. For now, hold the idea that “the model is the data” is more than a slogan, and so is the question of whose data.

Compute

Compute is the raw computer time used to train the model. It is measured loosely in floating-point operations per second (FLOPS), more practically in dollar costs, and most practically in “how many specialized chips were running for how many weeks.” The chips are graphics processing units (GPUs) made primarily by Nvidia, with newer entrants from AMD and Google’s custom Tensor Processing Units in the mix.

The compute story for the present moment is staggering. Stanford’s 2025 AI Index Report (Stanford HAI, 2025) documented that training compute for the highest-profile AI models doubled approximately every five months in the years leading up to 2024. The trend predates large language models: an analysis by Sevilla and colleagues at Epoch AI (Sevilla et al., 2022) showed that compute used for the largest ML training runs has gone through three distinct eras, with the most recent era seeing a sustained doubling on the order of months rather than the years that characterized earlier decades.

What this means concretely is that the difference between a training run today and a training run five years ago is not a small percentage. It is a factor of dozens to hundreds. The largest frontier models are estimated to cost in the tens of millions of dollars to train. The next generation will reportedly cost more.

This is the reason a small number of companies build the largest models. The reason is not secret algorithms (the algorithms are mostly public). The reason is not unique data (most labs scrape similar sources). The reason is compute, and compute is expensive. The shape of the AI industry in 2026 is, in large part, a shape determined by who can afford to keep doubling the chip bill.

This is a Lesson 4 setup; we will deepen the concentration-of-control thread in Lesson 33 (titled in our syllabus “Concentration of Power”). For now, register that compute is the ingredient that decides who plays at the frontier.

Algorithms

The algorithm is the recipe for what kind of model gets built from the data and the compute. It is the part that academia historically owns. It is the part that gets named: backpropagation, convolutional neural network, transformer, diffusion model. Most of the famous AI architectures you have heard of are algorithms. The transformer (introduced in 2017, which we covered last week) is an algorithm. The diffusion model behind most image generators is an algorithm. The reinforcement-learning loop behind reasoning models is an algorithm.

Algorithms have two useful properties that compute and data do not have. First, they are mostly published in academic papers, which means that once an architectural idea works, every other lab can read about it and adopt it within months. The transformer paper was free to download on the day it was posted. So was the diffusion-model paper. So were almost all the influential AI architectures of the past decade.

Second, algorithms can produce step-changes. A new architecture can make a previously impossible task possible. The 2012 AlexNet result and the 2017 transformer paper, both from Lesson 3, are step-change algorithms. Between them they unlocked image recognition and language modeling at scales that older approaches could not reach.

The catch is that algorithms alone do nothing. The transformer paper was published in June 2017; the public would not have access to a transformer-based chatbot for another five years. The algorithm needed enough data (large text corpora) and enough compute (GPU clusters) to actually be useful at the scale where the transformer’s advantages over older approaches showed up. The algorithm was the ignition; the data and compute were the fuel.

Why all three together

A useful exercise is to take any of the historical waves from last week and ask which of the three ingredients was missing.

The symbolic-AI work of the 1960s and 70s had algorithms (rule-based logic was understood well) but lacked data (the world had not been digitized yet) and lacked compute (a 1970 computer is comically slow by 2026 standards). The work produced narrow successes and failed to generalize, and the field cooled.

The expert-systems boom of the 1980s had algorithms (production rules were well understood), had a little more data (databases existed), and had a little more compute (LISP machines). It did not have enough data for the systems to generalize beyond the narrow domains the rules were written for, and it did not have enough compute to make brittleness affordable. The systems failed in production, and the field cooled.

The 2012 deep-learning shift had a new amount of all three. The algorithms had been around since the 1980s, refined for thirty years. Consumer GPUs (originally built for video games) suddenly made compute cheap enough at scale. The web and digital cameras suddenly made data plentiful and labeled at scale. The three ingredients met for the first time in roughly the right proportions, and a problem that had been hard for fifty years (recognizing objects in photographs) became tractable in a single afternoon’s training run.

The 2017–2022 language-modeling moment was the same recipe a generation later. The transformer algorithm met massive text corpora met GPU clusters that the major labs could afford. Output: ChatGPT, on a Wednesday in November.

Whenever you read a claim that “AI is going to revolutionize” anything in the next five years, ask which of the three ingredients is supposed to make the difference. New algorithm? New kind of data? More compute? Sometimes the claim has one of those clearly. Often it has none, and the claim is just hype dressed in the present tense.

Going Deeper (optional)

The “three ingredients” framing is durable but incomplete. There is a fourth ingredient that does not always get named and that matters as much as the other three: people.

Data does not arrive labeled. Someone has to label it. For image recognition that meant millions of crowdsourced human tags on photographs. For modern language models that means tens of thousands of paid labelers and ranking-rators, often in low-wage labor markets, evaluating model outputs to teach the model what humans actually want. The reinforcement learning from human feedback (“RLHF”) pass we will cover in Lesson 17 is, beneath the algorithm, a paid labor pipeline.

Algorithms do not write themselves either. The frontier labs employ small numbers of extremely highly paid researchers and large numbers of engineers, and the talent concentration is itself a structural fact about who can build frontier systems. We will pick this thread up at Lesson 29 (who is building this and why) and Lesson 33 (the concentration-of-control question).

So when you see the three ingredients framing this week, hold it as a working simplification: data plus compute plus algorithms, with a quiet fourth ingredient (human labor and talent) that often gets erased in the storytelling. The erasure is part of the story.

What you have, what comes next

Four weeks in, you have a triage kit. From Lesson 1: search or generative? From Lesson 2: which model, which product, which interface? From Lesson 3: which wave of AI history is this latest claim echoing? From this week: which of the three ingredients is supposed to be doing the work, and is that ingredient actually available?

In Lesson 5 we sharpen one of those distinctions. Generative AI is the kind that writes new content. Predictive AI is the older, more boring kind that classifies and ranks. The two are genuinely different in what they do, and the popular conversation conflates them constantly. Telling them apart is the move that lets you tell which kind of “AI feature” your bank or doctor or grocery app is actually using.

If You Want to Dig Deeper

For the algorithmic side of the deep-learning ingredient, the standard free reference is Deep Learning; the first few chapters are mathematical but readable, and Chapter 1 on its own pays for the visit. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. MIT Press. https://www.deeplearningbook.org/

For the compute ingredient told as a dollars-and-chips history, this Epoch AI study estimates the actual cost of training the largest models and shows how that cost has been rising; the broader Epoch portal tracks the trend live. Cottier, B., Rahman, R., Fattorini, L., Maslej, N., & Owen, D. (2024). The rising costs of training frontier AI models (preprint). arXiv. https://doi.org/10.48550/arXiv.2405.21015

For the human-labor angle on data that the Going Deeper section opened up, this paper documents the conditions under which much of the world’s labeled training data is produced. Hao, K., & Hernández, A. P. (2022, April 20). How the AI industry profits from catastrophe. MIT Technology Review. https://www.technologyreview.com/2022/04/20/1050392/ai-industry-appen-scale-data-labels/

Author Note:

Grace Ann Hansen is an independent researcher and writer, and an MBA & PhD graduate student in health informatics and artificial intelligence. She is also a published author, a professional musician, a gymnastics coach, and a queer transgender woman living in Sioux Falls, South Dakota. She corrects all her papers and articles with Grammarly, because even though she has deep thoughts, she has shallow patience for punctuation. She uses Anthropic’s Claude in Research mode for source location and verification on cited factual claims; all interpretation, argument, and prose are her own. Correspondence concerning this article should be addressed to Grace Ann Hansen at grace@graceannhansen.com.

References

Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. MIT Press. https://www.deeplearningbook.org/

Sevilla, J., Heim, L., Ho, A., Besiroglu, T., Hobbhahn, M., & Villalobos, P. (2022). Compute trends across three eras of machine learning. In 2022 International Joint Conference on Neural Networks (IJCNN) (pp. 1–8). IEEE. https://doi.org/10.48550/arXiv.2202.05924

Stanford Institute for Human-Centered Artificial Intelligence. (2025). The AI Index 2025 annual report. Stanford University. https://hai.stanford.edu/ai-index/2025-ai-index-report (accessed 2026–05–19)

Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An introduction (2nd ed.). MIT Press. http://incompleteideas.net/book/the-book-2nd.html