From Barnsley to Stable Diffusion

A reflection on four decades of “Find The Equation For The Image”.

“Image generated using OpenAI’s DALL·E.”

Today, a break from The Grand Game as we take a lesisurely walk down memory lane (before age, or wine, robs me of detailed recall) with a partial origin story before I entered the corporate world completely by accident.

It could have all be so very different.

What I Was Doing in 1995

In 1995 (ish) I finished a PhD around fractal image compression, though the write up would drag on a bit. As they do, but that’s another story.

The idea was simple enough to explain at a dinner party, though I didn’t actually go to dinner parties (and still don’t, as it happens, so don’t even think about it), but complicated enough to spend three years on.

Given any picture, find a small set of mathematical functions whose attractor — the image you get when you run them on themselves over and over in a feedback loop — is the picture you started with. Approximately. Then, store the functions, and throw away the picture. To get the picture back, run the functions over and over again and watch the attractor appear, one iteration at a time, out of a fog of random noise, or anything really, it doesn’t matter what you start with.

If that last sentence sounds suspiciously like what a modern diffusion model does, that is not an accident.

The Same Question, Asked Twice

The question (Michael) Barnsley was working on in the mid-1980s, and the one his student Arnaud Jacquin made practical in 1992, was geometric.

A picture is a point in an extremely high dimensional space. A 256×256 greyscale image lives in roughly 65,000 dimensions, one per pixel. Real world images don’t actually fill that space, they occupy a thin, contorted surface within it — what people now call the natural image manifold. Most points in the pixel space are noise. And, if you’re that way inclined, a very tiny fraction are actually pictures of cats.

The act of compression is the act of parameterising that manifold. If you can describe the surface with fewer numbers than there are pixels, then you have compression. The whole game, from Barnsley onwards, was to find a procedure whose output is the image you want, and whose description is shorter than the image.

Fractal (image) compression answered the question with contraction maps. You handcraft, or, after Jacquin, you automatically search for, a small set of functions that, run iteratively, converge to a fixed point that is approximately your image. The functions are then the compressed file and the image is what falls out when you run them.

Diffusion models answer the same question with a learned score function. You train one enormous neural network on a hundred million images and it learns the gradient of the natural-image manifold which itself is a vector field defined everywhere in pixel space that always points toward more image like territory. To generate a picture you start with Gaussian noise and walk along the gradient. To compress an image, you encode a short latent code that biases the walk toward a specific image. Run the walk and, again, watch the attractor appear.

It’s an iterative procedure converging to a target image. Run it long enough and out comes a picture. That was magic in the 1990s.

So, it really is the same question and just, essentially, it was the toolkit that changed.

Why Fractal Compression Lost

Fractal (image) compression had a fatal engineering problem, encoding an image was NP-hard. To compress an image you had to search a combinatorial space of possible function sets looking for one that produced your picture. Even with quadtree partitioning, classifier-based block matching, and lattice-restricted transforms — the bag of tricks that got the field through the 1990s — encoding a single image took minutes to hours. JPEG, meanwhile, was a discrete cosine transform you could compute in a single pass.

JPEG, unfortunately¹, ate fractal compression’s lunch on engineering grounds. The mathematics was beautiful, as mathematics tends to be, but the compute budget was atrocious. Especially on Sun workstations in the early 1990s, and I speak from long experience. By 2000 the field was effectively over as a commercial proposition. The papers kept coming for another decade out of academic inertia. JPEG won by default because fractal never showed up at scale (pun intended, if you know, you know).

I spent the late 1990s looking back and thinking, well, at least I was there².

Why Diffusion Wins Now

Modern diffusion has the same expensive encoding pattern in that training a state of the art image diffusion model costs millions of GPU hours. But, and this is the entire point, it costs that only once. Once trained, encoding a new image is a single forward pass through the model and decoding is just a fixed number of denoising steps. The cost shifted from per image combinatorial search to one time training amortised (oh, la la) over every image that will ever be compressed with it.

This is the pattern across the field, transformers beat recurrent neural networks the same way. Train once, infer cheaply. Gradient descent beats combinatorial search, eventually, given enough scale.

Fractal compression was not wrong, it was forty years too early and this is the point I’m slowly iterating toward (pun intended, as I try really hard with my puns — something you’d know if you’ve read my material previously³.)

A Lineage, But Not the Lineage You Think

I should be careful here, because if I am not careful the kind of reader who has actually read Sohl-Dickstein 2015 will fillet me in the comments⁴. Diffusion models do not descend from fractal compression intellectually, their actual lineage runs through non-equilibrium thermodynamics, score matching, and stochastic differential equations. Nobody at Google or OpenAI or Stability was reading Barnsley and Hurd’s Fractal Image Compression (1993) when they invented DDPMs. I doubt some of them were even born.

But intellectual lineage is not the only kind of lineage that matters, there is also lineage by problem. Two procedures, four decades apart, in this case attempting to solve the same geometric question with the same iterative structure. The connection is real, even when the usual citation chain is empty.

In March 2025 the academic community finally noticed as an ArXiv paper that month (2503.11321) introduced Fractal Frequency-Aware Band Learning inside a diffusion compressor, an explicit bridge between the two regimes and the loop finally closed. Forty years late, I grant you, but it closed.

What I Was Actually Doing

I spent three years at (the University of) Bath looking for polynomial fractal transforms, a generalisation of Jacquin’s affine block matching that could, in principle, capture more structure with fewer maps and it half worked. The thesis stands anyway, at least on my bookshelf, but nobody really implemented it commercially because, again, JPEG. I think (Microsoft) Encarta might have some fractally compressed images in it, but that’s fallen behind the legacy horizon now.

What I was actually doing, without quite knowing it (in many senses), was searching for a parameterisation of the natural image manifold using a function class that turned out to be too restricted. Contraction maps are a tiny corner of the space of functions that can approximate the manifold, but neural networks are a much larger corner. Score functions, learned by stochastic gradient descent on hundreds of millions of examples, are the right tool for this job. I did not have stochastic gradient descent, I did not have hundreds of millions of examples, and I definitely didn’t have the compute. No-one did then.

The question was right, but the function family was wrong.

It is a peculiar feeling to watch a field you worked on get vindicated, decades later, by a completely different set of people using a completely different toolkit. I’m not bitter, I think it’s great, and I’m highly amused as I always thought fractals in general were somehow special — fractional dimensions are cool, and I think higher dimensions are even cooler.

Anyway, the score function was always going to be the better answer, we just needed somebody to figure out how to learn one.

Sohl-Dickstein says what?

tl;dr? They showed you can teach a neural network to generate pictures by running a destruction process backwards, i.e. take a real image, scramble it into pure noise one tiny step at a time, then train the network to undo each step. Hand the trained network pure noise and it walks backwards through the steps until a picture appears, which is exactly the iterative procedure converging to an image pattern I’ve been rambling on about.

[1]: Unfortunately in that I’d probably have stayed in academia, which I really liked, but I got distracted, got a job, and ended up in The Grand Game of Software Engineering and never would have written this. You can’t change the past, therefore I have no regrets except that possibly I might have gone to the University of Edinburgh to do Observational Astronomy just when computing was really taking off in astronomy in 1992. Oh well. I’d probably still be remote working, and not at an actual observatory anyway! [2]: Playing netrek, I was in the European team. I wrote some papers too, honest. [3]: Love the puns, me. They keep me sane, and smiling. [4]: Then again, most people fillet most people in any comments these days, welcome to the 2020s, so I’ll just push on regardless.