Inside the Image Sensor

How a grid of silicon learns to see ..and why colour is a beautiful lie.

Last time, I told you about photodiodes. Tiny light-sensitive elements, millions of them packed into a grid. A photon hits one, an electron shakes loose, a charge builds up. That charge becomes a number. That number becomes a pixel.

Clean, right? Satisfying even.

But here’s the thing I glossed over. I left out the part that took me weeks to really sit with. Because when I first heard it, my reaction was: wait, that can’t be right.

A digital camera cannot see colour. Not natively. Not in the way your eye does.

Let me explain. And to do that, I need to take you back to the photodiode itself.

What a photodiode actually does

A photodiode is a semiconductor device ..typically silicon ..that responds to light. When a photon with enough energy strikes it, it knocks an electron loose from its crystal lattice. That loose electron becomes part of an electrical current. Capture enough of them and you get a measurable voltage. That voltage maps to a digital value: your pixel brightness.

There’s a beautiful simplicity to this. Silicon doesn’t have preferences. It doesn’t care if the photon was red, green, or blue. It just counts them.

Which is exactly the problem.

An unfiltered photodiode responds to a broad range of wavelengths-roughly 350 nm to 1100 nm, covering visible light and extending into near-infrared. What it gives you is intensity. Brightness. Not colour. A pixel with a value of 180 could be responding to red light, green light, or a mix of the two. There is no way to tell them apart.

So how do cameras see colour at all?

The Bayer filter -colour by deception

In 1974, an engineer at Kodak named Bryce Bayer came up with the solution that still dominates camera design today. And it is, in the most elegant sense, a trick.

His idea: put a coloured filter over each photodiode. Red filter over some pixels ,they only respond to red wavelengths. Green over others. Blue over the rest. Each pixel now captures only one colour channel. You’ve traded full-spectrum sensitivity for colour discrimination.

The pattern he chose arranges filters in a repeating 2×2 mosaic: one red, one blue, two green. Green gets double representation because human vision is most sensitive to green ,it’s how we perceive detail and brightness. This repeating grid is called a Colour Filter Array, and Bayer’s specific arrangement became so dominant it’s now simply called the Bayer pattern.

But here’s the part that stops people cold: each pixel now only knows one colour. The red pixel knows red. The blue pixel knows blue. Neither knows what the others saw.

So how do we get a full-colour image?

Demosaicing -the educated guess

We interpolate. We look at a pixel and its neighbours and we make an educated inference. A red pixel surrounded by bright green and dark blue neighbours? The algorithm estimates what the green and blue values at that red pixel’s location probably are, based on the surrounding context.

This process is called demosaicing and it runs on every single image your camera captures before you ever see it. It’s not reading colour. It’s inferring it.

For most scenes, the inference is excellent. Modern demosaicing algorithms are remarkably good. But push them-fine repeating textures, sharp diagonal edges, scenes with high-frequency colour variation and you start to see the seams. False colours. Moiré patterns. Artefacts that exist not in the scene but in the algorithm’s assumptions.

Every colour photograph you’ve ever seen was, at some level, a reconstruction. Not a recording-a reconstruction.

I find that thought genuinely humbling.

Full-well capacity -how much light is too much

Each photodiode is essentially a tiny bucket. When light hits it, electrons accumulate in that bucket. The maximum number of electrons it can hold before it overflows is called the full-well capacity.

When the bucket overflows, bad things happen. The excess charge spills into neighbouring pixels ,a phenomenon called blooming. You’ve seen this in photos: bright highlights bleeding into surrounding areas, white smearing into the frame.

Full-well capacity is a hard physical limit. Make the pixel smaller and the bucket gets smaller. This is the core tension in camera design that I’ve been circling around since the first article: more pixels means smaller pixels, smaller pixels mean smaller buckets, smaller buckets overflow faster.

Satellite sensors -which I’ll get to shortly navigate this tradeoff very differently from your phone camera. But the physics is identical.

Readout noise-the hum beneath the signal

Even in perfect darkness, a sensor generates electrons. Thermal vibration of silicon atoms mimics the signal from light. Electronics in the readout circuitry introduce small random fluctuations. The act of converting charge to voltage adds its own error.

All of this is noise. And it is always present.

The question isn’t whether noise exists,it’s whether your signal is loud enough to hear over it. Signal-to-noise ratio. We talked about SNR last time in the context of pixel size. Now you understand why pixel size matters so directly: bigger pixel, bigger bucket, more electrons captured per exposure, signal rises well above the noise floor.

Satellite sensors operating in low-light conditions dawn passes, forest canopy imaging, polar regions in winter ,deal with this constantly. The engineering choices around pixel size, full-well capacity, and readout circuitry in a satellite sensor aren’t arbitrary. They’re the careful resolution of competing constraints under genuinely brutal conditions.

CCD vs CMOS -the architecture question

I introduced these in Article 01 as two types of sensors. Now we can go one level deeper.

In a CCD — Charge-Coupled Device — electrons captured by each photodiode are moved in an orderly chain, passed bucket-to-bucket across the chip to a single readout circuit at the edge. The readout is serial and slow, but because there’s only one amplifier, noise is low and consistent. CCDs produce beautifully clean images. They draw a lot of power doing it.

In a CMOS — Complementary Metal-Oxide Semiconductor — each pixel has its own amplifier right there on the silicon. Readout can happen in parallel across many pixels simultaneously. This is why CMOS sensors are fast: they don’t need to move charge across the entire chip. And because amplifier fabrication uses standard chip-manufacturing processes, CMOS sensors can integrate processing logic directly — an entire ISP on a single piece of silicon.

The tradeoff used to be simple: CCD for quality, CMOS for speed and efficiency. But CMOS fabrication has improved so dramatically that modern CMOS sensors rival CCD quality in most conditions. Most satellite sensors built in the last decade are CMOS. Nearly every phone camera is CMOS. CCDs persist in scientific instruments where their noise characteristics and linearity still matter.

So there it is ,the sensor demystified. Silicon that counts photons but can’t see colour. Colour filters that lie productively. Algorithms that reconstruct what was never directly recorded. Buckets that overflow. Noise that never fully sleeps.

It’s messier than the clean abstraction of ‘pixels equal light.’ But it’s also more honest. And once you understand it, you start seeing the decisions embedded in every image the choices engineers made about pixel size, filter arrangement, readout architecture all of them visible in the photographs if you know how to look.

Coming up in Article 03 ↓ We take the camera out of the studio and strap it to a satellite. Why does altitude change everything? What is a pushbroom sensor? And how does Ground Sample Distance connect what we see in orbit to what we can measure on Earth?