Autonomous Vehicles: Perception Layer

The Three Layers of Autonomous Vehicles

When an autonomous vehicle drives, it’s constantly doing three things: perceiving the world around it, predicting what that world will do next, and planning the action it should take. Whether that’s slowing down, braking, or changing lanes. As humans, we run through the same sequence every time we drive, but we rarely think of the steps as distinct. For us, they blur into a single reflex. An AV has to break that reflex apart into explicit, separate stages.

These three layers, perception, prediction, and planning, are the backbone of how AVs work. Each owns a different part of the journey, and each hands its output to the next.

Perception: What is the car seeing?

Perception is the first layer, and its job is to observe the world and identify what’s in it. As the car drives, it sees other vehicles, pedestrians, stop signs, and traffic lights. The perception layer’s task is to detect each of these objects, classify them correctly, a stop sign as a stop sign, a red light as a red light, and locate them in space by drawing a bounding box around each one.

Importantly, perception doesn’t guess what happens next. It answers a more specific question: what is around the car right now, and where is it? It produces a structured snapshot of the current moment, along with a confidence score for each detection, and passes that snapshot to the next layer.

Prediction: What will the world do next?

Once the car knows what’s around it, the prediction layer forecasts what those objects will do. If perception has flagged a pedestrian on the sidewalk, prediction asks the follow-up questions: Will they keep walking? Will they step into the road? Will they wait at the crosswalk? It assigns a probability to each possible outcome, turning a static snapshot into a set of likely futures.

Planning: What should the car do?

Finally, the planning layer takes those predictions and decides how the car should act. If the pedestrian is likely to step into the road, planning relays the appropriate command to the vehicle: ease off the accelerator, brake, or steer around. This is the layer where all the upstream analysis becomes a physical action.

A Closer Look at Perception

Since perception is where everything begins, it’s worth understanding how the car actually “sees.” The layer is constantly analyzing and classifying objects while drawing bounding boxes to place them in space, and it does this using one of two sensor strategies.

The first is camera-only. The car relies entirely on cameras that continuously record its surroundings, and it infers depth the way our eyes partly do, by judging the size of objects relative to one another in the image. Tesla is the best-known proponent of this approach.

The second adds LIDAR, which measures depth directly by bouncing laser pulses off surrounding objects. Instead of inferring how far away something is, the car measures it, which improves accuracy especially in poor lighting or unusual conditions. Waymo is the standout example here.

Regardless of the approach the AV takes, the images are usually passed through the same technology: a convolutional neural network (CNN).

How Does the Car Actually Recognize Objects?

A convolutional neural network is the model that turns raw pixels into the structured snapshot perception hands downstream. It’s worth being precise about what that means: an image goes in, and what comes out is something like there is a pedestrian here and I’m 94% confident about it.

The core building block is the filter. A filter is a small grid that slides across the image looking for one specific pattern: a vertical edge, a sharp change in contrast, a particular curve. On its own, a single filter isn’t impressive. But stack enough of them in layers and something useful happens. The earliest layers detect primitive features like edges and corners. The next layers combine those edges into shapes. Deeper layers combine shapes into recognizable parts, and eventually into whole objects: a wheel arch, a windshield, a car.

The property that makes this work for driving is translation invariance. A filter that recognizes the hard edge of a stop sign works the same way whether the sign appears in the top-left corner of the frame or the bottom-right. The same filter is applied everywhere across the image, so the network learns what a stop sign looks like rather than where stop signs usually are. For a car, that distinction is everything. Objects can appear anywhere in the frame, at any scale, and the model can’t be caught off guard by a pedestrian showing up somewhere unusual.

Videos are Just Frames

A camera feed is a sequence of still images, typically around 30 per second, and each frame goes through the network as its own complete pass. This is where the constraint that shapes everything downstream comes into focus. At 30 frames per second, the system has roughly 33 milliseconds to fully process each frame. A car traveling at 65 mph covers about 95 feet every second — roughly three feet in that 33-millisecond window. Every millisecond the model spends thinking is distance the car travels blind.

In perception speed is a serious a safety constraint, and it puts a hard ceiling on how much model you can afford to run. Which sets up the trade-off at the center of every architecture decision an AV team makes: how much accuracy are you willing to give up to stay inside that budget?

Mobile Net vs. ResNet

These are complicated topics, but if I may distill it: ResNet is the accurate one. MobileNet is the fast one. That’s really the whole thing.

ResNet was built to go deep, more layers, more capacity to recognize complex scenes. It’s more accurate and it costs more compute. MobileNet was built for phones and embedded chips: it uses a cheaper way of doing the same convolution math, so it runs fast on small hardware. You give up a few points of accuracy to get there.

So the question becomes: what accuracy are we giving up to hit our latency budget, and is that still the right trade? That’s a product question as much as an engineering one, and it’s one that changes every time the hardware changes.

Final Thoughts

Working through this, the thing that stuck with me wasn’t any single piece of technology. It was how much of autonomous driving comes down to a chain of handoffs. Perception produces a snapshot of what’s out there. Prediction turns that snapshot into a set of likely futures. Planning turns those futures into a physical action. Each layer is only as useful as what it receives from the one before it, which means a perception failure doesn’t stay a perception failure. It becomes a bad prediction, and then a bad decision.