Designing Low-Latency ML Inference Systems for Autonomous Vehicles: A System Design Interview Walkthrough

System Design Question: How would you design a low-latency inference system that runs on an embedded GPU for example in the car?

Scope: From problem definition, let’s shortlist critical requirements: Functional:

1. preprocess camera data
2. Design memory management i.e. memory allocation strategy
3. Infer the command Non-Functional:
4. Redundancy and fallback
5. Model quantization and acceleration
6. Error handling
7. Performance Monitoring and so on…

Exclusions: (if any goes here)

Thermal/power constraints Deployment requirements Model constraints Are we allowed to modify the pre-trained model (quantization, pruning, distillation) or must we use it as-is?

First Step-Clarifying Questions(2–3min): As I highlighted above, you will ask the interviewer following questions:-

1. low-latency means how much 10 ms or 60 ms?

Interviewer: Latency requirement: We need end-to-end inference under 50ms for safety-critical decisions. The model should process sensor inputs and output actionable results within this window.

2. How many embedded GPUs one or many?

Interviewer: Hardware: You have one embedded GPU — think NVIDIA Jetson AGX Orin or similar automotive-grade hardware with about 8GB GPU memory and ~200 TOPS AI performance.

3. Do you have a pretrained model already?

Interviewer: Yes, we have a pre-trained perception model — it’s a multi-task network that takes camera images as input.

4. What the model is for? Is it a neural network for processing raw sensor data and directly output driving command.

Interviewer: Model purpose: The model performs object detection, semantic segmentation, and depth estimation from sensor data. It outputs bounding boxes, object classifications, segmentation masks, and depth maps that feed into our downstream planning pipeline. Interviewer: Now, walk me through your system design approach. How would you architect this to meet our latency requirements?

Other Clarifying questions may be Data throughput: What’s the input data rate? How many camera streams (1, 4, 8 cameras?) and at what resolution? What’s the LiDAR point cloud size and frequency?

Failure handling: What should happen if inference takes longer than 50ms? Do we need graceful degradation or can we drop frames?

How are you handling the data pipeline here? Are you processing sensors sequentially or in parallel?

High Level Approach

Fig. 1.0

Keep on the conversation. At this point the interviewer may say: This is a good start and shows the basic flow, but I need you to go deeper on several aspects. Candidate: Yes, I am expanding first block-input processing.

Let’s assume: ✅ multiple cameras (say, 4 to 6) around the car. ✅ want to process them together in a clean pipeline. Each camera has: Its own lens (so distortion) — distortion correction Its own position and orientation Its own timestamped images So you can think of preprocessing as a sequence of steps applied per camera, followed by multi-camera alignment.

Detailed Preprocessing Steps (Per Camera)

Below is how pipeline will look. Step 1: Ingestion Capture raw frames from each camera sensor. Attach metadata: Timestamp, Camera ID etc. Step 2: Undistortion Correct lens distortion (e.g., fisheye warping) Use intrinsic parameters (camera matrix, distortion coefficients). Output is an image with straight lines and consistent scale.

✅ In practice, this is done via OpenCV’s cv2.undistort or a GPU kernel.

Step-3 Resizing and Normalization Resize to the resolution expected by your neural network (e.g., 512×512) Normalize pixel intensities (e.g., subtract ImageNet mean, divide by std) Step 4: Timestamp Synchronization Align all camera frames to a common timestamp. Select nearest-in-time frames. Buffer images if necessary. ✅ This ensures all inputs correspond to the same real-world moment.

fig. 2 expanded input raw data pipeline

Always discuss tradeoffs one approach vs another, one algo versus another

interviewer: This is much better! You’ve shown good understanding of the computer vision preprocessing pipeline. I like that you’re thinking about the multi-camera challenges — distortion correction, synchronization, and calibration are all crucial. However, I have some concerns about meeting our 50ms latency requirement. Your steps 1–6 appear sequential (Sequential bottleneck). With 6 cameras, if each undistortion takes 5ms, that’s already 30ms just for step 2. How would you parallelize this?

[embed]Agents: All You Need | All You Need Agents | Pradeep Pujari | Substack Your guide to the evolving world of LLMs and AI agents. Click to read All You Need Agents, by Pradeep Pujari, a…agentsallyouneed.substack.com

Think loud…

Core idea of parallelism: Each camera is logically independent up to timestamp synchronization. 👉 So you can process them in parallel. Multi-threading or Multi-processing Approach-1: Use a thread pool or process pool. Each worker processes 1 camera stream . Undistortion . Rectification . Resizing . Normalization All in its own thread/process. Python: concurrent.futures.ThreadPoolExecutor C++: Thread pool CUDA: Launch per-camera kernels on the GPU

✅ This gives near-constant latency rather than linear scaling. Approach — 2: GPU Batch Processing Stack all camera frames into a batch tensor. Launch one batched CUDA kernel to undistort all images. Do resizing and normalization in the same pass. Benefit: (This is the tradeoff approach 1 and 2, important to mention) . Much faster than doing CPU preprocessing. . Memory coalescing improves throughput.

Synchronization overhead: Timestamp Synchronization could be expensive if you’re buffering and searching for timestamp matches across N=6 cameras. What’s your strategy here? If time left is too short, I would recommend briefly describe as below.

To minimize synchronization overhead, I maintain a time-sorted buffer per camera and define a reference timestamp — typically from a master camera like the front view. Then, I use a nearest-neighbor search or time-window filtering (e.g., ±10ms) to find the best matching frames from other cameras. The lookup is bounded and fast due to pre-sorted buffers.

For further performance, I parallelize the matching step across cameras and trigger batch formation only when enough synced frames are available.

With hardware timestamping and a common clock domain, this becomes even more robust and low-latency.

Future Ideas you can give at this point: The last minute of an interview is a golden opportunity to leave a strong impression. Talking about futuristic approaches shows you’re not just a coder — but a forward-thinking engineer. 🧠 1. Foundation Models for Perception We’re moving toward large-scale vision foundation models — like SAM, DINOv2, or OmniPose — that can generalize across tasks. Integrating one unified vision backbone that handles detection, depth, and segmentation could replace multiple specialized networks. ✅ Value: Better transfer learning, reduced compute cost over time. 🧠 2. Neural Radiance Fields (NeRF) for 3D Reconstruction NeRF-like techniques are enabling high-fidelity 3D scene representations from sparse cameras. With real-time NeRF variants, we could build dense volumetric maps from camera-only setups. ✅ Value: Detailed 3D understanding without LiDAR. 🧠 3. End-to-End Differentiable Planning Instead of separate perception, prediction, and planning blocks, recent trends point to end-to-end trainable pipelines — where perception feeds directly into trajectory generation via a learned policy. ✅ Value: Better coupling of perception and control, less manual tuning.

Close with confidence: I’d love the chance to work on systems like this — ones that not only solve the problems of today but anticipate the challenges of tomorrow.

**Go to previous ML System Design interview question**