What a Semester of Graduate Computer Vision at NYU Actually Taught Me

Note: This can be viewed as a primer on the historical evolution of the field. Given the rapid pace of progress in modern AI and computer vision, some specific methods, benchmarks, or claims discussed here may become outdated over time. However, the fundamental principles, motivations, and conceptual reasoning underlying these developments remain largely unchanged and continue to shape current research directions. Thank you Prof. Saining Xie for this amazing class.

Index

1. Introduction: What a Graduate Computer Vision Course Reveals
2. The Naming, Structure, Actions, and Matching Problem
3. Sixty-Five Years of Computer Vision History
4. Why Hand-Designed Features Were Doomed
5. Vision vs Language: Why Perception Is Harder Than It Looks
6. ConvNets and the Architectural Arms Race
7. Attention, Transformers, and Multimodal Unification
8. The Self-Supervised Learning Revolution
9. Detection, Segmentation, and the Hardness of Localization
10. Video Understanding and Temporal Complexity
11. Generative Models: GANs, VAEs, Diffusion, and DiT
12. The Persistent Challenge of 3D Vision
13. Visual Intelligence Beyond Traditional Computer Vision
14. Why the Hard Problems in Vision Still Matter
15. Conclusion: The Future of AI Needs Grounded Perception

If you sit through a full semester of a graduate computer vision course, you come out of it with a strange feeling. On the one hand, the field has had a spectacular decade. ImageNet top-5 error fell from 28.2% in 2010 to 2.3% by 2017, blowing past the 5.1% human baseline. We now have generative models that produce photorealistic images from text, detection systems robust enough for autonomous driving, and self-supervised representations that transfer effectively across a wide range of downstream tasks.

On the other hand, the central problem the field set itself in 1966, getting a computer to look at a scene and truly understand it, remains unsolved. Domestic humanoid robots are still largely confined to controlled demonstrations. And the entire foundation of modern visual intelligence rests on a recurring tension: vision is fundamentally different from language, in ways that profoundly shape the future trajectory of AI.

This article is a tour through that tension, organized around the actual arc of the course: history, fundamentals, deep learning, self-supervision, vision-language models, video, generative models, and 3D reasoning. The goal is not to summarize everything, but to explain why the field is structured the way it is, and what its most important open problems continue to be.

1. The naming, the structure, the actions, the matching

What do we even mean by Computer Vision? The textbook answer is something like “building artificial systems that process, perceive, and reason about visual data.” That is technically correct, but ultimately uninformative. A better way to think about the field is to ask what we actually want such a system to do.

Show a model a photo of a New York street corner with the Silver Center for Arts and Science in the frame. The first thing we want is naming: assigning semantic labels to what is present in the image. Some of these labels are generic, building, person, road. Others are highly specific. Some are even paragraph-length descriptions that a modern Vision-Language Model (VLM) can generate with little difficulty. Already, “naming” fragments into multiple subproblems, with increasingly fuzzy boundaries between classification, captioning, grounding, and semantic understanding.

Then we want 3D structure. A 2D photograph is merely a projection of a 3D world, and recovering that world, estimating depth, surface orientation, and occlusion relationships (what is in front of what), is fundamentally different from simply recognizing objects.

Next comes actions. What can I do in this scene? Can I cross the street? Sit on the steps? Walk through the front door? This is the language of affordances, and almost none of this information is explicitly encoded in the pixels themselves.

Then we arrive at matching. Given two photographs of the same scene from different viewpoints, can we identify the corresponding points between them? This problem forms the foundation of stereo vision, structure-from-motion, tracking, and much of classical 3D reconstruction.

This taxonomy turns out to be more useful than it first appears. The deep learning era did not collapse all of these problems into a single universal neural network. Instead, it produced different families of methods specialized for each of them.

2. Sixty-five years of trying

The standard history goes like this:

• In 1959, Hubel and Wiesel record from a cat’s visual cortex and discover simple cells (tuned to oriented edges) and complex cells (tuned to oriented edges with some translation invariance). This becomes the biological prior for what the rest of the field will spend decades trying to replicate.
• In 1963, Larry Roberts writes the first PhD thesis on extracting 3D structure from 2D photographs of polyhedral blocks.
• In 1966, Seymour Papert proposes the Summer Vision Project at MIT, intended to build a real piece of a visual system over ten weeks.
• In the 1970s, David Marr articulates a staged theory of vision:
• Primal sketch (zero crossings, edges, blobs)
• 2½-D sketch (local surface orientation)
• 3-D model representation
• The same decade also introduces generalized cylinders, pictorial structures, and recognition by parts.
• The 1980s are dominated by edge detection (Canny, 1986) and the first AI winter.
• The 1990s revive the field with grouping and segmentation, particularly Normalized Cuts (Shi and Malik, 1997).
• The late 1990s and early 2000s introduce:
• SIFT (Lowe, 1999), the first really good local feature descriptor
• Viola-Jones face detection (2001), one of the first major machine-learning wins in vision
• The mid-2000s bring standardized benchmarks:
• PASCAL VOC (2007)
• ImageNet (2009)
• Then 2012 happens. AlexNet halves the ImageNet error rate, and the field flips almost overnight from hand-designed features to learned representations.
• In 2018, the Turing Award goes to Bengio, Hinton, and LeCun for the foundational ideas underlying deep learning.

If you only take away one thing from this history, it should be this: the transition from designed features to learned features is the single most important event in modern computer vision.

3. Why hand-designed features were doomed

Classical filtering approaches like convolution, Gaussian smoothing, image gradients via Sobel-like filters, and the Canny edge detector are important to discuss in this context. These operations are not merely historical curiosities.

Convolution, in particular, is exactly the operation that a Convolutional Neural Network (ConvNet) learns to perform; the only difference is that the kernel weights are learned rather than manually specified. Understanding what a fixed 3×3 Gaussian or Sobel filter does to an image is one of the cleanest ways to build intuition for what the first layer of a ConvNet is doing when it learns its own kernels.

But classical computer vision did not stop at filtering. By the late 2000s, a pipeline for something like scene classification looked roughly like this:

• Extract GIST descriptors (24 Gabor-like filters at 8 orientations and 4 scales)
• Compute HOG2x2 features quantized into 300 visual words via k-means
• Build sparse SIFT histograms with chi-squared kernels
• Add LBP texture features
• Compute SSIM self-similarity descriptors
• Use dense SIFT in *CIE Lab** color space
• Include a tiny-images baseline
• Finally, feed everything into an SVM with carefully tuned kernels

Every single feature was hand-designed by researchers who had thought deeply about properties such as scale invariance, illumination invariance, rotation invariance, or other structural regularities of natural images.

The reason this paradigm ultimately had to fail is not that the features were bad. Many of them were, in fact, exceptionally good. The problem was fundamentally combinatorial. The number of distinguishable visual states is astronomically larger than the number of atoms in the universe. No human team manually designing rules, descriptors, and kernels could ever hope to cover that space exhaustively.

The only scalable solution was to let the model learn its own features directly from data, and to scale both the model capacity and the dataset size until robust representations emerged automatically.

This is why a 2015-era object detector could look at a goose floating on water and confidently output: “Car: 99%.” The local features were individually reasonable. But the system, as a whole, had no genuine understanding of the scene it was observing.

4. Vision and language are different problems

This is perhaps the single best framing for understanding where AI currently stands.

Language is fundamentally abstract knowledge. It has:

• High information density (every word matters)
• A relatively small gap between pre-training and evaluation (next-token prediction is essentially the task itself)
• A highly homogeneous data format on the web (text is text)
• Representations that are comparatively easy to compress and distill into models

It is also comparatively easy to impress humans with language. Language models appear intelligent because language itself is the medium humans use to evaluate intelligence.

By contrast, vision operates over natural signals. It has:

• Low information density (most pixels are redundant)
• A massive gap between pre-training and downstream evaluation (reconstructing pixels is not the same task as understanding objects)
• Numerous complications in data acquisition
• Severe distribution shift across environments and viewpoints

And unlike language, vision is difficult to impress humans with. Recognizing a cat in an image appears trivial to us, even though it is an extraordinarily difficult computational problem.

Hans Moravec observed a related phenomenon in 1988. Tasks humans consider intellectually difficult, such as chess, logic, and calculus, often turn out to be relatively easy for machines. Meanwhile, tasks humans perform effortlessly, such as perception, manipulation, and navigation, prove extraordinarily difficult.

Steven Pinker sharpened the observation even further: as new generations of AI emerge, stock analysts and petrochemical engineers may be replaced before gardeners and cooks.

Yann LeCun describes this asymmetry with a memorable analogy: sensory experience is the cake; language is the icing. Most of what biological organisms know comes not from text, but from interacting with the physical world. Language sits on top of a much deeper foundation of perceptual and embodied experience.

LeCun’s concern is that many current Vision-Language Models (VLMs) are effectively using language as a crutch, compensating for the fact that pure vision pretraining still lags behind the effectiveness of language pretraining. There is truth in this argument, although there are also reasonable counterarguments. Still, the underlying asymmetry between language and vision remains both real and foundational to the future of AI research.

5. ConvNets and the architectural arms race

Photo by Google DeepMind on Unsplash

Once you accept that representations must be learned, the next question becomes: what architecture should learn them?

A convolutional layer is characterized by a small set of hyperparameters:

• Kernel size KKK
• Number of filters CoutC_{out}Cout​
• Padding PPP
• Stride SSS

with a clean output-size relationship:

H′=H−K+2PS+1H’ = \frac{H — K + 2P}{S} + 1H′=SH−K+2P​+1

Standard practice eventually converged on 3×3 kernels with P = 1 (“same” padding) and channel counts scaling in powers of two. Pooling layers provided an additional mechanism for spatial downsampling. Stack enough of these operations together, interleave them with activation functions and normalization layers, and you obtain a Convolutional Neural Network (ConvNet).

What followed was effectively an architectural arms race across the deep learning era:

• AlexNet (2012) demonstrated that deep ConvNets could dominate ImageNet.
• VGG (2014) showed that extremely simple and uniform design rules could scale effectively to much deeper networks:
• all convolutions are 3×3, stride 1, padding 1
• all max-pooling layers are 2×2, stride 2
• double the number of channels after each pooling stage
• keep computation per spatial resolution roughly constant
• GoogLeNet / Inception introduced multi-branch architectures, allowing the network to process information at multiple receptive-field scales simultaneously.
• ResNet (2015) introduced perhaps the most important architectural insight of the decade. As plain networks became deeper, performance paradoxically degraded, not because of overfitting, but because of optimization difficulty. In principle, a deeper network should always be capable of emulating a shallower one simply by learning identity mappings in its extra layers. In practice, plain networks struggled to learn these identities.
• The solution was the residual block: introduce a skip connection so the network learns only the residual function on top of its input. This effectively makes the identity mapping the default behavior. That single modification unlocked extremely deep architectures with 50, 101, and even 152 layers.
• ResNeXt extended ResNet with grouped convolutions.
• Vision Transformers (ViT) (2020) replaced convolutions entirely with self-attention over image patches.
• ConvNeXt (2022) then posed a deeper question: how much of ViT’s success actually came from attention, and how much came from the surrounding ecosystem of modern training practices?
• By systematically modernizing a ResNet, using a patchify stem, larger kernels, fewer activations, layer normalization, AdamW, and longer training schedules, ConvNeXt was able to match or outperform Swin Transformers on ImageNet, while remaining dramatically simpler and transferring better to COCO detection and segmentation tasks.

The deeper lesson is not that convolutions are superior to attention, or vice versa. The more important observation is that architecture matters less than many people assume.

Once you combine:

• Residual connections
• Normalization layers
• Modern optimizers
• Sufficiently large datasets

a surprisingly wide range of architectures begin to converge toward similar levels of performance. The genuinely interesting differences lie less in raw accuracy and more in their inductive biases, computational properties, and scaling behavior.

6. Attention, transformers, and the unification of modalities

Pivot to attention and transformers is the most important conceptual shift after the move to learned representations. Self-attention has a few key properties:

• It treats its input as an unordered set of vectors, computing pairwise interactions via dot-product similarities and a softmax over them.
• It is permutation-equivariant, so you have to add positional information explicitly.
• Masked self-attention: preventing each position from “looking ahead” is what makes language modeling work as next-token prediction.
• The same architecture, with minor changes, handles language, images (as patches), audio (as spectrograms), video (as space-time patches), and so on.

This last point is why transformers ate the world. The pre-transformer landscape was a patchwork of architecture-per-modality: ConvNets for images, RNNs and LSTMs for language, 3D ConvNets for video, separate things for audio. Transformers collapsed all of that into a single recipe with modality-specific tokenization.

For vision specifically, this had two big consequences. First, ViT made it possible to train very large vision models with the same infrastructure used for language models. Second, vision-language models like CLIP became natural: encode images with a ViT, encode text with a transformer, contrast them in a shared embedding space, and you get zero-shot classification, retrieval, and a foundation for downstream multimodal systems.

7. The self-supervised learning detour

If learned representations are the key breakthrough, the next question becomes: where does the supervision come from?

Labels are expensive. Building something like ImageNet, with its 1,000 classes and 1.4 million annotated images, is not a process you want to repeat for every possible domain. This is the motivation behind Self-Supervised Learning (SSL): learning meaningful representations directly from raw unlabeled data.

Broadly speaking, SSL methods can be grouped into three major categories.

1. Generative Methods

These methods attempt to predict part of the input from the rest. Examples include:

• Autoencoders
• Sparse, denoising, and masked autoencoders
• Autoregressive models
• GANs
• Colorization
• Inpainting

The classic autoencoder objective minimizes the reconstruction error between the input and its reconstruction, typically using an L2 loss. A bottleneck in the latent representation forces the model to learn a compressed and informative feature space.

2. Discriminative Methods

These methods learn representations by predicting some property of the input itself. Examples include:

• Context prediction (predicting the relative position of image patches)
• Rotation prediction (predicting how much an image was rotated)
• Clustering objectives
• Contrastive learning

Among these, contrastive learning became particularly dominant for several years. The central idea is elegant:

• Take an image
• Apply two different augmentations
• Treat the two transformed versions as a positive pair
• Train an encoder so that the representations of the positive pair become similar, while representations from different images (negative samples) remain far apart

Methods such as SimCLR, MoCo, and many subsequent approaches were built around this principle.

3. Multimodal Methods

These approaches obtain supervision from a secondary signal, such as:

• Temporal consistency in video
• Audio
• Language

The supervision does not come from manual labels, but from the natural correlations that already exist between modalities.

Then came the major plot twist: Masked Autoencoders (MAE), introduced by He et al. (2022).

After years in which contrastive learning appeared to dominate SSL, MAE returned to what was essentially a modernized form of the denoising autoencoder, now applied to a Vision Transformer (ViT). The procedure was surprisingly simple:

• Randomly mask 75% of the image patches
• Encode only the visible patches using a ViT encoder
• Reconstruct the missing patches with a lightweight decoder

Despite its simplicity, the method worked remarkably well, outperforming many contrastive approaches on transfer benchmarks while being significantly easier to implement and train.

The deeper lesson is that Self-Supervised Learning is not a solved problem. It remains a rapidly evolving and highly empirical research area. Which pretext task performs best depends heavily on:

• The downstream task
• The model scale
• The dataset scale

What has stabilized, however, is the broader paradigm itself:

Learn representations directly from raw signals without labels, then transfer those representations to downstream tasks through linear probing or fine-tuning.

This has become the dominant framework not only for computer vision foundation models, but increasingly for modern AI systems more broadly.

8. Detection, segmentation, and the hardness of “where”

Naming is one thing. Naming and locating is another.

Object detection asks: given an image, output a set of detected objects, each with a category label from a fixed set and a bounding box (x, y, width, height). The standard way to evaluate is Intersection over Union (IoU): the area of overlap between the predicted box and the ground-truth box, divided by the area of their union. By convention, IoU > 0.5 is “decent,” > 0.7 is “pretty good,” > 0.9 is “almost perfect.”

The architectural lineage runs from R-CNN (Girshick 2014), generate ~2,000 region proposals via selective search, run a CNN on each, classify each region and regress a box correction, through Fast R-CNN (1 CNN pass over the whole image, then differentiable cropping of features for each proposal) to Faster R-CNN (the region proposal network is also learned, end-to-end) to Mask R-CNN (add a parallel branch that predicts a segmentation mask for each box, using RoIAlign instead of RoIPool to avoid quantization artifacts).

The other branch is single-stage detectors (YOLO and its descendants), which predict boxes and classes directly from a dense grid of anchors in a single pass, trading some accuracy for a lot of speed.

The thing to notice across all of these is that “where” is harder than “what.” Classification is essentially solved on ImageNet at the human-comparable level. Localization is still a hard, structured prediction problem with its own bestiary of architectures, losses, and tricks.

9. Video, motion, and the curse of the temporal dimension

Photo by Jakob Owens on Unsplash

Once you move from images to videos, almost every problem in computer vision becomes substantially harder.

The simplest baseline is surprisingly straightforward: train a 2D CNN on individual frames and average the predictions at test time. Embarrassingly, this often works remarkably well for action recognition. Many actions can already be inferred from a single frame because the surrounding context strongly constrains the interpretation. A still image of someone inside a swimming pool, for example, is already highly predictive of the action “swimming.”

More sophisticated approaches attempt to model the explicit temporal dimension of video. These include:

• 3D convolutions (extending convolution kernels across time as well as space)
• Two-stream networks (one stream processes RGB frames while another processes optical flow)
• Transformers over space-time patches

The broad lesson from this line of work is subtle but important: temporal information helps, but often less than expected, and usually at extremely high computational cost.

The study of video also exposes a deeper issue that is likely to become increasingly important for the future of AI: video data is enormously redundant. Consecutive frames are often nearly identical, and most pixels across a video contribute very little new information about the underlying scene.

This is one of the clearest examples of why vision has lower information density than language. A 30-second video contains vastly more raw bytes than a 30-second transcript of speech, yet frequently encodes less abstract semantic information.

The implication for modern pretraining is significant. Naively training on raw video streams is extraordinarily inefficient because the overwhelming majority of frames contain redundant signal. As a result, much of the recent research in video representation learning focuses on identifying and extracting the relatively small fraction of informative signal hidden inside the massive volume of redundant video data.

10. Generative models: from GANs to diffusion transformers

Photo by vackground.com on Unsplash

Generative modeling in deep learning has evolved through three major eras, each defined by a different approach to learning the underlying structure of visual data.

The first era was dominated by Generative Adversarial Networks (GANs), introduced by Goodfellow et al. in 2014. The central idea was elegant: jointly train two neural networks in competition with one another.

• A generator attempts to synthesize realistic images
• A discriminator attempts to distinguish real images from generated ones

Under ideal conditions, the generator eventually learns to approximate the true data distribution closely enough to fool the discriminator consistently.

GANs produced some of the most visually striking results of the mid-2010s. Systems such as StyleGAN generated synthetic human faces with remarkable realism and demonstrated that neural networks could model highly complex visual distributions. However, GANs also suffered from significant practical limitations:

• Training instability
• Mode collapse
• Sensitivity to hyperparameters
• The absence of an explicit likelihood function

The second era focused on likelihood-based generative models. This category included:

• Autoregressive models such as PixelRNN and PixelCNN
• Variational Autoencoders (VAEs)
• Flow-based models

These approaches provided a more principled probabilistic framework and were generally easier to train than GANs. Importantly, they offered explicit estimates of the data likelihood. However, for many years, the visual fidelity of their generated images lagged behind that of the best GAN-based systems.

The third and current era is defined by diffusion models. The intuition behind diffusion is surprisingly simple and mathematically elegant.

Start with a real image x0x_0x0​. Gradually corrupt it by adding Gaussian noise over many timesteps until the image becomes nearly indistinguishable from pure noise. Then train a neural network to reverse this process step-by-step: given a noisy image at timestep ttt, predict either:

• The noise that was added
• Or the cleaner image corresponding to timestep t−1t-1t−1

At inference time, generation proceeds in reverse. The model starts from random noise and iteratively denoises it into a coherent image.

This simple framework proved extraordinarily effective. Diffusion models rapidly surpassed GANs in image quality, diversity, and training stability, and they now underpin many of the strongest modern image generation systems.

A major evolution of this paradigm came with Diffusion Transformers (DiT). Traditional diffusion systems relied heavily on the U-Net architecture. DiT replaced this with a Transformer-based architecture operating over latent image representations, typically produced by a VAE encoder. Conditioning information, such as class labels or text prompts, is incorporated through mechanisms like adaptive layer normalization.

Compared to classical U-Net diffusion systems, DiTs demonstrated:

• Better scaling behavior
• Stronger cross-modal transfer
• Greater architectural flexibility

They now form the foundation of many of the most advanced image and video generation systems developed in recent years.

The broader lesson across all three eras is that generative modeling is no longer merely about synthesizing aesthetically pleasing images. A model capable of sampling from the data distribution is, in a meaningful sense, learning a model of the world itself.

By modeling the joint distribution over pixels, these systems implicitly encode an enormous amount of statistical structure about:

• Objects
• Geometry
• Lighting
• Texture
• Scene composition
• Physical regularities of the visual world

This is one reason why generative pretraining, particularly diffusion-based denoising objectives, is increasingly used not only for synthesis, but also as a mechanism for representation learning.

The distinction between a generative model and a world model is becoming increasingly blurred, and that convergence represents one of the most important and intellectually exciting research directions in modern AI.

11. The 3D problem we have never quite solved

The final lecture also explores 3D vision, an area where many classical computer vision methods such as multi-view geometry, structure-from-motion (SfM), and SLAM (Simultaneous Localization and Mapping) were already highly successful long before the deep learning era. What modern deep learning is now trying to do is integrate these geometric foundations into learned visual representations.

The course introduces the standard forms of 3D scene representation: meshes (vertices connected by faces), point clouds (unordered collections of 3D points), voxel grids (the 3D analogue of pixels), and various forms of implicit representations. Among the most important recent developments are Neural Radiance Fields (NeRF) and 3D Gaussian Splatting.

Photo by Sangharsh Lohakare on Unsplash

A NeRF models a scene as a neural network that maps a 3D coordinate and a viewing direction to a corresponding color and density value. To render a novel viewpoint, rays are cast through the scene, points are sampled along each ray, the network is queried at every sampled location, and the outputs are integrated to synthesize the final image. Trained on only a few dozen photographs, NeRFs can generate remarkably realistic novel-view renderings. Their main limitation, however, is computational efficiency. Because every pixel requires repeated neural network evaluations, NeRFs are both slow to train and slow to render.

3D Gaussian Splatting takes a different approach. Instead of representing the scene implicitly with an MLP, it uses an explicit representation composed of many anisotropic 3D Gaussians. Each Gaussian stores information such as position, covariance, color, and opacity. Rendering is then performed by projecting these Gaussians onto the image plane and compositing them together. This approach is dramatically faster than NeRF, often achieving real-time rendering at high resolution, which is why it has rapidly become the default representation for many modern novel-view synthesis systems.

These methods are significant not only because of their applications, such as photorealistic scene reconstruction, 3D scanning, and immersive media, but also because of what they reveal about the broader direction of the field. Deep learning is finally building meaningful bridges back to classical 3D geometry, and the resulting hybrid systems are often more powerful than either purely learned methods or purely geometric pipelines alone.

The course explicitly emphasizes this point, noting that “a strong foundation goes a long way in research, especially in areas like Gaussian Splatting and 3D generative models.” This reflects a broader shift in the field: topics once considered “old” or “obsolete”, such as camera models, projection geometry, calibration, and multi-view geometry, are once again becoming essential prerequisites for cutting-edge research in modern 3D vision and generative AI.

12. Visual intelligence is the goal, not just computer vision

The framing the course uses for what we are actually trying to build is visual intelligence: scalable, reliable systems that can interpret visual events, answer questions about them on demand, develop a common-sense understanding of the world, and interact with it. The related terms are spatial intelligence, embodied intelligence, 3D/4D intelligence.

The course offers a useful taxonomy of tasks along a language-vs-vision axis. Tasks like “Who won the game?” or “What does this remind you of?” lean heavily on language capability and abstract reasoning. Tasks like “Where can I buy this mug?” or “Which direction leads home?” lean heavily on visual intelligence — recognizing fine-grained objects, reading street-level scenes, reasoning about geometry. Modern systems are still much better at the first category than the second.

Photo by Andy Kelly on Unsplash

This is why, despite the success of LLMs, we still don’t have domestic robots that can do the chores list for a six-year-old (load the dishwasher, fold towels, mop floors, feed the cat). It is why Elon Musk’s 2019 prediction of a million robotaxis on the road within a year has not aged well. It is why Ernst Dickmanns was demonstrating autonomous highway driving in the 1980s and we are still working out the long tail of edge cases forty years later.

The honest takeaway from twelve lectures is that computer vision is in an extraordinary moment. We have tools that would have looked like science fiction in 2010. We also have a clear-eyed view of what we haven’t solved: robust generalization, sensory grounding, embodied perception, real 3D and 4D understanding, and the integration of all of this with the language models that have rightfully grabbed the spotlight.

The unsolved problems are the interesting ones, and they are exactly the ones that need to be solved if AI is going to do more than predict the next token. If you are working in this field in 2026, you are working on the problems that matter — the ones that vision started solving 65 years ago and that the rest of AI is finally catching up to.

References:

• Hubel & Wiesel (1959) — Simple and complex cells in cat visual cortex: Original paper at PMC
• Larry Roberts (1963) — Machine Perception of Three-Dimensional Solids: MIT Lincoln Laboratory Report
• Seymour Papert (1966) — Summer Vision Project: MIT CSAIL Memo
• Canny Edge Detector (1986): Wikipedia overview
• Normalized Cuts (Shi & Malik, 1997): Berkeley paper PDF
• SIFT (Lowe, 1999): ICCV paper PDF
• Viola-Jones Face Detection (2001): Academia overview

Datasets & Benchmarks

• PASCAL VOC (2007): Dataset on Kaggle
• ImageNet (2009): Wikipedia overview

Deep Learning Architectures

• AlexNet (2012): Video explanation
• VGG (2014): arXiv paper
• GoogLeNet/Inception (2014): arXiv paper
• ResNet (2015): arXiv paper
• Vision Transformer ViT (2020): arXiv paper PDF
• ConvNeXt (2022): arXiv paper

Self-Supervised Learning

• SimCLR & MoCo: Comparison article
• Masked Autoencoders (MAE, He et al. 2022): arXiv paper

Object Detection & Segmentation

• R-CNN (Girshick, 2014): arXiv paper
• Fast R-CNN, Faster R-CNN, Mask R-CNN: MATLAB documentation

Theoretical Frameworks

• David Marr’s Vision Theory (primal sketch, 2.5D sketch, 3D model): Edinburgh overview and arXiv analysis