Why Math Nerds Cry When Deep Learning Engineers Say ‘Tensor’

I have a confession to make. As someone standing at the intersection of mathematics and AI Engineering, the word “tensor” triggers a small existential crisis for me. Insert screeching meme here.

When a deep learning engineer says “tensor”, they mean an n-dimensional array living on a GPU. A cat image: (224, 224, 3). A batch: (32, 224, 224, 3). [1] Clean and practical! A bit different from what I’ve seen in differential geometry.

I’m not complaining, I’m investigating. Because once you look past the naming crime, you find something remarkable.

Deep learning networks, under the hood of backpropagation, actually do honor the true spirit of the tensor in a way.

What is a tensor?

The word comes from the Latin tensus. It means to stretch. It was born in the language of elasticity and deformation, long before anyone thought to use it for representing data. [2] (I’m leaving a youtube video link, explaining the cleanest way possible.)

For a mathematician, a tensor is a coordinate independent object. At a point on a manifold, it’s a multilinear map from tangent and cotangent spaces to real numbers. Change your coordinate system, and the tensors components shift covariantly, contravariantly, depending on the index but the object itself remains invariant. The transformation law isn’t a side property, it’s the tensor [3].

For the deep learning practitioner, a tensor is a data container. It holds RGB pixels, weight matrices, activations. It has a shape, it gets sliced, batched and parallelized across GPU cores. When you transpose a cat image with axes=[2,0,1], the pixels do not transform covariantly. The array is just re-indexed. [4]

So why we use the same word? Two reasons. First, the engineering convenience is real: multi-dimentional arrays generalize scalars, vectors and matrices in the exact same way tensors do in mathematics; the hierarchy feels natural. [4] Second and more interestingly, the operations performed on these arrays during training turn out to be geometrically honest in a way the raw data never is.

Which brings us to one of my favorite, the Jacobian.

The Jacobian and the geometry of backpropagation

Raw data doesn’t transform geometrically. But something in a neural network does and it lives in the derivatives.

Every layer of a neural network is a differentiable function. The derivative of that function, how the output changes as the input moves, is the Jacobian matrix. And the Jacobian matrix is not a passive container.

A multivariable function, itsJacobian matrix and gradient vector

It is a linear map from the tangent space of the input manifold to the tangent space of the output manifold. It tells you exactly how the space is being stretched and deformed at each point [5].

Scuba cat represented in a vector space

Backpropagation is nothing more than a chain of Jacobian multiplications, applied in reverse. At each layer, the gradient flowing backward gets transformed by the local Jacobian covariantly. (check torch.autograd documentation especially the vector calculus part and this video here, also this. Pretty cool sophisticated stuff.)

Lets imagine our Scuba Cat has a velocity vector. If we pass this vector through a PyTorch layer that alters its direction, watch how the gradient behaves:

# The gradient flowing through a linear layer
# dL/dx = dL/dy · dy/dx ← this multiplication IS a (1,1)-tensor contraction

import torch

# We take two adjacent pixel values from your actual Scuba Cat frame 
# to represent a local 'intensity vector' from the data.
sample_patch = dl_tensor[100:102, 100].clone().detach().requires_grad_(True)

# A transformation layer (e.g., a 90° CCW rotation matrix)
W_layer = torch.tensor([[0.0, -1.0],  
                        [1.0,  0.0]])

# Forward pass: the layer alters the vector's orientation
transformed_vector = W_layer @ sample_patch
loss = transformed_vector.sum()

# Backward pass: Differential forms pull back through the graph
loss.backward()

print(f"Original Data Vector (from GIF): {sample_patch.detach().numpy()}")
print(f"Transformed Vector:              {transformed_vector.detach().numpy()}")
print(f"Gradient (dLoss/dVector):        {sample_patch.grad.numpy()}  ← Obeys covariant pullback!")
print(f"W Layer Transpose (W.T):        \n{W_layer.T.numpy()}")
print(f"Are they equal?                 {torch.allclose(sample_patch.grad, W_layer.T @ torch.ones(2))}")
print("\nThis confirms that even in DL frameworks, gradients transform using the inverse/transpose laws of tensors.")

Original Data Vector (from GIF): [0.4117647  0.37908497]
Transformed Vector:              [-0.37908497  0.4117647 ]
Gradient (dLoss/dVector):        [ 1. -1.]  ← Obeys covariant pullback!
W Layer Transpose (W.T):        
[[ 0.  1.]
 [-1.  0.]]
Are they equal?                 True

The gradient doesn’t just wander aimlessly. It flows backward precisely through the transpose of the Jacobian tracking exactly how the layer streched and rotated the coordinate space.

This is where engineering and mathematics shake hands. PyTorch never materializes the massive full Jacobian for a network with millions of parameters. Instead, it computes only the vector Jacobian product. This is what .backward() actually does [5,6].

Computational graph and autograd. Source: https://datahacker.rs/004-computational-graph-and-autograd-with-pytorch/

I will write a separate blog about autograd! But lets move .forward()

The data is flat, geometry lives in the operators.

What If the Data Behaved Like a True Tensor?

We have established that the pixel array is a passive container, when you rotate it using torch.rot90, it doesn’t transform; the values are just physically shuffled to new memory indices.

BUT what if we treated our scuba cat not as a discrete array, but as a true mathematical tensor?

In differential geometry, an image isn’t a grid of pixels. It is a continous scalar field defined over a 2D manifold. If we model the cat this way, we can change our coordinate system (our camera for example) and the physical object the cat itself remains perfectly invariant.

Lets see differences between the worlds.

In world 1 rotating the array mutates the data positions. In world 2, whether you look at the cat through a straight grid or a tilted grid, the underlying intensity at a physical point p is invariant.

It sounds great! Not really sadly. For two reasons.

Problem One: The Pitfall of Discretization

Computers deal with discrete numbers in memory addresses. The transformation laws of differential geometry assume a smooth, continous manifold. No gaps.

When we force an array to act like a true tensor during spatial operations (like 45-degree pullback above), we must interpolate the lost data. Because our data is fundamentally discrete, interpolation forces us to invent numbers that weren’t there. Every rotation introduces artifacts, destroys high-frequency signals, and slowly degrades the data. We lose information from the data, and this has a high cost when it comes to model accuracy. The geometry demands continuity that digital data simply cannot provide. [7]

Problem Two: The 6 vs 9

A true rotational tensor is invariant under coordinate rotation. The object doesn’t change; only its representation does. If we enforced this on image data, a classifier would be forced to treat a rotated image as identical to the original.

Our scuba cat is happy, dancing. Stands straight. flipped 90 degrees, it reads entirely different. the motion, direction, physical context all change. We need semantic information, not the best symetrical world. (Which there are applications for that.) Orienation carries semantic information. [4]

Freeze the Data, Curve the Operator

If you can’t make the data geometrically honest, the only remaining option is to make the operators geometrically honest instead. Leave the input data alone. And bake the transformation laws into the layers that act on them.

Welcome to Geometric Deep Learning! [4]

The core insight is that most of the structure we care about comes from the symmetries of the space the data lives on, not from the values themselves.

A standard convolutional layer already does a version of this — it shares weights across spatial positions, making the operator translation-equivariant. If the Scuba Cat moves right by ten pixels, the feature map moves right by ten pixels too. The operator tracks the geometry. The pixel array just comes along for the ride.

Equivariant CNNs take this further. Group-equivariant convolutional networks extend the convolution operation to arbitrary symmetry groups, rotations and reflections. [7] The operator is explicitly constructed so that when the input transforms, the output transforms in a predictable, lawful way. That is a transformation law. That is a tensor in the full mathematical sense, built into the architecture itself.

Gauge CNNs go even further, extending equivariance to curved manifolds surfaces where the notion of “rotation” varies from point to point, like a 3D mesh. [8] Here the connection to differential geometry becomes total: the operator is defined in terms of parallel transport, the exact machinery physicists use to describe how vectors change as you move through curved spacetime.

The Scuba Cat, floating in its flat (T, H, W, C) array, has no idea any of this is happening. It doesn't need to.

Conclusion: Embracing the Ambiguity

The next time a math purist scoffs at PyTorch’s use of the word “tensor,” or a deep learning engineer dismisses differential geometry as over-complication, you can smile. Both of them are half right.

The raw data, your batch of embeddings is not a tensor in any geometrically meaningful sense. It’s an array. It holds numbers. It doesn’t know what a tangent space is, and it doesn’t need to. When you rotate it, the pixel at [0,0] just moves to [1,2] without explanation, without a transformation law, without ceremony.

But the moment you attach a loss function and run .backward(), something changes. The gradients flowing back through the network obey the transformation laws that define a tensor in the strictest mathematical sense. The network, under backpropagation, is performing a sequence of contractions across tangent spaces, stretching, compressing, and warping the high-dimensional space until the data becomes linearly separable. [4,5]

Deep learning engineers might have stretched the definition of the word. But the behavior of the network under training honors the true spirit of the tensor, tracking the elegant geometry of change, one Jacobian at a time.

The data stays flat, keeping its pure observation safe.

Sources

[1] PyTorch documentation: torch.Tensor — “A tensor is a multi-dimensional matrix containing elements of a single data type.”

[2] Bishop & Goldberg, Tensor Analysis on Manifolds (1980) — etymology and historical context, Ch. 1.

[3] Lee, Introduction to Smooth Manifolds, 2nd ed. — Chapter 12: Tensors.

[4] Bronstein et al., Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges (2021) — Section 2.1, data as signals on sets.

[5] Bishop & Goldberg, Tensor Analysis on Manifolds — Ch. 2: contravariant/covariant transformation rules.

[6] Baydin et al., “Automatic Differentiation in Machine Learning: a Survey,” JMLR 18 (2018) — VJP/JVP formulation.

[7] Cohen & Welling, “Group Equivariant Convolutional Networks,” ICML (2016) — motivation for equivariant operators over invariant data representations.

[8] Cohen et al., “Gauge Equivariant Convolutional Networks and the Icosahedral CNN,” ICML (2019).