Statistical Mechanics & Softmax: Entropy, Ensembles, and Machine Learning

Statistical mechanics is often described as “the art of turning microscopic laws into macroscopic predictions”. Give a physicist a box with (10^{23}) particles and a list of the forces between them, and they won’t try to solve all those equations exactly. Instead, they’ll average smartly — and that’s where statistical mechanics lives.

From gas laws and blackbody radiation to neural networks and softmax, the core idea is the same:

When you can’t track every detail, describe distributions over possibilities and extract the few numbers you care about.

This article weaves together four perspectives:

• Yuxi Liu’s beautifully modern “ensemble-centric” essay on stat mech
• The Stanford Encyclopedia’s philosophical deep dive into Boltzmann vs Gibbs
• David Tong’s “Fundamentals of Statistical Mechanics” lecture notes
• A DeepLearning.AI community thread connecting softmax to the Boltzmann distribution and the partition function

1. What statistical mechanics is trying to do

Thermodynamics talks about things like temperature, pressure, entropy and mostly assumes systems are in equilibrium. It doesn’t care what individual molecules are doing.

Statistical mechanics asks:

• Given the microscopic laws (Newton, Schrödinger…), how do aggregate quantities like temperature and pressure emerge?
• Why do systems move toward equilibrium and not spontaneously un-mix themselves?
• How can the world look irreversible when the underlying laws are reversible?

The Stanford Encyclopedia calls statistical mechanics one of the three pillars of modern physics (alongside quantum theory and relativity), specifically devoted to explaining macroscopic behavior using microscopic dynamics plus probabilistic assumptions.

Tong puts it more down‑to‑earth: stat mech is our translation dictionary between “fundamental laws” and “is this stuff hot, will it explode, and what happens if I squeeze it?”

2. Microstates, macrostates, and ensembles

To talk statistically, we first need a few basic ingredients.

2.1 State space and microstate

Take a classical gas of (n) molecules. At any moment, each molecule has a position ((x,y,z)) and momentum ((p_x,p_y,p_z)). Put all of that together:

• One point in a gigantic state space (phase space) with (6n) coordinates is a microstate: a complete specification of “where everything is and how it’s moving”.

2.2 Macrostate

A macrostate is what we can describe at human scale: things like

• total energy
• volume
• pressure
• magnetization, density, etc.

Many different microstates — swapping identical particles, tiny rearrangements — can correspond to the same macrostate. Philosophers formalize this as macrostate “supervening” on microstate: every microstate fixes a macrostate, but not vice versa.

2.3 Ensemble: probability over microstates

Now comes the conceptual leap. Instead of insisting “the system is in this exact microstate”, we say:

Treat our knowledge as a probability distribution over microstates.

That probability distribution is called an ensemble. As Yuxi Liu emphasizes, in ensemble-centric statistical mechanics, the central object is precisely this distribution (\rho(x)) over state space, not a single trajectory.

• The trajectory-centric view tracks one path through state space and tries to average over time.
• The ensemble-centric view works directly with (\rho(x)), the whole probability cloud, and asks how it evolves.

Most working physicists use the ensemble-centric, equilibrium-focused formulation in daily calculations; the trajectory/philosophical issues are important but mostly live in specialized literature.

3. Entropy: from counting states to missing information

With ensembles in hand, we can define entropy. For a discrete set of microstates with probabilities (p_i), the standard formula is:

[ S = -k_B \sum_i p_i ,\ln p_i, ]

and in continuous form:

[ S[\rho] = -\int \rho(x),\ln \rho(x),dx. ]

This is the Gibbs entropy in physics and the same functional used as Shannon entropy in information theory (up to constants).

So what does it mean?

• In thermodynamics, entropy tells you about heat, irreversibility, and how much energy is unavailable for work.
• In information theory, it measures uncertainty or missing information.
• In Yuxi’s language, it measures our “actionable ignorance”: how much useful structure we haven’t yet exploited.

He gives a vivid example: a beam (or gas) that looks “random” to us has high entropy relative to our toolbox, but if you have a clever optical gadget (a corner mirror) that can perfectly undo the scattering, in a sense the underlying entropy hasn’t changed — you just learned how to extract the hidden structure.

This hints at a subtle tension:

• Is entropy an objective property of the physical state?
• Or does it depend on what coarse-graining or measurements are practically available?

Philosophers and physicists argue about this in the context of Maxwell’s demon, Gibbs’ paradox, and the “cost of information”, but for practical calculations, a pragmatic view works well: fix the macroscopic variables you care about, and entropy tells you how many compatible microstates there are, in a log-averaged sense.

4. Equilibrium as maximum entropy

Here’s the key idea that powers a huge swath of statistical mechanics:

Among all probability distributions consistent with what we know, the equilibrium ensemble is the one with maximum entropy.

Yuxi’s essay makes this explicit: given constraints (like “the average energy is (U)” or “these particle numbers are fixed”), you can pose a constrained optimization problem: maximize (S[\rho]) subject to those constraints.

Using Lagrange multipliers, you find that the unique solution — when it exists — has the general form

[ \rho(x) = \frac{1}{Z(a_1,\dots,a_n)} \exp\Big(-\sum_i a_i A_i(x)\Big), ]

where the (A_i(x)) are the constrained quantities (like energy, particle number…) and the (a_i) are Lagrange multipliers chosen to enforce the desired averages. The normalizing constant

[ Z(a_1,\dots,a_n) = \int \exp\Big(-\sum_i a_i A_i(x)\Big),dx ]

is called the partition function.

Physicists recognize this as the Boltzmann–Gibbs distribution; statisticians know these as exponential family distributions.

Two especially important cases:

• Microcanonical ensemble (NVE)
• Constraints: fixed energy, fixed particle number, fixed volume.
• Distribution: uniform over all microstates with that exact energy.
• Canonical ensemble (NVT)
• Constraints: fixed average energy, fixed particle number and volume, in contact with a heat bath at temperature (T).
• Distribution: [ p_i = \frac{1}{Z} e^{-\beta E_i}, \quad \beta = 1/(k_BT). ]

The canonical ensemble is the classic Boltzmann distribution: higher-energy microstates are exponentially suppressed.

The Stanford entry frames this in a complementary way: in Boltzmann’s combinatorial argument, the equilibrium macrostate is the one with overwhelmingly many compatible microstates — so overwhelmingly that almost every randomly picked microstate with the right energy looks macroscopically like equilibrium.

5. Fundamental theorems that keep the story self‑consistent

Yuxi highlights a few core mathematical results that justify this whole approach.

5.1 Liouville’s theorem

In Hamiltonian mechanics, Liouville’s theorem says that phase‑space volume is preserved under time evolution. If you take a “blob” of initial conditions and let them evolve, the blob might stretch and twist, but its volume stays the same.

Consequence: for an evolving probability density (\rho_t(x)), the entropy

[ S[\rho_t] = -\int \rho_t(x)\ln\rho_t(x),dx ]

stays constant as long as the dynamics is purely Hamiltonian and you don’t do coarse‑graining.

This can sound paradoxical: how can entropy increase if the fine‑grained entropy doesn’t?

Answer: the physical entropy we talk about in thermodynamics is typically a coarse‑grained or macroscopic entropy, where we lump together many microstates into one macrostate. Boltzmann’s entropy, for example, is proportional to the log of the measure of the macro‑region in state space.

5.2 Maximum‑entropy distributions are stable under the dynamics

Yuxi points out a neat corollary: if your Hamiltonian dynamics preserves the constraints you used to define equilibrium (like total energy), then a distribution that maximizes entropy under those constraints stays maximizing as it evolves.

Intuitively:

• If you’re already at equilibrium (maximum entropy under the relevant constraints),
• and the dynamics doesn’t change those constraints,
• then there’s nowhere “more entropic” to go, and you just stay at equilibrium.

This is one reason equilibrium ensembles are consistent with the microscopic equations of motion.

6. The big ensembles, in plain language

Physicists talk about several standard ensembles, which are just different choices of what’s fixed and what’s allowed to fluctuate:

• Microcanonical (NVE): isolated system
• Fixed number of particles (N), volume (V), and energy (E).
• Equal probability for all microstates with that exact energy.
• Canonical (NVT): system in contact with a heat bath
• Fixed (N), (V), and temperature (T).
• Energy fluctuates; average energy determined by (T).
• Probabilities follow (p_i \propto e^{-\beta E_i}).
• Grand canonical (μVT): exchanging particles and heat
• Fixed temperature, volume, and chemical potential (\mu); particle number fluctuates.

From a modern perspective (Yuxi, Tong, Wikipedia all stress this), these ensembles aren’t different theories; they’re different optimization problems with different constraints. You always:

1. Pick the relevant macro‑constraints.
2. Maximize entropy subject to them.
3. Get an exponential family distribution with a partition function (Z).

In the thermodynamics chapter of Tong’s notes, he then shows how familiar quantities like free energy, pressure, and heat capacity can be expressed as derivatives of (\ln Z) with respect to the parameters (like (\beta), volume, etc.), mirroring what Yuxi calls “free entropies” in his essay.

7. Boltzmann vs Gibbs: two viewpoints on equilibrium

The Stanford Encyclopedia stresses that there is no single canonical formalism in statistical mechanics. Instead, there are several overlapping frameworks; the two big ones are:

• Boltzmannian statistical mechanics (BSM)
• Focus: microstates and macrostates.
• Boltzmann entropy (S_B(M) = k \log \mu(X_M)), where (X_M) is the region of phase space realizing macrostate (M).
• Equilibrium macrostate is the one whose region takes up almost all of state space; non‑equilibrium macrostates occupy tiny regions.
• Gibbsian statistical mechanics (GSM)
• Focus: probability measures (ensembles) on phase space.
• Uses distributions like the canonical ensemble, defined via entropy maximization and the partition function.

In many practical situations, these two views agree: the Gibbs equilibrium measure is heavily concentrated on Boltzmann’s largest macro‑region.

But they lead to different ways of thinking about:

• What exactly is equilibrium?
• How do we justify using phase averages as predictions for time averages (ergodicity, typicality)?
• How does irreversibility arise from time‑reversible microscopic laws?

Yuxi’s essay deliberately brackets most of these debates and takes a Gibbs/maximum‑entropy perspective: define a state space, define constraints, maximize entropy, and treat the resulting ensemble as physically meaningful.

8. Softmax, partition functions, and neural networks

Now for the machine learning twist.

In a DeepLearning.AI discussion thread, a learner noticed that the softmax activation function looks almost identical to the Boltzmann distribution in the canonical ensemble.

Recall:

• In statistical mechanics, the partition function for discrete energy levels (E_i) at inverse temperature (\beta) is [ Z = \sum_i e^{-\beta E_i}. ]
• The probability of being in state (i) is [ p_i = \frac{1}{Z} e^{-\beta E_i}. ]

In machine learning, given logits ({z_k}), softmax defines:

[ p_j = \frac{e^{z_j}}{\sum_k e^{z_k}}. ]

If you set

[ z_j = -\beta E_j, ]

then the softmax probabilities are exactly the Boltzmann probabilities of a system at temperature (T = 1/(k_B \beta)).

In the thread, the author literally relabels:

• (Z \to a_{\text{denominator}})
• (-\beta E_i \to z_i)

and shows that the same exponential sum appears in the denominator.

8.1 Cross‑entropy vs thermodynamic entropy

The learner also compares:

• The loss for a single labeled example, (-\ln a_j), where (a_j) is the softmax probability of the correct class, with
• The entropy term ( -p_j \ln p_j) in statistical mechanics (or information theory).

They’re related but not the same:

• Cross‑entropy loss focuses on the log probability of the true class.
• Thermodynamic entropy sums over all states, weighted by their probabilities.

But both stem from the same logarithmic structure and the same exponential family distribution shaped by constraints.

8.2 “Temperature” in neural networks

In ML practice, people often introduce a “temperature” parameter (T) in softmax:

[ p_j(T) = \frac{e^{z_j/T}}{\sum_k e^{z_k/T}}. ]

• Lower (T) → more “peaked” distribution (more confident, close to argmax).
• Higher (T) → more “flat” distribution (more exploratory).

This is mathematically the same as rescaling energies in the Boltzmann distribution. In physics, higher temperature makes high‑energy states more accessible; in ML, higher temperature spreads probability mass more evenly.

8.3 Do neural networks actually undergo phase transitions?

The thread’s author goes further and asks: if softmax is so similar to statistical mechanics, can neural networks show phase transitions — sharp changes in behavior as you vary parameters (like solids melting into liquids)?

Mathematically, there are notions of phase transitions in high‑dimensional optimization and learning:

• sudden changes in generalization behavior as model size crosses a threshold,
• abrupt changes in the landscape of minima,
• “double descent” in test error as you vary model capacity.

But as one responder notes, a neural net’s “states” are not physical molecules obeying a specific Hamiltonian; the analogy is powerful but not literal.

The safest takeaway is:

Softmax is a Boltzmann distribution on logits. That’s why words like “temperature”, “energy”, and “partition function” keep reappearing in machine learning, Bayesian statistics, and probabilistic modeling.

The physical intuition (systems favor lower energy states, but fluctuations are allowed) becomes a modeling intuition (models favor lower loss configurations, but uncertainty spreads probability mass).

9. How to “think like a statistical mechanician”

Putting all of this together, here’s a rough mental recipe drawn from Yuxi’s essay, Tong’s notes, and the philosophical framing:

1. Identify the microscopic description.

• What are the fundamental degrees of freedom? Atoms in a gas? Spins on a lattice? Parameters in a model?

1. Choose your macro‑constraints.

• Total energy, particle number, magnetization, etc.
• Or in ML: constraints on averages, regularization penalties, normalization conditions.

1. Define an ensemble (a probability distribution).

• Use the maximum entropy principle: pick the distribution over microstates that maximizes entropy subject to step 2’s constraints.
• This automatically gives you an exponential family / Boltzmann‑type form with a partition function (Z).

1. Compute the partition function (Z) or approximate it.

• Exactly for simple systems.
• Using series expansions, saddle‑point methods, or Monte Carlo for more complex ones.

1. Extract observables as derivatives of (\ln Z).

• Energy, heat capacity, susceptibilities, etc., are derivatives of (\ln Z) with respect to parameters like (\beta), external fields, and so on.

1. Remember that “more is different”.

• Phenomena like temperature, phase transitions, and critical exponents don’t exist at the level of a single particle; they emerge only when you consider huge numbers of degrees of freedom.

Yuxi’s essay pushes this mindset beyond physics: once you view statistical mechanics as a general theory of probability distributions constrained by partial information, you can apply the same machinery to biology, information theory, hypothesis testing, and more.

10. Where this leaves us

Statistical mechanics sits at a crossroads:

• As physics, it explains why gases fill containers, why magnets lose their magnetization when heated, and how blackbody radiation forced us to invent quantum mechanics.
• As philosophy, it grapples with time’s arrow, the meaning of probability in a deterministic world, and the status of entropy as objective vs subjective.
• As mathematics and information theory, it’s the study of exponential families, partition functions, and large deviations.
• As a language for machine learning, it explains why softmax looks like a Boltzmann distribution and why “temperature” tuning works the way it does.

Seen through the lens of the sources you provided, the unifying thought is simple but powerful:

Describe what you don’t know with entropy, describe what you do know with constraints, and let the maximum‑entropy principle choose the fairest distribution in between.

Everything else — from gas laws to neural network logits — is just working out the consequences.

References

1. Yuxi Liu — “Statistical Mechanics” https://yuxi.ml/essays/posts/statistical-mechanics/ A long, modern, ensemble‑centric tour of classical equilibrium statistical mechanics, with many theorems, examples, and information‑theoretic interpretations.
2. Stanford Encyclopedia of Philosophy — “Philosophy of Statistical Mechanics” https://plato.stanford.edu/entries/statphys-statmech/ A conceptual and historical overview of Boltzmannian vs Gibbsian frameworks, equilibrium, irreversibility, and foundational debates.
3. DeepLearning.AI Community — “How softmax relates to statistical mechanics” https://community.deeplearning.ai/t/how-softmax-relates-to-statistical-mechanics/741456 Discussion thread explicitly connecting the softmax function and cross‑entropy loss to the Boltzmann distribution and the partition function, and musing about phase transitions in neural networks.
4. David Tong — “Statistical Physics, Section 1: The Fundamentals of Statistical Mechanics” (Canonical HTML link as given) https://www.damtp.cam.ac.uk/user/tong/statphys/statmechhtml/S1.html Content accessed via lecture‑note reproductions: introductory material on how statistical mechanics bridges microscopic laws and everyday macroscopic phenomena.