Variational Autoencoders — the historical path
A 70-year story. VAE is not a single invention; it's the point where probabilistic modeling, Bayesian inference, and deep learning met. This document tells you which ideas had to be discovered, in which order, for the VAE to be possible.
This is the wide and shallow companion to vae.html. The other doc derives every piece from first principles. This one places the pieces in history — who, when, why, and what they were responding to. Read this for context; read the other for understanding.
1 The big problem — modeling realitysetting the stakes
The central question of generative modeling is one line:
- Generation.Sample $x \sim p$ to produce new examples — new images, new sentences, new molecules.
- Density / anomaly detection.Score $p(x)$ for new inputs. Low score = unlikely under the data distribution = potentially anomalous.
- Compression.Shannon's source coding theorem: optimal compression needs $-\log p(x)$ bits per sample. A good density model is a good compressor.
- Denoising / inpainting.Reconstruct corrupted $x$ as $\arg\max_x p(x \mid \tilde x)$ — Bayes' rule with a noise model.
- Representation learning.If $p(x)$ is built through latent variables $z$, then $z$ is a learned representation of $x$ — useful for downstream tasks.
- World models for RL & robotics.An agent that has $p(x_{t+1} \mid x_t, a_t)$ can simulate the future and plan against it without acting in the real world.
- Bayesian reasoning under uncertainty.Anywhere we want $p(\text{cause} \mid \text{effect})$, we need a generative model $p(\text{effect} \mid \text{cause})$ first.
Every item above reduces to "score or sample from $p$". The case for spending decades on this problem is exactly that: one good $p(x)$ unlocks all of it.
2 The first idea — latent variables1900s–1960s
Direct modeling of $p(x)$ for high-dimensional data is hopeless. Pixels in a face image are wildly entangled — you can't write down a joint density over them. The first foundational idea was to assume the entanglement comes from something simpler underneath.
Real-world high-dimensional data does not fill its ambient space. It lies on (or near) a low-dimensional manifold, parameterized by a handful of underlying factors. A 200,000-pixel face image varies with maybe 50 underlying things — pose, lighting, identity, expression.
Spearman proposed that performance on different mental tests was driven by a single underlying latent factor — "general intelligence" or $g$. This is essentially the first latent variable model in the statistical literature. The math was crude by modern standards, but the intuition — "many observations from one hidden cause" — is exactly the framework that VAEs use a century later.
3 Pre-deep-learning latent models1960s–2000s
Once people accepted "observations come from hidden causes," they spent ~40 years building increasingly sophisticated tools to fit such models.
3.1 Factor analysis — the linear ancestor
3.2 PCA → probabilistic PCA — the closest classical ancestor
PCA (Pearson 1901, Hotelling 1933) was originally a deterministic linear projection — no probability involved. Probabilistic PCA (Tipping & Bishop 1999) reframed it as a special case of factor analysis where the observation noise is isotropic: $\Psi = \sigma^2 I$.
The first model that's truly structurally identical to a VAE — linear decoder version. Same latent variable framework, same Gaussian prior $p(z) = \mathcal{N}(0, I)$, same Gaussian observation model $p(x \mid z) = \mathcal{N}(Wz + \mu, \sigma^2 I)$. The only difference: PPCA's decoder is a linear map; the VAE's decoder is a neural network.
| PPCA (1999) | VAE (2013) | |
|---|---|---|
| Prior $p(z)$ | $\mathcal{N}(0, I)$ | $\mathcal{N}(0, I)$ |
| Decoder $p(x\mid z)$ | $\mathcal{N}(Wz + \mu,\; \sigma^2 I)$ | $\mathcal{N}(f_\theta(z),\; \sigma^2 I)$ (neural net $f_\theta$) |
| Inference $p(z\mid x)$ | closed-form Gaussian | amortized $q_\phi(z \mid x)$ (neural encoder) |
| Training | EM or closed-form MLE | SGD on ELBO + reparameterization |
3.3 Gaussian mixture models — discrete latents
The other branch of classical latent variable models: $z$ is discrete, indexing a finite mixture of simple distributions.
The general framework: alternate between (E-step) computing the posterior over latents given the current parameters, and (M-step) updating the parameters as if the posterior were correct. Provably monotonic in the likelihood. EM is the direct algorithmic ancestor of variational inference — it's exactly coordinate ascent on the ELBO, but with $q$ set to the true posterior. When the posterior is intractable, EM breaks; VI is what generalizes EM to that case.
3.4 Helmholtz machines & wake-sleep — the unsung VAE ancestor
A neural network with two parts: a recognition network (input → latent) and a generative network (latent → output). Trained by the wake-sleep algorithm: in the "wake" phase, run the recognition network on data and update the generative network; in the "sleep" phase, sample from the generative network and update the recognition network to invert it.
Sound familiar? This is the encoder-decoder architecture, almost two decades before VAEs. The reason it didn't become "the" generative model: wake-sleep optimizes two different objectives for the two networks, with no guarantee they're compatible. The VAE's contribution — the ELBO — is the single joint objective that the Helmholtz machine was missing.
4 The inference bottleneckthe central problem
Build a latent variable model. Now you actually want to use it. Two questions:
- Marginal likelihood — $p(x)$."How likely is this data point?" Needed for training (MLE) and for scoring new data.
- Posterior — $p(z \mid x)$."Given this observation, what hidden cause produced it?" Needed for inference, for representation learning, and for everything Bayesian.
Both queries are linked by Bayes' rule:
Every algorithm in the rest of this document — MCMC, EM, VI, VAE, normalizing flows, diffusion — is a different attempt to dodge or approximate that single integral. If $p(x)$ were tractable, none of this would have been invented. It isn't, so all of it was.
5 The fork — sampling vs approximation1950s–1990s
Faced with the intractable integral, two parallel traditions developed. They produced different algorithms, different research communities, and (eventually) different generative model families.
| family | strategy | landmark | modern descendant |
|---|---|---|---|
| Sampling-based | Build a Markov chain whose stationary distribution is the posterior. Sample from it. | Metropolis–Hastings (1953) | HMC, MCMC, diffusion models |
| Variational | Pick a tractable family $\mathcal{Q}$. Optimize for the best $q \in \mathcal{Q}$. | Variational Bayes (1990s) | VAE, normalizing flows |
5.1 The sampling path — MCMC
Slow. Building a chain that mixes well over a high-dimensional, multi-modal posterior can take thousands or millions of steps per sample.
Hard to diagnose. You can't easily tell when the chain has converged. You compare runs and hope.
Doesn't compose with SGD. Neural network training needs gradients of an objective; MCMC produces samples, not objectives. The two paradigms don't fit together cleanly.
None of this means MCMC was abandoned — it powers modern Bayesian deep learning, probabilistic programming (Stan, PyMC, Pyro), and arguably diffusion models are MCMC reborn. But for the specific problem of "train a big neural latent variable model by gradient descent," MCMC was the wrong tool.
5.2 The variational path
The idea: stop trying to compute the posterior, start optimizing for it. Pick a parametric family $\mathcal{Q}$ of distributions we can work with (Gaussians, mean-field factorizations). Find the $q \in \mathcal{Q}$ closest to the true posterior. The "closest" is measured by KL divergence, and the optimization is doable because — as we'll see in Section 6 — we never need to know the true posterior to do it.
The name "variational" comes from the calculus of variations: optimization where the variable is itself a function (here, a distribution), not a number.
6 The ELBO — variational inference, properlyheart of the math
The variational program needs an objective. You can't minimize $\mathrm{KL}(q \,\|\, p(z \mid x))$ directly because you don't know $p(z \mid x)$ — that's why we're approximating it in the first place. So you derive an objective that has the same minimizer but only uses things you can compute.
The right-hand side is the ELBO:
$$\mathcal{L}(q) = \mathbb{E}_{q(z)}[\log p(x \mid z)] \;-\; \mathrm{KL}(q(z) \,\|\, p(z)).$$VAEs did not invent the ELBO. The ELBO had been the standard objective of variational Bayes for nearly 20 years before VAEs existed. Hinton was using it in the 1990s. What VAEs invented was a way to make the ELBO trainable by gradient descent with neural networks. The ELBO itself is an inheritance from the Bayesian statistics tradition, not a deep-learning contribution.
The original tractable variational family: assume $q(z) = \prod_j q_j(z_j)$ (fully factorized). For models in the exponential family, the coordinate-ascent updates have closed forms. Mean-field VI is what made variational inference deployable in the pre-neural-net era — every classical Bayesian topic model fit by VI used it.
7 The graphical models era1990s–2000s
Through the 1990s and 2000s, the dominant paradigm for probabilistic modeling was graphical models: structured combinations of conditional distributions, with topology specified by a graph.
- Bayesian networksDirected graphical models. Each node is a variable, each edge a conditional dependency. Pearl's 1988 book formalized them. Inference: belief propagation, junction tree, variational methods.
- Hidden Markov Models (HMMs)Sequence models with discrete latent states. The dominant model for speech recognition for ~30 years. Trained by EM (Baum–Welch).
- Latent Dirichlet Allocation (LDA)Blei, Ng, Jordan (2003). The topic-modeling success story that made variational inference famous outside theoretical statistics. Each document has a latent distribution over topics; each topic is a distribution over words. Fit by mean-field VI.
- Conditional Random Fields (CRFs)Lafferty, McCallum, Pereira (2001). Undirected discriminative models. Standard for structured prediction (NER, sequence labeling) before deep learning.
- Variational EMThe combination: when the E-step in EM is intractable, use VI to approximate the posterior. Bridge from EM to fully variational methods.
Graphical models were shallow. The conditional distributions $p(x_i \mid \text{parents})$ were almost always simple (linear-Gaussian, categorical, exponential family) so that closed-form updates existed. Trying to use arbitrary functions for these conditionals — say, deep neural networks — broke all the inference machinery. The expressiveness of the model and the tractability of inference were locked in opposition.
8 Deep learning meets probability2006–2013
Two streams of research converged in the early 2010s. They had been mostly disjoint communities.
| thread | strength | missing |
|---|---|---|
| Probabilistic modeling (statistics) | Principled objectives, uncertainty handling, Bayesian framework | Restricted to simple model families; couldn't scale to images |
| Deep learning (ML) | Flexible function approximation, gradient-based training at scale | No structured objective beyond classification loss; no uncertainty |
Each had what the other needed. The VAE is one of the first major fruits of their meeting.
An autoencoder is a neural network trained to reconstruct its input through a bottleneck. The bottleneck forces a compressed representation. Used for dimensionality reduction, representation learning, and (in stacked / deep form) pre-training for classification tasks before ReLU + good initialization made supervised pretraining unnecessary.
The autoencoder is not probabilistic — there's no $p(x)$, no $p(z)$, no sampling. It's just a deterministic encode-decode pipeline minimizing reconstruction error. The "auto-encoding" in "auto-encoding variational Bayes" is the bridge: the VAE looks like an autoencoder, but its objective comes from variational Bayes. The name is the only thing borrowed from the autoencoder tradition.
The first deep generative model that worked, sort of. Stacks of restricted Boltzmann machines (RBMs) trained greedily layer by layer. Often credited with launching the deep learning revival. Training was awkward (contrastive divergence, not real MLE). Sampling was MCMC and slow. Replaced almost overnight by VAEs and GANs once those appeared in 2013–14, but it set the stage by establishing "deep + probabilistic" as a viable direction.
9 Why naive neural latents failed~2012–2013
Once researchers started trying to combine neural networks with latent variable models, they ran into a specific, brutal obstruction.
Suppose your model is: $z \sim q_\phi(z \mid x)$, then $\hat x = f_\theta(z)$. To train by SGD you need
$$\nabla_\phi \mathbb{E}_{z \sim q_\phi}[\,\ell(\hat x, x)\,].$$The gradient with respect to $\phi$ — but $\phi$ controls the distribution being sampled from. The sampling step is a discrete, non-differentiable bottleneck. Backprop hits it and stops.
Use the identity (also known as REINFORCE):
$$\nabla_\phi \mathbb{E}_{q_\phi}[f(z)] = \mathbb{E}_{q_\phi}[\,f(z)\, \nabla_\phi \log q_\phi(z)\,].$$Mathematically unimpeachable. Practically, the variance is enormous — the estimator multiplies the loss by a (potentially large) log-derivative, and the resulting gradient is so noisy that training is unstable. Lots of variance-reduction tricks were tried (baselines, control variates) but none were as clean as what came next.
The REINFORCE / score-function estimator is the same algorithm as the policy gradient theorem in RL (Williams 1992). RL practitioners had been wrestling with its variance for two decades by the time generative-model researchers ran into it. The variance-reduction toolbox in RL (baselines, advantage functions, control variates, later PPO and GAE) is exactly what variational inference researchers tried to reuse — with limited success.
VAEs were lucky in a specific way: they used a Gaussian variational posterior, which is reparameterizable. RL with discrete actions (most of language-model RL, game-playing) does not have this luxury, which is why REINFORCE-style estimators remain a way of life in those communities.
10 The 2013 breakthrough — reparameterizationthe synthesis
The seminal paper. Note the title: "Auto-Encoding Variational Bayes," not "the variational autoencoder." Kingma and Welling were doing variational Bayes — they presented the architecture as a consequence, not the headline. The neural-net-as-encoder/decoder structure falls out of the math.
Two months after Kingma & Welling's preprint, Rezende et al. at DeepMind published essentially the same idea independently. "Stochastic Backpropagation and Approximate Inference in Deep Generative Models." The reparameterization trick appears under the name "stochastic backpropagation"; the ELBO derivation is the same. Two groups, two continents, two months apart, same idea — a classic sign that the field was ready for it.
The variational posterior is chosen to be Gaussian: $q_\phi(z \mid x) = \mathcal{N}(z \mid \mu_\phi(x), \sigma_\phi^2(x))$. Sampling $z \sim q_\phi(z \mid x)$ can be rewritten as:
$$z = \mu_\phi(x) + \sigma_\phi(x) \cdot \epsilon, \quad \epsilon \sim \mathcal{N}(0, I).$$Reparameterization moves the stochasticity to a parameter-free noise source $\epsilon$, leaving the rest of the computation deterministic. A latent variable model becomes a regular neural network with one extra input ($\epsilon$). Trainable by SGD. Variance dramatically lower than REINFORCE. The Helmholtz machine, the ELBO, the encoder-decoder architecture — all three pieces had existed for a decade or more. Reparameterization is what made them fit together into a thing you could train.
11 After VAEs (2014–now)the modern landscape
Once the VAE worked, the floodgates opened. Within a year, the field had multiple flavors of deep generative model — each making different tradeoffs in the (scoring vs sampling vs expressiveness) space we saw in vae.html §3.5.
VAEs lost the generation-quality race to GANs (briefly) and then to diffusion models (definitively). But the framework — amortized variational inference, encoder-decoder structure, latent representations, the ELBO — is everywhere. Stable Diffusion is a VAE encoder + diffusion in latent space + VAE decoder. Modern audio codecs (Descript, EnCodec) are VQ-VAE descendants. The VAE as a standalone generator is rare today; the VAE as a representation-learning and tokenization tool is in every foundation model.
12 The family treethe one diagram
Everything in one picture. Three colored streams converge at "VAE":
The VAE = PPCA's latent variable framework + variational Bayes' ELBO + the Helmholtz machine's encoder-decoder architecture + the reparameterization trick that made it all trainable. Each ingredient existed before 2013. The contribution of Kingma–Welling and Rezende-et-al. was identifying the one missing piece (reparameterization) and showing the resulting combination was practical.