Metrizing weak convergence of distributions

Author: Yen-Chi Chen (University of Washington)
Date: 07/22/2026

• This note is modified on my lecture note of STAT 581 since the topic may be of interest to some researchers. The full lecture note can be found here.
• We will discuss several metrics/distances of distributions that are commonly used in statsitics and machine learning.
• How these metrics compare to each other is in this Section.
• A guide on what metrics to be used from my past experience is in this Section

Metrics of weak convergence

By the Portmanteau theorem, we have seen that there are a number of ways to define convergence in distribution (weak convergence). However, the Portmanteau theorem does not directly describe the convergence as a convergence in metric/distance of distributions. In this section, we discuss the concept of convergence under distances of distributions.

Let $X_n$ be a sequence of random variables with CDF $F_{X_n}$ and let $Z$ be a fixed random variable with a CDF $F_Z$. We want to study the convergence of $d(F_{X_n}, F_Z) \rightarrow 0$, where $d$ is a distance of distributions. We first introduce three core distances whose convergence is equivalent to the convergence in distribution.

Note that Lévy distance is only defined for univariate distributions.

Note that the notation $\int f(x) dF(x) = \mathbb{E}(f(X)),$ where $X$ has a CDF $F$.

The Portmanteau theorem (Theorem \ref{thm::port}) implies the following result.

This Theorem shows that if distributions converge in any of the above three distances of distributions, this implies that the corresponding random variables converge in distribution. Formally, mathematicians often refer to this as these three distances inducing the weak topology, i.e., convergence in any of these distances implies convergence in distribution (weak convergence).

Metric related to the CDF: Kolmogorov-Smirnov

The Kolmogorov-Smirnov (KS) distance is another distance for distributions. Formally, it is defined as follows.

Note that if the CDFs are multivariate distributions, $t$ will be multivariate.

Clearly, convergence in KS distance implies convergence in distribution because the Portmanteau theorem states that convergence in distribution only requires the CDF to converge at every continuous point of the target distribution while KS distance requires uniform convergence, namely,
$$d_{KS}(F,G)\rightarrow 0 \Longrightarrow d_{BL}(F,G) \rightarrow0.$$

The KS distance is in fact a stronger distance than any of the above three distances! The following example shows that a random variable can converge in distribution but not in KS distance.

While the above example shows that KS distance is a stronger distance, if the converging target is a continuous random variable, they are equivalent. This result is known as Pólya's Theorem.

Metric related to moments: Wasserstein distances

In statistics, we often only need convergence in distribution for constructing confidence intervals and performing hypothesis tests. But convergence in distribution may not enough for applications beyond these two tasks. In many modern statistical and machine learning tasks, we are often interested in moments of random variables. The convergence in distribution is in fact NOT enough for convergences of moments!

To strengthen the convergence in distribution along with convergence of moment, people often use the Wasserstein distance.

The Wasserstein distance is also called Kantorovich distance, or Monge-Kantorovich distance, or the Mallows distance. It is a very important topic in optimal transport—you can think of the joint CDF $\pi$ as a transportation map from a distribution $F$ to another distribution $G$ and we want to minimizes the cost $\|x-y\|^p$.

We have the following equivalence theorem that is particularly useful to associate the Wasserstein distance to convergence in distribution.

Due to the above theorem, we see that convergence in Wasserstein distance is a stronger notion than convergence in distribution, i.e.,
$$d_{W_p}(F_{X_n}, F_Z) \rightarrow 0 \Longrightarrow d_{BL}(F_{X_n}, F_Z) \rightarrow 0$$
but not the other way around. But their difference primarily lies in the requirement on moment convergence or not.

The $1$-Wasserstein distance has an equivalent expression known as the Kantorovich duality theorem.

Kantorovich duality theorem turns the computation of Wasserstein distance into a maximization problem. This is the foundation of the modern Wasserstein generative adversarial neural networks (W-GAN). Essentially, W-GAN just uses neural networks to approximate all possible Lipschitz continuous functions $\|f\|_L\leq 1$.

Finally, we want to note that while the Wasserstein distances are often hard to compute for multivariate distributions, there is a simple formula for the univariate case.

Metric related to density

There are other popular distances that are related to probability density/mass functions. Here we cover two popular metrics that often appear in Statistics: the total variation (TV) distance and Hellinger distance.

When the two distributions $F,G$ have PDFs $f$ and $g$, the total variation distance has an equivalent expression using the PDFs.

The total variation distance has a nice interpretation that it tells us the bound on probability difference over any region of interest. This appealing feature makes it very popular for various application such as showing the convergence of a Markov chain (and hence convergence of the Markov chain Monte Carlo). If $d_{TV}(F,G) = 0.01$, we know that for any region, the probability mass difference between the two distribution $F$ and $G$ can differ by at most $0.01$.

Hellinger distance can be viewed as putting the square root of a PDF on an infinite dimensional sphere and using the Euclidean distance in that sphere. It is related to the concept of Quadratic Mean Differentiation (QMD).

Comparing metrics

Here is how all these metrics are related in $\mathbb{R}$ (or $\mathbb{R}^d$).

You may notice that the first inequality implies that $d_{BL}$ converges to $0$ if and only if $d_{Prok}$ converges. The fourth inequality shows that $d_{Levy}$ converges to $0$ when $d_{Prok}$ converges. Since all three of them induce the same weak topology (converge in distribution), you may be wondering if convergence in $d_{Levy}$ implies convergence in $d_{Prok}$. Unfortunately, this is NOT the case. Here is a counterexample that $d_{Levy}$ goes to $0$ while $d_{Prok}$ does not.

The above example seems to offer a conflicting example to the Portmanteau theorem but this is not true. The critical part of the above example is that we are considering two sequences. If $G_n$ is a fixed probability distribution $G$, then we have
$$d_{Levy}(F_n,G)\rightarrow 0 \Longleftrightarrow d_{Prok}(F_n,G) \rightarrow 0.$$
Formally, the above equation means that $d_{Levy}$ and $d_{Prok}$ are topologically equivalent. The topological equivalence only requires open sets, which is defined via a fixed convergent target.

For $d_{BL}$ and $d_{Prok}$, since we know that the convergence of one implies the convergence of another even for two sequences of distributions, they are uniformly equivalent. Note: the formal definition of uniformly equivalent is that there exists functions $\psi(t)\rightarrow 0 ,\phi(t)\rightarrow 0$ when $t\rightarrow0$ such that $\phi(d_1)\leq d_2 \leq \psi(d_1)$. For $d_{BL}$ and $d_{Prok}$, $\phi(t) = \frac{1}{2}t$ and $\psi(t) = \sqrt{t}$. Clearly, uniform equivalence implies topological equivalence.

When to use which distance?

With so many distances available to quantify the distance between distributions, selecting the right one depends entirely on the downstream statistical task. If the goal is simply to establish asymptotic normality to construct a basic confidence interval or hypothesis test, any distance that metrizes the weak topology is sufficient.

However, when we need finite-sample bounds, exact coverage rates, or control over moments, the choice of metric becomes critical. Here is general guidance on when to deploy which metric from my personal experience:

• Coverage bounds: Kolmogorov-Smirnov distance.
  While weak convergence guarantees that a bootstrap confidence interval is asymptotically valid, it does not quantify the finite-sample accuracy of that coverage. To bound the coverage error, we need the KS distance. Because KS directly measures the maximum vertical discrepancy between CDFs, it is the natural metric for theorems like the Berry-Esseen bound, which dictates exactly how fast the coverage deficiency shrinks (e.g., $O(n^{-1/2})$).

• Interpretation and state-space mixing: Total variation distance.
  The TV distance provides a strict, highly interpretable probability bound: a TV distance of $0.01$ guarantees that the two distributions will differ by at most $1\%$ on any measurable set, no matter how complex. This makes TV the gold standard for bounding the mixing times of Markov chains, as it ensures the chain has properly explored every subset of the state space relative to its stationary distribution.

• Moment convergence: Wasserstein distance.
  If a downstream task requires not just the probabilities to align, but the expectations (or variances) to converge, the Wasserstein distance is required. $W_p$ convergence guarantees the convergence of the $p$-th moment. The Kantorovich duality theorem is particularly powerful for deriving rates of convergence for generative models (like W-GANs or diffusion models) and for bounding distributional shifts.

• Multi-stage estimators: Bounded Lipschitz.
  For complex, multi-stage estimation procedures, the BL distance is powerful. Suppose an estimator behaves perfectly in a "good" scenario (stage one succeeds), but becomes wildly unstable in a "bad" scenario (stage one fails). If we evaluate this using an unbounded distance like $W_1$, the extreme values of the bad scenario could cause the distance to explode. However, because the BL distance restricts test functions to $\|f\|_\infty \leq 1$, the integral over the bad event is strictly bounded by the probability of the bad event occurring. As long as the probability of the first stage failing shrinks to $0$, the BL distance nullifies the bad scenario and guarantees weak convergence.

• Likelihood approximations and minimax lower bounds: Hellinger distance.
  Because the Hellinger distance operates on the square root of the probability density, it represents the geometry of maximum likelihood theory. In parametric statistics, it is the foundation of QMD. By defining smoothness under the Hellinger distance, QMD allows us to define Fisher information and establish local asymptotic normality even when densities are not pointwise differentiable. We will formally cover this in the next lecture. Furthermore, the Hellinger distance of $n$ IID observations enjoys an elegant form of product:
  $$d^2_{H}(F^n,G^n) = 2\left(1- \left(1 - \frac{d^2_{H}(F,G)}{2}\right)^n\right) = 2 (1- \rho(F,G)^n),$$
  where $\rho(F,G) = \int \sqrt{f(x) g(x)}dx$ is the Hellinger affinity and $f,g$ are the PDFs of $F$ and $G$. This product form leads to minimax lower bounds via Le Cam's method.