Introduction

This tutorial builds an intuitive and rigorous understanding of probability theory from the ground up. We start with the question “what is a random variable?” and end at cumulants and the Law of Total Cumulants — a powerful generalization that reveals deep structure in how information decomposes.

The central thread is the signal-to-noise ratio: given two random variables X and Y, how much does knowing X tell us about Y? Answering this question requires us to build up conditional expectations, conditional variance, and the Law of Total Variance.

What Is a Random Variable?

A common misconception is that a random variable is a function that returns probabilities. That’s actually a probability mass function (PMF) or probability density function (PDF). A random variable is something more fundamental.

The Setup

A probability space together with a random variable has four ingredients:

1. Sample space Ω: The set of all possible outcomes of an experiment
2. Event space ℱ: A σ-algebra of measurable subsets of Ω to which probabilities may be assigned
3. Probability measure P: Assigns probabilities to events in ℱ
4. Random variable X: Ω → ℝ: A measurable function that assigns a number to each outcome

Example: Rolling a Die

Multiple Random Variables on the Same Space

You can define multiple random variables on the same sample space, each extracting different information:

• X(ω) = the number showing (1, 2, 3, 4, 5, or 6)
• Y(ω) = 1 if even, 0 if odd

Expectation

The expectation (or expected value) of a discrete random variable X is the probability-weighted average of all its possible values:

Here the sum runs over all possible values x that X can take, and p(x) = P(X = x) is the probability of each value.

Example: Fair Die

Expectation of a Function of X

More generally, for any function g(X):

Conditional Expectation: E[Y|X]

This is where things get nuanced. There are two related but distinct concepts sharing very similar notation:

1. E[Y|X=x] — A Specific Number

For discrete variables and a value x with P(X=x) > 0, this is the expected value of Y given that X takes that specific, fixed value:

The sum is over all values of y (not x!), weighted by the conditional probability of y given x. This produces a single number.

2. E[Y|X] — A Random Variable

This is a function of X. Since X is random, E[Y|X] is also random. It takes the value E[Y|X=x] when X = x. As usual with conditional expectations, its values on events of probability zero can be chosen without changing the random variable probabilistically.

Formally, define:

Concrete Example

Suppose X can be 0 or 1, each with equal probability P(X=0) = P(X=1) = 0.5, and:

• When X = 0, Y is uniform on {1, 2, 3} (each with probability 1/3)
• When X = 1, Y is uniform on {4, 5, 6} (each with probability 1/3)

Step 1: Compute E[Y|X=x] for each x

Step 2: Assemble the random variable E[Y|X]

E[Y|X] is a random variable — a function of X. It takes the value g(X) where g(0) = 2 and g(1) = 5. Since X is random, we ask: what are the probabilities that E[Y|X] takes each of its values?

• E[Y|X] = 2 when X = 0, which happens with probability P(X=0) = 0.5
• E[Y|X] = 5 when X = 1, which happens with probability P(X=1) = 0.5

The Law of Total Expectation

Since E[Y|X] is a random variable, we can compute its expectation:

From the example above:

Variance

Variance measures the expected squared deviation from the mean. Think of it as the “expected variance” — how far values typically fall from the center, on average.

where μ = E[X].

An Equivalent Formula

Conditional Variance: Var(Y|X)

Following the same pattern as conditional expectation:

Var(Y|X=x) — A number

where μ_Y|x = E[Y|X=x].

This measures: “Given that X = x, how spread out is Y around its conditional mean?”

Var(Y|X) — A random variable

Just like E[Y|X], this is a function of X that returns Var(Y|X=x) when X = x. It is “waiting” for you to tell it what X is.

Signal-to-Noise Ratio & The Law of Total Variance

The Law of Total Variance is one of the most powerful decomposition results in probability. It breaks the total variance of Y into two interpretable components:

This decomposes total variance into:

A Variance-Based Signal-to-Noise Ratio

For prediction under squared-error loss, a natural signal-to-noise ratio compares variation in the conditional mean with the remaining conditional variation:

High SNR: shifts in the conditional mean explain most of Y’s variability — the signal dominates.
Low SNR: residual conditional variation dominates shifts in the conditional mean. X may still reveal other features, such as the conditional variance or shape of Y. If the denominator is zero, the ratio is infinite when the numerator is positive and undefined when both terms are zero.

Connection to Covariance and Correlation

To make the relationship between signal, noise, and dependence precise, we need two more definitions:

• Cov(X, Y) > 0: When X is above its mean, Y tends to be above its mean too
• Cov(X, Y) < 0: When X is above its mean, Y tends to be below its mean
• Cov(X, Y) = 0: No linear relationship (but nonlinear dependence may still exist!)

The problem with covariance is that its magnitude depends on the units of X and Y. To get a unit-free measure, we normalize:

where σ_X = √Var(X) and σ_Y = √Var(Y), assuming both variances are finite and nonzero.

When ρ = ±1 (perfect linear relationship), Y is a deterministic linear function of X, and the Law of Total Variance becomes extreme:

• Var(Y|X) = 0 — No residual variance (no noise)
• Var(E[Y|X]) = Var(Y) — All variance is “explained” (pure signal)

Analogy with Entropy

Why Do the Total Laws Have Different Forms?

Something curious has happened. We’ve seen two “total” laws, but they look very different:

One term. Clean. Simple.

Two terms. Why does variance need an extra term that expectation doesn’t?

The Root Cause: Squared Quantities Don’t Decompose Cleanly

Expectation is linear: E[X + Y] = E[X] + E[Y], always. But variance involves squared deviations, and squaring produces cross-terms.

For independent X and Y, the 1st moment (mean) adds cleanly:

But the 2nd raw moment does not:

The cross-term 2E[X]E[Y] ruins the additivity. Even though X and Y are independent, the 2nd raw moment of their sum is not the sum of their 2nd raw moments.

This pattern continues: the 3rd, 4th, and higher moments become increasingly messy to decompose. This raises a natural question: is there a systematic framework that explains why each “total law” takes the form it does?

The answer is yes — it comes from the theory of moments,generating functions, and cumulants. Understanding these tools will reveal that the Laws of Total Expectation and Total Variance are the first two cases of a beautiful general pattern: the Law of Total Cumulants.

Moments: The Shape of Distributions

To understand why the total laws have different forms, we need to formalize the idea of “higher-order” properties of distributions. Moments are the tool for this.

Why “Moments”?

Raw Moments

The nth raw moment is simply the expected power:

Central Moments

Central moments measure deviations from the mean:

Special Case: The Gaussian Distribution

Within the Gaussian (normal) family, a distribution is completely defined by its mean and variance. All higher moments are determined by these two parameters:

• Skewness (3rd central moment divided by σ³) = 0 (perfectly symmetric)
• Excess kurtosis (4th cumulant divided by σ⁴) = 0 (reference tail weight)
• All odd central moments = 0; all even central moments are determined by the variance

Taylor Expansion & the Moment Generating Function

We now know that moments characterize distribution shape, but raw moments don’t add cleanly for independent variables. To find quantities that do add, we need a tool that connects moments to derivatives: the Taylor expansion.

The Problem Taylor Expansion Solves

Imagine you have a complicated function f(x) — like sin(x) or e^x. These are hard to compute by hand. What if you could approximate them using only addition, subtraction, and multiplication (i.e., polynomials)?

The Formula

For a function analytic around a = 0, its Taylor series there (called a Maclaurin series) is:

Think of it as matching derivatives:

• The constant term matches f(0)
• The linear term matches the slope at 0
• The quadratic term matches the curvature at 0
• Each term corrects for higher-order behavior

Within the radius of convergence of an analytic function’s Taylor series, its partial sums converge to the function as more terms are included, although the error need not decrease after every individual term. Smoothness alone does not guarantee convergence to the original function.

Example: e^x

For f(x) = e^x, every derivative is e^x, so every derivative at 0 is 1:

The Moment Generating Function

Now we connect this to probability. When E[e^tX] is finite for t in an open interval around 0, define the Moment Generating Function (MGF):

Why e^tX?

Three powerful reasons:

1. Self-derivative property: Differentiating e^tX with respect to t produces X · e^tX. Differentiate again: X² · e^tX. At t = 0, these become E[X], E[X²], and so on. The exponential “spits out” powers of X when you differentiate. No other function does this so cleanly.
2. Sums become products: e^a+b = e^a · e^b. For independent X and Y: M_X+Y(t) = M_X(t) · M_Y(t).
3. Connection to Laplace transforms: The MGF is closely related to the Laplace transform already used in physics and engineering.

How the MGF Encodes Moments

Apply the Taylor expansion of e^tX:

Under this existence condition, take expectations term by term:

The moments appear as coefficients! Taking derivatives extracts them:

Cumulants: Quantities That Add for Independent Sums

We can now return to the motivating question: why do the Laws of Total Expectation and Total Variance have different forms? The MGF showed us that moments are derivatives, and that independent variables’ MGFs multiply. The final insight is that taking the logarithm converts products into sums — giving us quantities called cumulants that add for independent variables whenever the required generating functions and derivatives exist.

The Cumulant Generating Function

The nth cumulant κ_n = K⁽ⁿ⁾(0).

Why Does the Logarithm Work?

For independent X and Y:

Cumulants vs. Moments

The Law of Total Cumulants — Answering the Motivating Question

We can now finally answer why the two total laws have different forms. They are the first two cases of a general pattern: the Law of Total Cumulance sums over the set partitions of the conditioned arguments. For repeated copies of the same variable, symmetric partition terms can be grouped; the first three orders produce one, two, and three grouped terms, respectively, while the fourth already produces five grouped term types.

1st cumulant (mean) — 1 term

2nd cumulant (variance) — 2 terms

3rd cumulant — 3 terms

Var(Var(Y|X)) — Heteroscedasticity

An interesting quantity: the variance of the conditional variance. This measures whether the noise level itself depends on X; it is zero exactly when the conditional variance is constant almost surely.

For example, suppose X = 0 and X = 1 are equally likely:

• When X = 0, Y is uniform on {0, 1} → Var(Y|X=0) = 0.25
• When X = 1, Y is uniform on {0, 10} → Var(Y|X=1) = 25

Here Var(Y|X) takes the values 0.25 and 25 with equal probability, so Var(Var(Y|X)) = 153.14. This is heteroscedasticity — the noise level varies with the input.

The 3rd cumulant decomposition can capture one aspect of this: the covariance term Cov(κ₁(Y|X), κ₂(Y|X)) measures whether the noise level correlates with the prediction. Heteroscedasticity can also occur when this covariance is zero, however. The quantity Var(Var(Y|X)) appears directly in the 4th cumulant decomposition, so changing noise levels show up naturally in higher cumulant decompositions.

The Big Picture