A dataset is a finite collection of observations. By itself, it tells us what happened in one experiment, but it does not yet provide a compact rule for what similar experiments might produce. A statistical model supplies that rule by proposing a family of probability distributions. Different parameter values select different members of the family.
Once a family has been chosen, fitting becomes a search problem. The observations remain exactly where they are; only the candidate model changes. The first question is visible before it is algebraic: which candidate places its probability mass or density where the observed values actually landed?
One running dataset · one Gaussian family
Five observed values stay fixed while three candidate means change
observed (29, 31, 32, 34, 34)known width σ = 2unknown truth μ* is not shown
Candidate A: μ = 29.5normalized density over possible x · area = 1relative joint likelihood = 0.0201
Candidate B: μ = 30.4normalized density over possible x · area = 1relative joint likelihood = 0.2019
Candidate C: μ = 32.0normalized density over possible x · area = 1relative joint likelihood = 1.0000
Each curve is one normalized Gaussian probability density over every possible measurement x, using its labeled candidate mean and σ=2. The five gold dots mark the observations that actually occurred. The vertical gold lines carry each observation up to its density height; the marked points are the five values Li(μ)=p(xi∣μ,σ=2). Multiplying those heights produces the candidate's joint likelihood because it evaluates all five observations together. Dividing by the largest such product gives the displayed relative joint likelihood. Candidate C is the data-derived maximizer, not merely the strongest of the three displayed examples. The unknown true mean was not used in this calculation.
The figure suggests a comparison rule. Evaluate every candidate at the observations, combine those evaluations, and prefer the candidate with the strongest combined support. The method that carries out precisely this search is called maximum likelihood estimation.
THE ESTIMATION PROBLEM
data D=(x1,…,xn),model family M={pθ:θ∈Θ},unknown θ
What MLE can and cannot choose
MLE chooses the parameter inside the supplied family that best fits the observed data. With a finite sample, that parameter is an estimate of the unknown data-generating value, not a certificate that the truth has been recovered. Additional independent observations generally reduce the estimate's sampling uncertainty when the model is well specified, but MLE cannot establish from likelihood alone that the chosen family adequately describes reality. If the family is wrong, more data can concentrate the estimate around the best approximation available inside the wrong family. Model checking therefore remains a separate responsibility.
2
Reverse the generative question
Probability becomes a function of the candidate
A model is normally read forward: choose θ, then ask which values of the random variable X may occur. Estimation begins after an outcome has already occurred. It freezes that observed value and compares the parameter values that might have produced it.
The same model expression, read in two directions
What is allowed to vary changes the question
1Fix a model
θ fixedx varies
Ask which observations the chosen model would tend to produce.
forward sampling question
⇄
2Fix what happened
x observedθ varies
Ask which member of the model family makes that observation most plausible.
inverse fitting question
No probability formula has been changed. Only its role has changed: the observation becomes fixed evidence and the parameter becomes the candidate being compared.
This change of role motivates separate notation. When x varies and θ is fixed, pθ(x) is a probability mass function or probability density. When the observed x is fixed and θ varies, the same numerical expression is the likelihood function:
ONE OBSERVATION
L(θ;x)≡pθ(x)
Probability modelpθ(x)
Fix the parameter and compare possible outcomes. It must normalize over x.
Likelihood functionL(θ;x)
Fix the observed outcome and compare candidate parameters. It need not normalize over θ.
For continuous data, a density value may exceed one and is not the probability of observing one exact real number. Likelihood comparisons remain meaningful because every candidate is evaluated with the same measurement scale; ratios compare how much density the candidates place near what was observed.
3
Let every observation contribute
Independent evidence multiplies
The opening figure fixed one candidate mean and drew its probability density over possible measurements x. Its relative joint-likelihood number came from evaluating that one density at the five observed values and multiplying the five heights. The laboratory below exposes the same calculation from the other direction: each observation is now fixed while the candidate mean μ varies along the horizontal axis.
Each colored curve therefore tracks one factor Li(μ)=p(xi∣μ,σ=2) across every candidate mean. One observation leaves many candidates plausible; multiplying all five factors rewards a candidate that explains the complete dataset at once.
The five observed values become five likelihood contributions
Move one candidate mean through evidence supplied by the actual dataset
The curves remain fixed because the observations and known noise width remain fixed. Moving the slider moves only the dashed candidate line. Its colored intersections select one individual likelihood value from each upper curve; those five values multiply. The upper panel uses one shared vertical scale, and the cards report the actual Gaussian density values. After normalization by the largest joint product, the result is the gold intersection on the lower curve. The two curves for x4=x5=34 overlap exactly, so that contribution enters the product twice.
The gold curve is not a second construction. It repeats the opening figure's calculation continuously for every candidate mean. Candidates A, B, and C are three positions on this same gold curve; the earlier numbers are the gold curve's heights at those three positions. The complete rule is:
FROM INDIVIDUAL CONTRIBUTIONS TO THE ESTIMATE
Ln(θ;D)=i=1∏npθ(xi),θ^MLE∈θ∈ΘargmaxLn(θ;D)
Why the independence assumption matters
Multiplication is not a universal rule that lets us ignore dependence. It is the consequence of a factorized joint model. For a time series, spatial field, or repeated measurements from one subject, the correct likelihood must retain whatever dependence structure the model asserts.
4
Turn the search into calculus
The logarithm exposes the peak
Products of many densities rapidly become tiny and are awkward to differentiate. We want a new vertical scale that leaves the best candidate in place while converting the product of independent contributions into a sum that calculus can handle one term at a time.
The same five observations · the same Gaussian likelihood
The exact likelihood peak, score directions, and curvature for the running data
Both curves are calculated from (29,31,32,34,34) under the known-width Gaussian model. The upper curve is L(μ)/L(μ^); the lower curve is ℓ(μ)−ℓ(μ^), vertically rescaled to use the panel height. Logging changes the vertical scale but preserves the ordering of candidates, so both curves select μ^=32. The score is positive to its left, negative to its right, and zero at the peak. The constant negative second derivative confirms that this zero-slope point is the unique maximum. The unknown μ⋆ is neither needed nor shown.
The visible maximum has not moved: both curves select the same θ^. The transformed curve is the log-likelihood. It works because the logarithm is strictly increasing—larger positive likelihoods remain larger after transformation—and because the logarithm of a product is the sum of the logarithms of its factors.
ℓn(θ)≡logLn(θ)=logi=1∏npθ(xi)
Define the log-likelihood by taking the logarithm of the joint likelihood.
ℓn(θ)=i=1∑nlogpθ(xi)
The logarithm converts independent evidence from a product into a sum.
Un(θ)≡∇θℓn(θ)=i=1∑n∇θlogpθ(xi)
Differentiate the log-likelihood. The resulting score is the sum of the observations’ signed local recommendations.
The Gaussian curve shows what differentiation contributes to the search. At every candidate below 32, moving the mean a little to the right raises the log-likelihood; at every candidate above 32, moving a little to the left raises it. At 32, the slope—the total score—is zero. A horizontal tangent can also occur at the bottom of a valley or, with several parameters, at a saddle point. A negative second derivative verifies that a one-dimensional curve bends downward around the candidate. For several parameters, the corresponding curvature check uses the Hessian, the table containing every second derivative.
SLOPE AND CURVATURE CHECK
Un(θ^)=0,Hn(θ^)=∇θ2ℓn(θ^)≺0
Zero slope is a candidate, not a guarantee
The derivative tests describe the shape near one candidate. A zero score may identify a maximum, minimum, or saddle point. The MLE is the candidate with the largest likelihood overall, so every remaining zero-score candidate must still be compared.
1Write the joint modelrespect dependence
→
2Freeze the dataform the likelihood
→
3Take logsproducts become sums
→
4Optimizecompare every candidate
5
Concrete model: known noise
The Gaussian mean is where the votes balance
Now specialize the abstract family. Suppose each measurement equals one shared but unknown location plus independent Gaussian noise of known width. The observations in the laboratory below are fixed at (29,31,32,34,34). Their average is 32, a fact calculated from the observations rather than supplied by knowledge of the data-generating truth. Thus μ^=32 is the best-fitting mean for this sample, but five observations do not establish that μ⋆=32. Only the candidate mean moves in the laboratory.
unknown truthμ⋆generated the population; never used to draw the likelihood
candidate being testedμmoves continuously along the horizontal axis
estimate from these dataμ^=xˉ=32the candidate where this sample's likelihood peaks
Gaussian mean laboratory
Move the candidate mean and watch five score votes combine
sample meanx̄ = 32.00the likelihood peak
total scoreU(μ) = 2.000net vote: move μ right
relative likelihood0.2019peak normalized to 1
x1 = 29← 0.35
x2 = 31→ 0.15
x3 = 32→ 0.40
x4 = 34→ 0.90
x5 = 34→ 0.90
In the first panel, the five observations stay fixed while the candidate line moves. Each arrow is the signed residual xi−μ; moving the candidate changes the arrows because it changes the distance from the candidate to each fixed observation. In the second panel, μ is the center of the candidate Gaussian, so changing μ selects a different model and shifts its density curve horizontally. In the third panel, the likelihood curve itself stays fixed because the data stay fixed; only the gold point moves along it to show the currently selected candidate's relative likelihood. Dividing the residual arrows by σ2 produces the score votes shown below, and those signed votes add to the displayed total score.
The visible balance has an exact algebraic form. For Xi∣μ∼N(μ,σ2) with known σ, independence produces the following log-likelihood and score:
Ln(μ)=i=1∏n2πσ21exp[−2σ2(xi−μ)2]
Multiply the Gaussian density values assigned to all observed measurements.
ℓn(μ)=−2nlog(2πσ2)−2σ21i=1∑n(xi−μ)2
Take logs. Constants collect into one term and the exponential residuals become a sum of squares.
Un(μ)=∂μ∂ℓn=i=1∑nσ2xi−μ=σ2n(xˉ−μ)
Each observation casts the score vote (xᵢ−μ)/σ²; their sum depends only on the sample mean.
Un(μ^)=0⟹μ^MLE=xˉ
The signed votes balance exactly at the arithmetic mean.
∂μ2∂2ℓn=−σ2n<0
The log-likelihood bends downward everywhere, so the stationary point is the unique maximum.
Why σ changes confidence but not the answer
With one common known σ, every score contains the same factor 1/σ2. That factor changes the steepness of the combined log-likelihood but cancels from the zero-score equation. If observation i instead has its own known σi, its vote is weighted by precision 1/σi2, and the MLE becomes a precision-weighted mean.
6
Concrete model: unknown center and width
Fit both Gaussian parameters together
When the noise width is also unknown, every candidate is now a pair (μ,σ). Shifting μ moves the distribution; changing σ widens or narrows it. A width that is too small severely penalizes residual observations, while a width that is too large spreads density so thinly that every observation receives weak support.
Joint likelihood landscape
Center and width are two parameter directions through the same model family
best centerμ̂ = 32.00sample mean
best widthσ̂ = 1.90root mean squared residual
candidate / maximum0.2463relative likelihood
Brightness is relative likelihood for the fixed measurements (29,31,32,34,34). Moving horizontally changes the center; moving vertically changes the width. The marked optimum must balance both directions at once.
The bright region is the compromise between those two failures. To locate its peak algebraically, differentiate the same log-likelihood once in each parameter direction and solve the resulting equations together.
ℓn(μ,σ)=−2nlog(2π)−nlogσ−2σ21i=1∑n(xi−μ)2
The first term is constant; the second penalizes spreading density; the third penalizes residual error relative to the chosen width.
∂μ∂ℓn=σ21i=1∑n(xi−μ)=0⟹μ^=xˉ
Holding σ fixed while differentiating recovers the score balance for the center.
∂σ∂ℓn=−σn+σ31i=1∑n(xi−μ)2=0
Holding μ fixed exposes the tradeoff between the width penalty and the residual-error penalty.
−nσ^2+i=1∑n(xi−μ^)2=0
Multiply the scale equation by σ³ and evaluate it jointly with the fitted center.
The fitted Gaussian width is the root mean squared residual around the fitted mean.
Gaussian variance MLEn in the denominator
It maximizes the likelihood of the assumed Gaussian model for the observed data.
Unbiased sample variancen−1 in the denominator
It corrects the finite-sample downward bias caused by estimating the center from the same data.
One observation produces no finite width estimate
With only one observation, choosing μ=x1 makes the residual zero, after which the likelihood increases without bound as σ→0. There is no finite MLE for both parameters in that case. More generally, the Gaussian scale estimate requires genuine spread in the observations.
7
Return to the original surprise
MLE and Fisher information inspect the same surface
The score gave each observation a signed vote and gave the complete dataset a direction toward larger likelihood. At the MLE, those realized votes balance. That tells us where the peak lies, but not how decisively it is localized. A broad peak and a sharp peak can both have zero slope at their maxima.
Realized first derivativeScore
Un(θ)=∇θℓn(θ)
Which direction raises this dataset's likelihood?
Realized second derivativeObserved information
Jn(θ)=−∇θ2ℓn(θ)
How sharply does this dataset's log-likelihood bend?
Expected local sensitivityFisher information
In(θ)=Eθ[Jn(θ)]
How sharp is that evidence on average across repeated datasets?
Under the usual regularity conditions, the expected score is zero and Fisher information has two equivalent forms. The minus sign belongs to the curvature formula, not to the squared-score formula:
TWO VIEWS OF FISHER INFORMATION
In(θ)=Eθ[Un(θ)Un(θ)T]=−Eθ[∇θ2ℓn(θ)]
For the Gaussian mean with known width, the curvature does not depend on which values were observed. Observed and expected information coincide exactly:
Jn(μ)=−∂μ2∂2ℓn=σ2n
The negative log-likelihood curvature is the dataset’s observed information about μ.
In(μ)=Eμ[Jn(μ)]=σ2n
Because that curvature is data-independent here, averaging leaves it unchanged.
μ^=Xˉ∼N(μ,nσ2),Var(μ^)=In(μ)1
The MLE’s sampling variance is exactly the inverse information in this model.
Noise, information, and what more data accomplishes
One Gaussian observation contributes I1(μ)=1/σ2 information about the mean. Larger σ means more noise, a flatter likelihood, and less information per observation. For n independent observations, information adds to In(μ)=n/σ2, while the estimator's variance falls to Var(μ^)=σ2/n. Under the independent Gaussian model, gathering more observations makes Xˉ concentrate around and converge to the true μ⋆. Convergence is a long-run statement, not a guarantee that any finite sample equals the truth. With the five observations used here, 32 is the MLE; the data do not reveal whether the true mean is 32.
The compact relationship
MLE uses the realized score to find where the evidence balances. Fisher information uses score variability or expected curvature to measure how precisely that balance point can be located. Information geometry then uses this same expected curvature as the local ruler on the entire model family.
Continue the estimation and geometric stories
08
A reusable maximum-likelihood checklist
The Gaussian calculation is one instance of a general workflow. Before differentiating anything, identify the random variables, the observed values, the model family, and the dependence assumptions. The algebra becomes meaningful only after those roles are fixed.
1
Specify the model family
Write pθ, its parameter space Θ, and what each parameter changes.
2
Write the joint distribution
Factor only when the model actually grants conditional independence.
3
Condition on what was observed
Freeze the dataset and read the joint density as L(θ;D).
4
Take the logarithm
Preserve the maximizer while replacing products with numerically stable sums.
5
Find and compare candidates
Use the score equation when appropriate, but also inspect boundaries, curvature, and global likelihood values.
6
Quantify uncertainty and check fit
Use observed/Fisher information, intervals, simulation, residuals, and model criticism; a point estimate is not the entire analysis.
The sentence worth keeping
Maximum likelihood does not ask which parameter is intrinsically most probable. It asks which parameterized model assigns the greatest joint density or probability mass to the observations that occurred.
