Alongside the Central Limit Theorem, the Law of Large Numbers is equally important. The Weak Law of Large Numbers essentially states that the sample average converges in probability toward the expected value. There is also a Strong version which I won’t discuss for now. But, both versions are of great importance in science as they imply that large sample sizes are better for estimating population averages.
Mathematically, the Weak Law states that if we let be a sequence of iid random variables each having finite mean , for any
In order to demonstrate this, we shall first go through two Russian inequalities.
If is a random variable that takes only non-negative values then for any ,
Proof: For , let
and note that since ,
Taking expectations of he previous inequality yields
so we have
If is a random variable with finite mean and finite variance
then for any value ,
Proof: Since , we can apply Markov’s inequality with to obtain
Since iff , we have
With the above ingredients we are now ready to provide a proof of the Weak Law of Large Numbers:
Assuming that all random variables are iid with finite mean and finite variance,
and it follows from Chebyshev’s inequality that
Now, if we take the limit as we obtain
the desired result.