Strong Law of Large Numbers

Here I present a very useful version of the strong law of large numbers
where we assume that a random variable $X$ has finite
variance. Briefly, the Strong Law of Large Numbers states that for large
sample sizes the sample average converges almost surely to the expectation
of the random variable $X$ as $n \longrightarrow \infty$. In some cases, where we don’t have a strict upper-bounds for $|X|$ this may not be a good assumption and this is the case for many distributions encountered in Economics or Finance such as the Pareto distribution. But, I dare say that for most scientists this assumption holds for most of the distributions that they deal with.

Lemma: If $E[\sum_{i=1}^{n} |X_{i}|^{s}] < \infty$ , and $s > 0$ then $X_{n} \longrightarrow 0$ almost surely.

Proof:
By the monotone convergence theorem, $E[\sum_{i=1}^{n} |X_{i}|^{s}] < \infty$ which implies that $\sum_{i=1}^{n} |X_{i}|^{s}$ is finite with probability 1. Therefore, $|X_{n}|^{s}\longrightarrow 0$ almost surely which also implies that  $X_{n} \longrightarrow 0$ almost surely.

Now we’re ready to proceed with the proof. But, first we must rigorously state
the version of the Strong Law of Large Numbers to be proven.

Theorem:  Let $X_{1}, X_{2}, ...$ be i.i.d. random variables and assume that $E[|X|^{2}] < \infty$. Let $S_{n} = \sum_{i=1}^{n} X_{i}$, then $\frac{S_{n}}{n}$ converges almost surely to $E[X]$.

Proof:
Assuming that $E[|X|^{2}] < \infty$ we have $E[(\frac{S_{n}}{n}-\mu)^{2}] = \frac{var(X)}{n}$.

If we only consider values of n that are perfect squares, we obtain
$\sum_{i=1}^{\infty} E[(\frac{S_{i}}{i^{2}}-\mu)^{2}] = \sum_{i=1}^{\infty}\frac{var(X)}{i^{2}} < \infty$
which implies that $(\frac{S_{i}}{i^{2}}-E[X])^{2}$ converges to $0$ with probability $1$

Let’s suppose the variables $X_{i}$ are non-negative. Consider some $n$ such that $i^{2} \leq n \leq (i+1)^{2}$. We then have $S_{i^{2}} \leq S_{n} \leq S_{(i+1)^{2}}$. It follows that

$\frac{S_{i^{2}}}{(i+1)^{2}} \leq \frac{S_{n}}{n} \leq \frac{S_{(i+1)^{2}}}{i^{2}}$
or $\frac{i^{2}}{(i+1)^{2}}\frac{S_{i^{2}}}{i^{2}} \leq \frac{S_{n}}{n} \leq \frac{(i+1)^{2}}{i^{2}}\frac{S_{(i+1)^{2}}}{(i+1)^{2}}$

As $n \longrightarrow \infty$$\frac{i}{(i+1)}\longrightarrow 1$ and since $P(\frac{S_{i^{2}}}{i^{2}} \longrightarrow E[X]) = 1$ we have $P(\frac{S_{n}}{n}\longrightarrow E[X]) = 1$

Note: If $X \geq 0$ doesn’t always hold, you can apply the above method to the
positive and negative parts of $X$ where $X = X^{+} -X^{-}$ and show
that the Strong Law of Large Numbers holds for this variable as well due to the linearity of expectation.