Strong Law of Large Numbers

Here I present a very useful version of the strong law of large numbers
where we assume that a random variable X has finite
variance. Briefly, the Strong Law of Large Numbers states that for large
sample sizes the sample average converges almost surely to the expectation
of the random variable X as n \longrightarrow \infty. In some cases, where we don’t have a strict upper-bounds for |X| this may not be a good assumption and this is the case for many distributions encountered in Economics or Finance such as the Pareto distribution. But, I dare say that for most scientists this assumption holds for most of the distributions that they deal with.

Lemma: If E[\sum_{i=1}^{n} |X_{i}|^{s}] < \infty , and s > 0 then X_{n} \longrightarrow 0 almost surely.

Proof:
By the monotone convergence theorem, E[\sum_{i=1}^{n} |X_{i}|^{s}] < \infty which implies that \sum_{i=1}^{n} |X_{i}|^{s} is finite with probability 1. Therefore, |X_{n}|^{s}\longrightarrow 0 almost surely which also implies that  X_{n} \longrightarrow 0 almost surely.

Now we’re ready to proceed with the proof. But, first we must rigorously state
the version of the Strong Law of Large Numbers to be proven.

Theorem:  Let X_{1}, X_{2}, ... be i.i.d. random variables and assume that E[|X|^{2}] < \infty. Let S_{n} = \sum_{i=1}^{n} X_{i} , then \frac{S_{n}}{n} converges almost surely to E[X] .

Proof:
Assuming that E[|X|^{2}] < \infty we have E[(\frac{S_{n}}{n}-\mu)^{2}] = \frac{var(X)}{n}.

If we only consider values of n that are perfect squares, we obtain
\sum_{i=1}^{\infty} E[(\frac{S_{i}}{i^{2}}-\mu)^{2}] = \sum_{i=1}^{\infty}\frac{var(X)}{i^{2}} < \infty
which implies that (\frac{S_{i}}{i^{2}}-E[X])^{2} converges to 0 with probability 1

Let’s suppose the variables X_{i} are non-negative. Consider some n such that i^{2} \leq n \leq (i+1)^{2}. We then have S_{i^{2}} \leq S_{n} \leq S_{(i+1)^{2}}. It follows that

\frac{S_{i^{2}}}{(i+1)^{2}} \leq \frac{S_{n}}{n} \leq \frac{S_{(i+1)^{2}}}{i^{2}}
or \frac{i^{2}}{(i+1)^{2}}\frac{S_{i^{2}}}{i^{2}} \leq \frac{S_{n}}{n} \leq \frac{(i+1)^{2}}{i^{2}}\frac{S_{(i+1)^{2}}}{(i+1)^{2}}

As n \longrightarrow \infty\frac{i}{(i+1)}\longrightarrow 1 and since P(\frac{S_{i^{2}}}{i^{2}} \longrightarrow E[X]) = 1 we have P(\frac{S_{n}}{n}\longrightarrow E[X]) = 1

Note: If X \geq 0 doesn’t always hold, you can apply the above method to the
positive and negative parts of X where X = X^{+} -X^{-} and show
that the Strong Law of Large Numbers holds for this variable as well due to the linearity of expectation.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s