The sunrise problem is a problem first considered by Laplace that asks for the probability that the sun will rise tomorrow given a history of sunrises. Though this may seem like a silly problem it can serve to illustrate fundamental differences between Frequentists and Bayesians.
While Frequentists use probability only to model processes broadly described as ‘sampling’, Bayesians use probability to model both sampling and their ‘degree of belief’.
First, let’s consider the Frequentist approach. Now, the Frequentist guy has to
cheat in some way as this problem isn’t well-defined in the Frequentist
framework since ‘tomorrow’ is a sample of size one(and infinite standard
deviation). Some Frequentists try to define this probability by assuming that
there are many worlds with a sun potentially rising on each. But this is a really
silly bastardization of Laplace’s principle of insufficent reason. In all honesty,
the best this guy can do is to calculate the probability that the sun will rise on
any day, and not the probability that the sun rises on a particular day. Here we go:
1) Let’s assume that this phenomenon can be modeled as i.i.d. draws from a binomial distribution(i.e. a Bernoulli trial) where is the sum of
Bernoulli random variables and represents the number of days that the sun
rises out of observations.
2) By the Law of Large Numbers converges to the expected
number of sunrises, where is the probability that the sun rises on any day and is our estimate of this probability.
3) By the Central Limit Theorem, for large the sunrises should be normally distributed with mean and variance
4) Furthermore, for large we may construct confidence intervals
with coverage at least :
where is the percentile of the standard normal distribution. So, after a large number of sunrises the Frequentist can give a reasonable answer for the probability that the sun would rise on any day provided that his assumption holds true.
Now, compare this with the Bayesian solution which allows consideration of such questions:
1) is defined exactly as the Frequentist had it defined but in addition we can
define ,the event that the sun rises tomorrow, where equals 1 or 0.
2) Let be the probability of a sunrise on any given day.
3) We assume that before observing we had no prior information concerning
. Hence, by the principle of insufficient reason we may assume that
our prior is uniformly distributed on .
4) Now for the calculation of :
Note: the Bayesian approach isn’t better than the Frequentist solution. But, building a statistical model without any domain knowledge is doomed for failure whatever your approach…Bayesian, Frequentist, or otherwise.