Will the Sun Rise Tomorrow?

How an 18th-century puzzle reveals a smarter way to analyze uncertain data

Photo by Artem Sapegin on Unsplash

It sounds silly, but how likely is it that the sun will stop rising? I don’t mean an eclipse, I mean the probability that the sun could go away and never come back. Can we really reason about this mathematically? I mean, the sun has been continuously rising for millennia as far as we know. All our telescopes and physics don’t expect this to happen. Yet, how could we quantify this unlikely possibility?

The above problem is known as the Sunrise Problem and was solved by Pierre-Simon Laplace. The solution is known as the Rule of Succession, and it is used on problems in which some outcomes are very unlikely, poorly sampled, or never reported at all.

One of the most practical applications of this idea is improving how we interpret user ratings. For instance, consider you are buying a bicycle and you see the following options:

1. Bike A: 4.40/5.0 rating after 286 reviews
2. Bike B: 4.66/5.0 rating after 3 reviews

Which one is the best product? The first one has a lower rating but was reviewed by more people. Is it really a worse bike than the other one? Which one would you pick?

In such scenarios, the rule of succession gives us a reliable framework to disambiguate such scenarios and produce a more unbiased average.

Solving the Sunrise Problem

Setting the sun aside, the real problem solved by Laplace is the following:

How can we estimate the probability of an unlikely event?

Using different words, how can we estimate the probability of outcomes that we know are possible but have yet to happen?

The solution is deceptively simple: assume each outcome has happened at least once.

For the past ~5475 years, the sun has risen about 2 million times. Thus, we have 2M pieces of evidence in favor of it showing up and 0 for it not showing up. The naive approach is to say the sun rises 100% of the time (2M/2M) and won’t rise 0% of the time (0/2M).

Laplace’s trick is to boost each possible outcome by 1. Thus, we assume the sun has risen 2M + 1 times and not risen 0 + 1 times. These +1s mathematically represent that we believe at least once each outcome can happen.

Doing the math, the probability of the sun rising tomorrow under Laplace’s assumption is (2M+1)/ (2M+2), or 99.99995%, while the probability that it won’t rise is 1 / (2M+2), or 0.00005%.

This technique is the Rule of Succession, and, essentially, it allows us to encode into the computation our prior belief that each outcome is possible.

The more evidence we collect of the sun rising, the smaller the computed probability of it not rising will be. It will never be zero but will shrink infinitesimally towards 0.

From Binary to Categorical: User Ratings

The same idea extends naturally from two outcomes to many. If an item can receive ratings from 1 to 5 stars, we simply add one hypothetical rating for each possible score. For N outcomes, that means N new samples.

Consider the user ratings below:

• 4 users gave 5/5 stars
• 1 user gave 4/5 stars
• 3 users gave 3/5 stars
• 0 users gave 2/5 stars
• 0 users gave 1/5 stars

The math is straightforwad: (45 + 14 + 33 + 02 + 0*1) / 8 = 33 / 8 = 4.125

The procedure above is the simple/naive average. When applying the Rule of Succession, we will be assuming 5 extra data points: an extra 5/5, 4/5, 3/5, 2/5, and 1/5. The new ratings become:

• 4+1 users gave 5/5 rating
• 1+1 user gave 4/5 rating
• 3+1 users gave 3/5 rating
• 0+1 users gave 2/5 rating
• 0+1 users gave 1/5 rating

We now have 13 total ratings (8 + 5) and the new average is (55 + 24 + 43 + 12 + 1*1)/ 13 = 48/13 = 3.69 (!)

Laplace’s trick reduced the average by 0.435, almost half a star!

This new score encodes the ratings we have with the belief that someone somewhere would love this item, someone else would hate it, another person would find it so-so, and so on.

If you compare this to the sunrise problem, the main difference is the total number of observations. As the number of observations grows, the closer the simple and improved estimates will be. For very few observations, Laplace’s estimate will tend towards a neutral answer (a uniform distribution). In this case, 3 stars.

Revisiting the Bicycles Example

The data we have on buying a new bicycle is:

1. Bike A: 4.40/5.0 rating after 286 reviews
2. Bike B: 4.66/5.0 rating after 3 reviews

Using the Rule of Succession, we can now make more sense of Bike B's score, even though it has only 3 real reviews. In both cases, we will be adding 5 hypothetical ratings:

For Bike A, we get: (4.40 * 286 + 5+4+3+2+1) / (286 + 5) = 4.38

For Bike B, we get: (4.66 * 3 + 5+4+3+2+1) / (3 + 8) = 3.625

The improved estimate shrank down a full point for Bike B, while the widely reviewed Bike A evaluation almost didn’t change. If you had difficulty choosing, this technique gives you a principled way of looking at those review aggregates properly.

All Good Models have Regularization.

While this technique can be seen under many lights, I want to emphasize it as a regularization technique. When we include additional entries for each outcome, we are essentially creating a more well-behaved scenario.

Without using the rule of succession, we are putting 100% of our trust in the data. The more data we have, the safer it is to do this. However, as seen in the bicycles case, poorly sampled entries can be grossly overestimated.

Moreover, say a new bicycle was added to the system. It has 0 reviews, so how can we reason about that? Which rating does it have? This no-data situation is prone to analysis errors. Using the rule, we have data for all outcomes and a baseline of 3 stars for any new item.

We can go a step further and tune the trick to put more or less emphasis on our prior belief. This is known as Additive Smoothing.

The idea is that, instead of adding 1 to each outcome, we can add 0.5, 10, or any positive X, as long as we keep track of the new total. If the value is below one, we are downweighting the trick’s impact. If the value is above one, we are shifting more trust towards the prior belief and less towards the actual data.

If you are familiar with classical optimization techniques, this is very close in nature to the Lavenberg-Marquardt damping technique.

When Should You Use This?

The Rule of Succession lends itself naturally to data analysis problems that include frequency and averages among N options.

For instance, how likely are you to pick any of the 5 popcorn brands? Without any data, I can assume you might choose any of the 5, so I start with 5 datapoints and a prior belief of 20% chance for each. If you then choose popcorn C, I won’t naively assume you do that 100% of the time. Instead, its probability will grow to 33.3% while the others go down to 16.6%

This technique is meant for analysis only. I would refrain from actually showing it to users. It is more natural to say an item has 0 reviews than to say it has 0 reviews and 3 stars.

This is all for now. Feel free to leave a comment, follow me here on Medium, or connect with me on LinkedIn. If you like my writing, consider subscribing to get e-mail notifications whenever I publish. Cheers!