Last Uncertainty Wednesday, I introduced the beta distribution to model our prior belief about the probability of Heads in a coin toss. We saw that for the parameters α = β = 1 the beta distribution gives us a uniform prior, which expresses the highest degree of uncertainty (you may want to revisit the earlier post on entropy for that). 

Now I will toss an actual coin while writing this post. Wait. It came up tails (i.e. not heads). What should our new values be for α and β? 

As it turns out the updating formula is super simple. If we observe heads we increment α and if we observe tails we increment β. Here is what the Beta distribution looks like for α = 1 and β = 2, i.e. after we have observed tails:

What does this picture tell us? Going from a uniform prior, the observations of tails has shifted a fair bit of probability towards lower values of θ. Remember that θ is the parameter we are interested in. It is the probability that the coin will come up heads. Since we have seen only one outcome and that outcome was tails, we are updating by assigning a lot more probability to values of θ below 0.5 (on the horizontal axis). This is our updated belief.

Is this the only possible update we could have made? Well, if we use the Beta distribution to model our beliefs and the thing we observe has a binary outcome (such as a coin toss), then this is the precise updating as determined by Bayes’ Theorem. If you are so inclined you can find a very accessible derivation of this result here, which also shows how the simple updating rule results.

So let’s keep tossing our coin. I just did and as it turns out got heads. So our updated values are now α = 2 and β = 2 and our new distribution looks as follows

This is symmetric, which shouldn’t surprise us as we have observed both head and tails. It is also starting to shift probability away from the extremes and towards the middle.

Now if you want to play this game by yourself and see how the beta distribution changes after each toss, you can just head over to this query on WolframAlpha. Just add 1 to the value of α (alpha) each time you toss heads and 1 to the value of β (beta) each time you toss tails. Watch the distribution update!

I have just tossed my coin 30 times and observed 17 heads and 13 tails. Here is what the Beta distribution looks like for α = 18 and β = 14

So the beta distribution is starting to bunch up around 0.5, but we can see that the average is slightly above 0.5, in line with having observed more heads rather than tails (making heads somewhat more likely). Next Wednesday I will talk more about what we have learned at each step and also some of the limitations of this approach.

In the meantime, thanks to Eric Novik from Generable for helping me with my understanding of this!