It’s been six weeks since the last Uncertainty Wednesday, so I strongly suggest you go back first and read that post which provides an introduction to the idea of updating. Take your time, this new post won’t go away!
The distribution that we will use for modeling our coin is the Beta distribution. Why choose that one? Because the beta distribution when combined with likelihood function for a coin toss gives us another beta distribution. This is known technically as a so-called conjugate prior. That all sounds very technical but the idea is simple: when our prior is a beta distribution and the observations are of the 0 or 1 (head or tails) variety, then our posterior distribution after updating is once again a beta distribution.
Now the beta distribution has two parameters, which are commonly referred to as α and β. These parameters can take on lots of values, but there is straightforward choice for initial parameters. We will pick α = β = 1 because for those values, the beta distribution coincides with the uniform distribution. All the plots in this post are generated using Wolfram Alpha.
As a refresher: the x axis on this chart is the parameter θ for which we are expressing our belief. In the coin toss example θ would be the probability that the coin comes up Heads on any one toss (in which case 1 - θ is the probability that it comes up Tails). We know nothing about the coin right now so that probability θ could be anything between 0 (never see Heads) and 1 (every toss comes up Heads).
At this point you may be thoroughly confused. How do the parameters from the beta distribution relate to the parameter for the coin toss? Sometimes people call the α and β hyper-parameters, but I think a better term would have been meta-parameters or belief-parameters. Put differently α and β determine the shape of our belief about θ.
So now what remains to be done is to figure out how we should update α and β after we have observed some outcomes. We are looking for new values of α and β after we have observed either Heads or Tails. As we will see next Wednesday the updating of these values turns out to be super simple.
