Last Uncertainty Wednesday, I introduced the topic of spurious correlation. Since then I have discovered a site that gives some fantastic examples of (potentially) spurious correlations. Here is one:The coefficient of correlation is 0.9926, i.e. almost 1 (which would be perfectly correlated). Let’s remind ourselves what finding correlation in a sample of data means. It is simply a numerical measure that can be computed for any paired data. The formula produces a result that has nothing to wit...

Today’s Uncertainty Wednesday revisits a favorite topic of mine: correlation. I first wrote about the importance of thinking about correlation in modeling over 10 years ago, long before starting the Uncertainty Wednesday series (the reference to Excel is a giveaway). I then had a three part series about spurious correlation which you can find here: part 1, part 2 and part 3. Here is the key introductory paragraph:Well, as you have seen from the posts on sample mean and sample variance, whenev...

So the last two Uncertainty Wednesday posts have been about spurious correlation. Today, I want to give an example of easy it is to observe spurious correlation. To that end I wrote a little Python program which I will show below that rolls two independent dice. Each is rolled 10 times to give us two data series of 10 points each. This mimics the 10 data point series from last week’s example. The program runs some number of these and each times calculates and outputs the coefficient of correl...

After a short break of a few weeks, Uncertainty Wednesday is back! My last post had been the third part in a series on “Spurious Correlation” which ended with “Interestingly, this takes me to the edge of my own knowledge and so I have asked an expert in Bayesian estimation to help me. Stay tuned!” Well, that expert is Eric Novik from Generable. Generable is a company that uses Stan, the world’s leading Bayesian estimation too, to help companies make better decisions. Eric took on my challenge...

In the last Uncertainty Wednesday post on Sample Variance, I wrote that “Inference from data without explanations is how people go deeply wrong about reality.” It occurred to me that the best way to illustrate this is by writing about spurious correlation. To do that I first have to introduce the concept of correlation though. It may seem surprising that I have gotten this far into the series without doing so, but we spent a fair bit of time on a related concept, namely independence. If you d...