In last week’s Uncertainty Wednesday, I introduced the expected value EV of a random variable X. We saw that EV(X) is not a measure of uncertainty. The hypothetical investments I had described all had the same expected value of 0. It is trivial, given a random variable with EV(X) = μ to construct X’ so that EV(X’) = 0. That’s in fact how I constructed the first investment. I started with $0 with 99% probability and $100 with 1% probability, which has an EV of
EV = 0.99 * 0 + 0.01 * 100 = 1
and then I simply subtracted 1 from each possible outcome to get -$1 with 99% probability and $99 with 1% probability.
What we are looking for instead in order to measure uncertainty, is a number that captures how spread out the values are around the expected value. The obvious approach to this would be to form the weighted sum of the distances from the expected value as follows:
AAD(X) = ∑ P(X = x) * |x - EV(X)|
where | | denotes absolute value (meaning the magnitude without the sign). This metric is known as the Average Absolute Deviation (btw, instead of the shorthand P(x) I am now writing P(X = x) to show more clearly that it is the probability of the random variable X taking on the value x).
AAD is a one measure of dispersion around the expected value but it is not the most commonly used one. That instead is what is known as Variance, which is defined as follows
VAR(X) = sum Prob(X = x) * (x - EV(X))^2
Or expressed in words: the probability weighted sum of the squared distances of possible outcomes from the expected value. It turns out that for a variety of reasons using the square instead of the absolute value has some useful properties and also interesting physical interpretations (we may get to those at some later point).
Let’s take a look at both of these metrics for the random variables from our investment examples
Variance
Investment 1: 0.99 * (-1 - 0)^2 + 0.01 * (99 - 0)^2 = 0.99 * 1 + 0.01 * 9,801 = 99
Investment 2: 0.99 * (-100 - 0)^2 + 0.01 * (9,900 - 0)^2 = 0.99 * 10,000 + 0.01 * 98,010,000 = 990,000
Investment 3: 0.99 * (-10,000 - 0)^2 + 0.01 * (990,000 - 0)^2 = 0.99 * 100,000,000 + 0.01 * 980,100,000,000 = 9,900,000,000
Average Absolute Deviation
Investment 1: 0.99 * |-1 - 0| + 0.01 * |99 - 0| = 0.99 * 1 + 0.01 * 99 = 1.98
Investment 2: 0.99 * |-100 - 0| + 0.01 * |9,900 - 0| = 0.99 * 100 + 0.01 * 9,900 = 198
Investment 3: 0.99 * |-10,000 - 0| + 0.01 * |990,000 - 0| = 0.99 * 10,000 + 0.01 * 990,000 = 19,800
You might have previously noticed that Investment 2 is simply Investment 1 scaled by a factor of 100 (and ditto Investment 3 is 100x Investment 2). We see that AAD, as per its definition, follows that same linear scaling whereas variance grows in the square, meaning the variance of Investment 2 is 100^2 = 10,000x the variance of Investment 1.
Both of these are measures that pick up the values of the random variable as separate from the structure of the underlying probabilities. If that doesn’t make sense to you, go back and read the initial post about measuring uncertainty and then go back to the posts about entropy. The three hypothetical investments each have the same entropy as they share the same probabilities. But AAD and Variance pick up the difference in payouts between the investments.
