Continuations

The last two Uncertainty Wednesdays we have looked at entropy as a measure of uncertainty. Today I want to tie that back to the original communications context in which Shannon introduced it. We can use our really simple model of a world with two states and two signal values to do this. We have analyzed an example where the signal was a test. Now think of the signal as actually being an electric impulse sent down a wire by a local observer. The observer gets to see perfectly which state of th...

Today in Uncertainty Wednesday I want to start on the idea that just the shape of the probability distribution alone contains some measure of uncertainty. Let’s think about the simplest case again with just two states A and B and P(A) = p1 with P(B) = p2 = 1- p1. I am using indices because we will shortly expand the number of states. If p1 = 1 there is no uncertainty at all because you are certain that the world is in state A. Same holds true for p1 = 0 except that it is now state B (because ...

Last week in Uncertainty Wednesday, I introduced Shannon entropy as a measure of uncertainty that is based solely on the structure of the probability distribution. As a quick reminder, the formula is H = - K ∑ pi log pi where ∑ is over the i = 1…n of the probability distribution Now you may notice a potential problem here if the distribution includes a probability p that approaches 0, because log p will go to infinity. If you know limits and remember L'Hôpital’s rule, you can convince yo...