When dealing with probabilities or building mathematical models, the de-facto distribution is the normal/gaussian. The bell curve. But where does this come from? Why is it a better choice than any other distribution that centers on the average? It turns out that the Guassian distribution is the one that is the least biased choice if the only information you have are 1) the average and 2) the variance of a set of outcomes.
To get to this result, we will first establish a measure of our uncertainty. This will be useful because we will then (Part 2) choose a distribution to model our process that both maximizes this uncertainty function and simultaneously satisfies the constraints of our average/variance knowledge.
Criteria for a Measure of Uncertainty
How do you measure the amount of uncertainty in a probability distribution? We’ll start by establishing a few characteristics that would be required for a useful uncertainty function. From these we will derive the functional form that meets the criteria.
Suppose we have the possible outcomes of the process encoded by values represented by . For instance, if it were a 6-sided die, the values would be 1, 2, 3, 4, 5, and 6. One way to mathematically model the process of rolling the die would be to assign probabilities to each of the possible outcomes. In the case of a fair die, each probability is equal to 1/6. We assign a corresponding probability for each outcome and these probabilities represent our knowledge of the process. Let’s call our unknown uncertainty function “H”. We expect H to depend on our probabilities since we are trying to quantify our uncertainty in our knowledge and its our probabilities that represent that knowledge:
So which criteria does H have to meet so that it is useful and in line with our intentions?
- It must be a continuous function of the . This means a small change in one of the probabilities can’t give a huge change in our knowledge state.
- In the case that all outcomes are equally probable (as in the case of a fair 6-sided die), the uncertainty should increase as the number of possible outcomes increases. In other words, the uncertainty function monotonically increases with the number of equiprobable choices.
- This just means that if you have more choices, you’re more uncertain than if you had fewer choices. Guessing the outcome of a coin-flip is easier than picking the lottery numbers.
- No matter how you count the probabilities, you should arrive at the same value for the same state of knowledge.
- For instance, let’s say that instead of having to guess the outcome of a fair six sided die roll directly, you first guess whether it’s in the set or and then guess the final number from the smaller list. The first choice has two equal outcomes, so your probability of success is 1/2. The second choice has three equal outcomes, so your probability of success is 1/3. Your total probability of success is the product of these two and is 1/6. Essentially this is another way of modeling the exact same process. The requirement above just means that no matter how we model a process and break it up into sub-decisions, we should get one consistent value for H, whatever it ends up being.
The Uncertainty Function
It turns out, the function that satisfies these criteria is where is a positive constant that determines the units. If you’re interested in the derivation, see the Deriving the Uncertainty Function section below.
Let’s use a simple case where there are two outcomes with probabilities and . The uncertainty becomes
Plotting this function using and a logarithm of base-2, the following plot is generated with units of bits on the y-axis.
How does this fit our intuition? If we are certain in the outcome (i.e. or ) then our uncertainty function . We have no uncertainty. If, on the other hand, , then our function is maximized at bit. As we would expect, we are maximally uncertain when the two outcomes are equally probable.
Deriving the Uncertainty Function
Using the criteria stated above, let’s now derive our uncertainty function.
Uncertainty Function of Equally Probable Outcomes
Let’s assume we are dealing with equally probable events at first, like a fair die. We’ll simplify our notation so that instead of writing the uncertainty for equally probable events as , we will just use .
According to requirement 3 above, we can deduce that to decompose a choice from equally likely possibilities, you can transform it to be choices from equally likely possibilities: . In other words, if you have equally likely choices, you can decompose that into two subsequent choices, each from three possibilities. This decomposition is similar to what we did in the sub-point of requirement 3.
Now, for any specified number , we can find an to satisfy the following equation: . So if , we could choose and then , which satisfies the inequality.
By taking the logarithms of each term, we get . Dividing by , we get . Subtracting from each term we end up showing: . And since can be arbitrarily large, . So what?
Since requirement 2 says that our uncertainty function is monotonic that means the original inequality and subsequent forms of it apply: . Using our decomposition rule just derived: . Continuing with the same division/subtraction steps we just undertook we end up with the relation: .
These inequalities essentially state the following: The ratios of and are both within a distance of from the quantity . That means the furthest they can be from one another is . Or translating that to an inequality, .
So what does this tell us? We have shown that the difference between a ratio of our unknown functions are arbitrarily close to a ratio of known functions . This leads one to naturally conclude that the unknown functions are the known functions within some multiplicative constant. How else could they be arbitrarily close for any and ? Therefore:
Uncertainty of General Probability Distributions
So that’s the function that quantifies the uncertainty of a process with equally probable outcomes. But what about processes that don’t have equally probable outcomes? Using requirement 3 we see that the way you decompose a process is that you take the uncertainty of the first choice and add the uncertainties of the additional choices that are weighted by their probability of occurrence. So in the case of our game in the example of requirement 3, the uncertainty of the process of choosing 1 of 6 equally probable sides is the same as the sum of the uncertainties of the two step process: 1) choosing one of the two subgroups or and 2) choosing among the three elements of each subgroup multiplied by the probabilities of getting that subgroup in the first place.
Now imagine a process where you have outcomes where is the relative occurrence of the outcome. For instance, a 6 sided die where there are a single green, a single red, two blues and two yellows. The probabilities for each outcome are then straight-forward: . This also means that if you know you are on outcome , this is like choosing from equally probable events since there are occurrences of . To break down this process of outcomes we have . Let’s solve this for our unknown uncertainty function .
The only thing we did was recognize that a function weighted by its probabilities and summed is equal to the function itself (line 2) and substituted our equation of the that we found above (line 3).
So, finally, we have solved our problem. The uncertainty associated with a process that’s characterized by the set of probabilities is given by the function
where is a positive constant.
Our original goal was to find the least biased distribution to use to model a process when we only have knowledge of the average and variance of a sub-population. To do this we started by coming up with a metric for measuring our uncertainty when given a set of probabilities. In Part 2 of this treatment we will maximize this uncertainty function while using our knowledge of average/variance as constraints on the probabilities. In this way we work backwards from this uncertainty function and ultimately arrive at a functional form of our least-biased probability distribution.
This derivation and treatment follows almost exactly that found in Claud Shannon’s original work A Mathematical Theory of Communication. The image above was found on the wikipedia page on information entropy and can be found here.