Mutual information is the idea that learning something about one variable might tell you about another. For example, learning that it’s daytime might give you information about whether the sun is shining. it could still be cloudy, but you can be more sure that it’s sunny than before you learned it was daytime.<\/p>\n
Mathematically, mutual information is represented using the concept of entropy. The information gained about a variable X, assuming you learn Y, is given by: I(X;Y) = H(X) - H(X|Y)<\/span><\/p>\n In this case, H(.)<\/span> is a measure of the entropy. It is given by H(X) = \\sum_x p(x) \\log_2(\\frac{1}{p(x)})<\/span><\/p>\n Mutual information is supposed to be symmetric (I(X;Y) = I(Y;X)<\/span>), but I’m interested in how that works in a causal context.<\/p>\n Let’s say you have a lightbulb that can be turned on from either of two light switches. If either lightswitch is on, then the bulb is on. Learning that one light switch is on tells you the bulb is on, but learning that the bulb is on does *not* tell you that one specific light switch is on. It tells you that at least one is on (but not which one).<\/p>\n Let’s assume for the sake of argument that each light switch has a probability p(on) = 0.25 of being turned on (and equivalently a probability p(off) = 0.75 of being off). Assume also that they’re independent.<\/p>\n The entropy of switch one is<\/p>\n H(S1) = p(on)\\log_2(\\frac{1}{p(on)}) + p(off)\\log_2(\\frac{1}{p(off)})<\/span> Since either switch has a probability of 0.25 of being on, and they’re independent, the bulb itself has a probability of 7\/16 of being on.<\/p>\n The entropy of the bulb is<\/p>\n H(B) = p(on)\\log_2(\\frac{1}{p(on)}) + p(off)\\log_2(\\frac{1}{p(off)})<\/span> If you know switch 1’s state, then the information you have about the light is given by<\/p>\n
\nH(S1) = 1\/4* \\log_2(4) + 3\/4 * \\log_2(\\frac{4}{3})<\/span>
\nH(S1) = 0.811<\/span><\/p>\n
\nH(B) = 7\/16 * \\log_2(\\frac{16}{7}) + 9\/16 * \\log_2(\\frac{16}{9})<\/span>
\nH(B) = 0.989<\/span><\/p>\n