Information Theory Notes
Claude Shannon
1948
Shannon was an Engineer at Bell labs
who formulated he Mathematical Theory of Communication.
He wanted to know how to encode information so that it resists erosion by white noise encountered in telephone lines. As a result he developed his theory to explain general laws of information transmission and
invented a mathematical definition which turned out to be identical to definition of entropy.
Shannon's definition:
The more uncertainty there is about the contents of a message that is about to be received, the more information the message contains.
He was not talking about meaning.
Instead, he was talking about symbols in which the meaning is encoded.
For example, if an AP story is coming over the wire in Morse code, the first letter contains more information than the following letters because it could be any one of 26. If the first letter is a "q", the second letter contains little informat
ion because most of the time "q" is followed by "u." In other words, we already know what it will be.
Entropy
Entropy
The second law of thermodynamics gives a precise definition of a property called entropy. Entropy
can be thought of as a measure of how close a system is to equilibrium; it can also be thought of as a
measure of the disorder in the system. The law states that the entropy—that is, the disorder—of an
isolated system can never decrease. Thus, when an isolated system achieves a configuration of
maximum entropy, it can no longer undergo change: It has reached equilibrium. Nature, then, seems
to 'prefer' disorder or chaos. It can be shown that the second law stipulates that, in the absence of
work, heat cannot be transferred from a region at a lower temperature to one at a higher
temperature.
The second law poses an additional condition on thermodynamic processes. It is not enough to
conserve energy and thus obey the first law. A machine that would deliver work while violating the
second law is called a 'perpetual-motion machine of the second kind,' since, for example, energy
could then be continually drawn from a cold environment to do work in a hot environment at no
cost. The second law of thermodynamics is sometimes given as a statement that precludes
perpetual-motion machines of the second kind.
Shannon's Perspective
In a highly ordered (low entropy) system such as a glass of distilled water, we know what to expect: water molecules floating around.
In a high entropy system, such as a shovel load of dirt, we don't know what to expect.
The knowledge that it's dirt doesn't tell you what molecules to expect. Uncertainty is high.
We can turn the whole thing around mathematically by placing a minus sign up front, which might make it more intuitive,
so that Shannon equates info with negative entropy.
We can also include meaning in the formulation, however. The more choices we have, i.e., the more meaning we have, the more uncertainty we have. For example, the more we learn, the more we know that we have more to learn. Thus, the more meaning we acquire
, the more uncertain we SOMETIMES become. This implies, that we should keep messages simple and narrow the choices we offer if we wish to persuade. It's the KISS principle at work.
Norbert Wiener
Formulated the same definition at about the same time said:
"Just as the amount of information in a system is a measure of its degree of organization, so the entropy of a system is a measure of its degree of disorganization; and the one is simply the negative of the other." (Cybernetics)
Standard Definitions of Information
"to give form to, put into form or shape" (OED)
Does everything that has form qualify as information?
Do tires qualify?
Howabout light bulbs?
TV sets?
DNA?
Information is the prerequisite for structure
DNA, for example, advances and maintains structure.
All information, though, does not advance and maintain order?
Yelling "fire" in a theater.
Thus, information can lead to chaos.
Is a highly ordered system like a snowflake information?
Must meaning be a part of the definition of information?
Feedback
What about feedback?
Any system that pursues a goal must have feedback.
Thus, feedback is meaninful information.
Definition
Meaningful information is that which has form, can help create or maintain form, and does so by representing states of the environment and inducing behaviors appropriate to them.
Weather forcasts
What about lies?
What about a computer program?