Data compression/entropy

Entropy is a concept in information theory. It was named by anology with the concept of entropy in thermodynamics. The two concepts do actually have something in common, but this might not be obvious unless you are an expert in both fields.

Conceptually, entropy is the actual amount of (information theoretic) infomation in a piece of data. This is not necessarily the same as the number of bits in the data, for two main reasons.

1. Many of the bits in the data may not be conveying any infomation. For instance it is often the case that data structures store information redundantly, or have sections that are always the same regardless of the information in the data structure.

2. The amount of entropy is not always an integer number of bits.

A simple operational definition of entropy would be the size of the data if it was compressed using an ideal form of Data compression.

Another definition would be to take the logarithm of the number of different states that an object MIGHT be in. (This is only right if all the states are equally likely. If some are more likely than others the definition is messier.)

More information about entopy can be found on the page describing Information theory.

BTW Larry "I don't believe in subpages" Sanger should have a good look at this and maybe talk someone who knows both the entropies (or are there more than 2?) into finding a better home for this page.

Oh, yes, and the page is a mess, feel free to fix.