Compression of random binary data

[Steve D'Aprano]

On Sun, 29 Oct 2017 01:56 pm, Stefan Ram wrote:

>   If the entropy of an individual message is not defined,
>   than it is still available to be defined. I define it
>   to be log2(1/p), where p is the probability of this
>   message. I also choose a unit for it, which I call "bit".

That is exactly the definition of self-information:

https://en.wikipedia.org/wiki/Self-information

See also:

https://en.wikipedia.org/wiki/Entropy_(information_theory)

which lists several forms of related measures of information:

- the self-information of an individual message or symbol taken from a 
  given probability distribution;

- the entropy of a given probability distribution of messages or symbols;

- the entropy rate of a stochastic process.


It also suggests a connection between information entropy and thermodynamic
entropy, namely that the information entropy of a system is the amount of
additional information needed to determine the microstate of a system (the
states of all its particles), given the macrostate (identified by the bulk
thermodynamic parameters such as temperature, volume, energy).

More here: 

https://physics.stackexchange.com/questions/263197/is-information-entropy-the-same-as-thermodynamic-entropy




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.