Top Qs
Timeline
Chat
Perspective

Data processing inequality

Concept in information processing From Wikipedia, the free encyclopedia

Remove ads

The data processing inequality is an information theoretic concept that states that the information content of a signal cannot be increased via a local physical operation. This can be expressed concisely as 'post-processing cannot increase information'.[1]

Statement

Let three random variables form the Markov chain , implying that the conditional distribution of depends only on and is conditionally independent of . Specifically, we have such a Markov chain if the joint probability mass function can be written as

In this setting, no processing of , deterministic or random, can increase the information that contains about . Using the mutual information, this can be written as :

with the equality if and only if . That is, and contain the same information about , and also forms a Markov chain.[2]

Remove ads

Proof

One can apply the chain rule for mutual information to obtain two different decompositions of :

By the relationship , we know that and are conditionally independent, given , which means the conditional mutual information, . The data processing inequality then follows from the non-negativity of .

Remove ads

See also

References

Loading content...
Loading content...
Loading related searches...

Wikiwand - on

Seamless Wikipedia browsing. On steroids.

Remove ads