For discrete probability distributions $P$ and $Q$ defined on the same sample space, $\ {\mathcal {X}}\ ,$ the relative entropy from $Q$ to $P$ is defined^[11] to be

D_{\text{KL}}(P\parallel Q)=\sum _{x\in {\mathcal {X}}}P(x)\ \log \left({\frac {\ P(x)\ }{Q(x)}}\right)\ ,

which is equivalent to

D_{\text{KL}}(P\parallel Q)=-\sum _{x\in {\mathcal {X}}}P(x)\ \log \left({\frac {\ Q(x)\ }{P(x)}}\right)~.

In other words, it is the expectation of the logarithmic difference between the probabilities $P$ and $Q$ , where the expectation is taken using the probabilities $P$ .

Relative entropy is only defined in this way if, for all $x$ , $\ Q(x)=0\$ implies $\ P(x)=0\$ (absolute continuity). Otherwise, it is often defined as $+\infty$ ,^[1] but the value $\ +\infty \$ is possible even if $\ Q(x)\neq 0\$ everywhere,^[12]^[13] provided that $\ {\mathcal {X}}\$ is infinite in extent. Analogous comments apply to the continuous and general measure cases defined below.

Whenever $\ P(x)\$ is zero the contribution of the corresponding term is interpreted as zero because

\lim _{x\to 0^{+}}x\log(x)=0~.

For distributions $P$ and $Q$ of a continuous random variable, relative entropy is defined to be the integral^[14]

D_{\text{KL}}(P\parallel Q)=\int _{-\infty }^{\infty }p(x)\ \log \left({\frac {p(x)}{q(x)}}\right)\ \mathrm {d} \ \!x\ ,

where $p$ and $q$ denote the probability densities of $P$ and $Q$ .

More generally, if $P$ and $Q$ are probability measures on a measurable space $\ {\mathcal {X}}\ ,$ and $P$ is absolutely continuous with respect to $Q$ , then the relative entropy from $Q$ to $P$ is defined as

D_{\text{KL}}(P\parallel Q)=\int _{x\in {\mathcal {X}}}\ \log \left({\frac {P(\mathrm {d} \ \!x)}{Q(\mathrm {d} \ \!x)}}\right)\ P(\mathrm {d} \ \!x)\ ,

where $\ {\frac {\ P(\mathrm {d} \ \!x)\ }{Q(\mathrm {d} \ \!x)\ }}$ is the Radon–Nikodym derivative of $P$ with respect to $Q$ , i.e. the unique $Q$ almost everywhere defined function $r$ on $\ {\mathcal {X}}\$ such that $\ P(\mathrm {d} \ \!x)=r(x)Q(\mathrm {d} \ \!x)\$ which exists because $P$ is absolutely continuous with respect to $Q$ . Also we assume the expression on the right-hand side exists. Equivalently (by the chain rule), this can be written as

D_{\text{KL}}(P\parallel Q)=\int _{x\in {\mathcal {X}}}{\frac {P(\mathrm {d} \ \!x)}{Q(\mathrm {d} \ \!x)}}\ \log \left({\frac {P(\mathrm {d} \ \!x)}{Q(\mathrm {d} \ \!x)}}\right)\ Q(\mathrm {d} \ \!x)\ ,

which is the entropy of $P$ relative to $Q$ . Continuing in this case, if $\mu$ is any measure on ${\mathcal {X}}$ for which densities $p$ and $q$ with $\ P(\mathrm {d} \ \!x)=p(x)\mu (\mathrm {d} \ \!x)\$ and $\ Q(\mathrm {d} \ \!x)=q(x)\mu (\mathrm {d} \ \!x)\$ exist (meaning that $P$ and $Q$ are both absolutely continuous with respect to $\ \mu \$ ), then the relative entropy from $Q$ to $P$ is given as

D_{\text{KL}}(P\parallel Q)=\int _{x\in {\mathcal {X}}}p(x)\ \log \left({\frac {\ p(x)\ }{q(x)}}\right)\ \mu (\mathrm {d} \ \!x)~.

Note that such a measure $\mu$ for which densities can be defined always exists, since one can take $\ \mu ={\frac {1}{2}}\left(P+Q\right)\$ although in practice it will usually be one that in the context like counting measure for discrete distributions, or Lebesgue measure or a convenient variant thereof like Gaussian measure or the uniform measure on the sphere, Haar measure on a Lie group etc. for continuous distributions. The logarithms in these formulae are usually taken to base 2 if information is measured in units of bits, or to base $e$ if information is measured in nats. Most formulas involving relative entropy hold regardless of the base of the logarithm.

Various conventions exist for referring to $\ D_{\text{KL}}(P\parallel Q)\$ in words. Often it is referred to as the divergence between $P$ and $Q$ , but this fails to convey the fundamental asymmetry in the relation. Sometimes, as in this article, it may be described as the divergence of $P$ from $Q$ or as the divergence from $Q$ to $P$ . This reflects the asymmetry in Bayesian inference, which starts from a prior $Q$ and updates to the posterior $P$ . Another common way to refer to $\ D_{\text{KL}}(P\parallel Q)\$ is as the relative entropy of $P$ with respect to $Q$ or the information gain from $P$ over $Q$ .