Continuous Bernoulli distribution

Not to be confused with Bernoulli distribution.

In probability theory, statistics, and machine learning, the continuous Bernoulli distribution^[1][2][3] is a family of continuous probability distributions parameterized by a single shape parameter ${\displaystyle \lambda \in (0,1)}$, defined on the unit interval ${\displaystyle x\in [0,1]}$, by:

${\displaystyle p(x|\lambda )\propto \lambda ^{x}(1-\lambda )^{1-x}.}$

Continuous Bernoulli distribution
Probability density function
Parameters	${\displaystyle \lambda =1/(1+e^{-\theta })\in (0,1)}$	${\displaystyle \theta \in \mathbb {R} }$, natural parameter
Support	${\displaystyle x\in [0,1]}$	${\displaystyle x\in [0,1]}$
PDF	${\displaystyle C(\lambda )\lambda ^{x}(1-\lambda )^{1-x}\!}$ where ${\displaystyle C(\lambda )={\begin{cases}2&{\text{if }}\lambda ={\frac {1}{2}}\\{\frac {2\tanh ^{-1}(1-2\lambda )}{1-2\lambda }}&{\text{ otherwise}}\end{cases}}}$	${\displaystyle f(x\mid \theta )={\begin{cases}1&\theta =0\\\exp(x\theta -\log\{(e^{\theta }-1)/\theta \})&\theta \neq 0\end{cases}}}$
CDF	${\displaystyle F(x\mid \lambda )={\begin{cases}x,&\lambda ={\tfrac {1}{2}}\\[6pt]{\dfrac {\lambda ^{x}(1-\lambda )^{1-x}+\lambda -1}{2\lambda -1}},&{\text{otherwise}}\end{cases}}}$	${\displaystyle F(x\mid \theta )={\begin{cases}x&\theta =0\\(e^{\theta x}-1)/(e^{\theta }-1)&\theta \neq 0\end{cases}}}$
Mean	${\displaystyle \operatorname {E} [X]={\begin{cases}{\tfrac {1}{2}}&\lambda ={\tfrac {1}{2}}\\[6pt]{\dfrac {\lambda }{2\lambda -1}}+{\dfrac {1}{2\tanh ^{-1}(1-2\lambda )}},&{\text{otherwise}}\end{cases}}}$	${\displaystyle \operatorname {E} [X]={\begin{cases}1/2&\theta =0\\e^{\theta }/(e^{\theta }-1)-\theta ^{-1}&\theta \neq 0\end{cases}}}$
Variance	${\displaystyle \operatorname {Var} (X)={\begin{cases}{\tfrac {1}{12}},&\lambda ={\tfrac {1}{2}}\\[6pt]-{\dfrac {\lambda (1-\lambda )}{(1-2\lambda )^{2}}}+{\dfrac {1}{(2\tanh ^{-1}(1-2\lambda ))^{2}}},&{\text{otherwise}}\end{cases}}}$	${\displaystyle \operatorname {Var} (X)={\begin{cases}1/12&\theta =0\\(2-e^{\theta }-e^{-\theta })^{-1}+\theta ^{2}&\theta \neq 0\end{cases}}}$

The continuous Bernoulli distribution arises in deep learning and computer vision, specifically in the context of variational autoencoders,^[4][5] for modeling the pixel intensities of natural images. As such, it defines a proper probabilistic counterpart for the commonly used binary cross entropy loss, which is often applied to continuous, ${\displaystyle [0,1]}$-valued data.^[6][7][8][9] This practice amounts to ignoring the normalizing constant of the continuous Bernoulli distribution, since the binary cross entropy loss only defines a true log-likelihood for discrete, ${\displaystyle \{0,1\}}$-valued data.

The continuous Bernoulli also defines an exponential family of distributions. Writing ${\displaystyle \theta =\log \left(\lambda /(1-\lambda )\right)}$ for the natural parameter, the density can be rewritten in canonical form: ${\displaystyle p(x|\theta )\propto \exp(\theta x)}$. ^[10]

Statistical inference

Given an independent sample of ${\displaystyle n}$ points ${\displaystyle x_{1},\dots ,x_{n}}$ with ${\displaystyle x_{i}\in [0,1]\,\forall i}$ from continuous Bernoulli, the log-likelihood of the natural parameter ${\displaystyle \theta }$ is

${\displaystyle {\mathcal {L}}(\theta )=\theta \sum _{i=1}^{n}x_{i}-n\log\{(e^{\theta }-1)/\theta \}}$

and the maximum likelihood estimator of the natural parameter ${\displaystyle \theta }$ is the solution of ${\displaystyle {\mathcal {L}}'(\theta )=0}$, that is, ${\displaystyle {\hat {\theta }}}$ satisfies

${\displaystyle {\frac {e^{\hat {\theta }}}{e^{\hat {\theta }}-1}}-{\frac {1}{\hat {\theta }}}={\frac {1}{n}}\sum _{i=1}^{n}x_{i}}$

where the left hand side ${\displaystyle e^{\hat {\theta }}/(e^{\hat {\theta }}-1)-{\hat {\theta }}^{-1}}$ is the expected value of continuous Bernoulli with parameter ${\displaystyle {\hat {\theta }}}$. Although ${\displaystyle {\hat {\theta }}}$ does not admit a closed-form expression, it can be easily calculated with numerical inversion.

Further properties

The entropy of a continuous Bernoulli distribution is

${\displaystyle \operatorname {H} [X]={\begin{cases}0&{\text{ if }}\lambda ={\frac {1}{2}}\\{\frac {\lambda \log \left(\lambda \right)-\left(1-\lambda \right)\log \left(1-\lambda \right)}{1-2\lambda }}-\log \left({\frac {2\tanh ^{-1}\left(1-2\lambda \right)}{e\left(1-2\lambda \right)}}\right)&{\text{ otherwise}}\end{cases}}\!}$

Related distributions

Bernoulli distribution

The continuous Bernoulli can be thought of as a continuous relaxation of the Bernoulli distribution, which is defined on the discrete set ${\displaystyle \{0,1\}}$ by the probability mass function:

${\displaystyle p(x)=p^{x}(1-p)^{1-x},}$

where ${\displaystyle p}$ is a scalar parameter between 0 and 1. Applying this same functional form on the continuous interval ${\displaystyle [0,1]}$ results in the continuous Bernoulli probability density function, up to a normalizing constant.

Uniform distribution

The Uniform distribution between the unit interval [0,1] is a special case of continuous Bernoulli when ${\displaystyle \lambda =1/2}$ or ${\displaystyle \theta =0}$.

Exponential distribution

An exponential distribution with rate ${\displaystyle \Lambda }$ restricted to the unit interval [0,1] corresponds to a continuous Bernoulli distribution with natural parameter ${\displaystyle \theta =-\Lambda <0}$.

Continuous categorical distribution

The multivariate generalization of the continuous Bernoulli is called the continuous-categorical.^[11]