Deviance information criterion

The deviance information criterion (DIC) is a hierarchical modeling generalization of the Akaike information criterion (AIC). It is particularly useful in Bayesian model selection problems where the posterior distributions of the models have been obtained by Markov chain Monte Carlo (MCMC) simulation. DIC is an asymptotic approximation as the sample size becomes large, like AIC. It is only valid when the posterior distribution is approximately multivariate normal.

Definition

Define the deviance as ${\displaystyle D(\theta )=-2\log(p(y|\theta ))+C\,}$, where ${\displaystyle y}$ are the data, ${\displaystyle \theta }$ are the unknown parameters of the model and ${\displaystyle p(y|\theta )}$ is the likelihood function. ${\displaystyle C}$ is a constant that cancels out in all calculations that compare different models, and which therefore does not need to be known.

There are two calculations in common usage for the effective number of parameters of the model. The first, as described in Spiegelhalter et al. (2002, p. 587), is ${\displaystyle p_{D}={\overline {D(\theta )}}-D({\bar {\theta }})}$, where ${\displaystyle {\bar {\theta }}}$ is the expectation of ${\displaystyle \theta }$. The second, as described in Gelman et al. (2004, p. 182), is ${\displaystyle p_{D}=p_{V}={\frac {1}{2}}{\overline {\operatorname {var} \left(D(\theta )\right)}}}$. The larger the effective number of parameters is, the easier it is for the model to fit the data, and so the deviance needs to be penalized.

The deviance information criterion is calculated as

${\displaystyle \mathrm {DIC} =p_{D}+{\overline {D(\theta )}},}$

or equivalently as

${\displaystyle \mathrm {DIC} =D({\bar {\theta }})+2p_{D}.}$

From this latter form, the connection with AIC is more evident.

Motivation

The idea is that models with smaller DIC should be preferred to models with larger DIC. Models are penalized both by the value of ${\displaystyle {\bar {D}}}$, which favors a good fit, but also (similar to AIC) by the effective number of parameters ${\displaystyle p_{D}}$. Since ${\displaystyle {\bar {D}}}$ will decrease as the number of parameters in a model increases, the ${\displaystyle p_{D}}$ term compensates for this effect by favoring models with a smaller number of parameters.

An advantage of DIC over other criteria in the case of Bayesian model selection is that the DIC is easily calculated from the samples generated by a Markov chain Monte Carlo simulation. AIC requires calculating the likelihood at its maximum over ${\displaystyle \theta }$, which is not readily available from the MCMC simulation. But to calculate DIC, simply compute ${\displaystyle {\bar {D}}}$ as the average of ${\displaystyle D(\theta )}$ over the samples of ${\displaystyle \theta }$, and ${\displaystyle D({\bar {\theta }})}$ as the value of ${\displaystyle D}$ evaluated at the average of the samples of ${\displaystyle \theta }$. Then the DIC follows directly from these approximations. Claeskens and Hjort (2008, Ch. 3.5) show that the DIC is large-sample equivalent to the natural model-robust version of the AIC.

Assumptions

In the derivation of DIC, it is assumed that the specified parametric family of probability distributions that generate future observations encompasses the true model. This assumption does not always hold, and it is desirable to consider model assessment procedures in that scenario.

Also, the observed data are used both to construct the posterior distribution and to evaluate the estimated models. Therefore, DIC tends to select over-fitted models.

Extensions

A resolution to the issues above was suggested by Ando (2007), with the proposal of the Bayesian predictive information criterion (BPIC). Ando (2010, Ch. 8) provided a discussion of various Bayesian model selection criteria. To avoid the over-fitting problems of DIC, Ando (2011) developed Bayesian model selection criteria from a predictive view point. The criterion is calculated as

${\displaystyle {\mathit {IC}}={\bar {D}}+2p_{D}=-2\mathbf {E} ^{\theta }[\log(p(y|\theta ))]+2p_{D}.}$

The first term is a measure of how well the model fits the data, while the second term is a penalty on the model complexity. Note that the p in this expression is the predictive distribution rather than the likelihood above.

Other Applications of DIC

DIC was used in multiple S-Plus (and subsequently R) libraries for fitting likelihood-based models in the 1990's (having precedent over the Bayesian methods, to the extent they overlap); usually presented as a generalization of AIC. The DIC was defined by Hastie and Tibshirani (1990 p160, eqn 6.32)^[1] for the weighted smoothers used in Generalized Additive Models, and the requisite deviance and effective degrees-of-freedom calculations were incorporated into the GAM library (Hastie, 1991).^[2]

The aic method in S-Plus and R is credited (initially) to Pinheiro and Bates, developed in conjunction with nlme software,^[3] and subsequently backported to other libraries (some are simply AIC; others requiring DIC-style approximations).

In the context of local likelihood, a deviance information criterion is defined Loader (1999 p69, def'n 4.4),^[4] with a derivation based on jackknifed leave-one-out cross-validation, with effective degrees-of-freedom calculations explicitly in terms of likelihood derivatives. This leads to some small technical differences compared to the Hastie and Tibshirani approach.

Irizarry (2001) ^[5] also has an extensive development of information criteria for local likelihood. Unlike the global techniques in the above sources, the criteria developed by Irizarry are applied when estimating at a single point in the predictor space, and so are applicable to locally adaptive smoothers considered by other authors.

See also

• Akaike information criterion (AIC)
• Bayesian information criterion (BIC)
• Focused information criterion (FIC)
• Hannan-Quinn information criterion
• Kullback–Leibler divergence
• Jensen–Shannon divergence
• Watanabe–Akaike information criterion (WAIC)