Minimum mean square error
Estimation method that minimizes the mean square error / From Wikipedia, the free encyclopedia
Dear Wikiwand AI, let's keep it short by simply answering these key questions:
Can you list the top facts and stats about Minimum mean-square error?
Summarize this article for a 10 year old
In statistics and signal processing, a minimum mean square error (MMSE) estimator is an estimation method which minimizes the mean square error (MSE), which is a common measure of estimator quality, of the fitted values of a dependent variable. In the Bayesian setting, the term MMSE more specifically refers to estimation with quadratic loss function. In such case, the MMSE estimator is given by the posterior mean of the parameter to be estimated. Since the posterior mean is cumbersome to calculate, the form of the MMSE estimator is usually constrained to be within a certain class of functions. Linear MMSE estimators are a popular choice since they are easy to use, easy to calculate, and very versatile. It has given rise to many popular estimators such as the Wiener–Kolmogorov filter and Kalman filter.
The term MMSE more specifically refers to estimation in a Bayesian setting with quadratic cost function. The basic idea behind the Bayesian approach to estimation stems from practical situations where we often have some prior information about the parameter to be estimated. For instance, we may have prior information about the range that the parameter can assume; or we may have an old estimate of the parameter that we want to modify when a new observation is made available; or the statistics of an actual random signal such as speech. This is in contrast to the non-Bayesian approach like minimum-variance unbiased estimator (MVUE) where absolutely nothing is assumed to be known about the parameter in advance and which does not account for such situations. In the Bayesian approach, such prior information is captured by the prior probability density function of the parameters; and based directly on Bayes theorem, it allows us to make better posterior estimates as more observations become available. Thus unlike non-Bayesian approach where parameters of interest are assumed to be deterministic, but unknown constants, the Bayesian estimator seeks to estimate a parameter that is itself a random variable. Furthermore, Bayesian estimation can also deal with situations where the sequence of observations are not necessarily independent. Thus Bayesian estimation provides yet another alternative to the MVUE. This is useful when the MVUE does not exist or cannot be found.
Let be a hidden random vector variable, and let be a known random vector variable (the measurement or observation), both of them not necessarily of the same dimension. An estimator of is any function of the measurement . The estimation error vector is given by and its mean squared error (MSE) is given by the trace of error covariance matrix
where the expectation is taken over conditioned on . When is a scalar variable, the MSE expression simplifies to . Note that MSE can equivalently be defined in other ways, since
The MMSE estimator is then defined as the estimator achieving minimal MSE:
- When the means and variances are finite, the MMSE estimator is uniquely defined[1] and is given by:
- In other words, the MMSE estimator is the conditional expectation of given the known observed value of the measurements. Also, since is the posterior mean, the error covariance matrix is equal to the posterior covariance matrix,
- .
- The MMSE estimator is unbiased (under the regularity assumptions mentioned above):
- The MMSE estimator is asymptotically unbiased and it converges in distribution to the normal distribution:
- where is the Fisher information of . Thus, the MMSE estimator is asymptotically efficient.
- The orthogonality principle: When is a scalar, an estimator constrained to be of certain form is an optimal estimator, i.e. if and only if
- for all in closed, linear subspace of the measurements. For random vectors, since the MSE for estimation of a random vector is the sum of the MSEs of the coordinates, finding the MMSE estimator of a random vector decomposes into finding the MMSE estimators of the coordinates of X separately:
- for all i and j. More succinctly put, the cross-correlation between the minimum estimation error and the estimator should be zero,
- If and are jointly Gaussian, then the MMSE estimator is linear, i.e., it has the form for matrix and constant . This can be directly shown using the Bayes theorem. As a consequence, to find the MMSE estimator, it is sufficient to find the linear MMSE estimator.