Speaker: Dr. Stephen Portnoy
Title: When Are Unbiased Estimators Linear?
Abstract: The problem of fitting a linear model to data originated in the mid 18th century. The earliest approach was developed for simple linear models by Boscovitch and was based on minimizing the sum of absolute errors. Unfortunately, Boscovitch's geometric computational method was rather complicated, and did not apply when there were multiple x's. Thus, the least squares approach of Legendre and Gauss developed nearly 50 years later and became the standard method. In 1822, Gauss showed that the least squares estimator was a "Best linear unbiased estimator" (BLUE) in terms of variance minimization. Following the development of statistical theory in the mid-twentieth century, BLUE's are known to be optimal in many respects under normal assumptions. However, since variance minimization doesn’t depend on normality and unbiasedness is often considered reasonable, many statisticians have felt that BLUE’s ought to perform relatively well in some generality.
The result here considers the general linear model and shows that any measurable estimator that is unbiased over a moderately large (but finite dimensional) family of distributions must be linear. Thus, imposing unbiasedness cannot offer improvement over imposing linearity. The problem was suggested by Hansen, who showed that any estimator unbiased for nearly all error distributions (with finite covariance) must have a variance no smaller than that of the best linear estimator in some parametric subfamily. Specifically, the hypothesis of linearity can be dropped from the classical Gauss–Markov Theorem. This might suggest that the best unbiased estimator should provide superior performance, but the result here shows that the best unbiased regression estimator can be no better than the best linear estimator, and non-linear estimators are often substantially superior.
The result appeared in The American Statistician in late 2022, but a technical error in part of the proof was discovered. A new and much simpler proof will be presented and implications and generaliztions will be discussed.
Gauss probably used least squares around 1800, but didn't communicate it to other mathematicians until publishing it in 1809, after publications by Legnedre (1805) and the American Robert Adrain (1808).
Note: the idea of squaring errors does seem rather pointless and even unreasonable. It really makes sense only under normal assumptions. In fact, the idea of using absolute error to compare estimators was introduced by Galileo, and the use of absolute errors continued to be supported by Laplace and others. Nonetheless, methods to minimize the sum of absolute errors were not developed until the mid-twentieth century when they could be based on linear programming.