On mathematics, computation, and ideas.
Why fitting and evaluating on the same data always fools you.
Sample variance underestimates the population variance on average, and the standard fix is to divide by n − 1. The reason: estimating the mean costs exactly one sample's worth of bias, leaving n − 1 residual degrees of freedom. The next post extends this idea to full-rank linear least squares.
Ordinary least squares (OLS) fits a linear model by minimizing squared residuals. Geometrically, fitting k coefficients projects the observations onto a k-dimensional model space, leaving n − k perpendicular directions in which noise can remain. This is the same fact behind n − 1 in sample variance.