{"id":748,"date":"2022-03-01T00:01:50","date_gmt":"2022-03-01T05:01:50","guid":{"rendered":"https:\/\/mathvoices.ams.org\/featurecolumn\/?p=748"},"modified":"2022-03-02T16:55:09","modified_gmt":"2022-03-02T21:55:09","slug":"ordinary-least-squares","status":"publish","type":"post","link":"https:\/\/mathvoices.ams.org\/featurecolumn\/2022\/03\/01\/ordinary-least-squares\/","title":{"rendered":"The Origins of Ordinary Least Squares Assumptions"},"content":{"rendered":"

When we start to think more about it, more questions arise. What makes a line \u201cgood\u201d? How do we tell if a line is the \u201cbest\u201d?<\/em> <\/span><\/p>\n

The Origins of Ordinary Least Squares Assumptions<\/h1>\n

Some Are More Breakable Than Others<\/h2>\n

Sara Stoudt<\/strong>
\nBucknell University<\/strong><\/p>\n

\"Three<\/h1>\n

Introduction<\/h3>\n

Fitting a line to a set of points\u2026 how hard can it be? When those points represent the temperature outside and a town\u2019s ice cream consumption, I\u2019m really invested in that line helping me to understand the relationship between those two quantities. (What if my favorite flavor runs out?!) I might even want to predict new values of ice cream consumption based on new temperature values. A line can give us a way to do that too. But when we start to think more about it, more questions arise. What makes a line \u201cgood\u201d? How do we tell if a line is the \u201cbest\u201d?<\/span><\/p>\n

A technique called ordinary least squares (OLS), aka linear regression, is a principled way to pick the \u201cbest\u201d line where \u201cbest\u201d is defined as the one that minimizes the sum of the squared distances between the line and each point. We chant the assumptions of OLS and know what to look for in diagnostic plots, but where do these assumptions come from? Are there assumptions that are more hurtful than others if broken? These questions are something I even second guess myself, though I frequently use and teach linear regression.<\/span><\/p>\n

Bring on the Math<\/h3>\n

While proving \u201cgood\u201d performance of OLS, we need to make certain assumptions to streamline the process. Walking through the proof reveals what assumptions are most crucial and what the impact is of breaking each one.<\/span><\/p>\n

Let\u2019s first write out our theoretical model in matrix notation and recall what the dimensions of each piece of the puzzle are. We are looking for a linear relationship between $X$ and $Y$, but we know there may be some error, which we represent by $\\epsilon$.<\/span><\/p>\n

\"The<\/h1>\n

The OLS approach estimates the unknown parameters $\\beta$ by <\/span>$\\hat{\\beta} = (X\u2019X)^{-1}X\u2019Y$. Here $X\u2019$ is the transpose of $X$, so the product $X\u2019X$ is a square matrix.<\/span><\/p>\n

 <\/p>\n

\"The<\/h1>\n

Have we implicitly assumed anything yet? Actually yes, there is an inverse in play! What would $X\u2019X$ have to look like to be invertible? Let\u2019s put our linear algebra hats on.<\/span><\/p>\n

This matrix would have to be \u201cfull rank\u201d. Informally, that means that there is no redundant information in the columns of $X$. So if the number of predictor variables $p$ is greater than the number of data points $n$, that would be a problem. Even if $p < n$, the columns of $X$ still need to be linearly independent. Multicollinearity (when one covariate is highly correlated with another covariate) would make us suspicious of this. Let\u2019s keep that in mind just in case it comes back to haunt us.<\/span><\/p>\n

Now that we have our estimator for $\\beta$, there are two properties we want to check on.<\/span><\/p>\n

    \n
  1. We want to on average get the \u201cright\u201d answer.<\/span><\/li>\n
  2. We want to be able to describe the uncertainty in our estimate. In other words, how much would our estimate wiggle around if we happened to have a new sample?<\/span><\/li>\n<\/ol>\n

    Let\u2019s start with determining if we \u201cexpect\u201d to get the right answer on average. It will help to rewrite $Y$ based on the proposed model.<\/span><\/p>\n

    \"We<\/h1>\n

    As we break this expression down, we can see that we are on the right track. However, somehow that second term needs to have expectation zero.<\/span><\/p>\n

    \"The<\/h1>\n

    How can we make this happen? This is where two regression assumptions are born. First we need the errors, $\\epsilon$, to be independent of $X$. This seems plausible. If the errors depend on $X$, somehow we still have some information leftover that is not accounted for in the model. If the errors did<\/em> depend on $X$, that would be a form of heteroscedasticity (non-constant variance for short). That sets our OLS assumption alarm bells ringing. We also need to make an assumption about the magnitude of the errors themselves. It would be nice if they were not systematically positive or negative (that doesn\u2019t seem like a very good model), so assuming they are on average zero seems like a reasonable path forward.<\/span><\/p>\n

    \"Does<\/h1>\n

    With these assumptions in hand we now have an estimator that is unbiased<\/em>. We expect to get the right answer on average. So far we have seen the motivation for two assumptions. If either of those are broken, this unbiasedness is not guaranteed. So if our goal is to make predictions or interpret these coefficients in context, we will be out of luck if these assumptions aren\u2019t met. The next step is to understand the estimator\u2019s uncertainty. Let\u2019s see what other assumptions reveal themselves in the process.<\/span><\/p>\n

    \"The<\/h1>\n

    So let\u2019s recap what we\u2019ve had to assume about the errors so far.<\/span><\/p>\n

    \"Epsilon<\/h1>\n

    What haven\u2019t<\/em> we had to assume yet? Normality! Then why is everyone always worried about that when doing regression? Why all of those qqplots?! Well, we do need to know the whole sampling distribution of the estimates if we want to do inference, or say something about a more general population based on a sample. If we assume the distribution that those errors have is actually Normal, that lets us get a normal sampling distribution of $\\hat{\\beta}$.<\/span><\/p>\n

    \"A \u00a0<\/span><\/h1>\n

    Ah, but there is that pesky inverse again! If we have multicollinearity, that inverse might get unstable, affecting our understanding of the spread of the sampling distribution. And of course nothing in life is perfectly normal. However, we can often get an approximately normal sampling distribution if our sample size is large enough thanks to the Central Limit Theorem (an explainer for another time), so there is some robustness to breaking this particular assumption.<\/span><\/p>\n

    We\u2019ve now seen all of the regression assumptions unfold, so we should be able to build a sort of hierarchy of assumptions.\u00a0<\/span><\/p>\n

    \"The<\/h1>\n

     <\/p>\n

    This hierarchy is also borne out in simulation studies like <\/span>this one<\/span><\/a> that try to stress test OLS. You may even want to code up a simple simulation yourself if you want to look for any gaps between the theory and practice.\u00a0<\/span><\/p>\n

    Some broken assumptions are fixable though. We might transform variables to deal with a lack of linearity or use a Generalized Linear Model that allows the relationship between $X$ and $Y$ to take a non-linear form. We might just be happy with estimating the best linear projection of the true relationship. If errors aren\u2019t independent and\/or do not have constant variance, methods like generalized least squares (GLS) can step in. If errors are not normally distributed, they might be approximately normally distributed thanks to the Central Limit Theorem, or we might want to use a Generalized Linear Model (GLM) that can handle error distributions of other forms. We always have options!<\/span><\/p>\n

    It\u2019s great to see the theory behind the scenes informing the practice, and in general, assumptions of any method have to come from somewhere. Seeking out where in the math assumptions help make our lives easier can help demystify where these assumptions actually come from and gives some insight into which ones, if broken, are more dangerous than others. Happy line fitting!<\/span><\/p>\n

    All the Math in One Place<\/h3>\n

    $Y = X \\beta + \\epsilon$ <\/span><\/p>\n

    $\\hat{\\beta} = (X\u2019X)^{-1}X\u2019Y$<\/span><\/p>\n

     <\/p>\n

    $E[\\hat{\\beta} | X] = \\beta$<\/span><\/p>\n

    $\\hat{\\beta} = (X\u2019X)^{-1} X\u2019 Y = (X\u2019X)^{-1} X\u2019 (X\\beta + \\epsilon)$<\/span><\/p>\n

    $= (X\u2019X)^{-1} X\u2019X\\beta + (X\u2019X)^{-1} X\u2019\\epsilon$<\/span><\/p>\n

    $= \\beta + (X\u2019X)^{-1}X\u2019 \\epsilon$<\/span><\/p>\n

     <\/p>\n

    $E[\\hat{\\beta} | X] = E[(\\beta + (X\u2019X)^{-1} X\u2019 \\epsilon) | X]$<\/span><\/p>\n

    $ = E[\\beta | X] + E[(X\u2019X)^{-1}X\u2019 \\epsilon | X]$<\/span><\/p>\n

    $ = \\beta + (X\u2019X)^{-1} X\u2019 E[\\epsilon | X]$<\/span><\/p>\n

     <\/p>\n

    $E[\\epsilon | X] = 0?$<\/span><\/p>\n

    $\\epsilon_i \\stackrel{iid}{\\sim} N(0, \\sigma^2)$<\/p>\n

    $\\hat{\\beta} \\sim N(\\beta, \\sigma^2 (X’X)^{-1})$<\/p>\n

    References<\/h3>\n