If we were working with low dosages and using the correlation between dosage and outcomes as our metric of assessing effectiveness, we might underestimate how well the drug is actually working...
Footnote Finding
When Correlation and Regression Slopes Tell (Slightly) Different Stories
Sara Stoudt
Bucknell University
I recently read Robert P. Abelson’s book, Statistics as Principled Argument, and found a lot of interesting gems both about how to talk about statistics and the craft of doing statistics. If you are looking for some insight about how to present statistical results convincingly or for case studies of “gotcha” moments in statistics, it’s worth a read. I won’t get into a full book review here. Instead, I want to dig into one of the footnotes (pages 50 and 51) that piqued my curiosity. The note is about making a choice of how to measure the strength of a relationship between two things. For instance, if I want to say that taking a new drug is associated with improved metabolism, what type of statistic should I use to quantify that? It turns out, that choice matters (at least a little bit)!
Here is the footnote:
Correlation coefficients are sensitive to an artifact called ‘restriction of range’ (Cohen & Cohen, 1983, pp. 70-72): When the distribution of scores on variable $X$ is for some reason truncated so as to eliminate the tails, the correlations between $X$ and other variables decline. The same is not true for linear regressions, however. The slope (b) of the regression line giving the expected magnitude of some variable as a function of $X$ is not systematically affected by truncation.
In summary, if the tails of a distribution of $X$ are truncated before investigating $X$’s relationship to $Y$, the correlation between the truncated $X$ and $Y$ will dampen, but the estimated slope of the relationship between the truncated $X$ and $Y$ will not. Why should we care about this?
Abelson goes on to say:
The relevance of this point is that small cause sizes are analogous to restricted ranges. Rosenthal’s (1991) measure $r$ is therefore apt to come out too small whenever the independent variable is experimentally manipulated only modestly. By contrast, causal efficacy measures will (assuming linearity) tend to come out the same whether the cause sizes are small or large.
The “cause size” is the magnitude of the intervention. If we think about a case where we are trying to understand how well a drug works based on a dosage level, a small dosage would be a small cause size. If we were working with low dosages and using the correlation between dosage and outcomes as our metric of assessing effectiveness, we might underestimate how well the drug is actually working. However, if we used the magnitude of the slope between dosage and outcomes, we would have a more robust understanding of the effectiveness, even if we only observe outcomes for low dosages.
I wanted to see for myself what is going on here, so I coded up a little simulation to see what happens. The R code I used for this can be found here. Basically, I generated different relationships between a single covariate $X$ and a response $Y$ (varying how much variability there was in the error term of a regression model and the strength of the relationship as measured by the slope). I then compared the correlation between $X$ and $Y$ to the correlation between $X$ and $Y$ when the lowest and highest 5% of the data was removed (effectively truncating $X$). I did the same for estimating the slope between $X$ and $Y$ via linear regression.
When I compared the observed correlations of the full data to the correlations of the truncated data, Abelson’s footnote proved true. Below we see that when the correlation is negative, the truncated correlation is less extreme than the overall correlations (so the overall correlations are closer to -1). Similarly, for positive correlations, the original correlations are greater than the truncated correlations (so the overall correlations are closer to 1).
If keeping track of the sign is confusing, we can look at this on the absolute value scale and see that the truncated correlations are attenuated towards zero (all points are below the $y = x$ line on the absolute value scale).
However, if we look at the estimated regression slope for these situations, we see a different story that continues to support Abelson’s footnote. Now the estimated slope coefficient does not seem to be affected by the truncation of the data.
I didn’t expect the findings that Abelson mentioned to be so cut and dry in just a simple example, but we see exactly what we are supposed to. What’s the intuition behind all of this though? Well, we know that there is a relationship between the slope and the correlation in at least the simple linear regression case: the slope is equal to the correlation multiplied by the ratio of the response variable’s standard deviation to the explanatory variable’s standard deviation. If the slope is robust to truncation, but the correlation is dampened, that must mean that the ratio of standard deviations gets slightly magnified to compensate. It’s not immediately obvious to me why this would be true, but luckily for us, we can keep track of this ratio as part of our mini simulation study.
Below we actually do see this play out as predicted. The ratio of standard deviations is greater for the truncated cases than for the original cases. This effect isn’t dramatic, which is to be expected as the correlations were only dampened a little bit above.
Now we need to build some intuition about what makes this standard deviation ratio amplify a bit in the truncated scenarios. To me, it seems like standard deviations should shrink for truncated data in general, but it must be true that this happens more for the explanatory variable than it does for the response. Again, we turn to the simulation study to see if this is true.
To compare how much each standard deviation changes, we take the ratio of standard deviations (original v. truncated) for each variable and compare them. Below, we see that this ratio is bigger (meaning the truncated version shrinks more) for the explanatory variable than for the response (it’s the same on the y-axis because the explanatory variable was the same for every scenario in this simulation study).
Isn’t it so satisfying when a simulation so clearly backs up the theory or the general wisdom that we are trying to show? There are a few more in Abelson’s book that I may dig into in future posts, but in the meantime, reach out with any fun footnotes you have to share!
References
- Robert P. Abelson (1995) Statistics as Principled Argument. Hillsdale, NJ and Hove, UK: Lawrence Erlbaum Associates.
- Cohen, J. & Cohen, P. (1983). Applied multiple regression and correlation analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.
- Rosenthal, R. (1991). Meta-analytic procedures for social research (rev. ed.). Newbury Park, CA: Sage.