Ordinary Least Squares Linear Regression Walkthrough for Beginners

Ordinary Least Squares Linear Regression

Ordinary linear least squares regression is one of the first statistical methods that anyone learns who is trying to understand how two sets of numbers relate. It has an undeserved reputation of being able to foretell the future, but it is mathematically tractable, simple to apply, and often yields either directly usable results or signals the need for more sophisticated tools.

The idea is simple enough, find the line that passes through points in Cartesian (x,y 2-dimensional space) that minimizes the total distance (another over-simplification) between each point and the line. In slightly informal terms

\[y = mx + b\]

where y is the dependent variable, x is the independent variable, m is the correlation coefficient, or slope and b is the intercept, the value at which the slope intersects the y axis.

This neglects the term \[\epsilon\] arising from measurement error, random variation or both. I’ll set that aside for this discussion.

A regression, such as miles per gallon against horsepower, may give you plot of the paired point values (x,y) something like the following, indicating a trend.

Estimate Std. Error t value Pr(>|t|) (Intercept) 39.94 0.7175 55.66 1.22e-187 horsepower -0.1578 0.006446 -24.49 7.032e-81

Fitting linear model: mpg ~ horsepower Observations Residual Std. Error \(R^2\) Adjusted \(R^2\) 392 4.906 0.6059 0.6049

Even better, the model summary indicates a high coefficient of correlation, R², with a p-value (the unfortunately named “significance”) that is vanishingly small, and so unlikely due to chance.

In another post, I’ve discussed interpreting the model summary, but I haven’t mentioned the graphical methods that are a critical part of understanding what the data are telling you.

Ideally, in the two plots on the left hand side, we would like to see points clustered along the centerline. On the upper right, we would like to see most of the points – the residuals obtained from the distance of a point pair from the dashed line that represents a normal distirubtion – line up. On the lower right, we don’t want to see a curve being “pulled up” by far off outliers.¹

The departures from the ideal are easy enough to spot. What to do about them? Let’s look at the initial data again, this time without the trend line.

How would you draw a line by hand through the points? Would it be two straight lines? A decay curve?

Sometimes, we can improve fit through transforming the scale of one or both variables. A commonly used transformation is to express one variable as a natural logarithm (which is the default in R; other bases are available, such as decimal – log10()). For this dataset a log-log transformation has a slight advantage.

This results in an improvement of R² and a large decrease in the standard error. The plots show minimal non-linearity, the residuals are now very close to being normally distributed, and outliers show negligible effect.

Estimate Std. Error t value Pr(>|t|) (Intercept) 6.961 0.1215 57.3 5.168e-192 log(horsepower) -0.8418 0.02641 -31.88 1.128e-110

Fitting linear model: log(mpg) ~ log(horsepower) Observations Residual Std. Error \(R^2\) Adjusted \(R^2\) 392 0.1793 0.7227 0.722

Sometimes it’s not even this subtle

Here is the famous, contrived “anscombe” dataset that shows why plotting should be the first step in any linear regression. There are four datasets and four models, with practically identical output.

Estimate Std. Error t value Pr(>|t|) (Intercept) 3 1.125 2.667 0.02573 x1 0.5001 0.1179 4.241 0.00217

Fitting linear model: y1 ~ x1 Observations Residual Std. Error \(R^2\) Adjusted \(R^2\) 11 1.237 0.6665 0.6295

Estimate Std. Error t value Pr(>|t|) (Intercept) 3.001 1.125 2.667 0.02576 x2 0.5 0.118 4.239 0.002179

Fitting linear model: y2 ~ x2 Observations Residual Std. Error \(R^2\) Adjusted \(R^2\) 11 1.237 0.6662 0.6292

Estimate Std. Error t value Pr(>|t|) (Intercept) 3.002 1.124 2.67 0.02562 x3 0.4997 0.1179 4.239 0.002176

Fitting linear model: y3 ~ x3 Observations Residual Std. Error \(R^2\) Adjusted \(R^2\) 11 1.236 0.6663 0.6292

Estimate Std. Error t value Pr(>|t|) (Intercept) 3.002 1.124 2.671 0.02559 x4 0.4999 0.1178 4.243 0.002165

Fitting linear model: y4 ~ x4 Observations Residual Std. Error \(R^2\) Adjusted \(R^2\) 11 1.236 0.6667 0.6297

Looking before you leap:

Credits

Software (R Development Core Team (2008). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0, URL http://www.R-project.org.)

data Auto dataset from Gareth James, Daniela Witten, Trevor Hastie and Rob Tibshirani (2017). ISLR: Data for an Introduction to Statistical Learning with Applications in R. R package version 1.2. https://CRAN.R-project.org/package=ISLR; contrived data from Anscombe, Francis J. (1973). Graphs in statistical analysis. The American Statistician, 27, 17–21. doi: 10.2307/2682899, an included dataset in R.

trendline plot H. Wickham. ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York, 2016.

formating of linear regression summary Gergely Daróczi and Roman Tsegelskyi (2018). pander: An R ‘Pandoc’ Writer. R packageversion 0.6.2. https://CRAN.R-project.org/package=pander

1. The technical explanation of the plots I’ve save for another post.↩︎