Fitting a Simple Linear Regression

Concepts

• We have observed pairs $(x_i,y_i)$ and want one straight line that summarizes how $Y$ changes with $x$.
  • The population model assumes a true but unknown line plus random noise: $Y_i={\beta }_0+{\beta }_1{x}_i+{\epsilon }_i$.
  • ${\beta }_0,{\beta }_1$ are the true population intercept and slope — we never observe them directly.
  • From the sample, we estimate them with ${\hat{\beta }}_0,{\hat{\beta }}_1$ (the hat just means “estimated from data”).
  • The fitted line, our best guess at the true line, is $\hat{y}={\hat{\beta }}_0+{\hat{\beta }}_{1}x$.
  • End goal of least squares: choose ${\hat{\beta }}_0,{\hat{\beta }}_1$ so the fitted line has the smallest possible total squared vertical error from the observed points.
• The starting point is the residual — the vertical miss between a candidate line and the data.
  • For observation $i$, a candidate line predicts ${\hat{y}}_i={\beta }_0+{\beta }_1{x}_i$.
  • The residual is the gap between what actually happened and what the line predicted: $e_i=y_i-{\beta }_0-{\beta }_1{x}_i$.
  • Adding up the raw residuals is a bad way to measure total error, because a positive miss and a negative miss of the same size would cancel out and hide the fact that the line is bad.
  • Squaring each residual before adding fixes that, since squares are always positive.
  • This gives the sum of squared errors: $SSE({\beta }_0,{\beta }_1)=\sum _{i=1}^n(y_i-{\beta }_0-{\beta }_1{x}_i{)}^2.$
  • Everything that follows is machinery for making $SSE$ as small as possible.
• A few summation facts are needed before the derivation, all of which follow from the definition of a mean.
  • $\sum x_i=n\bar{x}$ and $\sum y_i=n\bar{y}$ — a sum is just the mean multiplied by how many terms there are.
  • $\sum (x_i-\bar{x})=0$ — deviations from the mean always cancel out exactly, by definition of the mean.
  • $S_{xx}=\sum (x_i-\bar{x}{)}^2=\sum x_i^2-n{\bar{x}}^2$ — this is the sum of squared deviations of $x$ from its own mean, written two equivalent ways (the second is a shortcut for hand computation).
  • $S_{xy}=\sum (x_i-\bar{x})(y_i-\bar{y})=\sum x_i{y}_i-n\bar{x}\bar{y}$ — the same idea, but for how $x$ and $y$ deviate together.
  • $S_{xx}$ and $S_{xy}$ are really just the sample variance and covariance before dividing by $n-1$. That’s why ${\hat{\beta }}_1=S_{xy}/S_{xx}$ later reads as “covariance over variance” — the same ratio as $\mathrm{cov}(x,y)/\mathrm{var}(x)$, since the $(n-1)$ in each cancels.
  • Pitfall: $\sum a_i{b}_i\ne (\sum a_i)(\sum b_i)$, and $\sum a_i^2\ne (\sum a_i{)}^2$ — you cannot pull a sum apart like that.
• Derive the estimators by minimizing $SSE$ with calculus: take a partial derivative with respect to each unknown, and set each one to zero.
  • Step 1 finds the condition on ${\beta }_0$.
    • Differentiate: $\frac{\partial SSE}{\partial {\beta }_0}=-2\sum (y_i-{\beta }_0-{\beta }_1{x}_i)$.
    • Set the derivative to $0$, since that’s what makes a point a minimum: $\sum (y_i-{\beta }_0-{\beta }_1{x}_i)=0$.
    • Expand and rearrange: $\sum y_i=n{\beta }_0+{\beta }_1\sum x_i$ — this is called the first normal equation.
    • Divide every term by $n$ and use the mean identities from above: $\bar{y}={\beta }_0+{\beta }_1\bar{x}$, so $\boxed{{\beta }_0=\bar{y}-{\beta }_1\bar{x}}$.
    • This equation says the fitted line is forced to pass through the point $(\bar{x},\bar{y})$, no matter what the slope turns out to be.
  • Step 2 finds the second condition, this time on ${\beta }_1$.
    • Differentiate with respect to ${\beta }_1$ instead: $\frac{\partial SSE}{\partial {\beta }_1}=-2\sum x_i(y_i-{\beta }_0-{\beta }_1{x}_i)$.
    • Set it to $0$ and rearrange: $\sum x_i{y}_i={\beta }_0\sum x_i+{\beta }_1\sum x_i^2$ — the second normal equation.
  • Step 3 solves the two normal equations together by substitution.
    • Plug ${\beta }_0=\bar{y}-{\beta }_1\bar{x}$ (from Step 1) and $\sum x_i=n\bar{x}$ into the second normal equation.
    • Expanding and collecting the ${\beta }_1$ terms on one side gives $(\sum x_i{y}_i-n\bar{x}\bar{y})-{\beta }_1(\sum x_i^2-n{\bar{x}}^2)=0$.
    • The two parentheses are exactly $S_{xy}$ and $S_{xx}$ from the summation facts above, so this reads $S_{xy}-{\beta }_1{S}_{xx}=0$.
    • Solving for ${\beta }_1$ gives $\boxed{{\hat{\beta }}_1=\frac{S_{xy}}{S_{xx}}}$.
  • Step 4 plugs the slope back into Step 1’s equation to recover the intercept: $\boxed{{\hat{\beta }}_0=\bar{y}-{\hat{\beta }}_1\bar{x}}$.
  • Step 5 assembles the fitted line: $\boxed{\hat{y}={\hat{\beta }}_0+{\hat{\beta }}_{1}x}$.
  • This stationary point really is a minimum (not a maximum or saddle) because $SSE$ is a quadratic bowl in $({\beta }_0,{\beta }_1)$, and as long as the $x_i$ aren’t all identical, $S_{xx}>0$, so the bowl opens upward.
• The two normal equations from Steps 1–2 have a direct restatement in terms of the residuals: $\sum {\hat{e}}_i=0$ and $\sum x_i{\hat{e}}_i=0$.
  • This is usually phrased as “the least-squares residual vector is orthogonal to the constant column and to $x$.”
  • In plain terms, that just means the leftover residual has nothing left to explain: no overall bias (the constant direction), and no leftover linear relationship with $x$ (the $x$ direction).
• Centering around the mean keeps appearing throughout this derivation for a reason.
  • $S_{xx}$ measures how spread out $x$ is around $\bar{x}$.
  • $S_{xy}$ measures how $x$ and $y$ move together around their means.
  • So ${\hat{\beta }}_1=S_{xy}/S_{xx}$ reads as “how much $y$ moves with $x$, relative to how much $x$ itself varies.”
• The exam workflow for a hand-computed fit is always the same chain of steps.
  • Compute the five raw sums: $n,\sum x_i,\sum y_i,\sum x_i^2,\sum x_i{y}_i$.
  • Convert to the means $\bar{x},\bar{y}$.
  • Convert to the centered sums $S_{xx},S_{xy}$.
  • Compute ${\hat{\beta }}_1=S_{xy}/S_{xx}$, then ${\hat{\beta }}_0=\bar{y}-{\hat{\beta }}_1\bar{x}$.
  • Write the fitted line, and if a residual is asked for, compute ${\hat{e}}_i=y_i-{\hat{y}}_i$ at the requested point.

Practice Problems

• Problem 1 — hand-compute a fit. Data $(-1,1),(0,2),(1,2),(2,3),(3,4)$, with $n=5$, $\sum x_i=5$, $\sum y_i=12$, $\sum x_i{y}_i=19$, $\sum x_i^2=15$. Find the fitted line and the residual at $x=2$.

  • Solution.
    • $\bar{x}=1$, $\bar{y}=2.4$
    • $S_{xy}=19-5(1)(2.4)=7$
    • $S_{xx}=15-5(1{)}^2=10$
    • ${\hat{\beta }}_1=7/10=0.7$
    • ${\hat{\beta }}_0=2.4-0.7(1)=1.7$
    • $\boxed{\hat{y}=1.7+0.7x}$
    • at $x=2$: $\hat{y}=3.1$, observed $y=3$, so $\hat{e}=3-3.1=-0.1$

• Problem 2 — derive from scratch, cold. Derive the two normal equations from $SSE=\sum (y_i-{\beta }_0-{\beta }_1{x}_i{)}^2$ and solve them for ${\hat{\beta }}_1$ and ${\hat{\beta }}_0$. Write it the way you would on the exam — the Concepts section above has the fully justified version; this is the fast, exam-pace version of the same argument.

  • Solution.
    • $\frac{\partial SSE}{\partial {\beta }_0}=-2\sum (y_i-{\beta }_0-{\beta }_1{x}_i)=0$
      • $\Rightarrow \sum y_i=n{\beta }_0+{\beta }_1\sum x_i$ — first normal equation
      • $\Rightarrow {\beta }_0=\bar{y}-{\beta }_1\bar{x}$
    • $\frac{\partial SSE}{\partial {\beta }_1}=-2\sum x_i(y_i-{\beta }_0-{\beta }_1{x}_i)=0$
      • $\Rightarrow \sum x_i{y}_i={\beta }_0\sum x_i+{\beta }_1\sum x_i^2$ — second normal equation
    • substitute ${\beta }_0=\bar{y}-{\beta }_1\bar{x}$ and $\sum x_i=n\bar{x}$ into the second equation
      • $\sum x_i{y}_i=(\bar{y}-{\beta }_1\bar{x})n\bar{x}+{\beta }_1\sum x_i^2$
      • $\sum x_i{y}_i-n\bar{x}\bar{y}={\beta }_1(\sum x_i^2-n{\bar{x}}^2)$
      • $S_{xy}={\beta }_1{S}_{xx}$
    • $\boxed{{\hat{\beta }}_1=\frac{S_{xy}}{S_{xx}}}$
    • $\boxed{{\hat{\beta }}_0=\bar{y}-{\hat{\beta }}_1\bar{x}}$