Residual and Sum-of-Squares Proofs

Concepts

• Least squares doesn’t just hand you a slope and intercept — the way it chooses them forces the residuals to satisfy a few exact cancellation identities.
  • Those identities are why the total variation in $Y$ splits cleanly into explained and unexplained pieces further down.
  • So ANOVA and $R^2$ (covered in a later note) are consequences of least squares, not separate formulas to memorize from scratch.
• A few definitions get reused in every proof below.
  • The fitted value is ${\hat{y}}_i={\hat{\beta }}_0+{\hat{\beta }}_1{x}_i$.
  • The residual is the leftover gap: ${\hat{e}}_i=y_i-{\hat{y}}_i$.
  • The two normal equations are $\sum {\hat{e}}_i=0$ and $\sum x_i{\hat{e}}_i=0$ — recall from the fitting derivation that these come directly from setting the two partial derivatives of SSE equal to zero.
• Proof 1 proves $\sum {\hat{e}}_i=0$ — the residuals sum to exactly zero.
  • Start from the definition of the residual: ${\hat{e}}_i=y_i-{\hat{\beta }}_0-{\hat{\beta }}_1{x}_i$.
  • Sum this over every observation: $\sum {\hat{e}}_i=\sum y_i-n{\hat{\beta }}_0-{\hat{\beta }}_1\sum x_i$.
  • The first normal equation says $n{\hat{\beta }}_0+{\hat{\beta }}_1\sum x_i=\sum y_i$ — substitute that in.
  • The right-hand side becomes $\sum y_i-\sum y_i=0$.
  • This proof is really just the first normal equation, read backwards: differentiating SSE with respect to ${\beta }_0$ and setting it to zero is the statement $\sum {\hat{e}}_i=0$.
  • In plain terms: positive and negative residuals exactly balance out once you’ve fit an intercept, so the average residual is zero.
• Proof 2 proves $\sum x_i{\hat{e}}_i=0$ — the residuals, each weighted by its own $x_i$, also sum to zero.
  • Multiply the residual definition by $x_i$ and sum: $\sum x_i{\hat{e}}_i=\sum x_i{y}_i-{\hat{\beta }}_0\sum x_i-{\hat{\beta }}_1\sum x_i^2$.
  • The second normal equation says ${\hat{\beta }}_0\sum x_i+{\hat{\beta }}_1\sum x_i^2=\sum x_i{y}_i$ — substitute that in, and the sum is $0$.
  • An alternative route that exposes $S_{xy}/S_{xx}$ directly: substitute ${\hat{\beta }}_0=\bar{y}-{\hat{\beta }}_1\bar{x}$ and $\sum x_i=n\bar{x}$ into the expansion instead, group terms into $S_{xy}-{\hat{\beta }}_1{S}_{xx}$, then substitute ${\hat{\beta }}_1=S_{xy}/S_{xx}$ to get $S_{xy}-S_{xy}=0$.
• Proof 3 proves $\sum {\hat{y}}_i{\hat{e}}_i=0$ — the residuals are also orthogonal to the fitted values themselves, not just to the raw $x$‘s.
  • Write ${\hat{y}}_i={\hat{\beta }}_0+{\hat{\beta }}_1{x}_i$ and multiply by ${\hat{e}}_i$: $\sum {\hat{y}}_i{\hat{e}}_i={\hat{\beta }}_0\sum {\hat{e}}_i+{\hat{\beta }}_1\sum x_i{\hat{e}}_i$.
  • Both of those sums are already known to be $0$, from Proofs 1 and 2 — so the whole thing is $0$.
  • This proof is “free” once you have Proofs 1 and 2: ${\hat{y}}_i$ is nothing but a combination of the constant and $x_i$, and the residual is already orthogonal to both of those pieces individually.
• Put together, the three proofs say the residual vector is orthogonal to the intercept column, to the predictor column, and therefore to the fitted values.
  • Geometrically: least squares splits the observed response into a fitted piece that lives in the “model space,” and a residual piece that is perpendicular to it.
• Before Proof 4, three sum-of-squares quantities need defining — each one is a sum of squared gaps, just measured from a different reference point.
  • $SST=\sum (y_i-\bar{y}{)}^2$ is the total variation: how far each $y_i$ sits from the overall mean $\bar{y}$.
  • $S{S}_{reg}=\sum ({\hat{y}}_i-\bar{y}{)}^2$ is the explained variation: how far the model’s fitted values sit from $\bar{y}$. Some sources call this $SSR$.
  • $RSS=\sum (y_i-{\hat{y}}_i{)}^2=\sum {\hat{e}}_i^2$ is the unexplained variation: the leftover residuals, squared. Some sources call this $SSE$.
  • Naming warning: textbooks disagree on whether “SSR” means regression or residual — rely on the formula itself, not the three letters.
• Proof 4 shows these three quantities fit together exactly: $SST=S{S}_{reg}+RSS$.
  • Step 1 splits each observation’s total deviation into two pieces, by inserting and subtracting the fitted value ${\hat{y}}_i$: $y_i-\bar{y}=(y_i-{\hat{y}}_i)+({\hat{y}}_i-\bar{y})={\hat{e}}_i+({\hat{y}}_i-\bar{y})$.
    • This rewrites “total deviation” as “unexplained deviation plus explained deviation.”
    • It’s the same telescoping trick used elsewhere in the course: insert and subtract a middle quantity so one unfamiliar gap becomes two gaps you already have names for.
  • Step 2 squares both sides of that split: $(y_i-\bar{y}{)}^2={\hat{e}}_i^2+2{\hat{e}}_i({\hat{y}}_i-\bar{y})+({\hat{y}}_i-\bar{y}{)}^2$.
  • Step 3 sums that equation over every observation: $SST=RSS+2\sum {\hat{e}}_i({\hat{y}}_i-\bar{y})+S{S}_{reg}$.
  • Step 4 shows the middle “cross term” is exactly zero.
    • Expand it: $\sum {\hat{e}}_i({\hat{y}}_i-\bar{y})=\sum {\hat{e}}_i{\hat{y}}_i-\bar{y}\sum {\hat{e}}_i$.
    • The first piece is $0$ by Proof 3, and the second is $0$ by Proof 1.
    • So the cross term vanishes entirely.
    • This step is the whole payoff of Proofs 1–3 — they exist specifically so this cross term dies, which is what makes $SST=S{S}_{reg}+RSS$ an exact equality rather than an approximation.
  • Step 5 drops the now-zero cross term to conclude: $\boxed{SST=S{S}_{reg}+RSS}$.
  • This works cleanly because least squares made the residuals orthogonal to the fitted values (Proof 3) — a model that wasn’t fit by least squares wouldn’t get this identity for free.

Practice Problems

• Problem 1 (HW1 Q1). Prove $\sum {\hat{e}}_i=0$ and $\sum x_i{\hat{e}}_i=0$ using the normal equations. Write it the way you would on the exam — the Concepts section above has the fully justified version; this is the fast, exam-pace version of the same two proofs.

  • Solution.
    • $\sum {\hat{e}}_i=0$:
      • ${\hat{e}}_i=y_i-{\hat{\beta }}_0-{\hat{\beta }}_1{x}_i$
      • $\sum {\hat{e}}_i=\sum y_i-n{\hat{\beta }}_0-{\hat{\beta }}_1\sum x_i$
      • first normal equation: $n{\hat{\beta }}_0+{\hat{\beta }}_1\sum x_i=\sum y_i$
      • $\Rightarrow \sum {\hat{e}}_i=\sum y_i-\sum y_i=0$ $■$
    • $\sum x_i{\hat{e}}_i=0$:
      • $x_i{\hat{e}}_i=x_i(y_i-{\hat{\beta }}_0-{\hat{\beta }}_1{x}_i)$
      • $\sum x_i{\hat{e}}_i=\sum x_i{y}_i-{\hat{\beta }}_0\sum x_i-{\hat{\beta }}_1\sum x_i^2$
      • second normal equation: ${\hat{\beta }}_0\sum x_i+{\hat{\beta }}_1\sum x_i^2=\sum x_i{y}_i$
      • $\Rightarrow \sum x_i{\hat{e}}_i=\sum x_i{y}_i-\sum x_i{y}_i=0$ $■$

• Problem 2 (HW1 Q3). Prove $SST=S{S}_{reg}+RSS$. Same exam-pace instruction as above.

  • Solution.
    • $y_i-\bar{y}=(y_i-{\hat{y}}_i)+({\hat{y}}_i-\bar{y})={\hat{e}}_i+({\hat{y}}_i-\bar{y})$
    • square both sides: $(y_i-\bar{y}{)}^2={\hat{e}}_i^2+2{\hat{e}}_i({\hat{y}}_i-\bar{y})+({\hat{y}}_i-\bar{y}{)}^2$
    • sum over $i$: $SST=RSS+2\sum {\hat{e}}_i({\hat{y}}_i-\bar{y})+S{S}_{reg}$
    • cross term: $\sum {\hat{e}}_i({\hat{y}}_i-\bar{y})=\sum {\hat{e}}_i{\hat{y}}_i-\bar{y}\sum {\hat{e}}_i=0-\bar{y}(0)=0$ (using $\sum {\hat{e}}_i{\hat{y}}_i=0$ and $\sum {\hat{e}}_i=0$)
    • $\boxed{SST=S{S}_{reg}+RSS}$ $■$

• Problem 3 — numeric verification (class data). Using the fitted line $\hat{y}=1.7+0.7x$ from the class dataset $(-1,1),(0,2),(1,2),(2,3),(3,4)$, compute all five residuals and verify both residual identities and the decomposition $SST=S{S}_{reg}+RSS$ numerically.

  • Solution.
    • residuals: $x=-1$: $\hat{e}=0$; $x=0$: $\hat{e}=0.3$; $x=1$: $\hat{e}=-0.4$; $x=2$: $\hat{e}=-0.1$; $x=3$: $\hat{e}=0.2$
    • check $\sum {\hat{e}}_i=0+0.3-0.4-0.1+0.2=0$
    • check $\sum x_i{\hat{e}}_i=(-1)(0)+(0)(0.3)+(1)(-0.4)+(2)(-0.1)+(3)(0.2)=0$
    • $RSS=0^2+0.3^2+(-0.4{)}^2+(-0.1{)}^2+0.2^2=0.30$
    • $\bar{y}=2.4$, so $SST=(1-2.4{)}^2+(2-2.4{)}^2+(2-2.4{)}^2+(3-2.4{)}^2+(4-2.4{)}^2=5.20$
    • $S{S}_{reg}=SST-RSS=5.20-0.30=4.90$
    • check: $5.20=4.90+0.30$ ✓