Regression Diagnostics and Leverage

Concepts

• Regression calculations will always hand you a fitted line, even when a straight-line model is secretly the wrong choice for the data.
  • Diagnostics are the follow-up check: do the assumptions behind that line and its inference actually hold, and does any individual observation have unusual influence on the result?
  • The end goal is to be able to look at a plot, or a single observation, and name exactly what kind of problem it represents.
• Residuals are the main diagnostic tool because they stand in for the errors we can never actually observe.
  • The true model is $Y_i={\beta }_0+{\beta }_1{x}_i+{\epsilon }_i$, where ${\epsilon }_i$ is the true, unobservable random error.
  • The residual is ${\hat{e}}_i=Y_i-{\hat{Y}}_i$ — the gap between the data and our fitted line, which we can compute.
  • If the fitted line is close to the true mean line, then ${\hat{Y}}_i\approx {\beta }_0+{\beta }_1{x}_i$, which makes ${\hat{e}}_i\approx {\epsilon }_i$.
  • So as long as the model is reasonable, the residuals are a usable, observable stand-in for the errors we can’t see.
• A healthy residual plot has a specific, recognizable look.
  • Roughly random scatter centered around $0$, with no systematic curvature and no trend against $x$ or the fitted values.
  • Roughly constant vertical spread across the whole plot.
  • A few individually large residuals are normal — the problem sign is a systematic pattern, not occasional size.
• Each kind of departure from that healthy look points to a specific underlying problem.
  • Curvature (a U or inverted-U shape) suggests the true mean function isn’t actually linear.
  • A funnel shape (spread that grows or shrinks) suggests the error variance isn’t constant.
  • One isolated, large vertical residual suggests a response outlier.
  • Runs or clusters suggest the errors are correlated, if the observations have a natural order.
  • A Q-Q plot that bends away from a straight line suggests the errors aren’t normally distributed.
• A concrete example of the first problem: fitting a line to data that’s actually quadratic leaves a visible pattern in the residuals.
  • Suppose the truth is $Y_i={\beta }_0+{\beta }_1{x}_i+{\beta }_2{x}_i^2+{\epsilon }_i$, but we only fit ${\hat{Y}}_i\approx {\beta }_0+{\beta }_1{x}_i$ (missing the $x^2$ term).
  • The residual then works out to ${\hat{e}}_i\approx {\beta }_2{x}_i^2+{\epsilon }_i$ — it still carries a systematic $x_i^2$ term that the line never captured.
  • That leftover $x_i^2$ term is exactly why curvature shows up in the residual plot: a pattern in the residuals means the model failed to explain a systematic part of the response, not just random noise.
• The Q-Q plot checks the normality assumption by comparing the residuals’ shape to what a normal distribution’s shape should look like.
  • Plot the ordered standardized residuals against the order statistics you’d expect from a standard normal distribution.
  • Points sitting close to a straight line are consistent with normality; systematic bending away from the line is evidence against it.
  • The matching works like this: with $n$ points total, the $k$-th smallest residual is compared against roughly the $k/n$ quantile of the normal distribution — e.g. with $n=200$ points, the 6th-smallest one is compared against the $6/200=3$rd percentile.
  • R’s plot(m) produces four such diagnostic panels automatically: residuals vs. fitted (checks model shape), Q-Q (checks normality), a scale-location plot (checks constant variance, using $\sqrt{|r_i|}$), and residuals vs. leverage.
• The constant-variance diagnostic checks whether the error variance actually stays the same across the data, which is one of the model’s core assumptions.
  • Plot residuals (or, better, standardized residuals — explained below) against fitted or predictor values.
  • The warning sign is a cloud that visibly widens or narrows as the fitted values increase.
  • When that happens, the standard fix is a transformation — a common rule of thumb is to take the square root of both variables, $\sqrt{y}=\alpha +\beta \sqrt{x}$, not just $y$ alone.
  • A more general transform, Box–Cox, exists for this same problem but is out of scope here.
• Leverage asks a different question from a residual: is this observation’s $x$-value itself unusual, regardless of what its $y$-value turned out to be.
  • A residual is about the response — is $y_i$ unusual given its $x_i$?
  • Leverage is about the predictor — is $x_i$ itself far from the rest of the $x$‘s?
  • In simple linear regression, leverage has a closed form: $\boxed{h_{ii}=\frac{1}{n}+\frac{(x_i-\bar{x}{)}^2}{S_{xx}}}$.
  • Reading the formula: $h_{ii}$ is smallest when $x_i=\bar{x}$ and grows as $x_i$ moves away from the center, and it depends only on the $x$‘s — the $y$-values never enter into it.
• Where that leverage formula actually comes from — every fitted value turns out to be a weighted sum of every observed response, and $h_{ii}$ is just one observation’s own weight in that sum.
  • Start from the fitted value written around the sample mean: ${\hat{y}}_i=\bar{y}+{\hat{\beta }}_1(x_i-\bar{x})$.
  • Write both pieces as sums over all observations $j$: $\bar{y}={\sum }_j\frac{1}{n}{y}_j$ and ${\hat{\beta }}_1=\frac{1}{S_{xx}}{\sum }_j(x_j-\bar{x})y_j$.
  • Substitute both in and combine into one sum: ${\hat{y}}_i={\sum }_j\left[\frac{1}{n}+\frac{(x_i-\bar{x})(x_j-\bar{x})}{S_{xx}}\right]y_j$.
  • Define that bracketed quantity as $\boxed{h_{ij}=\frac{1}{n}+\frac{(x_i-\bar{x})(x_j-\bar{x})}{S_{xx}}}$, so ${\hat{y}}_i={\sum }_j{h}_{ij}{y}_j$.
  • Setting $j=i$ gives the observation’s own weight in determining its own fitted value, $h_{ii}$ — which is why a point with large $h_{ii}$ can pull the fitted line toward itself.
• The leverages across all $n$ observations always add up to exactly $2$ in simple linear regression.
  • $\sum h_{ii}=\sum \frac{1}{n}+\frac{1}{S_{xx}}\sum (x_i-\bar{x}{)}^2=1+1=2$.
  • So the average leverage is $2/n$, and the $2$ corresponds to the two parameters being fit (intercept and slope).
• The high-leverage flag is a rule of thumb for deciding when a point’s leverage counts as “unusually large.”
  • Flag observation $i$ when its leverage exceeds twice the average: $\boxed{h_{ii}>2\times \frac{2}{n}=\frac{4}{n}}$.
  • This is a screening rule, not a theorem — a flagged, high-leverage point can still lie exactly on the fitted trend and cause no real problem.
• Raw residuals don’t all have the same variance, which is why they need to be standardized before comparing their sizes.
  • The exact variance of a residual is $\mathrm{Var}({\hat{e}}_i)={\sigma }^2(1-h_{ii})$.
  • As $h_{ii}\to 1$ (very high leverage), $\mathrm{Var}({\hat{e}}_i)\to 0$ — a high-leverage point’s residual is artificially squeezed toward zero, even if the true errors all have the same constant variance.
  • Dividing out that effect gives the standardized residual: $\boxed{r_i=\frac{{\hat{e}}_i}{S\sqrt{1-h_{ii}}}}$, where $S=\sqrt{RSS/(n-2)}$.
  • $r_i$ is the residual measured in units of its own estimated standard deviation, which makes residuals at different leverage points directly comparable; flag $|r_i|>2$.
• Three related but distinct ideas are easy to mix up: outlier, leverage, and influence.
  • A large residual means an unusual $y$-value, given its $x$.
  • High leverage means an unusual $x$-value, regardless of $y$.
  • An influential point is one whose removal would substantially change the fit.
  • A point can have high leverage and still sit right on the trend line (harmless); a point can have a large residual at an entirely ordinary $x$; a point with both traits at once is the one most capable of being seriously influential.
• Cook’s distance combines outlyingness and leverage into one number, specifically to catch the dangerous combination of the two.
  • Conceptually, it’s defined by leaving one observation out, refitting the whole model without it, and measuring how far all the fitted values moved as a result: $D_i=\frac{{\sum }_j({\hat{y}}_{j(i)}-{\hat{y}}_j{)}^2}{2S^2}$.
  • That’s expensive to compute directly, but it equals a simple shortcut: $\boxed{D_i=\frac{r_i^2}{2}\cdot \frac{h_{ii}}{1-h_{ii}}}$.
  • Reading the shortcut: $D_i$ gets large from a big standardized residual $r_i$, a big leverage $h_{ii}$, or both together.
  • Flag $D_i>\frac{4}{n-2}$.
  • Using Cook’s distance this way relies on the errors being normal, since that’s what makes the standardized residual inside it a meaningful quantity in the first place.

Practice Problems

• Problem 1 — leverage (Huber data, YBad, $n=6$). Predictor values $x=-4,-3,-2,-1,0,10$. Compute each $h_{ii}$ and flag any leverage points against the $4/n$ cutoff.

  • Solution.
    • $\bar{x}=0$, $S_{xx}=130$
    • hat values: $0.2897,0.2359,0.1974,0.1744,0.1667,0.9359$ (sum $=2.0$, confirming the sum-to-2 rule)
    • cutoff $4/6\approx 0.667$
    • only $x_6=10$ exceeds it ($h_{66}=0.9359$) — a clear high-leverage point, matching the slides

• Problem 2 — standardized residuals and Cook’s distance on the same data. Using the Huber YBad fit, compute the standardized residuals and Cook’s distances, and decide whether point 6 is a bad leverage point.

  • Solution.
    • standardized residuals: $1.597,0.308,-0.195,-1.129,-0.981,1.902$ — point 6’s $|r_6|=1.90$ is just under the $|2|$ flag on its own
    • Cook’s distances: $0.520,0.015,0.005,0.135,0.096,26.40$ — $D_6\approx 26.4$ is enormous against the cutoff $4/(n-2)=1$
    • teaching point: the standardized residual alone almost misses point 6, but Cook’s distance (which also weights in $h_{66}/(1-h_{66})$) flags it unmistakably as a bad leverage point

• Problem 3 — applying the cutoff on a bigger dataset (bonds data, $n=35$). The cutoff is $4/35\approx 0.11$. Cases 4, 5, 13, and 35 have leverage values above $0.11$. What does this tell you, and what would you check next?

  • Solution.
    • these four cases are flagged as leverage points purely from having unusual $x$ (coupon rate) values — three are the left-most points and one is the right-most point in the scatterplot
    • leverage alone doesn’t say they’re bad
    • next step: check their standardized residuals (or Cook’s distance) to see whether their $y$-values also break the pattern, before considering removing or refitting around them