Regression Inference and ANOVA

Concepts

• The fitted coefficients ${\hat{\beta }}_0,{\hat{\beta }}_1$ describe the sample at hand, but inference means making a statement about the true, unknown population relationship.
  • To do that, we need an estimate of the error variance, standard errors for the estimators, a reference distribution, and a way to measure how much variation the regression actually explains.
  • The goal of this whole note is to move fluently among coefficient inference, $R^2$, and ANOVA — they all come from the same handful of ingredients below, not three unrelated topics.
• The first ingredient is an estimate of the error variance ${\sigma }^2$, since every standard error below needs it and ${\sigma }^2$ itself is never known.
  • Recall $RSS=\sum {\hat{e}}_i^2$ is the sum of squared residuals.
  • Fitting two parameters (${\hat{\beta }}_0$ and ${\hat{\beta }}_1$) costs two degrees of freedom out of $n$, so the unbiased estimate divides by $n-2$ instead of $n$: $\boxed{S^2=MSE=\frac{RSS}{n-2}}$.
  • The residual standard error is just the square root: $\boxed{S=\sqrt{MSE}}$.
• Inference for the slope asks whether the true ${\beta }_1$ could plausibly equal some specific value.
  • Its standard error, from Regression Estimator Properties, is $SE({\hat{\beta }}_1)=\frac{S}{\sqrt{S_{xx}}}$.
  • To test $H_0:{\beta }_1={\beta }_{1,0}$ against $H_a:{\beta }_1\ne {\beta }_{1,0}$, the test statistic is $\boxed{t=\frac{{\hat{\beta }}_1-{\beta }_{1,0}}{S/\sqrt{S_{xx}}}}$, compared against a $t_{n-2}$ distribution.
  • The most common null is ${\beta }_1=0$ (is there any linear relationship at all?), but ${\beta }_1=1$ shows up too (e.g. testing whether two measurements are on a 1-to-1 scale) — same formula, just a different number subtracted in the numerator.
  • Reject $H_0$ for a large $|t|$, which is the same thing as a small two-sided $p$-value.
• A confidence interval for the slope is built from the same standard error.
  • The interval is ${\hat{\beta }}_1\pm t_{n-2}^{\ast }S/\sqrt{S_{xx}}$, where $t_{n-2}^{\ast }$ is the critical value for the chosen confidence level.
  • Test–CI connection: a two-sided level-$\alpha$ test rejects $H_0:{\beta }_1={\beta }_{1,0}$ exactly when ${\beta }_{1,0}$ falls outside this interval — so you can read off a test’s conclusion directly from the CI.
• Inference for the intercept works the same way, with its own standard error from the same earlier note: $SE({\hat{\beta }}_0)=S\sqrt{\frac{1}{n}+\frac{{\bar{x}}^2}{S_{xx}}}$.
  • The $t$-test and CI machinery is identical to the slope’s, just with this standard error substituted in.
  • Interpretation warning: the intercept is the expected response at $x=0$, which can be scientifically meaningless if $x=0$ sits far outside the range of data you actually observed.
• A confidence interval for the mean response at some new $x_0$ answers “where does the true regression line sit at $x_0$?”
  • The fitted mean response there is ${\hat{y}}_0={\hat{\beta }}_0+{\hat{\beta }}_1{x}_0$.
  • Its standard error is $S{E}_{mean}=S\sqrt{\frac{1}{n}+\frac{(x_0-\bar{x}{)}^2}{S_{xx}}}$, and the interval is ${\hat{y}}_0\pm t_{n-2}^{\ast }S{E}_{mean}$.
  • This interval is narrowest when $x_0=\bar{x}$, since $(x_0-\bar{x}{)}^2$ is smallest there — the fitted line is best pinned down near the center of the data, and gets less certain the further you move from it.
• A prediction interval for one new observation at $x_0$ answers a different question: “where will one actual new data point land?”
  • A single new response carries two separate sources of uncertainty: the uncertainty in where the mean line itself sits, and that one observation’s own random error around the line.
  • Its standard error adds an extra $1$ to account for that second source: $S{E}_{pred}=S\sqrt{1+\frac{1}{n}+\frac{(x_0-\bar{x}{)}^2}{S_{xx}}}$, giving the interval ${\hat{y}}_0\pm t_{n-2}^{\ast }S{E}_{pred}$.
  • Because of that extra $1$ under the root, a prediction interval is always wider than the confidence interval for the mean response at the same $x_0$.
• $R^2$ falls directly out of the sum-of-squares decomposition covered in the earlier proofs note.
  • Recall $SST=S{S}_{reg}+RSS$.
  • Dividing through by $SST$ gives $\boxed{R^2=\frac{S{S}_{reg}}{SST}=1-\frac{RSS}{SST}}$.
  • Read it as ”$R^2\times 100\%$ of the total sample variability in $Y$ is explained by the fitted regression on $x$.”
  • In simple linear regression with an intercept, $R^2$ is also exactly the squared correlation coefficient: $R^2=r_{xy}^2$.
  • What $R^2$ does not tell you: it doesn’t prove causation, and it doesn’t verify linearity, normality, independence, or constant variance — a high $R^2$ can still coexist with a badly misspecified model.
• The ANOVA table is just the same three sum-of-squares quantities organized into rows, with degrees of freedom and averages attached.
  • The regression row has SS $=S{S}_{reg}$, df $=1$ (one predictor), and MS $=S{S}_{reg}/1=S{S}_{reg}$.
  • The residual row has SS $=RSS$, df $=n-2$, and MS $=RSS/(n-2)=MSE$.
  • The total row has SS $=SST$, df $=n-1$.
  • The $F$ statistic compares the two mean squares: $\boxed{F=M{S}_{reg}/MSE}$.
• Reconstructing a partially blank ANOVA table is mostly a matter of knowing which quantities determine which others.
  • The degrees of freedom tell you $n$: total df is $n-1$, residual df is $n-2$.
  • $SST=S{S}_{reg}+RSS$ lets you fill in whichever SS is missing once the other two are known.
  • $M{S}_{reg}=S{S}_{reg}/1$ and $MSE=RSS/(n-2)$ convert sums of squares into mean squares.
  • $F=M{S}_{reg}/MSE$ connects the two mean squares.
  • If $R^2$ is given instead of an SS value, use $R^2=S{S}_{reg}/SST$ or $1-RSS/SST$ to back it out.
• In simple linear regression, the overall $F$-test and the slope’s two-sided $t$-test are secretly the same test — this is worth deriving once so it’s not just a memorized fact.
  • Since ${\hat{\beta }}_0=\bar{y}-{\hat{\beta }}_1\bar{x}$, the fitted deviation from the mean simplifies to ${\hat{y}}_i-\bar{y}={\hat{\beta }}_1(x_i-\bar{x})$.
  • Squaring and summing that gives $S{S}_{reg}=\sum ({\hat{y}}_i-\bar{y}{)}^2={\hat{\beta }}_1^2{S}_{xx}$.
  • Since the regression row has only $1$ degree of freedom, $M{S}_{reg}=S{S}_{reg}/1={\hat{\beta }}_1^2{S}_{xx}$.
  • Dividing by $MSE=S^2$: $F={\hat{\beta }}_1^2{S}_{xx}/S^2={\left({\hat{\beta }}_1/(S/\sqrt{S_{xx}})\right)}^2$.
  • The expression in parentheses is exactly the slope’s $t$-statistic under $H_0:{\beta }_1=0$, so $\boxed{F=t^2}$.
  • In other words: with only one predictor, “does the model explain anything” (the $F$-test) and “is the slope nonzero” (the $t$-test) are the same question asked two different ways.

Practice Problems

• Problem 1 (HW1 Q8 — airfares: Fare ~ Distance, $n=17$). Fitted line $\widehat{Fare}=48.9718+0.2197Distance$. Test $H_0:{\beta }_1=0$; give a 95% CI for ${\beta }_1$; report and interpret $R^2$ and the residual standard error; predict the fare for a 500-mile flight with a 95% prediction interval, checking first whether 500 lies in the observed range.

  • Solution.
    • $t_{obs}={\hat{\beta }}_1/SE=0.219687/0.004421\approx 49.69$ on $n-2=15$ df, $p\ll 0.05$ — reject $H_0$, strong evidence of a positive linear association
    • 95% CI for ${\beta }_1$: $0.219687\pm 2.131(0.004421)=(0.2103,0.2291)$, using $t_{0.025,15}^{\ast }=2.131$ — $0$ is far outside it, same conclusion as the test
    • $R^2=0.994$: distance explains about 99.4% of the variation in fare
    • residual standard error $=10.41$ on $15$ df: typical miss of about $10.41
    • observed distances run 90 to 1828 miles, so $500$ is inside the range (interpolation, reliable)
    • fitted value at $500$: $48.9718+0.2197(500)\approx 158.82$ — about $158.82
    • 95% prediction interval $\approx (135.79,181.85)$ — about $135.79 to $181.85

• Problem 2 (HW1 Q6 — mtcars: mpg ~ wt, $n=32$). Fitted line $\widehat{mpg}=37.285-5.344wt$. Interpret the slope and intercept; report and interpret $R^2$.

  • Solution.
    • slope $-5.344$: each additional 1000 lbs of weight is associated with about $5.34$ fewer mpg, on average
    • intercept $37.285$: predicted mpg at $wt=0$, not meaningful since no real car weighs nothing and all observed weights are well above zero
    • $R^2=0.753$: weight explains about 75.3% of the variation in mpg
    • residual standard error $\approx 3.05$ on $30$ df
    • $F=91.38$ on $1,30$ df, $p\approx 1.3\times 10^{-10}$

• Problem 3 (Case Study A — test2 ~ test1, $n=100$, a non-zero null). Test both $H_0:{\beta }_0=0$ (no bias) and $H_0:{\beta }_1=1$ (perfect agreement), and report $R^2$.

  • Solution.
    • intercept test (against $0$ by default): $t\approx 1.65$, $p\approx 0.103$ — no evidence of a constant offset
    • slope test against $1$ (not the default $0$, since the question is whether the relationship is proportional): $t_{obs}=({\hat{\beta }}_1-1)/SE({\hat{\beta }}_1)\approx -0.565$ on $98$ df, $p\approx 0.573$ — fail to reject, no evidence the tests drift apart
    • 95% CI for the slope $\approx (0.891,1.060)$, which contains $1$
    • $R^2\approx 0.843$: test1 explains about 84% of the variation in test2

• Problem 4 — reconstruct a full ANOVA table and compare the two kinds of interval (Production data: RunTime ~ RunSize, $n=20$). The SLR slides quote only the slope test (${\hat{\beta }}_1=0.25924$, $SE=0.03714$, $t=6.98$, $p<0.0001$) for this dataset. Fill in the full ANOVA table, and find both a 95% CI for the mean response and a 95% prediction interval at $RunSize=200$.

  • Solution.
    • $F=t^2=6.98^2\approx 48.72$ on $1,18$ df (matches $F=t^2$ above)
    • regression: SS $=12,868.4$, df $1$, MS $=12,868.4$
    • residual: SS $=4754.6$, df $18$, MS $=264.1$
    • total: SS $=17,623.0$, df $19$
    • $R^2=12,868.4/17,623.0\approx 0.730$
    • at $RunSize=200$: fitted value ${\hat{y}}_0\approx 201.60$
    • 95% CI for the mean response $\approx (193.96,209.23)$
    • 95% prediction interval for one new run $\approx (166.61,236.59)$ — visibly wider, exactly because it carries the extra $1$ under the square root for the new observation’s own error