Formula and Identity Sheet

• A one-page reference for the notation, identities, and assumptions that connect every note. The derivations and worked examples live in the linked notes.

1. Core notation

   • Observed data: $(x_i,y_i)$ for $i=1,\dots ,n$
   • Sample means
     • $\bar{x}=\frac{1}{n}\sum _{i=1}^n{x}_i$
     • $\bar{y}=\frac{1}{n}\sum _{i=1}^n{y}_i$
   • Centered sums
     • $S_{xx}=\sum _{i=1}^n(x_i-\bar{x}{)}^2$
     • $S_{xy}=\sum _{i=1}^n(x_i-\bar{x})(y_i-\bar{y})$
   • Least-squares estimators
     • ${\hat{\beta }}_1=\frac{S_{xy}}{S_{xx}}$
     • ${\hat{\beta }}_0=\bar{y}-{\hat{\beta }}_1\bar{x}$
   • Fitted value and residual
     • ${\hat{y}}_i={\hat{\beta }}_0+{\hat{\beta }}_1{x}_i$
     • ${\hat{e}}_i=y_i-{\hat{y}}_i$

2. Core identities that connect the notes

   • Least squares / normal equations imply
     • $\sum {\hat{e}}_i=0$
     • $\sum x_i{\hat{e}}_i=0$
     • $\sum {\hat{y}}_i{\hat{e}}_i=0$
   • Those identities make the cross term vanish in
     • $SST=S{S}_{reg}+RSS$
   • The decomposition gives
     • $R^2=\frac{S{S}_{reg}}{SST}=1-\frac{RSS}{SST}$
   • Residual variation estimates ${\sigma }^2$
     • $S^2=MSE=\frac{RSS}{n-2}$
   • Slope uncertainty is
     • $\mathrm{SE}({\hat{\beta }}_1)=\frac{S}{\sqrt{S_{xx}}}$
   • Therefore the slope test statistic is
     • $t=\frac{{\hat{\beta }}_1-{\beta }_{1,0}}{S/\sqrt{S_{xx}}}$
   • In SLR, the overall regression $F$ test for $H_0:{\beta }_1=0$ satisfies
     • $F=t^2$
   • Leverage of observation $i$
     • $h_{ii}=\frac{1}{n}+\frac{(x_i-\bar{x}{)}^2}{S_{xx}},\sum _i{h}_{ii}=2$

3. Model assumptions to remember

   • The mean response is linear in $x$.
   • Errors have mean zero.
     • $E({\epsilon }_i)=0$
   • Errors have constant variance.
     • $\mathrm{Var}({\epsilon }_i)={\sigma }^2$
   • Errors are independent.
   • For the exact small-sample $t$ and $F$ inference, errors are normally distributed.
     • ${\epsilon }_i\sim N(0,{\sigma }^2)$

4. Notation warning on SSR / SSE

   • Textbooks and software disagree on the three-letter names.
   • These notes use
     • $S{S}_{reg}$ = explained / regression variation $=\sum ({\hat{y}}_i-\bar{y}{)}^2$
     • $RSS$ = residual / unexplained variation $=\sum {\hat{e}}_i^2$
   • Some sources write $SSR$ for the regression sum of squares and $SSE$ for the error sum of squares.
   • Rely on the actual formula, not the abbreviation.

5. Multiple linear regression (MLR) — notation for $k$ predictors

   • Matrix form: $Y=X\beta +ϵ$, with $X$ an $n\times (k+1)$ design matrix (intercept column plus $k$ predictor columns)
   • Least-squares estimator
     • $\hat{\beta }=(X^{\prime }X{)}^{-1}{X}^{\prime }Y$
   • Hat matrix and its two defining properties
     • $H=X(X^{\prime }X{)}^{-1}{X}^{\prime },\hat{Y}=HY,\hat{e}=(I-H)Y$
     • $H^{\prime }=H$ (symmetric), $H^2=H$ (idempotent), $HX=X$, $\mathrm{tr}(H)=p$ where $p=k+1$
   • Leverage of observation $i$ is $h_{ii}$, the $i$-th diagonal entry of $H$
     • $\sum _i{h}_{ii}=p,$ flag $h_{ii}>\frac{2p}{n}$ (SLR’s $4/n$ is this with $p=2$)
   • ANOVA degrees of freedom: regression $k$, residual $n-k-1$, total $n-1$ ($k=1$ recovers every SLR formula above)

6. MLR global $F$-test and $R_{adj}^2$

   • Global $F$-test, $H_0:{\beta }_1=\cdots ={\beta }_k=0$
     • $F=\frac{S{S}_{reg}/k}{RSS/(n-k-1)}=\frac{n-k-1}{k}\cdot \frac{R^2}{1-R^2}\sim F_{k,n-k-1}$
   • Adjusted $R^2$
     • $R_{adj}^2=1-\frac{RSS/(n-k-1)}{SST/(n-1)}=1-\frac{n-1}{n-k-1}(1-R^2)$
     • Always $R_{adj}^2\le R^2$, with equality only at $k=0$ or $R^2=1$
   • Partial $F$-test for a subset of $q$ predictors, reduced model $R$ vs. full model $F$
     • $F=\frac{(RS{S}_R-RS{S}_F)/q}{RS{S}_F/(n-p-q-1)}\sim F_{q,n-p-q-1}$
     • For $q=1$: $F=t^2$, same test as the coefficient’s own $t$-test
   • Interpreting any ${\hat{\beta }}_j$ with $k>1$ predictors requires “holding the other predictors constant” explicitly