Regression Estimator Properties

• Mean, variance, and covariance of the least-squares estimators, derived via covariance bilinearity.

Concepts

• Once we fit a line, ${\hat{\beta }}_0$ and ${\hat{\beta }}_1$ are just numbers for the sample we happened to observe.
  • But before the sample is observed, the responses $Y_i$ are random — a different sample would give different numbers.
  • That makes the estimators ${\hat{\beta }}_0,{\hat{\beta }}_1$ random variables too, with their own mean, variance, and distribution.
  • This is exactly where the standard errors used later in inference come from.
• The model assumptions that everything below relies on:
  • The predictor values $x_i$ are treated as fixed, known constants — not random.
  • $Y_i={\beta }_0+{\beta }_1{x}_i+{\epsilon }_i$, with $E({\epsilon }_i)=0$ and $\mathrm{Var}({\epsilon }_i)={\sigma }^2$, and the errors independent of each other.
  • These assumptions carry over directly to $Y_i$: $E(Y_i)={\beta }_0+{\beta }_1{x}_i$, $\mathrm{Var}(Y_i)={\sigma }^2$, and $Y_i,Y_j$ independent for $i\ne j$.
• The key move: rewrite the slope as a weighted sum of the responses, since that makes every property below easy to derive.
  • Start from the familiar formula ${\hat{\beta }}_1=S_{xy}/S_{xx}$.
  • Expand the numerator: $S_{xy}=\sum (x_i-\bar{x})(Y_i-\bar{Y})$.
  • The $\bar{Y}$ part of that expansion drops out, because $\sum (x_i-\bar{x})=0$ — leaving $S_{xy}=\sum (x_i-\bar{x})Y_i$.
  • Define a fixed weight for each observation: $c_i=\frac{x_i-\bar{x}}{S_{xx}}$ — fixed because it only depends on the (non-random) $x_i$‘s.
  • That gives $\boxed{{\hat{\beta }}_1=\sum _i{c}_i{Y}_i}$ — the slope written as a weighted sum of the random $Y_i$‘s, with fixed weights.
  • This is the whole reason the $c_i$ trick exists: once ${\hat{\beta }}_1$ is a weighted sum of the $Y_i$‘s, every property below follows from ordinary rules for linear combinations of random variables, instead of fighting with $S_{xy}/S_{xx}$ directly.
• Three facts about the $c_i$ weights get used repeatedly below — each is just algebra on the definition of $c_i$.
  • $\sum c_i=0$, since $\sum (x_i-\bar{x})=0$.
  • $\sum c_i{x}_i=1$ — write $x_i=(x_i-\bar{x})+\bar{x}$, expand, and the $\bar{x}\sum (x_i-\bar{x})$ term vanishes, leaving $S_{xx}/S_{xx}=1$.
  • $\sum c_i^2=1/S_{xx}$ — direct from squaring $c_i=(x_i-\bar{x})/S_{xx}$ and summing.
• The slope estimator is unbiased: on average, across repeated samples, ${\hat{\beta }}_1$ centers on the true ${\beta }_1$.
  • Take the expectation of the weighted-sum form: $E({\hat{\beta }}_1)=E(\sum c_i{Y}_i)=\sum c_{i}E(Y_i)$.
  • Substitute $E(Y_i)={\beta }_0+{\beta }_1{x}_i$: $=\sum c_i({\beta }_0+{\beta }_1{x}_i)={\beta }_0\sum c_i+{\beta }_1\sum c_i{x}_i$.
  • Using the two $c_i$ facts above, $\sum c_i=0$ and $\sum c_i{x}_i=1$: $={\beta }_0(0)+{\beta }_1(1)={\beta }_1$.
  • So $\boxed{E({\hat{\beta }}_1)={\beta }_1}$.
• The variance of the slope measures how much ${\hat{\beta }}_1$ would bounce around from one sample to the next.
  • Since the $Y_i$ are independent, variance distributes over the sum without any cross terms: $\mathrm{Var}({\hat{\beta }}_1)=\sum c_i^2\mathrm{Var}(Y_i)$.
  • Every $Y_i$ has the same variance ${\sigma }^2$, so this is ${\sigma }^2\sum c_i^2$.
  • Using the third $c_i$ fact, $\sum c_i^2=1/S_{xx}$: $\boxed{\mathrm{Var}({\hat{\beta }}_1)=\frac{{\sigma }^2}{S_{xx}}}$.
  • Click point: a bigger spread in the $x$-values means a larger $S_{xx}$, which means a smaller variance — spreading your $x$‘s out gives the slope estimate more leverage to be pinned down precisely.
• The intercept can be rewritten as a weighted sum too, the same way the slope was.
  • Start from ${\hat{\beta }}_0=\bar{Y}-{\hat{\beta }}_1\bar{x}$.
  • Substitute $\bar{Y}=\sum \frac{1}{n}{Y}_i$ and ${\hat{\beta }}_1=\sum c_i{Y}_i$ and collect terms: ${\hat{\beta }}_0=\sum (\frac{1}{n}-\bar{x}{c}_i)Y_i$.
  • Define $d_i=\frac{1}{n}-\bar{x}{c}_i$, so ${\hat{\beta }}_0=\sum d_i{Y}_i$ — same trick, new weights.
• The intercept estimator is unbiased too, by the same kind of argument.
  • Using the simpler form ${\hat{\beta }}_0=\bar{Y}-\bar{x}{\hat{\beta }}_1$: $E({\hat{\beta }}_0)=E(\bar{Y})-\bar{x}E({\hat{\beta }}_1)$.
  • $E(\bar{Y})={\beta }_0+{\beta }_1\bar{x}$ (the average of the model’s mean response), and $E({\hat{\beta }}_1)={\beta }_1$ from above.
  • So $E({\hat{\beta }}_0)=({\beta }_0+{\beta }_1\bar{x})-\bar{x}{\beta }_1={\beta }_0$.
• Covariance between two weighted sums follows four bilinearity rules, which every proof from here on leans on.
  • Constants pull outside: $\mathrm{Cov}(aX,bY)=ab\mathrm{Cov}(X,Y)$.
  • Covariance distributes over a sum, the same way multiplication distributes over addition.
  • Covariance of something with itself is just its variance: $\mathrm{Cov}(X,X)=\mathrm{Var}(X)$.
  • Independent random variables have covariance $0$ — so for independent $Y_i$, $\mathrm{Cov}(\sum a_i{Y}_i,\sum b_i{Y}_i)=\sum a_i{b}_i\mathrm{Var}(Y_i)$, since every cross term with $i\ne j$ vanishes.
• An intermediate result, $\mathrm{Cov}(\bar{Y},{\hat{\beta }}_1)=0$, makes the next two proofs simple.
  • Write both as weighted sums: $\bar{Y}=\sum \frac{1}{n}{Y}_i$ and ${\hat{\beta }}_1=\sum c_i{Y}_i$.
  • By independence, $\mathrm{Cov}(\bar{Y},{\hat{\beta }}_1)=\sum \frac{1}{n}{c}_i\mathrm{Var}(Y_i)=\frac{{\sigma }^2}{n}\sum c_i$.
  • Using $\sum c_i=0$ from above, this is $0$.
• The variance of the intercept follows from the usual variance-of-a-difference formula, now that the covariance piece is known.
  • $\mathrm{Var}({\hat{\beta }}_0)=\mathrm{Var}(\bar{Y})+{\bar{x}}^2\mathrm{Var}({\hat{\beta }}_1)-2\bar{x}\mathrm{Cov}(\bar{Y},{\hat{\beta }}_1)$.
  • The covariance term is $0$ (just shown), and $\mathrm{Var}(\bar{Y})={\sigma }^2/n$ since the $Y_i$ are independent with common variance.
  • So $\boxed{\mathrm{Var}({\hat{\beta }}_0)={\sigma }^2\left(\frac{1}{n}+\frac{{\bar{x}}^2}{S_{xx}}\right)}$.
• The covariance between the intercept and slope — the bilinearity proof itself.
  • Start from ${\hat{\beta }}_0=\bar{Y}-\bar{x}{\hat{\beta }}_1$, and take the covariance of both sides with ${\hat{\beta }}_1$: $\mathrm{Cov}({\hat{\beta }}_0,{\hat{\beta }}_1)=\mathrm{Cov}(\bar{Y},{\hat{\beta }}_1)-\bar{x}\mathrm{Cov}({\hat{\beta }}_1,{\hat{\beta }}_1)$.
  • The first term is $0$ (shown above), and the second term is just $\mathrm{Var}({\hat{\beta }}_1)={\sigma }^2/S_{xx}$.
  • So $\boxed{\mathrm{Cov}({\hat{\beta }}_0,{\hat{\beta }}_1)=-\frac{\bar{x}{\sigma }^2}{S_{xx}}}$.
  • Sanity check: if $\bar{x}>0$, the covariance is negative — tilting the slope up forces the intercept down so the line can still pass through $(\bar{x},\bar{y})$.
• Once errors are assumed normal, the estimators have a known sampling distribution, which is what makes $t$-based inference exact in small samples.
  • Linear combinations of jointly normal variables are themselves normal, and ${\hat{\beta }}_1,{\hat{\beta }}_0$ are both linear combinations of the $Y_i$‘s.
  • So ${\hat{\beta }}_1\sim N({\beta }_1,{\sigma }^2/S_{xx})$ and ${\hat{\beta }}_0\sim N({\beta }_0,{\sigma }^2(\frac{1}{n}+\frac{{\bar{x}}^2}{S_{xx}}))$.
• In practice ${\sigma }^2$ is never actually known, so one more substitution is needed before any of this is usable.
  • Estimate it with $S^2=RSS/(n-2)$.
  • Replace $\sigma$ by $S$ inside each standard deviation above: $SE({\hat{\beta }}_1)=S/\sqrt{S_{xx}}$ and $SE({\hat{\beta }}_0)=S\sqrt{\frac{1}{n}+\frac{{\bar{x}}^2}{S_{xx}}}$.
  • That substitution is also what changes the reference distribution from normal to $t_{n-2}$ in the inference formulas later.

Practice Problems

• Problem 1 (HW1 Q2). Derive $\mathrm{Cov}({\hat{\beta }}_0,{\hat{\beta }}_1)$ using bilinearity rather than memorizing it. Write it the way you would on the exam — the Concepts section above has the fully justified version; this is the fast, exam-pace version of the same derivation.

  • Solution.
    • ${\hat{\beta }}_0=\bar{Y}-\bar{x}{\hat{\beta }}_1$
    • $\mathrm{Cov}({\hat{\beta }}_0,{\hat{\beta }}_1)=\mathrm{Cov}(\bar{Y}-\bar{x}{\hat{\beta }}_1,{\hat{\beta }}_1)$
    • $=\mathrm{Cov}(\bar{Y},{\hat{\beta }}_1)-\bar{x}\mathrm{Cov}({\hat{\beta }}_1,{\hat{\beta }}_1)$
    • $\mathrm{Cov}(\bar{Y},{\hat{\beta }}_1)=0$ (shown earlier via $\sum c_i=0$)
    • $\mathrm{Cov}({\hat{\beta }}_1,{\hat{\beta }}_1)=\mathrm{Var}({\hat{\beta }}_1)={\sigma }^2/S_{xx}$
    • $\boxed{\mathrm{Cov}({\hat{\beta }}_0,{\hat{\beta }}_1)=-\frac{\bar{x}{\sigma }^2}{S_{xx}}}$ $■$

• Problem 2 — verify a real printed standard error (Production data, $n=20$). The production data (RunTime ~ RunSize) has $S_{xx}=191,473.8$ and R reports $S^2=MSE=264.1$ and a slope standard error of $0.03714$. Using $\mathrm{Var}({\hat{\beta }}_1)={\sigma }^2/S_{xx}$ with $S^2$ in place of the unknown ${\sigma }^2$, verify this printed value by hand.

  • Solution.
    • $SE({\hat{\beta }}_1)=\sqrt{S^2/S_{xx}}$
    • $=\sqrt{264.1/191,473.8}$
    • $=\sqrt{0.0013799}$
    • $\approx 0.03714$ — matches R’s printed Std. Error exactly