STAT 211 Topic 10

« previous | Thursday, April 14, 2011 | next »

Simple Linear Regression

Predicting linear relationships between two variables $X$ and $Y$: $Y=A+BX$, where rvs $A,B\sim \mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l}$.

Goal: Find the line such that the squared distance (so distance values are positive) between the predicted and observed values is minimized.

What the line tells us:

• Estimates/Predicts values (interpolation within range; extrapolation outside of range)
• Describe ratio between two variables (slope)

$y={\beta }_0+{\beta }_{1}x+ϵ$
Regression Model Equation

• β₀, β₁, and x are assumed to be unknown fixed parameters.
• ε Error variable: $ϵ\sim \mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l}(0,{\sigma }^2)$ (σ² is unknown)
• Therefore $y\sim \mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l}({\beta }_0+{\beta }_{1}x,{\sigma }^2)$

We have estimates for $x$ (sample data), but we need estimates for ${\beta }_0$ and ${\beta }_1$.

Least Squares

Given points $(x_1,y_1),(x_2,y_2),\dots ,(x_n,y_n)$, our regression model is:

$y_i={\beta }_0+{\beta }_1{x}_i+{ϵ}_i$

Squared distance between points is given by $(y_i-({\beta }_0+{\beta }_1{x}_i){)}^2$. Therefore the sum of squared differences is:

$S({\beta }_0,{\beta }_1)=\sum _{i=1}^n(y_i-{\beta }_0-{\beta }_1{x}_i{)}^2$

Find ${\beta }_0$ and ${\beta }_1$ such that $S({\beta }_0,{\beta }_1)$ is minimized. (Take derivative, equate to zero, and solve)

Estimated Solutions: (both have normal distribution regardless of sample size)

$\begin{array}{rlr}{\hat{\beta }}_0& =\frac{\sum (x_i-\overline{x})(y_i-\overline{y})}{\sum (x_i-\overline{x}{)}^2}& \sim \mathrm{N}({\beta }_0,)\\ {\hat{\beta }}_1& =\overline{y}-{\hat{\beta }}_1\overline{x}& \sim \mathrm{N}({\beta }_1,\frac{{\sigma }^2}{\sum (x_i-\overline{x}{)}^2})\end{array}$

Therefore, the final equation becomes

${\hat{y}}_i={\hat{\beta }}_0+{\hat{\beta }}_1{x}_i$

The fitted (predicted) value of $y_i$ is ${\hat{y}}_i={\hat{\beta }}_0+{\hat{\beta }}_1{x}_i$

Residual (observed − fitted; $y_i-{\hat{y}}_i$) can be plotted to check how well the line fits.

All computer stat systems will return a coefficient of determination $R^2$ value. This value is always between 0 and 1, and $R^2$ close to 1 implies a better fitting line.

Tests for β₁

Since we don't know σ, we can substitute an estimate $s_{ϵ}$ (see below), then the standard deviation of β₁ is a T-distribution with df $n-2$.

Therefore, we can do confidence intervals $({\hat{\beta }}_1\pm t_{\alpha /2}\cdot s_{{\hat{\beta }}_1})$ and t-test for H₀: β₁ = β₁₀ $(t=\frac{{\hat{\beta }}_1-{\beta }_{10}}{s_{{\hat{\beta }}_1}})$

Estimating σ²

Minimum value for $D$ is the Error Sum of Squares:

$D=\sum _{i=0}^n(y_i-({\hat{\beta }}_0+{\hat{\beta }}_1{x}_i){)}^2=SSE$

To obtain estimate for σ², we divide SSE by the (remaining) degrees of freedom (after estimating β₁ and β₂) $n-2$:

$s_{ϵ}=\hat{\sigma }=\sqrt{\frac{SSE}{n-2}}$

ANOVA for Regression

$\begin{array}{rlr}SST=SSR+SSE& =\sum _{i=1}^n(y_i-\overline{y}{)}^2& \text{Total Sum of Squares}\\ SSE& =\sum _{i=1}^n(y_i-({\hat{\beta }}_0+{\hat{\beta }}_1{x}_i){)}^2& \text{Error Sum of Squares}\\ SSR=SST-SSE& =\sum _{i=1}^n({\hat{y}}_i-\overline{y}{)}^2& \text{Regression Sum of Squares}\\ R^2& =\frac{SSR}{SST}& \text{Coefficient of Determination}\end{array}$

Source	Sum of Squares	Degrees of Freedom	Mean Square	f statistic
Regression Error	SSR	1	MSR	MSR / MSE
Error	SSE	$n-2$	MSE
Total	SST	$n-1$

Predicting with Regression

Suppose we have a new value that we want to estimate: x*

2 possible ways to calculate this:

1. mean response (only think of value of regression model line): $E(y|x=x^{*})={\hat{\beta }}_0+{\hat{\beta }}_1{x}^{*}$
2. prediction (include error variable ε): $\hat{y}={\hat{\beta }}_0+{\hat{\beta }}_1{x}^{*}$

Basically plugging in a new $x$ value into the model equation to obtain a $\hat{y}$ estimate.

Mean Response

We want to narrow the range and reduce variance by as much as possible, so we disregard the variance of ε.

Calculate 100(1-α)% confidence interval for response with:

${\hat{\beta }}_0+{\hat{\beta }}_1{x}^{*}\pm t_{\alpha /2,n-2}\cdot s_{ϵ}\sqrt{\frac{1}{n}+\frac{(x^{*}-\overline{x}{)}^2}{\sum (x_i-\overline{x}{)}^2}}$

Prediction

Because the regression model has the ε term, the prediction is actually within a range.

${\hat{\beta }}_0+{\hat{\beta }}_1{x}^{*}\pm t_{\alpha /2,n-2}\cdot s_{ϵ}\sqrt{1+\frac{1}{n}+\frac{(x^{*}-\overline{x}{)}^2}{\sum (x_i-\overline{x}{)}^2}}$

Correlation Coefficient

Show how strongly related two random variables are.

Note: If X and Y are independent, then correlation is 0

$\mathrm{C}\mathrm{o}\mathrm{r}\mathrm{r}(X,Y)={\rho }_{X,Y}=\frac{\mathrm{C}\mathrm{o}\mathrm{v}(X,Y)}{\sqrt{V(X)V(Y)}}$

where Covariance is

$\mathrm{C}\mathrm{o}\mathrm{v}(X,Y)=E(XY)-E(X)E(Y)$

For a population where we do not know the pdf (ƒ(x, y)), we can estimate the sample correlation coefficient using

$r=\frac{\sum _{i=1}^n(x_i-\overline{x})(y_i-\overline{y})}{\sqrt{\sum _{i=1}^n(x_i-\overline{x}{)}^2\sum _{i=1}^n(y_i-\overline{y}{)}^2}}$