PURPOSE: to show how the estimator formulas for the slope and intercept of an ordinary least squares regression are derived. You will not be required to reproduce these derivations; they are provided so that you can appreciate the intuition of the process that leads to the eventual formulas that are used, in the background, by SHAZAM and other regression software packages when you ask the program to find the "best-fitting linear relationship" between Y and X. Except for one very simple homework exercise, you will never again be using these formulas explicitly. Let the software do it for you.

CAUTION: newly transformed to HTML; please report any remaining errors.

 
Sample Regression Function:    Y_i = b₁ + b₂ X_i + e_i      implies e_i = Y_i - b₁ - b₂ X_i

All sums (S) below are from i=1 to i=n (i.e. over the entire sample).

OLS:    min (over   b₁, b₂ )  S e_i²  = min S (Y_i - b₁ - b₂ X_i)²  =   min [ * ]

Recipe: take the first partial derivatives (slopes) of the minimand function [*] and set them equal to zero; then solve for the values of b₁ and b₂  that make these equations true.

First, with respect to b₁:    ¶ */¶ b₁:       S 2 (Y_i - b₁ - b₂ X_i) (-1) = 0
 
                                                             same as:         S Y_i - n b₁ - b₂ S X_i = 0

Then, with respect to  b₂:        ¶ */¶ b₂:       S 2 (Y_i - b₁ - b₂ X_i) (-X_i) = 0

                                                              same as:       S X_i Y_i - b₁ S X_i - b₂ S X_i²  = 0

These derivatives, set equal to zero, give us the so-called "normal equations" for ordinary least squares:

(1)      S Y_i = n b₁ + b₂ S X_i

(2)       S X_i Y_i = b₁ S X_i + b₂ S X_i²

As soon as you have a sample of data on X and Y, you will know the X_i, the Y_i, and n.

The "normal equations" are two equations in two unknowns: b₁ and b₂

From (1), b₁   =   ( S Y_i - b₂ S X_i)/n   =   S Y_i/n - b₂ (S X_i/n)   =   - b₂ 

From (2), b₂   =   (S X_i Y_i - b₁ S X_i)/(S X_i² )   =   (S X_i Y_i)/(S X_i² ) - b₁ (S X_i)/(S X_i² )

With these rearrangements, we recognize a system of two linear equations in b₁ and b₂. Each unknown parameter is expressed as a linear function of the other, with the "coefficients" in this case being functions of the data on X_i and Y_i.

Strategy is to substitute the expression derived for b₁ (in terms of b₂) into the expression for b₂, leaving b₂ the only unknown in (2) after the substitution is made. This is just a messy version of high school algebra. The final result for b₂ can then be expressed in several different ways:

b₂   =   [  n S X_i Y_i - S X_i S Y_i ] / [  n S X_i²  - (S X_i)²  ];     or, dividing all terms by n  

b₂   =  [  S X_i Y_i - n    ]  /  [  S X_i²  - n X² ]     or, rearranging

b₂   =  [  S (X_i - )(Y_i - ) ]  /  [  S (X_i - X)² ] ,    

or,  if we define x_i = X_i -  and y_i = Y_i -  ,

b₂   =   (S x_iy_i)/(S x_i² )               ... easiest version of slope estimator (MEMORIZE)

Then the intercept estimate,  b₁ =  - b₂ ,  is then trivial to calculate.      (MEMORIZE)