← All posts

Rebuilding OLS from the objective up

Understanding Linear Regression from scratch, deriving the error function, and finding the optimal slope and intercept using Ordinary Least Squares (OLS) and calculus.

Updated 25 Oct 2026 linear algebramachine learningfrom scratch

Derivation

There are two common ways to find the values of (m) and (b): Ordinary Least Squares (OLS) and Gradient Descent. Scikit-learn uses the least squares method in its “LinearRegression” class to find the best values of (m) and (b), while “SGDRegressor” uses Stochastic Gradient Descent. Let’s build the mathematical intuition behind Linear Regression.

Suppose we have the following data about the hours of study by students and the marks they have obtained:

Hours Studied (x)12345...
Marks (y)23564...

Each row in this table is one observation, and we have a total of (n) observations. We will plot these observations on a graph with (x) on the horizontal axis and (y) on the vertical axis. Since the data follows a variable pattern, it is likely that these points will not lie exactly on a straight line. And that is exactly what we want to do in Linear Regression: find the best possible line that comes as close as possible to all the points.

We all know that the equation of a straight line is:

y^=mx+b \boxed{\hat{y} = mx + b}

Here, (x) is the input or independent variable, (y) is the actual observed output, (\hat{y}) (read as y-hat) is the output predicted by our line, (m) is the slope of the line, and (b) is the y-intercept. In machine learning terminology, finding the values of (m) and (b) is called training the model.

Residual

Let’s assume that we have drawn a straight line through the data points in the graph, and for every data point, the line predicts a value (\hat{y}_i). Therefore, for the (i)-th observation,

y^i=mxi+b \boxed{\hat{y}_i = mx_i + b}

Here, (\hat{y}_i) is the predicted value and (y_i) is the actual observed value. The difference between the actual and predicted values is called the residual.

di=yi−y^i \boxed{d_i = y_i - \hat{y}_i}

Substituting the equation of the line gives us:

di=yi−mxi−b \boxed{d_i = y_i - mx_i - b}

Let’s take an example. Suppose an observation has (x = 2) and its actual (y = 3), and our current line is:

y^=1.5x+0.5 \hat{y} = 1.5x + 0.5

Therefore, the predicted value for this observation will be:

y^=1.5(2)+0.5=3.5 \hat{y} = 1.5(2) + 0.5 = 3.5

But the actual value is (3). So the residual becomes:

d=3−3.5=−0.5 d = 3 - 3.5 = -0.5

The negative sign suggests that our line has predicted a value greater than the actual value. Now, this was just one observation. But what if we have thousands of observations? How do we measure how good our line is across all these points? We need an error function for this.

Error Function

One way to calculate the total error is to add the absolute values of all the residuals.

E=∣d1∣+∣d2∣+∣d3∣+⋯+∣dn∣ E = |d_1| + |d_2| + |d_3| + \cdots + |d_n|

But there is a problem with using absolute values. The absolute value function is not differentiable at zero, which makes it less convenient to minimize directly using standard derivative-based methods.

So, instead of taking absolute values, we square each residual and then add them together. The squared-error function is differentiable, which makes it easier to find the minimum error using calculus. Also, squaring the residuals penalizes larger errors more heavily, which means that outliers can have a much greater impact on the total error.

Therefore, our error function becomes:

E=d12+d22+d32+⋯+dn2 E = d_1^2 + d_2^2 + d_3^2 + \cdots + d_n^2

Using summation notation, we can write it as:

E=∑i=1ndi2 \boxed{E = \sum_{i=1}^{n} d_i^2}

Now, substituting the residual equation into this expression gives us:

E(m,b)=∑i=1n(yi−mxi−b)2 \boxed{E(m,b) = \sum_{i=1}^{n}(y_i - mx_i - b)^2}

This is the Sum of Squared Errors (SSE). Notice that our error function depends on (m) and (b). Different values of (m) and (b) will give us different lines, each with a different total error. Our goal is to find the values of (m) and (b) that produce the minimum error.

Finding (m) and (b)

Think of the error function as a surface where the two axes represent (m) and (b), and the height represents the error (E). Every point on this surface represents some combination of (m) and (b), and we want to find the lowest point on this surface.

For a differentiable function, the first derivatives must be zero at an interior minimum. Since our error function depends on two variables, we take the partial derivatives with respect to both (m) and (b) and set them equal to zero:

∂E∂m=0and∂E∂b=0 \boxed{ \frac{\partial E}{\partial m}=0 \qquad \text{and} \qquad \frac{\partial E}{\partial b}=0 }

Finding (b)

We have:

E(m,b)=∑i=1n(yi−mxi−b)2 E(m,b) = \sum_{i=1}^{n}(y_i - mx_i - b)^2

Taking the partial derivative with respect to (b) gives us:

∂E∂b=−2∑i=1n(yi−mxi−b)=0 \frac{\partial E}{\partial b} = -2\sum_{i=1}^{n}(y_i - mx_i - b) = 0

Dividing both sides by (-2), we get:

∑i=1n(yi−mxi−b)=0 \sum_{i=1}^{n}(y_i - mx_i - b) = 0

Expanding the summation gives us:

∑i=1nyi−m∑i=1nxi−nb=0 \sum_{i=1}^{n}y_i - m\sum_{i=1}^{n}x_i - nb = 0

Rearranging the terms:

b=∑i=1nyin−m∑i=1nxin b = \frac{\sum_{i=1}^{n}y_i}{n} - m\frac{\sum_{i=1}^{n}x_i}{n}

We know that the mean of (x) is represented by (\bar{x}) and the mean of (y) is represented by (\bar{y}). Therefore, we can write the above equation as:

b=yˉ−mxˉ \boxed{b = \bar{y} - m\bar{x}}

And there we have our first result. We can find the intercept if we know the slope and the means of (x) and (y).

Finding (m)

We already know that:

b=yˉ−mxˉ b = \bar{y} - m\bar{x}

Substituting this value of (b) into our error function gives us:

E(m)=∑i=1n[(yi−yˉ)−m(xi−xˉ)]2 E(m) = \sum_{i=1}^{n} \left[(y_i-\bar{y})-m(x_i-\bar{x})\right]^2

Now, we take the derivative with respect to (m) and set it equal to zero. Applying the chain rule gives us:

dEdm−2∑i=1n(xi−xˉ)[(yi−yˉ)−m(xi−xˉ)]=0 \frac{dE}{dm} -2\sum_{i=1}^{n} (x_i-\bar{x}) \left[(y_i-\bar{y})-m(x_i-\bar{x})\right] = 0

Dividing both sides by (-2) and expanding the expression gives us:

∑i=1n(xi−xˉ)(yi−yˉ)m∑i=1n(xi−xˉ)2=0 \sum_{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y}) m\sum_{i=1}^{n}(x_i-\bar{x})^2 = 0

Rearranging the terms, we get:

∑i=1n(xi−xˉ)(yi−yˉ)m∑i=1n(xi−xˉ)2 \sum_{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y}) m\sum_{i=1}^{n}(x_i-\bar{x})^2

Finally, dividing both sides by the sum of squared deviations of (x) gives us:

m=∑i=1n(xi−xˉ)(yi−yˉ)∑i=1n(xi−xˉ)2 \boxed{ m = \frac{ \sum_{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y}) }{ \sum_{i=1}^{n}(x_i-\bar{x})^2 } }

And there we have it. We have derived the formula for the slope using OLS. Once we have both (m) and (b), we can plug them back into our original equation:

y^=mx+b \boxed{\hat{y} = mx + b}

And this is how we find the best-fitting straight line using Ordinary Least Squares.