Regression

linear regression
polynomial regression
logistic regression

Linear regression is a method in machine learning where we fit a straight line to a set of data points that appear to be concentrated around the line. This model helps to predict the outcome for future values.

Equation of line:

where is scaler used to transform the line.
and (coefficient of x) is slope of the line

Linear regression graph showing best fit line with actual data points, predicted values, slope (beta), and y-intercept

(Input)				.	.
(Outcome)				.	.

In the figure, n data points are plotted. We are trying to fit the line to this set of discrete data points.

Given Data:
- (Actual Data): The value of the output variable where , as provided by the data set.
Line Equation:
- (Predicted Data): The value of the outcome as predicted by the line , corresponding to .
Error Calculation:
- Error: The difference between (actual data point) and (predicted value from the line).
- Square of Errors:
- Sum of Square of Errors:

Where:

Goal

Now, we are trying to find out those value of and for which is minimum.

Partial Derivative with respect to :

using the power rule:

Differentiate with respect to
Apply the Chain Rule:
Differentiate the Inner Term:
Put the inner term:
Simplify:

Divide both sides by :

Equation^1:\bar{y}=\alpha+\beta{\bar{x}}

Partial Derivative with respect to :

Differentiate the Error Function with respect to
Apply the Chain Rule:
Differentiate the Inner Term:
Put the inner term:
Simplify:

Divide both sides by :

Equation^2:\overline{(x y)}=\alpha\overline{x}+\beta\overline{(x^2)}

Equation 1:

Equation 2:

Multiply Equation 1 by :

Subtracting from Equation 2:

Equation^3:\beta=\frac{\overline{(xy)} - \overline{x} . \overline{y}}{(\overline{(x^2)} - \overline{x}^2)}

Variance

Equation^4:Var(x)=\overline{(x^2)}-\bar{x}^2

Co-variance

cov(x,y)=\frac{1}{n}\sum_{i=1}^{n}\left(x_{i}-\overline{x})(y_{i}-\overline{y}\right)

cov(x,y)=\frac{1}{n}\sum_{i=1}^{n}(x_{i}y_{i}-x_{i}\overline{y}-\overline{x}y_{i}+\overline{x}.\overline{y})

cov(x,y)=\frac{1}{n}\left(\sum_{i=1}^{n}x_{i}y_{i}-\sum_{i=1}^{n}x_{i}\overline{y}-\sum_{i=1}^{n}\overline{x}y_{i}+\sum_{i=1}^{n}\overline{x}.\overline{y})\right)

cov(x,y)=\frac{1}{n}\left(\sum_{i=1}^{n}x_{i}y_{i}-\overline{y}\sum_{i=1}^{n}x_{i}-\overline{x}\sum_{i=1}^{n}y_{i}+\overline{x}.\overline{y}\sum_{i=1}^{n}.1\right)

cov(x,y)=\overline{(xy)}-\overline{y}.\overline{x}-\overline{x}.\overline{y}+\overline{x}.\overline{y}

cov(x,y)=\overline{(xy)}-\overline{x}.\overline{y}

Equation^5:cov(x,y)=\overline{(xy)}-\overline{x}.\overline{y}

Equation^1:\bar{y}=\alpha+\beta{\bar{x}}

Equation^2:\overline{(x y)}=\alpha\overline{x}+\beta\overline{(x^2)}

Equation^3:\beta=\frac{\overline{(xy)}-\overline{x}.\overline{y}}{\left(\overline{(x^2)}-\overline{x}^2\right)}

Equation^4:Var(x)=\overline{(x^2)}-\bar{x}^2

Equation^5:cov(x,y)=\overline{(xy)}-\overline{x}.\overline{y}

\beta=\frac{\overline{(xy)}-\overline{y}.\overline{y}}{\left(\overline{(x^2)}-\overline{x}^2\right)}

\beta=\frac{cov(x, y)}{var(x)}

Put the value of in :

\alpha=\overline{y}-\frac{cov(x, y)}{var(x)}\overline{x}

\alpha = \overline{y} - \frac{cov(x, y)}{\overline{(x^2)} - \overline{x}^2} \times \overline{x}

\alpha=\overline{y}-\frac{cov(x, y)}{\overline{(x^2)} - \overline{x}}

Final Value

\beta=\frac{cov(x,y)}{var\left(x\right)}

\alpha=\overline{y}-\beta\overline{x}