Linear Regression: Explained and Implemented from Scratch
๐ What is Linear Regression?
Linear Regression is a supervised learning algorithm used for predicting a continuous value based on one or more input features.
๐ง The Idea:
We fit a straight line (in 2D) or hyperplane (in higher dimensions) to the data that best describes the relationship between input X and output y.
Equation (for simple linear regression):
๐ฆ
=
๐ค
⋅
๐ฅ
+
๐
y=w⋅x+b
Where:
x is the input feature
y is the target
w is the weight (slope)
b is the bias (intercept)
๐ฏ Goal of Training
We want to find the best values of w and b that minimize the difference between predicted values and actual values.
This is done by minimizing the Mean Squared Error (MSE):
MSE
=
1
๐
∑
๐
=
1
๐
(
๐ฆ
๐
−
๐ฆ
^
๐
)
2
MSE=
n
1
i=1
∑
n
(y
i
−
y
^
i
)
2
Where:
๐ฆ
๐
y
i
is the actual value
๐ฆ
^
๐
y
^
i
is the predicted value
๐งฎ Gradient Descent (Optimization)
We use Gradient Descent to minimize the MSE by updating weights:
๐ค
=
๐ค
−
๐ผ
⋅
∂
MSE
∂
๐ค
w=w−ฮฑ⋅
∂w
∂MSE
๐
=
๐
−
๐ผ
⋅
∂
MSE
∂
๐
b=b−ฮฑ⋅
∂b
∂MSE
Where
๐ผ
ฮฑ is the learning rate.
๐ ️ Linear Regression from Scratch in Python
Here’s a full implementation:
✅ Step-by-step Python Code
import numpy as np
import matplotlib.pyplot as plt
# Generate sample data
np.random.seed(42)
X = 2 * np.random.rand(100, 1)
true_w = 3.5
true_b = 1.25
y = true_w * X + true_b + np.random.randn(100, 1) * 0.5 # Add noise
# Initialize parameters
w = 0.0
b = 0.0
learning_rate = 0.05
epochs = 100
n = X.shape[0]
# Training loop
for epoch in range(epochs):
y_pred = w * X + b
error = y_pred - y
# Gradients
dw = (2/n) * np.sum(error * X)
db = (2/n) * np.sum(error)
# Update parameters
w -= learning_rate * dw
b -= learning_rate * db
if epoch % 10 == 0:
loss = np.mean(error ** 2)
print(f"Epoch {epoch:03d} | Loss: {loss:.4f} | w: {w:.4f}, b: {b:.4f}")
# Final parameters
print(f"\nFinal model: y = {w:.2f}x + {b:.2f}")
# Plot the results
plt.scatter(X, y, color='blue', label='Data')
plt.plot(X, w * X + b, color='red', label='Fitted line')
plt.title("Linear Regression from Scratch")
plt.xlabel("X")
plt.ylabel("y")
plt.legend()
plt.show()
๐งพ Output (Example):
Epoch 000 | Loss: 15.0326 | w: 0.6813, b: 0.1707
Epoch 010 | Loss: 0.4849 | w: 3.1655, b: 1.1435
Epoch 020 | Loss: 0.2543 | w: 3.3891, b: 1.2187
...
Final model: y = 3.45x + 1.23
๐ง What You Just Did:
✅ Created synthetic data
✅ Implemented linear regression from scratch
✅ Used gradient descent to optimize weights
✅ Plotted the learned model
๐งช Next Steps
Extend to Multiple Linear Regression (more than 1 feature)
Add regularization (e.g., Ridge, Lasso)
Compare with sklearn.LinearRegression
Evaluate with metrics like R², MAE, RMSE
Learn Data Science Course in Hyderabad
Read More
Deep dive into specific algorithms with clear explanations and code.
Automating Your Data Pipeline with Python Scripts
Web Scraping with BeautifulSoup and Scrapy
Visit Our Quality Thought Training Institute in Hyderabad
Comments
Post a Comment