Feature Selection Techniques: Filter, Wrapper, and Embedded Methods

 🎯 What is Feature Selection?

Feature selection is the process of choosing the most relevant and important features (variables) from your dataset. It helps to:

Improve model performance

Reduce overfitting

Decrease training time

Improve model interpretability

🔍 1. Filter Methods

These select features based on statistical measures, without involving any machine learning model.

🧪 How it works:

Evaluate each feature independently from the model.

Use statistical tests (like correlation, Chi-square, ANOVA) to rank features.

📊 Common Techniques:

Correlation coefficient (e.g., Pearson)

Chi-square test

ANOVA F-test

Mutual information

✅ Pros:

Fast and simple

Doesn’t depend on model choice

❌ Cons:

Ignores feature interactions

May select irrelevant features for your specific model

🧰 2. Wrapper Methods

These use a machine learning model to evaluate feature subsets by training and testing the model on different combinations.

🔁 How it works:

Try different feature combinations

Select the set that gives the best model performance (accuracy, F1, etc.)

📊 Common Techniques:

Forward selection: Start with none, add one at a time

Backward elimination: Start with all, remove one at a time

Recursive Feature Elimination (RFE)

python

Copy

Edit

from sklearn.feature_selection import RFE

from sklearn.linear_model import LogisticRegression

model = LogisticRegression()

rfe = RFE(model, n_features_to_select=5)

fit = rfe.fit(X, y)

✅ Pros:

Takes feature interactions into account

Usually more accurate for a specific model

❌ Cons:

Very computationally expensive

Risk of overfitting on small datasets

🧩 3. Embedded Methods

These perform feature selection during model training — it’s “built into” the learning algorithm.

🧪 How it works:

The model penalizes irrelevant features or assigns importance scores during training.

📊 Common Techniques:

Lasso (L1 regularization) – forces some coefficients to zero

Decision tree feature importance

ElasticNet (L1 + L2 regularization)

python

Copy

Edit

from sklearn.linear_model import Lasso

model = Lasso(alpha=0.01)

model.fit(X, y)

print(model.coef_)  # Zero coefficients = unimportant features

✅ Pros:

More efficient than wrapper methods

Good balance of performance and speed

❌ Cons:

Tied to a specific model

May not generalize well to other models

🧠 Summary Table

Method Uses Model? Speed Feature Interaction Example

Filter ❌ No ✅ Fast ❌ No Correlation, Chi2

Wrapper ✅ Yes ❌ Slow ✅ Yes RFE, Forward Selection

Embedded ✅ Yes ⚖️ Medium ✅ Yes Lasso, Tree Importances

💡 Final Tip:

Use Filter methods for a quick pre-selection, Wrapper methods for best performance, and Embedded methods for model-specific tuning.

Learn Data Science Course in Hyderabad

Read More

How to Use Principal Component Analysis (PCA) for Dimensionality Reduction

One-Hot Encoding vs. Label Encoding: When to Use Them

How to Select the Right Features for Machine Learning Models

Feature Engineering and Model Optimization

Visit Our Quality Thought Training Institute in Hyderabad

Get Directions