The Problem with Standard Linear Regression
Ordinary Least Squares (OLS) regression works fine for clean datasets with few features. It estimates parameters by minimizing the sum of squared residuals: Σ(yi - ŷi)².
But real enterprise datasets rarely cooperate. When you're predicting house prices or customer churn with dozens of correlated features and noisy inputs, OLS becomes unstable. The model overfits: it memorizes training data patterns that don't generalize.
Regularization techniques solve this by adding a penalty term to the loss function. The model now balances prediction accuracy against complexity. Two approaches dominate: Ridge (L2) and Lasso (L1).
Ridge Regression: Shrink Everything
Ridge adds a penalty proportional to the sum of squared coefficients:
Loss = Σ(yi - ŷi)² + λΣβj²
The lambda parameter controls regularization strength. Higher lambda means more aggressive shrinkage.
What it does: Reduces all coefficients toward zero without eliminating any. If you have ten features, you'll still have ten features, just with smaller, more stable coefficients.
When to use it: Your features are highly correlated (multicollinearity), most variables legitimately influence the outcome, and you need stable predictions. Ridge distributes weight across correlated features rather than picking arbitrarily.
What it doesn't do: Feature selection. Ridge shrinks coefficients close to zero but never hits exactly zero.
In sklearn: Ridge(alpha=1.0) where alpha is your lambda parameter. Use RidgeCV for automatic cross-validation tuning.
Lasso Regression: Cut the Weak Links
Lasso uses L1 regularization, penalizing absolute coefficient values:
Loss = Σ(yi - ŷi)² + λΣ|βj|
What it does: Forces some coefficients to exactly zero. This is automatic feature selection. A Lasso model might decide that only 3 of your 10 features actually matter.
When to use it: You suspect most features are noise, you need an interpretable model for stakeholders, or you're working with high-dimensional data where only a subset of predictors truly drives outcomes.
The trade-off: With correlated features, Lasso arbitrarily picks one and zeros out the others. If house size and number of rooms correlate at 0.85, Lasso might give size a coefficient of $180/sq ft and rooms exactly $0. Ridge would split the difference more evenly.
In sklearn: Lasso(alpha=0.1). Start with smaller alpha values than Ridge.
The Geometry Matters
Lasso's diamond-shaped constraint region has corners where coefficients hit zero exactly. Ridge's circular constraint smoothly shrinks coefficients but they approach zero asymptotically. This geometric difference explains why Lasso performs feature selection and Ridge doesn't.
What About Elastic Net?
Elastic Net combines both penalties: Loss = RSS + λ₁Σ|βj| + λ₂Σβj². This handles correlated features better than pure Lasso while still enabling feature selection. Worth considering when you have grouped correlated predictors.
Implementation Notes
Always scale your features first. Regularization penalizes coefficient magnitude, so features on different scales (house size in thousands vs number of bedrooms) will be penalized unequally.
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
Don't penalize the intercept. Both sklearn's Ridge and Lasso handle this correctly by default.
Tune lambda via cross-validation. Don't guess. RidgeCV and LassoCV automate this:
from sklearn.linear_model import RidgeCV
ridge = RidgeCV(alphas=[0.1, 1.0, 10.0])
ridge.fit(X_train_scaled, y_train)
The Real Question
Ridge or Lasso? Ask: do I need all my features (Ridge), or should the model tell me which ones matter (Lasso)? For exploratory work, Lasso's feature selection provides signal. For production models with known important predictors, Ridge's stability often wins.
Neither is universally better. The question is what your data needs.