Overfitting & Underfitting: Bias-Variance, Cross-Validation and Data Leakage

Two students prepare for an exam. One memorises last year's question paper word for word — every answer, in order. The other understands the concepts. On last year's paper the memoriser scores 100%. On this year's, they are lost.

That is overfitting, and it is the most-asked concept in ML placement interviews for a good reason: it is the failure mode that every model, every algorithm and every project has to deal with.

The three fits

Underfitting — the model is too simple to capture the pattern. It performs poorly on training data and on new data.

Good fit — it captures the underlying trend without chasing individual points. Slightly better on training than on test, which is normal and expected.

Overfitting — the model has learned the noise as well as the signal. Near-perfect on training data, poor on anything new.

Diagnosing from two numbers

You need only the training score and the test score, and the diagnosis is entirely in the gap.

Two habits follow. Always compute both numbers — a candidate who reports only training performance has told you nothing. And always ask whether the test score is good relative to a baseline: 85% is excellent if a simple rule gets 60%, and worthless if the rule gets 84%.

The bias-variance tradeoff

The formal framing behind those two failures, and a guaranteed interview question.

Bias is error from wrong assumptions — the model is too simple to represent the truth. Fitting a straight line to a curved relationship has high bias no matter how much data you give it.

Variance is sensitivity to the particular training sample — train on a slightly different set and you get a very different model. High-variance models chase noise.

Increasing model complexity lowers bias and raises variance. Total error is minimised somewhere in between — and validation data is what finds that point for you, not intuition.

💡 The analogy that lands in an interview: high bias is an archer who consistently hits the same spot away from the bullseye. High variance is one whose arrows scatter all over the target. You want tight and centred, and improving one usually costs a little of the other.

Fixing each one

That last cell on the right matters more than it looks. If a model underfits no matter what you do, the honest possibility is that the features simply don't contain the answer — and no algorithm rescues that. Teams lose months to model tuning on problems the data cannot support.

Cross-validation

One train-test split gives you one number, and with a few hundred rows that number depends heavily on which rows happened to land where. K-fold cross-validation splits the data k ways, trains k times, and averages.

from sklearn.model_selection import cross_val_score

scores = cross_val_score(model, X, y, cv=5, scoring="f1")

print("folds:", scores.round(3))
print("mean :", scores.mean().round(3), "±", scores.std().round(3))

Two things to read there. The mean is a more trustworthy estimate than any single split. And the spread is information — that fourth fold at 0.641 is worth investigating, because a wide variance across folds usually means the data is heterogeneous or the model is unstable.

Use stratified k-fold for classification so each fold keeps the class balance. And for time-series data, use a time-aware split — training on future data to predict the past is something production can never do.

Regularisation

The general idea: penalise complexity, so the model only takes on complexity that genuinely pays for itself in reduced error.

The L1-versus-L2 distinction is asked often: Lasso can zero features out; Ridge shrinks but keeps them. Choose L1 when you suspect many features are useless, L2 when features are correlated and you want them shrunk together.

Wait — my model got 99.9%. Is that good?

On a problem experts find hard? Almost certainly not. Extraordinary results demand extraordinary scrutiny, and near-perfect scores are usually a bug rather than a breakthrough.

The usual culprit is data leakage — information from the answer sneaking into the features. Four ways it happens:

# 1. A FEATURE THAT ENCODES THE ANSWER
#    Predicting churn using "account_closed_date" — recorded when
#    the customer churned. The model is reading the answer.
#    TEST: would this value be known, with this value, on the day
#    I make the prediction?

# 2. PREPROCESSING BEFORE SPLITTING
scaler.fit_transform(df)                    # ✗ learns from test rows
X_train, X_test = train_test_split(df)

X_train, X_test = train_test_split(df)      # ✓ split first
scaler.fit(X_train); scaler.transform(X_test)

# 3. DUPLICATE ROWS SPLIT ACROSS TRAIN AND TEST
#    The model has literally seen the test row before.

# 4. RANDOM SPLIT ON TIME-DEPENDENT DATA
#    Trains on the future to predict the past. Split by time.

Leakage is the single most common reason a brilliant offline model fails in production, because the leaked feature does not exist — or does not have that value — at prediction time.

🎯 Selection-round radar: "What is overfitting?" is near-certain. Give the definition, then the diagnosis (high training score, low test score — the gap is the signal), then three fixes. And if they follow up with a 99% result on a hard problem, the expected answer is suspect data leakage, not congratulations.

Common mistakes

• Reporting only the training score.
• Using the test set repeatedly for tuning — it stops being held out.
• Preprocessing (scaling, imputing, encoding) before splitting.
• Random splits on time-dependent data.
• Duplicate or near-duplicate rows straddling train and test.
• Treating a near-perfect score as success rather than as a bug report.
• Adding model complexity when the features don't contain the signal.

Quick recap

Practice Zone — PYQs from real selection rounds

Six MCQs, then two tasks: diagnose four models from their scores, and find every leak in a real pipeline.

Q1Overfittingeasy

What is overfitting?

Asked in

aUsing too much training databThe model learns the noise and specifics of the training data instead of the general pattern, so it performs well on training data and poorly on new datacThe model runs out of memorydTraining for too little time

Q2Underfittingeasy

A model scores 62% on training data and 61% on test data. This is:

Asked in

aOverfittingbUnderfitting — it performs poorly on both, meaning the model is too simple to capture the patterncA perfect modeldImpossible

Q3Bias-variancemedium

What is the bias-variance tradeoff?

Asked in

aA tradeoff between fairness and speedbSimple models have high bias and underfit; complex models have high variance and overfit — total error is minimised somewhere in betweencA choice between two algorithmsdThe tradeoff between training and inference cost

Q4Cross-validationmedium

Why use k-fold cross-validation instead of one train-test split?

Asked in

aIt trains fasterbIt splits the data k ways and averages the results, so the estimate doesn't depend on one lucky or unlucky split — especially valuable with limited datacIt removes the need for a test set foreverdIt prevents overfitting by itself

Q5Regularisationhard

What is the difference between L1 (Lasso) and L2 (Ridge) regularisation?

Asked in

aL1 is faster to computebL1 can drive coefficients to exactly zero, performing feature selection; L2 shrinks them toward zero but rarely to zero, keeping all features with reduced influencecL2 only works on classificationdThey produce identical results

Q6Overfittinghard

A model achieves 99.8% test accuracy on a hard problem. What should you check first?

Asked in

aNothing — celebratebData leakage — a feature that encodes the answer, or test data that overlaps the training setcThe GPU temperaturedWhether to add more layers

Hands-on tasks:

df = pd.read_csv("customers.csv")

scaler = StandardScaler()
df[num_cols] = scaler.fit_transform(df[num_cols])      # (1)
df = df.fillna(df.mean())                              # (2)

X = df.drop("churned", axis=1)
y = df["churned"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42)              # (3)

model.fit(X_train, y_train)
print(model.score(X_test, y_test))    # 0.99

FAQ

Next lesson: the numbers you report, and why accuracy is usually the wrong one — Lesson 6: Evaluation Metrics →