Model Evaluation Metrics: Confusion Matrix, Precision, Recall, F1 & ROC-AUC

A fraud model is 98% accurate. Sounds excellent — until you notice that 98% of transactions are legitimate, so a model that approves everything scores exactly the same and catches no fraud at all.

This lesson is about the numbers that don't lie, and about the idea underneath all of them: which metric you optimise is a business decision, not a technical one.

The confusion matrix

Four numbers, and every classification metric is built from them:

• True positive (68) — fraud, correctly caught.
• False negative (132) — fraud, missed. A miss.
• False positive (42) — legitimate, wrongly flagged. A false alarm.
• True negative (9,758) — legitimate, correctly passed.

Accuracy collapses these four into one number and hides which kind of mistake you are making — which is precisely the information you need, because the two kinds of mistake almost never cost the same.

Precision and recall

precision = TP / (TP + FP)     # 68 / 110 = 0.618
recall    = TP / (TP + FN)     # 68 / 200 = 0.340
accuracy  = (TP + TN) / total  # 9826 / 10000 = 0.983

In words, which is how you should say it in an interview:

Precision — of everything we flagged, how much really was fraud? Here, 62%. So about 4 in 10 flags are legitimate customers being inconvenienced.

Recall — of all the fraud that happened, how much did we catch? Here, 34%. So 132 frauds went through.

The framing worth memorising: precision is about the cost of a false alarm; recall is about the cost of a miss.

💡 A memory aid that survives exam pressure: precision reads down the predicted column, recall reads across the actual row. Both have TP on top; only the denominator differs.

Which one matters for your problem

Notice rows 3 and 4: the same fraud model, opposite priorities, because the action differs. Blocking a payment hurts a real customer immediately; flagging for review costs a few minutes of an analyst's time. The metric follows the consequence, not the model.

F1 — and why it's a harmonic mean

f1 = 2 * (precision * recall) / (precision + recall)
#    = 2 * (0.618 * 0.340) / 0.958 = 0.439

F1 combines both into one number for when you need a single figure. The important detail is that it is the harmonic mean, not the arithmetic one — and that is the whole point:

precision = 1.00   recall = 0.02

arithmetic mean = 0.51    <- looks acceptable
harmonic mean   = 0.04    <- correctly calls it a bad model

A model that flags one transaction, correctly, and misses everything else has perfect precision and useless recall. The harmonic mean refuses to be fooled by it.

Use F1 when you need one number and both errors matter roughly equally. Use precision and recall separately when they don't — which is most of the time.

Wait — I can change the answer without retraining?

Yes, and it is the most under-used lever in applied ML.

A classifier doesn't really output a class. It outputs a probability, and a threshold turns that into a class. The default is 0.5, but nothing is sacred about 0.5.

probs = model.predict_proba(X_test)[:, 1]

for t in [0.20, 0.35, 0.50, 0.65, 0.80]:
    preds = (probs >= t).astype(int)
    print(f"threshold {t:.2f}  "
          f"precision {precision_score(y_test, preds):.2f}  "
          f"recall {recall_score(y_test, preds):.2f}")

Same model, five completely different products. Lower the threshold and you catch more fraud while annoying more customers; raise it and the reverse. No retraining, no new features — a single number.

And the threshold should come from economics, not from a default. If a missed fraud costs ₹8,000 and reviewing a false alarm costs ₹200, the arithmetic says catch far more fraud and accept the alarms. Compute expected cost at each threshold and pick the minimum.

💡 A refinement worth knowing: if your review team can only handle 50 flags a day, the useful metric is precision@50 — how many of the top 50 highest-risk cases are real. Recall is capped by their capacity anyway, and no model can raise that ceiling.

ROC-AUC and PR-AUC

Every metric so far depends on a chosen threshold. ROC-AUC summarises performance across all thresholds at once, which makes it good for comparing models.

Its clearest interpretation: ROC-AUC is the probability that the model ranks a random positive case above a random negative one. 0.5 is random guessing; 1.0 is perfect.

The catch, and the follow-up question: on heavily imbalanced data ROC-AUC can look reassuring while the model performs badly on the class you care about. With 0.1% positives, the enormous number of true negatives keeps the false-positive rate low and flatters the curve. PR-AUC — the area under the precision-recall curve — is the honest view there, because it ignores true negatives entirely.

Regression metrics

And a case worth knowing about: when errors are asymmetric — being 20 minutes late is worse than 20 minutes early — neither MAE nor RMSE captures that. A quantile loss, predicting the 80th percentile rather than the mean, matches the business better.

🎯 Selection-round radar: "Precision vs recall" and "why is accuracy bad for imbalanced data" are two of the most-asked ML questions anywhere. Define both, give the cancer-screening and spam-filter examples in the same breath to show the costs flip, and add the closer: "and I'd tune the threshold from the cost of each error rather than leaving it at 0.5."

Common mistakes

• Reporting accuracy on imbalanced data.
• Never looking at the confusion matrix.
• Leaving the threshold at 0.5 because it is the default.
• Choosing a metric without asking what each kind of error costs.
• Using ROC-AUC on extreme imbalance where PR-AUC is more honest.
• Reporting one aggregate that hides a class collapsing.
• Optimising a metric the business never asked for.

Quick recap

Practice Zone — PYQs from real selection rounds

Six MCQs, then two tasks: compute every metric from a confusion matrix and report it usefully, and choose metrics for four different systems.

Q1Evaluation metricseasy

What does a confusion matrix show?

Asked in

aWhich features confused the modelbCounts of true positives, false positives, true negatives and false negatives — the full breakdown behind a single accuracy numbercThe correlation between featuresdTraining time per class

Q2Evaluation metricsmedium

What is the difference between precision and recall?

Asked in

aThey are the same thing measured differentlybPrecision: of everything flagged positive, how much really was. Recall: of everything actually positive, how much did we catchcPrecision is for regression, recall for classificationdRecall measures training speed

Q3Evaluation metricsmedium

For cancer screening, which metric matters most and why?

Asked in

aPrecision — false alarms waste moneybRecall — missing a real case can be fatal, while a false positive leads to a follow-up testcAccuracy — it is the standard metricdTraining time

Q4Evaluation metricsmedium

What is the F1 score?

Asked in

aThe average of precision and recallbThe harmonic mean of precision and recall — it punishes a low value in either, so a model can't score well by sacrificing onecThe first metric computed during trainingdAccuracy adjusted for class balance

Q5Evaluation metricshard

What does ROC-AUC measure, and when is PR-AUC better?

Asked in

aROC-AUC measures training speedbROC-AUC is the probability the model ranks a random positive above a random negative, across all thresholds; PR-AUC is more informative on heavily imbalanced data, where ROC-AUC can look good despite poor positive-class performancecThey are identicaldPR-AUC only works for regression

Q6Evaluation metricshard

Your fraud model has 92% precision and 34% recall. The business wants more fraud caught. What do you do first?

Asked in

aRetrain with a bigger modelbLower the classification threshold — recall rises and precision falls, and the right threshold comes from the cost of a missed fraud versus the cost of reviewing a false alarmcCollect more featuresdSwitch to accuracy as the metric

Hands-on tasks:

                  Predicted
                Fraud    Legit
Actual Fraud      68       132
       Legit      42      9758

FAQ

Next lesson: the work that improves models more than any algorithm change — Lesson 7: Feature Engineering →