A doctor works through symptoms in order — fever? how long? any rash? — and narrows to a diagnosis. A pharmacist recognises a customer's complaint because it resembles a hundred previous ones. A panel of three specialists votes and takes the majority.
Those are, almost exactly, a decision tree, K-nearest neighbours, and an ensemble. This lesson covers the main classification algorithms — what each does, where each wins, and the question interviewers actually care about: how you choose.
Decision trees
A flowchart the algorithm builds itself. At each step it picks the question that best separates the classes, then repeats on each branch.
[existing_loans <= 1?]
/ \
yes no
| |
[income <= 40000?] [income <= 120000?]
/ \ / \
REJECT APPROVE REJECT APPROVE
(n=310) (n=1240) (n=520) (n=180)Its great virtue: you can read it. No scaling needed, it handles mixed numeric and categorical features, and you can hand the path to a customer as an explanation. That is why trees survive in regulated settings where a more accurate model would be unusable.
Its weaknesses are equally real. A deep tree overfits badly — given enough depth it will memorise the training data. Thresholds are hard cliffs, so ₹119,999 and ₹120,001 get opposite outcomes. And small changes in the data can produce a completely different tree.
Random forests
The fix for a tree's instability is not a better tree — it is many trees. A random forest trains hundreds, each on a random subset of rows and considering a random subset of features at each split, then takes the majority vote.
Why it works: each individual tree overfits, but they overfit differently. Averaging keeps the signal they agree on and cancels the noise they don't. This is called bagging, and it reduces variance.
from sklearn.ensemble import RandomForestClassifier
model = RandomForestClassifier(
n_estimators=300, # number of trees
max_depth=None, # let them grow; averaging controls overfitting
n_jobs=-1, # trees are independent -> parallel
).fit(X_train, y_train)
for name, imp in sorted(zip(features, model.feature_importances_),
key=lambda t: -t[1])[:5]:
print(f"{name:24s} {imp:.3f}")Result
Random forests are the reliable default: they rarely embarrass you, need little tuning, handle mixed data, and give feature importances for free. The cost is interpretability — you cannot show a customer 300 trees.
Boosting — and bagging vs boosting
Boosting also builds many trees, but sequentially: each new tree focuses on the examples the previous ones got wrong. XGBoost, LightGBM and CatBoost are the implementations you will meet.
| Bagging (random forest) | Boosting (XGBoost) | |
|---|---|---|
| Trees are built | independently, in parallel | sequentially, each correcting the last |
| Primarily reduces | variance | bias |
| Typical accuracy | very good | usually the best on tabular data |
| Tuning needed | little | more — learning rate, depth, estimators |
| Sensitive to noisy labels | fairly robust | yes — it chases the errors, including the wrong ones |
Bagging averages away variance; boosting attacks bias by iterating on mistakes. That one line answers the interview question cleanly. The practical implication: gradient boosting usually scores highest on tabular data, and a random forest is the safer starting point.
K-Nearest Neighbours
The simplest idea in the course: to classify a new point, find the k closest training points and take their majority vote.
It is a lazy learner — there is essentially no training, just storage. All the work happens at prediction time, which makes it slow to predict and slower as the dataset grows. That rules it out for real-time systems.
💡 Two things KNN requires that people forget. Scaling is mandatory, because it measures distance — salary in lakhs next to age in years means the distance is effectively salary alone. And it degrades in high dimensions, where everything becomes roughly equidistant from everything else.
Naive Bayes
Uses probability to ask: given these features, which class is most likely? The "naive" part is an assumption — that all features are independent of each other given the class.
That assumption is almost always false. In spam detection, "free" and "offer" obviously co-occur. And yet the classification often lands correctly anyway, because being wrong about the exact probabilities doesn't necessarily change which class comes out on top.
A wrong assumption can still produce the right decision — a nice thing to be able to say in an interview. Its practical strengths: extremely fast, works with little data, and handles high-dimensional text well, which is why it remains a strong text baseline.
Support Vector Machines
SVM finds the boundary that separates the classes with the widest possible margin — the biggest gap between the boundary and the nearest points of each class. Those nearest points are the "support vectors", and they alone determine the boundary.
The kernel trick lets it handle non-linearly separable data by implicitly working in a higher-dimensional space where a straight boundary does exist.
SVMs shine on small-to-medium datasets with clear margins and many features. They scale poorly to very large datasets, need scaling, and don't naturally produce probabilities. In practice they have been largely displaced by gradient boosting for tabular work, but they are still asked about.
Wait — so which one do I actually pick?
This is the question that matters, and the honest answer is that the constraint usually picks the algorithm before accuracy gets a vote.
| Constraint | Choose |
|---|---|
| Every decision must be explained to a regulator | decision tree or logistic regression |
| Tabular data, accuracy is what matters | gradient boosting (XGBoost / LightGBM) |
| Need a reliable answer with minimal tuning | random forest |
| Text classification, need a fast baseline | Naive Bayes or logistic regression on TF-IDF |
| Prediction must return in milliseconds | a linear model — never KNN |
| Images, audio, long text | a neural network, ideally pretrained (lesson 9) |
| Very little data | simple models — logistic regression, Naive Bayes |
And whatever you choose, fit a simple baseline alongside. If logistic regression gets 84% and your tuned ensemble gets 86%, you now know exactly what the complexity bought — and whether two points are worth a model nobody can explain.
🎯 Selection-round radar: "Which algorithm would you use and why?" is a judgement question, not a knowledge one. Never answer with one algorithm for every scenario. Ask (or state) the constraints first — data type, size, explainability requirement, latency — then choose, and mention the baseline you would compare against.
Common mistakes
- Answering "XGBoost" to every scenario question.
- Using KNN where prediction latency matters.
- Forgetting to scale for KNN and SVM (trees don't need it).
- Deploying a single deep decision tree — it will overfit.
- Choosing a black-box model where decisions must be explained.
- Skipping the simple baseline, so the complex model's value is unknown.
- Reading random forest feature importances as causal.
Quick recap
| Algorithm | One-liner |
|---|---|
| Decision tree | readable flowchart; overfits alone; hard threshold cliffs |
| Random forest | many independent trees averaged — bagging, reduces variance |
| Boosting | sequential trees correcting errors — reduces bias; best on tables |
| KNN | lazy; no training, slow prediction; needs scaling |
| Naive Bayes | assumes independent features — usually false, often works; fast on text |
| SVM | widest-margin boundary; kernel trick for non-linear data |
| Choosing | constraints first — explainability, latency, data type — then accuracy |
Practice Zone — PYQs from real selection rounds
Six MCQs, then two tasks: pick algorithms for four constrained scenarios, and read a decision tree critically.
How does a decision tree make a prediction?
Asked in
Why does a random forest usually beat a single decision tree?
Asked in
What is the key difference between bagging and boosting?
Asked in
Why is Naive Bayes called 'naive'?
Asked in
Which statement about K-Nearest Neighbours is correct?
Asked in
For a tabular business dataset with 50,000 rows and 30 mixed numeric and categorical features, what is the most sensible first choice?
Asked in
Hands-on tasks:
Choose an algorithm and give the reason for each: (1) a bank must explain every loan rejection to the regulator, (2) 200,000 rows of tabular telecom data, accuracy is all that matters, (3) classifying 50,000 support emails by topic, (4) a real-time recommender that must respond in under 10ms.
Asked in
Explain what this tree does, predict the outcome for a 34-year-old with monthly income ₹80,000 and 2 existing loans, and name one weakness of the tree as shown.
Asked in
[existing_loans <= 1?]
/ \
yes no
| |
[income <= 40000?] [income <= 120000?]
/ \ / \
yes no yes no
| | | |
REJECT APPROVE REJECT APPROVE
(n=310) (n=1240) (n=520) (n=180)FAQ
Do these algorithms work for multi-class problems?
Yes — trees, forests, boosting, KNN and Naive Bayes handle multiple classes natively. Logistic regression and SVM are binary at heart and are extended by strategies like one-vs-rest, which scikit-learn applies for you. The metrics change more than the algorithms do: you will want per-class recall rather than one aggregate.
How do I handle imbalanced classes?
Several levers, and you usually combine them. Set class_weight="balanced", which most scikit-learn classifiers support. Adjust the decision threshold — often the cheapest and most effective fix. Resample (oversample the minority, or synthesise with SMOTE), fitting the resampling on training data only. And above all, stop using accuracy (lesson 6).
Should I always try every algorithm?
No — that is a leaderboard habit, not engineering. Start with one interpretable baseline and one strong default (logistic regression plus gradient boosting covers most tabular work). Spend the time you saved on features and on the evaluation setup, which move the number far more than a fifth algorithm will.
Next lesson: the failure mode that affects every algorithm here, and how to detect it — Lesson 5: Overfitting & Underfitting →