Clustering & PCA: K-Means, Choosing k, DBSCAN and Dimensionality Reduction

A shop owner who has never heard of machine learning can tell you about her customers: there are the daily regulars, the weekend families, the students who only come for offers, and the ones who appear once at Diwali. Nobody defined those groups. She noticed them.

That is clustering. And its companion — reducing forty confusing columns to two you can plot — is dimensionality reduction. Both work without labels, and both need a human to say whether the result is actually useful.

K-Means

The most-used clustering algorithm, and pleasingly simple:

1. Pick k starting points (centroids).
2. Assign every data point to its nearest centroid.
3. Move each centroid to the mean of the points assigned to it.
4. Repeat 2 and 3 until nothing moves.

from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler

X = StandardScaler().fit_transform(df[["recency_days", "orders", "spend"]])

km = KMeans(n_clusters=4, n_init=10, random_state=42).fit(X)
df["segment"] = km.labels_

print(df.groupby("segment")[["recency_days", "orders", "spend"]].mean())
print(df["segment"].value_counts())

Two things that scaling line is doing, and both are essential. K-Means measures distance, so unscaled spend in thousands next to orders in single digits would mean it clusters on spend alone. And n_init=10 runs it ten times from different starting points and keeps the best, because the result depends on where the centroids started.

Choosing k

You must supply k in advance, and there is no formula that gives the right answer. Two tools narrow the range:

The elbow method. Plot within-cluster variance against k. It always falls as k rises (at k = number of points it hits zero), so what you look for is the "elbow" — the point after which adding clusters stops buying much.

Silhouette score. Measures how tight each cluster is relative to how far it sits from the next one. Higher is better.

for k in range(2, 9):
    km = KMeans(n_clusters=k, n_init=10, random_state=42).fit(X)
    print(f"k={k}  inertia={km.inertia_:>10.0f}  "
          f"silhouette={silhouette_score(X, km.labels_):.3f}")

Both point to k = 4 here. But that is guidance, not the decision. The deciding question is not mathematical: can the business run a different action for each group? Six tidy segments marketing cannot build separate campaigns for are worse than four they can. Also check segment sizes — a cluster holding 0.5% of customers is rarely worth a campaign.

Where K-Means breaks

Worth knowing, because a bad clustering result usually has one of these causes:

• It assumes roughly spherical, similarly-sized clusters. Elongated or crescent-shaped groups defeat it.
• It is sensitive to outliers. A handful of extreme spenders will each capture a centroid, leaving everyone else in one enormous cluster — the classic symptom.
• It requires scaling. Same reason as KNN: it measures distance.
• Every point gets assigned, including genuine noise. There is no "this one belongs nowhere".

💡 The diagnostic pattern to recognise: one giant cluster and several tiny ones almost always means unscaled features or outliers. Fix those and re-cluster before concluding anything about your customers.

Other clustering methods

DBSCAN is the one worth naming in an interview alongside K-Means, because it fixes two of K-Means' weaknesses at once: it does not need k, and it explicitly marks points that belong to no cluster.

PCA

Principal Component Analysis takes many correlated features and produces a smaller number of new ones — components — ordered so the first captures the most variance in the data, the second the next most, and so on.

from sklearn.decomposition import PCA

X_scaled = StandardScaler().fit_transform(X_train)   # scaling first, always

pca = PCA(n_components=0.95)        # keep enough to explain 95% of variance
X_reduced = pca.fit_transform(X_scaled)

print("original :", X_train.shape[1], "features")
print("reduced  :", X_reduced.shape[1], "components")
print("explained:", pca.explained_variance_ratio_[:5].round(3))

Sixty columns down to fourteen while keeping 95% of the variance. Less noise, faster training, and — if you keep two or three — something you can actually plot.

The distinction that gets tested: PCA is feature extraction, not featureselection. It does not pick your best columns; it builds new ones from combinations of all of them.

Wait — what does PCA cost me?

Interpretability, completely. Component 1 is a weighted mixture of all sixty original features. You can no longer say "income drove this prediction", only "component 1 was −0.42", which means nothing to a customer, a regulator or a colleague.

In a regulated setting that cost is usually decisive, and it is the reason PCA is a poor fit for credit scoring however many features you have.

Two other requirements worth stating: scale before applying it, or high-variance columns dominate the components purely because of their units; and fit it on training data only, then transform the test set — it learns from the data, so it leaks like any other fitted transform.

Use PCA when dimensionality is genuinely a problem: many correlated features, and interpretability you can afford to lose. Not by default, and never on eight features that already train in three seconds.

🎯 Selection-round radar: "Explain K-Means" and "what is PCA" are both standard. For K-Means, give the four steps and then add the two things that show experience: you must choose k, and results depend on initialisation and on scaling. For PCA, say "extraction, not selection" and name the cost — you lose interpretability — before they ask.

Making the result useful

Unsupervised output is not a deliverable. Four steps turn it into one:

1. Profile each cluster — the mean of each feature per group, as in the output above.
2. Name them from the data. "Lapsing high-value", "discount-driven regulars", "new and exploring". A name people can say out loud is what makes a segment real to them.
3. Attach an action to each. If two segments get the same treatment, they are one segment.
4. Validate with a person who knows the business. There is no ground truth here, so a human confirming the groups are real and distinct is the only validation available.

And plan to re-run it. Customers move between segments, so refresh on a schedule and track migration — that movement is often more interesting than the segments themselves.

Common mistakes

• Clustering without scaling.
• Leaving outliers in and getting one giant cluster.
• Choosing k purely from a silhouette score, ignoring usefulness.
• Expecting the algorithm to name or explain its groups.
• Applying PCA where interpretability is a hard requirement.
• Fitting PCA or the scaler on the full dataset before splitting.
• Using PCA on a handful of features that were never a problem.
• Presenting clusters as findings without a business person validating them.

Quick recap

Practice Zone — PYQs from real selection rounds

Six MCQs, then two tasks: design a customer segmentation end to end, and decide where PCA belongs.

Q1Clusteringeasy

How does K-Means work?

Asked in

aIt builds a decision treebIt places k centroids, assigns each point to its nearest centroid, moves each centroid to the mean of its points, and repeats until nothing movescIt fits a line through the datadIt sorts data alphabetically

Q2Clusteringmedium

How do you choose the number of clusters k?

Asked in

aAlways use 3bUse the elbow method or silhouette score as guidance, then check whether the resulting groups are meaningful and actionable to the businesscUse the number of featuresdThere is a formula that gives the exact answer

Q3Clusteringmedium

Which is a real limitation of K-Means?

Asked in

aIt cannot handle numeric databIt assumes roughly spherical, similarly-sized clusters and is sensitive to outliers and to feature scalingcIt requires labelled datadIt only works with two features

Q4PCAmedium

What does PCA do?

Asked in

aIt selects the most important original featuresbIt creates new features — combinations of the originals — ordered so the first few capture most of the variance, letting you keep fewer dimensionscIt clusters the datadIt removes missing values

Q5PCAhard

What is the main cost of using PCA?

Asked in

aIt requires labelled databYou lose interpretability — components are mixtures of all original features, so you can no longer say 'income drove this prediction'cIt always reduces accuracydIt only works on images

Q6Clusteringhard

You cluster customers and get one enormous cluster and four tiny ones. What is the likely cause?

Asked in

aK-Means is brokenbUnscaled features or outliers — a few extreme points pull centroids toward themselves, leaving everything else in one groupcThere are genuinely five segmentsdThe dataset is too small

Hands-on tasks:

FAQ

Next lesson: the model family behind deep learning, and where it genuinely wins — Lesson 9: Neural Networks Basics →