ML Project Lifecycle: From Problem Definition to Production and Drift

Most ML projects that fail don't fail at the model. They fail because nobody defined what "churn" meant, or because a feature that existed in the historical table doesn't exist at prediction time, or because the model was excellent and no one changed a single decision because of it.

This lesson is the shape of a real project — and the parts that interviews probe hardest are, unsurprisingly, the ones furthest from the algorithm.

The lifecycle

Two things that diagram is telling you. Deployment is the middle of the project, not the end — monitoring feeds back into data, and the loop continues for as long as the model runs. And the effort bar at the bottom is the honest proportion: data work dominates, model training is a slice.

1. Define the problem

The step most often rushed, and the one that sinks the most projects. Four questions, answered before anything else:

• What decision will change because of this model? If nobody can name one, stop. A model nobody acts on has no value however accurate it is.
• What exactly is the target? "Churn" is not a label. "Has not recharged for 30 consecutive days" is.
• When is the prediction made? 48 hours before the appointment, at checkout, on the 1st of the month. This defines which features can legally exist.
• What does success mean in business terms? Fewer wasted slots, more fraud caught per analyst hour — not "92% accuracy".

💡 Question 3 is quietly the most technical of the four. Fixing the prediction point up front is what makes data leakage detectable later: every feature has to be answerable at that moment.

2. Data — where the time goes

Collecting, cleaning, joining, labelling and engineering features is the bulk of the work in essentially every real project.

Split by time whenever the problem is temporal. Training on future rows to predict past ones is something production can never do, and it inflates every number you report.

3. Baseline, then model

Build the dumbest reasonable thing first. Predict the majority class. Predict last month's value. Apply the one rule the business already uses.

Without a baseline, 85% accuracy is a number with no meaning. If the existing rule gets 84%, your model bought one point in exchange for a system somebody has to maintain forever — and knowing that before you build it is worth a lot.

Then model: one interpretable option and one strong default. For tabular data that is usually logistic regression plus gradient boosting. Resist trying nine algorithms; features and evaluation move the number much more than a fifth model will.

4. Evaluate

On held-out data, with a metric that matches the cost of each kind of error (lesson 6), reported per segment rather than as one aggregate.

And there is a second evaluation nobody teaches: does the model beat the baseline by enough to justify existing? A model is a permanent liability — someone monitors it, retrains it, and explains it when it goes wrong. Two points over a rule is often not worth that.

5. Deploy

1. Shadow run. Score live traffic, act on nothing, compare predictions to what actually happened. This is what catches training-serving skew before it reaches anyone.
2. A/B test. A fraction of traffic gets the model-driven treatment, the rest gets current practice. Measure the business outcome — retention, revenue, resolution rate — not the model's accuracy.
3. Ramp, with a one-switch rollback.

Training-serving skew deserves naming, because it is the most common reason a good offline model disappoints. A feature computed with a 30-day window in your notebook and a 28-day window in the production pipeline is enough — the model silently receives inputs unlike anything it trained on, and no error is ever raised.

TRAINING (notebook)          PRODUCTION (pipeline)
  avg over last 30 days   vs   avg over last 28 days
  includes today          vs   excludes today
  nulls -> median         vs   nulls -> 0
  city cleaned/lowercased vs   raw string from the form

Any one of these is enough to degrade the model silently.
The fix: ONE code path computing features for both — a feature
store, or literally the same function called in both places.

Wait — it worked for months, then got worse

Expected, not surprising. Three causes, and one of them is not what people assume.

Data drift — the inputs changed. New customer segment, new product range, a price change, a competitor promotion.

Concept drift — the relationship changed even though the inputs look similar. A demand model met both at once during the pandemic.

A pipeline bug — and check this first, because it is the most common and the cheapest to find. An upstream source changed format, or a feature is silently defaulting to zero. Compare live feature distributions against training ones; a column whose mean shifted or whose variance collapsed is the usual smoking gun.

There is also a fourth, sneakier possibility: a feedback loop, where the model's own actions change the data it will next be trained on. If low forecasts cause low stock, low stock causes low sales, and low sales confirm the forecast. Keeping a small untreated control group is how you detect it.

🎯 Selection-round radar: "Your model works offline but not in production — why?" is a favourite. Three answers, in order: training-serving skew (features computed differently), data leakage that inflated the offline score, and distribution shift between historical and live data. Then add the prevention: one code path for features, and shadow-run before switching.

The non-technical risks

These get asked more than students expect, especially at companies deploying models that affect people.

• Fairness. A model can be accurate overall and systematically worse for one group. Features often proxy for protected attributes — pincode for community, distance from a clinic for income. Evaluate error rates across groups, not just overall.
• Explainability. If a person is refused a loan or a job, someone has to explain why. Decide the explainability requirement before choosing an algorithm, not after.
• Feedback loops. The model changes the world it predicts. Reserve a control group.
• Asymmetric harm. Ask who is hurt when the model is wrong, and whether that person had any say. An overbooked appointment slot means someone waits.
• Privacy and governance. Who can see a score? Does it enter a permanent record? What can be deleted?

Common mistakes

• Starting with the algorithm instead of the decision it will change.
• A vague target definition — "churn" with no rule behind it.
• No baseline, so the model's value is unknown.
• Random splits on time-dependent data.
• Features unavailable at prediction time.
• Two code paths computing features — training-serving skew.
• Treating deployment as the end of the project.
• Monitoring only errors, not feature distributions.
• Never checking performance across demographic groups.

Quick recap

Practice Zone — PYQs from real selection rounds

Six MCQs, then two tasks: plan a full project including the risks that aren't technical, and investigate a model that degraded in production.

Q1ML lifecycleeasy

What is the first step in a machine learning project?

Asked in

aChoosing an algorithmbDefining the business problem precisely — what decision will change because of this model, and what does success mean in business termscCollecting as much data as possibledSetting up a GPU

Q2ML lifecyclemedium

Roughly how is effort distributed across a typical ML project?

Asked in

a80% model training, 20% everything elsebThe majority goes to problem definition, data collection, cleaning and feature engineering; model training is usually a small fractioncIt is evenly split across all stagesdMost time goes to choosing hyperparameters

Q3ML lifecyclemedium

What is a baseline, and why build one first?

Asked in

aThe first neural network you trybThe simplest reasonable approach — a rule, the majority class, or last month's value — that tells you what your sophisticated model must beat to be worth its complexitycThe minimum accuracy required by lawdThe data before cleaning

Q4ML lifecyclemedium

What is model drift?

Asked in

aThe model file becoming corruptedbPerformance degrading over time because the real world changed — the input distribution moved, or the relationship between features and target didcThe model getting slowerdWeights changing during inference

Q5ML lifecyclehard

Your model performs well offline but poorly in production. What is the most likely cause?

Asked in

aThe production server is slowbTraining-serving skew — features computed differently in production than in training, data leakage that inflated the offline score, or a distribution shift between historical and live datacThe model file was truncateddProduction always performs worse

Q6ML lifecyclehard

How should a new model be rolled out?

Asked in

aReplace the old one everywhere immediatelybShadow-run it against live traffic without acting on its output, then A/B test on a fraction of traffic measuring the business outcome, with a fast rollback pathcDeploy on Friday eveningdLet users opt in manually

Hands-on tasks:

FAQ

That's the course. Next: the company pages — reported questions from each company's rounds, with full answers. Start with TCS Machine Learning Interview Questions →

Or continue the AI track: Generative AI, RAG, Prompt Engineering and Fine-Tuning & LLMOps.