Message Queues & Asynchronous Processing Explained (Kafka, RabbitMQ)

You place an order on Flipkart. Behind that one tap, the system needs to: charge your card, update inventory, send you a confirmation email, notify the seller, and log the event for analytics. Should your "place order" button really sit there spinning while all five of those happen, one after another, in the same request? Almost certainly not — and message queues are exactly how you say no.

Synchronous vs asynchronous work

Some work genuinely must finish before you respond to the user — charging their card is one of them; the user needs to know immediately if payment failed. Other work does NOT need to finish before responding — sending a confirmation email can happen a second (or ten seconds) later with zero real difference to the user's experience. The trick is telling these two categories apart, and moving everything in the second category off the critical request path.

Producer, consumer, and the queue between them

A producer (your order-placement code) pushes a message ("send confirmation email for order #4821") onto a queue and immediately moves on — it does not wait for the email to actually be sent. A separate consumer (a worker process, possibly a whole fleet of them) picks messages off the queue whenever it's ready and does the actual work. This decouples the two: the producer's request finishes fast, and the consumer can be scaled independently based on how much background work is piling up.

Real queue technologies

Apache KafkaRabbitMQ

Kafka vs a traditional queue like RabbitMQ

RabbitMQ is a traditional message broker: a message is typically delivered to one consumer and then removed from the queue — a good fit for "do this task exactly once, by someone." Kafka is built around a durable, ordered, replayable log of events, organized into partitions — multiple independent consumer groups can each read the SAME stream of events independently, at their own pace, which makes Kafka a good fit for event-driven architectures where several different services all care about the same event ("an order was placed" might interest inventory, analytics, AND notifications, all separately).

Retries and duplicate messages

Queues generally guarantee "at-least-once" delivery, not "exactly-once" — if a consumer crashes after doing the work but before acknowledging the message, the queue will redeliver it, and the SAME work could run twice. This is why consumers processing messages that have real side effects (like charging money) need to be written idempotently — using an idempotency key so processing the same message twice has the same effect as processing it once.

Dead-letter queues

Sometimes a specific message can never be processed successfully — a corrupted file, malformed data — and retrying it forever just clogs the queue, slowing down every healthy message behind it. After a bounded number of retry attempts, a well-designed system moves that message to a separate dead-letter queue instead, where it can be investigated deliberately (alerting, manual review) rather than silently retried forever or silently lost.

Backpressure

If producers push messages faster than consumers can process them, the queue keeps growing — unboundedly, if nothing intervenes. Backpressure is any mechanism that pushes back on that growth: capping queue size and rejecting or slowing new producers once it's full, or auto-scaling the consumer fleet based on queue depth. Without some form of backpressure, a temporary slowdown in consumers can turn into an unbounded memory problem for the queue itself.

Common mistakes

• Making every side effect synchronous, so a slow or down email service breaks the entire checkout flow.
• Writing consumers that aren't idempotent, breaking under at-least-once redelivery.
• No dead-letter queue, so one permanently-broken message clogs the whole pipeline forever.
• No backpressure, letting a slow consumer fleet cause unbounded queue growth.

Quick recap

Practice Zone

Five MCQs, then two applied questions.

Q1Sync vs asynceasy

Why would a system move some work from synchronous (in the request path) to asynchronous (via a queue)?

Asked in

aTo make the code more complex on purposebSo the user doesn't have to wait for slow, non-essential work (like sending an email or writing an analytics event) before getting a responsecAsync processing is always more consistentdThere's no real benefit, it's just a trend

Q2Kafka conceptsmedium

In Kafka, what is a 'consumer group'?

Asked in

aA single consumer reading every message alonebA set of consumers that split the work of reading a topic's partitions between them, so each partition is read by only one consumer in the groupcA backup copy of the whole topicdA way to encrypt messages

Q3Message orderinghard

Why might a queue/stream deliver messages out of order, and what's a common mitigation?

Asked in

aOrdering is never a real concern for any systembMultiple partitions/consumers process independently in parallel, so global order across the whole topic isn't guaranteed — mitigation: route related messages (e.g. all events for one order id) to the same partition using a partition key, which preserves order within that keycOrdering is only a concern for SQL databasesdAdding more consumers always fixes ordering

Q4Duplicate messagesmedium

A consumer processes a message, but crashes right before acknowledging it. What typically happens, and what must the consumer handle?

Asked in

aThe message is permanently lostbThe message is redelivered (since it was never acknowledged), so the consumer may process it a second time — it must be written to handle duplicates safely (idempotency)cThe queue automatically prevents any duplicate processingdThis scenario is impossible in any queue system

Q5Dead-letter queuesmedium

What is a dead-letter queue (DLQ) for?

Asked in

aIt stores messages that a consumer repeatedly fails to process, so they can be inspected later instead of blocking or endlessly retrying the main queuebIt's a queue that has been permanently shut downcIt's used only for encrypting messagesdIt automatically fixes broken messages

FAQ

Next: Lesson 8 — CDN & Object Storage →