03.3 Message queues & async processing

Junior · Component

Core idea

A message queue decouples the producer (who creates work) from the consumer (who does the work). This enables async processing, load leveling, and fault isolation. The producer doesn’t wait for the work to complete.

Mental model

A restaurant: customers (producers) place orders, the queue is the order tickets on the rail, chefs (consumers) process tickets at their own pace. During lunch rush, tickets pile up but nobody waits for a chef to finish before ordering. Remove a chef → orders slow but don’t fail. Add a chef → tickets clear faster.

Why it matters

  • Synchronous calls create tight coupling — if downstream fails, upstream fails
  • Queues absorb traffic spikes — consumers work at their capacity, not the burst rate
  • Enables retry logic, dead-letter queues, and replay — robust error handling

What to know

  • Point-to-point (SQS): one producer, one consumer group — good for task queues
  • Pub/sub (Kafka, SNS): one producer, many consumer groups — good for event streaming
  • At-least-once delivery: message may be delivered more than once → consumers must be idempotent
  • Exactly-once: harder guarantee, often with deduplication overhead
  • Kafka: log-based, persistent, consumers pull at own pace, replay past events — high throughput
  • Dead-letter queue: failed messages moved here after N retries — prevents infinite retry loops
  • Backpressure: consumers signal producers to slow down — prevents queue overflow

Common traps

  • Not making consumers idempotent — duplicate processing corrupts data
  • Queue becomes a bottleneck if it’s a single point — partition it (Kafka partitions)
  • Ignoring dead-letter queue — failed messages silently disappear

Recognise it

Recognition trigger: Two services communicate + you don’t need immediate response = async + queue. Email sending, image processing, payment notifications, analytics events — all should be async. If failure in one service should not fail the other → queue between them.

Practice question

A user uploads a video. We need to transcode it to 4 resolutions. How do you architect this? Why not do it synchronously?

Apply at work

Answer these for the system you work on.

Go deeper