Kafka Consumer Lag Keeps Growing

Your Kafka dashboard shows consumer lag climbing steadily. Messages are piling up in the topic faster than your consumer group can process them. Left alone, latency grows without bound and eventually you breach retention and lose data. Here is how to diagnose and fix it.

What Lag Means

Lag is the difference between the latest offset a producer wrote and the offset your consumer group has committed. A flat, small lag is healthy. A steadily rising lag means your consumers cannot keep up with the incoming rate — you have a throughput deficit.

kafka-consumer-groups.sh --bootstrap-server broker:9092 \
  --describe --group orders-processor
# Watch the LAG column per partition

Cause 1: Not Enough Consumers

A topic's parallelism is capped by its partition count. If you have 12 partitions but only 3 consumers, each consumer handles 4 partitions. Add consumers up to the partition count:

partitions = 12  →  scale the consumer group toward 12 members

Beyond the partition count, extra consumers sit idle — so if you are already at parity, you need more partitions, not more consumers.

Cause 2: Slow Per-Message Processing

If each message triggers a slow database write or an external API call, your throughput is bottlenecked downstream. Batch the work:

// Process records in batches instead of one at a time
records.partitions().forEach(partition -> {
    List<Record> batch = records.records(partition);
    repository.saveAll(batch);   // one round-trip, not N
});

Cause 3: Frequent Rebalances

If consumers keep joining and leaving, the group spends its time rebalancing instead of consuming. Tune max.poll.interval.ms and session.timeout.ms so a slow batch does not get the consumer kicked out mid-work.

Summary

Growing lag is a throughput deficit: producers are outrunning consumers. Scale the consumer group toward the partition count, speed up per-message work by batching downstream calls, and stop needless rebalances. If you are already maxed on partitions and still behind, add partitions to unlock more parallelism. Close the deficit and the lag flattens back out.

Kafka Consumer Lag Keeps Growing — Here's Why