Kafka Consumer Groups in Java, Explained Clearly

Consumer groups are the single idea that makes Kafka's read side make sense. Once you understand how a group maps consumers to partitions, offsets and rebalancing stop being mysterious. Here is the model.

A Group Shares the Partitions of a Topic

A topic is split into partitions. A consumer group is a set of consumers that cooperate to read one topic, and Kafka guarantees that each partition is read by exactly one consumer in the group at a time. That single rule drives everything else.

If a topic has 6 partitions and your group has 3 consumers, each consumer gets 2 partitions. Add a fourth consumer, and the load redistributes to 2/2/1/1:

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "order-processors"); // the group
props.put("key.deserializer", StringDeserializer.class.getName());
props.put("value.deserializer", StringDeserializer.class.getName());

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(List.of("orders"));

Scaling Stops at the Partition Count

Because a partition maps to at most one consumer per group, you cannot have more useful consumers than partitions. A group of 10 consumers on a 6-partition topic leaves 4 consumers idle. Partition count is your ceiling on parallelism — plan it up front.

Offsets Track Progress Per Group

Each group independently tracks how far it has read in each partition, stored as a committed offset. Two different groups reading the same topic each get every message and keep separate progress — that is how you fan out to multiple independent services:

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(200));
    for (ConsumerRecord<String, String> record : records) {
        process(record.value());
    }
    consumer.commitSync(); // advance this group's offsets
}

Rebalancing Redistributes Work

When a consumer joins or leaves (deploy, crash, scale-up), Kafka triggers a rebalance: it pauses consumption briefly and reassigns partitions across the current members. This is how the group heals and rescales automatically — but it also means processing stalls momentarily, so avoid unnecessarily frequent restarts.

Summary

A consumer group shares a topic's partitions so each partition is read by exactly one member, which caps parallelism at the partition count. Offsets are tracked per group, letting independent services each consume the full stream. Rebalancing reassigns partitions automatically as members come and go. Keep those four facts in mind and Kafka's consumer model becomes predictable.