ConsumerRebalanceListener

class kafka.ConsumerRebalanceListener

A callback interface that the user can implement to trigger custom actions when the set of partitions assigned to the consumer changes.

This is applicable when the consumer is having Kafka auto-manage group membership. If the consumer’s directly assign partitions, those partitions will never be reassigned and this callback is not applicable.

When Kafka is managing the group membership, a partition re-assignment will be triggered any time the members of the group changes or the subscription of the members changes. This can occur when processes die, new process instances are added or old instances come back to life after failure. Rebalances can also be triggered by changes affecting the subscribed topics (e.g. when then number of partitions is administratively adjusted).

There are many uses for this functionality. One common use is saving offsets in a custom store. By saving offsets in the on_partitions_revoked(), call we can ensure that any time partition assignment changes the offset gets saved.

Another use is flushing out any kind of cache of intermediate results the consumer may be keeping. For example, consider a case where the consumer is subscribed to a topic containing user page views, and the goal is to count the number of page views per users for each five minute window. Let’s say the topic is partitioned by the user id so that all events for a particular user will go to a single consumer instance. The consumer can keep in memory a running tally of actions per user and only flush these out to a remote data store when its cache gets too big. However if a partition is reassigned it may want to automatically trigger a flush of this cache, before the new owner takes over consumption.

Threading: callbacks run on the consumer’s IO event loop, the same loop that drives heartbeats. Sync listener methods must return promptly – blocking IO inside a sync listener will block heartbeats for the duration and can cause the consumer to be kicked from the group if the listener runs longer than session_timeout_ms. For listeners that need to do blocking work (e.g. flushing state to a database), prefer AsyncConsumerRebalanceListener, which lets you await while keeping the loop responsive, or wrap the blocking call in your own worker thread.

It is guaranteed that all consumer processes will invoke on_partitions_revoked() prior to any process invoking on_partitions_assigned(). So if offsets or other state is saved in the on_partitions_revoked() call, it should be saved by the time the process taking over that partition has their on_partitions_assigned() callback called to load the state.

abstract on_partitions_assigned(assigned)

A callback method the user can implement to provide handling of customized offsets on completion of a successful partition re-assignment. This method will be called after an offset re-assignment completes and before the consumer starts fetching data.

It is guaranteed that all the processes in a consumer group will execute their on_partitions_revoked() callback before any instance executes its on_partitions_assigned() callback.

Parameters:

assigned (list of TopicPartition) – the partitions assigned to the consumer (may include partitions that were previously assigned)

on_partitions_lost(lost)

KIP-429: called when the consumer has been forcibly removed from the group (heartbeat session expiry, UnknownMemberIdError, IllegalGenerationError, FencedInstanceIdError) and the partitions cannot be cleanly committed. on_partitions_revoked implies the user can still commit; on_partitions_lost makes explicit that the member has been booted and any in-flight state for these partitions should be discarded.

Default behaviour is to delegate to on_partitions_revoked so listeners written before KIP-429 keep working unchanged. Override for cleanup that is specific to the forced-eviction case (e.g. skipping a commit attempt that will fail anyway).

Parameters:

lost (set of TopicPartition) – the partitions that were assigned but have been lost due to forced eviction.

abstract on_partitions_revoked(revoked)

A callback method the user can implement to provide handling of offset commits to a customized store on the start of a rebalance operation. This method will be called before a rebalance operation starts and after the consumer stops fetching data. It is recommended that offsets should be committed in this callback to either Kafka or a custom offset store to prevent duplicate data.

NOTE: This method is called before each rebalance and also when the consumer is closing (KafkaConsumer.close()), so that offsets / state can be committed before the partitions are given up. If the group membership has already been lost (forced eviction), on_partitions_lost() is called instead.

Parameters:

revoked (list of TopicPartition) – the partitions that were assigned to the consumer on the last rebalance

class kafka.AsyncConsumerRebalanceListener

Async variant of ConsumerRebalanceListener.

Implement this when your rebalance hooks need to perform IO that would otherwise block the consumer’s event loop – e.g. flushing state to a database, calling an external service, or coordinating with other async code. The coordinator detects coroutine functions and await s them instead of calling inline, so other tasks on the loop (notably the heartbeat coroutine) continue to run while your listener is suspended.

Same lifecycle and ordering guarantees as the sync listener: all consumers in the group invoke on_partitions_revoked before any invokes on_partitions_assigned. Both methods must be defined as async def; otherwise use ConsumerRebalanceListener.

abstract async on_partitions_assigned(assigned)

Async-callback for the completion of a partition re-assignment.

See ConsumerRebalanceListener.on_partitions_assigned() for semantics.

Parameters:

assigned (set of TopicPartition) – the partitions assigned to the consumer (may include partitions that were previously assigned).

async on_partitions_lost(lost)

Async variant of ConsumerRebalanceListener.on_partitions_lost(). Default implementation awaits on_partitions_revoked for backward compatibility with listeners written before KIP-429.

Parameters:

lost (set of TopicPartition) – the partitions that were assigned but have been lost due to forced eviction.

abstract async on_partitions_revoked(revoked)

Async-callback for the start of a rebalance operation.

See ConsumerRebalanceListener.on_partitions_revoked() for semantics. The coordinator awaits this method, so non-blocking IO via await keeps the heartbeat loop responsive during the call.

Parameters:

revoked (set of TopicPartition) – the partitions that were assigned to the consumer on the last rebalance.