kafka.coordinator.assignors package

Submodules

kafka.coordinator.assignors.abstract module

class kafka.coordinator.assignors.abstract.AbstractPartitionAssignor

Bases: object

Abstract assignor implementation which does some common grunt work (in particular collecting partition counts which are always needed in assignors).

assign(cluster, members)

Perform group assignment given cluster metadata and member subscriptions

Parameters:

• cluster (ClusterMetadata) – metadata for use in assignment
• (dict of {member_id (members) – MemberMetadata}): decoded metadata for each member in the group.

Returns:

{member_id: MemberAssignment}

Return type:

dict

metadata(topics)

Generate ProtocolMetadata to be submitted via JoinGroupRequest.

Parameters:topics (set) – a member’s subscribed topics Returns:MemberMetadata struct

name

.name should be a string identifying the assignor

on_assignment(assignment)

Callback that runs on each assignment.

This method can be used to update internal state, if any, of the partition assignor.

Parameters:assignment (MemberAssignment) – the member’s assignment

kafka.coordinator.assignors.roundrobin module

class kafka.coordinator.assignors.roundrobin.RoundRobinPartitionAssignor

Bases: kafka.coordinator.assignors.abstract.AbstractPartitionAssignor

The roundrobin assignor lays out all the available partitions and all the available consumers. It then proceeds to do a roundrobin assignment from partition to consumer. If the subscriptions of all consumer instances are identical, then the partitions will be uniformly distributed. (i.e., the partition ownership counts will be within a delta of exactly one across all consumers.)

For example, suppose there are two consumers C0 and C1, two topics t0 and t1, and each topic has 3 partitions, resulting in partitions t0p0, t0p1, t0p2, t1p0, t1p1, and t1p2.

The assignment will be:

C0: [t0p0, t0p2, t1p1] C1: [t0p1, t1p0, t1p2]

When subscriptions differ across consumer instances, the assignment process still considers each consumer instance in round robin fashion but skips over an instance if it is not subscribed to the topic. Unlike the case when subscriptions are identical, this can result in imbalanced assignments.

For example, suppose we have three consumers C0, C1, C2, and three topics t0, t1, t2, with unbalanced partitions t0p0, t1p0, t1p1, t2p0, t2p1, t2p2, where C0 is subscribed to t0; C1 is subscribed to t0, t1; and C2 is subscribed to t0, t1, t2.

The assignment will be:

C0: [t0p0] C1: [t1p0] C2: [t1p1, t2p0, t2p1, t2p2]

classmethod assign(cluster, member_metadata)

classmethod metadata(topics)

name = 'roundrobin'

classmethod on_assignment(assignment)

version = 0

Module contents