kafka.partitioner package

Submodules

kafka.partitioner.base module

class kafka.partitioner.base.Partitioner(partitions=None)

Bases: object

Base class for a partitioner

__call__(key, all_partitions=None, available_partitions=None)

Takes a string key, num_partitions and available_partitions as argument and returns a partition to be used for the message

Parameters:

• key – the key to use for partitioning.
• all_partitions – a list of the topic’s partitions.
• available_partitions – a list of the broker’s currently avaliable partitions(optional).

kafka.partitioner.hashed module

kafka.partitioner.hashed.HashedPartitioner

alias of LegacyPartitioner

class kafka.partitioner.hashed.LegacyPartitioner(partitions)

Bases: object

DEPRECATED – See Issue 374

Implements a partitioner which selects the target partition based on the hash of the key

partition(key, partitions=None)

class kafka.partitioner.hashed.Murmur2Partitioner(partitions=None)

Bases: kafka.partitioner.base.Partitioner

Implements a partitioner which selects the target partition based on the hash of the key. Attempts to apply the same hashing function as mainline java client.

partition(key, partitions=None)

kafka.partitioner.hashed.murmur2(data)

Pure-python Murmur2 implementation.

Based on java client, see org.apache.kafka.common.utils.Utils.murmur2

Parameters:data (bytes) – opaque bytes

Returns: MurmurHash2 of data

kafka.partitioner.roundrobin module

class kafka.partitioner.roundrobin.CachedPartitionCycler(partitions=None)

Bases: object

next()

set_partitions(partitions)

class kafka.partitioner.roundrobin.RoundRobinPartitioner(partitions=None)

Bases: kafka.partitioner.base.Partitioner

partition(key, all_partitions=None, available_partitions=None)

Module contents

class kafka.partitioner.DefaultPartitioner

Bases: object

Default partitioner.

Hashes key to partition using murmur2 hashing (from java client) If key is None, selects partition randomly from available, or from all partitions if none are currently available

classmethod __call__(key, all_partitions, available)

Get the partition corresponding to key :param key: partitioning key :param all_partitions: list of all partitions sorted by partition ID :param available: list of available partitions in no particular order :return: one of the values from all_partitions or available

class kafka.partitioner.RoundRobinPartitioner(partitions=None)

Bases: kafka.partitioner.base.Partitioner

partition(key, all_partitions=None, available_partitions=None)

kafka.partitioner.HashedPartitioner

alias of LegacyPartitioner

class kafka.partitioner.Murmur2Partitioner(partitions=None)

Bases: kafka.partitioner.base.Partitioner

Implements a partitioner which selects the target partition based on the hash of the key. Attempts to apply the same hashing function as mainline java client.

partition(key, partitions=None)

class kafka.partitioner.LegacyPartitioner(partitions)

Bases: object

DEPRECATED – See Issue 374

Implements a partitioner which selects the target partition based on the hash of the key

partition(key, partitions=None)