kafka-python API

The kafka package exposes a small set of public classes for working with Apache Kafka. The following sections group them by role; each class links to its own API reference page.

Application code typically interacts with one of three top-level clients. Each owns a background IO thread and a shared async networking layer. The sections below group each client with the data types and extension points specific to it.

kafka.consumer

The high-level, group-aware message consumer and the data types it produces.

  • KafkaConsumer - high-level, group-aware message consumer. Iterable, with manual or automatic offset commits, cooperative rebalance, pluggable deserializers, and transactional-read isolation.

  • ConsumerRecord - a single record consumed from a topic partition, as yielded by poll() and consumer iteration.

  • ConsumerRebalanceListener - base class for receiving partition join/revoke callbacks during a group rebalance. Also includes the async interface AsyncConsumerRebalanceListener.

kafka.producer

The high-level, asynchronous message producer and the data types and extension points specific to it.

  • KafkaProducer - high-level, asynchronous message producer. Batches records into a background sender thread, with optional idempotence, transactions, compression, and pluggable serializers.

  • FutureRecordMetadata - asynchronous handle returned by send(); resolves to a RecordMetadata once the record is acknowledged.

  • RecordMetadata - metadata about a produced record after the broker has acknowledged it.

  • Partitioner - base class for pluggable partition selection; controls which partition a record is assigned to.

kafka.admin

Cluster administration operations.

  • KafkaAdminClient - admin operations: topic, ACL, config, consumer group, partition, quota, log-directory, and quorum management.

kafka.net

The clients share a single async networking layer (kafka.net). These classes are exposed for advanced use cases - embedding the connection pool, building a custom client on top of the kafka.net event loop, or driving the protocol layer directly from the REPL.

  • manager - connection pool and high-level facade over the shared IO event loop. Each top-level client owns one.

  • connection - per-broker async connection: state machine, request/response correlation, and SASL handshake.

  • transport - Async socket I/O with write buffering, pause/resume hooks, and the asyncio-shaped protocol callback surface.

  • inet - DNS lookup + non-blocking connect, plus a URL-scheme registry that resolves proxy_url to socket factories.

  • http_connect - Tunnels broker connections through an HTTP CONNECT proxy (RFC 7231).

  • socks5 - SOCKS5 client with optional username/password authentication.

other / misc

Lightweight data types shared across the clients (and useful when working with the lower-level protocol layer).