kafka-python API

The kafka package exposes a small set of public classes for working with Apache Kafka. The following sections group them by role; each class links to its own API reference page.

kafka

Application code typically interacts with one of three top-level clients. Each owns a background IO thread and a shared async networking layer.

  • KafkaConsumer - high-level, group-aware message consumer. Iterable, with manual or automatic offset commits, cooperative rebalance, pluggable deserializers, and transactional-read isolation.

  • KafkaProducer - high-level, asynchronous message producer. Batches records into a background sender thread, with optional idempotence, transactions, compression, and pluggable serializers.

  • KafkaAdminClient - admin operations: topic, ACL, config, consumer group, partition, quota, log-directory, and quorum management.

kafka.net

The clients share a single async networking layer (kafka.net). These classes are exposed for advanced use cases - embedding the connection pool, building a custom client on top of the kafka.net event loop, or driving the protocol layer directly from the REPL.

  • manager - connection pool and high-level facade over the shared IO event loop. Each top-level client owns one.

  • connection - per-broker async connection: state machine, request/response correlation, and SASL handshake.

  • transport - Async socket I/O with write buffering, pause/resume hooks, and the asyncio-shaped protocol callback surface.

  • inet - DNS lookup + non-blocking connect, plus a URL-scheme registry that resolves proxy_url to socket factories.

  • http_connect - Tunnels broker connections through an HTTP CONNECT proxy (RFC 7231).

  • socks5 - SOCKS5 client with optional username/password authentication.

other / misc

Lightweight data types used throughout the client APIs (and useful when working with the lower-level protocol layer).

  • ClusterMetadata - in-memory cache of brokers, topics, partitions, and the active controller. Refreshes itself on the shared IO thread.

  • TopicPartition - namedtuple identifying a partition as (topic, partition).

  • OffsetAndMetadata - committed-offset record (offset, metadata, leader_epoch).

  • OffsetSpec - enum for partition offset queries.

  • IsolationLevel - enum for transactional isolation.

  • Serializer - base class for serialization / deserialization of key and value bytes. Includes helper classes ~kafka.DefaultSerializer and ~kafka.JsonSerializer.

  • ConsumerRebalanceListener - base class for consumer class to receive join/rebalance group hooks. Also includes async interface ~kafka.AsyncConsumerRebalanceListener.