Kafka integration with asyncio

Navigation

Verified details

These details have been verified by PyPI
Owner
organization icon aio-libs
Maintainers
Avatar for Andrew.Svetlov from gravatar.com Andrew.Svetlov Avatar for otkds from gravatar.com otkds Avatar for tvoinarovskyi from gravatar.com tvoinarovskyi

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: Apache Software License ( Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR ...)
  • Author: Andrew Svetlov
  • Requires: Python >=3.8
Classifiers

Project description

|Build status| |Coverage| |Chat on Gitter|

asyncio client for Kafka

AIOKafkaProducer

AIOKafkaProducer is a high-level, asynchronous message producer.

Example of AIOKafkaProducer usage:

from aiokafka import AIOKafkaProducer
import asyncio

async def send_one():
    producer = AIOKafkaProducer(bootstrap_servers='localhost:9092')
    # Get cluster layout and initial topic/partition leadership information
    await producer.start()
    try:
        # Produce message
        await producer.send_and_wait("my_topic", b"Super message")
    finally:
        # Wait for all pending messages to be delivered or expire.
        await producer.stop()

asyncio.run(send_one())

AIOKafkaConsumer

AIOKafkaConsumer is a high-level, asynchronous message consumer. It interacts with the assigned Kafka Group Coordinator node to allow multiple consumers to load balance consumption of topics (requires kafka >= 0.9.0.0).

Example of AIOKafkaConsumer usage:

from aiokafka import AIOKafkaConsumer
import asyncio

async def consume():
    consumer = AIOKafkaConsumer(
        'my_topic', 'my_other_topic',
        bootstrap_servers='localhost:9092',
        group_id="my-group")
    # Get cluster layout and join group `my-group`
    await consumer.start()
    try:
        # Consume messages
        async for msg in consumer:
            print("consumed: ", msg.topic, msg.partition, msg.offset,
                  msg.key, msg.value, msg.timestamp)
    finally:
        # Will leave consumer group; perform autocommit if enabled.
        await consumer.stop()

asyncio.run(consume())

Running tests

Docker is required to run tests. See https://docs.docker.com/engine/installation for installation notes. Also note, that lz4 compression libraries for python will require python-dev package, or python source header files for compilation on Linux. NOTE: You will also need a valid java installation. It’s required for the keytool utility, used to generate ssh keys for some tests.

Setting up tests requirements (assuming you’re within virtualenv on ubuntu 14.04+):

sudo apt-get install -y libsnappy-dev libzstd-dev libkrb5-dev krb5-user
make setup

Running tests with coverage:

make cov

To run tests with a specific version of Kafka (default one is 1.0.2) use KAFKA_VERSION variable:

make cov KAFKA_VERSION=0.10.2.1

Test running cheatsheat:

  • make test FLAGS="-l -x --ff" - run until 1 failure, rerun failed tests first. Great for cleaning up a lot of errors, say after a big refactor.

  • make test FLAGS="-k consumer" - run only the consumer tests.

  • make test FLAGS="-m 'not ssl'" - run tests excluding ssl.

  • make test FLAGS="--no-pull" - do not try to pull new docker image before test run.

Project details

Verified details

These details have been verified by PyPI
Owner
organization icon aio-libs
Maintainers
Avatar for Andrew.Svetlov from gravatar.com Andrew.Svetlov Avatar for otkds from gravatar.com otkds Avatar for tvoinarovskyi from gravatar.com tvoinarovskyi

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: Apache Software License ( Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR ...)
  • Author: Andrew Svetlov
  • Requires: Python >=3.8
Classifiers

Release history Release notifications | RSS feed

0.12.0

0.11.0

0.10.0

0.10.0a0 pre-release

0.9.0

This version

0.9.0rc1 pre-release

0.9.0rc0 pre-release

0.8.1

0.8.0

0.7.2

0.7.1

0.7.0

0.6.0

0.5.2

0.5.1

0.5.0

0.4.3

0.4.2

0.4.2.dev0 pre-release

0.4.1

0.4.0

0.3.1

0.3.0

0.2.3

0.2.2

0.2.1

0.2.0

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.1

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

aiokafka-0.9.0rc1.tar.gz (557.6 kB view hashes)

Uploaded Source

Built Distributions

aiokafka-0.9.0rc1-cp312-cp312-win_amd64.whl (355.0 kB view hashes)

Uploaded CPython 3.12 Windows x86-64

aiokafka-0.9.0rc1-cp312-cp312-win32.whl (336.7 kB view hashes)

Uploaded CPython 3.12 Windows x86

aiokafka-0.9.0rc1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB view hashes)

Uploaded CPython 3.12 manylinux: glibc 2.17+ x86-64

aiokafka-0.9.0rc1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.2 MB view hashes)

Uploaded CPython 3.12 manylinux: glibc 2.17+ ARM64

aiokafka-0.9.0rc1-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl (1.2 MB view hashes)

Uploaded CPython 3.12 manylinux: glibc 2.17+ i686 manylinux: glibc 2.5+ i686

aiokafka-0.9.0rc1-cp312-cp312-macosx_10_9_x86_64.whl (363.5 kB view hashes)

Uploaded CPython 3.12 macOS 10.9+ x86-64

aiokafka-0.9.0rc1-cp311-cp311-win_amd64.whl (357.5 kB view hashes)

Uploaded CPython 3.11 Windows x86-64

aiokafka-0.9.0rc1-cp311-cp311-win32.whl (337.6 kB view hashes)

Uploaded CPython 3.11 Windows x86

aiokafka-0.9.0rc1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

aiokafka-0.9.0rc1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.1 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ARM64

aiokafka-0.9.0rc1-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl (1.1 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ i686 manylinux: glibc 2.5+ i686

aiokafka-0.9.0rc1-cp311-cp311-macosx_10_9_x86_64.whl (365.9 kB view hashes)

Uploaded CPython 3.11 macOS 10.9+ x86-64

aiokafka-0.9.0rc1-cp310-cp310-win_amd64.whl (357.6 kB view hashes)

Uploaded CPython 3.10 Windows x86-64

aiokafka-0.9.0rc1-cp310-cp310-win32.whl (338.6 kB view hashes)

Uploaded CPython 3.10 Windows x86

aiokafka-0.9.0rc1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

aiokafka-0.9.0rc1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.1 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

aiokafka-0.9.0rc1-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl (1.0 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ i686 manylinux: glibc 2.5+ i686

aiokafka-0.9.0rc1-cp310-cp310-macosx_10_9_x86_64.whl (365.7 kB view hashes)

Uploaded CPython 3.10 macOS 10.9+ x86-64

aiokafka-0.9.0rc1-cp39-cp39-win_amd64.whl (359.1 kB view hashes)

Uploaded CPython 3.9 Windows x86-64

aiokafka-0.9.0rc1-cp39-cp39-win32.whl (339.9 kB view hashes)

Uploaded CPython 3.9 Windows x86

aiokafka-0.9.0rc1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

aiokafka-0.9.0rc1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.1 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

aiokafka-0.9.0rc1-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl (1.0 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ i686 manylinux: glibc 2.5+ i686

aiokafka-0.9.0rc1-cp39-cp39-macosx_10_9_x86_64.whl (367.0 kB view hashes)

Uploaded CPython 3.9 macOS 10.9+ x86-64

aiokafka-0.9.0rc1-cp38-cp38-win_amd64.whl (359.8 kB view hashes)

Uploaded CPython 3.8 Windows x86-64

aiokafka-0.9.0rc1-cp38-cp38-win32.whl (340.0 kB view hashes)

Uploaded CPython 3.8 Windows x86

aiokafka-0.9.0rc1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

aiokafka-0.9.0rc1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.1 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

aiokafka-0.9.0rc1-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl (1.1 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ i686 manylinux: glibc 2.5+ i686

aiokafka-0.9.0rc1-cp38-cp38-macosx_10_9_x86_64.whl (368.6 kB view hashes)

Uploaded CPython 3.8 macOS 10.9+ x86-64

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page