Skip to main content

Kafkit helps you write Kafka producers and consumers in Python with asyncio.

Project description

Kafkit

Kafkit helps you write Kafka producers and consumers in Python with asyncio:

  • Kafkit provides a client for the Confluent Schema Registry’s HTTP API. The RegistryApi client includes both high-level methods for managing subjects and schemas in a Registry, and direct low-level access to HTTP methods (GET, POST, PUT, PATCH, and DELETE). The high-level methods use caching so you can use the client as an integral part of your application’s schema management. RegistryApi is implemented around aiohttp, but since the base class is designed with a sans IO architecture, a Registry client can be implemented with any asyncio HTTP library.

  • Kafkit provides Avro message serializers and deserializers that integrate with the Confluent Schema Registry: Deserializer, Serializer, and PolySerializer.

  • The RecordNameSchemaManager is a streamlined tool for serializing messages using the schemas maintained by your app, while also integrating with the Confluent Schema Registry.

  • The kafkit.ssl module helps you connect to SSL-secured Kafka brokers.

Learn more about Kafkit at https://kafkit.lsst.io.

Change log

0.2.0 (unreleased)

  • Python versions 3.6 and earlier are no longer supported because Kafkit is adopting the annotations import from __future__. Kafkit is explicitly tested with Python 3.7 and 3.8.

  • We’ve added a kafkit.ssl module to help connect to Kafka brokers over TLS. The associated documentation includes a tutorial for working with the SSL certificates generated in a Kafka cluster managed by Strimzi.

  • The brand-new kafkit.registry.manager.RecordNameSchemaManager provides a streamlined workflow for serializing Avro messages using Avro schemas that are maintained in your app’s codebase. The manager handles schema registration for you. To serialize a message, you simply need to provide the data and the name of the schema.

  • A new kafkit.registry.sansio.CompatibilityType Enum helps you write use valid Schema Registry compatibility types.

  • We’ve significantly improved Kafkit’s packaging and infrastructure:

    • Migrate packaging metadata from setup.py to setup.cfg.

    • Support PEP 518 with pyproject.toml (Kafkit continues to be a setuptools-based project).

    • Tox now runs tasks like tests, in conjunction with the existing Pytest set up.

    • Pre-commit hooks lint and format the code base.

    • Code style is now handled by Black (and in the documentation with blacken-docs).

    • Full support for type annotations! tox -e typing validates Kafkit’s type annotations with Mypy.

    • We’ve migrated from Travis CI to GitHub Actions.

  • The documentation now includes a development guide.

0.1.1 (2019-02-13)

Several fixes:

  • RegistryApi.put was doing a PATCH behind the scenes. That’s fixed now.

  • The RegistryApi.put, patch, and delete methods weren’t returning data. That’s fixed now as well.

  • All of the RegistryApi’s low-level HTTP methods have more thorough unit testing now to avoid these issues in the future.

0.1.0 (2019-01-30)

Initial release of Kafkit!

This release includes the kafkit.registry package with a working Confluent Schema Registry API client implemented with a sans I/O design. There are two client implementations. One is designed for aiohttp users (kafkit.registry.aiohttp.RegistryClient), and the other is for I/O-free unit testing (kafkit.registry.sansio.MockRegistryApi). The clients include schema caches so they can be used as both local stores of schemas, in addition to accessors for remote schemas. The release also includes a suite of Avro message serializers and deserializers that integrate with Confluent Schema Registry and the Confluent Wire Format (kafkit.registry.serializer).

MIT License

Copyright (c) 2019-2020 Association of Universities for Research in Astronomy (AURA), Inc.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

kafkit-0.2.0b2.tar.gz (44.7 kB view details)

Uploaded Source

Built Distribution

kafkit-0.2.0b2-py3-none-any.whl (22.1 kB view details)

Uploaded Python 3

File details

Details for the file kafkit-0.2.0b2.tar.gz.

File metadata

  • Download URL: kafkit-0.2.0b2.tar.gz
  • Upload date:
  • Size: 44.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.44.1 CPython/3.7.6

File hashes

Hashes for kafkit-0.2.0b2.tar.gz
Algorithm Hash digest
SHA256 7dbacbbcaa1e207ac65453ed5f83a895cbde97d7ef25be6e00e2312a8f6cc5d3
MD5 8a90f85170d693d4981fdcc702b1f0b9
BLAKE2b-256 621d2fbb9a42923bbd8fc14b48e6b6ecf783d95bfe7d7e87d3833d80bbaa7b8d

See more details on using hashes here.

File details

Details for the file kafkit-0.2.0b2-py3-none-any.whl.

File metadata

  • Download URL: kafkit-0.2.0b2-py3-none-any.whl
  • Upload date:
  • Size: 22.1 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.44.1 CPython/3.7.6

File hashes

Hashes for kafkit-0.2.0b2-py3-none-any.whl
Algorithm Hash digest
SHA256 25b8a8da3ca6c4461408ec3aa1a262a916cffde47f7620746bb71016a787e756
MD5 0f6536b9944989bedc9ec34580172d4f
BLAKE2b-256 70217cf2bd1a36a177a04448d61faf380628cfc183d4edcd44112a6760e0423f

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page