Conventions for Python + Apache Beam
Project description
The klio Library
The library for implementing Klio-ified Apache Beam transforms with decorators, helper transforms, and leverage Klio’s message-handling logic.
As the klio library is not meant to be installed directly, check out the installation guide for how to setup installation. There is also the user guide and the API documentation for more information.
What is Klio?
Klio is an ecosystem that allows you to process audio files – or any binary files – easily and at scale.
Klio jobs are opinionated data pipelines in Python (streaming or batch) built upon Apache Beam and tuned for audio and binary file processing.
Klio was built by Spotify to run our large-scale audio intelligence systems and is used by teams of engineers and audio researchers to help develop and deploy next generation audio algorithms.
Read more about Klio here.
Get Started
Check out our Quick Start guide on how to start using Klio.
Resources
Have a look at our complete documentation.
Check out our contributors’ guide.
Join the #klio channel in the Spotify FOSS Slack organization.
Read up on some of our FAQs.
Review our Code of Conduct - this is how we roll.
Release Information
21.8.0 (UNRELEASED)
Added
Add KlioReadFromPubSub and KlioWriteToPubSub IO transforms.
Add default metrics to be collected in Klio’s IO transforms.
Add default metrics to be collected in Klio’s helper transforms.
Add default metrics to be collected in Klio’s decorators.
Add support for using Beam’s metrics API directly.
Add DirectGKERunner, runs direct runner on GKE with added logic to only ack pub/sub message when: * the pipeline successfully ran, but before any write to event output happens if any, * the message is skipped because output data already exists, * the message is dropped because input data does not exist, or * the message is dropped because it was not the intended recipient.
Add metrics interface for shumway
Fixed
KlioTriggerUpstream no longer raises a pickling error when trying to log.
Changed
The Beam metrics client will always be used, no matter the configured runner.
Marked Klio’s support for Stackdriver log-based metrics for deprecation and eventual removal.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file klio-21.8.0rc1.tar.gz.
File metadata
- Download URL: klio-21.8.0rc1.tar.gz
- Upload date:
- Size: 74.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.2 CPython/3.6.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 17c8991d91d45f7f89f7eb3be0786ece1f0c28961ab14b2957e278241dba32bb |
|
| MD5 | 905ebd68eaef8c4b30b37d6d6340dfec |
|
| BLAKE2b-256 | ad8d049981867fe27d8c68c161ec6d8a37a9f04d4d17c6d9af8c768d23248d54 |
File details
Details for the file klio-21.8.0rc1-py2.py3-none-any.whl.
File metadata
- Download URL: klio-21.8.0rc1-py2.py3-none-any.whl
- Upload date:
- Size: 69.4 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.2 CPython/3.6.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 145164762622f57ee0e260a24254a2c4d553c08b93503b54d2f210f10cb36f11 |
|
| MD5 | a7b356678eeaa546c382baa603493cfe |
|
| BLAKE2b-256 | df2084359baf525dcdb7f390eb4f4552a1d2cc0430f3cd14b3817754fc32b11e |
