Humilis kinesis stream processor plugin
Project description
Kinesis event stream processor
===================================
.. |Build Status| image:: https://travis-ci.org/humilis/humilis-kinesis-processor.svg?branch=master
:target: https://travis-ci.org/humilis/humilis-kinesis-processor
.. |PyPI| image:: https://img.shields.io/pypi/v/humilis-kinesis-processor.svg?style=flat
:target: https://pypi-hypernode.com/pypi/humilis-kinesis-processor
|Build Status| |PyPI|
A `humilis <https://github.com/humilis/humilis>`__ plugin to deploy a
`Lambda <https://aws.amazon.com/documentation/lambda/>`__ function that
maps events in a `Kinesis <https://aws.amazon.com/documentation/kinesis/>`__
event stream using a list of Python callables.
Installation
------------
::
pip install humilis-kinesis-processor
Development
-----------
Assuming you have
`virtualenv <https://virtualenv.readthedocs.org/en/latest/>`__ installed:
::
make develop
Configure humilis::
.env/bin/humilis configure --local
Testing
-------
Unit tests
~~~~~~~~~~
To run the local test suite::
make test
Integration tests
~~~~~~~~~~~~~~~~~
Before running the integration test suite you need to set a few deployment
secrets using the command::
s3keyring set [group]/[STAGE] [key] [secret]
In group ``humilis-kinesis-processor`` the following secrets need to be set:
* ``sentry/dsn``: The `Sentry DSN <https://docs.getsentry.com/hosted/quickstart/#configure-the-dsn>`__.
By the default, the integration tests will deploy on a stage called ``DEV`` so
the command to set the Sentry DSN is::
s3keyring set humilis-kinesis-processor:DEV sentry.dsn [SENTRYDSN]
To run the integration test suite::
make testi
The command above will deploy a Kinesis processor to your AWS account, and will
also create additional resources (such as several Kinesis streams) needed to
test that the deployment was successful. Once deployed, the integration tests
will run, and once they have completed the test environment will be destroyed.
If you do not want the test environment to be destroyed after tests have
completed you should run instead::
make testi DESTROY=no
You can also modify the name of the deployment stage by setting the ``STAGE``
environment variable. For instance, to deploy to a ``TEST`` stage::
make testi STAGE=TEST
More information
----------------
See `humilis <https://github.com/humilis/humilis>`__ documentation.
Contact
-------
If you have questions, bug reports, suggestions, etc. please create an issue on
the `GitHub project page <http://github.com/humilis/humilis-kinesis-processor>`_.
License
-------
This software is licensed under the `MIT license <http://en.wikipedia.org/wiki/MIT_License>`_
See `License file <https://github.com/humilis/humilis-kinesis-processor/blob/master/LICENSE.txt>`_
© 2016 Anatoly Bubenkov, German Gomez-Herrero, `FindHotel <http://company.findhotel.net>`_ and others.
Authors
=======
`Anatoly Bubenkov <bubenkoff@gmail.com>`_
initial idea and first alpha implementation
* `German Gomez-Herrero <german.gomezherrero@gmail.com>`_
actual implementation and further developments
These people have contributed to `humilis-kinesis-mapper`, in alphabetical order:
Changelog
=========
0.7.7
-----
- Support for multiple delivery streams for input, error, and output(s)
0.7.6
-----
- Support for one-to-many mappers
0.7.5
-----
- Users can also specify input/output/error streams by name instead of as
references to another layer outputs.
0.6.8
-----
- Set environment variables for the humilis environment/layer/stage
0.4.0
-----
- Major bugfix: permission to put records in delivery streams
0.3.0
-----
- Major bugfix: give permission to write state to DynamodDB
0.2.0
-----
- Package becomes a generic map-multiplex-map processor (germangh)
0.1.1
-----
- Cleanup package data (bubenkoff)
0.1.0
-----
- Major refactoring to simplify and clean the code (germangh)
0.0.1
-----
- Initial release (bubenkoff)
===================================
.. |Build Status| image:: https://travis-ci.org/humilis/humilis-kinesis-processor.svg?branch=master
:target: https://travis-ci.org/humilis/humilis-kinesis-processor
.. |PyPI| image:: https://img.shields.io/pypi/v/humilis-kinesis-processor.svg?style=flat
:target: https://pypi-hypernode.com/pypi/humilis-kinesis-processor
|Build Status| |PyPI|
A `humilis <https://github.com/humilis/humilis>`__ plugin to deploy a
`Lambda <https://aws.amazon.com/documentation/lambda/>`__ function that
maps events in a `Kinesis <https://aws.amazon.com/documentation/kinesis/>`__
event stream using a list of Python callables.
Installation
------------
::
pip install humilis-kinesis-processor
Development
-----------
Assuming you have
`virtualenv <https://virtualenv.readthedocs.org/en/latest/>`__ installed:
::
make develop
Configure humilis::
.env/bin/humilis configure --local
Testing
-------
Unit tests
~~~~~~~~~~
To run the local test suite::
make test
Integration tests
~~~~~~~~~~~~~~~~~
Before running the integration test suite you need to set a few deployment
secrets using the command::
s3keyring set [group]/[STAGE] [key] [secret]
In group ``humilis-kinesis-processor`` the following secrets need to be set:
* ``sentry/dsn``: The `Sentry DSN <https://docs.getsentry.com/hosted/quickstart/#configure-the-dsn>`__.
By the default, the integration tests will deploy on a stage called ``DEV`` so
the command to set the Sentry DSN is::
s3keyring set humilis-kinesis-processor:DEV sentry.dsn [SENTRYDSN]
To run the integration test suite::
make testi
The command above will deploy a Kinesis processor to your AWS account, and will
also create additional resources (such as several Kinesis streams) needed to
test that the deployment was successful. Once deployed, the integration tests
will run, and once they have completed the test environment will be destroyed.
If you do not want the test environment to be destroyed after tests have
completed you should run instead::
make testi DESTROY=no
You can also modify the name of the deployment stage by setting the ``STAGE``
environment variable. For instance, to deploy to a ``TEST`` stage::
make testi STAGE=TEST
More information
----------------
See `humilis <https://github.com/humilis/humilis>`__ documentation.
Contact
-------
If you have questions, bug reports, suggestions, etc. please create an issue on
the `GitHub project page <http://github.com/humilis/humilis-kinesis-processor>`_.
License
-------
This software is licensed under the `MIT license <http://en.wikipedia.org/wiki/MIT_License>`_
See `License file <https://github.com/humilis/humilis-kinesis-processor/blob/master/LICENSE.txt>`_
© 2016 Anatoly Bubenkov, German Gomez-Herrero, `FindHotel <http://company.findhotel.net>`_ and others.
Authors
=======
`Anatoly Bubenkov <bubenkoff@gmail.com>`_
initial idea and first alpha implementation
* `German Gomez-Herrero <german.gomezherrero@gmail.com>`_
actual implementation and further developments
These people have contributed to `humilis-kinesis-mapper`, in alphabetical order:
Changelog
=========
0.7.7
-----
- Support for multiple delivery streams for input, error, and output(s)
0.7.6
-----
- Support for one-to-many mappers
0.7.5
-----
- Users can also specify input/output/error streams by name instead of as
references to another layer outputs.
0.6.8
-----
- Set environment variables for the humilis environment/layer/stage
0.4.0
-----
- Major bugfix: permission to put records in delivery streams
0.3.0
-----
- Major bugfix: give permission to write state to DynamodDB
0.2.0
-----
- Package becomes a generic map-multiplex-map processor (germangh)
0.1.1
-----
- Cleanup package data (bubenkoff)
0.1.0
-----
- Major refactoring to simplify and clean the code (germangh)
0.0.1
-----
- Initial release (bubenkoff)
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
File details
Details for the file humilis-kinesis-processor-0.8.2.tar.gz.
File metadata
- Download URL: humilis-kinesis-processor-0.8.2.tar.gz
- Upload date:
- Size: 10.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 0806cba6b36706a205811258cdf208015dddf4623d460feef265827037c60339 |
|
| MD5 | b4cbaa7e5edcf760f4fcacbf3a261564 |
|
| BLAKE2b-256 | 3adb49d1ba02fa34599eeb616d2f19643c9c743162379149cb03d4e389716120 |
