Cryptocurrency Exchange Feed Handler

Handles multiple cryptocurrency exchange data feeds and returns normalized and standardized results to client registered callbacks for events like trades, book updates, ticker updates, etc. Utilizes websockets when possible, but can also poll data via REST endpoints if a websocket is not provided.

Installation

pip install cryptofeed

or a safer installation:

python3 -m pip install --user --upgrade cryptofeed

To install Cryptofeed along with all its optional dependencies in one bundle:

pip install cryptofeed[all]

See more options, explanations and Pipenv usage in INSTALL.md.

Examples

Please see the examples for more code samples and the documentation for more information about the library useage. The FAQ contains a few oddities/gotchas as well as answers to common questions.

from cryptofeed import FeedHandler

fh = FeedHandler()

# ticker, trade, and book are user defined functions that
# will be called when ticker, trade and book updates are received
ticker_cb = {TICKER: TickerCallback(ticker)}
trade_cb = {TRADES: TradeCallback(trade)}
gemini_cb = {TRADES: TradeCallback(trade), L2_BOOK: BookCallback(book)}


fh.add_feed(Coinbase(pairs=['BTC-USD'], channels=[TICKER], callbacks=ticker_cb))
fh.add_feed(Bitfinex(pairs=['BTC-USD'], channels=[TICKER], callbacks=ticker_cb))
fh.add_feed(Poloniex(pairs=['BTC-USDT'], channels=[TRADES], callbacks=trade_cb))
fh.add_feed(Gemini(pairs=['BTC-USD', 'ETH-USD'], channels=[TRADES, L2_BOOK], callbacks=gemini_cb))

fh.run()

To see an example of an application using cryptofeed to aggregate and store cryptocurrency data to a database, please look at Cryptostore.

Supported exchanges

• Binance
• Binance US
• Binance Jersey
• Binance Futures
• Bitcoin.com
• Bitfinex
• BitMax
• BitMEX
• Bitstamp
• Bittrex
• Blockchain
• Bybit
• Coinbase
• Deribit
• EXX
• FTX
• FTX US
• Gemini
• HitBTC
• Huobi
• HuobiDM
• Kraken
• Kraken Futures
• OKCoin
• OKEx
• Poloniex
• Upbit

National Best Bid/Offer (NBBO)

Cryptofeed also provides a synthetic NBBO (National Best Bid/Offer) feed that aggregates the best bids and asks from the user specified feeds.

from cryptofeed.feedhandler import FeedHandler
from cryptofeed.exchanges import Coinbase, Bitfinex, HitBTC


def nbbo_ticker(pair, bid, ask, bid_feed, ask_feed):
    print('Pair: {} Bid: {} Bid Feed: {} Ask: {} Ask Feed: {}'.format(pair,
                                                                      bid,
                                                                      bid_feed,
                                                                      ask,
                                                                      ask_feed))


fh = FeedHandler()
fh.add_nbbo([Coinbase, Bitfinex, HitBTC], ['BTC-USD'], nbbo_ticker)
fh.run()

Supported Channels

Cryptofeed supports the following channels:

• L2_BOOK - Price aggregated sizes. Some exchanges provide the entire depth, some provide a subset.
• L3_BOOK - Price aggregated orders. Like the L2 book, some exchanges may only provide partial depth.
• TRADES - Note this reports the taker's side, even for exchanges that report the maker side
• TICKER
• VOLUME
• FUNDING
• BOOK_DELTA - Subscribed to with L2 or L3 books, receive book deltas rather than the entire book on updates. Full updates will be periodically sent on the L2 or L3 channel. If BOOK_DELTA is enabled, only L2 or L3 book can be enabled, not both. To receive both create two feedhandler objects. Not all exchanges are supported, as some exchanges send complete books on every update.
• *_SWAP (L2/L3 Books, Trades, Ticker) - Swap data on supporting exchanges
• *_FUTURES (L2/L3 Books, Trades, Ticker) - Futures data on supporting exchanges
• OPEN_INTEREST - Open interest data

Backends

Cryptofeed supports backend callbacks that will write directly to storage or other interfaces

Supported Backends:

• Redis (Streams and Sorted Sets)
• Arctic
• ZeroMQ
• UDP Sockets
• TCP Sockets
• Unix Domain Sockets
• InfluxDB (v1 and v2)
• MongoDB
• Kafka
• Elastic Search
• RabbitMQ
• PostgreSQL

Rest API

Cryptofeed supports some REST interfaces for retrieving historical data and placing orders. See the rest package.

Planned Work

Future Feeds

• CEX
• BTCC
• Many more...

REST

Continue to build out rest endpoints and standardize exchange interfaces and data

Additional Callback Methods / Backends

• Pulsar
• More ZMQ improvements/options

Contributing

Issues and PRs are welcomed. If you'd like to discuss ongoing development please join the slack

This wouldn't have been possible with the many contributors! I owe them and all who have contributed in other ways my thanks!

Cryptofeed installation

The Cryptofeed library is intended to be used by Python developers.

Multiple ways to get and use Cryptofeed:

• using Pip, such as pip install cryptofeed
• using Git, such as git clone https://github.com/bmoscon/cryptofeed
• source code in a ZIP archive: https://github.com/bmoscon/cryptofeed/archive/master.zip
• using Pipenv

The following chapters provide more details about Pip and Pipenv usage, in a pedagogical way.

Installing using Pip

The recommended way to install or upgrade the Cryptofeed library:

python3 -m pip install --user --upgrade cryptofeed

In order to minimize the amount of dependencies to download, the dependencies required by the Rest API and the Cryptofeed backends are optional, but easy to install.

See the file setup.py for the exhaustive list of the extra dependencies.

Install all optional dependencies

To install Cryptofeed along with all optional dependencies in one bundle:

python3 -m pip install --user --upgrade cryptofeed[all]

Rest API

The Rest API feature of Cryptofeed allows to retrieve historical market data and to place order. See also the chaptper Rest API) in the main README.md for more details.

python3 -m pip install --user --upgrade cryptofeed[rest_api]

Arctic backend

To install Cryptofeed along with Arctic in one bundle:

python3 -m pip install --user --upgrade cryptofeed[arctic]

Redis backend

python3 -m pip install --user --upgrade cryptofeed[redis]

ZeroMQ backend

python3 -m pip install --user --upgrade cryptofeed[zmq]

RabbitMQ backend

python3 -m pip install --user --upgrade cryptofeed[zmq]

MongoDB backend

python3 -m pip install --user --upgrade cryptofeed[mongo]

PostgreSQL backend

python3 -m pip install --user --upgrade cryptofeed[postgres]

Kafka backend

python3 -m pip install --user --upgrade cryptofeed[kafka]

Pipenv

The Pipenv tool allows to install Cryptofeed and its dependencies without impacting your daily Python environment. Pipenv is based on pip and virtualenv.

Install Pipenv

The Cryptofeed team recommends to install the latest Pip and Pipenv versions on the user Python environment to limit conflicts with the Operating System:

python3 -m pip install --user --upgrade pip
python3 -m pip install --user --upgrade pipenv

Install the runtime dependencies

Once you get the Cryptofeed source code, you can install the dependencies within a dedicated Python virtual environment using the following command line:

cd your/path/to/cryptofeed
python3 -m pipenv install

Attention: the default Pipfile is configured to install all dependencies. You edit the Pipfile before installing all optional dependencies.

Uninstall the unused dependencies

Edit the Pipfile and comment some (or all) dependencies above the line # Optional dependencies.

After, run:

cd your/path/to/cryptofeed
python3 -m pipenv clean

You may also copy and hack this Pipfile within you own project. The Pipfile content is in the public domain.

Attention: See the LICENSE for the rest of the Cryptofeed files.

Update dependencies

Once a week, you may update the dependency versions.

cd your/path/to/cryptofeed
python3 -m pipenv update

Dependency graph

You can also check the whole dependency tree.

cd your/path/to/cryptofeed
python3 -m pipenv graph

Run a script

In the following example, we run the demo.py script.

cd your/path/to/cryptofeed
PYTONPATH=. python3 -m pipenv run python3 examples/demo.py

To use shorter command lines, you may prefer to enter in the Python virtual environment sub-shell:

cd your/path/to/cryptofeed
python3 -m pipenv shell
export PYTONPATH=$PWD
cd path/to/your/awesome/project
python your-awesome-script.py
[...]
exit    # or [Ctrl] + [D]

Do not forget you are in a virtual environment sub-shell.
To exit this sub-shell: use the command exit or the keyboard shortcut Ctrl + D.

Install the development dependencies

cd your/path/to/cryptofeed
python3 -m pipenv install --dev

Unit-test

Pytest is listed as a dependency in Pipfile. There is also a Pytest plugin, pytest-asyncio, allowing us to write unit-tests for asyncio functions.

Once the the development dependencies are installed, you run the unit-tests in two ways.

One way is to use a long python command line:

cd your/path/to/cryptofeed
python3 -m pipenv run python3 -m pytest tests

The second way is to activate the Python virtual environment within a dedicated shell:

cd your/path/to/cryptofeed
python3 -m pipenv shell
pytest
[...]
exit   # or [Ctrl] + [D]

Static-Analysis

The development dependencies in Pipfile include Pylint and many Pylint plugins for a relevant static-analysis.

Therefore you can detect potential bugs and error-prone coding style.

cd your/path/to/cryptofeed
python3 -m pipenv run python3 -m pylint --output-format=colorized ./cryptofeed/exchange

You may want to reduce the amount of issues by disabling the minor ones with option --disable:

cd your/path/to/cryptofeed
python3 -m pipenv run python3 -m pylint --output-format=colorized --disable=C0111,C0301,C0103,R0903,R0913,R0912  ./cryptofeed/exchange

Analyze more folders:

cd your/path/to/cryptofeed
python3 -m pipenv run python3 -m pylint --output-format=colorized ./cryptofeed ./examples ./tools

Enable the Pylint plugins with option --load-plugins:

cd your/path/to/cryptofeed
export PYTONPATH=.
python3 -m pipenv run python3 -m pylint --verbose --output-format=colorized --load-plugins=pylint_topology,pylint_import_modules,pylint_google_style_guide_imports_enforcing,pylint_unittest,pylint_requests,pylint_args,string_spaces_checkers ./cryptofeed

When almost all issues are fixed, speed up the Pylint using option --jobs=8. However, when there is many issues, this options mixes the Pylint output.

Optimize the import sections

One more thing: the Pipfile also provides the isort tool.

The following isort options apply the same formatting as black but only on the import sections:

cd your/path/to/cryptofeed
python3 -m pipenv run python3 -m isort --jobs=8 --atomic --multi-line 3 --force-grid-wrap 0 --trailing-comma --use-parentheses --apply --recursive .

Support / Contribute

If you have an issue installing / hacking Cryptofeed, please feel free to open a new issue:
https://github.com/bmoscon/cryptofeed/issues/

Pull Requests are also welcome, even for minor changes.

Changelog

1.5.0 (2020-07-31)

• Feature: New Exchange - FTX US
• Feature: Add funding data to rest library
• Bugfix: DSX updated their api, websocket no longer supported. Removing DSX
• Feature: Websocket client now uses unbounded message queue
• Feature: Support for HuobiDM next quarter contracts
• Bugfix: Fix datetime fields in elasticsearch
• Feature: BinanceFutures: support ticker, open interest and Liquidation, FTX: support open interest and liquidations, Deribit: liquidations support
• Bugfix: Fix receipt timestamps in Postgres backend
• Bugfix: Huobi Swap Init

1.4.1 (2020-05-22)

• Feature: Support for disabling timeouts on feeds
• Bugfix: #224 Ignore newly added trading pairs in Poloniex while running
• Feature: New exchange, DSX
• Bugfix: Bybit updated their API, websocket subscription to L2 book data needed to be updated
• Bugfix: Deribit subscription condensed into a single message to avoid issues with rate limit
• Bugfix: Funding interval for bitmex not converted to integer
• Bugfix: HuobiSwap missing from feedhandler
• Feature: Optional flag on Feed to enable check for crossed books
• Feature: Blockchain Exchange

1.3.1 (2020-03-17)

• Feature: Add missing update detection to orderbooks in Binance
• Feature: REST support for FTX
• Feature: Added new field, receipt timestamp, to all callbacks. This contains the time the message was received by cryptofeed.
• Feature: Upbit Exchange Support

1.3.0 (2020-02-11)

• Bugfix: Enabling multiple symbols on Bitmex with deltas and max depth configured could cause crashes.
• Bugfix: Default open interest callback missing
• Change: Mongo backend stores book data in BSON
• Feature: Open Interest callbacks added to all backends
• Change: Instrument removed in favor of open interest
• Bugfix: Huobi feedhandlers not properly setting forced indicator for book updates, breaking deltas
• Bugfix: Some kraken futures funding fields not always populated
• Feature: Open interest updates for kraken futures
• Feature: Open interest updates for Deribit
• Bugfix: FTX ticker can have Nones for bid/ask
• Feature: InfluxDB 2.0 support
• Bugfix: Deribit funding only available on perpetuals
• Feature: Enable deltas (with out max depth) on exchanges that do not support them

1.2.0 (2020-01-18)

• Feature: New exchange: Binance Futures
• Feature: New Exchange: Binance Jersey
• Feature: Funding data on Kraken Futures
• Feature: User defined pair seperator (default still -)
• Feature: Postgres backend
• Feature: Deribit Funding
• Bugfix: Deribit subscriptions using config subscribed to symbols incorrectly
• Bugfix: Some RabbitMQ messages were missing symbol and exchange data
• Feature: Open interest data for OKEX swaps

1.1.0 (2019-11-14)

• Feature: User enabled logging of exchange messages on error
• Refactor: Overhaul of backends - new base classes and simplified code
• Bugfix: Handle i messages from poloniex more correctly
• Bugfix: Report bittrex errors correctly
• Feature: New exchange: Bitcoin.com
• Feature: New exchange: BinanceUS
• Feature: New exchange: Bitmax
• Feature: Ability to store raw messages from exchanges

1.0.1 (2019-09-30)

• Feature: Backfill Bitmex historical trade data from S3 Bucket
• Feature: RabbitMQ backend
• Feature: Custom Depth and deltas for all L2 book updates
• Feature: Support new 100ms book diff channel on binance
• Feature: Bittrex exchange support
• Feature: Ticker support in Redis and Kafka Backends
• Feature: Ticker callbacks require/contain timestamp
• Feature: Renko Aggregation
• Bugfix: Max Depth without deltas should only send updates when book changes
• Bugfix: Update count and previous book now associated with pair

1.0.0 (2019-08-18)

• Bugfix #113: Fix remaining exchanges who are not reporting timestamps correctly
• Feature: Generated timestamps now based on message receipt by feedhandler
• Feature: Multi-callback support
• Feature: Rework ZMQ using pub/sub with topics
• Feature: FTX Exchange
• Feature: Gemini subscriptions now work like all other exchanges
• Feature: Use unique id for each feed (as opposed to feed id/name)
• Bugfix: fix Poloniex historical trade timestamps
• Bugfix: Bitmex L2 channel incorrectly classified
• Feature: Kraken Futures
• Feature: Redis backend supports UDS
• Feature: Binance full book (L2) with deltas
• Feature: Allow user to start event loop themselves (potentially scheduling other tasks before/after).

0.25.0 (2019-07-06)

• Feature: Rest Endpoints for Historical Deribit data
• Feature: Specify numeric datatype for InfluxDB
• Bugfix: Greatly improve performance of book writes for InfluxDB
• Feature: Bybit exchange support
• Bugfix: Deribit now returning floats in decimal.Decimal
• Feature: Elastic Search backend

0.24.0 (2019-06-19)

• Bugfix: Book Delta Conversion issue in backends
• Bugfix: Tweak BitMEX rest api to handle more errors more gracefully
• Feature: Deribit Exchange support
• Feature: Instrument channel
• Bugfix: support Kraken websocket API changes
• Bugfix: correct USDT symbol mappings for Bitfinex
• Bugfix: Fixed mongo book backend
• Feature: Book delta support for mongo, sockets, ZMQ

0.23.0 (2019-06-03)

• Feature: Book delta support for InfluxDB
• Feature: Swaps on OkEX

0.22.2 (2019-05-23)

• Bugfix: Fix tagging issue in InfluxDB
• Bugfix: Fix book updates in InfluxDB
• Feature: Book delta support in Redis backends
• Feature: Book delta support in Kafka backend

0.22.1 (2019-05-19)

• Feature: Cleanup callback code
• Feature: Poloniex subscription now behaves like other exchanges
• Feature: Kafka Backend

0.22.0 (2019-05-04)

• Bugfix: Timestamp normalization for backends were losing subsecond fidelity
• Feature: All exchanges report timestamps in floating point unix time
• Bugfix: Implement change in OkEx's trading pair endpoint for pair generation

0.21.1 (2019-04-28)

• Feature: Config support for Coinbene, Binance, EXX, BitMEX, Bitfinex, Bitstamp, HitBTC
• Feature: Complete clean up of public REST endpoints
• Feature: Improved book delta example
• Feature: Bitstamp Websocket V2 - L3 books now supported
• Bugfix: Incorrect book building in Kraken

0.21.0 (2019-04-07)

• Bugfix: Coinbase L3 Book would get in cycle of reconnecting due to missing sequence numbers
• Feature: Kraken L2 Book Deltas
• Feature: Book deltas streamlined and retain ordering
• Feature: OKCoin exchange support
• Feature: OKEx exchange support
• Feature: Coinbene exchange support
• Feature: Support Huobi Global and Huobi USA

0.20.2 (2019-03-19)

• Bugfix: Kraken REST api using wrong symbol for trades
• Feature: Complete work on standardizing Bitfinex rest API
• Bugfix: Allow index symbols on Bitmex

0.20.1 (2019-02-16)

• Feature: Trades sides are now labeled as Buy / Sell instead of Bid / Ask.
• Feature: Support for the Huobi exchange
• Bugfix: Change how exchange pairs are mapped for REST module - only map exchanges that are used
• Bugfix #67: Ensure all trades report the taker's side

0.20.0 (2019-02-04)

• Feature #57: Write updates directly to MongoDB via new backend support
• Feature #56: Experimental support for fine grained configuration per exchange
• Feature #58: Support Kraken websocket API
• Feature: Only generate trading pair conversions for configured exchanges
• Feature: Historical trade data on REST api for Kraken

0.19.2 (2019-01-21)

• Feature #55: OHLCV aggregation method in backends plus support for user defined aggregators
• Feature: EXX exchange support

0.19.1 (2019-01-11)

• Bugfix: Poloniex logging had bug that prevented reconnect on missing sequence number

0.19.0 (2019-01-10)

• Feature #50: Support multiple streams per websocket connection on Binance
• Bugfix #51: Fix pairs on streams in Binance

0.18.0 (2018-12-15)

• Feature: InfluxDB support via backend
• Feature: Aggregation backend wrappers
• Bugfix: BookDelta callback no longer needs to be an instance of BookUpdateCallback
• Bugfix: REST module was creating duplicate log handlers
• Bugfix: Bitfinex REST now properly handles cases when there are more than 1000 updates for a single tick

0.17.4 (2018-11-17)

• Readme change for long description rendering issue

0.17.3 (2018-11-17)

• Feature #41: Rework trading pairs to generate them dynamically (as opposed to hard coded)
• Feature: When book depth configured Redis, ZMQ and UDP backends only report book changes when changed occurred in depth window
• Feature: TCP socket backend support
• Feature: UDS backend support

0.17.2 (2018-11-03)

• Bugfix #45: Bitstamp prices and sizes in L2 book are string, not decimal.Decimal
• Feature: Binance support

0.17.1 (2018-10-19)

• Bugfix #43: Coinbase L2 book used "0" rather than 0 for comparisons against decimal.Decimal
• Feature: REST feed market data supported via normal subscription methods
• Feature: Kraken support
• Bugfix: Bitfinex book timestamps match expected bitfinex timestamps (in ms)

0.17.0 (2018-10-13)

• Feature: Timestamps for orderbooks and book deltas
• Feature #40: NBBO now uses best bid/ask from L2 books
• Feature #28: GDAX now renamed Coinbase and uses coinbase endpoints
• Feature: ZeroMQ backend. Write updates directly to zmq connection
• Feature: UDP Socket backend. Write updates directy to UDP socket

0.16.0 (2018-10-4)

• Feature: L2 books are now all price aggregted amounts, L3 books are price aggregated orders
• Book deltas supported on all feeds
• Bugfix: Fix NBBO feed

0.15.0 (2018-09-29)

• Feature: GDAX/Coinbase rest support - trades, order status, etc
• Feature: Arctic backend, supports writing to arctic directly on trade/funding updates
• Bugfix: #36 Update poloniex to use new trading pairs and handle sequence numbers
• Bugfix: Improve Bitfinex orderbooks and handle sequence numbers
• Bugfix: GDAX and Bitmex orderbook and logging improvements

0.14.1 (2018-09-14)

• Added some docstrings
• Feature: Add exchanges by name to feedhandler. Easier to instantiate a feedhandler from config
• Logging improvements
• Bugfix: non-gathered futures were suppressing exceptions when multiple feeds are configured. Changed to tasks
• Redis backend uses a connection pool

0.14.0 (2018-09-04)

• Feature: support for writing order books directly to redis
• Feature: ability to specify book depth for redis updates

0.13.3 (2018-08-31)

• Feature: normalize bitfinex funding symbols

0.13.2 (2018-08-31)

• Bugfix: fix symbol in bitfinex rest

0.13.1 (2018-08-31)

• Feature: access rest endpoints via getitem / []
• Bugfix: #31 - funding channel broke gemini
• Feature: Book deltas for GDAX
• Bugfix: Fix intervals on Bitmex (rest)

0.13.0 (2018-08-22)

• Feature: Funding data from Bitmex on ws
• Feature: Funding historical data via rest
• Bugfix: Python 3.7 compatibility
• Feature: Rest trade APIs are now generators
• Feature: funding data on bitfinex - ws and rest

0.12.0 (2018-08-20)

• Bugfix: Handle 429s in Bitmex (REST)
• Feature: Redis backend for trades to write updates directly to redis
• Bugfix: issue #27 - Bitmex trades missing timestamps

0.11.1 (2018-08-18)

• Bitfinex and Bitmex historical trade data via REST
• Bugfix: interval incorrect for rest time ranges
• Bugfix: lowercase attrs in Rest interface

0.11.0 (2018-08-05)

• Feature: Support for delta updates for order books
• REST api work started

0.10.2

• Bugfix: Clear data structures on reconnect in bitmex
• Feature: Support reconnecting on more connection errors
• Feature: Timestamp support on trade feeds
• Feature: Connection watcher will terminate and re-open idle connections

0.10.1 (2018-5-11)

• Feature: Reconnect when a connection is lost
• Bugfix #22: Check for additional connection failures
• Feature #4: Trade ID support
• Feature: Account for new gemini message type

0.10.0 (2018-03-18)

• Feature: Bitmex

0.9.2 (2018-03-13)

• Bugfix #10: Change from float to decimal.Decimal in GDAX
• Feature #5: use sorted dictionaries for order books
• Feature #17: logging support
• Bugfix: Gemini order books now work
• Bugfix: All json floats parsed to Decimal
• Bugfix: Fix Bitstamp pair parsing
• Feature: Major clean up of channel, exchange, and trading pair names

0.9.1 (2018-01-27)

• Bugfix #4: produce ticker from trades channel on GDAX
• Feature: Bitstamp feed

0.8.0 (2018-01-07)

• Feature: HitBTC feed
• Feature: Poloniex Orderbook support

0.6.0 (2018-01-02)

• Feature: Gemini Feed

0.5.0 (2018-01-02)

• Initial release: GDAX, Poloniex, Bitfinex Support
• Feature: NBBO support