Skip to main content

TensorFlow IO

Project description




TensorFlow I/O

GitHub CI PyPI License Documentation

TensorFlow I/O is a collection of file systems and file formats that are not available in TensorFlow's built-in support. A full list of supported file systems and file formats by TensorFlow I/O can be found here.

The use of tensorflow-io is straightforward with keras. Below is an example to Get Started with TensorFlow with the data processing aspect replaced by tensorflow-io:

import tensorflow as tf
import tensorflow_io as tfio

# Read the MNIST data into the IODataset.
dataset_url = "https://storage.googleapis.com/cvdf-datasets/mnist/"
d_train = tfio.IODataset.from_mnist(
    dataset_url + "train-images-idx3-ubyte.gz",
    dataset_url + "train-labels-idx1-ubyte.gz",
)

# Shuffle the elements of the dataset.
d_train = d_train.shuffle(buffer_size=1024)

# By default image data is uint8, so convert to float32 using map().
d_train = d_train.map(lambda x, y: (tf.image.convert_image_dtype(x, tf.float32), y))

# prepare batches the data just like any other tf.data.Dataset
d_train = d_train.batch(32)

# Build the model.
model = tf.keras.models.Sequential(
    [
        tf.keras.layers.Flatten(input_shape=(28, 28)),
        tf.keras.layers.Dense(512, activation=tf.nn.relu),
        tf.keras.layers.Dropout(0.2),
        tf.keras.layers.Dense(10, activation=tf.nn.softmax),
    ]
)

# Compile the model.
model.compile(
    optimizer="adam", loss="sparse_categorical_crossentropy", metrics=["accuracy"]
)

# Fit the model.
model.fit(d_train, epochs=5, steps_per_epoch=200)

In the above MNIST example, the URL's to access the dataset files are passed directly to the tfio.IODataset.from_mnist API call. This is due to the inherent support that tensorflow-io provides for HTTP/HTTPS file system, thus eliminating the need for downloading and saving datasets on a local directory.

NOTE: Since tensorflow-io is able to detect and uncompress the MNIST dataset automatically if needed, we can pass the URL's for the compressed files (gzip) to the API call as is.

Please check the official documentation for more detailed and interesting usages of the package.

Installation

Python Package

The tensorflow-io Python package can be installed with pip directly using:

$ pip install tensorflow-io

People who are a little more adventurous can also try our nightly binaries:

$ pip install tensorflow-io-nightly

To ensure you have a version of TensorFlow that is compatible with TensorFlow-IO, you can specify the tensorflow extra requirement during install:

pip install tensorflow-io[tensorflow]

Similar extras exist for the tensorflow-gpu, tensorflow-cpu and tensorflow-rocm packages.

Docker Images

In addition to the pip packages, the docker images can be used to quickly get started.

For stable builds:

$ docker pull tfsigio/tfio:latest
$ docker run -it --rm --name tfio-latest tfsigio/tfio:latest

For nightly builds:

$ docker pull tfsigio/tfio:nightly
$ docker run -it --rm --name tfio-nightly tfsigio/tfio:nightly

R Package

Once the tensorflow-io Python package has been successfully installed, you can install the development version of the R package from GitHub via the following:

if (!require("remotes")) install.packages("remotes")
remotes::install_github("tensorflow/io", subdir = "R-package")

TensorFlow Version Compatibility

To ensure compatibility with TensorFlow, it is recommended to install a matching version of TensorFlow I/O according to the table below. You can find the list of releases here.

TensorFlow I/O Version TensorFlow Compatibility Release Date
0.24.0 2.8.x Feb 04, 2022
0.23.1 2.7.x Dec 15, 2021
0.23.0 2.7.x Dec 14, 2021
0.22.0 2.7.x Nov 10, 2021
0.21.0 2.6.x Sep 12, 2021
0.20.0 2.6.x Aug 11, 2021
0.19.1 2.5.x Jul 25, 2021
0.19.0 2.5.x Jun 25, 2021
0.18.0 2.5.x May 13, 2021
0.17.1 2.4.x Apr 16, 2021
0.17.0 2.4.x Dec 14, 2020
0.16.0 2.3.x Oct 23, 2020
0.15.0 2.3.x Aug 03, 2020
0.14.0 2.2.x Jul 08, 2020
0.13.0 2.2.x May 10, 2020
0.12.0 2.1.x Feb 28, 2020
0.11.0 2.1.x Jan 10, 2020
0.10.0 2.0.x Dec 05, 2019
0.9.1 2.0.x Nov 15, 2019
0.9.0 2.0.x Oct 18, 2019
0.8.1 1.15.x Nov 15, 2019
0.8.0 1.15.x Oct 17, 2019
0.7.2 1.14.x Nov 15, 2019
0.7.1 1.14.x Oct 18, 2019
0.7.0 1.14.x Jul 14, 2019
0.6.0 1.13.x May 29, 2019
0.5.0 1.13.x Apr 12, 2019
0.4.0 1.13.x Mar 01, 2019
0.3.0 1.12.0 Feb 15, 2019
0.2.0 1.12.0 Jan 29, 2019
0.1.0 1.12.0 Dec 16, 2018

Performance Benchmarking

We use github-pages to document the results of API performance benchmarks. The benchmark job is triggered on every commit to master branch and facilitates tracking performance w.r.t commits.

Contributing

Tensorflow I/O is a community led open source project. As such, the project depends on public contributions, bug-fixes, and documentation. Please see:

Build Status and CI

Build Status
Linux CPU Python 2 Status
Linux CPU Python 3 Status
Linux GPU Python 2 Status
Linux GPU Python 3 Status

Because of manylinux2010 requirement, TensorFlow I/O is built with Ubuntu:16.04 + Developer Toolset 7 (GCC 7.3) on Linux. Configuration with Ubuntu 16.04 with Developer Toolset 7 is not exactly straightforward. If the system have docker installed, then the following command will automatically build manylinux2010 compatible whl package:

#!/usr/bin/env bash

ls dist/*
for f in dist/*.whl; do
  docker run -i --rm -v $PWD:/v -w /v --net=host quay.io/pypa/manylinux2010_x86_64 bash -x -e /v/tools/build/auditwheel repair --plat manylinux2010_x86_64 $f
done
sudo chown -R $(id -nu):$(id -ng) .
ls wheelhouse/*

It takes some time to build, but once complete, there will be python 3.5, 3.6, 3.7 compatible whl packages available in wheelhouse directory.

On macOS, the same command could be used. However, the script expects python in shell and will only generate a whl package that matches the version of python in shell. If you want to build a whl package for a specific python then you have to alias this version of python to python in shell. See .github/workflows/build.yml Auditwheel step for instructions how to do that.

Note the above command is also the command we use when releasing packages for Linux and macOS.

TensorFlow I/O uses both GitHub Workflows and Google CI (Kokoro) for continuous integration. GitHub Workflows is used for macOS build and test. Kokoro is used for Linux build and test. Again, because of the manylinux2010 requirement, on Linux whl packages are always built with Ubuntu 16.04 + Developer Toolset 7. Tests are done on a variatiy of systems with different python3 versions to ensure a good coverage:

Python Ubuntu 18.04 Ubuntu 20.04 macOS + osx9 Windows-2019
2.7 :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: N/A
3.7 :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark:
3.8 :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark:

TensorFlow I/O has integrations with many systems and cloud vendors such as Prometheus, Apache Kafka, Apache Ignite, Google Cloud PubSub, AWS Kinesis, Microsoft Azure Storage, Alibaba Cloud OSS etc.

We tried our best to test against those systems in our continuous integration whenever possible. Some tests such as Prometheus, Kafka, and Ignite are done with live systems, meaning we install Prometheus/Kafka/Ignite on CI machine before the test is run. Some tests such as Kinesis, PubSub, and Azure Storage are done through official or non-official emulators. Offline tests are also performed whenever possible, though systems covered through offine tests may not have the same level of coverage as live systems or emulators.

Live System Emulator CI Integration Offline
Apache Kafka :heavy_check_mark: :heavy_check_mark:
Apache Ignite :heavy_check_mark: :heavy_check_mark:
Prometheus :heavy_check_mark: :heavy_check_mark:
Google PubSub :heavy_check_mark: :heavy_check_mark:
Azure Storage :heavy_check_mark: :heavy_check_mark:
AWS Kinesis :heavy_check_mark: :heavy_check_mark:
Alibaba Cloud OSS :heavy_check_mark:
Google BigTable/BigQuery to be added
Elasticsearch (experimental) :heavy_check_mark: :heavy_check_mark:
MongoDB (experimental) :heavy_check_mark: :heavy_check_mark:

References for emulators:

Community

Additional Information

License

Apache License 2.0

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

tensorflow_io_nightly-0.24.0.dev20220209214902-cp310-cp310-win_amd64.whl (21.8 MB view details)

Uploaded CPython 3.10 Windows x86-64

tensorflow_io_nightly-0.24.0.dev20220209214902-cp310-cp310-macosx_10_14_x86_64.whl (24.1 MB view details)

Uploaded CPython 3.10 macOS 10.14+ x86-64

tensorflow_io_nightly-0.24.0.dev20220209214902-cp39-cp39-win_amd64.whl (21.8 MB view details)

Uploaded CPython 3.9 Windows x86-64

tensorflow_io_nightly-0.24.0.dev20220209214902-cp39-cp39-macosx_10_14_x86_64.whl (24.1 MB view details)

Uploaded CPython 3.9 macOS 10.14+ x86-64

tensorflow_io_nightly-0.24.0.dev20220209214902-cp38-cp38-win_amd64.whl (21.8 MB view details)

Uploaded CPython 3.8 Windows x86-64

tensorflow_io_nightly-0.24.0.dev20220209214902-cp38-cp38-macosx_10_14_x86_64.whl (24.1 MB view details)

Uploaded CPython 3.8 macOS 10.14+ x86-64

tensorflow_io_nightly-0.24.0.dev20220209214902-cp37-cp37m-win_amd64.whl (21.8 MB view details)

Uploaded CPython 3.7m Windows x86-64

tensorflow_io_nightly-0.24.0.dev20220209214902-cp37-cp37m-macosx_10_14_x86_64.whl (24.1 MB view details)

Uploaded CPython 3.7m macOS 10.14+ x86-64

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220209214902-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220209214902-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 21.8 MB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.10.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220209214902-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 f4a8926689c5a796dc0982e37dfa0c23ac538b130aaf494fb3b7f6f13bd7f799
MD5 4101bf0e6c23a6bfbdd61bc1436c9c1a
BLAKE2b-256 72c09cfd34bb911052c36611154b5a09252c5b3d3233d334576cfed06fb0fca6

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220209214902-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220209214902-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 e8bf9897e6639d0534af2b7391ffd338b11b056e507137d002802f30f60b80a6
MD5 c6c44457d27f79c5334c30565ce12b24
BLAKE2b-256 9f822fa50b952bd7447f7a92cb552fa962c1af623a268ca952a12e86f13d2d31

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220209214902-cp310-cp310-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220209214902-cp310-cp310-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 24.1 MB
  • Tags: CPython 3.10, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.10.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220209214902-cp310-cp310-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 15487135adcd6d949b3cacedda3898d8bfbf09ac359d9f8b5221395331c2cdee
MD5 b7b42f3c58aaca6d6d27b5cc210fd4b0
BLAKE2b-256 fa10057aee968aa15236b273b50676600976fbecede6e7c47c7dc1ad8f4f6b1f

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220209214902-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220209214902-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 21.8 MB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.10.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220209214902-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 48fa266a7183a40457821018acce07aeea8e1d4c6720b6054e4ff865edc8d796
MD5 bc6f3779c11fdd08c1c2cc64e0f9bb8f
BLAKE2b-256 39910da15fc583ade499d8074e593ce01de86bd2106ddcb5f2f0f36fa80e3106

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220209214902-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220209214902-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 3f0d0d873e603fcd927464d1c74a87eefb4892392a33cd1406d79a8b13afea67
MD5 a0cbe6b4b7c69cf4be2668439e1b0ba2
BLAKE2b-256 335973a4dd1e03c1e2631503db082496770b136db64595cdbce6f808e9ede622

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220209214902-cp39-cp39-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220209214902-cp39-cp39-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 24.1 MB
  • Tags: CPython 3.9, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.10.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220209214902-cp39-cp39-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 366e6e7eb1033549080ca8561afdb288652e8127f1bb7ecbe2b31ac6e0f9e9e4
MD5 29890b316df1bb455a1b19076facf5ae
BLAKE2b-256 580adb9e41d988daa7c4b93461b28faaeb8ab97ba7a61f98d8a5d58cf265aaed

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220209214902-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220209214902-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 21.8 MB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.10.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220209214902-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 1d2f8d88c868a54b48ac5de88e6dfcdcf07846a51d866abdbf4190d2af529083
MD5 d5984a32d955bce08ac9907d62b15f01
BLAKE2b-256 48251b6a1e26f6a2bfb4e29ec1b140b23eba001cb036f153aba2784c0d7c0d2a

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220209214902-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220209214902-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 bd3a25da43494d03fd68699fc0f941d66e9d539933d9f53e65a4fec0964ec899
MD5 dd3679f455cf7b258ee36e9775b41e4a
BLAKE2b-256 fa00525db8c5bae16e3c1309c8073f5b8d3dbb721400d0b71cd6a78eb1c4a605

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220209214902-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220209214902-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 24.1 MB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.10.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220209214902-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 6a773ead10bd04baa366aea21a6d33b28355adda3c11857575239e51543abf17
MD5 86e8c6324c469dd70cfc63374892ef32
BLAKE2b-256 4b572aa66d54bab0d79d1ac3b1535448d6998e6fd349ef764ea5ab08b7184d9b

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220209214902-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220209214902-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 21.8 MB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.10.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220209214902-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 11def66872c2c8ce5d4184c5cbcf338806fc2b39cca2cf5ee3f13775e5ef58ea
MD5 59f47d07724ee00385fd091ff7c11e5a
BLAKE2b-256 3e3d9536fcf759cc1005fd2b36c9a15866fbb9f64e344ffe19eb983ffda63400

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220209214902-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220209214902-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 85b823ff023a6bd6920f4f89106e057cfb091ed54c7e6a7dd56b28b0b5feb229
MD5 b809dde5e370139aa17b8556858df47d
BLAKE2b-256 b29940b3a9764f706cc1daeb867a90759bdda7d58622f0b4afb3f9b22253ab9d

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220209214902-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220209214902-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 24.1 MB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.10.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220209214902-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 4cb647e6b5d6d9192b9017917d5fd4844dd96984b99c2d37b70e9979551309bc
MD5 9dd82fb6b9e75026b49ae2c8b9cd7049
BLAKE2b-256 7ab8316bfa8a8114b9b059c16367ef7268588bd8546505195cda07a6a17d2d36

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page