Skip to main content

TensorFlow IO

Project description




TensorFlow I/O

GitHub CI PyPI License Documentation

TensorFlow I/O is a collection of file systems and file formats that are not available in TensorFlow's built-in support. A full list of supported file systems and file formats by TensorFlow I/O can be found here.

The use of tensorflow-io is straightforward with keras. Below is an example to Get Started with TensorFlow with the data processing aspect replaced by tensorflow-io:

import tensorflow as tf
import tensorflow_io as tfio

# Read the MNIST data into the IODataset.
dataset_url = "https://storage.googleapis.com/cvdf-datasets/mnist/"
d_train = tfio.IODataset.from_mnist(
    dataset_url + "train-images-idx3-ubyte.gz",
    dataset_url + "train-labels-idx1-ubyte.gz",
)

# Shuffle the elements of the dataset.
d_train = d_train.shuffle(buffer_size=1024)

# By default image data is uint8, so convert to float32 using map().
d_train = d_train.map(lambda x, y: (tf.image.convert_image_dtype(x, tf.float32), y))

# prepare batches the data just like any other tf.data.Dataset
d_train = d_train.batch(32)

# Build the model.
model = tf.keras.models.Sequential(
    [
        tf.keras.layers.Flatten(input_shape=(28, 28)),
        tf.keras.layers.Dense(512, activation=tf.nn.relu),
        tf.keras.layers.Dropout(0.2),
        tf.keras.layers.Dense(10, activation=tf.nn.softmax),
    ]
)

# Compile the model.
model.compile(
    optimizer="adam", loss="sparse_categorical_crossentropy", metrics=["accuracy"]
)

# Fit the model.
model.fit(d_train, epochs=5, steps_per_epoch=200)

In the above MNIST example, the URL's to access the dataset files are passed directly to the tfio.IODataset.from_mnist API call. This is due to the inherent support that tensorflow-io provides for HTTP/HTTPS file system, thus eliminating the need for downloading and saving datasets on a local directory.

NOTE: Since tensorflow-io is able to detect and uncompress the MNIST dataset automatically if needed, we can pass the URL's for the compressed files (gzip) to the API call as is.

Please check the official documentation for more detailed and interesting usages of the package.

Installation

Python Package

The tensorflow-io Python package can be installed with pip directly using:

$ pip install tensorflow-io

People who are a little more adventurous can also try our nightly binaries:

$ pip install tensorflow-io-nightly

To ensure you have a version of TensorFlow that is compatible with TensorFlow-IO, you can specify the tensorflow extra requirement during install:

pip install tensorflow-io[tensorflow]

Similar extras exist for the tensorflow-gpu, tensorflow-cpu and tensorflow-rocm packages.

Docker Images

In addition to the pip packages, the docker images can be used to quickly get started.

For stable builds:

$ docker pull tfsigio/tfio:latest
$ docker run -it --rm --name tfio-latest tfsigio/tfio:latest

For nightly builds:

$ docker pull tfsigio/tfio:nightly
$ docker run -it --rm --name tfio-nightly tfsigio/tfio:nightly

R Package

Once the tensorflow-io Python package has been successfully installed, you can install the development version of the R package from GitHub via the following:

if (!require("remotes")) install.packages("remotes")
remotes::install_github("tensorflow/io", subdir = "R-package")

TensorFlow Version Compatibility

To ensure compatibility with TensorFlow, it is recommended to install a matching version of TensorFlow I/O according to the table below. You can find the list of releases here.

TensorFlow I/O Version TensorFlow Compatibility Release Date
0.24.0 2.8.x Feb 04, 2022
0.23.1 2.7.x Dec 15, 2021
0.23.0 2.7.x Dec 14, 2021
0.22.0 2.7.x Nov 10, 2021
0.21.0 2.6.x Sep 12, 2021
0.20.0 2.6.x Aug 11, 2021
0.19.1 2.5.x Jul 25, 2021
0.19.0 2.5.x Jun 25, 2021
0.18.0 2.5.x May 13, 2021
0.17.1 2.4.x Apr 16, 2021
0.17.0 2.4.x Dec 14, 2020
0.16.0 2.3.x Oct 23, 2020
0.15.0 2.3.x Aug 03, 2020
0.14.0 2.2.x Jul 08, 2020
0.13.0 2.2.x May 10, 2020
0.12.0 2.1.x Feb 28, 2020
0.11.0 2.1.x Jan 10, 2020
0.10.0 2.0.x Dec 05, 2019
0.9.1 2.0.x Nov 15, 2019
0.9.0 2.0.x Oct 18, 2019
0.8.1 1.15.x Nov 15, 2019
0.8.0 1.15.x Oct 17, 2019
0.7.2 1.14.x Nov 15, 2019
0.7.1 1.14.x Oct 18, 2019
0.7.0 1.14.x Jul 14, 2019
0.6.0 1.13.x May 29, 2019
0.5.0 1.13.x Apr 12, 2019
0.4.0 1.13.x Mar 01, 2019
0.3.0 1.12.0 Feb 15, 2019
0.2.0 1.12.0 Jan 29, 2019
0.1.0 1.12.0 Dec 16, 2018

Performance Benchmarking

We use github-pages to document the results of API performance benchmarks. The benchmark job is triggered on every commit to master branch and facilitates tracking performance w.r.t commits.

Contributing

Tensorflow I/O is a community led open source project. As such, the project depends on public contributions, bug-fixes, and documentation. Please see:

Build Status and CI

Build Status
Linux CPU Python 2 Status
Linux CPU Python 3 Status
Linux GPU Python 2 Status
Linux GPU Python 3 Status

Because of manylinux2010 requirement, TensorFlow I/O is built with Ubuntu:16.04 + Developer Toolset 7 (GCC 7.3) on Linux. Configuration with Ubuntu 16.04 with Developer Toolset 7 is not exactly straightforward. If the system have docker installed, then the following command will automatically build manylinux2010 compatible whl package:

#!/usr/bin/env bash

ls dist/*
for f in dist/*.whl; do
  docker run -i --rm -v $PWD:/v -w /v --net=host quay.io/pypa/manylinux2010_x86_64 bash -x -e /v/tools/build/auditwheel repair --plat manylinux2010_x86_64 $f
done
sudo chown -R $(id -nu):$(id -ng) .
ls wheelhouse/*

It takes some time to build, but once complete, there will be python 3.5, 3.6, 3.7 compatible whl packages available in wheelhouse directory.

On macOS, the same command could be used. However, the script expects python in shell and will only generate a whl package that matches the version of python in shell. If you want to build a whl package for a specific python then you have to alias this version of python to python in shell. See .github/workflows/build.yml Auditwheel step for instructions how to do that.

Note the above command is also the command we use when releasing packages for Linux and macOS.

TensorFlow I/O uses both GitHub Workflows and Google CI (Kokoro) for continuous integration. GitHub Workflows is used for macOS build and test. Kokoro is used for Linux build and test. Again, because of the manylinux2010 requirement, on Linux whl packages are always built with Ubuntu 16.04 + Developer Toolset 7. Tests are done on a variatiy of systems with different python3 versions to ensure a good coverage:

Python Ubuntu 18.04 Ubuntu 20.04 macOS + osx9 Windows-2019
2.7 :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: N/A
3.7 :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark:
3.8 :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark:

TensorFlow I/O has integrations with many systems and cloud vendors such as Prometheus, Apache Kafka, Apache Ignite, Google Cloud PubSub, AWS Kinesis, Microsoft Azure Storage, Alibaba Cloud OSS etc.

We tried our best to test against those systems in our continuous integration whenever possible. Some tests such as Prometheus, Kafka, and Ignite are done with live systems, meaning we install Prometheus/Kafka/Ignite on CI machine before the test is run. Some tests such as Kinesis, PubSub, and Azure Storage are done through official or non-official emulators. Offline tests are also performed whenever possible, though systems covered through offine tests may not have the same level of coverage as live systems or emulators.

Live System Emulator CI Integration Offline
Apache Kafka :heavy_check_mark: :heavy_check_mark:
Apache Ignite :heavy_check_mark: :heavy_check_mark:
Prometheus :heavy_check_mark: :heavy_check_mark:
Google PubSub :heavy_check_mark: :heavy_check_mark:
Azure Storage :heavy_check_mark: :heavy_check_mark:
AWS Kinesis :heavy_check_mark: :heavy_check_mark:
Alibaba Cloud OSS :heavy_check_mark:
Google BigTable/BigQuery to be added
Elasticsearch (experimental) :heavy_check_mark: :heavy_check_mark:
MongoDB (experimental) :heavy_check_mark: :heavy_check_mark:

References for emulators:

Community

Additional Information

License

Apache License 2.0

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

tensorflow_io_nightly-0.24.0.dev20220209004551-cp310-cp310-win_amd64.whl (21.8 MB view details)

Uploaded CPython 3.10 Windows x86-64

tensorflow_io_nightly-0.24.0.dev20220209004551-cp310-cp310-macosx_10_14_x86_64.whl (24.1 MB view details)

Uploaded CPython 3.10 macOS 10.14+ x86-64

tensorflow_io_nightly-0.24.0.dev20220209004551-cp39-cp39-win_amd64.whl (21.8 MB view details)

Uploaded CPython 3.9 Windows x86-64

tensorflow_io_nightly-0.24.0.dev20220209004551-cp39-cp39-macosx_10_14_x86_64.whl (24.1 MB view details)

Uploaded CPython 3.9 macOS 10.14+ x86-64

tensorflow_io_nightly-0.24.0.dev20220209004551-cp38-cp38-win_amd64.whl (21.8 MB view details)

Uploaded CPython 3.8 Windows x86-64

tensorflow_io_nightly-0.24.0.dev20220209004551-cp38-cp38-macosx_10_14_x86_64.whl (24.1 MB view details)

Uploaded CPython 3.8 macOS 10.14+ x86-64

tensorflow_io_nightly-0.24.0.dev20220209004551-cp37-cp37m-win_amd64.whl (21.8 MB view details)

Uploaded CPython 3.7m Windows x86-64

tensorflow_io_nightly-0.24.0.dev20220209004551-cp37-cp37m-macosx_10_14_x86_64.whl (24.1 MB view details)

Uploaded CPython 3.7m macOS 10.14+ x86-64

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220209004551-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220209004551-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 21.8 MB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.10.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220209004551-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 b4044afa8a1fb2d28d238f21f85502b67eb28b04bc77f9eb4b1a15c98837da34
MD5 0d9a5bfc20ab0321417f0371d389319f
BLAKE2b-256 7bec6b300ab46ba8199bd29b5e47fb10a087583a9c24605912299457dcbceebc

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220209004551-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220209004551-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 ebd4b1d95afe3dce878e304a9bd31a4c3d619d3a82eb321dd77451a8a8fd5722
MD5 bd462a9ea64453616cdd7220dee26d08
BLAKE2b-256 afb6ac0e0dd9cf0defb17e3f3521c7fd08c5dfb6246fe138029fbc0480b07281

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220209004551-cp310-cp310-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220209004551-cp310-cp310-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 24.1 MB
  • Tags: CPython 3.10, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.10.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220209004551-cp310-cp310-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 8eb915d951ab62ff978686ff2120ecf6bf5d50dc4fba3357ae9ddc98bebafe82
MD5 de24a4f79eb0d24c30e9d6156fa9f663
BLAKE2b-256 06f748d0fa509b671a531848ca3e3957d98a49203fc084992b1cb832825e504e

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220209004551-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220209004551-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 21.8 MB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.10.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220209004551-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 7bfb2dc76aece77da631237df527711e51c5475a76f0543badddce9a229a5d6c
MD5 a8da8aff06b91053519eaa1790109d80
BLAKE2b-256 15810f3e5d68586f9e2d319094c576cd490731421967cd3195e74ba9d7ee72e4

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220209004551-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220209004551-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 1747db45e4e35470518f3721e8fa69de343a43c1f84a10289679e7cdad8b337b
MD5 8913e079570a6a551b8ede1fc657863d
BLAKE2b-256 5455b784c62daca5bdc0d11155ad873bd7891b27f78efec841fad43374d64855

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220209004551-cp39-cp39-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220209004551-cp39-cp39-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 24.1 MB
  • Tags: CPython 3.9, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.10.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220209004551-cp39-cp39-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 c97b0eed83e3a4717f0fb1d98d52bedc2a7da2843368feb42f1c74c97e40c7a9
MD5 d333423aac660c6df499b5facdd2d141
BLAKE2b-256 abebaba0cc5fb014eb38d5ac4aca6ae794c61a5eee8e96a1fa902d7eca88729b

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220209004551-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220209004551-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 21.8 MB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.10.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220209004551-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 4bf49efded80cbfc0c31aa737662d9c842e96b36006d1c22c35ecd3a6094b412
MD5 72a4ef61b2ae680bc4e5a3894ef5b34e
BLAKE2b-256 b85e4c8b3e9ae22334098d685538d647cfbb1e2fb74492ded2485dbda3389f53

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220209004551-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220209004551-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 bda22aae65f5ef7f417033425750b5725b82905b2213a6ca670d6d85df5cd824
MD5 9ce828311da445d4aeab9a110a3a06c0
BLAKE2b-256 4b74642d9314011b87880c7d61e2870a414f3201b583dfb2ff5edd39ff82eafa

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220209004551-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220209004551-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 24.1 MB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.10.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220209004551-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 fa7d719263869edf5688048e364523a46a9c4fd8149df4b573ef16d19886c0f7
MD5 d2415ccf79f76d385a2ae62dcd9424db
BLAKE2b-256 80a05865545a4350da3c909aa7fc4bdadf4f44b2c3e68a28047f0b0e97ff68dd

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220209004551-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220209004551-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 21.8 MB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.10.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220209004551-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 78c363aec52daf51901d175e2add84e574750e2a0eda8eb85746bf05e961b715
MD5 af9ee7d6831406c6b58b033d6a44b4ef
BLAKE2b-256 6090bd6f54a4b085e14a5e61d649f649c6341f5719dbfd0bcdd06a5a78bbab0c

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220209004551-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220209004551-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 6912ba722ccff2ddc72be74e635eb2bdffe792ff23fde5be0cbb4938c27bab9d
MD5 286df3a196a25b374855b29267f78608
BLAKE2b-256 0d5c35253f1baf2870f5be4307d97face2c729aca86ce52bb04e56ff27fb29fc

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220209004551-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220209004551-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 24.1 MB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.10.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220209004551-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 76660cff76f60401c383d6d4ba56b70cfc9adccf17dcbaf388bfbd6390c4dab5
MD5 dc7c0107be4265c39f53050f348d9702
BLAKE2b-256 073a7d4c549b56ae9bdd409c178eac7bdefe05aec6fb82d9a9c1bf082fe362a8

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page