Skip to main content

TensorFlow IO

Project description




TensorFlow I/O

GitHub CI PyPI License Documentation

TensorFlow I/O is a collection of file systems and file formats that are not available in TensorFlow's built-in support. A full list of supported file systems and file formats by TensorFlow I/O can be found here.

The use of tensorflow-io is straightforward with keras. Below is an example to Get Started with TensorFlow with the data processing aspect replaced by tensorflow-io:

import tensorflow as tf
import tensorflow_io as tfio

# Read the MNIST data into the IODataset.
dataset_url = "https://storage.googleapis.com/cvdf-datasets/mnist/"
d_train = tfio.IODataset.from_mnist(
    dataset_url + "train-images-idx3-ubyte.gz",
    dataset_url + "train-labels-idx1-ubyte.gz",
)

# Shuffle the elements of the dataset.
d_train = d_train.shuffle(buffer_size=1024)

# By default image data is uint8, so convert to float32 using map().
d_train = d_train.map(lambda x, y: (tf.image.convert_image_dtype(x, tf.float32), y))

# prepare batches the data just like any other tf.data.Dataset
d_train = d_train.batch(32)

# Build the model.
model = tf.keras.models.Sequential(
    [
        tf.keras.layers.Flatten(input_shape=(28, 28)),
        tf.keras.layers.Dense(512, activation=tf.nn.relu),
        tf.keras.layers.Dropout(0.2),
        tf.keras.layers.Dense(10, activation=tf.nn.softmax),
    ]
)

# Compile the model.
model.compile(
    optimizer="adam", loss="sparse_categorical_crossentropy", metrics=["accuracy"]
)

# Fit the model.
model.fit(d_train, epochs=5, steps_per_epoch=200)

In the above MNIST example, the URL's to access the dataset files are passed directly to the tfio.IODataset.from_mnist API call. This is due to the inherent support that tensorflow-io provides for HTTP/HTTPS file system, thus eliminating the need for downloading and saving datasets on a local directory.

NOTE: Since tensorflow-io is able to detect and uncompress the MNIST dataset automatically if needed, we can pass the URL's for the compressed files (gzip) to the API call as is.

Please check the official documentation for more detailed and interesting usages of the package.

Installation

Python Package

The tensorflow-io Python package can be installed with pip directly using:

$ pip install tensorflow-io

People who are a little more adventurous can also try our nightly binaries:

$ pip install tensorflow-io-nightly

To ensure you have a version of TensorFlow that is compatible with TensorFlow-IO, you can specify the tensorflow extra requirement during install:

pip install tensorflow-io[tensorflow]

Similar extras exist for the tensorflow-gpu, tensorflow-cpu and tensorflow-rocm packages.

Docker Images

In addition to the pip packages, the docker images can be used to quickly get started.

For stable builds:

$ docker pull tfsigio/tfio:latest
$ docker run -it --rm --name tfio-latest tfsigio/tfio:latest

For nightly builds:

$ docker pull tfsigio/tfio:nightly
$ docker run -it --rm --name tfio-nightly tfsigio/tfio:nightly

R Package

Once the tensorflow-io Python package has been successfully installed, you can install the development version of the R package from GitHub via the following:

if (!require("remotes")) install.packages("remotes")
remotes::install_github("tensorflow/io", subdir = "R-package")

TensorFlow Version Compatibility

To ensure compatibility with TensorFlow, it is recommended to install a matching version of TensorFlow I/O according to the table below. You can find the list of releases here.

TensorFlow I/O Version TensorFlow Compatibility Release Date
0.24.0 2.8.x Feb 04, 2022
0.23.1 2.7.x Dec 15, 2021
0.23.0 2.7.x Dec 14, 2021
0.22.0 2.7.x Nov 10, 2021
0.21.0 2.6.x Sep 12, 2021
0.20.0 2.6.x Aug 11, 2021
0.19.1 2.5.x Jul 25, 2021
0.19.0 2.5.x Jun 25, 2021
0.18.0 2.5.x May 13, 2021
0.17.1 2.4.x Apr 16, 2021
0.17.0 2.4.x Dec 14, 2020
0.16.0 2.3.x Oct 23, 2020
0.15.0 2.3.x Aug 03, 2020
0.14.0 2.2.x Jul 08, 2020
0.13.0 2.2.x May 10, 2020
0.12.0 2.1.x Feb 28, 2020
0.11.0 2.1.x Jan 10, 2020
0.10.0 2.0.x Dec 05, 2019
0.9.1 2.0.x Nov 15, 2019
0.9.0 2.0.x Oct 18, 2019
0.8.1 1.15.x Nov 15, 2019
0.8.0 1.15.x Oct 17, 2019
0.7.2 1.14.x Nov 15, 2019
0.7.1 1.14.x Oct 18, 2019
0.7.0 1.14.x Jul 14, 2019
0.6.0 1.13.x May 29, 2019
0.5.0 1.13.x Apr 12, 2019
0.4.0 1.13.x Mar 01, 2019
0.3.0 1.12.0 Feb 15, 2019
0.2.0 1.12.0 Jan 29, 2019
0.1.0 1.12.0 Dec 16, 2018

Performance Benchmarking

We use github-pages to document the results of API performance benchmarks. The benchmark job is triggered on every commit to master branch and facilitates tracking performance w.r.t commits.

Contributing

Tensorflow I/O is a community led open source project. As such, the project depends on public contributions, bug-fixes, and documentation. Please see:

Build Status and CI

Build Status
Linux CPU Python 2 Status
Linux CPU Python 3 Status
Linux GPU Python 2 Status
Linux GPU Python 3 Status

Because of manylinux2010 requirement, TensorFlow I/O is built with Ubuntu:16.04 + Developer Toolset 7 (GCC 7.3) on Linux. Configuration with Ubuntu 16.04 with Developer Toolset 7 is not exactly straightforward. If the system have docker installed, then the following command will automatically build manylinux2010 compatible whl package:

#!/usr/bin/env bash

ls dist/*
for f in dist/*.whl; do
  docker run -i --rm -v $PWD:/v -w /v --net=host quay.io/pypa/manylinux2010_x86_64 bash -x -e /v/tools/build/auditwheel repair --plat manylinux2010_x86_64 $f
done
sudo chown -R $(id -nu):$(id -ng) .
ls wheelhouse/*

It takes some time to build, but once complete, there will be python 3.5, 3.6, 3.7 compatible whl packages available in wheelhouse directory.

On macOS, the same command could be used. However, the script expects python in shell and will only generate a whl package that matches the version of python in shell. If you want to build a whl package for a specific python then you have to alias this version of python to python in shell. See .github/workflows/build.yml Auditwheel step for instructions how to do that.

Note the above command is also the command we use when releasing packages for Linux and macOS.

TensorFlow I/O uses both GitHub Workflows and Google CI (Kokoro) for continuous integration. GitHub Workflows is used for macOS build and test. Kokoro is used for Linux build and test. Again, because of the manylinux2010 requirement, on Linux whl packages are always built with Ubuntu 16.04 + Developer Toolset 7. Tests are done on a variatiy of systems with different python3 versions to ensure a good coverage:

Python Ubuntu 18.04 Ubuntu 20.04 macOS + osx9 Windows-2019
2.7 :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: N/A
3.7 :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark:
3.8 :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark:

TensorFlow I/O has integrations with many systems and cloud vendors such as Prometheus, Apache Kafka, Apache Ignite, Google Cloud PubSub, AWS Kinesis, Microsoft Azure Storage, Alibaba Cloud OSS etc.

We tried our best to test against those systems in our continuous integration whenever possible. Some tests such as Prometheus, Kafka, and Ignite are done with live systems, meaning we install Prometheus/Kafka/Ignite on CI machine before the test is run. Some tests such as Kinesis, PubSub, and Azure Storage are done through official or non-official emulators. Offline tests are also performed whenever possible, though systems covered through offine tests may not have the same level of coverage as live systems or emulators.

Live System Emulator CI Integration Offline
Apache Kafka :heavy_check_mark: :heavy_check_mark:
Apache Ignite :heavy_check_mark: :heavy_check_mark:
Prometheus :heavy_check_mark: :heavy_check_mark:
Google PubSub :heavy_check_mark: :heavy_check_mark:
Azure Storage :heavy_check_mark: :heavy_check_mark:
AWS Kinesis :heavy_check_mark: :heavy_check_mark:
Alibaba Cloud OSS :heavy_check_mark:
Google BigTable/BigQuery to be added
Elasticsearch (experimental) :heavy_check_mark: :heavy_check_mark:
MongoDB (experimental) :heavy_check_mark: :heavy_check_mark:

References for emulators:

Community

Additional Information

License

Apache License 2.0

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

tensorflow_io_nightly-0.24.0.dev20220211221418-cp310-cp310-win_amd64.whl (21.8 MB view details)

Uploaded CPython 3.10 Windows x86-64

tensorflow_io_nightly-0.24.0.dev20220211221418-cp310-cp310-macosx_10_14_x86_64.whl (24.1 MB view details)

Uploaded CPython 3.10 macOS 10.14+ x86-64

tensorflow_io_nightly-0.24.0.dev20220211221418-cp39-cp39-win_amd64.whl (21.8 MB view details)

Uploaded CPython 3.9 Windows x86-64

tensorflow_io_nightly-0.24.0.dev20220211221418-cp39-cp39-macosx_10_14_x86_64.whl (24.1 MB view details)

Uploaded CPython 3.9 macOS 10.14+ x86-64

tensorflow_io_nightly-0.24.0.dev20220211221418-cp38-cp38-win_amd64.whl (21.8 MB view details)

Uploaded CPython 3.8 Windows x86-64

tensorflow_io_nightly-0.24.0.dev20220211221418-cp38-cp38-macosx_10_14_x86_64.whl (24.1 MB view details)

Uploaded CPython 3.8 macOS 10.14+ x86-64

tensorflow_io_nightly-0.24.0.dev20220211221418-cp37-cp37m-win_amd64.whl (21.8 MB view details)

Uploaded CPython 3.7m Windows x86-64

tensorflow_io_nightly-0.24.0.dev20220211221418-cp37-cp37m-macosx_10_14_x86_64.whl (24.1 MB view details)

Uploaded CPython 3.7m macOS 10.14+ x86-64

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220211221418-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220211221418-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 21.8 MB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.0 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220211221418-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 0b41a468ee202e4a0c45c7e4bd5fd5ec7561c62af1e49af9f39abd1a03368ffa
MD5 cacb648331202372d0ed15f21e85c4e3
BLAKE2b-256 1548f54d1b45749be6ff6201de0f3f3f096023d4cf21869fd2a6b6f2ec4503e6

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220211221418-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220211221418-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 d54fe7a1e2cb4f1cc4316bf25a02f018e22da1583f617b6803acd1fe23ac532c
MD5 832c7e92aeffb077d5560013a68f97fa
BLAKE2b-256 63a20fdaf9b3b8196ba8d42e01b2f4674f75a9f6dfda6dc87fa6f5f48e226d1c

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220211221418-cp310-cp310-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220211221418-cp310-cp310-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 24.1 MB
  • Tags: CPython 3.10, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.0 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220211221418-cp310-cp310-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 180e8adbd80fa875da0dde83b6a798871e0fb98d3860366828471d212b8f43b7
MD5 d970bbcc466ca86ef7ddd21e316cb3d2
BLAKE2b-256 ce6ee875ccb19bf459ac43de6bbb1d4a015615bddf033af0bc1289580a50bdfe

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220211221418-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220211221418-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 21.8 MB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.0 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220211221418-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 99c25ca0d49230925079b98c3f527ef30dde42edbfccefc377f104c3c7223362
MD5 5b0f0ca078c3992a1807831c290f9e65
BLAKE2b-256 10952bf15f52df206b95291a04a682d1cffa96e0ab1c92df4ae6560015fff225

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220211221418-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220211221418-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 ffeca7b78eb9dfe1477efbfc23ab573956d2842e2058bfb339c3facc370b84e2
MD5 b4aefe1edeb86664be357407bb027944
BLAKE2b-256 c1de8e8ee624cac68a17ec00892806d96baa3a2878277e5063f996e1b43db3d8

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220211221418-cp39-cp39-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220211221418-cp39-cp39-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 24.1 MB
  • Tags: CPython 3.9, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.0 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220211221418-cp39-cp39-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 4324e2b7552a16f26c05357054817c889f83a7c61020ca7d43564182a9e25cb9
MD5 3626fb545d031d3bd436c8b324bb07cf
BLAKE2b-256 02dd49447f47627dc1e773e21108a8bac061b18ff79f3954e926f43b75b19bbf

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220211221418-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220211221418-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 21.8 MB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.0 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220211221418-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 f3c2aebfaae318eca9829c776fd778848229232a94d4e3c3ac145ec0e708ed32
MD5 62cc4afe112620335de0b6df2bcd5cf8
BLAKE2b-256 0793a3619b953a3a44c865e6fb3089bafe0a86d4d0520361db1df0dedd62be01

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220211221418-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220211221418-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 7033c1d3a7f998edaacc867695e9ee7439595726488ff7138c6b198e16ac2e44
MD5 1a537db695c4f144a2a1243e79dcb425
BLAKE2b-256 be3d023f7d7ae318c553518bfed4cc20c417e1879b6f298abb7b5e158adee198

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220211221418-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220211221418-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 24.1 MB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.0 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220211221418-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 2478cb5b28a4d2fe78a5a86b7881c6fb5b5a3c47dfb93f0324d38f16a170d45a
MD5 a3e8144443b896a6f43e0a9fa7d388ed
BLAKE2b-256 5981afa403afa29e28e9c23602ac0252014c6d7dd10ea02210f5770c61105c7d

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220211221418-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220211221418-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 21.8 MB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.0 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220211221418-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 aefdf2ec91ae3bb77115e560b356e7019ecde93b7fd4c021992749358e68f7c8
MD5 ab505e118d648a54ceb20d013b93d2d0
BLAKE2b-256 5a088d3d1971a9d8752d7e40b3e3240b35ea691e8d7b49bf0ccb2cd95ab693b6

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220211221418-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220211221418-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 e315dbfb5b583e732c8cf97a1e80b4faad655ee7cb946d814cd944a3b52a99d4
MD5 9a3e6e39a8ee97a7fe99e078979ddb63
BLAKE2b-256 30402808e98412816b935b94ca7f10a33fe54af048bb83fb9bf0b6d8e0bd7f83

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220211221418-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220211221418-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 24.1 MB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.0 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220211221418-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 1fc5ded8a60a4c63c8114d35530817d6ff1071a7f3e00b5a257250dba168a85f
MD5 0fe368054119925a6e439daa2f80d2da
BLAKE2b-256 c5acb36400dbbeefb804db46210f481809ed86079c03bafb1b5125aeb7f5bc14

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page