Skip to main content

TensorFlow IO

Project description




TensorFlow I/O

GitHub CI PyPI License Documentation

TensorFlow I/O is a collection of file systems and file formats that are not available in TensorFlow's built-in support. A full list of supported file systems and file formats by TensorFlow I/O can be found here.

The use of tensorflow-io is straightforward with keras. Below is an example to Get Started with TensorFlow with the data processing aspect replaced by tensorflow-io:

import tensorflow as tf
import tensorflow_io as tfio

# Read the MNIST data into the IODataset.
dataset_url = "https://storage.googleapis.com/cvdf-datasets/mnist/"
d_train = tfio.IODataset.from_mnist(
    dataset_url + "train-images-idx3-ubyte.gz",
    dataset_url + "train-labels-idx1-ubyte.gz",
)

# Shuffle the elements of the dataset.
d_train = d_train.shuffle(buffer_size=1024)

# By default image data is uint8, so convert to float32 using map().
d_train = d_train.map(lambda x, y: (tf.image.convert_image_dtype(x, tf.float32), y))

# prepare batches the data just like any other tf.data.Dataset
d_train = d_train.batch(32)

# Build the model.
model = tf.keras.models.Sequential(
    [
        tf.keras.layers.Flatten(input_shape=(28, 28)),
        tf.keras.layers.Dense(512, activation=tf.nn.relu),
        tf.keras.layers.Dropout(0.2),
        tf.keras.layers.Dense(10, activation=tf.nn.softmax),
    ]
)

# Compile the model.
model.compile(
    optimizer="adam", loss="sparse_categorical_crossentropy", metrics=["accuracy"]
)

# Fit the model.
model.fit(d_train, epochs=5, steps_per_epoch=200)

In the above MNIST example, the URL's to access the dataset files are passed directly to the tfio.IODataset.from_mnist API call. This is due to the inherent support that tensorflow-io provides for HTTP/HTTPS file system, thus eliminating the need for downloading and saving datasets on a local directory.

NOTE: Since tensorflow-io is able to detect and uncompress the MNIST dataset automatically if needed, we can pass the URL's for the compressed files (gzip) to the API call as is.

Please check the official documentation for more detailed and interesting usages of the package.

Installation

Python Package

The tensorflow-io Python package can be installed with pip directly using:

$ pip install tensorflow-io

People who are a little more adventurous can also try our nightly binaries:

$ pip install tensorflow-io-nightly

To ensure you have a version of TensorFlow that is compatible with TensorFlow-IO, you can specify the tensorflow extra requirement during install:

pip install tensorflow-io[tensorflow]

Similar extras exist for the tensorflow-gpu, tensorflow-cpu and tensorflow-rocm packages.

Docker Images

In addition to the pip packages, the docker images can be used to quickly get started.

For stable builds:

$ docker pull tfsigio/tfio:latest
$ docker run -it --rm --name tfio-latest tfsigio/tfio:latest

For nightly builds:

$ docker pull tfsigio/tfio:nightly
$ docker run -it --rm --name tfio-nightly tfsigio/tfio:nightly

R Package

Once the tensorflow-io Python package has been successfully installed, you can install the development version of the R package from GitHub via the following:

if (!require("remotes")) install.packages("remotes")
remotes::install_github("tensorflow/io", subdir = "R-package")

TensorFlow Version Compatibility

To ensure compatibility with TensorFlow, it is recommended to install a matching version of TensorFlow I/O according to the table below. You can find the list of releases here.

TensorFlow I/O Version TensorFlow Compatibility Release Date
0.24.0 2.8.x Feb 04, 2022
0.23.1 2.7.x Dec 15, 2021
0.23.0 2.7.x Dec 14, 2021
0.22.0 2.7.x Nov 10, 2021
0.21.0 2.6.x Sep 12, 2021
0.20.0 2.6.x Aug 11, 2021
0.19.1 2.5.x Jul 25, 2021
0.19.0 2.5.x Jun 25, 2021
0.18.0 2.5.x May 13, 2021
0.17.1 2.4.x Apr 16, 2021
0.17.0 2.4.x Dec 14, 2020
0.16.0 2.3.x Oct 23, 2020
0.15.0 2.3.x Aug 03, 2020
0.14.0 2.2.x Jul 08, 2020
0.13.0 2.2.x May 10, 2020
0.12.0 2.1.x Feb 28, 2020
0.11.0 2.1.x Jan 10, 2020
0.10.0 2.0.x Dec 05, 2019
0.9.1 2.0.x Nov 15, 2019
0.9.0 2.0.x Oct 18, 2019
0.8.1 1.15.x Nov 15, 2019
0.8.0 1.15.x Oct 17, 2019
0.7.2 1.14.x Nov 15, 2019
0.7.1 1.14.x Oct 18, 2019
0.7.0 1.14.x Jul 14, 2019
0.6.0 1.13.x May 29, 2019
0.5.0 1.13.x Apr 12, 2019
0.4.0 1.13.x Mar 01, 2019
0.3.0 1.12.0 Feb 15, 2019
0.2.0 1.12.0 Jan 29, 2019
0.1.0 1.12.0 Dec 16, 2018

Performance Benchmarking

We use github-pages to document the results of API performance benchmarks. The benchmark job is triggered on every commit to master branch and facilitates tracking performance w.r.t commits.

Contributing

Tensorflow I/O is a community led open source project. As such, the project depends on public contributions, bug-fixes, and documentation. Please see:

Build Status and CI

Build Status
Linux CPU Python 2 Status
Linux CPU Python 3 Status
Linux GPU Python 2 Status
Linux GPU Python 3 Status

Because of manylinux2010 requirement, TensorFlow I/O is built with Ubuntu:16.04 + Developer Toolset 7 (GCC 7.3) on Linux. Configuration with Ubuntu 16.04 with Developer Toolset 7 is not exactly straightforward. If the system have docker installed, then the following command will automatically build manylinux2010 compatible whl package:

#!/usr/bin/env bash

ls dist/*
for f in dist/*.whl; do
  docker run -i --rm -v $PWD:/v -w /v --net=host quay.io/pypa/manylinux2010_x86_64 bash -x -e /v/tools/build/auditwheel repair --plat manylinux2010_x86_64 $f
done
sudo chown -R $(id -nu):$(id -ng) .
ls wheelhouse/*

It takes some time to build, but once complete, there will be python 3.5, 3.6, 3.7 compatible whl packages available in wheelhouse directory.

On macOS, the same command could be used. However, the script expects python in shell and will only generate a whl package that matches the version of python in shell. If you want to build a whl package for a specific python then you have to alias this version of python to python in shell. See .github/workflows/build.yml Auditwheel step for instructions how to do that.

Note the above command is also the command we use when releasing packages for Linux and macOS.

TensorFlow I/O uses both GitHub Workflows and Google CI (Kokoro) for continuous integration. GitHub Workflows is used for macOS build and test. Kokoro is used for Linux build and test. Again, because of the manylinux2010 requirement, on Linux whl packages are always built with Ubuntu 16.04 + Developer Toolset 7. Tests are done on a variatiy of systems with different python3 versions to ensure a good coverage:

Python Ubuntu 18.04 Ubuntu 20.04 macOS + osx9 Windows-2019
2.7 :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: N/A
3.7 :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark:
3.8 :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark:

TensorFlow I/O has integrations with many systems and cloud vendors such as Prometheus, Apache Kafka, Apache Ignite, Google Cloud PubSub, AWS Kinesis, Microsoft Azure Storage, Alibaba Cloud OSS etc.

We tried our best to test against those systems in our continuous integration whenever possible. Some tests such as Prometheus, Kafka, and Ignite are done with live systems, meaning we install Prometheus/Kafka/Ignite on CI machine before the test is run. Some tests such as Kinesis, PubSub, and Azure Storage are done through official or non-official emulators. Offline tests are also performed whenever possible, though systems covered through offine tests may not have the same level of coverage as live systems or emulators.

Live System Emulator CI Integration Offline
Apache Kafka :heavy_check_mark: :heavy_check_mark:
Apache Ignite :heavy_check_mark: :heavy_check_mark:
Prometheus :heavy_check_mark: :heavy_check_mark:
Google PubSub :heavy_check_mark: :heavy_check_mark:
Azure Storage :heavy_check_mark: :heavy_check_mark:
AWS Kinesis :heavy_check_mark: :heavy_check_mark:
Alibaba Cloud OSS :heavy_check_mark:
Google BigTable/BigQuery to be added
Elasticsearch (experimental) :heavy_check_mark: :heavy_check_mark:
MongoDB (experimental) :heavy_check_mark: :heavy_check_mark:

References for emulators:

Community

Additional Information

License

Apache License 2.0

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

tensorflow_io_nightly-0.24.0.dev20220208203249-cp310-cp310-win_amd64.whl (21.8 MB view details)

Uploaded CPython 3.10 Windows x86-64

tensorflow_io_nightly-0.24.0.dev20220208203249-cp310-cp310-macosx_10_14_x86_64.whl (24.1 MB view details)

Uploaded CPython 3.10 macOS 10.14+ x86-64

tensorflow_io_nightly-0.24.0.dev20220208203249-cp39-cp39-win_amd64.whl (21.8 MB view details)

Uploaded CPython 3.9 Windows x86-64

tensorflow_io_nightly-0.24.0.dev20220208203249-cp39-cp39-macosx_10_14_x86_64.whl (24.1 MB view details)

Uploaded CPython 3.9 macOS 10.14+ x86-64

tensorflow_io_nightly-0.24.0.dev20220208203249-cp38-cp38-win_amd64.whl (21.8 MB view details)

Uploaded CPython 3.8 Windows x86-64

tensorflow_io_nightly-0.24.0.dev20220208203249-cp38-cp38-macosx_10_14_x86_64.whl (24.1 MB view details)

Uploaded CPython 3.8 macOS 10.14+ x86-64

tensorflow_io_nightly-0.24.0.dev20220208203249-cp37-cp37m-win_amd64.whl (21.8 MB view details)

Uploaded CPython 3.7m Windows x86-64

tensorflow_io_nightly-0.24.0.dev20220208203249-cp37-cp37m-macosx_10_14_x86_64.whl (24.1 MB view details)

Uploaded CPython 3.7m macOS 10.14+ x86-64

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220208203249-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220208203249-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 21.8 MB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.10.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220208203249-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 ac99443b9f2e4784aaab3022b51ca96036c458f0ac95cdb2af6de6aba0a4fd62
MD5 44a3d472de1453685e4c1471426de112
BLAKE2b-256 4c1b3fa0a2b18ceb7a0203d1aa1e51e5b3c26ad28bd0a2a9e987f9418eb58506

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220208203249-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220208203249-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 b67f8fb274f2a0cc3e571c51cfc23409ee9328241719020e7824b15dc1fc39f7
MD5 b8c72f1b6e4e9201098bf760779b3ce8
BLAKE2b-256 f3f5bd16c32d51b9a228dd5f4fbfb092f5067ccbc6dbc25a6ebeeff249059e97

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220208203249-cp310-cp310-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220208203249-cp310-cp310-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 24.1 MB
  • Tags: CPython 3.10, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.10.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220208203249-cp310-cp310-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 0f4354a702f35a605fde13e2917018b8ce64e54b4a1fd42b02bac1a5f60f13db
MD5 1fbf3644a062a26e2bd844cfae53ac0e
BLAKE2b-256 cdffdbebda7d361d7d832a3eeefe1e5a5a82e77a010c2a34cf6a7a33c4b49eec

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220208203249-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220208203249-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 21.8 MB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.10.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220208203249-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 916d60e485ba7eea08a04cb338b6ff8cb093ad80c56a3f04cf895f8df0904842
MD5 6e9483b368827ecacc29196698d62140
BLAKE2b-256 592dcb6f5f908d692199f6e2835212e419cdbd2a40e9e62d0b7602a2d3538a88

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220208203249-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220208203249-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 013d39ef9558ab94b8c3c6302c53180aab1f97a135c5daf747122baa032e1307
MD5 766d0a64bfc4184fa673ef55f4184659
BLAKE2b-256 b14d05ef985d0842eaabbb3a77e09f963a6ac14eae5e5d3ffcd11fa9a057bcd7

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220208203249-cp39-cp39-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220208203249-cp39-cp39-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 24.1 MB
  • Tags: CPython 3.9, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.10.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220208203249-cp39-cp39-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 f34b13342cd447612686a016f7550e906204ceb09605334da7700b01780ba35a
MD5 332f0ef9b76638579a7912ca8533e2cd
BLAKE2b-256 88d49afd3fed25dd4aad43b5df865daf9ba9777323a5f3420f5cf17d28446d85

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220208203249-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220208203249-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 21.8 MB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.10.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220208203249-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 2e619867a46d6eb1a98729a588a7842481860be2972a0e25be588ccf8aa7644d
MD5 044c21e3d4086dc05669f211277bfd22
BLAKE2b-256 793a12175e257cf63ee5fc506ab4cd7e0486121df169e738efada1676ddfc83f

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220208203249-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220208203249-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 a29f4ec924201d92a32e1efe8e81c35075abcc5b3a1eb3fc455fbd083ec9e063
MD5 dc47b64b132fee941664840e206486ef
BLAKE2b-256 c4a7b25cfdaa212d8fb15e977b9e1567c33d7b50db787b7ce9606d4a6b8b9e62

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220208203249-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220208203249-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 24.1 MB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.10.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220208203249-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 2ca3cb310c726bedd74fa61f4613ac6b9d75b3a66b1613b0e937ca7b456ff57d
MD5 54e1f268250cdc7bbd6c9d343808ec26
BLAKE2b-256 f75db683dadf262b5d2fec939a036b7146567dfb891ae2becef5c72d6b7026e7

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220208203249-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220208203249-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 21.8 MB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.10.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220208203249-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 2bcc41f07fffecf2adfeea3baccf2d3c74449fe21bdd74f048b318bf29cfcef0
MD5 f0cbfa75a3738d620dbf8451bddeae19
BLAKE2b-256 7691291ddd5a27bc18fcf01f1bacea4d8004d1c19d08ddc593e18d03c1e592dc

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220208203249-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220208203249-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 a150a8f119a17d451dcf8d3eb79a0febcf3330aef07d89343cdd43c8942ed16f
MD5 b2589255802debe371ff5379b88e8836
BLAKE2b-256 30a9d945fd91659bd7cb4b28cc2b95d0050f80d4a1c9bf8cc51221983c8cdf38

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220208203249-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220208203249-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 24.1 MB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.10.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220208203249-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 38114848b1fea53ac446220c01fd032a8bec3b8cdecaeecdb2bb01be197aa02f
MD5 ca7ccdd06047f933225d78c1602f6432
BLAKE2b-256 f9185cf36d750a69f20d0b4a96f8688ed1f8ad49d57183a512b2196f182764a2

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page