Skip to main content

TensorFlow IO

Project description




TensorFlow I/O

GitHub CI PyPI License Documentation

TensorFlow I/O is a collection of file systems and file formats that are not available in TensorFlow's built-in support. A full list of supported file systems and file formats by TensorFlow I/O can be found here.

The use of tensorflow-io is straightforward with keras. Below is an example to Get Started with TensorFlow with the data processing aspect replaced by tensorflow-io:

import tensorflow as tf
import tensorflow_io as tfio

# Read the MNIST data into the IODataset.
dataset_url = "https://storage.googleapis.com/cvdf-datasets/mnist/"
d_train = tfio.IODataset.from_mnist(
    dataset_url + "train-images-idx3-ubyte.gz",
    dataset_url + "train-labels-idx1-ubyte.gz",
)

# Shuffle the elements of the dataset.
d_train = d_train.shuffle(buffer_size=1024)

# By default image data is uint8, so convert to float32 using map().
d_train = d_train.map(lambda x, y: (tf.image.convert_image_dtype(x, tf.float32), y))

# prepare batches the data just like any other tf.data.Dataset
d_train = d_train.batch(32)

# Build the model.
model = tf.keras.models.Sequential(
    [
        tf.keras.layers.Flatten(input_shape=(28, 28)),
        tf.keras.layers.Dense(512, activation=tf.nn.relu),
        tf.keras.layers.Dropout(0.2),
        tf.keras.layers.Dense(10, activation=tf.nn.softmax),
    ]
)

# Compile the model.
model.compile(
    optimizer="adam", loss="sparse_categorical_crossentropy", metrics=["accuracy"]
)

# Fit the model.
model.fit(d_train, epochs=5, steps_per_epoch=200)

In the above MNIST example, the URL's to access the dataset files are passed directly to the tfio.IODataset.from_mnist API call. This is due to the inherent support that tensorflow-io provides for HTTP/HTTPS file system, thus eliminating the need for downloading and saving datasets on a local directory.

NOTE: Since tensorflow-io is able to detect and uncompress the MNIST dataset automatically if needed, we can pass the URL's for the compressed files (gzip) to the API call as is.

Please check the official documentation for more detailed and interesting usages of the package.

Installation

Python Package

The tensorflow-io Python package can be installed with pip directly using:

$ pip install tensorflow-io

People who are a little more adventurous can also try our nightly binaries:

$ pip install tensorflow-io-nightly

To ensure you have a version of TensorFlow that is compatible with TensorFlow-IO, you can specify the tensorflow extra requirement during install:

pip install tensorflow-io[tensorflow]

Similar extras exist for the tensorflow-gpu, tensorflow-cpu and tensorflow-rocm packages.

Docker Images

In addition to the pip packages, the docker images can be used to quickly get started.

For stable builds:

$ docker pull tfsigio/tfio:latest
$ docker run -it --rm --name tfio-latest tfsigio/tfio:latest

For nightly builds:

$ docker pull tfsigio/tfio:nightly
$ docker run -it --rm --name tfio-nightly tfsigio/tfio:nightly

R Package

Once the tensorflow-io Python package has been successfully installed, you can install the development version of the R package from GitHub via the following:

if (!require("remotes")) install.packages("remotes")
remotes::install_github("tensorflow/io", subdir = "R-package")

TensorFlow Version Compatibility

To ensure compatibility with TensorFlow, it is recommended to install a matching version of TensorFlow I/O according to the table below. You can find the list of releases here.

TensorFlow I/O Version TensorFlow Compatibility Release Date
0.24.0 2.8.x Feb 04, 2022
0.23.1 2.7.x Dec 15, 2021
0.23.0 2.7.x Dec 14, 2021
0.22.0 2.7.x Nov 10, 2021
0.21.0 2.6.x Sep 12, 2021
0.20.0 2.6.x Aug 11, 2021
0.19.1 2.5.x Jul 25, 2021
0.19.0 2.5.x Jun 25, 2021
0.18.0 2.5.x May 13, 2021
0.17.1 2.4.x Apr 16, 2021
0.17.0 2.4.x Dec 14, 2020
0.16.0 2.3.x Oct 23, 2020
0.15.0 2.3.x Aug 03, 2020
0.14.0 2.2.x Jul 08, 2020
0.13.0 2.2.x May 10, 2020
0.12.0 2.1.x Feb 28, 2020
0.11.0 2.1.x Jan 10, 2020
0.10.0 2.0.x Dec 05, 2019
0.9.1 2.0.x Nov 15, 2019
0.9.0 2.0.x Oct 18, 2019
0.8.1 1.15.x Nov 15, 2019
0.8.0 1.15.x Oct 17, 2019
0.7.2 1.14.x Nov 15, 2019
0.7.1 1.14.x Oct 18, 2019
0.7.0 1.14.x Jul 14, 2019
0.6.0 1.13.x May 29, 2019
0.5.0 1.13.x Apr 12, 2019
0.4.0 1.13.x Mar 01, 2019
0.3.0 1.12.0 Feb 15, 2019
0.2.0 1.12.0 Jan 29, 2019
0.1.0 1.12.0 Dec 16, 2018

Performance Benchmarking

We use github-pages to document the results of API performance benchmarks. The benchmark job is triggered on every commit to master branch and facilitates tracking performance w.r.t commits.

Contributing

Tensorflow I/O is a community led open source project. As such, the project depends on public contributions, bug-fixes, and documentation. Please see:

Build Status and CI

Build Status
Linux CPU Python 2 Status
Linux CPU Python 3 Status
Linux GPU Python 2 Status
Linux GPU Python 3 Status

Because of manylinux2010 requirement, TensorFlow I/O is built with Ubuntu:16.04 + Developer Toolset 7 (GCC 7.3) on Linux. Configuration with Ubuntu 16.04 with Developer Toolset 7 is not exactly straightforward. If the system have docker installed, then the following command will automatically build manylinux2010 compatible whl package:

#!/usr/bin/env bash

ls dist/*
for f in dist/*.whl; do
  docker run -i --rm -v $PWD:/v -w /v --net=host quay.io/pypa/manylinux2010_x86_64 bash -x -e /v/tools/build/auditwheel repair --plat manylinux2010_x86_64 $f
done
sudo chown -R $(id -nu):$(id -ng) .
ls wheelhouse/*

It takes some time to build, but once complete, there will be python 3.5, 3.6, 3.7 compatible whl packages available in wheelhouse directory.

On macOS, the same command could be used. However, the script expects python in shell and will only generate a whl package that matches the version of python in shell. If you want to build a whl package for a specific python then you have to alias this version of python to python in shell. See .github/workflows/build.yml Auditwheel step for instructions how to do that.

Note the above command is also the command we use when releasing packages for Linux and macOS.

TensorFlow I/O uses both GitHub Workflows and Google CI (Kokoro) for continuous integration. GitHub Workflows is used for macOS build and test. Kokoro is used for Linux build and test. Again, because of the manylinux2010 requirement, on Linux whl packages are always built with Ubuntu 16.04 + Developer Toolset 7. Tests are done on a variatiy of systems with different python3 versions to ensure a good coverage:

Python Ubuntu 18.04 Ubuntu 20.04 macOS + osx9 Windows-2019
2.7 :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: N/A
3.7 :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark:
3.8 :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark:

TensorFlow I/O has integrations with many systems and cloud vendors such as Prometheus, Apache Kafka, Apache Ignite, Google Cloud PubSub, AWS Kinesis, Microsoft Azure Storage, Alibaba Cloud OSS etc.

We tried our best to test against those systems in our continuous integration whenever possible. Some tests such as Prometheus, Kafka, and Ignite are done with live systems, meaning we install Prometheus/Kafka/Ignite on CI machine before the test is run. Some tests such as Kinesis, PubSub, and Azure Storage are done through official or non-official emulators. Offline tests are also performed whenever possible, though systems covered through offine tests may not have the same level of coverage as live systems or emulators.

Live System Emulator CI Integration Offline
Apache Kafka :heavy_check_mark: :heavy_check_mark:
Apache Ignite :heavy_check_mark: :heavy_check_mark:
Prometheus :heavy_check_mark: :heavy_check_mark:
Google PubSub :heavy_check_mark: :heavy_check_mark:
Azure Storage :heavy_check_mark: :heavy_check_mark:
AWS Kinesis :heavy_check_mark: :heavy_check_mark:
Alibaba Cloud OSS :heavy_check_mark:
Google BigTable/BigQuery to be added
Elasticsearch (experimental) :heavy_check_mark: :heavy_check_mark:
MongoDB (experimental) :heavy_check_mark: :heavy_check_mark:

References for emulators:

Community

Additional Information

License

Apache License 2.0

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

tensorflow_io_nightly-0.24.0.dev20220307180833-cp310-cp310-win_amd64.whl (21.8 MB view details)

Uploaded CPython 3.10 Windows x86-64

tensorflow_io_nightly-0.24.0.dev20220307180833-cp310-cp310-macosx_10_14_x86_64.whl (24.1 MB view details)

Uploaded CPython 3.10 macOS 10.14+ x86-64

tensorflow_io_nightly-0.24.0.dev20220307180833-cp39-cp39-win_amd64.whl (21.8 MB view details)

Uploaded CPython 3.9 Windows x86-64

tensorflow_io_nightly-0.24.0.dev20220307180833-cp39-cp39-macosx_10_14_x86_64.whl (24.1 MB view details)

Uploaded CPython 3.9 macOS 10.14+ x86-64

tensorflow_io_nightly-0.24.0.dev20220307180833-cp38-cp38-win_amd64.whl (21.8 MB view details)

Uploaded CPython 3.8 Windows x86-64

tensorflow_io_nightly-0.24.0.dev20220307180833-cp38-cp38-macosx_10_14_x86_64.whl (24.1 MB view details)

Uploaded CPython 3.8 macOS 10.14+ x86-64

tensorflow_io_nightly-0.24.0.dev20220307180833-cp37-cp37m-win_amd64.whl (21.8 MB view details)

Uploaded CPython 3.7m Windows x86-64

tensorflow_io_nightly-0.24.0.dev20220307180833-cp37-cp37m-macosx_10_14_x86_64.whl (24.1 MB view details)

Uploaded CPython 3.7m macOS 10.14+ x86-64

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220307180833-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220307180833-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 21.8 MB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/33.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.11.2 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220307180833-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 75431f4e6812dab4890b1b312e2495446218e68e75d8f2c1603ef175e3674b4c
MD5 e05da99957724247ab381c0c23405946
BLAKE2b-256 f78102216d57291144e0164edc94606743dade5450097e86b3cf786ffcf1332f

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220307180833-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220307180833-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 5fcef070f331bc678daa8f4b8b6d750dbe0d73b9096f751298688a50a52bb5f7
MD5 3d8e8fe0dc39135bdc337a0232bd4925
BLAKE2b-256 ffa1aa48836429fdfd127c9e691da952766dc00346d7032bcda80665985bd568

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220307180833-cp310-cp310-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220307180833-cp310-cp310-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 24.1 MB
  • Tags: CPython 3.10, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/33.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.11.2 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220307180833-cp310-cp310-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 0cf74146fc588e8d87f54dde32115b8ca3b336044b48d64361c1998225987e0b
MD5 b2d6c8a75c9b2f65decaf676324be047
BLAKE2b-256 1f222bf00adef1e93b5330ee08c9b2d7c7abe9bc6fe244745eff68045c1b76e9

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220307180833-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220307180833-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 21.8 MB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/33.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.11.2 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220307180833-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 358e8e8cf5561e43d4c235e7390d8926e6a5bc3bf802c7a31d0d83b2e43a28a7
MD5 b9ef80b5eb1ef1fa51b34df7a0c699b3
BLAKE2b-256 0c557fe1efa83283e3199229c5222f4adb351eb0a559d147a66ba148cf053fc8

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220307180833-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220307180833-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 d9257cdf1f245abd85876ff27b6f3e85169f045c4d3a25030f459dfa68845941
MD5 fd5a4854d81bd7fd64f92065ba734ccc
BLAKE2b-256 1a8ac9cc507fe7bee56eb67431095f6eec297e1f1792b45f35d3469dbf2f43e9

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220307180833-cp39-cp39-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220307180833-cp39-cp39-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 24.1 MB
  • Tags: CPython 3.9, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/33.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.11.2 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220307180833-cp39-cp39-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 1c8c6e77af415f978895706e65d91aaa3d64830eb67bf7511ad0364e507de874
MD5 fa4a65047b6aef1fb7335dd21ec6cfa4
BLAKE2b-256 125946b1f271a83a89a31e6d6e86cfbd49530b1398e8daba2616f772425af17a

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220307180833-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220307180833-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 21.8 MB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/33.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.11.2 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220307180833-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 6a68a2985f773ca47cbb7184ef20272cfd118f9917538a216a05bcf20861df53
MD5 5d4cbccee4bbb736454a1d8c11573400
BLAKE2b-256 77e0533fa37667ceb9f64c431c56c2f596946da9aa11ef1c999abcb6e03f37f4

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220307180833-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220307180833-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 5a9bfc208856eb5af083c56c3247636963dc5a8cd357fd4fab82ce940eb28264
MD5 6b02b56c7a68659844909b79c7038ed5
BLAKE2b-256 03ef7bce783135c83b8ac964c3fda58db3ec5089a2da9303fe83af662fdb39d4

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220307180833-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220307180833-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 24.1 MB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/33.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.11.2 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220307180833-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 0cb6a9cc77f73519854afc06c0047c9053010b7be5dbf3f5ef98d933e2b402ef
MD5 9207f38e8b934c0bf1ccb023f87d72e4
BLAKE2b-256 7183df94b48ae955ffca378bdb44ceafbbab854ef7b266eddfd3f1826f10fc6c

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220307180833-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220307180833-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 21.8 MB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/33.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.11.2 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220307180833-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 138559ed90f0d7a78d41e3916023edce3ba1e06180846cd3452d8901a13e9072
MD5 dc027fa5e15558ed92ec65d90a057de4
BLAKE2b-256 826e06ee878d1b1dd113b9d9f67d60e1f4c5ea925c718e2e80795636f1eb58e2

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220307180833-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220307180833-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 b9067f8f61188baf68e4584a59fae0f34da4a62deafbccf842a4b4a2ff846744
MD5 88a72851ad2ea3a89e640da6e2f60cfa
BLAKE2b-256 7e1c5432b84f6fe7614342da70f09bbd3e01c0ed7067cce29e132382b5abbfa8

See more details on using hashes here.

File details

Details for the file tensorflow_io_nightly-0.24.0.dev20220307180833-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: tensorflow_io_nightly-0.24.0.dev20220307180833-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 24.1 MB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/33.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.11.2 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for tensorflow_io_nightly-0.24.0.dev20220307180833-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 82be2eb2dbe721ab52e64ce4a627d58306109f2d2a3fa5466446ebf8cee79ded
MD5 a1f580e0ee1a0f1c22e229ac162e835c
BLAKE2b-256 6e48aee029279c12c029e03afe2221a9bfef03b155b5994f3d3ef5fa8073e9e1

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page