Read GCS and local paths with the same interface, clone of tensorflow.io.gfile
Project description
blobfile
This is a standalone clone of TensorFlow's gfile
, supporting both local paths and gs://
paths.
Writing to a remote path will not actually perform the write incrementally, so don't write to a log file this way. By default reads and writes are streamed, set streaming=False
to BlobFile
to do a single copy operation per file instead.
The main function is BlobFile
, a replacement for GFile
. There are also a few additional functions, basename
, dirname
, and join
, which mostly do the same thing as their os.path
namesakes, only they also support gs://
paths. There are also a few extra functions:
cache_key
- returns a cache key that can be used for the path (this is not guaranteed to change when the content changes, but should hopefully do that)retry
- useful for retrying since network operations often failget_url
- returns a url for a pathmd5
- get the md5 hash for a path, for GCS this is fast, but for other backends this may be slow
A number of existing gfile
functions are currently not implemented.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.