Read GCS and local paths with the same interface, clone of tensorflow.io.gfile
Project description
blobfile
This is a standalone clone of TensorFlow's gfile, supporting both local paths and gs:// (Google Cloud Storage) paths.
The main function is BlobFile, a replacement for GFile. There are also a few additional functions, basename, dirname, and join, which mostly do the same thing as their os.path namesakes, only they also support gs:// paths.
By default reads copy the entire source file on creation and writes on close(). Set streaming=True to BlobFile to stream reads and writes instead. GCS files are written in large chunks though, so be careful if you do a log file this way as the end could be truncated.
Example usage:
import blobfile as bf
with bf.BlobFile("gs://my-bucket-name/cats", "wb") as w:
w.write(b"meow!")
Here are the functions:
BlobFile- likeopen()but works withgs://paths toocopy- copy a file from one path to anotherexists- returnsTrueif the file or directory existsglob- return files matching a pattern, on GCS this only supports the*operator and can be slow if the*appears early in the pattern since GCS can only do prefix matches, all additional filtering must happen locallyisdir- returnsTrueif the path is a directorylistdir- list contents of a directorymakedirs- ensure that a directory and all parent directories existremove- remove a filerename- move a file from one path to another (source and destination must be both local or both on GCS), not atomic on GCScopytree- copy a directory tree from one path to anotherrmtree- remove a directory treestat- get the size and modification time of a filewalk- walk a directory tree, yielding(dirpath, dirnames, filenames tuples)basename- get the final component of a pathdirname- get the path except for the final componentjoin- join 2 or more paths together, inserting directory separators between each componentcache_key- returns a cache key that can be used for the path (this is not guaranteed to change when the content changes, but should hopefully do that)get_url- returns a url for a pathmd5- get the md5 hash for a path, for GCS this is fast, but for other backends this may be slowset_log_callback- set a log callback functionlog(msg: string)to use instead of printing to stdout
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
File details
Details for the file blobfile-0.3.3-py3-none-any.whl.
File metadata
- Download URL: blobfile-0.3.3-py3-none-any.whl
- Upload date:
- Size: 21.0 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.7.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | f1bd607d1437fd43efaba45d51cb69ea553d4733d97f3249071c072dc5f8a5b8 |
|
| MD5 | deed69bd7d4adc313e7d06f2a3994c4d |
|
| BLAKE2b-256 | f5a1f7fb95112a22c3b57710b8b9453ca96a67559a86f59c2c4335c36af1b81f |
