Read GCS and local paths with the same interface, clone of tensorflow.io.gfile
Project description
blobfile
This is a standalone clone of TensorFlow's gfile, supporting both local paths and gs:// (Google Cloud Storage) paths.
The main function is BlobFile, a replacement for GFile. There are also a few additional functions, basename, dirname, and join, which mostly do the same thing as their os.path namesakes, only they also support gs:// paths.
Installation:
pip install blobfile
Usage:
import blobfile as bf
with bf.BlobFile("gs://my-bucket-name/cats", "wb") as w:
w.write(b"meow!")
Here are the functions:
BlobFile- likeopen()but works withgs://paths too, data can be streamed to/from the remote file. It accepts the following arguments:streaming:- The default for
streamingisTruewhenmodeis in"r", "rb"andFalsewhenmodeis in"w", "wb", "a", "ab". streaming=True:- Reading is done without downloading the entire remote file.
- Writing is done to the remote file directly, but only in chunks of a few MB in size.
flush()will not cause an early write. - Appending is not implemented.
streaming=False:- Reading is done by downloading the remote file to a local file during the constructor.
- Writing is done by uploading the file on
close()or during destruction. - Appending is done by downloading the file during construction and uploading on
close().
- The default for
buffer_size: number of bytes to buffer, this can potentially make reading more efficient.cache_dir: a directory in which to cache files for reading, only valid ifstreaming=Falseandmodeis in"r", "rb". You are reponsible for cleaning up the cache directory.
Some are inspired by existing os.path and shutil functions:
copy- copy a file from one path to another, will do a remote copy between two remote paths on the same blob storage serviceexists- returnsTrueif the file or directory existsglob- return files matching a glob-style pattern as a generator. Globs can have surprising performance characteristics when used with blob storage. Character ranges are not supported in patterns.isdir- returnsTrueif the path is a directorylistdir- list contents of a directory as a generatormakedirs- ensure that a directory and all parent directories existremove- remove a filermdir- remove an empty directoryrmtree- remove a directory treestat- get the size and modification time of a filewalk- walk a directory tree with a generator that yields(dirpath, dirnames, filenames)tuplesbasename- get the final component of a pathdirname- get the path except for the final componentjoin- join 2 or more paths together, inserting directory separators between each component
There are a few bonus functions:
get_url- returns a url for a path along with the expiration for that url (or None)md5- get the md5 hash for a path, for GCS this is fast, but for other backends this may be slowset_log_callback- set a log callback functionlog(msg: string)to use instead of printing to stdout
Errors
Error- base class for library-specific exceptionsRequestFailure- a request has failed permanently, hasmessage:str,request:Request, andresponse:urllib3.HTTPResponseattributes.- The following generic exceptions are raised from some functions to make the behavior similar to the original versions:
FileNotFoundError,FileExistsError,IsADirectoryError,NotADirectoryError,OSError,ValueError,io.UnsupportedOperation
Examples
Write and read a file:
import blobfile as bf
with bf.BlobFile("gs://my-bucket/file.name", "wb") as f:
f.write(b"meow")
print("exists:", bf.exists("gs://my-bucket/file.name"))
print("contents:", bf.BlobFile("gs://my-bucket/file.name", "rb").read())
Parallel execution:
import blobfile as bf
import multiprocessing as mp
import tqdm
filenames = [f"{i}.ext" for i in range(1000)]
with mp.Pool() as pool:
for filename, exists in tqdm.tqdm(zip(filenames, pool.imap(bf.exists, filenames)), total=len(filenames)):
pass
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
No source distribution files available for this release.See tutorial on generating distribution archives.
Built Distribution
blobfile-0.16.2-py3-none-any.whl
(38.9 kB
view hashes)
Close
Hashes for blobfile-0.16.2-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 9ac5292fd70a989472556277de1d69dad02698886defa9b189a4c092b88fff7c |
|
| MD5 | 778a6a7c0f9c7d0167a4ee5440018bab |
|
| BLAKE2b-256 | 6c6ccea7e0248382b556d7e65c2b60d201275f1c9e5b4a539bca8b2d24f5a2d7 |
