Streaming read/writes to Google Storage blobs with ascynchronous buffering.
Project description
gs-chunked-io: Streams for Google Storage
gs-chunked-io provides transparently chunked io streams for google storage objects. Writable streams are managed as multipart objects that are composed when the stream is closed.
import gs_chunked_io as gscio
from google.cloud.storage import Client
client = Client()
bucket = client.bucket("my-bucket")
blob = bucket.get_blob("my-key)
# Readable stream:
with gscio.Reader(blob) as fh:
fh.read(size)
# Readable stream, download in background:
with gscio.AsyncReader(blob) as fh:
fh.read(size)
# Writable stream:
with gscio.Writer("my_new_key", bucket) as fh:
fh.write(data)
# Writable stream, upload in background:
with gscio.AsyncWriter("my_new_key", bucket) as fh:
fh.write(data)
# Process blob in chunks:
with gscio.Reader(blob) as reader:
for chunk in reader.for_each_chunk():
my_chunk_processor(chunk)
# Multipart copy with processing:
dst_bucket = client.bucket("my_dest_bucket")
with gscio.Writer("my_dest_key", dst_bucket) fh_write:
with gscio.AsyncReader(blob) as reader:
for chunk in reader.for_each_chunk(blob):
process_my_chunk(chunk)
fh_write(chunk)
# Extract .tar.gz on the fly:
import gzip
import tarfile
with gscio.AsyncReader(blob) as fh:
gzip_reader = gzip.GzipFile(fileobj=fh)
tf = tarfile.TarFile(fileobj=gzip_reader)
for tarinfo in tf:
process_my_tarinfo(tarinfo)
Installation
pip install gs-chunked-io
Links
Project home page GitHub
Package distribution PyPI
Bugs
Please report bugs, issues, feature requests, etc. on GitHub.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
gs-chunked-io-0.2.12.tar.gz
(5.1 kB
view details)
Built Distribution
File details
Details for the file gs-chunked-io-0.2.12.tar.gz.
File metadata
- Download URL: gs-chunked-io-0.2.12.tar.gz
- Upload date:
- Size: 5.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.7.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | e0a84fce2114d735298979c4fa08535fe3aef4ce50be252a1ca1e93dc24f0c87 |
|
| MD5 | f1db4d940aecb7d7580a0780ef7ccb22 |
|
| BLAKE2b-256 | fce102c198a4f361495c16afea8910533ab85f441dbc62522622ce9ec902d86f |
Provenance
File details
Details for the file gs_chunked_io-0.2.12-py3-none-any.whl.
File metadata
- Download URL: gs_chunked_io-0.2.12-py3-none-any.whl
- Upload date:
- Size: 7.1 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.7.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 45d7e5ae66880f4006e74b80bd07770e155ca32bb52366e418148f921bee4c26 |
|
| MD5 | 24c6dfb7860e575801a6d356cd2f3cf1 |
|
| BLAKE2b-256 | f44f2421c8fdf03e74e83ba7201a9db8a4dddfe5b91d05619f956eb7c695db66 |
