Skip to main content

JSON Cache Loader

Project description

jsoncache

Python cache control for cloud storage models

This library exposes a multithreaded JSON object loader that support Amazon S3 and Google Cloud Storage.

Why do I care?

Because loading JSON files from the cloud is more annoying than you realize.

  • Sometimes you're gonna get errors - log those errors.
  • Sometimes you're going to have compressed JSON blobs because Google Cloud Storage has unmanageable timeouts for uploads (https://github.com/googleapis/python-storage/issues/74)
  • You want your application to behave as if read errors from the cloud weren't a problem, but you want those errors to show up in logging.

Quick Start

  1. Import the ThreadedObjectCache class.
  2. Instantiate it passing in the cloud type, bucket, path and time to live in seconds.
  3. Call .get() on the ThreadedObjectCache instace.

You can optionally pass in a custom implementation of the time module to override how time.time() works.

You can optionally pass in a custom callable transformer that will apply the transformer function to the data before it's returned. Typical use cases might involve initializing a sklearn model.

You can optionally pass in block_until_cached=True so that the constructor will block until a model is loaded successfully from the network.

All background threads are marked as daemon threads so using this code won't cause your application to wait for thread death.

Python 3.7.8 | packaged by conda-forge | (default, Jul 31 2020, 02:37:09)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.17.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from jsoncache import *

In [2]: t = ThreadedObjectCache('s3', 'telemetry-parquet', 'taar/similarity/lr_curves.json', 10)

In [3]: 2020-08-05 16:07:14,369 - botocore.credentials - INFO - Found credentials in environment variables.
In [3]:

In [3]: t.get()
Out[3]:
[[0.0, [0.029045735469752962, 0.02468400347868071]],
 [0.005000778819764661, [0.029530930135620918, 0.025088940785616222]],
 ...

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mozilla-jsoncache-0.1.7.tar.gz (7.9 kB view details)

Uploaded Source

File details

Details for the file mozilla-jsoncache-0.1.7.tar.gz.

File metadata

  • Download URL: mozilla-jsoncache-0.1.7.tar.gz
  • Upload date:
  • Size: 7.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.2.1.post20200802 requests-toolbelt/0.9.1 tqdm/4.48.2 CPython/3.7.8

File hashes

Hashes for mozilla-jsoncache-0.1.7.tar.gz
Algorithm Hash digest
SHA256 f7ec7dfdb43ed0bb7312bed414199a8265ffdbc355ab7ee5248645a267931b98
MD5 48cd871ca635da67dba733edf7288dbf
BLAKE2b-256 727a881640be10888832826c1ff08ce736742f6ac78d0aa3eaeb3b545f9f26f4

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page