File System Backends for Dogpile Cache

Filesystem-based backends for dogpile cache.

The generic variant of the backend, paylogic.filesystem, will accept any picklable value and it will store it in the file system.

The raw variant paylogic.raw_filesystem will only work with file-like values and it will avoid the pickling phase. This is useful when you are generating a big file and you don't want to keep in memory the contents of this file.

Both variants use fcntl.lockf operations, therefore it is compatible with UNIX-like systems only. The lockf system call allows to allocate an arbitrary number of locks using the same file, avoiding problems that arise when deleting lock files.

Installation

Install with pip:

$ pip install dogpile_filesystem

Usage

Generic variant

Configure a region to use paylogic.filesystem:

from dogpile.cache import make_region
import datetime

region = make_region().configure(
    'paylogic.filesystem',
    arguments = {
        "base_dir": "/path/to/cachedir",  # Make sure this directory is only for this region
        # Optional parameters
        "cache_size": 1024**3,  # Defaults to 1 Gb
        "expiration_time": datetime.timedelta(seconds=30),  # Defaults to no expiration
        "distributed_lock": True,  # Defaults to true
    }
)

@region.cache_on_arguments()
def my_function(args):
    return 42

Raw variant

Configure a region to use dogpile_filesystem:

from dogpile.cache import make_region
import datetime
import tempfile

region = make_region().configure(
    'paylogic.raw_filesystem',
    arguments = {
        "base_dir": "/path/to/cachedir",  # Make sure this directory is only for this region
        # Optional parameters
        "cache_size": 1024**3,  # Defaults to 1 Gb
        "file_movable": True,  # Whether the backend can freely move the file.
                               # When True, the backend will move the file to the cache
                               # directory directly using os.rename(file.name).
                               # When False (default), the content of the file will be copied to 
                               # the cache directory.
        "expiration_time": datetime.timedelta(seconds=30),  # Defaults to no expiration
        "distributed_lock": True,  # Defaults to true
    }
)

@region.cache_on_arguments()
def big_file_operation(args):
    # When using `file_movable=True`, we must make sure that NamedTemporaryFile does not delete the file on close,
    # otherwise it will complain that it cannot find the file.
    f = tempfile.NamedTemporaryFile(delete=False)
    # fill the file
    f.flush()
    f.seek(0)
    return f

Development

Install the dev requirements and the project in development mode:

$ pip install -r requirements_dev.txt -e .

Run tests:

$ pytest tests

Optionally run tests for all supported configurations:

$ tox

Changelog

All notable changes to this project will be documented in this file.

This project adheres to Semantic Versioning.

Unreleased

0.2.0

• Removed logic that falls back to copying a file when file_movable=True but the file does not appear to be movable. Do not try to be smart.

0.1.1

• Let PyPI know that we use markdown

0.1.0

• Initial release