aiohttp-client-cache

See full documentation at https://aiohttp-client-cache.readthedocs.io

aiohttp-client-cache is an async persistent cache for aiohttp requests, based on requests-cache.

Not to be confused with aiohttp-cache, which is a cache for the aiohttp web server. This package is, as you might guess, specifically for the aiohttp client.

Development Status

This is an early work in progress!

The current state is a working drop-in replacement (or mixin) for aiohttp.ClientSession, with multiple asynchronous cache backends.

Breaking changes should be expected until a 1.0 release.

Installation

Requires python 3.7+

Install the latest stable version with pip:

pip install aiohttp-client-cache

Note: You will need additional dependencies depending on which backend you want to use; See Cache Backends section below for details. To install with extra dependencies for all supported backends:

pip install aiohttp-client-cache[backends]

See Contributing for setup info for local development.

Usage Examples

See the examples folder for more detailed usage examples.

Here is a simple example using an endpoint that takes 1 second to fetch. After the first request, subsequent requests to the same URL will return near-instantly; so, fetching it 10 times will only take ~1 second instead of 10.

from aiohttp_client_cache import CachedSession, SQLiteBackend

async with CachedSession(cache=SQLiteBackend()) as session:
    for i in range(10):
        await session.get('http://httpbin.org/delay/1')

Here is an example with more customized behavior:

cache = SQLiteBackend(
    cache_name='~/.cache/aiohttp-requests.db',     # For SQLite, this will be used as the filename
    expire_after=24,                               # By default, cached responses expire in a day
    expire_after_urls={'*.site.com/static': 24*7}, # Requests with this pattern will expire in a week
    ignored_params=['auth_token'],                 # Ignore this param when caching responses
)

async with CachedSession(cache=cache) as session:
    await session.get('https://site.com/index.html')             # Expires in a day
    await session.get('https://img.site.com/static/a27bf6.jpg')  # Expires in a week

See CacheBackend for more usage details.

aiohttp-client-cache can also be used as a mixin, if you happen have other mixin classes that you want to combine with it:

from aiohttp import ClientSession
from aiohttp_client_cache import CacheMixin

class CustomSession(CacheMixin, CustomMixin, ClientSession):
    pass

Cache Backends

Several backends are available. If one isn't specified, a non-persistent in-memory cache will be used.

• SQLiteBackend: Uses a SQLite database (requires aiosqlite)
• RedisBackend: Uses a Redis cache (requires redis-py)
• MongoDBBackend: Uses a MongoDB database (requires motor)

Incomplete:

• DynamoDBBackend: Uses a Amazon DynamoDB database (requires boto3)
• GridFSBackend: Uses a MongoDB GridFS database, which enables storage of documents greater than 16MB (requires pymongo)

You can also provide your own backend by subclassing aiohttp_client_cache.backends.BaseCache.

Expiration

If you are using the expire_after parameter, expired responses are removed from the storage the next time the same request is made. If you want to manually purge all expired items, you can use CachedSession.delete_expired_responses. Example:

session = CachedSession(expire_after=3)   # Cached responses expire after 3 hours
await session.remove_expired_responses()  # Remove any responses over 3 hours old

Conditional Caching

Caching behavior can be customized by defining various conditions:

• Response status codes
• Request HTTP methods
• Request headers
• Specific request parameters
• Custom filter function

See CacheBackend docs for details.

Credits

Thanks to Roman Haritonov and contributors for the original requests-cache!

This project is licensed under the MIT license, with the exception of storage backend code adapted from requests-cache, which is licensed under the BSD license (copy included).

History

0.2.0 (2021-02-28)

• Refactor SQLite backend to use aiosqlite for async cache operations
• Refactor MongoDB backend to use motor for async cache operations
• Refactor Redis backend to use aiosqlite for async cache operations
• Add integration tests and docker-compose for local test servers

0.1.0 (2020-11-14)

• Initial PyPI release
• First pass at a general refactor and conversion from requests to aiohttp
• Basic features are functional, but some backends do not actually operate asynchronously

requests-cache

See requests-cache development history for details on prior changes.