aiohttp-client-cache

aiohttp-client-cache is an async persistent cache for aiohttp client requests.

See full documentation at https://aiohttp-client-cache.readthedocs.io

Development Status

This is an early work in progress!

The current state is a working drop-in replacement (or mixin) for aiohttp.ClientSession, with multiple asynchronous cache backends.

Breaking changes should be expected until a 1.0 release.

Installation

Requires python 3.7+

Install the latest stable version with pip:

pip install aiohttp-client-cache

Note: You will need additional dependencies depending on which backend you want to use; See Cache Backends section below for details. To install with extra dependencies for all supported backends:

pip install aiohttp-client-cache[backends]

See Contributing for setup info for local development.

Usage Examples

See the examples folder for more detailed usage examples.

Here is a simple example using an endpoint that takes 1 second to fetch. After the first request, subsequent requests to the same URL will return near-instantly; so, fetching it 10 times will only take ~1 second instead of 10.

from aiohttp_client_cache import CachedSession, SQLiteBackend

async with CachedSession(cache=SQLiteBackend()) as session:
    for i in range(10):
        await session.get('http://httpbin.org/delay/1')

Here is an example with more customized behavior:

cache = SQLiteBackend(
    cache_name='~/.cache/aiohttp-requests.db',     # For SQLite, this will be used as the filename
    expire_after=24,                               # By default, cached responses expire in a day
    expire_after_urls={'*.site.com/static': 24*7}, # Requests with this pattern will expire in a week
    ignored_params=['auth_token'],                 # Ignore this param when caching responses
)

async with CachedSession(cache=cache) as session:
    await session.get('https://site.com/index.html')             # Expires in a day
    await session.get('https://img.site.com/static/a27bf6.jpg')  # Expires in a week

See CacheBackend documentation for more usage details.

aiohttp-client-cache can also be used as a mixin, if you happen have other mixin classes that you want to combine with it:

from aiohttp import ClientSession
from aiohttp_client_cache import CacheMixin

class CustomSession(CacheMixin, CustomMixin, ClientSession):
    pass

Cache Backends

Several backends are available. If one isn't specified, a non-persistent in-memory cache will be used.

• SQLiteBackend: Uses a SQLite database (requires aiosqlite)
• RedisBackend: Uses a Redis cache (requires redis-py)
• MongoDBBackend: Uses a MongoDB database (requires motor)

Incomplete:

• DynamoDBBackend: Uses a Amazon DynamoDB database (requires boto3)
• GridFSBackend: Uses a MongoDB GridFS database, which enables storage of documents greater than 16MB (requires pymongo)

You can also provide your own backend by subclassing aiohttp_client_cache.backends.BaseCache.

Expiration

If you are using the expire_after parameter, expired responses are removed from the storage the next time the same request is made. If you want to manually purge all expired items, you can use CachedSession.delete_expired_responses. Example:

session = CachedSession(expire_after=3)   # Cached responses expire after 3 hours
await session.remove_expired_responses()  # Remove any responses over 3 hours old

Conditional Caching

Caching behavior can be customized by defining various conditions:

• Response status codes
• Request HTTP methods
• Request headers
• Specific request parameters
• Custom filter function

See CacheBackend docs for details.

Related Projects

Other python cache projects you may want to check out:

• aiohttp-cache: A server-side async HTTP cache for the aiohttp web server
• diskcache: A general-purpose (not HTTP-specific) file-based cache built on SQLite
• aiocache: General-purpose (not HTTP-specific) async cache backends
• requests-cache An HTTP cache for requests; also served as a starting point for making aiohttp-client-cache
• CacheControl: An HTTP cache for requests that caches according to uses HTTP headers and status codes

Credits

Thanks to Roman Haritonov and contributors for the original requests-cache!

This project is licensed under the MIT license, with the exception of portions of storage backend code adapted from requests-cache, which is licensed under the BSD license (copy included).

History

0.3.0 (TBD)

See all issues & PRs here

• Add support for setting different expiration times based on URL patterns
• Add support for serializing/deserializing ClientSession.links
• Add case-insensitive response headers for compatibility with aiohttp.ClientResponse.headers

0.2.0 (2021-02-28)

See all issues & PRs here

• Refactor SQLite backend to use aiosqlite for async cache operations
• Refactor MongoDB backend to use motor for async cache operations
• Refactor Redis backend to use aiosqlite for async cache operations
• Add integration tests and docker-compose for local test servers

0.1.0 (2020-11-14)

• Initial PyPI release
• First pass at a general refactor and conversion from requests to aiohttp
• Basic features are functional, but some backends do not actually operate asynchronously

requests-cache

See requests-cache development history for details on prior changes.