async_generator 1.0

The async_generator library

This is a tiny library to add “async generators” to Python 3.5. What are those?

Option 1: my 5-minute lightning talk demo from PyCon 2015

Option 2: read on!

Python’s iterators are great to use – but manually implementing the iterator protocol (__iter__, __next__) can be very annoying. No-one wants to do that all the time.

Fortunately, Python has generators, which make it easy and straightforward to create an iterator by writing a function. E.g., if you have a file where each line is a JSON document, you can make an iterator over the decoded bodies with:

def load_json_lines(fileobj):
    for line in fileobj:
        yield json.loads(line)

Starting in v3.5, Python has added *async iterators* and *async functions*. These are like regular iterators and functions, except that they have magic powers that let them do asynchronous I/O without twisting your control flow into knots.

Asynchronous I/O code is all about incrementally processing streaming data, so async iterators are super handy. But manually implementing the async iterator protocol (__aiter__, __anext__) can be very annoying, which is why we want async generators, which make it easy to create an async iterator by writing an async function. For example, suppose that in our example above, we want to read the documents from a network connection, instead of the local filesystem. Using the asyncio.StreamReader interface we can write:

async def load_json_lines(asyncio_stream_reader):
    async for line in asyncio_stream_reader:
        async yield json.loads(line)

BUT! the above DOESN’T WORK in Python 3.5 – you just get a syntax error. In 3.5, the only way to make an async generator is to manually define __aiter__ and __anext__.

Until now.

This is a little library which implements async generators in Python 3.5, by emulating the above syntax. The two changes are that you have to decorate your async generators with @async_generator, and instead of writing async yield x you write await yield_(x):

# Same example as before, but works in Python 3.5
from async_generator import async_generator, yield_

@async_generator
async def load_json_lines(asyncio_stream_reader):
    async for line in asyncio_stream_reader:
        await yield_(json.loads(line))

Semantics

In addition to being super handy, this library is intended to help us get some experience with writing and using async generators, so that we can figure out exactly what semantics they should have, in hopes of getting them added to the language properly in ~3.6/3.7. So here are some notes on the exact design decisions taken (for now):

Changes