Asynchronous file operations.
Project description
Real asynchronous file operations with asyncio support.
Status
Development - Stable
Features
Since version 2.0.0 using caio, is contain linux libaio and two thread-based implementations (c-based and pure-python).
AIOFile has no internal pointer. You should pass offset and chunk_size for each operation or use helpers (Reader or Writer). The simples way is use async_open for create object with file-like interface.
For Linux using implementation based on libaio.
For POSIX (MacOS X and optional Linux) using implementation using on threadpool.
Otherwise using pure-python thread-based implementation.
Implementation chooses automatically depending on system compatibility.
Limitations
Linux native AIO implementation not able to open special files. Asynchronous operations against special fs like /proc/ /sys/ not supported by the kernel. It’s not a aiofile`s or `caio issue. To In this cases, you might switch to thread-based implementations (see troubleshooting section). However, when used on supported file systems, the linux implementation has a smaller overhead and preferred but it’s not a silver bullet.
Code examples
All code examples requires python 3.6+.
High-level API
async_open helper
The async_open helper creates file like object with file-like methods:
import asyncio
from aiofile import async_open
async def main():
async with async_open("/tmp/hello.txt", 'w+') as afp:
await afp.write("Hello ")
await afp.write("world")
afp.seek(0)
print(await afp.read())
await afp.write("Hello from\nasync world")
print(await afp.readline())
print(await afp.readline())
loop = asyncio.get_event_loop()
loop.run_until_complete(main())Supported methods:
async def read(length = -1) - reading chunk from file, when length is -1 will be read file to the end.
async def write(data) - write chunk to file
def seek(offset) - set file pointer position
def tell() - returns current file pointer position
async def readline(size=-1, newline="\n") - read chunks until newline or EOF. Suboptimal for small lines because doesn’t reuse read buffer. When you want to read file by lines please avoid to use async_open use LineReader instead.
Reader and Writer
When you want to read or write file linearly following example might be helpful.
import asyncio
from aiofile import AIOFile, Reader, Writer
async def main():
async with AIOFile("/tmp/hello.txt", 'w+') as afp:
writer = Writer(afp)
reader = Reader(afp, chunk_size=8)
await writer("Hello")
await writer(" ")
await writer("World")
await afp.fsync()
async for chunk in reader:
print(chunk)
loop = asyncio.get_event_loop()
loop.run_until_complete(main())LineReader - read file line by line
LineReader is a helper that is very effective when you want to read a file linearly and line by line.
It contains a buffer and will read the fragments of the file chunk by chunk into the buffer, where it will try to find lines.
The default chunk size is 4KB.
import asyncio
from aiofile import AIOFile, LineReader, Writer
async def main():
async with AIOFile("/tmp/hello.txt", 'w+') as afp:
writer = Writer(afp)
await writer("Hello")
await writer(" ")
await writer("World")
await writer("\n")
await writer("\n")
await writer("From async world")
await afp.fsync()
async for line in LineReader(afp):
print(line)
loop = asyncio.get_event_loop()
loop.run_until_complete(main())When you want to read file by lines please avoid to use async_open use LineReader instead.
Low-level API
Following API is just little bit sugared caio API.
Write and Read
import asyncio
from aiofile import AIOFile
async def main():
async with AIOFile("/tmp/hello.txt", 'w+') as afp:
await afp.write("Hello ")
await afp.write("world", offset=7)
await afp.fsync()
print(await afp.read())
loop = asyncio.get_event_loop()
loop.run_until_complete(main())Read file line by line
import asyncio
from aiofile import AIOFile, LineReader, Writer
async def main():
async with AIOFile("/tmp/hello.txt", 'w') as afp:
writer = Writer(afp)
for i in range(10):
await writer("%d Hello World\n" % i)
await writer("Tail-less string")
async with AIOFile("/tmp/hello.txt", 'r') as afp:
async for line in LineReader(afp):
print(line[:-1])
loop = asyncio.get_event_loop()
loop.run_until_complete(main())More examples
Useful examples with aiofile
Async CSV Dict Reader
import asyncio
import io
from csv import DictReader
from aiofile import AIOFile, LineReader
class AsyncDictReader:
def __init__(self, afp, **kwargs):
self.buffer = io.BytesIO()
self.file_reader = LineReader(
afp, line_sep=kwargs.pop('line_sep', '\n'),
chunk_size=kwargs.pop('chunk_size', 4096),
offset=kwargs.pop('offset', 0),
)
self.reader = DictReader(
io.TextIOWrapper(
self.buffer,
encoding=kwargs.pop('encoding', 'utf-8'),
errors=kwargs.pop('errors', 'replace'),
), **kwargs,
)
self.line_num = 0
def __aiter__(self):
return self
async def __anext__(self):
if self.line_num == 0:
header = await self.file_reader.readline()
self.buffer.write(header)
line = await self.file_reader.readline()
if not line:
raise StopAsyncIteration
self.buffer.write(line)
self.buffer.seek(0)
try:
result = next(self.reader)
except StopIteration as e:
raise StopAsyncIteration from e
self.buffer.seek(0)
self.buffer.truncate(0)
self.line_num = self.reader.line_num
return result
async def main():
async with AIOFile('sample.csv', 'rb') as afp:
async for item in AsyncDictReader(afp):
print(item)
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
loop.run_until_complete(main())
finally:
# Shutting down and closing file descriptors after interrupt
loop.run_until_complete(loop.shutdown_asyncgens())
loop.close()Troubleshooting
The caio linux implementation works normal for modern linux kernel versions and file systems. So you may have problems specific for your environment. It’s not a bug and might be resolved some ways:
Upgrade the kernel
Use compatible file system
Use threads based or pure python implementation.
The caio since version 0.7.0 contains some ways to do this.
1. In runtime use the environment variable CAIO_IMPL with possible values:
linux - use native linux kernels aio mechanism
thread - use thread based implementation written in C
python - use pure python implementation
2. File default_implementation located near __init__.py in caio installation path. It’s useful for distros package maintainers. This file might contains comments (lines starts with # symbol) and the first line should be one of linux thread or python.
You might manually manage contexts:
import asyncio
from aiofile import async_open
from caio import linux_aio, thread_aio
async def main():
linux_ctx = linux_aio.Context()
threads_ctx = thread_aio.Context()
async with async_open("/tmp/test.txt", "a", context=linux_ctx) as afp:
await afp.write("Hello")
async with async_open("/tmp/test.txt", "a", context=threads_ctx) as afp:
print(await afp.read())
asyncio.run(main())Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file aiofile-3.6.0.tar.gz.
File metadata
- Download URL: aiofile-3.6.0.tar.gz
- Upload date:
- Size: 12.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.1 importlib_metadata/3.10.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.59.0 CPython/3.9.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 864940bf81c3d20fb079c8a0e9773a63539d1cfcef03ceeffead2551846bc513 |
|
| MD5 | fb9756f8465617b6bbba0781ca94827e |
|
| BLAKE2b-256 | 931bb2a71f4c0a4467025dfb2451e9f08817aca9cd01acbc071e84760a199eca |
File details
Details for the file aiofile-3.6.0-py3-none-any.whl.
File metadata
- Download URL: aiofile-3.6.0-py3-none-any.whl
- Upload date:
- Size: 17.7 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.1 importlib_metadata/3.10.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.59.0 CPython/3.9.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 8300ed179d4003560db1ea4ffb52f0f42cd23d754b63b63ea18b2358b2b53926 |
|
| MD5 | cb3650d2a18fcfcf509221a4076b357a |
|
| BLAKE2b-256 | ee78229e82c8c74a8b62d187f4031fcec4caa40118e20ae65951d189a2c5d728 |
