PFN IO library
Project description
PFIO
PFIO is an IO abstraction library developed by PFN, optimized for deep learning training with batteries included. It supports
- Filesystem API abstraction with unified error semantics,
- Explicit user-land caching system,
- IO performance tracing and metrics stats, and
- Fileset container utilities to save metadata.
Dependency
- HDFS client and libhdfs for HDFS access
- Python 3
Installation and Document build
Installation
$ git clone https://github.com/pfnet/pfio.git
$ cd pfio
$ pip install .
Documentation
$ cd pfio/docs
$ make html
$ open build/html/index.html
Test
$ cd pfio
$ pip install .[test]
$ pytest
How to use
Please refer to the Documentation for more information about the usage.
Also you can find some examples in
examples directory for usage in Chainer training script examples.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
pfio-1.0.1.tar.gz
(22.3 kB
view details)
Built Distribution
pfio-1.0.1-py3-none-any.whl
(30.1 kB
view details)
File details
Details for the file pfio-1.0.1.tar.gz.
File metadata
- Download URL: pfio-1.0.1.tar.gz
- Upload date:
- Size: 22.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.0.0 requests-toolbelt/0.9.1 tqdm/4.43.0 CPython/3.7.4
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 9b2a9f2e3478d27d343e7ebca5a23f835663f3a0dd6bf8b53d65b5a283126fc2 |
|
| MD5 | ce2bd4fc17fea4e1b27c0e333b0a4c99 |
|
| BLAKE2b-256 | dcaadb73a9b3cb2d1b3965468dc3015356fbaaa8e906d708917c341289b21080 |
File details
Details for the file pfio-1.0.1-py3-none-any.whl.
File metadata
- Download URL: pfio-1.0.1-py3-none-any.whl
- Upload date:
- Size: 30.1 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.0.0 requests-toolbelt/0.9.1 tqdm/4.43.0 CPython/3.7.4
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | d883fb9ecc998a8a9793388f7e0170d0ca1d1bf7b148adf932ee43afce15adf3 |
|
| MD5 | 4525a2c226377f73362394c968dae1e3 |
|
| BLAKE2b-256 | 055bcaf44f2763f3d44fbf894a5e73cee9a5fe485c8f8ab1b22dfd0060d1c601 |
