pfio

PFN IO library

These details have been verified by PyPI

Maintainers

belldandyxtq ksugihara kuenishi y1r

These details have not been verified by PyPI

Project links

Homepage

Development Status
- 3 - Alpha
Intended Audience
- Developers
License
- OSI Approved :: MIT License
Operating System
- POSIX
- POSIX :: Linux
Programming Language
Topic
- Software Development :: Build Tools
- System :: Filesystems

Project description

PFIO is an IO abstraction library developed by PFN, optimized for deep learning training with batteries included. It supports

Filesystem API abstraction with unified error semantics,
Explicit user-land caching system,
IO performance tracing and metrics stats, and
Fileset container utilities to save metadata.

Dependency

HDFS client and libhdfs for HDFS access
Python 3

Installation and Document build

Installation

$ git clone https://github.com/chainer/pfio.git
$ cd pfio
$ pip install .

Documentation

$ cd pfio/docs
$ make html
$ open build/html/index.html

Test

$ cd pfio
$ pip install .[test]
$ pytest

How to use

Please refer to the Documentation for more information about the usage. Also you can find some examples in examples directory for usage in Chainer training script examples.

Project details

These details have been verified by PyPI

Maintainers

belldandyxtq ksugihara kuenishi y1r

These details have not been verified by PyPI

Project links

Homepage

Development Status
- 3 - Alpha
Intended Audience
- Developers
License
- OSI Approved :: MIT License
Operating System
- POSIX
- POSIX :: Linux
Programming Language
Topic
- Software Development :: Build Tools
- System :: Filesystems

Release history Release notifications | RSS feed

2.9.0

Oct 29, 2024

2.8.0

Apr 17, 2024

2.7.1

Oct 6, 2023

2.7.0

Oct 2, 2023

2.6.0

Jun 26, 2023

2.5.1

Feb 7, 2023

2.5.0

Jan 31, 2023

2.4.0

Oct 24, 2022

2.4.0rc1 pre-release

Oct 6, 2022

2.3.0

Sep 26, 2022

2.3.0rc1 pre-release

Jul 26, 2022

2.2.0

Feb 24, 2022

2.1.2

Jan 11, 2022

2.1.1

Jan 4, 2022

2.1.0

Jan 4, 2022

2.0.1

Dec 23, 2021

2.0.0

Oct 28, 2021

2.0.0a3 pre-release

Oct 19, 2021

2.0.0a2 pre-release

Oct 4, 2021

2.0.0a1 pre-release

Sep 8, 2021

1.6.0

Sep 1, 2021

1.5.0

Jun 8, 2021

1.4.3

Jun 7, 2021

1.4.1

May 24, 2021

1.4.0

Apr 21, 2021

1.3.0

Mar 15, 2021

1.2.2

Mar 11, 2021

1.2.1

Mar 10, 2021

1.2.0

Mar 10, 2021

1.1.0

Feb 19, 2021

1.0.1

Oct 8, 2020

This version

1.0.0

May 1, 2020

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pfio-1.0.0.tar.gz (20.0 kB view details)

Uploaded May 1, 2020 Source

Built Distribution

pfio-1.0.0-py3-none-any.whl (26.4 kB view details)

Uploaded May 1, 2020 Python 3

File details

Details for the file pfio-1.0.0.tar.gz.

File metadata

Download URL: pfio-1.0.0.tar.gz
Upload date: May 1, 2020
Size: 20.0 kB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.0.0 requests-toolbelt/0.9.1 tqdm/4.43.0 CPython/3.7.4

File hashes

Hashes for pfio-1.0.0.tar.gz
Algorithm	Hash digest
SHA256	`7cac33f23e7674b128d04ee76f23df615f9c9de817c6b6b80a9ef46c876996ad`
MD5	`533b5bf338c7e92a3684e3aaeeea89b9`
BLAKE2b-256	`ca3f1a7ddd16e2cd5d97b30e9ce0806dc4ec3253f0cbaaa030b6fb4572bdda3c`

See more details on using hashes here.

File details

Details for the file pfio-1.0.0-py3-none-any.whl.

File metadata

Download URL: pfio-1.0.0-py3-none-any.whl
Upload date: May 1, 2020
Size: 26.4 kB
Tags: Python 3
Uploaded using Trusted Publishing? No
Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.0.0 requests-toolbelt/0.9.1 tqdm/4.43.0 CPython/3.7.4

File hashes

Hashes for pfio-1.0.0-py3-none-any.whl
Algorithm	Hash digest
SHA256	`69aa261675a608c9505f8abef5d43481b2b2578cc74031ae98c0418dee41b537`
MD5	`9750216c2f205ebcf8d2da8a8f7bb306`
BLAKE2b-256	`c3caa8bcc9af24ebf2eb8843e9015999fb9ca0cb9c077e35e4d299ca4257babd`