PFN IO library

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for belldandyxtq from gravatar.com belldandyxtq Avatar for ksugihara from gravatar.com ksugihara Avatar for kuenishi from gravatar.com kuenishi Avatar for y1r from gravatar.com y1r

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License
  • Author: Tianqi Xu, Kota Uenishi
  • Tags filesystem, hdfs, chainer, development
  • Requires: Python >=3.6
Classifiers

Project description

PFIO

PFIO is an IO abstraction library developed by PFN, optimized for deep learning training with batteries included. It supports

  • Filesystem API abstraction with unified error semantics,
  • Explicit user-land caching system,
  • IO performance tracing and metrics stats, and
  • Fileset container utilities to save metadata.

Dependency

  • HDFS client and libhdfs for HDFS access
  • CPython >= 3.6

Installation and Document build

Installation

$ git clone https://github.com/pfnet/pfio.git
$ cd pfio
$ pip install .

Documentation

$ cd pfio/docs
$ make html
$ open build/html/index.html

Test

$ cd pfio
$ pip install .[test]
$ pytest tests/

How to use

Please refer to the official document for more information about the usage.

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for belldandyxtq from gravatar.com belldandyxtq Avatar for ksugihara from gravatar.com ksugihara Avatar for kuenishi from gravatar.com kuenishi Avatar for y1r from gravatar.com y1r

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License
  • Author: Tianqi Xu, Kota Uenishi
  • Tags filesystem, hdfs, chainer, development
  • Requires: Python >=3.6
Classifiers

Release history Release notifications | RSS feed

2.9.0

2.8.0

2.7.1

2.7.0

2.6.0

2.5.1

2.5.0

2.4.0

2.4.0rc1 pre-release

2.3.0

2.3.0rc1 pre-release

2.2.0

2.1.2

2.1.1

2.1.0

2.0.1

2.0.0

2.0.0a3 pre-release

2.0.0a2 pre-release

2.0.0a1 pre-release

1.6.0

1.5.0

1.4.3

1.4.1

1.4.0

This version

1.3.0

1.2.2

1.2.1

1.2.0

1.1.0

1.0.1

1.0.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pfio-1.3.0.tar.gz (33.7 kB view details)

Uploaded Source

Built Distribution

pfio-1.3.0-py3-none-any.whl (45.4 kB view details)

Uploaded Python 3

File details

Details for the file pfio-1.3.0.tar.gz.

File metadata

  • Download URL: pfio-1.3.0.tar.gz
  • Upload date:
  • Size: 33.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/53.1.0 requests-toolbelt/0.9.1 tqdm/4.57.0 CPython/3.9.2

File hashes

Hashes for pfio-1.3.0.tar.gz
Algorithm Hash digest
SHA256 34387b288ba1df02affd14b71e787b929dd4cf3e739082e988a707f39c57950e
MD5 0a97e582d87514668a233dfbd7c5ffee
BLAKE2b-256 14d0aa4ed54aa7ee9c1baed7d75203301ebdfdc6f83e5ba8d5e6ad26f9b6ceda

See more details on using hashes here.

File details

Details for the file pfio-1.3.0-py3-none-any.whl.

File metadata

  • Download URL: pfio-1.3.0-py3-none-any.whl
  • Upload date:
  • Size: 45.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/53.1.0 requests-toolbelt/0.9.1 tqdm/4.57.0 CPython/3.9.2

File hashes

Hashes for pfio-1.3.0-py3-none-any.whl
Algorithm Hash digest
SHA256 d673025b0d509e0c5a0b1e60c9361c76cb783e544c4474b7c72083723b518fc3
MD5 f1c3c0b87f5541fa6fbeff3fb3995fd8
BLAKE2b-256 aad161547210990c0a664377aebd748a86d086ef9ef3dde6ae47babe7b552fd2

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page