PFN IO library

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for belldandyxtq from gravatar.com belldandyxtq Avatar for ksugihara from gravatar.com ksugihara Avatar for kuenishi from gravatar.com kuenishi Avatar for y1r from gravatar.com y1r

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License
  • Author: Tianqi Xu, Kota Uenishi
  • Tags filesystem, hdfs, chainer, development
  • Requires: Python >=3.6
Classifiers

Project description

PFIO

PFIO is an IO abstraction library developed by PFN, optimized for deep learning training with batteries included. It supports

  • Filesystem API abstraction with unified error semantics,
  • Explicit user-land caching system,
  • IO performance tracing and metrics stats, and
  • Fileset container utilities to save metadata.

Dependency

  • HDFS client and libhdfs for HDFS access
  • CPython >= 3.6

Installation and Document build

Installation

$ git clone https://github.com/pfnet/pfio.git
$ cd pfio
$ pip install .

Documentation

$ cd pfio/docs
$ make html
$ open build/html/index.html

Test

$ cd pfio
$ pip install .[test]
$ pytest tests/

How to use

Please refer to the official document for more information about the usage.

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for belldandyxtq from gravatar.com belldandyxtq Avatar for ksugihara from gravatar.com ksugihara Avatar for kuenishi from gravatar.com kuenishi Avatar for y1r from gravatar.com y1r

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License
  • Author: Tianqi Xu, Kota Uenishi
  • Tags filesystem, hdfs, chainer, development
  • Requires: Python >=3.6
Classifiers

Release history Release notifications | RSS feed

2.9.0

2.8.0

2.7.1

2.7.0

2.6.0

2.5.1

2.5.0

2.4.0

2.4.0rc1 pre-release

2.3.0

2.3.0rc1 pre-release

2.2.0

2.1.2

2.1.1

2.1.0

2.0.1

2.0.0

2.0.0a3 pre-release

2.0.0a2 pre-release

2.0.0a1 pre-release

1.6.0

1.5.0

1.4.3

1.4.1

1.4.0

1.3.0

1.2.2

1.2.1

1.2.0

This version

1.1.0

1.0.1

1.0.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pfio-1.1.0.tar.gz (23.9 kB view details)

Uploaded Source

Built Distribution

pfio-1.1.0-py3-none-any.whl (31.4 kB view details)

Uploaded Python 3

File details

Details for the file pfio-1.1.0.tar.gz.

File metadata

  • Download URL: pfio-1.1.0.tar.gz
  • Upload date:
  • Size: 23.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/53.0.0 requests-toolbelt/0.9.1 tqdm/4.57.0 CPython/3.9.1

File hashes

Hashes for pfio-1.1.0.tar.gz
Algorithm Hash digest
SHA256 8d67b62c039818910db91fd76a93bb731f74d1bafe1018d3017bed1a6da8f54b
MD5 c8196364d3b9635778b7f06a19315410
BLAKE2b-256 b7dfcbf725502ae966fbd57d0d04184cb97601815d2d144f8a37465ea3b253f7

See more details on using hashes here.

File details

Details for the file pfio-1.1.0-py3-none-any.whl.

File metadata

  • Download URL: pfio-1.1.0-py3-none-any.whl
  • Upload date:
  • Size: 31.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/53.0.0 requests-toolbelt/0.9.1 tqdm/4.57.0 CPython/3.9.1

File hashes

Hashes for pfio-1.1.0-py3-none-any.whl
Algorithm Hash digest
SHA256 ebae84f8a8c80918bbda448df6b94390a6c1b5a27920106a15e14915ddc9565f
MD5 45b19f4a66acd9285b769c4c77d6e8d8
BLAKE2b-256 2c6a7247a0df0b14c45483b573c2b86c70cfb47ed7028d1c7e1cea70a37da08a

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page