Skip to main content

File-system specification

Project description

filesystem_spec

Build Status Docs

A specification for pythonic filesystems.

Install

pip install fsspec

or

conda install -c conda-forge fsspec

Purpose

To produce a template or specification for a file-system interface, that specific implementations should follow, so that applications making use of them can rely on a common behaviour and not have to worry about the specific internal implementation decisions with any given backend. Many such implementations are included in this package, or in sister projects such as s3fs and gcsfs.

In addition, if this is well-designed, then additional functionality, such as a key-value store or FUSE mounting of the file-system implementation may be available for all implementations "for free".

Documentation

Please refer to RTD

Develop

fsspec uses tox and tox-conda to manage dev and test environments. First, install conda with tox and tox-conda in a base environment (eg. conda install -c conda-forge tox tox-conda). Calls to tox can then be used to configure a development environment and run tests.

First, setup a development conda environment via tox -e dev. This will install fspec dependencies, test & dev tools, and install fsspec in develop mode. Then, activate the dev environment under .tox/dev via conda activate .tox/dev.

Testing

Tests can be run directly in the activated dev environment via pytest fsspec.

The full fsspec test suite can be run via tox, which will setup and execute tests against multiple dependency versions in isolated environment. Run tox -av to list available test environments, select environments via tox -e <env>.

The full fsspec suite requires a system-level docker, docker-compose, and fuse installation. See ci/install.sh for a detailed installation example.

Code Formatting

fsspec uses Black to ensure a consistent code format throughout the project. black is automatically installed in the tox dev env, activated via conda activate .tox/dev.

Then, run black fsspec from the root of the filesystem_spec repository to auto-format your code. Additionally, many editors have plugins that will apply black as you edit files.

Optionally, you may wish to setup pre-commit hooks to automatically run black when you make a git commit. black is automatically installed in the tox dev env, activated via conda activate .tox/dev.

Then, run pre-commit install --install-hooks from the root of the filesystem_spec repository to setup pre-commit hooks. black will now be run before you commit, reformatting any changed files. You can format without committing via pre-commit run or skip these checks with git commit --no-verify.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fsspec-0.6.3.tar.gz (103.7 kB view details)

Uploaded Source

Built Distribution

fsspec-0.6.3-py3-none-any.whl (65.1 kB view details)

Uploaded Python 3

File details

Details for the file fsspec-0.6.3.tar.gz.

File metadata

  • Download URL: fsspec-0.6.3.tar.gz
  • Upload date:
  • Size: 103.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/45.2.0.post20200209 requests-toolbelt/0.9.1 tqdm/4.42.1 CPython/3.7.6

File hashes

Hashes for fsspec-0.6.3.tar.gz
Algorithm Hash digest
SHA256 7a6d48af15b2416c5007706039e6d46d083c89229e7cdf6474b5ef5e7e687b00
MD5 ddf7a2314cf0035b21a10829dfe29e1b
BLAKE2b-256 2b5b229c396ae4f986dc31879b1a1bb899b126382f1d7299912cad847452c9d9

See more details on using hashes here.

File details

Details for the file fsspec-0.6.3-py3-none-any.whl.

File metadata

  • Download URL: fsspec-0.6.3-py3-none-any.whl
  • Upload date:
  • Size: 65.1 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/45.2.0.post20200209 requests-toolbelt/0.9.1 tqdm/4.42.1 CPython/3.7.6

File hashes

Hashes for fsspec-0.6.3-py3-none-any.whl
Algorithm Hash digest
SHA256 89b502e4bdf51bb3cb1698d1427ec07923fa9ee05e532672e0075e642e78134d
MD5 717f0e0ec0ba209e628a91b256a6b294
BLAKE2b-256 850fea31396f3fc46c375604ce4fb6f6714cbe0b01e8376ca277341e462e28db

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page