Skip to main content

Provides utility functions for accessing data repository for Project Pythia examples/notebooks

Project description

CI GitHub Workflow Status GitHub Workflow Status Code Coverage Status
Docs Documentation Status
Package Conda PyPI
License License

pythia-datasets

Data repository for Project Pythia examples/notebooks

Sample data sets

These files are used as sample data in Pythia Project examples/notebooks and are downloaded by pythia_datasets package:

  • NARR_19930313_0000.nc
  • enso_data.csv
  • jan-17-co-asos.txt.xz
  • CESM2_sst_data.nc
  • CESM2_grid_variables.nc

Adding new datasets

To add a new dataset file, please follow these steps:

  1. Add the dataset file to the data/ directory
  2. From the command line, run python make_registry.py script to update the registry file residing in pythia_datasets/registry.txt
  3. Commit and push your changes to GitHub

Using datasets in notebooks and/or scripts

  • Ensure the pythia_datasets package is installed in your environment

    python -m pip install pythia-datasets
    
    # or
    
    python -m pip install git+https://github.com/ProjectPythia/pythia-datasets
    
  • Import DATASETS and inspect the registry to find out which datasets are available

    In [1]: from pythia_datasets import DATASETS
    
    In [2]: DATASETS.registry_files
    Out[2]: ['jan-17-co-asos.txt.xz', 'NARR_19930313_0000.nc']
    
  • To fetch a data file of interest, use the .fetch method and provide the filename of the data file. This will

    • download and cache the file if it doesn't exist already.
    • retrieve and return the local path
    In [4]: filepath = DATASETS.fetch('jan-17-co-asos.txt.xz')
    
    In [5]: filepath
    Out[5]: '/Users/abanihi/Library/Caches/pythia-datasets/jan-17-co-asos.txt.xz'
    
  • Once you have access to the local filepath, you can then use it to load your dataset into pandas or xarray or your package of choice:

    In [6]: df = pd.read_csv(filepath)
    

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pythia-datasets-2021.9.21.tar.gz (12.0 kB view details)

Uploaded Source

Built Distribution

pythia_datasets-2021.9.21-py3-none-any.whl (8.7 kB view details)

Uploaded Python 3

File details

Details for the file pythia-datasets-2021.9.21.tar.gz.

File metadata

  • Download URL: pythia-datasets-2021.9.21.tar.gz
  • Upload date:
  • Size: 12.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.7

File hashes

Hashes for pythia-datasets-2021.9.21.tar.gz
Algorithm Hash digest
SHA256 19fcb86b0db71aa3251dcab528b51bf66639e2dc3c8654d6fc80b8a3379ac650
MD5 28aa736e264964a7699031ce22b102d9
BLAKE2b-256 0b45fd687d08636413a2b612b56ce67e83fe1724158a962199d9fc6e45f04d5b

See more details on using hashes here.

File details

Details for the file pythia_datasets-2021.9.21-py3-none-any.whl.

File metadata

  • Download URL: pythia_datasets-2021.9.21-py3-none-any.whl
  • Upload date:
  • Size: 8.7 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.7

File hashes

Hashes for pythia_datasets-2021.9.21-py3-none-any.whl
Algorithm Hash digest
SHA256 179dbe46538ad2310fd11075472807c425ffa956db6b90957d52211e22441f5b
MD5 2c6f7fc3afb67dc36253c5b152d26516
BLAKE2b-256 e540fc8ffc2c2fb3bbb0dec75bec4a5fb5336f035ceafac5397400f9c2b12dab

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page