Skip to main content

Provides utility functions for accessing data repository for ARM data examples/notebooks

Project description

arm-test-data

CI PyPI Version Conda Version

A place to share atmospheric data with the community, shared throughout the Atmospheric Radiation Measurement user facility and beyond!

Sample data sets

These files are used as sample data in openradar examples/notebooks and are downloaded by arm-test-data package:

  • 201509021500.bi
  • AAFNAV_COR_20181104_R0.ict
  • NEON.D18.BARR.DP1.00002.001.000.010.001.SAAT_1min.2022-10.expanded.20221107T205629Z.csv
  • NEON.D18.BARR.DP1.00002.001.sensor_positions.20221107T205629Z.csv
  • NEON.D18.BARR.DP1.00002.001.variables.20221201T110553Z.csv
  • anltwr_mar19met.data
  • ayp22199.21m
  • ayp22200.00m
  • brw21001.dat
  • brw_12_2020_hour.dat
  • brw_CCl4_Day.dat
  • co2_brw_surface-insitu_1_ccgg_MonthlyData.txt
  • ctd21125.15w
  • ctd22187.00t.txt
  • enametC1.b1.20221109.000000.cdf
  • gucmetM1.b1.20230301.000000.cdf
  • list_of_files.txt
  • maraosmetM1.a1.20180201.000000.nc
  • marirtsstM1.b1.20190320.000000.nc
  • marnavM1.a1.20180201.000000.nc
  • met_brw_insitu_1_obop_hour_2020.txt
  • met_lcl.nc
  • mosaossp2M1.00.20191216.000601.raw.20191216000000.ini
  • mosaossp2M1.00.20191216.130601.raw.20191216x193.sp2b
  • mosaossp2auxM1.00.20191217.010801.raw.20191216000000.hk
  • nsacloudphaseC1.c1.20180601.000000.nc
  • nsasurfspecalb1mlawerC1.c1.20160609.080000.nc
  • sgp30ebbrE13.b1.20190601.000000.nc
  • sgp30ebbrE32.b1.20191125.000000.nc
  • sgp30ebbrE32.b1.20191130.000000.nc
  • sgp30ecorE14.b1.20190601.000000.cdf
  • sgpaerich1C1.b1.20190501.000342.nc
  • sgpaosacsmE13.b2.20230420.000109.nc
  • sgpaosccn2colaE13.b1.20170903.000000.nc
  • sgpbrsC1.b1.20190705.000000.cdf
  • sgpceilC1.b1.20190101.000000.nc
  • sgpco2flx4mC1.b1.20201007.001500.nc
  • sgpdlppiC1.b1.20191015.120023.cdf
  • sgpdlppiC1.b1.20191015.121506.cdf
  • sgpirt25m20sC1.a0.20190601.000000.cdf
  • sgpmetE13.b1.20190101.000000.cdf
  • sgpmetE13.b1.20190102.000000.cdf
  • sgpmetE13.b1.20190103.000000.cdf
  • sgpmetE13.b1.20190104.000000.cdf
  • sgpmetE13.b1.20190105.000000.cdf
  • sgpmetE13.b1.20190106.000000.cdf
  • sgpmetE13.b1.20190107.000000.cdf
  • sgpmetE13.b1.20190508.000000.cdf
  • sgpmetE13.b1.20210401.000000.csv
  • sgpmetE13.b1.yaml
  • sgpmetE15.b1.20190508.000000.cdf
  • sgpmetE31.b1.20190508.000000.cdf
  • sgpmetE32.b1.20190508.000000.cdf
  • sgpmetE33.b1.20190508.000000.cdf
  • sgpmetE34.b1.20190508.000000.cdf
  • sgpmetE35.b1.20190508.000000.cdf
  • sgpmetE36.b1.20190508.000000.cdf
  • sgpmetE37.b1.20190508.000000.cdf
  • sgpmetE38.b1.20190508.000000.cdf
  • sgpmetE39.b1.20190508.000000.cdf
  • sgpmetE40.b1.20190508.000000.cdf
  • sgpmetE9.b1.20190508.000000.cdf
  • sgpmet_no_time.nc
  • sgpmet_test_time.nc
  • sgpmfrsr7nchE11.b1.20210329.070000.nc
  • sgpmmcrC1.b1.1.cdf
  • sgpmmcrC1.b1.2.cdf
  • sgpmplpolfsC1.b1.20190502.000000.cdf
  • sgprlC1.a0.20160131.000000.nc
  • sgpsebsE14.b1.20190601.000000.cdf
  • sgpsirsE13.b1.20190101.000000.cdf
  • sgpsondewnpnC1.b1.20190101.053200.cdf
  • sgpstampE13.b1.20200101.000000.nc
  • sgpstampE31.b1.20200101.000000.nc
  • sgpstampE32.b1.20200101.000000.nc
  • sgpstampE33.b1.20200101.000000.nc
  • sgpstampE34.b1.20200101.000000.nc
  • sgpstampE9.b1.20200101.000000.nc
  • sodar.20230404.mnd
  • twpsondewnpnC3.b1.20060119.050300.custom.cdf
  • twpsondewnpnC3.b1.20060119.112000.custom.cdf
  • twpsondewnpnC3.b1.20060119.163300.custom.cdf
  • twpsondewnpnC3.b1.20060119.231600.custom.cdf
  • twpsondewnpnC3.b1.20060120.043800.custom.cdf
  • twpsondewnpnC3.b1.20060120.111900.custom.cdf
  • twpsondewnpnC3.b1.20060120.170800.custom.cdf
  • twpsondewnpnC3.b1.20060120.231500.custom.cdf
  • twpsondewnpnC3.b1.20060121.051500.custom.cdf
  • twpsondewnpnC3.b1.20060121.111600.custom.cdf
  • twpsondewnpnC3.b1.20060121.171600.custom.cdf
  • twpsondewnpnC3.b1.20060121.231600.custom.cdf
  • twpsondewnpnC3.b1.20060122.052600.custom.cdf
  • twpsondewnpnC3.b1.20060122.111500.custom.cdf
  • twpsondewnpnC3.b1.20060122.171800.custom.cdf
  • twpsondewnpnC3.b1.20060122.232600.custom.cdf
  • twpsondewnpnC3.b1.20060123.052500.custom.cdf
  • twpsondewnpnC3.b1.20060123.111700.custom.cdf
  • twpsondewnpnC3.b1.20060123.171600.custom.cdf
  • twpsondewnpnC3.b1.20060123.231500.custom.cdf
  • twpsondewnpnC3.b1.20060124.051500.custom.cdf
  • twpsondewnpnC3.b1.20060124.111800.custom.cdf
  • twpsondewnpnC3.b1.20060124.171700.custom.cdf
  • twpsondewnpnC3.b1.20060124.231500.custom.cdf
  • twpvisstgridirtemp.c1.20050705.002500.nc
  • vdis.b1

Adding new datasets

To add a new dataset file, please follow these steps:

  1. Add the dataset file to the data/ directory
  2. From the command line, run python make_registry.py script to update the registry file residing in arm-test-data/registry.txt
  3. Commit and push your changes to GitHub

Using datasets in notebooks and/or scripts

  • Ensure the arm-test-data package is installed in your environment

    python -m pip install arm-test-data
    
    # or
    
    python -m pip install git+https://github.com/ARM-DOE/arm-test-data
    
    # or
    
    conda install -c conda-forge arm-test-data
    
  • Import DATASETS and inspect the registry to find out which datasets are available

    In [1]: from arm_test_data import DATASETS
    
    In [2]: DATASETS.registry_files
    Out[2]: ['sample_file.nc`]
    
  • To fetch a data file of interest, use the .fetch method and provide the filename of the data file. This will

    • download and cache the file if it doesn't exist already.
    • retrieve and return the local path
    In [4]: filepath = DATASETS.fetch('sample_data.nc')
    
    In [5]: filepath
    Out[5]: '/Users/mgrover/Library/Caches/arm-test-data/sample_sgp_data.nc'
    
  • Once you have access to the local filepath, you can then use it to load your dataset into pandas or xarray or your package of choice:

    In [6]: radar = pyart.io.read(filepath)
    

Changing the default data cache location

The default cache location (where the data are saved on your local system) is dependent on the operating system. You can use the locate() method to identify it:

from arm_test_data import locate
locate()

The location can be overwritten by the ACT_TEST_DATA_DIR environment variable to the desired destination.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

arm_test_data-0.0.10.tar.gz (16.7 kB view details)

Uploaded Source

Built Distribution

arm_test_data-0.0.10-py3-none-any.whl (11.9 kB view details)

Uploaded Python 3

File details

Details for the file arm_test_data-0.0.10.tar.gz.

File metadata

  • Download URL: arm_test_data-0.0.10.tar.gz
  • Upload date:
  • Size: 16.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.9

File hashes

Hashes for arm_test_data-0.0.10.tar.gz
Algorithm Hash digest
SHA256 0b66ca4e52ee2a3c3e0164a11fe6073f006530dd4dc16f40045b2654f3f61085
MD5 db9d31e68befe54d8537b081732f6ad9
BLAKE2b-256 c85225bff9c87f8d58c6a590a4de598fb1bff493b03a6a3b8bcb4ec9a202658c

See more details on using hashes here.

Provenance

File details

Details for the file arm_test_data-0.0.10-py3-none-any.whl.

File metadata

File hashes

Hashes for arm_test_data-0.0.10-py3-none-any.whl
Algorithm Hash digest
SHA256 4d48ee29c681c7195d7ea1382ef03082352b848ad32b0649a210326f3a957fcf
MD5 4e04a8d0fe824877d4d3253eaa1c8cea
BLAKE2b-256 2194a79d817f3d3b76e5fa0775f3fa38bc1cbdc6f88258bd5f2b187fef6faeb3

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page