Provides utility functions for accessing data repository for ARM data examples/notebooks
Project description
arm-test-data
A place to share atmospheric data with the community, shared throughout the Atmospheric Radiation Measurement user facility and beyond!
Sample data sets
These files are used as sample data in openradar examples/notebooks and are downloaded by arm-test-data
package:
201509021500.bi
AAFNAV_COR_20181104_R0.ict
NEON.D18.BARR.DP1.00002.001.000.010.001.SAAT_1min.2022-10.expanded.20221107T205629Z.csv
NEON.D18.BARR.DP1.00002.001.sensor_positions.20221107T205629Z.csv
NEON.D18.BARR.DP1.00002.001.variables.20221201T110553Z.csv
anltwr_mar19met.data
ayp22199.21m
ayp22200.00m
brw21001.dat
brw_12_2020_hour.dat
brw_CCl4_Day.dat
co2_brw_surface-insitu_1_ccgg_MonthlyData.txt
ctd21125.15w
ctd22187.00t.txt
enametC1.b1.20221109.000000.cdf
gucmetM1.b1.20230301.000000.cdf
list_of_files.txt
maraosmetM1.a1.20180201.000000.nc
marirtsstM1.b1.20190320.000000.nc
marnavM1.a1.20180201.000000.nc
met_brw_insitu_1_obop_hour_2020.txt
met_lcl.nc
mosaossp2M1.00.20191216.000601.raw.20191216000000.ini
mosaossp2M1.00.20191216.130601.raw.20191216x193.sp2b
mosaossp2auxM1.00.20191217.010801.raw.20191216000000.hk
nsacloudphaseC1.c1.20180601.000000.nc
nsasurfspecalb1mlawerC1.c1.20160609.080000.nc
sgp30ebbrE13.b1.20190601.000000.nc
sgp30ebbrE32.b1.20191125.000000.nc
sgp30ebbrE32.b1.20191130.000000.nc
sgp30ecorE14.b1.20190601.000000.cdf
sgpaerich1C1.b1.20190501.000342.nc
sgpaosacsmE13.b2.20230420.000109.nc
sgpaosccn2colaE13.b1.20170903.000000.nc
sgpbrsC1.b1.20190705.000000.cdf
sgpceilC1.b1.20190101.000000.nc
sgpco2flx4mC1.b1.20201007.001500.nc
sgpdlppiC1.b1.20191015.120023.cdf
sgpdlppiC1.b1.20191015.121506.cdf
sgpirt25m20sC1.a0.20190601.000000.cdf
sgpmetE13.b1.20190101.000000.cdf
sgpmetE13.b1.20190102.000000.cdf
sgpmetE13.b1.20190103.000000.cdf
sgpmetE13.b1.20190104.000000.cdf
sgpmetE13.b1.20190105.000000.cdf
sgpmetE13.b1.20190106.000000.cdf
sgpmetE13.b1.20190107.000000.cdf
sgpmetE13.b1.20190508.000000.cdf
sgpmetE13.b1.20210401.000000.csv
sgpmetE13.b1.yaml
sgpmetE15.b1.20190508.000000.cdf
sgpmetE31.b1.20190508.000000.cdf
sgpmetE32.b1.20190508.000000.cdf
sgpmetE33.b1.20190508.000000.cdf
sgpmetE34.b1.20190508.000000.cdf
sgpmetE35.b1.20190508.000000.cdf
sgpmetE36.b1.20190508.000000.cdf
sgpmetE37.b1.20190508.000000.cdf
sgpmetE38.b1.20190508.000000.cdf
sgpmetE39.b1.20190508.000000.cdf
sgpmetE40.b1.20190508.000000.cdf
sgpmetE9.b1.20190508.000000.cdf
sgpmet_no_time.nc
sgpmet_test_time.nc
sgpmfrsr7nchE11.b1.20210329.070000.nc
sgpmmcrC1.b1.1.cdf
sgpmmcrC1.b1.2.cdf
sgpmplpolfsC1.b1.20190502.000000.cdf
sgprlC1.a0.20160131.000000.nc
sgpsebsE14.b1.20190601.000000.cdf
sgpsirsE13.b1.20190101.000000.cdf
sgpsondewnpnC1.b1.20190101.053200.cdf
sgpstampE13.b1.20200101.000000.nc
sgpstampE31.b1.20200101.000000.nc
sgpstampE32.b1.20200101.000000.nc
sgpstampE33.b1.20200101.000000.nc
sgpstampE34.b1.20200101.000000.nc
sgpstampE9.b1.20200101.000000.nc
sodar.20230404.mnd
twpsondewnpnC3.b1.20060119.050300.custom.cdf
twpsondewnpnC3.b1.20060119.112000.custom.cdf
twpsondewnpnC3.b1.20060119.163300.custom.cdf
twpsondewnpnC3.b1.20060119.231600.custom.cdf
twpsondewnpnC3.b1.20060120.043800.custom.cdf
twpsondewnpnC3.b1.20060120.111900.custom.cdf
twpsondewnpnC3.b1.20060120.170800.custom.cdf
twpsondewnpnC3.b1.20060120.231500.custom.cdf
twpsondewnpnC3.b1.20060121.051500.custom.cdf
twpsondewnpnC3.b1.20060121.111600.custom.cdf
twpsondewnpnC3.b1.20060121.171600.custom.cdf
twpsondewnpnC3.b1.20060121.231600.custom.cdf
twpsondewnpnC3.b1.20060122.052600.custom.cdf
twpsondewnpnC3.b1.20060122.111500.custom.cdf
twpsondewnpnC3.b1.20060122.171800.custom.cdf
twpsondewnpnC3.b1.20060122.232600.custom.cdf
twpsondewnpnC3.b1.20060123.052500.custom.cdf
twpsondewnpnC3.b1.20060123.111700.custom.cdf
twpsondewnpnC3.b1.20060123.171600.custom.cdf
twpsondewnpnC3.b1.20060123.231500.custom.cdf
twpsondewnpnC3.b1.20060124.051500.custom.cdf
twpsondewnpnC3.b1.20060124.111800.custom.cdf
twpsondewnpnC3.b1.20060124.171700.custom.cdf
twpsondewnpnC3.b1.20060124.231500.custom.cdf
twpvisstgridirtemp.c1.20050705.002500.nc
vdis.b1
Adding new datasets
To add a new dataset file, please follow these steps:
- Add the dataset file to the
data/
directory - From the command line, run
python make_registry.py
script to update the registry file residing inarm-test-data/registry.txt
- Commit and push your changes to GitHub
Using datasets in notebooks and/or scripts
-
Ensure the
arm-test-data
package is installed in your environmentpython -m pip install arm-test-data # or python -m pip install git+https://github.com/ARM-DOE/arm-test-data # or conda install -c conda-forge arm-test-data
-
Import
DATASETS
and inspect the registry to find out which datasets are availableIn [1]: from arm_test_data import DATASETS In [2]: DATASETS.registry_files Out[2]: ['sample_file.nc`]
-
To fetch a data file of interest, use the
.fetch
method and provide the filename of the data file. This will- download and cache the file if it doesn't exist already.
- retrieve and return the local path
In [4]: filepath = DATASETS.fetch('sample_data.nc') In [5]: filepath Out[5]: '/Users/mgrover/Library/Caches/arm-test-data/sample_sgp_data.nc'
-
Once you have access to the local filepath, you can then use it to load your dataset into pandas or xarray or your package of choice:
In [6]: radar = pyart.io.read(filepath)
Changing the default data cache location
The default cache location (where the data are saved on your local system) is dependent on the operating system. You can use the locate()
method to identify it:
from arm_test_data import locate
locate()
The location can be overwritten by the ACT_TEST_DATA_DIR
environment
variable to the desired destination.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for arm_test_data-0.0.10-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4d48ee29c681c7195d7ea1382ef03082352b848ad32b0649a210326f3a957fcf |
|
MD5 | 4e04a8d0fe824877d4d3253eaa1c8cea |
|
BLAKE2b-256 | 2194a79d817f3d3b76e5fa0775f3fa38bc1cbdc6f88258bd5f2b187fef6faeb3 |