phenodata 0.13.0

Acknowledgements

Thanks to the many observers of »Deutscher Wetterdienst« (DWD), the »Global Phenological Monitoring programme« (GPM), and all people working behind the scenes for their commitment on recording observations and making the excellent datasets available to the community. You know who you are.

Notes

Please note that phenodata is beta-quality software, and a work in progress. Contributions of all kinds are welcome, in order to make it more solid.

Breaking changes should be expected until a 1.0 release, so version pinning is recommended, especially when you use phenodata as a library.

Output example

Phenodata can produce output in different formats. This is a table in reStructuredText format.

Introduction

For most acquisition tasks, you will have to select one of two different datasets of DWD, annual-reporters or immediate-reporters. Further, the data partition has to be selected, it is either recent, or historical.

Currently, as of 2023, the historical datasets extend from the past until 2021. All subsequent observations are stored within the recent dataset partition.

The DWD publishes data in files separated by species, this means each plant’s data will be in a different file. By default, phenodata will acquire data for all species (plants), in order to be able to respond to all kinds of queries across the whole dataset.

If you are only interested in a limited set of species (plants), you can improve data acquisition performance by using the filename option to only select specific files for retrieval.

For example, when using --filename=Hasel,Schneegloeckchen, only file names containing Hasel or Schneegloeckchen will be retrieved, thus minimizing the effort needed to acquire all files.

Install

To install the software from PyPI, invoke:

pip install 'phenodata[sql]' --upgrade

Library use

This snippet demonstrates how to use phenodata as a library within individual programs. For ready-to-run code examples, please have a look into the examples directory.

>>> import pandas as pd
>>> from phenodata.ftp import FTPSession
>>> from phenodata.dwd.cdc import DwdCdcClient
>>> from phenodata.dwd.pheno import DwdPhenoData

>>> cdc_client = DwdCdcClient(ftp=FTPSession())
>>> client = DwdPhenoData(cdc=cdc_client, humanizer=None, dataset="immediate")
>>> options = {
...     # Select data partition.
...     "partition": "recent",
...
...     # Filter by file names and years.
...     "filename": ["Hasel", "Raps", "Mais"],
...     "year": [2018, 2019, 2020],
...
...     # Filter by station identifier.
...     "station-id": [13346]
... }

>>> observations: pd.DataFrame = client.get_observations(options, humanize=False)
>>> observations.info()
[...]
>>> observations
[...]

Command-line use

This section gives you an idea about how to use the phenodata program on the command-line.

$ phenodata --help

Usage:
  phenodata info
  phenodata list-species --source=dwd [--format=csv]
  phenodata list-phases --source=dwd [--format=csv]
  phenodata list-stations --source=dwd --dataset=immediate [--all] [--filter=berlin] [--sort=Stationsname] [--format=csv]
  phenodata nearest-station --source=dwd --dataset=immediate --latitude=52.520007 --longitude=13.404954 [--format=csv]
  phenodata nearest-stations --source=dwd --dataset=immediate --latitude=52.520007 --longitude=13.404954 [--all] [--limit=10] [--format=csv]
  phenodata list-quality-levels --source=dwd [--format=csv]
  phenodata list-quality-bytes --source=dwd [--format=csv]
  phenodata list-filenames --source=dwd --dataset=immediate --partition=recent [--filename=Hasel,Schneegloeckchen] [--year=2017]
  phenodata list-urls --source=dwd --dataset=immediate --partition=recent [--filename=Hasel,Schneegloeckchen] [--year=2017]
  phenodata (observations|forecast) --source=dwd --dataset=immediate --partition=recent [--filename=Hasel,Schneegloeckchen] [--station-id=164,717] [--species-id=113,127] [--phase-id=5] [--quality-level=10] [--quality-byte=1,2,3] [--station=berlin,brandenburg] [--species=hazel,snowdrop] [--species-preset=mellifera-de-primary] [--phase=flowering] [--quality=ROUTKLI] [--year=2017] [--forecast-year=2021] [--humanize] [--show-ids] [--language=german] [--long-station] [--sort=Datum] [--sql=sql] [--format=csv] [--verbose]
  phenodata drop-cache --source=dwd
  phenodata --version
  phenodata (-h | --help)

Data acquisition options:
  --source=<source>         Data source. Currently, only "dwd" is a valid identifier.
  --dataset=<dataset>       Data set. Use "immediate" or "annual" for "--source=dwd".
  --partition=<dataset>     Partition. Use "recent" or "historical" for "--source=dwd".
  --filename=<file>         Filter by file names (comma-separated list)

Direct filtering options:
  --year=<year>             Filter by year (comma-separated list)
  --station-id=<station-id> Filter by station identifiers (comma-separated list)
  --species-id=<species-id> Filter by species identifiers (comma-separated list)
  --phase-id=<phase-id>     Filter by phase identifiers (comma-separated list)

Humanized filtering options:
  --station=<station>       Filter by strings from "stations" data (comma-separated list)
  --species=<species>       Filter by strings from "species" data (comma-separated list)
  --phase=<phase>           Filter by strings from "phases" data (comma-separated list)
  --species-preset=<preset> Filter by strings from "species" data (comma-separated list)
                            The preset will get loaded from the "presets.json" file.

Forecasting options:
  --forecast-year=<year>    Use as designated forecast year.

Postprocess filtering options:
  --sql=<sql>               Apply given SQL query before output.

Data output options:
  --format=<format>         Output data in designated format. Choose one of "tabular", "json",
                            "csv", or "string". Use "md" for Markdown output, or "rst" for
                            reStructuredText. With "tabular:foo", it is also possible to specify
                            other tabular output formats.  [default: tabular:psql]
  --sort=<sort>             Sort by given field names. (comma-separated list)
  --humanize                Resolve identifier-based fields to human-readable labels.
  --show-ids                Show identifiers alongside resolved labels, when using "--humanize".
  --language=<language>     Use labels in designated language, when using "--humanize"
                            [default: english].
  --long-station            Use long station name including "Naturraumgruppe" and "Naturraum".
  --limit=<limit>           Limit output of "nearest-stations" to designated number of entries.
                            [default: 10]
  --verbose                 Turn on verbose output.

Metadata

Display list of species, with their German, English, and Latin names:

phenodata list-species --source=dwd

Display list of phases, with their German and English names:

phenodata list-phases --source=dwd

List of all reporting/monitoring stations:

phenodata list-stations --source=dwd --dataset=immediate

List of stations, with filtering:

phenodata list-stations --source=dwd --dataset=annual --filter="Fränkische Alb"

Display nearest station for given position:

phenodata nearest-station --source=dwd --dataset=immediate \
    --latitude=52.520007 --longitude=13.404954

Display 20 nearest stations for given position:

phenodata nearest-stations \
    --source=dwd --dataset=immediate \
    --latitude=52.520007 --longitude=13.404954 --limit=20

List of file names of recent observations by the annual reporters:

phenodata list-filenames \
    --source=dwd --dataset=annual --partition=recent

Same as above, but with filtering by file name:

phenodata list-filenames \
    --source=dwd --dataset=annual --partition=recent \
    --filename=Hasel,Kornelkirsche,Loewenzahn,Schneegloeckchen

List full URLs instead of only file names:

phenodata list-urls \
    --source=dwd --dataset=annual --partition=recent \
    --filename=Hasel,Kornelkirsche,Loewenzahn,Schneegloeckchen

Observations

phenodata observations \
    --source=dwd --dataset=annual --partition=recent \
    --filename=Hasel,Schneegloeckchen

phenodata observations \
    --source=dwd --dataset=annual --partition=recent \
    --filename=Hasel,Schneegloeckchen --station-id=7521,7532

phenodata observations \
    --source=dwd --dataset=annual --partition=recent \
    --station-id=7521,7532 --year=2020,2021

phenodata observations \
    --source=dwd --dataset=annual --partition=recent \
    --station-id=7521,7532 --species-id=113,127

phenodata list-quality-bytes --source=dwd
phenodata observations \
    --source=dwd --dataset=annual --partition=recent \
    --quality-byte=5,6,7,8

phenodata observations \
    --source=dwd --dataset=annual --partition=recent \
    --filename=Hasel,Schneegloeckchen,Apfel,Birne \
    --station-id=12132 \
    --humanize \
    --language=german

phenodata observations \
    --source=dwd --dataset=annual --partition=recent \
    --filename=Hasel,Schneegloeckchen \
    --station=berlin,brandenburg \
    --humanize --sort=Datum

phenodata observations \
    --source=dwd --dataset=annual --partition=recent \
    --station=münchen \
    --species=hazel,snowdrop \
    --year=2022 \
    --humanize --sort=Datum

phenodata observations \
    --source=dwd --dataset=annual --partition=recent \
    --station=münchen \
    --phase=flowering \
    --year=2022 \
    --humanize --sort=Datum

phenodata observations \
    --source=dwd --dataset=annual --partition=recent \
    --station=münchen \
    --phase=flowering \
    --quality="nicht beanstandet" \
    --year=2022 \
    --humanize --sort=Datum

phenodata observations \
    --source=dwd --dataset=annual --partition=recent \
    --station=münchen \
    --phase=flowering \
    --quality=korrigiert \
    --humanize --sort=Datum

Hasel, Schneeglöckchen, Sal-Weide, Löwenzahn, Süßkirsche, Apfel, Winterraps, Robinie, Winter-Linde, Heidekraut

phenodata observations \
    --source=dwd --dataset=annual --partition=recent \
    --phase="beginning of flowering" \
    --year=2021,2022,2023 \
    --station=köln \
    --species-preset=mellifera-de-primary \
    --humanize --language=german --sort=Spezies,Datum

phenodata observations \
    --source=dwd --dataset=annual --partition=recent \
    --year=2019,2020,2021,2022,2023 \
    --species-preset=mellifera-de-primary --phase="beginning of flowering" \
    --humanize --language=german \
    --sql="SELECT * FROM data WHERE Station LIKE '%Berlin%' ORDER BY Datum" \
    --format=md

Resources

• Source code

• Documentation

• Python Package Index (PyPI)

Contributions

If you would like to contribute, you are most welcome. Spend some time taking a look around, locate a bug, design issue or spelling mistake and then send us a pull request or create an issue. Thank you in advance for your efforts, the authors really appreciate any kind of help and feedback.

Discussions

Discussions around the development of phenodata and its applications are taking place at the Hiveeyes forum. Enjoy reading them, and don’t hesitate to write in, if you think you may be able to contribute a thing or another, or to share what you have been doing with the program in form of a “show and tell” post.

• https://community.hiveeyes.org/t/phanologischer-kalender-fur-trachtpflanzen/664

• https://community.hiveeyes.org/t/phenodata-ein-datenbezug-und-manipulations-toolkit-fur-open-access-phanologiedaten/2892

• https://community.hiveeyes.org/t/phanologischer-kalender-2020/2893

• https://community.hiveeyes.org/t/klimadatenkalender-zur-anzeige-der-phanologischen-daten-des-deutschen-wetterdienstes/948

• https://community.hiveeyes.org/t/phanologie-und-imkerliche-eingriffe-bei-den-bienen/705

• https://community.hiveeyes.org/t/phenological-calendar-for-france/800

Development

In order to setup a development environment on your workstation, please head over to the development sandbox documentation. When you see the software tests succeed, you should be ready to start hacking.

Code license

The project is licensed under the terms of the GNU AGPL license, see LICENSE.

Data license

The DWD has information about their data re-use policy in German and English. Please refer to the respective Disclaimer (de, en) and Copyright (de, en) information.

Disclaimer

The project and its authors are not affiliated with DWD, GPM, USA-NPN, or any other organization in any way. It is a sole project conceived by the community, in order to make data more accessible, in the spirit of open data and open scientific data. The authors believe the world would be a better place if public data could be loaded into pandas dataframes and Xarray datasets easily.