Skip to main content

Process data from live API of luftdaten.info

Project description

https://img.shields.io/badge/Python-2.7,%203.6-green.svg https://img.shields.io/pypi/v/luftdatenpumpe.svg https://img.shields.io/github/tag/hiveeyes/luftdatenpumpe.svg

Luftdatenpumpe

https://cdn.jsdelivr.net/gh/hiveeyes/luftdatenpumpe@master/doc/logo.svg

About

  1. Luftdatenpumpe acquires the current window of measurement readings from the livedata API of luftdaten.info.

  2. While iterating the readings, it collects information about all stations and sensors they are originating from.

  3. Then, each stations location information gets enhanced by

    • attaching its geospatial position as a Geohash.

    • attaching a synthetic real-world address resolved using the reverse geocoding service Nominatim by OpenStreetMap.

  4. Information about stations can be

    • displayed on STDOUT or STDERR in JSON format.

    • filtered and transformed interactively through jq, the swiss army knife of JSON manipulation.

    • stored into RDBMS databases using the fine dataset package. Being built on top of SQLAlchemy, this supports all major databases.

  5. Measurement readings can be

    • displayed on STDOUT or STDERR in JSON format, which allows for piping into jq again.

    • forwarded to MQTT.

Demo

Live Data

List of stations

Synopsis

Overview

# List stations
luftdatenpumpe stations --station=28,297 --reverse-geocode

# Write list of stations and metadata to PostgreSQL database
luftdatenpumpe stations --station=28,1071 --reverse-geocode --target=postgresql:///weatherbase

# Forward readings to MQTT
luftdatenpumpe readings --station=28,1071 --target=mqtt://mqtt.example.org/luftdaten.info

Details

$ luftdatenpumpe --help
    Usage:
      luftdatenpumpe stations [options] [--target=<target>]...
      luftdatenpumpe readings [options] [--target=<target>]...
      luftdatenpumpe --version
      luftdatenpumpe (-h | --help)

    Options:
      --station=<stations>          Filter data by given location ids, comma-separated.
      --sensor=<sensors>            Filter data by given sensor ids, comma-separated.
      --reverse-geocode             Compute geographical address using the Nominatim reverse geocoder and add to MQTT message
      --target=<target>             Data output target
      --progress                    Show progress bar
      --version                     Show version information
      --dry-run                     Run data acquisition and postprocessing but skip publishing to MQTT bus
      --debug                       Enable debug messages
      -h --help                     Show this screen

    Station list examples:

      # Display metadata for given stations in JSON format
      luftdatenpumpe stations --station=28,1071 --reverse-geocode

      # Display metadata for given sensors in JSON format
      luftdatenpumpe stations --sensor=657,2130 --reverse-geocode

      # Display list of stations in JSON format, suitable for integrating with Grafana
      luftdatenpumpe stations --station=28,1071 --reverse-geocode --target=json.grafana+stream://sys.stdout

      # Write list of stations and metadata to PostgreSQL database, also display on STDERR
      luftdatenpumpe stations --station=28,1071 --reverse-geocode --target=postgresql:///weatherbase --target=json+stream://sys.stderr

    Data examples:

      # Publish data to topic "luftdaten.info" at MQTT broker "mqtt.example.org"
      luftdatenpumpe readings --station=28,1071 --target=json+stream://sys.stderr --target=mqtt://mqtt.example.org/luftdaten.info

      # MQTT publishing, with authentication
      luftdatenpumpe readings --station=28,1071 --target=mqtt://username:password@mqtt.example.org/luftdaten.info

Setup

Prerequisites

Debian packages:

apt install postgis redis-server redis-tools
Postgres database

Create database:

createuser --no-createdb --pwprompt hiveeyes
createdb --owner hiveeyes weatherbase

Create read-only user:

su - postgres
psql

postgres=# \c weatherbase
weatherbase=# CREATE ROLE readonly WITH LOGIN PASSWORD 'XXX';
weatherbase=# GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public TO readonly;
weatherbase=# GRANT SELECT ON ALL TABLES IN SCHEMA public TO readonly;
Redis cache

This program extensively uses a runtime cache based on Redis. To make this work best, you should enable data durability with your Redis instance.

The append-only file is an alternative, fully-durable strategy for Redis. It became available in version 1.1. You can turn on the AOF in your Redis configuration file (e.g. /etc/redis/redis.conf):

appendonly yes

Python module

# Not published yet, please install from repository
pip install luftdatenpumpe

References

Upstream luftdaten.info

Standing on the shoulders of giants

Development

License

This program is free software; you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with this program; if not, see: <http://www.gnu.org/licenses/agpl-3.0.txt>, or write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA

Content attributions

The copyright of particular images and pictograms are held by their respective owners, unless otherwise noted.

Icons and pictograms

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

luftdatenpumpe-0.4.3.tar.gz (19.4 kB view details)

Uploaded Source

File details

Details for the file luftdatenpumpe-0.4.3.tar.gz.

File metadata

  • Download URL: luftdatenpumpe-0.4.3.tar.gz
  • Upload date:
  • Size: 19.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.21.0 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.28.1 CPython/2.7.15

File hashes

Hashes for luftdatenpumpe-0.4.3.tar.gz
Algorithm Hash digest
SHA256 38bac907342b53ab68866dd2548e1dd3556fd9d53a1bd3bff931b44015e01e12
MD5 86926fab0e1e790b10a64feaa7a7b505
BLAKE2b-256 8b18dde208700ca0d863fe54e683129a18c364e1fc5854547993a660b953a5f9

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page