Skip to main content

Import and export data into/from InfluxDB

Project description

Build status Coverage License PyPI Downloads per month

Python Version Status PyPI Version

About

You can use influxio to import and export data into/from InfluxDB. It can be used both as a standalone program, and as a library.

influxio is, amongst others, based on the excellent dask, fsspec, influxdb-client, influx-line, line-protocol-parser, pandas, and SQLAlchemy packages.

Please note that influxio is alpha-quality software, and a work in progress. Contributions of all kinds are very welcome, in order to make it more solid. Breaking changes should be expected until a 1.0 release, so version pinning is recommended, especially when you use it as a library.

Synopsis

# Export from API to database.
influxio copy \
    "http://example:token@localhost:8086/testdrive/demo" \
    "sqlite://export.sqlite?table=demo"

# Export from data directory to line protocol format.
influxio copy \
    "file:///path/to/influxdb/engine?bucket-id=372d1908eab801a6&measurement=demo" \
    "file://export.lp"

Quickstart

If you are in a hurry, and want to run influxio without any installation, just use the OCI image on Podman or Docker.

docker run --rm --network=host ghcr.io/daq-tools/influxio \
    influxio copy \
    "http://example:token@localhost:8086/testdrive/demo" \
    "crate://crate@localhost:4200/testdrive/demo"

Setup

Install influxio from PyPI.

pip install influxio

Usage

This section outlines some example invocations of influxio, both on the command line, and per library use. Other than the resources available from the web, testing data can be acquired from the repository’s testdata folder.

Prerequisites

For properly running some of the example invocations outlined below, you will need an InfluxDB and a CrateDB server. The easiest way to spin up those instances is to use Podman or Docker.

Please visit the docs/development.rst documentation to learn about how to spin up corresponding sandbox instances on your workstation.

Command line use

Help

influxio --help
influxio info
influxio copy --help

Import

# From test data to API.
# Choose one of dummy, mixed, dateindex, wide.
influxio copy \
    "testdata://dateindex/" \
    "http://example:token@localhost:8086/testdrive/demo"

# With selected amount of rows.
influxio copy \
    "testdata://dateindex/?rows=42" \
    "http://example:token@localhost:8086/testdrive/demo"

# With selected amount of rows and columns (only supported by certain test data sources).
influxio copy \
    "testdata://wide/?rows=42&columns=42" \
    "http://example:token@localhost:8086/testdrive/demo"

# From line protocol file to InfluxDB API.
influxio copy \
    "file://tests/testdata/basic.lp" \
    "http://example:token@localhost:8086/testdrive/demo"

# From line protocol file to InfluxDB API.
influxio copy \
    "https://github.com/influxdata/influxdb2-sample-data/raw/master/air-sensor-data/air-sensor-data.lp" \
    "http://example:token@localhost:8086/testdrive/demo"

# From line protocol file to any database supported by SQLAlchemy.
influxio copy \
    "file://export.lp" \
    "sqlite://export.sqlite?table=export"

Export from API

# From API to database file.
influxio copy \
    "http://example:token@localhost:8086/testdrive/demo" \
    "sqlite:///export.sqlite?table=demo"

# From API to database server.
influxio copy \
    "http://example:token@localhost:8086/testdrive/demo" \
    "crate://crate@localhost:4200/testdrive?table=demo"

# From API to line protocol file.
influxio copy \
    "http://example:token@localhost:8086/testdrive/demo" \
    "file://export.lp"

# From API to line protocol on stdout.
influxio copy \
    "http://example:token@localhost:8086/testdrive/demo" \
    "file://-?format=lp"

Export from data directory

# From InfluxDB data directory to line protocol file.
influxio copy \
    "file:///path/to/influxdb/engine?bucket-id=372d1908eab801a6&measurement=demo" \
    "file://export.lp"

# From InfluxDB data directory to line protocol file, compressed with gzip.
influxio copy \
    "file:///path/to/influxdb/engine?bucket-id=372d1908eab801a6&measurement=demo" \
    "file://export.lp.gz"

# From InfluxDB data directory to line protocol on stdout.
influxio copy \
    "file:///path/to/influxdb/engine?bucket-id=372d1908eab801a6&measurement=demo" \
    ""file://-?format=lp"

OCI

OCI images are available on the GitHub Container Registry (GHCR). In order to run them on Podman or Docker, invoke:

docker run --rm --network=host ghcr.io/daq-tools/influxio \
    influxio copy \
    "http://example:token@localhost:8086/testdrive/demo" \
    "stdout://export.lp"

If you want to work with files on your filesystem, you will need to either mount the working directory into the container using the --volume option, or use the --interactive option to consume STDIN, like:

docker run --rm --volume=$(pwd):/data ghcr.io/daq-tools/influxio \
    influxio copy "file:///data/export.lp" "sqlite:///data/export.sqlite?table=export"

cat export.lp | \
docker run --rm --interactive --network=host ghcr.io/daq-tools/influxio \
    influxio copy "stdin://?format=lp" "crate://crate@localhost:4200/testdrive/export"

In order to always run the latest nightly development version, and to use a shortcut for that, this section outlines how to use an alias for influxio, and a variable for storing the input URL. It may be useful to save a few keystrokes on subsequent invocations.

docker pull ghcr.io/daq-tools/influxio:nightly
alias influxio="docker run --rm --interactive ghcr.io/daq-tools/influxio:nightly influxio"
SOURCE=https://github.com/daq-tools/influxio/raw/main/tests/testdata/basic.lp
TARGET=crate://crate@localhost:4200/testdrive/basic

influxio copy "${SOURCE}" "${TARGET}"

Project information

Development

For installing the project from source, please follow the development documentation.

Prior art

There are a few other projects which are aiming at similar goals.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

influxio-0.2.0.tar.gz (23.8 kB view details)

Uploaded Source

Built Distribution

influxio-0.2.0-py3-none-any.whl (20.7 kB view details)

Uploaded Python 3

File details

Details for the file influxio-0.2.0.tar.gz.

File metadata

  • Download URL: influxio-0.2.0.tar.gz
  • Upload date:
  • Size: 23.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.7

File hashes

Hashes for influxio-0.2.0.tar.gz
Algorithm Hash digest
SHA256 79525a7cebe353c1f100ab7c8dd1c02167179a01b64f9fe86b4222939976860f
MD5 efd81a552e4162ca0dae86e7dc67c285
BLAKE2b-256 921eb68aab39a96409a3df10e104519d251efd02cee90ac39329bd1ab76645e9

See more details on using hashes here.

Provenance

File details

Details for the file influxio-0.2.0-py3-none-any.whl.

File metadata

  • Download URL: influxio-0.2.0-py3-none-any.whl
  • Upload date:
  • Size: 20.7 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.7

File hashes

Hashes for influxio-0.2.0-py3-none-any.whl
Algorithm Hash digest
SHA256 5d0b6e0a7779cdf065d373ba0a32d95f06934e1866e1ee8b06e67e99b197f22d
MD5 41e8060eef179fc5844f4028849d0e9b
BLAKE2b-256 1cd55a72c04f31d61a7ba312df993f2e7901468fa3c2f3eccba59e7f21f0f23d

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page