Skip to main content

Consistent partial database dump utility

Project description

XDump

Build Status Coverage Status Documentation Status Latest PyPI version

XDump is a utility to make a consistent partial dump and load it into the database.

The idea is to provide an ability to specify what to include in the dump via SQL queries.

Installation

XDump can be obtained with pip:

$ pip install xdump

Usage example

Make a dump (on production replica for example):

>>> from xdump.postgresql import PostgreSQLBackend
>>>
>>> backend = PostgreSQLBackend(dbname='app_db', user='prod', password='pass', host='127.0.0.1', port='5432')
>>> backend.dump(
    '/path/to/dump.zip',
    full_tables=['groups'],
    partial_tables={'employees': 'SELECT * FROM employees ORDER BY id DESC LIMIT 2'}
)

Load a dump on your local machine:

>>> backend = PostgreSQLBackend(dbname='app_db', user='local', password='pass', host='127.0.0.1', port='5432')
# If you need a clear DB
>>> backend.recreate_database()  # or `backend.truncate()`
>>> backend.load('/path/to/dump.zip')

Dump is compressed by default. Compression level could be changed with passing compression argument to dump method. Valid options are zipfile.ZIP_STORED, zipfile.ZIP_DEFLATED, zipfile.ZIP_BZIP2 and zipfile.ZIP_LZMA.

The verbosity of the output could be customized via verbosity (with values 0, 1 or 2) argument of a backend class.

There are two options to control the content of the dump:

  • dump_schema - controls if the schema should be included

  • dump_data - controls if the data should be included

Command Line Interface

xload provides an ability to create a dump.

Signature:

xdump [postgres|sqlite] [OPTIONS]

Common options:

-o, --output TEXT               output file name  [required]
-f, --full TEXT                 table name to be fully dumped. Could be used
                                multiple times
-p, --partial TEXT              partial tables specification in a form
                                "table_name:select SQL". Could be used
                                multiple times
-c, --compression [deflated|stored|bzip2|lzma]
                                dump compression level
--schema / --no-schema          include / exclude the schema from the dump
--data / --no-data              include / exclude the data from the dump
-D, --dbname TEXT               database to work with  [required]
-v, --verbosity                 verbosity level

PostgreSQL-specific options:

-U, --user TEXT                 connect as specified database user
                                [required]
-W, --password TEXT             password for the DB connection
-H, --host TEXT                 database server host or socket directory
-P, --port TEXT                 database server port number

xload loads a dump into a database.

Signature:

xload [postgres|sqlite] [OPTIONS]

Common options:

-i, --input TEXT                input file name  [required]
-m, --cleanup-method [recreate|truncate]
                                method of DB cleaning up
-D, --dbname TEXT               database to work with  [required]
-v, --verbosity                 verbosity level

PostgreSQL-specific options are the same as for xdump.

RDBMS support

At the moment only the following are supported:

  • PostgreSQL

  • SQLite >= 3.8.3

Django support

Add xdump.extra.django to your INSTALLED_APPS settings:

INSTALLED_APPS = [
   ...,
   'xdump.extra.django',
]

Add XDUMP to your project settings file. It should contain minimum two entries:

  • FULL_TABLES - a list of tables that should be fully dumped.

  • PARTIAL_TABLES - a dictionary with table_name: select SQL

XDUMP = {
    'FULL_TABLES': ['groups'],
    'PARTIAL_TABLES': {'employees': 'SELECT * FROM employees WHERE id > 100'}
}

Optionally you could use a custom backend:

XDUMP = {
    ...,
    'BACKEND': 'importable.string',
}

Run xdump command:

$ ./manage.py xdump dump.zip

Run xload command:

$ ./manage.py xload dump.zip

Possible options to both commands:

  • -a/--alias - allows you to choose database config from DATABASES, that is used during the execution;

  • -b/--backend - importable string, that leads to custom dump backend class.

Options for xdump command:

  • -s/--dump-schema - controls if the schema should be included;

  • -d/--dump-data - controls if the data should be included.

Options for xload command:

  • -m/--cleanup-method - optionally re-creates DB or truncates the data.

NOTE. If the dump has no schema inside, DB won’t be re-created.

The following make command could be useful to get a configured dump from production to your local machine:

sync-production:
    ssh -t $(TARGET) "DJANGO_SETTINGS_MODULE=settings.production /path/to/manage.py xdump /tmp/dump.zip"
    scp $(TARGET):/tmp/dump.zip ./dump.zip
    ssh -t $(TARGET) "rm /tmp/dump.zip"
    DJANGO_SETTINGS_MODULE=settings.local $(PYTHON) manage.py xload ./dump.zip

And the usage is:

$ make sync-production TARGET=john@production.com PYTHON=/path/to/python/in/venv

Python support

XDump supports Python 2.7, 3.4 - 3.7 and PyPy 2 & 3.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

xdump-0.6.0.tar.gz (17.1 kB view details)

Uploaded Source

Built Distribution

xdump-0.6.0-py2.py3-none-any.whl (19.2 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file xdump-0.6.0.tar.gz.

File metadata

  • Download URL: xdump-0.6.0.tar.gz
  • Upload date:
  • Size: 17.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.18.4 setuptools/39.0.1 requests-toolbelt/0.8.0 tqdm/4.23.3 CPython/3.6.5

File hashes

Hashes for xdump-0.6.0.tar.gz
Algorithm Hash digest
SHA256 059729a4d18ea358aee262bcb537b59411606faed40a6eba43b7362cd9c7654b
MD5 07e3588fe061ca52fbc2c9c84307230a
BLAKE2b-256 4625022f9ef5db00f802c7279a56bbea1758ac4b7e8be4d2c3e869f8203dc57b

See more details on using hashes here.

File details

Details for the file xdump-0.6.0-py2.py3-none-any.whl.

File metadata

  • Download URL: xdump-0.6.0-py2.py3-none-any.whl
  • Upload date:
  • Size: 19.2 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.18.4 setuptools/39.0.1 requests-toolbelt/0.8.0 tqdm/4.23.3 CPython/3.6.5

File hashes

Hashes for xdump-0.6.0-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 60d0c98084e8b98bb9159bdae01171b3ab246d8f74f6e0ae1925b1ef59603dab
MD5 39ffd77a2451f732900f5eefecd579a1
BLAKE2b-256 46bc86d43a197c2948e2c10749bd7ff8fd213a97baf2bad37bffc351b8ec79d8

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page