Consistent partial database dump utility
Project description
XDump
XDump is a utility to make a consistent partial dump and load it into the database.
The idea is to provide an ability to specify what to include in the dump via SQL queries.
Installation
XDump can be obtained with pip:
$ pip install xdump
Usage example
Make a dump (on production replica for example):
>>> from xdump.postgresql import PostgreSQLBackend
>>>
>>> backend = PostgreSQLBackend(dbname='app_db', user='prod', password='pass', host='127.0.0.1', port='5432')
>>> backend.dump(
'/path/to/dump.zip',
full_tables=['groups'],
partial_tables={'employees': 'SELECT * FROM employees ORDER BY id DESC LIMIT 2'}
)
Load a dump on your local machine:
>>> backend = PostgreSQLBackend(dbname='app_db', user='local', password='pass', host='127.0.0.1', port='5432')
# If you need a clear DB
>>> backend.recreate_database() # or `backend.truncate()`
>>> backend.load('/path/to/dump.zip')
Dump is compressed by default. Compression level could be changed with passing compression argument to dump method. Valid options are zipfile.ZIP_STORED, zipfile.ZIP_DEFLATED, zipfile.ZIP_BZIP2 and zipfile.ZIP_LZMA.
The verbosity of the output could be customized via verbosity (with values 0, 1 or 2) argument of a backend class.
There are two options to control the content of the dump:
dump_schema - controls if the schema should be included
dump_data - controls if the data should be included
Command Line Interface
xload provides an ability to create a dump.
Signature:
xdump [postgres|sqlite] [OPTIONS]
Common options:
-o, --output TEXT output file name [required] -f, --full TEXT table name to be fully dumped. Could be used multiple times -p, --partial TEXT partial tables specification in a form "table_name:select SQL". Could be used multiple times -c, --compression [deflated|stored|bzip2|lzma] dump compression level --schema / --no-schema include / exclude the schema from the dump --data / --no-data include / exclude the data from the dump -D, --dbname TEXT database to work with [required] -v, --verbosity verbosity level
PostgreSQL-specific options:
-U, --user TEXT connect as specified database user [required] -W, --password TEXT password for the DB connection -H, --host TEXT database server host or socket directory -P, --port TEXT database server port number
xload loads a dump into a database.
Signature:
xload [postgres|sqlite] [OPTIONS]
Common options:
-i, --input TEXT input file name [required] -m, --cleanup-method [recreate|truncate] method of DB cleaning up -D, --dbname TEXT database to work with [required] -v, --verbosity verbosity level
PostgreSQL-specific options are the same as for xdump.
RDBMS support
At the moment only the following are supported:
PostgreSQL
SQLite >= 3.8.3
Django support
Add xdump.extra.django to your INSTALLED_APPS settings:
INSTALLED_APPS = [
...,
'xdump.extra.django',
]
Add XDUMP to your project settings file. It should contain minimum two entries:
FULL_TABLES - a list of tables that should be fully dumped.
PARTIAL_TABLES - a dictionary with table_name: select SQL
XDUMP = {
'FULL_TABLES': ['groups'],
'PARTIAL_TABLES': {'employees': 'SELECT * FROM employees WHERE id > 100'}
}
Optionally you could use a custom backend:
XDUMP = {
...,
'BACKEND': 'importable.string',
}
Run xdump command:
$ ./manage.py xdump dump.zip
Run xload command:
$ ./manage.py xload dump.zip
Possible options to both commands:
-a/--alias - allows you to choose database config from DATABASES, that is used during the execution;
-b/--backend - importable string, that leads to custom dump backend class.
Options for xdump command:
-s/--dump-schema - controls if the schema should be included;
-d/--dump-data - controls if the data should be included.
Options for xload command:
-m/--cleanup-method - optionally re-creates DB or truncates the data.
NOTE. If the dump has no schema inside, DB won’t be re-created.
The following make command could be useful to get a configured dump from production to your local machine:
sync-production:
ssh -t $(TARGET) "DJANGO_SETTINGS_MODULE=settings.production /path/to/manage.py xdump /tmp/dump.zip"
scp $(TARGET):/tmp/dump.zip ./dump.zip
ssh -t $(TARGET) "rm /tmp/dump.zip"
DJANGO_SETTINGS_MODULE=settings.local $(PYTHON) manage.py xload ./dump.zip
And the usage is:
$ make sync-production TARGET=john@production.com PYTHON=/path/to/python/in/venv
Python support
XDump supports Python 2.7, 3.4 - 3.7 and PyPy 2 & 3.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file xdump-0.6.0.tar.gz
.
File metadata
- Download URL: xdump-0.6.0.tar.gz
- Upload date:
- Size: 17.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.18.4 setuptools/39.0.1 requests-toolbelt/0.8.0 tqdm/4.23.3 CPython/3.6.5
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 059729a4d18ea358aee262bcb537b59411606faed40a6eba43b7362cd9c7654b |
|
MD5 | 07e3588fe061ca52fbc2c9c84307230a |
|
BLAKE2b-256 | 4625022f9ef5db00f802c7279a56bbea1758ac4b7e8be4d2c3e869f8203dc57b |
File details
Details for the file xdump-0.6.0-py2.py3-none-any.whl
.
File metadata
- Download URL: xdump-0.6.0-py2.py3-none-any.whl
- Upload date:
- Size: 19.2 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.18.4 setuptools/39.0.1 requests-toolbelt/0.8.0 tqdm/4.23.3 CPython/3.6.5
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 60d0c98084e8b98bb9159bdae01171b3ab246d8f74f6e0ae1925b1ef59603dab |
|
MD5 | 39ffd77a2451f732900f5eefecd579a1 |
|
BLAKE2b-256 | 46bc86d43a197c2948e2c10749bd7ff8fd213a97baf2bad37bffc351b8ec79d8 |