Skip to main content

No project description provided

Project description

CI Status

Why

Utilities for loading and dumping database data as JSON.

These utilities (partially) replace Django’s built-in dumpdata and loaddata management commands.

Suppose you want to move data between systems incrementally. In this case it isn’t sufficient to only know the data which has been created or updated; you also want to know which data has been deleted in the meantime. Django’s dumpdata and loaddata management commands only support the former case, not the latter. They also do not including dependent objects in the dump.

This package offers utilities and management commands to address these shortcomings.

How

pip install feincms3-data.

Add feincms3_data to INSTALLED_APPS so that the included management commands are discovered.

Add datasets somewhere describing the models and relationships you want to dump, e.g. in a module named app.f3datasets:

from feincms3_data.data import (
    specs_for_app_models,
    specs_for_derived_models,
    specs_for_models,
)
from app.dashboard import models as dashboard_models
from app.world import models as world_models


def districts(args):
    pks = [int(arg) for arg in args.split(",") if arg]
    return [
        *specs_for_models(
            [world_models.District],
            {
                "filter": {"pk__in": pks},
                "delete_missing": True,
            },
        ),
        *specs_for_models(
            [world_models.Exercise],
            {
                "filter": {"district__in": pks},
                "delete_missing": True,
            },
        ),
        # All derived non-abstract models which aren't proxies:
        *specs_for_derived_models(
            world_models.ExercisePlugin,
            {
                "filter": {"parent__district__in": pks},
                "delete_missing": True,
            },
        ),
    ]


def datasets():
    return {
        "articles": {
            "specs": lambda args: specs_for_app_models(
                "articles",
                {"delete_missing": True},
            ),
        },
        "pages": {
            "specs": lambda args: specs_for_app_models(
                "pages",
                {"delete_missing": True},
            ),
        },
        "teachingmaterials": {
            "specs": lambda args: specs_for_models(
                [
                    dashboard_models.TeachingMaterialGroup,
                    dashboard_models.TeachingMaterial,
                ],
                {"delete_missing": True},
            ),
        },
        "districts": {
            "specs": districts,
        },
    }

Add a setting with the Python module path to the specs function:

FEINCMS3_DATA_DATASETS = "app.f3datasets.datasets"

Now, to dump e.g. pages you would run:

./manage.py f3dumpdata pages > tmp/pages.json

To dump the districts with the primary key of 42 and 43 you would run:

./manage.py f3dumpdata districts:42,43 > tmp/districts.json

The resulting JSON file has three top-level keys:

  • "version": 1: The version of the dump, because not versioning dumps is a recipe for pain down the road.

  • "specs": [...]: A list of model specs.

  • "objects": [...]: A list of model instances; uses the same serializer as Django’s dumpdata, everything looks the same.

Model specs consist of the following fields:

  • "model": The lowercased label (app_label.model_name) of a model.

  • "filter": A dictionary which can be passed to the .filter() queryset method as keyword arguments; used for determining the objects to dump and the objects to remove after loading.

  • "delete_missing": This flag makes the loader delete all objects matching "filter" which do not exist in the dump.

  • "ignore_missing_m2m": A list of field names where deletions of related models should be ignored when restoring. This may be especially useful when only transferring content partially between databases.

  • "save_as_new": If present and truish, objects are inserted using new primary keys into the database instead of (potentially) overwriting pre-existing objects.

  • "defer_values": A list of fields which should receive random garbage when loading initially and only receive their real value later. This is especially useful to avoid unique constraint errors when loading partial graphs.

The dumps can be loaded back into the database by running:

./manage.py f3loaddata -v2 tmp/pages.json tmp/districts.json

Each dump is processed in an individual transaction. The data is first loaded into the database; at the end, data matching the filters but whose primary key wasn’t contained in the dump is deleted from the database (if "delete_missing": True).

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

feincms3_data-0.7.0.tar.gz (15.9 kB view details)

Uploaded Source

Built Distribution

feincms3_data-0.7.0-py3-none-any.whl (8.9 kB view details)

Uploaded Python 3

File details

Details for the file feincms3_data-0.7.0.tar.gz.

File metadata

  • Download URL: feincms3_data-0.7.0.tar.gz
  • Upload date:
  • Size: 15.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: python-httpx/0.27.2

File hashes

Hashes for feincms3_data-0.7.0.tar.gz
Algorithm Hash digest
SHA256 99518f09f8981d98ca7339be3450005c27c0758d82fdbd98af049c33bdaaa40a
MD5 877e7ed3d7e97bf7d9aae4449cf54ea1
BLAKE2b-256 b7749842d2509838e51e3a18feea2184a2f853f99f45aea0992c30b3cbb60a2f

See more details on using hashes here.

Provenance

File details

Details for the file feincms3_data-0.7.0-py3-none-any.whl.

File metadata

File hashes

Hashes for feincms3_data-0.7.0-py3-none-any.whl
Algorithm Hash digest
SHA256 df32fda3c6dc4800d7690eac1dc5985c5f3d2821013b69fd5a128e87c8252bd4
MD5 9b114fbcbda4ba9b381aa41468f41cde
BLAKE2b-256 364e304ad08ce3590ed91efef30f1791c41128f4c39ce0a7d3e08aa60aad971b

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page