Skip to main content

Package for simplify data structures migrations

Project description

Codecov Test Code Style Documentation Status PyPI version Conda-forge version

This support package simplifies data persistence between user sessions and software version updates.

The main idea of this package is simplify data migration between versions, and allow to define migration information next to data structure definition.

Basic usage (data serialization)

If You only need to serialize data, then you could use only JSON hooks

import json

from pydantic import BaseModel
from local_migrator import Encoder, object_hook


class SampleModel(BaseModel):
    field1: int
    field2: str


data = SampleModel(field1=4, field2="abc")

with open("sample.json", "w") as f_p:
    json.dump(data, f_p, cls=Encoder)

with open("sample.json") as f_p:
    data2 = json.load(f_p, object_hook=object_hook)

assert data == data2

Migrations

To register this information there is register_class decorator. It has 4 parameters:

  • version - version of data structure

  • migration_list - list of tuple (version. migration_function).

  • old_paths - list of fully qualified python paths to previous class definitions. This is to allow move class during code refactoring.

  • use_parent_migrations - if True, then parent class migrations will be used.

Lets imagine that we have such code

from local_migrator import Encoder, object_hook

class SampleModel(BaseModel):
    field1: int
    field_ca_1: str
    field_ca_2: float

with open("sample.json", "w") as f_p:
    json.dump(data, f_p, cls=Encoder)

But there is decision to move both ca field to sub structure:

class CaModel(BaseModel):
    field_1: str
    field_2: float

class SampleModel(BaseModel):
    field1: int
    field_ca: CaModel

Then with local_migrator code may look:

from local_migrator import object_hook, register_class

class CaModel(BaseModel):
    field_1: str
    field_2: float

def ca_migration_function(dkt):
    dkt["field_ca"] = CaModel(field1=dkt.pop("field_ca_1"),
                              field2=dkt.pop("field_ca_2"))
    return dkt

@register_class("0.0.1", [("0.0.1", ca_migration_function)])
class SampleModel(BaseModel):
    field1: int
    field_ca: CaModel

with open("sample.json") as f_p:
    data = json.load(f_p, object_hook=object_hook)

Assume that there is decision to rename field1 to id. Then code may look:

from local_migrator import object_hook, register_class, rename_key

class CaModel(BaseModel):
    field_1: str
    field_2: float

def ca_migration_function(dkt):
    dkt["field_ca"] = CaModel(field1=dkt.pop("field_ca_1"),
                              field2=dkt.pop("field_ca_2"))
    return dkt

@register_class("0.0.2", [("0.0.1", ca_migration_function), ("0.0.2", rename_key("field1", "id"))])
class SampleModel(BaseModel):
    id: int
    field_ca: CaModel

with open("sample.json") as f_p:
    data = json.load(f_p, object_hook=object_hook)

More examples could be found in examples section of documentation

Additional functions

  • rename_key(from_key: str, to_key: str, optional=False) -> Callable[[Dict], Dict] - helper function for rename field migrations.

  • update_argument(argument_name:str)(func: Callable) -> Callable - decorator to keep backward compatibility by converting dict argument to some class base on function type annotation

Contributing

Contributions are encouraged! Please create pull request or open issue. For PR please remember to add tests and documentation.

Additional notes

This package is originally named nme but was rename to clarify its purpose.

This package is extracted from PartSeg project for simplify reuse it in another projects.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

local-migrator-0.1.10.tar.gz (24.6 kB view details)

Uploaded Source

Built Distribution

local_migrator-0.1.10-py3-none-any.whl (11.7 kB view details)

Uploaded Python 3

File details

Details for the file local-migrator-0.1.10.tar.gz.

File metadata

  • Download URL: local-migrator-0.1.10.tar.gz
  • Upload date:
  • Size: 24.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/4.0.2 CPython/3.11.7

File hashes

Hashes for local-migrator-0.1.10.tar.gz
Algorithm Hash digest
SHA256 b836adf9ff9c6bacf7922dfd16c26f236df47e5fb492c27c50b9b98ed9ba254c
MD5 5b5a62682f0e309f219dcb91918513aa
BLAKE2b-256 1eb44d6d5ef31e3de8c5f94e89b41e5c31f31cfaab9fd68bd1ba2a03ec0b67e1

See more details on using hashes here.

File details

Details for the file local_migrator-0.1.10-py3-none-any.whl.

File metadata

File hashes

Hashes for local_migrator-0.1.10-py3-none-any.whl
Algorithm Hash digest
SHA256 e91861ed84b6d35df165371f174f7ad6ae005bc61ce4aee1e87bd1384166339d
MD5 74532dd5ad3fbd07ca9312274055fd6f
BLAKE2b-256 06369a8f451d012549aa7e1782a95b61f482bd1001f6fc7fa7ac1bad3e399dab

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page