django-syzygy 1.1.0

Installation

pip install django-syzygy

Usage

Add 'syzygy' to your INSTALLED_APPS

# settings.py
INSTALLED_APPS = [
    ...
    'syzygy',
    ...
]

Setup you deployment pipeline to run migrate --pre-deploy before rolling out your code changes and migrate afterwards to apply the postponed migrations.

Concept

When dealing with database migrations in the context of an highly available application managed through continuous deployment the Django migration leaves a lot to be desired in terms of the sequencing of operations it generates.

The automatically generated schema alterations for field additions, removals, renames, and others do not account for deployments where versions of the old and the new code must co-exist for a short period of time.

For example, adding a field with a default does not persist a database level default which prevents INSERT from the pre-existing code which ignores the existence of tentatively added field from succeeding.

Figuring out the proper sequencing of operations is doable but non-trivial and error prone. Syzygy ought to provide a solution to this problem by introducing a notion of prerequisite and postponed migrations with regards to deployment and generating migrations that are aware of this sequencing.

A migration is assumed to be a prerequisite to deployment unless it contains a destructive operation or the migration has its stage class attribute set to Stage.POST_DEPLOY. When this attribute is defined it will bypass operations based heuristics.

e.g. this migration would be considered a prerequisite

class Migration(migrations.Migration):
    operations = [
        AddField('model', 'field', models.IntegerField(null=True))
    ]

while the following migrations would be postponed

class Migration(migrations.Migration):
    operations = [
        RemoveField('model', 'field'),
    ]

from syzygy import Stage

class Migration(migrations.Migration):
    stage = Stage.POST_DEPLOY

    operations = [
        RunSQL(...),
    ]

To take advantage of this new notion of migration stage the migrate command allows migrations meant to be run before a deployment to be targeted using –pre-deploy flag.

What it does and doesn’t do

Specialized operations

Syzygy overrides the makemigrations command to automatically split and organize operations in a way that allows them to safely be applied in pre and post-deployment stages.

ALTER TABLE "author" ADD COLUMN "dob" int NOT NULL DEFAULT 1988;
ALTER TABLE "author" ALTER COLUMN "dob" DROP DEFAULT;

ALTER TABLE "author" DROP COLUMN "dob";

Checks

In order to prevent the creation of migrations mixing operations of different stages this package registers system checks. These checks will generate an error for every migration with an ambiguous stage.

e.g. a migration mixing inferred stages would result in a check error

class Migration(migrations.Migration):
    operations = [
        AddField('model', 'other_field', models.IntegerField(null=True)),
        RemoveField('model', 'field'),
    ]

By default, syzygy should not generate automatically migrations and you should only run into check failures when manually creating migrations or adding syzygy to an historical project.

For migrations that are part of your project and trigger a failure of this check it is recommended to manually annotate them with proper stage: syzygy.stageStage annotations. For third party migrations you should refer to the following section.

Third-party migrations

As long as the adoption of migration stages concept is not generalized your project might depend on third-party apps containing migrations with an ambiguous sequence of operations.

Since an explicit stage cannot be explicitly assigned by editing these migrations a fallback or an override stage can be specified through the respective MIGRATION_STAGES_FALLBACK and MIGRATION_STAGES_OVERRIDE settings.

By default third-party app migrations with an ambiguous sequence of operations will fallback to Stage.PRE_DEPLOY but this behavior can be changed by setting MIGRATION_THIRD_PARTY_STAGES_FALLBACK to Stage.POST_DEPLOY or disabled by setting it to None.

Reverts

Migration revert are also supported and result in inverting the nature of migrations. A migration that is normally considered a prerequisite would then be postponed when reverted.

CI Integration

In order to ensure that no feature branch includes an ambiguous sequence of operations users are encouraged to include a job that attempts to run the migrate --pre-deploy command against a database that only includes the changes from the target branch.

For example, given a feature branch add-shiny-feature and a target branch of main a script would look like

git checkout main
python manage.py migrate
git checkout add-shiny-feature
python manage.py migrate --pre-deploy

Assuming the feature branch contains a sequence of operations that cannot be applied in a single atomic deployment consisting of pre-deployment, deployment, and post-deployment stages the migrate --pre-deploy command will fail with an AmbiguousPlan exception detailing the ambiguity and resolution paths.

Migration quorum

When deploying migrations to multiple clusters sharing the same database it’s important that:

1. Migrations are applied only once

2. Pre-deployment migrations are applied before deployment in any clusters is takes place

3. Post-deployment migrations are only applied once all clusters are done deploying

The built-in migrate command doesn’t offer any guarantees with regards to serializability of invocations, in other words naively calling migrate from multiple clusters before or after a deployment could cause some migrations to be attempted to be applied twice.

To circumvent this limitation Syzygy introduces a --quorum <N:int> flag to the migrate command that allow clusters coordination to take place.

When specified the migrate --quorum <N:int> command will wait for at least N number invocations of migrate for the planned migrations before proceeding with applying them once and blocking on all callers until the operation completes.

In order to use the --quorum feature you must configure the MIGRATION_QUORUM_BACKEND setting to point to a quorum backend such as cache based one provided by Sygyzy

MIGRATION_QUORUM_BACKEND = 'syzygy.quorum.backends.cache.CacheQuorum'

or

CACHES = {
    ...,
    'quorum': {
        ...
    },
}
MIGRATION_QUORUM_BACKEND = {
    'backend': 'syzygy.quorum.backends.cache.CacheQuorum',
    'alias': 'quorum',
}

Development

Make your changes, and then run tests via tox:

tox