pytest-postgresql 3.1.0

Populating database for tests

With SQLAlchemy

This example shows how to populate database and create an SQLAlchemy’s ORM connection:

Sample below is simplified session fixture from pyramid_fullauth tests:

from sqlalchemy import create_engine
from sqlalchemy.orm import scoped_session, sessionmaker
from sqlalchemy.pool import NullPool
from zope.sqlalchemy import register


@pytest.fixture
def db_session(postgresql):
    """Session for SQLAlchemy."""
    from pyramid_fullauth.models import Base  # pylint:disable=import-outside-toplevel

    # NOTE: this fstring assumes that psycopg2 >= 2.8 is used. Not sure about it's support in psycopg2cffi (PyPy)
    connection = f'postgresql+psycopg2://{postgresql.info.user}:@{postgresql.info.host}:{postgresql.info.port}/{postgresql.info.dbname}'

    engine = create_engine(connection, echo=False, poolclass=NullPool)
    pyramid_basemodel.Session = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))
    pyramid_basemodel.bind_engine(
        engine, pyramid_basemodel.Session, should_create=True, should_drop=True)

    yield pyramid_basemodel.Session

    transaction.commit()
    Base.metadata.drop_all(engine)


@pytest.fixture
def user(db_session):
    """Test user fixture."""
    from pyramid_fullauth.models import User
    from tests.tools import DEFAULT_USER

    new_user = User(**DEFAULT_USER)
    db_session.add(new_user)
    transaction.commit()
    return new_user


def test_remove_last_admin(db_session, user):
    """
    Sample test checks internal login, but shows usage in tests with SQLAlchemy
    """
    user = db_session.merge(user)
    user.is_admin = True
    transaction.commit()
    user = db_session.merge(user)

    with pytest.raises(AttributeError):
        user.is_admin = False

Maintaining database state outside of the fixtures

It is possible and appears it’s used in other libraries for tests, to maintain database state with the use of the pytest-postgresql database managing functionality:

For this import DatabaseJanitor and use its init and drop methods:

import pytest
from pytest_postgresql.janitor import DatabaseJanitor

@pytest.fixture
func database(postgresql_proc):
    # variable definition

    janitor = DatabaseJanitor(
        postgresql_proc.user,
        postgresql_proc.host,
        postgresql_proc.port,
        "my_test_database",
        postgresql_proc.version,
        password="secret_password,
    ):
    janitor.init()
    yield psycopg2.connect(
        dbname="my_test_database",
        user=postgresql_proc.user,
        password="secret_password",
        host=postgresql_proc.host,
        port=postgresql_proc.port,
    )
    janitor.drop()

or use it as a context manager:

import pytest
from pytest_postgresql.janitor import DatabaseJanitor

@pytest.fixture
func database(postgresql_proc):
    # variable definition

    with DatabaseJanitor(
        postgresql_proc.user,
        postgresql_proc.host,
        postgresql_proc.port,
        "my_test_database",
        postgresql_proc.version,
        password="secret_password,
    ):
        yield psycopg2.connect(
            dbname="my_test_database",
            user=postgresql_proc.user,
            password="secret_password",
            host=postgresql_proc.host,
            port=postgresql_proc.port,
        )

Connecting to Postgresql (in a docker)

To connect to a docker run postgresql and run test on it, use noproc fixtures.

docker run --name some-postgres -e POSTGRES_PASSWORD=mysecretpassword -d postgres

This will start postgresql in a docker container, however using a postgresql installed locally is not much different.

In tests, make sure that all your tests are using postgresql_noproc fixture like that:

postgresql_in_docker = factories.postgresql_noproc()
postresql = factories.postgresql("postgresql_in_docker", db_name="test")


def test_postgres_docker(postresql):
    """Run test."""
    cur = postgresql.cursor()
    cur.execute("CREATE TABLE test (id serial PRIMARY KEY, num integer, data varchar);")
    postgresql.commit()
    cur.close()

And run tests:

pytest --postgresql-host=172.17.0.2 --postgresql-password=mysecretpassword

Using a common database initialisation between tests

If you’ve got several tests that require common initialisation, you need to define a load and pass it to your custom postgresql process fixture:

from pytest_postgresql.factories import postgresql, postgresql_proc
def load_database(**kwargs):
    db_connection: connection = psycopg2.connect(**kwargs)
    with db_connection.cursor() as cur:
        cur.execute("CREATE TABLE stories (id serial PRIMARY KEY, name varchar);")
        cur.execute(
            "INSERT INTO stories (name) VALUES"
            "('Silmarillion'), ('Star Wars'), ('The Expanse'), ('Battlestar Galactica')"
        )
        db_connection.commit()

postgresql_proc = postgresql_proc(
    load=[load_database],
)

postgresql = postgresql(
    "postgresql_proc",
)

You can also define your own database name by passing same dbname value to both factories.

The way this will work is that the process fixture will populate template database, which in turn will be used automatically by client fixture to create a test database from scratch. Fast, clean and no dangling transactions, that could be accidentally rolled back.

Same approach will work with noproces fixture, while connecting to already running postgresql instance whether it’ll be on a docker machine or running remotely or locally.

3.1.0

3.0.2

Bugfix

• Changed UPDATE pg_database SET to ALTER. System tables should not be updated.

3.0.1

3.0.0

2.6.1

• [bugfix] To not fail loading code if no postgresql version is installed. Fallback for janitor and process fixture only, if called upon.

2.6.0

• [enhancement] add ability to pass options to pg_ctl’s -o flag to send arguments to the underlying postgres executable Use postgres_options as fixture argument, –postgresql-postgres-options as pytest starting option or postgresql_postgres_options as pytest.ini configuration option

2.5.3

• [enhancement] Add ability to set up isolation level for fixture and janitor

2.5.2

• [fix] Status checks for running postgres depend on pg_ctl status code, not on pg_ctl log language. Fixes starting on systems without C locale. Thanks @Martin Meyries.

2.5.1

• [fix] Added LC_* env vars to running initdb and other utilities. Now all tools and server are using same, C locale

2.5.0

• [feature] Ability to define default schema to initialize database with

• [docs] Added more examples to readme on how to use the plugin

2.4.1

• [enhancement] extract NoopExecutor into it’s own submodule

• [bugfix] Ignore occasional ProcessFinishedWithError error on executor exit.

• [bugfix] Fixed setting custom password for process fixture

• [bugfix] Fix version detection, to allow for two-digit minor version part

2.4.0

• [feature] Drop support for pyhon 3.5

• [enhancement] require at least mirakuru 2.3.0 (executor’s stop method parameter’s change)

• [bug] pass password to DatabaseJanitor in client’s factory

2.3.0

• [feature] Allow to set password for postgresql. Use it throughout the flow.

• [bugfix] Default Janitor’s connections to postgres database. When using custom users, postgres attempts to use user’s database and it might not exist.

• [bugfix] NoopExecutor connects to read version by context manager to properly handle cases where it can’t connect to the server.

2.2.1

• [bugfix] Fix drop_postgresql_database to actually use DatabaseJanitor.drop instead of an init

2.2.0

• [feature] ability to properly connect to already existing postgresql server using postgresql_nooproc fixture.

2.1.0

• [enhancement] Gather helper functions maintaining postgresql database in DatabaseJanitor class.

• [deprecate] Deprecate init_postgresql_database in favour of DatabaseJanitor.init

• [deprecate] Deprecate drop_postgresql_database in favour of DatabaseJanitor.drop

2.0.0

• [feature] Drop support for python 2.7. From now on, only support python 3.5 and up

• [feature] Ability to configure database name through plugin options

• [enhancement] Use tmpdir_factory. Drop logsdir parameter

• [ehnancement] Support only Postgresql 9.0 and up

• [bugfix] Always start postgresql with LC_ALL, LC_TYPE and LANG set to C.UTF-8. It makes postgresql start in english.

1.4.1

• [bugfix] Allow creating test databse with hyphens

1.4.0

• [enhancements] Ability to configure additional options for postgresql process and connection

• [bugfix] - removed hard dependency on psycopg2, allowing any of its alternative packages, like psycopg2-binary, to be used.

• [maintenance] Drop support for python 3.4 and use 3.7 instead

1.3.4

• [bugfix] properly detect if executor running and clean after executor is being stopped

1.3.3

• [enhancement] use executor’s context manager to start/stop postrgesql server in a fixture

1.3.2

• [bugfix] version regexp to correctly catch postgresql 10

1.3.1

• [enhancement] explicitly turn off logging_collector

1.3.0

• [feature] pypy compatibility

1.2.0

• [bugfix] - disallow connection to database before it gets dropped.

• [cleanup] - removed path.py dependency

1.1.1

• [bugfix] - Fixing the default pg_ctl path creation

1.1.0

• [feature] - migrate usage of getfuncargvalue to getfixturevalue. require at least pytest 3.0.0

1.0.0

• create command line and pytest.ini configuration options for postgresql starting parameters

• create command line and pytest.ini configuration options for postgresql username

• make the port random by default

• create command line and pytest.ini configuration options for executable

• create command line and pytest.ini configuration options for host

• create command line and pytest.ini configuration options for port

• Extracted code from pytest-dbfixtures