Skip to main content

Detecting the n+1 queries problem in Python

Project description

Latest version Travis-CI Code coverage

nplusone is a library for detecting the n+1 queries problem in Python ORMs, including SQLAlchemy, Peewee, and the Django ORM.

The Problem

Many object-relational mapping (ORM) libraries default to lazy loading for relationships. This pattern can be efficient when related rows are rarely accessed, but quickly becomes inefficient as relationships are accessed more frequently. In these cases, loading related rows eagerly using a JOIN can be vastly more performant. Unfortunately, understanding when to use lazy versus eager loading can be challenging: you might not notice the problem until your app has slowed to a crawl.

nplusone is an ORM profiling tool to help diagnose and improve poor performance caused by inappropriate lazy loading. nplusone monitors applications using Django or SQLAlchemy and sends warnings when potentially expensive lazy loads are emitted. It can identify the offending relationship attribute and specific lines of code behind the problem, and recommend fixes for better performance.

nplusone also detects inappropriate eager loading for Flask-SQLAlchemy and the Django ORM, emitting a warning when related data are eagerly loaded but never accessed within the current request.

Installation

pip install -U nplusone

nplusone supports Python >= 2.7 or >= 3.3.

Usage

Note: nplusone should only be used for development and should not be deployed to production environments.

Django

Note: nplusone supports Django 1.7 and 1.8.

Add nplusone to INSTALLED_APPS:

INSTALLED_APPS = (
    ...
    'nplusone.ext.django',
)

Add NPlusOneMiddleware:

MIDDLEWARE_CLASSES = (
    ...
    'nplusone.ext.django.NPlusOneMiddleware',
)

Optionally configure logging settings:

NPLUSONE_LOGGER = logging.getLogger('nplusone')
NPLUSONE_LOG_LEVEL = logging.WARN

Configure logging handlers:

LOGGING = {
    'version': 1,
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
        },
    },
    'loggers': {
        'nplusone': {
            'handlers': ['console'],
            'level': 'WARN',
        },
    },
}

When your app loads data lazily, nplusone will emit a warning:

Potential n+1 query detected on `<model>.<field>`

Consider using select_related or prefetch_related in this case.

When your app eagerly loads related data without accessing it, nplusone will log a warning:

Potential unnecessary eager load detected on `<model>.<field>`

Flask-SQLAlchemy

Wrap application with NPlusOne:

from flask import Flask
from nplusone.ext.flask_sqlalchemy import NPlusOne

app = Flask(__name__)
NPlusOne(app)

Optionally configure logging settings:

app = Flask(__name__)
app.config['NPLUSONE_LOGGER'] = logging.getLogger('app.nplusone')
app.config['NPLUSONE_LOG_LEVEL'] = logging.ERROR
NPlusOne(app)

When your app loads data lazily, nplusone will emit a warning:

Potential n+1 query detected on `<model>.<field>`

Consider using subqueryload or joinedload in this case; see SQLAlchemy’s guide to relationship loading for complete documentation.

When your app eagerly loads related data without accessing it, nplusone will log a warning:

Potential unnecessary eager load detected on `<model>.<field>`

Ignoring Warnings

To suppress warnings thrown by intentional lazy loading, use the ignore context manager:

from nplusone.core import signals

with signals.ignore(signals.lazy_load):
    # lazy-load rows
    # ...

License

MIT licensed. See the bundled LICENSE file for more details.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

nplusone-0.2.1.tar.gz (8.3 kB view details)

Uploaded Source

Built Distribution

nplusone-0.2.1-py2.py3-none-any.whl (12.9 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file nplusone-0.2.1.tar.gz.

File metadata

  • Download URL: nplusone-0.2.1.tar.gz
  • Upload date:
  • Size: 8.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for nplusone-0.2.1.tar.gz
Algorithm Hash digest
SHA256 ab177a2190dddfe562523110b89aba70ebd647b27b086ec0f619b0b955fcd1d1
MD5 c8d854384d7ce7eb9cd22af500aee5ad
BLAKE2b-256 2a8f571eb33bfecb89fbdff38ef93005eda02f43cbc2ea0f70a775c960349e45

See more details on using hashes here.

File details

Details for the file nplusone-0.2.1-py2.py3-none-any.whl.

File metadata

File hashes

Hashes for nplusone-0.2.1-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 a3906a97a541a41fe6e6998210bdfe3e1cb6f14d09d20b81d223bb29a63aa846
MD5 b9b0f75c3df15b0076e96e0195289350
BLAKE2b-256 c2c0c0740cfee99261dc22a59221a0ffd6414da7949ecdc807635bf2ebc6eaab

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page