Skip to main content

uWSGI log parser to find memory hogs

Project description

pyuwsgimemhog

https://img.shields.io/pypi/v/pyuwsgimemhog.svg https://github.com/xrmx/pyuwsgimemhog/actions/workflows/ci.yml/badge.svg

Do you want to find out which path of your application running under uWSGI is leaking memory?

pyuwsgimemhog parses uWSGI logs to point out which paths contributes to uWSGI processes RSS memory increase.

Installation

pip install pyuwsgimemhog

Requirements

In order to have the needed information you have to run uWSGI with the memory-report enabled.

Please note that it’s possible to have meaningful results only if you are using one thread per process.

Use from the command line

You need to pass a single uWSGI log file to pyuwsgimemhog:

pyuwsgimemhog --logfile /path/to/log
/api 975 200 4.9
/another-api 502 2 251

That means that /api contributed to increase the memory usage by 975 MB, it has been accounted 200 times and it contributed 4.9 MB per call. /another-api contributed 502 MB in two occurences so 251MB per call.

Use as a library

In addition to using pyuwsgimemhog as a command line utility, it can also be used as a library. This allows passing a custom path normalization function into the log analyzer in order to group requests based on that URL path.

A trivial example of using the Django URL resolver in order to normalize URLs would be the following:

from django.urls import resolve, Resolver404
from pyuwsgimemhog.pyuwsgimemhog import normalize_path, uwsgimemhog
from urllib.parse import urlparse

def normalize(url):
    try:
        return resolve(urlparse(url).path).view_name
    except Resolver404:
        # This view was not handled by Django so fall back to the default
        # normalization. Use normalize_path_with_nums to normalize numbers
        # in path to 0.
        return normalize_path(url)

with open(logfile, 'r') as f:
    for view, memory, count in uwsgimemhog(
            f, threshold * 1_000_000, normalize):
        print('{} {} {} {:.1f}'.format(
            view, memory // 1_000_000, count, memory / count / 1_000_000
        ))

License

MIT license

Credits

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.

History

0.3.0 (2021-04-24)

  • Make it possible to pass a custom normalization function (Terence D. Honles)

  • Fix regular expression to require RSS (Terence D. Honles)

  • Use Github actions instead of Travis (Terence D. Honles)

0.2.0 (2020-01-10)

  • Count and show occurrences and per call leak count.

0.1.0 (2019-04-12)

  • First release on PyPI.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pyuwsgimemhog-0.3.0.tar.gz (6.6 kB view details)

Uploaded Source

File details

Details for the file pyuwsgimemhog-0.3.0.tar.gz.

File metadata

  • Download URL: pyuwsgimemhog-0.3.0.tar.gz
  • Upload date:
  • Size: 6.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.4.2 requests/2.24.0 setuptools/52.0.0 requests-toolbelt/0.9.1 tqdm/4.57.0 CPython/3.9.2

File hashes

Hashes for pyuwsgimemhog-0.3.0.tar.gz
Algorithm Hash digest
SHA256 5309f4d16ad2a930273895c3594ba3825aba508e755a22e0415bba5280b8f02a
MD5 f061307d3beb0dad03a01b1033884acf
BLAKE2b-256 decea70ff4cedb08441a6f741d0a3f3ec152584dfdae9cb066e63dbbc70c3d79

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page