Skip to main content

A memory profiler for data batch processing applications.

Project description

The Fil memory profiler for Python

Your code reads some data, processes it, and uses too much memory. In order to reduce memory usage, you need to figure out:

  1. Where peak memory usage is, also known as the high-water mark.
  2. What code was responsible for allocating the memory that was present at that peak moment.

That's exactly what Fil will help you find. Fil an open source memory profiler designed for data processing applications written in Python, and includes native support for Jupyter.

At the moment it only runs on Linux and macOS, and while it supports threading, it does not yet support multiprocessing or multiple processes in general.

"Within minutes of using your tool, I was able to identify a major memory bottleneck that I never would have thought existed. The ability to track memory allocated via the Python interface and also C allocation is awesome, especially for my NumPy / Pandas programs."

—Derrick Kondo

For more information, including an example of the output, see https://pythonspeed.com/products/filmemoryprofiler/

Fil vs. other Python memory tools

There are two distinct patterns of Python usage, each with its own source of memory problems.

In a long-running server, memory usage can grow indefinitely due to memory leaks. That is, some memory is not being freed.

  • If the issue is in Python code, tools like tracemalloc and Pympler can tell you which objects are leaking and what is preventing them from being leaked.
  • If you're leaking memory in C code, you can use tools like Valgrind.

Fil, however, is not aimed at memory leaks, but at the other use case: data processing applications. These applications load in data, process it somehow, and then finish running.

The problem with these applications is that they can, on purpose or by mistake, allocate huge amounts of memory. It might get freed soon after, but if you allocate 16GB RAM and only have 8GB in your computer, the lack of leaks doesn't help you.

Fil will therefore tell you, in an easy to understand way:

  1. Where peak memory usage is, also known as the high-water mark.
  2. What code was responsible for allocating the memory that was present at that peak moment.
  3. This includes C/Fortran/C++/whatever extensions that don't use Python's memory allocation API (tracemalloc only does Python memory APIs).

This allows you to optimize that code in a variety of ways.

Installation

Assuming you're on macOS or Linux, and are using Python 3.6 or later, you can use either Conda or pip (or any tool that is pip-compatible and can install manylinux2010 wheels).

Conda

To install on Conda:

$ conda install -c conda-forge filprofiler

Pip

To install the latest version of Fil you'll need Pip 19 or newer. You can check like this:

$ pip --version
pip 19.3.0

If you're using something older than v19, you can upgrade by doing:

$ pip install --upgrade pip

If that doesn't work, try running your code in a virtualenv:

$ python3 -m venv venv/
$ . venv/bin/activate
(venv) $ pip install --upgrade pip

Assuming you have a new enough version of pip:

$ pip install filprofiler

Using Fil

Profiling in Jupyter

To measure peak memory usage of some code in Jupyter you need to do three things:

  1. Use an alternative kernel, "Python 3 with Fil". You can choose this kernel when you create a new notebook, or you can switch an existing notebook in the Kernel menu; there should be a "Change Kernel" option in there in both Jupyter Notebook and JupyterLab.
  2. Load the extension by doing %load_ext filprofiler.
  3. Add the %%filprofile magic to the top of the cell with the code you wish to profile.

Screenshot of JupyterLab

Profiling complete Python programs

Instead of doing:

$ python yourscript.py --input-file=yourfile

Just do:

$ fil-profile run yourscript.py --input-file=yourfile

And it will generate a report and automatically try to open it in for you in a browser. Reports will be stored in the fil-result/ directory in your current working directory.

As of version 0.11, you can also run it like this:

$ python -m filprofiler run yourscript.py --input-file=yourfile

API for profiling specific Python functions

You can also measure memory usage in part of your program; this requires version 0.15 or later. This requires two steps.

1. Add profiling in your code

Let's you have some code that does the following:

def main():
    config = load_config()
    result = run_processing(config)
    generate_report(result)

You only want to get memory profiling for the run_processing() call.

You can do so in the code like so:

from filprofiler.api import profile

def main():
    config = load_config()
    result = profile(lambda: run_processing(config), "/tmp/fil-result")
    generate_report(result)

You could also make it conditional, e.g. based on an environment variable:

import os
from filprofiler.api import profile

def main():
    config = load_config()
    if os.environ.get("FIL_PROFILE"):
        result = profile(lambda: run_processing(config), "/tmp/fil-result")
    else:
        result = run_processing(config)
    generate_report(result)

2. Run your script with Fil

You still need to run your program in a special way. If previously you did:

$ python yourscript.py --config=myconfig

Now you would do:

$ filprofiler python yourscript.py --config=myconfig

Notice that you're doing filprofiler python, rather than filprofiler run as you would if you were profiling the full script. Only functions explicitly called with the filprofiler.api.profile() will have memory profiling enabled; the rest of the code will run at (close) to normal speed and configuration. Each call to profile() will generate a separate report.

The memory profiling report will be written to the directory specified as the output destination when calling profile(); in or example above that was "/tmp/fil-result". Unlike full-program profiling:

  1. The directory you give will be used directly, there won't be timestamped sub-directories. If there are multiple calls to profile(), it is your responsibility to ensure each call writes to a unique directory.
  2. The report(s) will not be opened in a browser automatically, on the presumption you're running this in an automated fashion.

Debugging out-of-memory crashes

New in v0.14 and later: Just run your program under Fil, and it will generate a SVG at the point in time when memory runs out, and then exit with exit code 53:

$ fil-profile run oom.py 
...
=fil-profile= Wrote memory usage flamegraph to fil-result/2020-06-15T12:37:13.033/out-of-memory.svg

Fil uses three heuristics to determine if the process is close to running out of memory:

  • A failed allocation, indicating insufficient memory is available.
  • The operating system or memory-limited cgroup (e.g. a Docker container) only has 100MB of RAM available.
  • The process swap is larger than available memory, indicating heavy swapping by the process. In general you want to avoid swapping, and e.g. explicitly use mmap() if you expect to be using disk as a backfill for memory.

Reducing memory usage in your code

You've found where memory usage is coming from—now what?

If you're using data processing or scientific computing libraries, I have written a relevant guide to reducing memory usage.

How Fil works

Fil uses the LD_PRELOAD/DYLD_INSERT_LIBRARIES mechanism to preload a shared library at process startup. This shared library captures all memory allocations and deallocations and keeps track of them.

At the same time, the Python tracing infrastructure (used e.g. by cProfile and coverage.py) to figure out which Python callstack/backtrace is responsible for each allocation.

For performance reasons, only the largest allocations are reported, with a minimum of 99% of allocated memory reported. The remaining <1% is highly unlikely to be relevant when trying to reduce usage; it's effectively noise.

Fil and threading, with notes on NumPy and Zarr {#threading}

In general, Fil will track allocations in threads correctly.

First, if you start a thread via Python, running Python code, that thread will get its own callstack for tracking who is responsible for a memory allocation.

Second, if you start a C thread, the calling Python code is considered responsible for any memory allocations in that thread. This works fine... except for thread pools. If you start a pool of threads that are not Python threads, the Python code that created those threads will be responsible for all allocations created during the thread pool's lifetime.

Therefore, in order to ensure correct memory tracking, Fil disables thread pools in BLAS (used by NumPy), BLOSC (used e.g. by Zarr), OpenMP, and numexpr. They are all set to use 1 thread, so calls should run in the calling Python thread and everything should be tracked correctly.

This has some costs:

  1. This can reduce performance in some cases, since you're doing computation with one CPU instead of many.
  2. Insofar as these libraries allocate memory proportional to number of threads, the measured memory usage might be wrong.

Fil does this for the whole program when using fil-profile run. When using the Jupyter kernel, anything run with the %%filprofile magic will have thread pools disabled, but other code should run normally.

What Fil tracks

Fil will track memory allocated by:

  • Normal Python code.
  • C code using malloc()/calloc()/realloc()/posix_memalign().
  • C++ code using new (including via aligned_alloc()).
  • Anonymous mmap()s.
  • Fortran 90 explicitly allocated memory (tested with gcc's gfortran).

Still not supported, but planned:

  • mremap() (resizing of mmap()).
  • File-backed mmap(). The semantics are somewhat different than normal allocations or anonymous mmap(), since the OS can swap it in or out from disk transparently, so supporting this will involve a different kind of resource usage and reporting.
  • Other forms of shared memory, need to investigate if any of them allow sufficient allocation.
  • Anonymous mmap()s created via /dev/zero (not common, since it's not cross-platform, e.g. macOS doesn't support this).
  • memfd_create(), a Linux-only mechanism for creating in-memory files.
  • Possibly memalign, valloc(), pvalloc(), reallocarray(). These are all rarely used, as far as I can tell.

License

Copyright 2020 Hyphenated Enterprises LLC

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

 http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

filprofiler-2021.3.0-cp39-cp39-manylinux2010_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.12+ x86-64

filprofiler-2021.3.0-cp39-cp39-macosx_10_14_x86_64.whl (373.5 kB view details)

Uploaded CPython 3.9 macOS 10.14+ x86-64

filprofiler-2021.3.0-cp38-cp38-manylinux2010_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.12+ x86-64

filprofiler-2021.3.0-cp38-cp38-macosx_10_14_x86_64.whl (373.5 kB view details)

Uploaded CPython 3.8 macOS 10.14+ x86-64

filprofiler-2021.3.0-cp37-cp37m-manylinux2010_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.12+ x86-64

filprofiler-2021.3.0-cp37-cp37m-macosx_10_14_x86_64.whl (373.5 kB view details)

Uploaded CPython 3.7m macOS 10.14+ x86-64

filprofiler-2021.3.0-cp36-cp36m-manylinux2010_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.6m manylinux: glibc 2.12+ x86-64

filprofiler-2021.3.0-cp36-cp36m-macosx_10_14_x86_64.whl (373.4 kB view details)

Uploaded CPython 3.6m macOS 10.14+ x86-64

File details

Details for the file filprofiler-2021.3.0-cp39-cp39-manylinux2010_x86_64.whl.

File metadata

  • Download URL: filprofiler-2021.3.0-cp39-cp39-manylinux2010_x86_64.whl
  • Upload date:
  • Size: 2.7 MB
  • Tags: CPython 3.9, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/3.7.3 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.59.0 CPython/3.8.8

File hashes

Hashes for filprofiler-2021.3.0-cp39-cp39-manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 910c5d1861e74310f20dce6e2813bb0346f3ea96c13c94031f3af7de4aa25ebd
MD5 ba13cabff642f57738f5b4a0ac2511d7
BLAKE2b-256 0e9f3751b62968c5e5ef45e6293fa414df4c77c953f72181d04fd06b28f497f0

See more details on using hashes here.

Provenance

File details

Details for the file filprofiler-2021.3.0-cp39-cp39-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: filprofiler-2021.3.0-cp39-cp39-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 373.5 kB
  • Tags: CPython 3.9, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/3.7.3 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.59.0 CPython/3.9.2

File hashes

Hashes for filprofiler-2021.3.0-cp39-cp39-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 8597daaa96ee514e42238403a82215828b1e7aa931529596622068ad5a4d860d
MD5 c51410cb338bc6c2ab147037bb5877f4
BLAKE2b-256 656fde2d9d8551687208314b2bbff9e678fb37a17112c0b39e32b3195c52a280

See more details on using hashes here.

Provenance

File details

Details for the file filprofiler-2021.3.0-cp38-cp38-manylinux2010_x86_64.whl.

File metadata

  • Download URL: filprofiler-2021.3.0-cp38-cp38-manylinux2010_x86_64.whl
  • Upload date:
  • Size: 2.7 MB
  • Tags: CPython 3.8, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/3.7.3 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.59.0 CPython/3.8.8

File hashes

Hashes for filprofiler-2021.3.0-cp38-cp38-manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 082b1a02643ab0ae1887112fa94f676457a80f0d52a1da1d7288b86a8c890a26
MD5 3c94bc11ba84adf906d01bb1745bb1dc
BLAKE2b-256 ba5023312f9ee12f3feec8626b1630aae5fc9d0705369cfffd0203eb1332947d

See more details on using hashes here.

Provenance

File details

Details for the file filprofiler-2021.3.0-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: filprofiler-2021.3.0-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 373.5 kB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/3.7.3 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.59.0 CPython/3.8.8

File hashes

Hashes for filprofiler-2021.3.0-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 a4b552fd7b7090c0a4df52a1454903871d406de21e3e3b1c1acee403ff100cd0
MD5 13fde8a53413f70eebb55f23bda57f12
BLAKE2b-256 0746492ba6df36db4adc54fd3564b3d78eafa1a060400cb0d6c0b3507f5f2412

See more details on using hashes here.

Provenance

File details

Details for the file filprofiler-2021.3.0-cp37-cp37m-manylinux2010_x86_64.whl.

File metadata

  • Download URL: filprofiler-2021.3.0-cp37-cp37m-manylinux2010_x86_64.whl
  • Upload date:
  • Size: 2.7 MB
  • Tags: CPython 3.7m, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/3.7.3 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.59.0 CPython/3.8.8

File hashes

Hashes for filprofiler-2021.3.0-cp37-cp37m-manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 8f680f52923b935fc8a620082937ec798682405f2880bb21baab7f02f64f9a14
MD5 ace7a0244a924a6000a3f359af4e7ea7
BLAKE2b-256 f3993fa2721504fada5b2bf9b72966621f4bc6b74ea27c899a5404c517ad437b

See more details on using hashes here.

Provenance

File details

Details for the file filprofiler-2021.3.0-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: filprofiler-2021.3.0-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 373.5 kB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/3.7.3 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.59.0 CPython/3.7.10

File hashes

Hashes for filprofiler-2021.3.0-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 5a5b88fd10f0a66c890f1db5594ef386824e93c9c6a550853fe9e9d6bf3a1aba
MD5 9601b57903b4e01991205ae3f8fb0dd3
BLAKE2b-256 4a7f2860210bdbddb8891d554964d25712a3115dbcede09601898782c4fcacb0

See more details on using hashes here.

Provenance

File details

Details for the file filprofiler-2021.3.0-cp36-cp36m-manylinux2010_x86_64.whl.

File metadata

  • Download URL: filprofiler-2021.3.0-cp36-cp36m-manylinux2010_x86_64.whl
  • Upload date:
  • Size: 2.7 MB
  • Tags: CPython 3.6m, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/3.7.3 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.59.0 CPython/3.8.8

File hashes

Hashes for filprofiler-2021.3.0-cp36-cp36m-manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 59206d3b255443a123f7a8f77b84ddd37c4a670800c250b410c0dd4f44844eeb
MD5 f8683693fe36d5a1d1c35d9b97f56df5
BLAKE2b-256 c28221472f56ac10d33ebfc1973f25b9d69e4932ccb0e24189c844ae7c58da0e

See more details on using hashes here.

Provenance

File details

Details for the file filprofiler-2021.3.0-cp36-cp36m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: filprofiler-2021.3.0-cp36-cp36m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 373.4 kB
  • Tags: CPython 3.6m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/3.7.3 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.59.0 CPython/3.6.13

File hashes

Hashes for filprofiler-2021.3.0-cp36-cp36m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 ee0b65f40ecf9968695ad17f2c172dff1c6d06bc333c4250b657558bc2a94289
MD5 1179984e47a8f2c782809058501b377f
BLAKE2b-256 0ff810757ba1447476e1ca94d1fd7d55dbce3f4c7549e3ee1a8a728cb4c8d360

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page