Skip to main content

A memory profiler for data batch processing applications.

Project description

The Fil memory profiler for Python

Fil a memory profiler designed for data processing applications. At the moment it only runs on Linux and macOS.

Your code reads some data, processes it, and—uses too much memory. What you need to reduce is peak memory usage.

And that's exactly what Fil will help you find: exactly which code was responsible for allocating memory at peak memory usage.

For more information see https://pythonspeed.com/products/filmemoryprofiler/

What Fil tracks

Fil will track memory allocated by:

  • Normal Python code.
  • C code using malloc()/calloc()/realloc()/posix_memalign().
  • C++ code using new (including via aligned_alloc()).
  • Anonymous mmap()s.
  • Fortran 90 explicitly allocated memory (tested with gcc's gfortran).

Still not supported, but planned:

  • mremap() (resizing of mmap()).
  • File-backed mmap(). The usage here is inconsistent since the OS can swap it in or out, so probably supporting this will involve a different kind of resource usage.
  • Other forms of shared memory, need to investigate if any of them allow sufficient allocation.
  • Anonymous mmap()s created via /dev/zero (not common, since it's not cross-platform, e.g. macOS doesn't support this).
  • memfd_create().
  • Possibly memalign, valloc(), pvalloc(), reallocarray().

Installation

Prerequisites

To install the latest version of Fil you'll need Pip 19 or newer. You can check like this:

$ pip --version
pip 20.0.2

If you're using something older than that, do:

$ pip install --upgrade pip

If that doesn't work, try running that a virtualenv or Conda env.

Installing Fil

Assuming you're on macOS or Linux, and are using Python 3.6 or later:

$ pip install filprofiler

Measuring peak (high-water mark) memory usage

Instead of doing:

$ python yourscript.py --input-file=yourfile

Just do:

$ fil-profile run yourscript.py --input-file=yourfile

And it will generate a report.

Debugging out-of-memory crashes

First, run free to figure out how much memory is available—in this case about 6.3GB—and then set a corresponding limit on virtual memory with ulimit:

$ free -h
       total   used   free  shared  buff/cache  available
Mem:   7.7Gi  1.1Gi  6.3Gi    50Mi       334Mi      6.3Gi
Swap:  3.9Gi  3.0Gi  871Mi
$ ulimit -Sv 6300000

Then, run your program under Fil, and it will generate a SVG at the point in time when memory runs out:

$ fil-profile run oom.py 
...
=fil-profile= Wrote memory usage flamegraph to fil-result/2020-06-15T12:37:13.033/out-of-memory.svg

You've found where memory usage is coming from—now what?

If you're using data processing or scientific computing libraries, I have written a relevant guide to reducing memory usage.

License

Copyright 2020 Hyphenated Enterprises LLC

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

 http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

filprofiler-0.9.0-cp38-cp38-manylinux2010_x86_64.whl (1.8 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.12+ x86-64

filprofiler-0.9.0-cp38-cp38-macosx_10_14_x86_64.whl (263.9 kB view details)

Uploaded CPython 3.8 macOS 10.14+ x86-64

filprofiler-0.9.0-cp37-cp37m-manylinux2010_x86_64.whl (1.8 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.12+ x86-64

filprofiler-0.9.0-cp37-cp37m-macosx_10_14_x86_64.whl (263.9 kB view details)

Uploaded CPython 3.7m macOS 10.14+ x86-64

filprofiler-0.9.0-cp36-cp36m-manylinux2010_x86_64.whl (1.8 MB view details)

Uploaded CPython 3.6m manylinux: glibc 2.12+ x86-64

filprofiler-0.9.0-cp36-cp36m-macosx_10_14_x86_64.whl (263.9 kB view details)

Uploaded CPython 3.6m macOS 10.14+ x86-64

File details

Details for the file filprofiler-0.9.0-cp38-cp38-manylinux2010_x86_64.whl.

File metadata

  • Download URL: filprofiler-0.9.0-cp38-cp38-manylinux2010_x86_64.whl
  • Upload date:
  • Size: 1.8 MB
  • Tags: CPython 3.8, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.48.2 CPython/3.8.5

File hashes

Hashes for filprofiler-0.9.0-cp38-cp38-manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 4166db4be46294b6d0ab7a1e685b28cb27a195e78c6ee290fb993f94a60b66bb
MD5 fa522ad062949dec4fc0c65554aca962
BLAKE2b-256 9f05950c64cf20e717787a05d823229ca6be6327cc39d966d8f6e53a329246d2

See more details on using hashes here.

Provenance

File details

Details for the file filprofiler-0.9.0-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: filprofiler-0.9.0-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 263.9 kB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.48.2 CPython/3.8.5

File hashes

Hashes for filprofiler-0.9.0-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 fe8f009e820016501bcc87e354a4c0ee8955febfa693583554458fcb86c7c7f8
MD5 a820744d6a019c5a0775df5f218b5441
BLAKE2b-256 0818c192c608d18eaf2b5e88c7841f91bdd15d98e4631c2fa560d51a1272d53e

See more details on using hashes here.

Provenance

File details

Details for the file filprofiler-0.9.0-cp37-cp37m-manylinux2010_x86_64.whl.

File metadata

  • Download URL: filprofiler-0.9.0-cp37-cp37m-manylinux2010_x86_64.whl
  • Upload date:
  • Size: 1.8 MB
  • Tags: CPython 3.7m, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.48.2 CPython/3.8.5

File hashes

Hashes for filprofiler-0.9.0-cp37-cp37m-manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 06bea75042c8a9342c14a437dee57e59dc06c53fdc9a52ea461a5451ee968a14
MD5 6b5b2a53adfcf25d7fa0814ffa1c08dc
BLAKE2b-256 615658b21b09ec4d8354d38fa2f12981d29e26922d93928a1501541b33f1cc70

See more details on using hashes here.

Provenance

File details

Details for the file filprofiler-0.9.0-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: filprofiler-0.9.0-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 263.9 kB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.48.2 CPython/3.7.8

File hashes

Hashes for filprofiler-0.9.0-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 5f31f9f658380a28e01df0e8e6746a6fa877e52ba35c66ad9b2513cec142a9a3
MD5 661b3a872f677999e6a05238ce37541e
BLAKE2b-256 9aca4fa1d1bdca8a2765560076716d3f38d6a3a49edc226a7d92b6a6f5740c1a

See more details on using hashes here.

Provenance

File details

Details for the file filprofiler-0.9.0-cp36-cp36m-manylinux2010_x86_64.whl.

File metadata

  • Download URL: filprofiler-0.9.0-cp36-cp36m-manylinux2010_x86_64.whl
  • Upload date:
  • Size: 1.8 MB
  • Tags: CPython 3.6m, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.48.2 CPython/3.8.5

File hashes

Hashes for filprofiler-0.9.0-cp36-cp36m-manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 e56745bb921a7f00f791213c9cbce239de4f73b177234bc08b02589bf1fdd826
MD5 6b246d473c963c5a5eac71cb35bfff4c
BLAKE2b-256 9d9eb03cc67f4b6e7ec7272a2172e9b003e387a799851085670cc247dc585e6d

See more details on using hashes here.

Provenance

File details

Details for the file filprofiler-0.9.0-cp36-cp36m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: filprofiler-0.9.0-cp36-cp36m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 263.9 kB
  • Tags: CPython 3.6m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/40.6.2 requests-toolbelt/0.9.1 tqdm/4.48.2 CPython/3.6.11

File hashes

Hashes for filprofiler-0.9.0-cp36-cp36m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 a4b1f53adf2556a98d14d605d54880fddd961e2738ad178c27e9aad31c61615d
MD5 433e9b934f00fec0c5ba244e66786114
BLAKE2b-256 c8c824bc49069e4953ecf52e12c2b724f8b90bd0f67b768bfd76c9259b29d538

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page