A memory profiler for data batch processing applications.
Project description
The Fil memory profiler for Python
Your code reads some data, processes it, and uses too much memory. In order to reduce memory usage, you need to learn what code is responsible, and specifically what code is responsible for peak memory usage.
And that's exactly what Fil will help you find. Fil an open source memory profiler designed for data processing applications written in Python, and includes native support for Jupyter.
At the moment it only runs on Linux and macOS.
For more information, including an example of the output, see https://pythonspeed.com/products/filmemoryprofiler/
Installation
Assuming you're on macOS or Linux, and are using Python 3.6 or later, you can use either Conda or pip (or any tool that is pip-compatible and can install manylinux2010 wheels).
Conda
To install on Conda:
$ conda install -c conda-forge filprofiler
Pip
To install the latest version of Fil you'll need Pip 19 or newer. You can check like this:
$ pip --version
pip 19.3.0
If you're using something older than v19, you can upgrade by doing:
$ pip install --upgrade pip
If that doesn't work, try running that in a virtualenv.
Assuming you have a new enough version of pip:
$ pip install filprofiler
Using Fil
Measuring peak (high-water mark) memory usage in Jupyter
To measure memory usage of some code in Jupyter you need to do three things:
- Use an alternative kernel, "Python 3 with Fil". You can choose this kernel when you create a new notebook, or you can switch an existing notebook in the Kernel menu; there should be a "Change Kernel" option in there in both Jupyter Notebook and JupyterLab.
- Load the extension by doing
%load_ext filprofiler. - Add the
%%filprofilemagic to the top of the cell with the code you wish to profile.
Measuring peak (high-water mark) memory usage for Python scripts
Instead of doing:
$ python yourscript.py --input-file=yourfile
Just do:
$ fil-profile run yourscript.py --input-file=yourfile
And it will generate a report.
Debugging out-of-memory crashes
First, run free to figure out how much memory is available—in this case about 6.3GB—and then set a corresponding limit on virtual memory with ulimit:
$ free -h
total used free shared buff/cache available
Mem: 7.7Gi 1.1Gi 6.3Gi 50Mi 334Mi 6.3Gi
Swap: 3.9Gi 3.0Gi 871Mi
$ ulimit -Sv 6300000
Then, run your program under Fil, and it will generate a SVG at the point in time when memory runs out:
$ fil-profile run oom.py
...
=fil-profile= Wrote memory usage flamegraph to fil-result/2020-06-15T12:37:13.033/out-of-memory.svg
Reducing memory usage in your code
You've found where memory usage is coming from—now what?
If you're using data processing or scientific computing libraries, I have written a relevant guide to reducing memory usage.
What Fil tracks
Fil will track memory allocated by:
- Normal Python code.
- C code using
malloc()/calloc()/realloc()/posix_memalign(). - C++ code using
new(including viaaligned_alloc()). - Anonymous
mmap()s. - Fortran 90 explicitly allocated memory (tested with gcc's
gfortran).
Still not supported, but planned:
mremap()(resizing ofmmap()).- File-backed
mmap(). The usage here is inconsistent since the OS can swap it in or out, so probably supporting this will involve a different kind of resource usage. - Other forms of shared memory, need to investigate if any of them allow sufficient allocation.
- Anonymous
mmap()s created via/dev/zero(not common, since it's not cross-platform, e.g. macOS doesn't support this). memfd_create().- Possibly
memalign,valloc(),pvalloc(),reallocarray().
License
Copyright 2020 Hyphenated Enterprises LLC
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for filprofiler-0.10.0-cp38-cp38-manylinux2010_x86_64.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 5f4e81c4599885e80197ec62f6cdb214e15517b642b82a172d921725bc5ba6f1 |
|
| MD5 | d0d186103b5da89f064664e2e158e997 |
|
| BLAKE2b-256 | 17b08988c810bb1af164a79d8d15cb84f9f533b17f28b28d9b0baa7267a80422 |
Hashes for filprofiler-0.10.0-cp38-cp38-macosx_10_14_x86_64.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 3171992dac1c04d8affc5960b5dfcd550722c4d2223cf8c6e8ea190c694c79e5 |
|
| MD5 | 9ebd5469de30544347844a63c7828093 |
|
| BLAKE2b-256 | 50059b6cb09c0ce4411e837cc5ac57fc66656b41614cf228d3dbcc289cd9cc25 |
Hashes for filprofiler-0.10.0-cp37-cp37m-manylinux2010_x86_64.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 451501f1bf6757f731f4274cf189e6e0ed3594a17f90b6c0d0cc070ab5ab7a69 |
|
| MD5 | fccca0883747680edf816eab12705001 |
|
| BLAKE2b-256 | dc1a7d832ebe80e1cf316f36e70d596b5f8142c8ac1444acd10cac5fa9653963 |
Hashes for filprofiler-0.10.0-cp37-cp37m-macosx_10_14_x86_64.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 00639f4422cb361e08fccaadc147651b88a06cd98a5dbff0cb7525e9562e5d03 |
|
| MD5 | 550934a8ed883396aab8b07ff9026832 |
|
| BLAKE2b-256 | 985902fd377b69462c8cb94e63b3a35bfc9b97b43a24da3605aafb31fab141bc |
Hashes for filprofiler-0.10.0-cp36-cp36m-manylinux2010_x86_64.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 08ab9e83515e2f1039c3f5b479e6cb359a9e894c6a01920c4f6f89d586b502a6 |
|
| MD5 | 3cc675a893b0d62d03a55035aa5a9796 |
|
| BLAKE2b-256 | f79e7fa3aa8fef5e965a5f7e3804b8c7786a022f4d0d1a6de6af8d0f4a49e94c |
Hashes for filprofiler-0.10.0-cp36-cp36m-macosx_10_14_x86_64.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 259572ce6181b8f4488160a540b982482145f26fd44224c4b2f2396622c4fda6 |
|
| MD5 | 5d075071305baab4752682ebb40cd2cb |
|
| BLAKE2b-256 | 3c9a4091ca013c23089a1c7b78491a5a2ba24631751cdc7072e6304821179de3 |
