memory_profiler | Monitor Memory usage of Python code

I would like to use some code line speed indicator related libraries such as scalene and tamppa to evaluate which code lines consume more times. We will need to run some command lines in the terminal, before and after the code execution, for seeing the results. For example using tamppa library, if we have the following code (test.py) and execute it in PyCharm:

I'm trying to load a dask dataframe from a 30gb csv file into a 3D barchart using matplotlib.

The problem is the task has been running for days with no end in sight as soon as it gets to the 'color settings' portion of the code.

I have tried to make it use only a limited number of rows from the dataframe but dask doesn't seem to allow for row indexing, only column indexing.

So I split the partition and used the partition size to limit the row size. However even with only 100 rows it takes days.

I have my doubts that it would take days for 100 rows of color settings to be calculated (not even to the plotting portion yet)

So clearly I am doing something wrong.

Here is what the dataframe looks like

Here is the code

Which python module can I use to calculate the amount of memory spent executing a function? I'm using memory_profiler, but it shows the amount of memory spent by each line of the algorithm, in this case I want one that shows the total amount spent.

I have the following code from the example folder with the exception that I added @profile. I am just trying to make this example run because in my code which is more complex I have the same error and I would like to know how much memory is used on each line.

SYSTEM:

Python: 3.9

memory-profiler: 0.58

OS: Manjaro

CODE:

I am working on Ptera Software, an open-source aerodynamics solver. This is the first package I have distributed, and I'm having some issues related to memory management.

Specifically, importing my package takes up an absurd amount of memory. The last time I checked, it took around 136 MB of RAM. PyPI lists the package size as 118 MB, which also seems crazy high. For reference, NumPy is only 87 MB.

At first, I thought that maybe I had accidentally included some huge file in the package. So I downloaded every version's tar.gz files from PyPI and extracted them. None was over 1 MB unzipped.

This leads me to believe that there's something wrong with how I am importing my requirements. My REQUIREMENTS.txt file looks like this:

I want to filter a Dask dataframe based on a value of a column, i.e.

Recently I started having some problems with Django (3.1) tests, which I finally tracked down to some kind of memory leak. I normally run my suite (roughly 4000 tests at the moment) with --parallel=4 which results in a high memory watermark of roughly 3GB (starting from 500MB or so). For auditing purposes, though, I occasionally run it with --parallel=1 - when I do this, the memory usage keeps increasing, ending up over the VM's allocated 6GB.

I spent some time looking at the data and it became clear that the culprit is, somehow, Webtest - more specifically, its response.html and response.forms: each call during the test case might allocate a few MBs (two or three, generally) which don't get released at the end of the test method and, more importantly, not even at the end of the TestCase.

I've tried everything I could think of - gc.collect() with gc.DEBUG_LEAK shows me a whole lot of collectable items, but it frees no memory at all; using delattr() on various TestCase and TestResponse attributes and so on resulted in no change at all, etc.

I'm quite literally at my wits' end, so any pointer to solve this (beside editing the thousand or so tests which use WebTest responses, which is really not feasible) would be very much appreciated.

(please note that I also tried using guppy and tracemalloc and memory_profiler but neither gave me any kind of actionable information.)

I found that one of our EC2 testing instances isn't affected by the problem, so I spent some more time trying to figure this out. Initially, I tried to find the "sensible" potential causes - for instance, the cached template loader, which was enabled on my local VM and disabled on the EC2 instance - without success. Then I went all in: I replicated the EC2 virtualenv (with pip freeze) and the settings (copying the dotenv), and checked out the same commit where the tests were running normally on the EC2.

Et voilà! THE MEMORY LEAK IS STILL THERE!

Now, I'm officially giving up and will use --parallel=2 for future tests until some absolute guru can point me in the right directions.

And now the memory leak is there even with --parallel=2. I guess that's somehow better, since it looks increasingly like it's a system problem rather than an application problem. Doesn't solve it but at least I know it's not my fault.

Thanks to Tim Boddy's reply to this question I tried using chap to figure out what's making memory grow. Unfortunately I can't "read" the results properly but it looks like some non-python library is actually causing the problem. So, this is what I've seen analyzing the core after a few minutes running the tests that I know cause the leak:

I have a very simple query, in a very simple table:

I want to benchmark QuickSort, and BubbleSort. My task it's measure time, CPU usage, and memory usage for both. I wrote code, and in BubbleSort everything works fine. But I have a problem with QuickSort. Example script result: