gc.h | Header-only Non-moving & Precise GC for C | Game Engine library

There is no effect of using -XX:+UseG1GC on IBM JVM. It will just be silently swallowed. The JVM will default to Gencon GC policy.

You can verify that by running -verbose:gc, what will reported GC policy being used.

The closest IBM's GC policy to Hotspot's G1GC is Balanced one, the main distinguishing characteristics being they are region based (unlike Gencon that has two distinct ares of heap for old and new objects).

As far as concurrency, all 3 (G1GC, Balanced, Gencon) are similar: global GCs are mostly concurrent and partial/local GCs are STW (Stop-The-World).

Reasons to use region based GC policy are to reduce worst case pause time. They are capable of doing some global type operations incrementally in partial GCs. Most notably, they can incrementally de-fragment heap, unlike Gencon, that it does in global GC via optional STW compact operation. Most of applications will not require such global compact, hence Gencon is default. But, if long pauses due to global compaction are observed in Gencon run, Balanced should be tried. Balanced GC will however slightly compromise the application throughput.

Due to the way that the CPython interpreter manages memory, it very rarely actually frees any allocated memory. Generally CPython processes will keep growing and growing in memory usage

Since you are using Gunicorn you can set the max_requests setting which will regularly restart your workers and alleviate some "memory leak" issues

When I installed libgc on mac, as you did, the files were installed to /usr/local/Cellar/bdw-gc/. Then, when it came time to compile my code I had to run:

It would be better to not recreate the list each time. Look at this version : only the needed cells are created dynamically while existing one are kept :

So I have read a bit into the CPython implementation (https://github.com/python/cpython/blob/master/Modules/gcmodule.c) and this is what I've learned:

Basically get_count (impl here: https://github.com/python/cpython/blob/master/Modules/gcmodule.c#L1636-L1645) measures the amount of collections happened in a one level lower generation until that generation itself gets collected (See here: https://github.com/python/cpython/blob/master/Modules/gcmodule.c#L1211-L1212).

So for example, when gen 0 (first gen) is collected, the count for gen 1 increases by 1. The count for gen 0 increases upon allocation, and decreases on deallocation (the collection starts when the #allocations - #deallocations > threashold).

This answers question (1) - the discrepancy is because they are totally different things.

Now that question 1 is answered, question 2 is actually not relevant when asked as is.

However, we might ask a different question which is "how do I track which objects are collected for a specific generation?".

With Python 3.8 this is possible since the interface of get_objects has changed and it's possible to get the objects that "belong" to a specific generation. With that in mind, one can register a callback (via the gc.callbacks.append(callback_method)) that will track the collection of that specific gen by getting the objects before they are cleaned (but note that you don't want to actually strong reference these objects or else you will change behaviour just by measuring), getting them afterwards, and comparing the results.

I will leave this answer unaccepted for some time to give chance for other answers, since I'm answering my own question.

Try this:

Because your computer has a double resolution (Retina) screen.

Memory Management The Memory Management report shows the calls where memory management blocks occurred, along with the total blocking times of each call stack. Use this information to identify areas that have excessive paging or garbage collection issues.

Further more

Memory management time

These segments in the timeline are associated with blocking times that are categorized as Memory Management. This scenario implies that a thread is blocked by an event that is associated with a memory management operation such as Paging. During this time, a thread has been blocked in an API or kernel state that the Concurrency Visualizer is counting as memory management. These include events such as paging and memory allocation. Examine the associated call stacks and profile reports to better understand the underlying reasons for blocks that are categorized as Memory Management.

Yes, allocating less will likely have a large benefit on your resources and efficiency, but that is almost always the case on hot paths and thrashed applications

Heap allocations and particular Large Object Heap (LOB) allocations are costly, it also creates extra work for your The Garbage Collector and can fragment your memory causing even more inefficiency. The less you allocate, or reuse memory, or use the stack the better you are (in general).

This is also where you would learn to use a good memory profiler and get to know your garbage collector.

On saying that this would not be the only tool you would use to make your application less allocatey. A good memory profiler will go a long way, combined with learning how to read the results and affect changes based on the results.

Creating minimal allocation code is an artform, and one worth your learning

Also as @mjwills pointed out in the comments, you would run any change through your benchmark software as well, removing allocations at the cost of CPU time wont make sense. There are a lot of ways to speed up code, and low allocation is just one of a lot of approaches that may help.

Lastly, I would suggest following Marc Gravell and his blogs as a start (Mr DeAllocation), get to know your Garbage Collector and how the generations wortk, and tools like memory profilers and benchmarkers for performant silky smooth production code

This is a hard-coded version that can execute MessageBoxA