stress-ng | ng upstream project git repository | Testing library

It worked after I completely removed mine and applied the permissions as shown here: https://chaos-engineering.workshop.aws/en/030_basic_content/030_basic_experiment/10-permissions.html

Posting this for the benefit of others who might run into similar issues.

stress-ng --vm-bytes 10% will use sysconf(_SC_AVPHYS_PAGES) to determine the available memory. This sysconf() system call will return the number of pages that the application can use without hindering any other process. So this is approximately what the free command is returning for the free memory statistic.

Note that stress-ng will allocate the memory with mmap, so it may be that during run time mmap'd pages may not necessarily be physically backed at the time you check how much real memory is being used.

It may be worth trying to also use the --vm-populate option; this will try and ensure the pages are physically populated on the mmap'd memory that stress-ng is exercising. Also try --vm-madvise willneed to use the madvise() system call to hint that the pages will be required fairly soon.

It's important to remember that modern CPUs, especially those made by Intel, have variable clock frequencies. The CPU will run slowly when lightly loaded to conserve power, which extends battery life, but can ramp up under load.

The limiting factor is thermals, that is the CPU will only be allowed to get so hot before the frequency is trimmed to reduce power consumption, and by extension, heat generation.

On a chip with more than one core, a single core can be run very quickly without hitting thermal throttling. Two cores must run slower, they're producing effectively twice the heat, and when using all four cores each has to share a smaller slice of the overall thermal budget.

It's worth checking your CPU temperature as the tests are running as it will likely be hitting some kind of cap.

I solved this problem by also capturing the event syscalls:sys_enter_execve. Between the two of them I was able to get every instance of exec called.

MemoryPressure is a node metrics. You could run the stress command on the node directly to create an artificial memory pressure on the node. Kubelet will notice this(with a caveat) and should report that the node is under MemoryPressure.

The caveat as per the doc is

The kubelet currently polls cAdvisor to collect memory usage stats at a regular interval. If memory usage increases within that window rapidly, the kubelet may not observe MemoryPressure fast enough, and the OOMKiller will still be invoked. We intend to integrate with the memcg notification API in a future release to reduce this latency, and instead have the kernel tell us when a threshold has been crossed immediately

Carrying on from comments by @osgx,

As is mentioned here, by default, the perf stat command will monitor not only all the threads of the process to be monitored, but also its child processes and threads.

The problem in this situation is that by running perf stat and monitoring the docker run stress-ng command, you are not monitoring the actual stress-ng process. It is important to note that, the processes running as part of the container, will actually not be started by the docker client, but rather by the docker-containerd-shim process (which is a grandchild process of the dockerd process).

If you run the docker command to run stress-ng inside the container and observe the process-tree, it becomes evident.