cadvisor | Analyzes resource usage and performance characteristics of running containers | Continuous Deployment library

I'm having issues with Prometheus alerting rules. I have various cAdvisor specific alerts set up, for example:

I have deployed istio on kubernetes, and I installed prometheus from istio addons. My goal is to only monitor some pods of one application(such as all pods of bookinfo application). The job definition for monitoring pods is as below:

I'm trying to find out and understand how OOM-killer works on the container.

To figuring it out, I've read lots of articles and I found out that OOM-killer kills container based on the oom_score. And oom_score is determined by oom_score_adj and memory usage of that process.

And there're two metrics container_memory_working_set_bytes, container_memory_rss from the cAdvisor for monitoring memory usage of the container.

It seems that RSS memory (container_memory_rss) has impact on oom_score so I can understand that with the container_memory_rss metric, if that metric reached to memory limit, the OOM-killer will kill the process.

• https://github.com/torvalds/linux/blob/v3.10/fs/proc/base.c#L439
• https://github.com/torvalds/linux/blob/v3.10/mm/oom_kill.c#L141
• https://github.com/torvalds/linux/blob/v3.10/include/linux/mm.h#L1136

But from the articles like below:

• https://faun.pub/how-much-is-too-much-the-linux-oomkiller-and-used-memory-d32186f29c9d
• https://blog.freshtracks.io/a-deep-dive-into-kubernetes-metrics-part-3-container-resource-metrics-361c5ee46e66

The better metric is container_memory_working_set_bytes as this is what the OOM killer is watching for.

I cannot understand the fact that OOM-killer is watching for container's working set memory. I think I'm not understand the meaning of the working set memory on the container which is 'total usage - inactive file'.

• https://github.com/google/cadvisor/issues/2582#issuecomment-644883028

Where can I find the reference? Or could you explain the relationship between working set memory and OOM-kill on the container?

I'm trying to create a json file that has the following structure:

I want to monitor docker containers running on multiple servers lets say i have a,b servers and containers running inside them, now I add one server (d) I want to monitor all docker containers inside all servers (A,B) only from server c. I have configured the docker to expose logs on all servers followed this docker docs not using cAdvisor . The target status shows 'ok' on all the servers, but the problem is as expression is same for all the containers of docker Prometheus is not able differentiate between the servers can anyone share the sample Prometheus rule file with expression i.e number of stopped containers should not be less then x .This is my current rule file

I would like to have Prometheus and Grafana running on my developer machine using docker-images / docker-for-windows.

I have system-under-development, ASP.Net core, running on localhost:5001 and metrics are showing just fine on https://localhost:5001/metrics.

Docker-compose.yml and prometheus.yml listed below.

• If I include network_mode: host in docker-compose.yml, I can't access Prometheus on my physical machine via localhost:9090
• If I exclude network_mode and instead use ports: , I can access Prometheus on my physical machine via localhost:9090, but checking http://localhost:9090/targets, it shows https://localhost:5001/metrics as being down.

What am I doing wrong? Any comments welcome!

When staring the cAdvisor, I am getting Factory "docker" was unable to handle container "/system.slice/kdump.service". I am trying to understand what these are for, and ... How can resolve?

Any pointers will be appreciated.

I have two deployments running a v1 and v2 of the same service in istio. I have set up a custom-metric 'istio-total-requests' to be collected through prometheus adapater.

I have set up an HPA to scale the v2 deployment and can see the target value increasing when I send requests, but what is not happening is the HPA is not scaling the number of pods/replicas.

I am running kubernetes v1.19 on minikube v1.13.1, and cannot understand why its not scaling.

I have cloudflared my site which is hosted on GoDaddy.

I also set up an email for this site:

info@domain.com

It sends email fine.

No incoming email is received.

On Cloudflare it says: Add an MX record for your root domain so that mail can reach @trackpython.com addresses.

Update: I have added the correct MX record on CLOUDFLARE - but still no luck.

What else do I need to do to resolve the problem of no incoming email.

The GoDaddy CAdvisor says the "email routing" is wrong...but offers no further help.

Thanks in advance.