uptime-kuma | A fancy self-hosted monitoring tool | Monitoring library

You can use VPC Flow logs to monitor the traffic to the VPC (which will include the traffic that is going to the EC2 instance).

As far as I know, psutil can only be used for gathering information about local processes, and is not suitable for retrieving information about processes running on other hosts. If you want to check whether or not a process is running on another host, there are many ways to approach this problem, and the solution depends on how deep you want to go (or need to go), and what your local situation is. From the top of my head, here are some ideas:

If you are only dealing with network services with exposed ports:

• A very simple solution would involve using a script and a port scanner (nmap); if a port that a service is listening behind, is open, then we can assume that the service is running. Run the script every once in a while to check up on the services, and do your thing.

• If you want to stay in Python, you can achieve the same end result by using Python's socket module to try and connect to a given host and port to determine whether or not the port that a service is listening behind, is open.

• A Python package or tool for monitoring network services on other hosts like this probably already exists.

If you want more information and need to go deeper, or you want to check up on local services, your solution will have to involve a local monitor process on each host, and connecting to that process to gather information.

• You can use your code to implement a server that lets clients connect to it, to check up on the services running on that host. (Check the socket module's official documentation for examples on how to implement clients and servers.)

Here's the big thing though. Based on your question and how it was asked, I would assume that you do not have the experience nor the insight to implement this in a secure way yet. If you're using this for a simple hobby/student project, roll out your own solution, and learn. Otherwise, I would recommend that you check out an existing solution like Nagios, and follow the security recommendations very closely.

After some digging I found there is an option you can enable via the init script called enable_query_name_tag which is disabled by default as it can cause there to be a ton of tags created when you are not using query names.

The modification is shown here:

Try the following query:

There is a second config file with experimental options related to HTTP server, and it has options to enable TLS:

The documentation on this file is located at the HTTPS AND AUTHENTICATION article. Note that after creating this file, you have to start Prometheus with the extra option:

The above is to be configured on the receiving part. The sending part needs a client TLS certificate configured in its remote_write:

The solution to this problem is to use an actual service discovery instead of static targets. This way Prometheus will scrape each replica during each iteration.

If it is just docker-compose (I mean, not Swarm), you can use DNS service discovery (dns_sd_config) to obtain all IPs belonging to a service:

This is the simplest way to get things up and running.

Moving next, you can use the dedicated Docker service discovery: docker_sd_config. Apart from the target list, it gives you more data in labels (e.g. container name, image version, etc) but it also requires a connection to the Docker daemon to get this data. In my opinion, this is an overkill for a development environment, but it might be essential in production. Here is an example configuration, boldly copy-pasted from https://github.com/prometheus/prometheus/blob/release-2.33/documentation/examples/prometheus-docker.yml :

At last, there is the dockerswarm_sd_config which is to be used, obviously, with Docker Swarm. This is the most complex thing of the trio and thus, there is a comprehensive official setup guide. Like the docker_sd_config it has additional information about containers in labels and even more than that (for example, it can tell on which node the container is). An example configuration is available here: https://github.com/prometheus/prometheus/blob/release-2.33/documentation/examples/prometheus-dockerswarm.yml , but you should really read the docs to be able to understand it and tune for yourself.

There is a short description along with each of the metrics. You can see them if you open node exporter in browser or just curl http://my-node-exporter:9100/metrics. You will see all the exported metrics and lines with # HELP are the description ones:

Grafana can show this help message in the editor: Prometheus (with recent experimental editor) can show it too: And this works for all metrics, not just node exporter's. If you need more technical details about those values, I recommend searching for the information in Google and man pages (if you're on Linux). Node exporter takes most of the metrics from /proc almost as-is and it is not difficult to find the details. Take for example node_memory_KReclaimable_bytes. 'Bytes' suffix is obviously the unit, node_memory is just a namespace prefix, and KReclaimable is the actual metric name. Using man -K KReclaimable will bring you to the proc(5) man page, where you can find that:

Finally, if this intention to learn more about the metrics is inspired by the desire to configure alerts for your hardware, you can skip to the last part and grab some alerts shared by the community from here: https://awesome-prometheus-alerts.grep.to/rules#host-and-hardware

You need the following query:

The and binary operation is used for matching right-hand time series to the left-hand series with the same set of labels. See these docs for details on matching rules.

Let's substitute rps_total and rps_failure with the actual time series given matching rules mentioned above.

• The rps_total is substituted with sum(nginx_ingress_controller_requests{prometheus_from="$cluster", ingress="brand-safety-phoenix-service"}) without (status) . The sum(...) without (status) is needed in order to sum metrics across all the status labels grouped by the remaining labels.

• The rps_failure is substituted with sum(nginx_ingress_controller_requests{prometheus_from="$cluster", ingress="brand-safety-phoenix-service", status!="200"}) without (status)

Then the final PromQL query will look like:

Recapping for better visibility:

OpsGenie supports multiple tools, including Google Stackdriver.
Instruction on how to integrate it with Stackdriver webhooks can be found here.

As far as i know there is no direct way to figure out a PVC is used by which pod To get that info possible workaround is grep through all the pods for the respective pvc :