node_exporter | Exporter for machine metrics | Dashboard library

There are two problems with your expr:

1. You use literal match operator (=) instead of regex match (=~).
2. You need to put the list inside parentheses, otherwise \\.service applies only to the last item.

Try this:

According your description of the syntax problem and the comment about YAML multiline strings, I've created a small test with Block Scalar Style: folded and Block Chomping: strip

Grafana Agent works with the same metric format Prometheus does. It is focused on scraping metrics instead of Prometheus and pushing them (remote_write) to the Prometheus instance that Grafana Cloud hosts for you. This is mentioned in the list of product features.

You can use Prometheus PHP library to create metrics and avoid troubles with the raw format. The best practices are also applicable.

Once you are done with creating metrics, you need to instruct the agent to scrape them from your server. Use these docs (one, two) for reference.

that was interesting

it was part of the OPS team infrastructure backups

setting the ambiance: we run on-prem on top of VMWare vCenter cluster backing up on the infra side with VMWare VM snapshot and Veeamm VM backup where the vmdk files\ESXi datastores reside on a NetApp storage based on NFS.

when checking node exporter metrics in Node Exporter Full Dashboard I saw network traffic drop in the order of up to 2 packets per second for about 5 to 15 minutes consistently through the last few months, increasing dramatically in phenomenon length in the last month (around the same time late at night).

Rough illustration:

After checking again with our OPS team they indicated it could be the host configurations\Veeam Backups.

It turns out that because the storage for the VMs (including the one that runs the Veeam backup) is attached via network and not directly on the ESXi hosts, the final snapshot saved\consolidated at that late-night time -

node detaches every night at a specific time

With the way NFS works with disk locking (limiting IOPs to existing data) along with the high IOPs requirements from the Veeam backup cause the server to hang\freeze and sometimes on rare occasions even a VM restart. here's the quote from the Veeam issue doc:

The snapshot removal process significantly lowers the total IOPS that can be delivered by the VM because of additional locks on the VMFS storage due to the increase in metadata updates

the snapshot removal process will easily push that into the 80%+ mark and likely much higher. Most storage arrays will see a significant latency penalty once IOP's get into the 80%+ mark which will of course be detrimental to application performance.

This issue occurs when the target virtual machine and the backup appliance [proxy] reside on two different hosts, and the NFSv3 protocol is used to mount NFS datastores. A limitation in the NFSv3 locking method causes a lock timeout, which pauses the virtual machine being backed up [during snapshot removal].

Obviously, that would interfere at the very least with Postgres functionality especially configured as a cluster with replication that requires a near-constant connection between the Postgres servers. A similar thread on SO using a different DB server

a solution is suggested including solving the issue presented in the last quote in this link, though for the time being, we removed the usage of Veeam backup for sensitive VMs until the solution can be verified locally (will update in the future if tried and fixed issue)

additional incidents documentation: similar issue case, suggested solution info from Veeam, third party site solution (around the same problem as a temp fix as I see it), Reddit thread acknowledging the issue and suggesting options

I would suggest using set_fact module to save the contents of the URL in a variable, then we can extract the tag_name out of it.

I have a Python script [...] that needs to continuously read changing values inside a file located on the host file system.

Just run it. Most Linux systems have Python preinstalled. You don't need Docker here. You can use tools like Python virtual environments if your application has Python library dependencies that need to be installed.

Is it possible to get a Docker container to read a file from host file system not using Volumes?

You need some kind of mount, perhaps a bind mount; docker run -v /host/path:/container/path image-name. Make sure to not overwrite the application's code in the image when you do this, since the mount will completely hide anything in the underlying image.

Without a bind mount, you can't access the host filesystem at all. This filesystem isolation is a key feature of Docker.

...the [Prometheus] node_exporter container...

Reading from the GitHub link in the question, "It's not recommended to deploy it as a Docker container because it requires access to the host system." The docker run example there uses a bind mount to access the entire host filesystem, circumventing Docker's filesystem isolation.

You have it all; instead of looping over the array and calling a script - execute that script content.

It may be that you need only find a system that allows you to graph data. Prometheus has a sibling technology called Grafana.

Prometheus is a tool that is configured to periodically scrape (read) measurements from endpoints (==targets) of systems. These measurements describe the state of the system at the time it is scraped. Prometheus records these measurements in time-series databases(s) and provides an expressive query language that enables insights to be found in the data. Prometheus includes basic charting of this time-series data too. Frequently Prometheus is combined with Grafana.

Your question suggests that you're not looking to monitor the state of the database but to rather graph the (statistics) data contained in database tables. If so, Prometheus may not be what you need.

To answer your questions, Prometheus' requires systems to publish sets of metrics in a simple text format called Exposition Format. Any system that generates data in this format and can serve the metrics on an HTTP endpoint can be scraped by Prometheus.

Node Exporter is one example of such a system that publishes metrics that can be scraped by Prometheus. It publishes node (==host) metrics.

ArangoDB is another system that can publish metrics (about its state; the state of the database system) in Prometheus' Exposition Format, see:

https://www.arangodb.com/2020/03/monitoring-arangodb-prometheus-grafana/

I suspect, what you need is a solution to graph a set of data that you already have stored in ArangoDB tables. For this, you may be able to connect a system like Grafana to your database and then use SQL queries to surface the data in a form that can be graphed.

You can create a relatively simple table with label values like this one:

1. Create a new panel, select source, change format from Time series to Table, select Table visualisation, and enable instant mode:

2. Open Transform tab, add two transforms in this order: 1) Labels to fields, 2) Organize fields. See the picture:

3. Disable unnecessary fields and rename ones you'd like to see.