procfs | Rust library for reading the Linux procfs filesystem | File Utils library

This issue was solved recently. Here is more information: https://github.com/prometheus-community/helm-charts/issues/467 and here: https://github.com/prometheus-community/helm-charts/pull/757

Here is the solution (https://github.com/prometheus-community/helm-charts/issues/467#issuecomment-802642666):

[you need to] opt-out the rootfs host mount (preventing the crash). In order to do that you need to specify the following value in values.yaml file:

Taking this as a question:

So, in any chance those four columns will show different UIDs.

Yes. Subject to various limitations, processes can change their effective and saved UIDs and GIDs. This is what the setuid(), setgid(), seteuid(), and setegid() functions do.

The filesystem uid and gid are Linux-specific features that are used mainly, if not entirely, in the context of NFS (see filesystem uid and gid in linux). These can be manipulated with setfsuid() and setfsgid(), subject, again, to limitations.

For most processes, all the UIDs will the same and all the GIDs will be the same, but it is conceivable that they would all be different. It is a function of the behavior of the process.

Quite some more digging turned up this answer to "About mounting and unmounting inherited mounts inside a newly-created mount namespace" which points in the correct direction, but needs additional explanations (not least due to basing on a misleading paragraph about mount namespaces being hierarchical from man pages which Michael Kerrisk fixed some time ago).

Our starting point is when runc sets up the (test) container, for masking system paths especially in the container's future /proc tree, it creates a set of new mounts to either mask out individual files using /dev/null or subdirectories using tmpfs. This results in procfs being mounted on /proc, as well as further sub-mounts.

Now the test container starts and at some point a process unshares into a new user namespace. Please keep in mind that this new user namespace (again) belongs to the (real) root user with UID 0, as a default Docker installation won't enable running containers in new user namespaces.

Next, the test process also unshares into a new mount namespace, so this new mount namespace belongs to the newly created user namespace, but not to the initial user namespace. According to section "restrictions on mount namespaces" in mount_namespaces(7):

If the new namespace and the namespace from which the mount point list was copied are owned by different user namespaces, then the new mount namespace is considered less privileged.

Please note that the criterion here is: the "donor" mount namespace and the new mount namespace have different user namespaces; it doesn't matter whether they have the same owner user (UID), or not.

The important clue now is:

Mounts that come as a single unit from a more privileged mount namespace are locked together and may not be separated in a less privileged mount namespace. (The unshare(2) CLONE_NEWNS operation brings across all of the mounts from the original mount namespace as a single unit, and recursive mounts that propagate between mount namespaces propagate as a single unit.)

As it now is not possible anymore to separate the /proc mountpoint as well as the masking submounts, it's not possible to (re)mount /proc (question 1). In the same sense, it is impossible to unmount /proc/kcore, because that would allow unmasking (question 2).

Now, when deploying the test container using --security-opt systempaths=unconfined this results in a single /proc mount only, without any of the masking submounts. In consequence and according to the man page rules cited above, there is only a single mount which we are allowed to (re)mount, subject to the CAP_SYS_ADMIN capability including also mounting (besides tons of other interesting functionality).

Please note that it is possible to unmount masked /proc/ paths inside the container while still in the original (=initial) user namespace and when possessing (not surprisingly) CAP_SYS_ADMIN. The (b)lock only kicks in with a separate user namespace, hence some projects striving for deploying containers in their own new user namespaces (which unfortunately has effects not least on container networking).

As posted in the comments:

Please add to the question the steps that you followed (editing any values in the Helm chart etc). Also please check if the nodes are not over the limit of pods that can be scheduled on it. Here you can find the link for more reference: LINK.

no processes occupying 9100 on the given node. @DawidKruk The POD limit was reached. Thanks! I expected them to give me some error regarding that rather than vague node selector property not matching

Not really sure why the following messages were displayed:

• node(s) didn't have free ports for the requested pod ports
• node(s) didn't match node selector

The issue that Pods couldn't be scheduled on the nodes (Pending state) was connected with the Insufficient pods message in the $ kubectl get events command.

Above message is displayed when the nodes reached their maximum capacity of pods (example: node1 can schedule maximum of 30 pods).

More on the Insufficient Pods can be found in this github issue comment:

That's true. That's because the CNI implementation on EKS. Max pods number is limited by the network interfaces attached to instance multiplied by the number of ips per ENI - which varies depending on the size of instance. It's apparent for small instances, this number can be quite a low number.

Docs.aws.amazon.com: AWSEC2: User Guide: Using ENI: Available IP per ENI

-- Github.com: Kubernetes: Autoscaler: Issue 1576: Comment 454100551

Additional resources:

• Stackoverflow.com: Questions: Pod limit on node AWS EKS

Since you didn't mention it above: did you instrument a Go application? The Elastic APM Go "Agent" is a package which you use to instrument your application source code. It is not an independent process, but runs within your application.

So, first (if you haven't already) instrument your application. See https://www.elastic.co/guide/en/apm/agent/go/current/getting-started.html#instrumenting-source

Here's an example web server using Echo, and the apmechov4 instrumentation module:

There are several ways to achieve what you want (taking into account different aspects, like module can be compiled in or be a module).

1. Convert Out-Of-Tree module to be In-Tree one (in your custom kernel build). This will require simple export and import as you basically done and nothing special is required, just maybe providing a header with the symbol and depmod -a run after module installation. Note, you have to use modprobe in-tree which reads and satisfies dependencies.
2. Turn other way around, i.e. export symbol from in-tree module and file it in the out-of-tree. In this case you simply have to check if it has been filed or not (since it's a MAC address the check against all 0's will work, no additional flags needed)

BUT, these ways are simply wrong. The driver and even your patch clearly show that it supports OF (Device Tree) and your board has support of it. So, this is a first part of the solution, you may provide correct MAC to the network card using Device Tree.

In the case you want to change it runtime the procfs approach is very strange to begin with. Network device interface in Linux has all means to update MAC from user space at any time user wants to do it. Just use ip command, like /sbin/ip link set <$ETH> addr <$MACADDR>, where <$ETH> is a network interface, for example, eth0 and <$MACADDR> is a desired address to set.

So, if this question rather about module symbols, you need to find better example for it because it's really depends to use case. You may consider to read How to export symbol from Linux kernel module in this case? as an alternative way to exporting. Another possibility how to do it right is to use software nodes (it's a new concept in recent Linux kernel).

You can try using the env variable like AUTO_SERVER_HOST and call this in your code

Typical solution is to create a new file in the same directory, then rename (hardlink) it over the old one.

This way, processes see either an old one or a new one, never a mix; and it only depends on the moment when they open the file.

The Linux kernel takes care of the caching, so if the file is accessed often, it will be in RAM (page cache). The writer must, however, remember to delete the file when it exits.

A better approach is to use fcntl()-based advisory record locks (typically over the entire file, i.e. .l_whence = SEEK_SET, .l_start = 0, .l_len = 0).

The writer will grab a write/exclusive lock before truncating and rewriting the contents, and readers a read/shared lock before reading the contents.

This requires cooperation, however, and the writer must be prepared to not be able to lock (or grabbing the lock may take undefined amount of time).

A Linux-only scheme would be to use atomic replacement (via rename/hardlinking), and file leases.

(When the writer process has an exclusive lease on an open file, it gets a signal whenever another process wants to open that same file (inode, not file name). It has at least a few seconds to downgrade or release the lease, at which point the opener gets access to the contents.)

Basically, the writer process creates an empty status file, and obtains exclusive lease on it. Whenever the writer receives a signal that a reader wants to access the status file, it writes the current status to the file, releases the lease, creates a new empty file in the same directory (same mount suffices) as the status file, obtains an exclusive lease on that one, and renames/hardlinks it over the status file.

If the status file contents do not change all the time, only periodically, then the writer process creates an empty status file, and obtains exclusive lease on it. Whenever the writer receives a signal that a reader wants to access the (empty) status file, it writes the current status to the file, and releases the lease. Then, when the writer process' status is updated, and there is no lease yet, it creates a new empty file in the status file directory, takes an exclusive lease on it, and renames/hardlinks over the status file.

This way, the status file is always updated just before a reader opens it, and only then. If there are multiple readers at the same time, they can open the status file without interruption when the writer releases the lease.

It is important to note that the status information should be collected in a single structure or similar, so that writing it out to the status file is efficient. Leases are automatically broken if not released soon enough (but there are a few seconds at least to react), and the lease is on the inode – file contents – not the file name, so we still need the atomic replacement.

Here's a crude example implementation:

3 node(s) didn't have free ports for the requested pod ports.

From the error it shows that allocated node port is already in use. As you define hostPort: 9100, it limits the number of places the pod can be scheduled, because each combination must be unique. Ref: https://kubernetes.io/docs/concepts/configuration/overview/#services