procfs | Go package for accessing the /proc virtual filesystem | File Utils library

This is probably not a good way to be deploying your services onto virtual machines. You don't want to just launch your GitLab CI job and then hope that it results in what you want. Managing each VM separately is both going to be tedious and error-prone.

You probably want to do is have a method that has a declarative way to define/describe your infrastructure and the state of how that infrastructure should be configured and the applications running on it.

For example, you could:

1. Use a proper orchestrator, such as docker swarm or Kubernetes AND/OR
2. Use a provisioning tool, such as Ansible connected to each VM, or if your VMs run in the cloud, Terraform or similar.

In both these examples, you can leverage these tools from a single GitLab CI job and deploy changes to all of your VMs/clusters at once.

For example, instead of running your docker-compose on 20 hosts, you can join all 20 VMs to the same docker swarm.

Then in your compose file, you create a deploy key specifying how many replicas you want across the swarm, including numbers per node. Or use mode: global to simply specify you want one container of the service per host in your cluster.

This issue has been resolved. The reason for the confusion was the log output as it was hard to read, but eventually I found this:

It is very likely that pip3 is pointing to a different python instance.

Imagine you had python, python3, python3.6 and python3.8 all installed on your system. Which one would pip3 install packages for? (who knows?)

It is almost always safer to do python3.8 -m pip list/install since you can be sure that python3.8 somefile.py will be using the same files you just saw. (even better, do python3.8 -m venv /path/to/some/virtualenv and then make sure that is activated, then you can be sure pip points to the same python)

If you want to disallow reading, you can just omit explicitly setting the .read field of struct file_operation. If the structure is defined as static and therefore initialized to 0, all fields that are not explicitly overridden will default to NULL, and the kernel will simply not do anything and return an error (I believe -EINVAL) whenever user code tries to call read on your open file.

Alternatively, if you want to return a custom error, you can define a dummy function which only returns an error (like for example return -EFAULT;).

Do you think the way I am "writing" the key into the buffer is the right way to do it ?

This is wrong for multiple reasons.

First, your copy_from_user() is blindly trusting the user count, so this result in a kernel buffer overflow on the temp variable, which is pretty bad. You need to check and/or limit the size first. You are also not checking the return value of copy_from_user(), which you should (and it is not an int, but rather an unsigned long).

This issue was solved recently. Here is more information: https://github.com/prometheus-community/helm-charts/issues/467 and here: https://github.com/prometheus-community/helm-charts/pull/757

Here is the solution (https://github.com/prometheus-community/helm-charts/issues/467#issuecomment-802642666):

[you need to] opt-out the rootfs host mount (preventing the crash). In order to do that you need to specify the following value in values.yaml file:

Taking this as a question:

So, in any chance those four columns will show different UIDs.

Yes. Subject to various limitations, processes can change their effective and saved UIDs and GIDs. This is what the setuid(), setgid(), seteuid(), and setegid() functions do.

The filesystem uid and gid are Linux-specific features that are used mainly, if not entirely, in the context of NFS (see filesystem uid and gid in linux). These can be manipulated with setfsuid() and setfsgid(), subject, again, to limitations.

For most processes, all the UIDs will the same and all the GIDs will be the same, but it is conceivable that they would all be different. It is a function of the behavior of the process.

Quite some more digging turned up this answer to "About mounting and unmounting inherited mounts inside a newly-created mount namespace" which points in the correct direction, but needs additional explanations (not least due to basing on a misleading paragraph about mount namespaces being hierarchical from man pages which Michael Kerrisk fixed some time ago).

Our starting point is when runc sets up the (test) container, for masking system paths especially in the container's future /proc tree, it creates a set of new mounts to either mask out individual files using /dev/null or subdirectories using tmpfs. This results in procfs being mounted on /proc, as well as further sub-mounts.

Now the test container starts and at some point a process unshares into a new user namespace. Please keep in mind that this new user namespace (again) belongs to the (real) root user with UID 0, as a default Docker installation won't enable running containers in new user namespaces.

Next, the test process also unshares into a new mount namespace, so this new mount namespace belongs to the newly created user namespace, but not to the initial user namespace. According to section "restrictions on mount namespaces" in mount_namespaces(7):

If the new namespace and the namespace from which the mount point list was copied are owned by different user namespaces, then the new mount namespace is considered less privileged.

Please note that the criterion here is: the "donor" mount namespace and the new mount namespace have different user namespaces; it doesn't matter whether they have the same owner user (UID), or not.

The important clue now is:

Mounts that come as a single unit from a more privileged mount namespace are locked together and may not be separated in a less privileged mount namespace. (The unshare(2) CLONE_NEWNS operation brings across all of the mounts from the original mount namespace as a single unit, and recursive mounts that propagate between mount namespaces propagate as a single unit.)

As it now is not possible anymore to separate the /proc mountpoint as well as the masking submounts, it's not possible to (re)mount /proc (question 1). In the same sense, it is impossible to unmount /proc/kcore, because that would allow unmasking (question 2).

Now, when deploying the test container using --security-opt systempaths=unconfined this results in a single /proc mount only, without any of the masking submounts. In consequence and according to the man page rules cited above, there is only a single mount which we are allowed to (re)mount, subject to the CAP_SYS_ADMIN capability including also mounting (besides tons of other interesting functionality).

Please note that it is possible to unmount masked /proc/ paths inside the container while still in the original (=initial) user namespace and when possessing (not surprisingly) CAP_SYS_ADMIN. The (b)lock only kicks in with a separate user namespace, hence some projects striving for deploying containers in their own new user namespaces (which unfortunately has effects not least on container networking).

As posted in the comments:

Please add to the question the steps that you followed (editing any values in the Helm chart etc). Also please check if the nodes are not over the limit of pods that can be scheduled on it. Here you can find the link for more reference: LINK.

no processes occupying 9100 on the given node. @DawidKruk The POD limit was reached. Thanks! I expected them to give me some error regarding that rather than vague node selector property not matching

Not really sure why the following messages were displayed:

• node(s) didn't have free ports for the requested pod ports
• node(s) didn't match node selector

The issue that Pods couldn't be scheduled on the nodes (Pending state) was connected with the Insufficient pods message in the $ kubectl get events command.

Above message is displayed when the nodes reached their maximum capacity of pods (example: node1 can schedule maximum of 30 pods).

More on the Insufficient Pods can be found in this github issue comment:

That's true. That's because the CNI implementation on EKS. Max pods number is limited by the network interfaces attached to instance multiplied by the number of ips per ENI - which varies depending on the size of instance. It's apparent for small instances, this number can be quite a low number.

Docs.aws.amazon.com: AWSEC2: User Guide: Using ENI: Available IP per ENI

-- Github.com: Kubernetes: Autoscaler: Issue 1576: Comment 454100551

Additional resources:

• Stackoverflow.com: Questions: Pod limit on node AWS EKS

Since you didn't mention it above: did you instrument a Go application? The Elastic APM Go "Agent" is a package which you use to instrument your application source code. It is not an independent process, but runs within your application.

So, first (if you haven't already) instrument your application. See https://www.elastic.co/guide/en/apm/agent/go/current/getting-started.html#instrumenting-source

Here's an example web server using Echo, and the apmechov4 instrumentation module: