local-path-provisioner | Dynamically provisioning persistent local storage | Storage library

The key to your problem was updating PSP.

I would like to add something about PSP:

According to this documentation and this blog:

As of Kubernetes version 1.21, PodSecurityPolicy (beta) is deprecated. The Kubernetes project aims to shut the feature down in version 1.25.

However I haven't found any information in Rancher's case (the documentation is up to date).

Rancher ships with two default Pod Security Policies (PSPs): the restricted and unrestricted policies.

See also:

• The benefits of Pod Security Policy
• Secure Kubernetes cluster PSP
• Pod Security Policies

Even in an empty cluster with just one master node, there are at least 5 components that reach out to the API server on a regular basis:

• kubelet for the master node
• Controller manager
• Scheduler
• CoreDNS
• Kube proxy

This is because API Server acts as the only entry point for all components in Kubernetes to know what the cluster state should be and take action if needed.

If you are interested in the details, you could enable audit logs in the API server and get a very verbose file with all the requests being made. How to do so is not the goal of this answer, but you can start from the apiserver documentation.

I think you can just delete one of the storage class.

There are mandatory data to be maintain when a cluster is running (eg. default service token). When you shutdown (eg. systemctl stop k3s) the cluster (not just delete pods) these will be released.

Turns out that the issue was not a kubernetes one, but a configuration issue. I was able to solve the problem I was experiencing by modifying the postgres hba configuration file to have a broader range of ip addresses. As mentioned by @gohm'c there are many resources online describing how to fix the connection error that I was encountering. Thanks to @gohm'c for their help in debugging.

There is an open issue for offline installation- default gateway need to be set.

follow this comment, it should work.

[aiops@7 ~]$ ip route
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
192.168.100.0/24 dev ens192 proto kernel scope link src 192.168.100.7 metric 100
sudo ip route add default via 192.168.100.1

1. I think you need to restart K3s via systemd if you want your cluster back after kill. Try command:
   sudo systemctl restart k3s This is supported by the installation script for systemd and openrc. Refer rancher doc

2. The pod-xxx id will remain same as k3s-killall.sh doesn't uninstall k3s (you can verify this, after k3s-killall script k3s -v will return output) and it only restart the pods with same image. The Restarts column will increase the count of all pods.

When you install k3s based on the instructions here it won't install Calico CNI by default. There is a need to install Calico CNI separately.

To answer you question, let's analyse /usr/local/bin/k3s-killall.sh file, especially the part with iptables command:

To reset a pod, you can just delete it. If it's managed by deployment (pods in your question should be), they should be recreated automatically.