k3s | Lightweight Kubernetes | Continuous Deployment library

We have a 2 node K3S cluster with one master and one worker node and would like "reasonable availability" in that, if one or the other nodes goes down the cluster still works i.e. ingress reaches the services and pods which we have replicated across both nodes. We have an external load balancer (F5) which does active health checks on each node and only sends traffic to up nodes.

Unfortunately, if the master goes down the worker will not serve any traffic (ingress).

This is strange because all the service pods (which ingress feeds) on the worker node are running.

We suspect the reason is that key services such as the traefik ingress controller and coredns are only running on the master.

Indeed when we simulated a master failure, restoring it from a backup, none of the pods on the worker could do any DNS resolution. Only a reboot of the worker solved this.

We've tried to increase the number of replicas of the traefik and coredns deployment which helps a bit BUT:

• This gets lost on the next reboot
• The worker still functions when the master is down but every 2nd ingress request fails
  • It seems the worker still blindly (round-robin) sends traffic to a non-existant master

We would appreciate some advice and explanation:

• Should not key services such as traefik and coredns be DaemonSets by default?
• How can we change the service description (e.g. replica count) in a persistent way that does not get lost
• How can we get intelligent traffic routing with ingress to only "up" nodes
• Would it make sense to make this a 2-master cluster?

UPDATE: Ingress Description:

I want to start a server and a client using docker run rancher/k3s:latest server and docker run -e K3S_TOKEN=xyz -e K3S_URL=https://:6443 rancher/k3s:latest agent

But for some reason, the server and the client aren't able to communicate with each other even if I deploy it on a separate network. Any suggestions as to what can be done?

I've accidently deleted kube-proxy from my k3s cluster. How can I restore it? Any object type is non-existent anymore, this command gives an empty result:

So I am trying to deploy rancher on my K3S cluster.
I installed it using the documentation and helm: Rancher documentation While I am getting access using my loadbalancer. I cannot find the secret to insert into the setup.

They discribe the following command for getting the token:

I have a basic tensorflow serving docker container exposing a model on a kubernetes pod.

I'm experimenting with kubernetes and a minio deployment. I have a k3s 4 node cluster, each one with 4 50GB disk. Following the instructions here I have done this:

1. First I installed krew in order to install the minio and the directpv operators.

2. I installed those two without a problem.

3. I formatted every Available hdd in the node using kubectl directpv drives format --drives /dev/vd{b...e} --nodes k3s{1...4}

4. I then proceed to make the deployment, first I create the namespace with kubectl create namespace minio-tenant-1, and then I actually create the tenant with:

   kubectl minio tenant create minio-tenant-1 --servers 4 --volumes 8 --capacity 10Gi --storage-class direct-csi-min-io --namespace minio-tenant-1

5. The only thing I need to do then is expose the port to access, which I do with: kubectl port-forward service/minio 443:443 (I'm guessing it should be a better way to achieve this, as the last command isn't apparently permanent, maybe using a LoadBalancer or NodePort type services in the kubernetes cluster).

So far so good, but I'm facing some problems:

• When I try to create an alias to the server using mc the prompt answer me back with:

mc: Unable to initialize new alias from the provided credentials. Get "https://127.0.0.1/probe-bucket-sign-9aplsepjlq65/?location=": x509: cannot validate certificate for 127.0.0.1 because it doesn't contain any IP SANs

I can surpass this with simply adding the --insecure option, but I don't know why it throws me this error, I guess is something how k3s manage the TLS auto-signed certificates.

• Once created the alias (I named it test) of the server with the --insecure option I try to create a bucket, but the server always answer me back with:

  mc mb test/hello

  mc: Unable to make bucket \test/hello. The specified bucket does not exist.

So... I can't really use it... Any help will be appreciated, I need to know what I'm doing wrong.

I'm running my Backend on Kubernetes on around 250 pods under 15 deployments, backend in written in NODEJS.

Sometimes after X number of hours (5<30) I'm getting ENOTFOUND in one of the PODS, as follows:

I am having trouble trying to figure out what the values should be for 'Valid Redirect URIs', 'Base URL', 'Backchannel Logout URL'. I am using Keycloak 15.02 along with 10 Spring Boot applications, and 2 Realms. The suite of applications and Keycloak are deployed to our customer sites, and may have more than 2 realms in some cases.

In our dev environment we have two hosts (api.dev, and web.dev) that are running Keycloak, and client apps. Everything is running Docker containers.

The client config for `Valid Redirect URIs', and 'Backchannel Logout URL' currently include the host name web.dev. I'd like to be able to remove that host name to make the Realm configs portable between environments. Having to configure each client in each realm makes for a lot of repetitive and mistake-prone work.

But when I remove the hostname, I get the error: Invalid parameter: redirect_uri.

The redirect URL shown by Keyloak in the request parameters looks the same for both configurations so I dont really understand why its telling me that its invalid.

This works:

That configuration produces the redirect_uri value seen in the following request:

I'm trying to use Ansible to deploy a small k3s cluster with just two server nodes at the moment. Deploying the first server node, which I refer to as "master" is easy to set up with Ansible. However, setting up the second server node, which I refer to as "node" is giving me a challenge because I need to pull the value of the node-token from the master and use it to call the k3s install command on the "node" vm.

I'm using Ansible roles, and this is what my playbook looks like: