kubernetes | Production-Grade Container Scheduling and Management | Continuous Deployment library

I just switched from ForkPool to gevent with concurrency (5) as the pool method for Celery workers running in Kubernetes pods. After the switch I've been getting a non recoverable erro in the worker:

amqp.exceptions.PreconditionFailed: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more

The broker logs gives basically the same message:

2021-11-01 22:26:17.251 [warning] <0.18574.1> Consumer None4 on channel 1 has timed out waiting for delivery acknowledgement. Timeout used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more

I have the CELERY_ACK_LATE set up, but was not familiar with the necessity to set a timeout for the acknowledgement period. And that never happened before using processes. Tasks can be fairly long (60-120 seconds sometimes), but I can't find a specific setting to allow that.

I've read in another post in other forum a user who set the timeout on the broker configuration to a huge number (like 24 hours), and was also having the same problem, so that makes me think there may be something else related to the issue.

Any ideas or suggestions on how to make worker more resilient?

Github Actions were working in my repository till yesterday. I didnt make any changes in .github/workflows/dev.yml file or in DockerFile.

But, suddenly in recent pushes, my Github Actions fail with the error

Setup, Build, Publish, and Deploy

...

I run prometheus locally as http://localhost:9090/targets with

Whenever I am trying to run the docker images, it is exiting in immediately.

init container is a great feature in Kubernetes and I wonder whether docker-compose supports it? it allows me to run some command before launch the main application.

I come cross this PR https://github.com/docker/compose-cli/issues/1499 which mentions to support init container. But I can't find related doc in their reference.

In my gitlab CI I always get this hint messages. Yes, I see I have to set git config --global init.defaultBranch main, but everything I'm adding in my stages / jobs of the CI gitlab config is executed after fetching.

I found a source describing that the default gc used changes depending on the available resources. It seems that the jvm uses either g1gc or serial gc dependnig on hardware and os.

The serial collector is selected by default on certain hardware and operating system configurations

Can someone point out a more detailed source on what the specific criteria is and how that would apply in a dockerized/kubernetes enivronment. In other words:

Could setting resource requests of the pod in k8s to eg. 1500 mCpu make the jvm use serial gc and changing to 2 Cpu change the default gc to g1gc? Do the limits on when which gc is used change depending on jvm version (11 vs 17)?

I have just set up a kubernetes cluster on bare metal using kubeadm, Flannel and MetalLB. Next step for me is to install ArgoCD.

I installed the ArgoCD yaml from the "Getting Started" page and logged in.

When adding my Git repositories ArgoCD gives me very weird error messages: The error message seems to suggest that ArgoCD for some reason is resolving github.com to my public IP address (I am not exposing SSH, therefore connection refused).

I can not find any reason why it would do this. When using https:// instead of SSH I get the same result, but on port 443.

I have put a dummy pod in the same namespace as ArgoCD and made some DNS queries. These queries resolved correctly.

What makes ArgoCD think that github.com resolves to my public IP address?

EDIT:

I have also checked for network policies in the argocd namespace and found no policy that was restricting egress.

I have had this working on clusters in the same network previously and have not changed my router firewall since then.

Both replica set and deployment have the attribute replica: 3, what's the difference between deployment and replica set? Does deployment work via replica set under the hood?

configuration of deployment