fluentd | Fluentd : Unified Logging Layer

I have checked your windows server version. you are using windows server 2016 (1607 version). since you are using the 1607 version you cant use WSL, Hyper-V, LinuxKit, Docker Desktop to run the Linux container image i.e (node, alpine, Nginx, etc..)

Please refer this StackOverflow question. you will find the solution.

I was able to resolve this, unfortunately I won't be able to find out why this happened.

I tried removing and installing docker-ce and dependencies again. I wasn't able to remove due to containerd.service not stopping. I found it was set to always restart, which would normally make sense. I then ran sudo systemctl disable docker containerd and rebooted. I confirmed those services were no longer running by following journalctl output, looking for the usual restarting and core-dump errors from docker and containerd.

I ran sudo apt remove docker-ce and sudo apt autoremove again, then ran docker's get-docker.sh which reinstalled docker. I then ran sudo systemctl enable docker containerd and sudo systemctl start docker containerd. Docker is the same version it was before and the hello-world container and other containers of mine that wasn't previously running is now running successfully.

I faced the same problem, but I used to make exactly the same image where everything works to this day. I can't figure out what has changed.

But if you need to urgently solve the problem, use my in-person image:

It is a little different in Elasticsearch 6.X as it had mapping types, which is not used anymore.

Try something like this:

You can receive logs directly in elasticsearch (without even having to format them to json) through the syslog plugin. This probably would be the most straightforward solution to your problem.

If for some reason u need to use some kind of log aggregator, I personally would not recommend fluentd, as it can bring unecessary complexity with it.

But you could use logstash which is supported by elasticsearch and you can find plenty of documentation about it.

Nuking /var/lib/docker seems to get the system working again - this removes all images and lots of other docker-related data.

This doesn't feel like a great solution, but it'll have to do for now.

To scrape logs only from specific Pods, you can use:

If a pod crashes, all logs still will be accessible in efk. No need to add a persistent volume to the pod with your application only for storing log file.

Main question is how to get logs from this file. There are two main approaches which are suggested and based on kubernetes documentation:

1. Use a sidecar container.

   Containers in pod have the same file system and sidecar container will be streaming logs from file to stdout and/or stderr (depends on implementation) and after logs will be picked up by kubelet.

   Please find streaming sidecar container and example how it works.

2. Use a sidecar container with a logging agent.

   Please find Sidecar container with a logging agent and configuration example using fluentd. In this case logs will be collected by fluentd and they won't be available by kubectl logs commands since kubelet is not responsible for these logs.

Most important part of this flag is it works only within one pod, all containers within one pod will share processes between each other.

In described approach job is supposed to be used. Job creates a separate pod so it won't work this way. Container should be a part of the "main" pod with all other containers to have access to running processes of that pod.

More details about process sharing.

It's possible to get processes from the containers directly using kubectl command.

Below is an example how to check state of the processes using pgrep command. The pgrepContainer container needs to have the pgrep command already installed.

job.yaml: