fluentd | Docker image with docker-gen and fluentd | Continuous Deployment library

In our fluentd kubernetes deamonset we wanted to store pod name in a separate field named as application_id. But type of this field must be keyword. For that we need to provide index mapping to elasticsearch output plugin to create index as per our need. But we did not found any key in elasticsearch output plugin to provide index mapping. Can anyone help us to resolve this issue?

I use my custom fluentd plugin and it does not work with ubuntu20 but for other ubuntu version is no problem.

Here is my error

I installed Docker on a CentOS 7 machine and DNS is not working within containers.

So, if I run nslookup google.com on my host, it resolves correctly. However, if I do docker container run busybox nslookup google.com I get:

I am publishing a data to elasticsearch using fluentd. It has a field Data.CPU which is currently set to string. Index name is health_gateway

I have made some changes in python code which is generating the data so now this field Data.CPU has now become integer. But still elasticsearch is showing it as string. How can I update it data type.

I tried running below commands in kibana dev tools:

I have the following json message on Fluentd input

{"foo":{"bar":{"abc":"[\n {\n \"ip\":\"192.168.1.1\",\n \"hostname\":\"pc\",\n \"mac\":\"01:02:03:04:05:06\"\n} \n]"}}}

And want to get the output message

We're using https://github.com/fluent/fluentd-kubernetes-daemonset to deploy Fluentd in our K8s cluster. We have 5 nodes in the cluster, which means there are 5 Fluentd pods.

Each Fluentd pod in the DaemonSet exposes Prometheus metrics on localhost:24231/metrics endpoint via fluentd prometheus plugin. I'm having trouble finding the relevant bits of documentation on how to configure Prometheus to collect those metrics from every Pod's localhost:24321/metrics endpoint.

• there are N pods in a DaemonSet
• each pod has prometheus metrics exposed on localhost:24321

I need to configure Prometheus so it's able to scrape those metrics somehow. Any tips on how to solve this or examples of such configurations would be much appreciated!

hope you're all well during this pandemic.

I've got a kubernetes cluster running. The comunication between pods is done through kafka. It is currently logging to stdout only. No files. no kafka topic. This is obviously pretty bad.

I want to setup a grafana instance that lets me centralize all logs there. The storage would be Loki + S3

In order to do that, I found that many people use tools like Fluentd, FluentBit and Promtail, which centralizes the logs and sends them to Loki. However, I already have Kafka running. I can't see why I'd use some tool like fluentd if I can send all logs to kafka through a "logging" topic.

My question is: How could I send all messages inside the logging topic to Loki? Fluentd cannot get input from kafka.

Would I have to setup some script that runs periodically, sorts data and sends it to loki directly?

I have to run two CronJobs in Kubernetes (AWS-EKS) and I have below configuration. When I apply the template, only one CronJob is getting created. The one that gets created is always the second one. So it looks like the first one is getting overwritten by the second. I am unable to figure out what am I doing wrong.

I am trying to bring up an on-prem k8 cluster using kubespray with 3 master and 5 worker nodes. The node IPs are from 2 different subnets.

Ansible inventory: