cloud-on-k8s | Elastic Cloud on Kubernetes | Continuous Deployment library

I'm trying to deploy the ELK stack to my developing kubernetes cluster. It seems that I do everything as described in the tutorials, however, the pods keep failing with Java errors (see below). I will describe the whole process from installing the cluster until the error happens.

Step 1: Installing the cluster

I'm running this tutorial https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-deploy-elasticsearch.html and found that the elasticsearch operator comes included with a pre-defined secret which is accessed through kubectl get secret quickstart-es-elastic-user -o go-template='{{.data.elastic | base64decode}}'. I was wondering how I can access it in a manifest file for a pod that will make use of this as an env var. The pod's manifest is as follows:

I'm playing with the Elasticsearch operator Kubernetes and created two stateful sets (see https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-orchestration.html):

I have deployed ECK (using helm) on my k8s cluster and i am attempting to install elasticsearch following the docs. https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-deploy-elasticsearch.html

I have externally exposed service/elasticsearch-prod-es-http so that i can connect to it from outside of my k8s cluster. However as you can see when i try to connect to it either from curl or the browser i receive an error "502 Bad Gateway" error.

After reading some documentation regarding Persistent Volumes in Kubernetes I am wondering which one would be the best setup (storage speaking) for running a highly available ElasticSearch cluster. I am not running the typical EFK (or ELK) setup, but I am using ElasticSearch as a proper full-text search engine.

I've read the official Elastic Documentation, but I find it quite lacking of clarification. According to "Kubernetes in Action", Chapter 6:

When an application running in a pod needs to persist data to disk and have that same data available even when the pod is rescheduled to another node, you can’t use any of the volume types we’ve mentioned so far. Because this data needs to be accessible from any cluster node, it must be stored on some type of network-attached storage (NAS).

So if I am not mistaken, I need a Volume and access it through PersistentVolumes and PersistentVolumeClaim with Retain policies.

When looking at Official Volumes, I get a feeling that one should define the Volume type him/herself. Though, when looking at a DigitalOcean guide, it does not seem there was any Volume setup there. I picked that tutorial, but there are dozens on Medium that are all doing the same thing.

So: which one is the best setup for an ElasticSearch cluster? Of course keeping in mind order to not loose any data within an index, and being able to add pods(Kubernetes) or nodes (ElasticSearch) that can access the index.

I am trying to install the Elastic Cloud on Kubernetes (ECK) Kubernetes operator with the all-in-one.yaml file, as per the tutorial: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-install-all-in-one.html But I am getting an error:

Error from server: error when creating "https://download.elastic.co/downloads/eck/1.3.1/all-in-one.yaml": the server responded with the status code 413 but did not return more information (post customresourcedefinitions.apiextensions.k8s.io)

I am a bit lost as to how to proceed solving this issue...

Command:

ive followed this guide https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-deploy-eck.html, then apply this manifest:

I deployed Elasticsearch with the following the page below to my Azure Kubernetes environment.

I installed minikube on my Mac and I'd like to deploy elasticsearch on this k8s cluster. I followed this instruction: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-deploy-elasticsearch.html

The file I created is: