docker-elk | The Elastic stack powered by Docker and Compose | Continuous Deployment library

If you are using a text editor then you are probably creating a new file each time you exit it.

That could be an inode reuse issue. There are links to various issues in the META issue 211. Especially see 251.

Tracking which files have been read when those files can get rotated is an extremely hard problem. Way harder than most folks would initially think. A good option to get it right is to checksum the file contents (although this is not foolproof). The file input does not do that, because it can get ridiculously expensive. Instead it implements a very cheap technique that almost always gets it right (but in a few cases it decides it has already read a file that it has not read).

There are other cases where it gets it wrong by duplicating data (which is what you are hitting). As I said, it is a really hard problem.

The port 9600 is the port for the Logstash API, for monitoring logstash, not the port for the http input.

If you want to use the http input and since you didn't specify a port in the configuration, you should use the port 8080, which is the default port for this input.

You will need to expose this port also in your docker configuration.

While non-Docker Elasticsearch logs to /var/log/elasticsearch/elasticsearch.log by default (on Linux), the Docker containers write their logs to STDOUT , which is generally a Docker best practice.

Those logs should be in /var/lib/docker/containers/, but note that on Mac this is inside the small VM layer that Docker is using, so you can't access it directly.

How do you "stop and remove all the docker containers" and still "the ELK server still contains all the old logs"? docker-compose down -v should remove everything and do you see the logs in docker logs or somewhere else?

You need to mount your pipelines.yml file to the container as well. The default location Logstash is looking for a possible pipelines.yml file is /usr/share/logstash/config/ (the same folder you've already mounted the logstash.yml file to).

Please note that you also have to update your current, local pipelines.yml file to the correct paths of the pipelines inside the container. To be precise, you need to change

path.config: "/etc/logstash/my-first-pipeline.config"

to

path.config: "/usr/share/logstash/pipeline/my-first-pipeline.config"

Also, have a look at these official guides for running Logstash with Docker and how to configure multiple pipelines:

• https://www.elastic.co/guide/en/logstash/current/docker-config.html#docker-config

• https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html

I hope I could help you!

EDIT:

The official documentations call the file pipelines.yml instead of pipeline.yml