vm-memory | Virtual machine 's guest memory crate | Emulator library

Got this answer from AWS Support

I checked the particular metric and can also see the JVM increasing from the last few days. However, I do not think this is an issue as JVM is expected to increase over time. Also, the garbage collection in ES runs once the JVM reaches 75% (currently its around 69%), after which you would see a drop in the JVM metric of your cluster. If JVM is being continuously > 75 % and not coming down after GC's is a problem and should be investigated.

The other thing which you mentioned about clearing the cache for fielddata for all indices was not helping in reducing JVM, that is because the dedicated master nodes do not hold any indices data and their related caches. Clearing caches should help in reducing JVM on the data nodes.

In a container, MaxRAMPercentage is calculated basing on cgroup memory.limit_in_bytes value.

It is limits Kubernetes configuration that affects cgroup memory limit. So, in your case, the maximum heap size will be set to 256M (50% of the limit).

I finally found my issue and I want to share it so others can benefit in some way from this.

As I found on my last edit I had a thread problem that was causing all the memory consumption over time, specifically we was using an asynchronous method from a third party library without properly taking care those resources (ensure those calls was ending correctly in this case).

I was able to detect the issue because I used a memory limit on my kubernete deployment from the beginning (which is a good practice on production environments) and then I monitored very closely my app memory consumption using tools like jstat, jcmd, visualvm, kill -3 and most importantly the -XX:NativeMemoryTracking=summary flag that gave me so much detail in this regard.

This is what one of my cluster nodes looks like from heap perspective:

You can use the execute-count of you sink component to measure the output of your topology. If each of your components have a 1:1 input:output ratio then this will be your throughput.

However, if you are windowing tuples into batches or splitting tuples (like separating sentences into individual words) then things get a little more complicated. You can get the input into your topology by looking at the emit-count of your spout components. You could then use this in comparison to you bolt execute-counts to create your own throughput metric.

An easy way to get programmatic access to these metrics is via the Heron Tracker REST API. You can use your chosen language's HTTP library (like Requests for Python) to query the last 3 hours of data for a running topology. If you require more than 3 hours of data (the maximum stored by the topology TMaster) you will need to use one of the other metrics sinks to send metrics to an external database. Heron currently provides sinks for saving to local files, Graphite or Prometheus. InfluxDB support is in the works.

It's not an 'instruction number'. It is an instruction address, and it is incremented by the length of the previous instruction.

1. Sure.
2. ulimit -n . See Increasing ulimit on CentOS.
3. There really isn't one - as many as needed depending on a lot of factors, such as your mergefactor (if you have many files, the number of open files will be large as well - this is especially true for the indices that aren't full imports. Check the number of files in your data directories and issue an optimize if the same index has become very fragmented and have a large mergefactor), number of searchers, other software running on the same server, etc.
4. It could. Yes (or at least it won't function properly, as it won't be able to open any new files). No. In practice you'll get a message about being unable to open a file with the message "Too many open files".

As you can see from URL this plugin is not developed by VisualVM devs. It was a separate project at java.net. Unfortunately java.net was discontinued and the author of the memory pool plugin did not set up new project yet. There is already a discussion about the memory pool plugin at new visualvm home at github.