offheap | Used to be called go

Our Spring Boot application shall be able to fetch data from a remote Ehcache instance. For this, the following configuration has been added.

I recently changed from running Ignite with Multicast IP Finder to using Static IP Finder to make district sets of clusters per machine.

Prior to this change I was able to establish a connection between the client and the Ignite server. However, after specifying the Static IP finder in the configuration of the server and the client, when I attempt to connect to the server a IgniteCheckedException: No session found is thrown and takes down the JVM killing my application. The Ignite server however stays up.

Just to test I tried to revert to the Multicast IP finder but I am now getting the same error.

I have been able to connect to Ignite other clusters, but not the one local to the client.

This is the client configuration:

I'm using Ehcache as buffer to pass data for all the clients connected to a WebSocket(Spring).

CacheEventListener implementation:

I start the Apache Ingnite node, a server node, and another client node.

My scenario is: Close the client node, and how to update the service node Topology Snapshot at the same time.

Now, the Topology Snapshot is refreshed only when the NodeFailed event is received by the server after 20 seconds.

What method or configuration on the server side can receive the NodeFailed event immediately or refresh the Topology Snapshot?

This is server log:

We use Ignite 2.7.6 in both server and client modes: two server and six clients.

At first, each app node with client Ignite inside had 2G heap. Each Ignite server node had 24G offheap and 2G heap.

With last app update we introduced new functionality which required about 2000 caches of 20 entires (user groups). Cache entry has small size up to 10 integers inside. These caches are created via ignite.getOrCreateCache(name) method, so they have default cache configurations (off-heap, partitioned).

But in an hour after update we got OOM error on a server node:

when I search off-heap in Spark configuration, there are two properties related(spark.executor.memoryOverhead and spark.memory.offHeap.size), I am not sure the relationship between these two.

If I enable spark.memory.offHeap.enabled, will spark.memory.offHeap.size be part of spark.executor.memoryOverhead? or these two types of off-heap memory are independent(thus the total off-heap memory is the sum of the two)

I am trying to set the configuration of a few spark parameters inside the pyspark shell.

I tried the following

spark.conf.set("spark.executor.memory", "16g")

To check if the executor memory has been set, I did the following spark.conf.get("spark.executor.memory")

which returned "16g".

I tried to check it through sc using sc._conf.get("spark.executor.memory")

and that returned "4g".

Why do these two return different values and whats the correct way to set these configurations.

Also, I am fiddling with a bunch of parameters like "spark.executor.instances" "spark.executor.cores" "spark.executor.memory" "spark.executor.memoryOverhead" "spark.driver.memory" "spark.driver.cores" "spark.driver.memoryOverhead" "spark.memory.offHeap.size" "spark.memory.fraction" "spark.task.cpus" "spark.memory.offHeap.enabled " "spark.rpc.io.serverThreads" "spark.shuffle.file.buffer"

Is there a way that will set the configurations for all the variables.

EDIT

I need to set the configuration programmatically. How do I change it after I have done spark-submit or started the pyspark shell? I am trying to reduce the runtime of my jobs for which I am going through multiple iterations changing the spark configuration and recording the runtimes.

I have allocated 5 GB Non heap memory to Ignite as can be verified at application startup.

...

I m trying to use apache ignite, I m generating my node using the ignite web console.

I needed to configure 2 caches from database and enabled the persistence storage since the two table have lot of data.

Here is what I have done (the console)