ilock | Inter-process named lock library | File Utils library

As you already know, CPSubsystem is a CP system in terms of CAP theorem. It has to be enabled explicitly to use, because it has some limitations and prerequisites. One of them is, at least 3 Hazelcast members should exist in the cluster. Actually, 2 members is sufficient but Hazelcast's CPSubsystem rejects to work with 2 members because majority of 2 members is 2 again, and it's prone to be unavailable once one of the members crashes.

HazelcastInstance.getLock() uses async replication of Hazelcast and cannot provide CP guarantees under failures. This is fine for some systems/applications but not for all. That's why choosing between a best-effort locking mechanism vs CP based locking mechanism should be explicit and applications relying on the lock should be designed depending on this choice. See Daniel Abadi's The dangers of conditional consistency guarantees post related to this choice. That's why, CPSubsystem().getLock() does not fallback to best-effort/unsafe locking mechanism when cluster size is below 3.

HazelcastInstance.getLock() is deprecated in 3.12 and will be removed in 4.0. But Hazelcast will provide an unsafe (development) mode for CP data structures, which will work with any number of members and will be based on async replication similar to Hazelcast AP data structures.

Since Hazelcast locks are distributed, they are protected by thread ownership so that only the thread who locked it can unlock.

In your code, you use tryLock which may return false. However, finally block executes unlock operation in any case. So, each time you can't get the lock (because it is locked by some other thread in the cluster), you will try to unlock it and you will get that exception.

It is a good idea to use lock.isLockedByCurrentThread() to check the ownership.

fd==1 refers to stdout, or Standard Out. It's a common feature of Unix-like Operatin Systems. The kernel knows that it's not a real file. Writes to stdout are mapped to terminal output.

it is a correct approach when using Reactor, because you took care of offsetting the blocking portion into a dedicated Scheduler/Thread.

But I'd say mutually exclusive code like this is not a very good fit for reactive programming in general: you lose one of the key benefits of doing more with less threads, you risk blocking other parts of the application should you forget to publishOn a dedicated thread, etc...

System.getTimestamp() might not be unique. You can call it twice and get the same result.

You could get Hazelcast to generate the id for you https://docs.hazelcast.org//docs/3.10.5/javadoc/com/hazelcast/flakeidgen/FlakeIdGenerator.html

As we discussed in comments (with @mdogan), there's no built-in mechanism for that.

To achieve what I wanted, I have to:

1. Store of all acquired lock keys by current Hazelcast member in ConcurrentSkipListSet.
2. Before shutdown, loop through ConcurrentSkipListSet check if ILock.isLocked(), if true then wait for release.
3. Shutdown current Hazelcast member.

First I'll qualify that I'm not exactly an expert on which threads these are executed on and when so some may disagree but here're my thoughts on this so anyone please chime in as this looks to be an interesting case. Your first solution mixes the Hazelcast event threading with it's operation threading. In fact you're triggering three operations to be invoked as a result of the single event. If you put some arbitrary latency in your call to updateProcductDeviceTable, you'll see that eventually, it will slow down but resume up again after some time. This will cause your local event queue to pile up while operations are invoked. You could put everything you're doing in a separate thread which you can "wake" up on #itemAdded or if you can afford to have a bit of latency, do what you're doing on your second solution. I would, however, make a couple changes in

listenToDiagnosticsRuns() method:

If it is OK to process the task outside the scope of the current request, i.e. to queue it for later, then you can think of a sequence like this¹:

Implement a resource lock (monitor) and a List of tasks.

1. For each request:

2. Lock the List, Add current task to the List, remember nr. of tasks in the List, unlock the List.

3. Try to acquire the lock.

4. If unsuccessful:

   • If the nr. of tasks in the list < threshold X, then Return.
   • Else Acquire the Lock (will block)

5. Lock the List, move it's contents to a temp list, unlock the List.

6. If temp list is not empty

   • Execute the tasks in the temp list.

   • Repeat from step 5.

7. Release the lock.

The first request will go through the whole sequence. Subsequent requests, if the first is still executing, will short-circuit at step 4.

Tune for the optimal threshold X (or change it to a time-based threshold).

¹ If you need to wait for the task in the scope of the request, then you need to extend the process slightly:

Add two fields to the Task class: completion flag and exception.

At step 4, before Returning, wait for the task to complete (Monitor.Wait) until its completion flag becomes true. If exception is not null, throw it.

At step 6, for each task, set the completion flag and optionally the exception and then notify the waiters (Monitor.PulseAll).

After spending days investigating this JNI issue, we have finally found out the reason and I would like to share our experience here so that hopefully it will help others.

First of all, the reason why we needed to use JNI in the first place was because we needed to make use of a 3rd party network library that was a Linux native lib, and unfortunately that was the cause of our problem.

The library provided us a callback handle that we implemented to receive incoming network messages from it, and this callback, we later found out, was simply a signal handler. So, it means that this signal handler would get called whenever a signal popped up, even during gc.

Since C threads keep running during safepoints in JVM, it would have been fine if those C threads weren't attached to the JVM, otherwise disasters would certainly strike.

Here is kind of what we thought had happened. (everything below happened in the JNI layer)

1. App starts. We init and cached JNI resources, e.g. Jmv*, Method ID, etc.
2. We registered a C function to the library to receive messages. The C function is a function that would call JNI APIs to allocate memory to accommodate the received messages and pass them onto Java. After that, we just started to wait for incoming messages.
3. When a message finally arrived, the C function mentioned above was called to handle the message, but wait... what was this thread that's handling the callback. That would have been the main thread or hmm... any available threads.
4. As taught in any JNI textbook, we attached the thread to JVM first if not yet done so before calling any JNI APIs. Great!
5. Now, during a GC, all Java threads were blocked, but the C layer was still running. At this critical moment, if a message arrived, some thread (any threads) was called up to handle the message. But what threads were still available during gc? All application threads were blocked and the only ones that were still running at this moment (our guess) were unfortunately the gc threads.

The gdb stacktrace that we were seeing was basically what happened when a gc thread that was actually in a middle of doing some work on the heap and then got a call from our application to do some application work and then a few JNI API calls... BOOM

Solution:

• Have a C thread that handles the library callback
• Never attach that thread to JVM
• Have other threads attach to the JVM to do the Native-Java transition.

p.s. maybe some of the details weren't exactly accurate, so any JVM expert advice is welcomed. I will try to correct them as advised.

Thanks

Update.1 (@apangin): We have another gdb stacktrace here. Just wondering if the GangWorker at #18 was a parallel GC thread.