DotNetty | DotNetty project – a port | TCP library

I have this application using Akka.net cluster feature. The people who wrote the code have left the company. I am trying to understand the code and we are planning a deployment.

The cluster has 2 types of nodes

QueueServicer: supports sharding and only these nodes should participate in sharding.
LightHouse: They are just seed nodes, nothing else.

Lighthouse : 2 nodes
QueueServicer : 3 Nodes

I see one of the QueueServicer node unable to join the cluster. Both lighthouse nodes are refusing connection. It constantly tries to join and never succeeds. This has been happening for the last 5 days or so and the node is never dying also. Its CPU and memory usage is high. Also It doesn't have any queue processor actors running when filtered search through the log. It takes long hours for Garbage collection etc. I see in the log for this node, the following.

{"timestamp":"2021-09-08T22:26:59.025Z", "logger":"Akka.Event.DummyClassForStringSources", "message":Tried to associate with unreachable remote address [akka.tcp://myapp@lighthouse-1:7892]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: [Association failed with akka.tcp://myapp@lighthouse-1:7892] Caused by: [System.AggregateException: One or more errors occurred. (Connection refused akka.tcp://myapp@lighthouse-1:7892) ---> Akka.Remote.Transport.InvalidAssociationException: Connection refused akka.tcp://myapp@lighthouse-1:7892 at Akka.Remote.Transport.DotNetty.TcpTransport.AssociateInternal(Address remoteAddress) at Akka.Remote.Transport.DotNetty.DotNettyTransport.Associate(Address remoteAddress) --- End of inner exception stack trace --- at System.Threading.Tasks.Task1.GetResultCore(Boolean waitCompletionNotification) at Akka.Remote.Transport.ProtocolStateActor.<>c.b__12_18(Task1 result) at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke() at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)

{"timestamp":"2021-09-08T22:26:59.025Z", "logger":"Akka.Event.DummyClassForStringSources", "message":Tried to associate with unreachable remote address [akka.tcp://myapp@lighthouse-0:7892]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: [Association failed with akka.tcp://myapp@lighthouse-0:7892] Caused by: [System.AggregateException: One or more errors occurred. (Connection refused akka.tcp://myapp@lighthouse-0:7892) ---> Akka.Remote.Transport.InvalidAssociationException: Connection refused akka.tcp://myapp@lighthouse-0:7892 at Akka.Remote.Transport.DotNetty.TcpTransport.AssociateInternal(Address remoteAddress) at Akka.Remote.Transport.DotNetty.DotNettyTransport.Associate(Address remoteAddress) --- End of inner exception stack trace --- at System.Threading.Tasks.Task1.GetResultCore(Boolean waitCompletionNotification) at Akka.Remote.Transport.ProtocolStateActor.<>c.b__12_18(Task1 result) at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke() at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)

There are other "Now supervising", "Stopping" "Started" logs which I am omitting here.

Can you please verify if the HCON config is correct for split brain resolver and Sharding?

I think LightHouse/SeeNodes should not have the sharding configuration specified. I think it is a mistake. I also think, split brain resolver configuration might be wrong in LightHouse/SeedNodes and should not be specified for seed nodes.

I appreciate your help.

Here is the HOCON for QueueServicer Trimmed

akka {
    loggers = ["Akka.Logger.log4net.Log4NetLogger, Akka.Logger.log4net"]
    log-config-on-start = on
    loglevel = "DEBUG"
    actor {
        provider = cluster
        serializers {
            hyperion = "Akka.Serialization.HyperionSerializer, Akka.Serialization.Hyperion"
        }
        serialization-bindings {
            "System.Object" = hyperion
        }
    }

I'm using DotNetty in Unity, and has the following client bootstrap, when the server is down, it will take the "clientBootstrap.ConnectAsync()" around 17 seconds to give up trying to establish a connection, during which, the client is frozen. Actually, it doesn't need to try for 17 seconds to know there can be no connection. How can I configurate so the "clientBootstrap.ConnectAsync()" trys only for maybe 5 seconds? I added ".Option(ChannelOption.ConnectTimeout, TimeSpan.FromMilliseconds(5000))", but still, it takes 17 seconds to stop trying. The option only triggered an extra timeout exception.

As we know that DotNetty is a port of Netty to .net, I searched a lot, but couldn't find out which version of Netty is equal to the DotNetty 0.6.0 (latest DotNetty version).

If it's Netty 5, then there might be some problems, Netty 5 was abandoned.

Thanks.