round-robin | faster round-robin balancing algorithm written in golang | SMS library

As per the question

RSS did not have a good load balance between CPU cores case the rx packets has been modified by insert tags between mac and ip

There are couple of items which needs to be clarified. so let me explain

1. Certain Physical and virtual NIC exposes RSS via DPDK RX offload for fixed tuples like IP, Protocol, TCP|UDP|SCTP port number.
2. Certain NIC allows to configure the hash reta algorithm to better suit to needs (example when the source destination IP address is fixed we can skip and use others).
3. As I recollect from DPDK 18.11, RTE_FLOW is been introduced to support RSS on selected RX queues (example Q1,Q2,Q3 can be RSS for TCP packets while Q4,Q5 can be used for UDP). But again this is based on the either Inner or Outer IP+PORT number
4. For DPDK version 19.11 onwards RTE_FLOW is been enhanced to support RAW Pattern. The intend of this feature is support Special protocol which by default the NIC does not understand like (VXLAN, GENEVE, RTP and other protocols).
5. For NIC like Fortville & Columbiaville (from Intel) allows loading of special firmware via DDP (Dynamic Device Personation) to configure special fabric headers or MPLS like headers (between ethernet and ip) to be parsed, lookup and used as seed for RSS (allowing better distribution).
6. There are NIC which do support L2 layer but these would be limited SMAC, DMAC, VLAN1,VLAn2, MPLS only and not custom header.

Hence depending upon NIC, vendor, RSS support for L2 and firmware the ability calculating RSS on fields between varies in port init or RTE_FLOW specific configuration. For example RSS ETH supported on

1. I40E is I40E_INSET_DMAC | I40E_INSET_SMAC
2. DPAA2 is NH_FLD_ETH_TYPE and NET_PROT_ETH
3. CNXK is RSS_DMAC_INDEX
4. OCTEONX2 is FLOW_KEY_TYPE_VLAN and FLOW_KEY_TYPE_CH_LEN_90B

Hence for NIC ixgbe and XXV710 there is no ready support for the custom header between ethernet and IP.

Alternatives:

1. Use a smart NIC or FPGA: that is programmed to parse and RSS on your specific headers to RSS on multiple RX queue
2. Work with Intel using XXV710 (Fortville): to create DDP which can parse your specific headers as RSS on multiple RX queue.
3. Identify DPDK NIC: which can parse RAW header as defined 12.2.6.2. Once the support is added by the vendor you can create a simple traffic spread tool which will ensure traffic distribution across the desired RX queues in round robin fashion.
4. Use SW to support the missing hardware.

note: I am not recommending use of HW based static Round Robin as it will create 2 fold problem

• if it is pure DPDK BOND Round ROBIN you will not have any flow pinning
• If you use hash based pinning, there are chances elephant flow can pushed 1 or a few queues causing performance drops in cpu processing.
• My recommendation is to use EVENTDEV model with atomic mode, which ensure better cache locality (at a given instance same flows will fall onto worker thread) and almost linear performance performance. sample app

For option 4 (Software Model):

1. Disable RSS in port_init function
2. Use single RX queue to receive all the packet either custom rx thread.
3. calculate the hash based on the desired header and update mbuf hash field.
4. Use rte_distributor library to spread traffic based on the custom.
5. or use rte_eventdev with atomic model to spread the work load on multiple worker.

[Clarification from Comments]:

1. I have asked relevant practitioners， they said modifying the pmd driver can solves my problem, its the only way?

[ANSWER] Since you are using custom header and not generic VLAN|IP|Port this suggestion is not correct. As you have clarified in question and comments you want to distribute like RSS for custom header.

1. I haven't written any code about rte_flow distribution yet, i read the rte_flow example, don't see the code to configure round-robin

[ANSWER] as explained above not all nic support RSS and RAW. Since your current NIC is ixgbe and i40e the function of parsing and executing RSS for custom header is unlikely. you can try option 2 (work with intel create new ddp) for i40e to achieve the same or implement in SW as suggested in option 4.

1. im not ask for solution, I just want to know how to set up round robin by 'RTE_FLOW' Can you give me a few API

[ANSWER] normally one updates with code snippets or steps used for reproducing the error. But the current question is more like clarification. Please refer above.

It sounds to me like everything is working as designed. I believe your expectations are being thwarted because you're using pub/sub (i.e. JMS topics).

Let me provide a bit of background. When a JMS client creates a subscription on a topic the broker responds by creating a multicast queue on the address with the same name. The queue is named according to the kind of subscription it is. If it is a non-durable subscription then the queue is named with a UUID. If it is a durable subscription then the queue is named according to the subscription name provided by the client and the client ID (if available). When a message is sent to the address it is put in all the multicast queues bound to that address.

Therefore, when a new non-durable subscription is created a new queue for that subscription is also created which means that the subscriber will receive none of the messages sent to the topic prior to the creation of the subscription. This is the expected behavior for JMS topics (i.e. normal pub/sub semantics). Also, since the queue for a non-durable subscription is only available while the subscriber is connected that means there's no way to enforce LVQ semantics since any message which arrives in the queue will be immediately dispatched to the consumer. In short, LVQ with JMS topics doesn't make a lot of sense.

The behavior changes when you use a JMS queue because the queue is always there to receive messages. Consumers can come and go as they please while the broker enforces LVQ semantics.

One possible solution would be to create a special "initialization" queue where consumers could initially connect to get the latest information and after that they could subscribe to the JMS topic to get the pub/sub semantics you need. You could use a divert to make this transparent for the applications sending the messages so they can continue to just send to the JMS topic. Here's sample configuration:

A headless service will return all Pod IPs that are associated through the selector. The order is not stable, so if a client is making repeated DNS queries and uses only the first returned IP, this will result in some kind of load balancing as well.

Regarding your second question: That is correct. In general, if a client does not need to know all instances - and handle the unstable IPs - a regular service provides more benefits.

This is due to usual traffic throttling from Event Hubs, take a look at the limits for 1 TU https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-quotas, you can increase the number of TUs to 2 and then go from there. If you think this is unexpected throttling then open a support ticket for the issue.

I think I know what the issue is here - you don't have any akka.cluster.roles defined nor is your /commander router configured with the use-role setting - so as a result, every Nth message is being dropped because it's trying to route a message to itself and does not have a /user/cluster actor present to receive it.

To fix this properly, we should do the following:

1. Have all nodes that can process the PrimeCalculationEntry declare akka.cluster.roles=[prime]
2. Have the node with the /commander router change its HOCON to:

if you don't want to use slice then I think something like this will work:- (playground)

Final SQL with the schema (and expected result) given in the question:

The final fiddle

So the key here is network alias.

Let me share a view on how ZeroMQ could meet the above defined Intention.

Let's rather use ZeroMQ Scalable Formal Communication Pattern Archetypes ( as they are RTO now, not as we may wish them to be at some, yet unsure, point in (a just potentially happening) future evolution state ).

We need not hesitate to use many more ZeroMQ-based connections among a herd of coming/leaving client-instance(s) and the server

For example :

Client .connect()-s a REQ-socket to Server-address:port-A to ask for a "job"-ticket processing over this connection

Client .connect()-s a SUB-socket to Server-address:port-B to listen ( if present ) about published announcements about already completed "job"-tickets that are Server-ready to deliver results for

Client exposes another REQ-socket to request upon an already broadcast "job"-ticket completion announcement message, it has just heard about over the SUB-socket, to get "job"-ticket results finally delivered, if proving itself, by providing a proper / matching job-ticket-AUTH-key to proof its right to receive the publicly announced results' availability, using this same socket to deliver a POSACK-message to Server upon client has correctly received this "job"-ticket results "in hands"

Server exposes REP-socket to respond each client ad-hoc upon a "job"-ticket request, notifying this way about having "accepted"-job-ticket, delivering also a job-ticket-AUTH-key for later pickup of results

Server exposes PUB-socket to announce any and all not yet picked-up "finished"-job-tickets

Server exposes another REP-socket to receive any possible attempt to request to deliver "job"-ticket-results. Upon verifying there delivered job-ticket-AUTH-key, Server decides whether the respective REQ-message had matching job-ticket-AUTH-key to indeed deliver a proper message with results, or whether a match did not happen, in which case a message will carry some other payload data ( logic is left for further thoughts, so as to prevent potential bruteforcing or eavesdropping and similar, less primitive attacks on stealing the results )

Clients need not stay waiting for results live/online and/or can survive certain amounts of LoS, L2/L3-errors or network-storm stresses

Clients need just to keep some kind of job-ticket-ID and job-ticket-AUTH-key for later retrieving of the Server-processes/maintained/auth-ed results

Server will keep listening for new jobs

Server will accept new job-tickets with providing a privately added job-ticket-AUTH-key

Server will process job-tickets as it will to do so

Server will maintain a circular-buffer of completed job-tickets to be announced

Server will announce, in due time and repeated as decided in public, job-tickets, that are ready for client-initiated retrieval

Server will accept new retrieval requests

Server will verify client-requests for matching any announced job-ticket-ID and testing if job-ticket-AUTH-key match either

Server will respond to either matching / non-matching job-ticket-ID results retrieval request(s)

Server will remove a job-ticket-ID from a circular-buffer only upon both POSACK-ed AUTH-match before a retrieval and a POSACK-message re-confirmed delivery to client