ipc-bench | : racehorse : Benchmarks for Inter-Process-Communication

I want to create an ultra fast message processing C++ solution which will be CPU bound and micro-services based. It will process lots of request/response messages that are small enough (32 bytes to 1kb per message).

Some services will be placed in different hosts. Some will be on the same host, but in different processes. And some inside the same process, but in different threads.

Currently I'm thinking about communication protocols for such services topology. My first idea was to use TCP-based communication which allows to use loopback for IPC communication on the same host and even for threading communication. The benefit is to have a single communication code which allows to experiment with services topology. It will be easy to host service inside some process or move it to the remote host.

However I understand that if I want to have a low-latency solution with max RPS and speed of message delivery I have to split transport depending on communication type:

• Threads communication - the best results could be achieved with circular buffer or LMAX Disruptor pattern.

• IPC communication - I think pipes or circular buffer in shared memory are good solutions. Is there a better way to do IPC?

• Remote communication - async TCP server/client for sequential message delivery and UDP for multicasting.

Also I'm thinking about Linux solution, but having cross-platform would be great!

I believe Asio C++ Library is a good starting point for remote communications.

My questions are the following:

1. Is it worth creating a custom IPC/threading communication solutions or should I start with TCP-based localhost communications?
2. Can anyone provide me with some performance comparison results of different IPC techniques (locahost tcp vs pipes vs shared memory) for best RPS of small messages up to 1kb? (E.g. shared memory will be 10-times faster than localhost TCP and 3-times faster than pipes or approximate RPS values to compare).
3. Maybe I missed some good low-lattency IPC/RPC technique or library that I should look into?
4. Or perhaps some production-ready solution exists for my problem that I can use or purchase the license?

Thanks in advance for your answers and suggestions!

I just found and performed low-level IPC benchmarks under Linux (Linux ubuntu 4.4.0 x86_64 i7-6700K 4.00GHz). Message size was 128 bytes and messages count is 1000000. I get following results:

Pipe benchmark: