SharedMemory | shared memory classes for sharing data

Hello I made small 'client-server' file transfer program in Linux. But something strange is happening. If I launch first client and then server everything work fine. shmget() giving same id for provided key. Unfortunately if I launch first server and then client I am getting different id in client and I cannot copy data. I am using IPC_CREAT flag so it should join to shared memory.

client

I did a python script that gets data from shared mem and convert it from bytes to floats. The main problem is that it very slow.

This is how I init the shared memory:

I'm currently writing an image processing program in Python 3.x that needs to process frames in real-time (30 FPS) with low-latency (<60ms). I have 1 parent process that reads frames and sends them to multiple child processes via a SharedMemory object. The computations done by the child processes are CPU bound and running all of them on a single core is not possible at 30 FPS. But since they work independently of each other, I decided to run them as separate processes.

Currently, I'm using Pipes to send commands to the child processes, most importantly to inform them whenever the frame is updated. On measuring the time between the send() command of the parent and the recv() command on the child, the latency is always >100ms. I used time.time_ns() for this.

This is a problem because the output feed will now always be lagging by >100ms + time taken by all the children to finish processing (another 20-30ms + the delays between all the send() functions).

The application is meant to be used on a live sports feed and therefore cannot introduce such a high latency. So I have exactly 2 questions:

1. Are Pipes actually that slow in Python? Or is something wrong with my implementation of them. (Note: I have tested the latency on an Intel i5 9th Gen as well as an Apple M1)

2. If Pipes indeed are this slow, do I have any other options in Python? Other than resorting to some form of sockets?

Thanks.

Edit:

I've added the code I've used to test the Pipe latency here.

I'm trying to load List[np.ndarray] into shared_memory such that other processes can directly access this shared_memory and recover the original List[np.ndarray] without copying List[np.ndarray] into every process. The detailed motivation is related to my previous question: share read-only generic complex python object with int, list of numpy array, tuple, etc. as instance field between multiprocessing

I wrote the following code(python version: 3.8.12, Numpy:1.20.3, MacOS):

encode_nd_arr_list(): given List[np.ndarray], I can get List of share_memory name. decode_nd_arr_list(): given List of share_memory name, I can recover original List[np.ndarray].

I'm initializing a SharedMemory in python to be shared between multiple processes and I've noticed that it always seems to be filled with zeros (which is fine), but I don't understand why this is occurring as the documentation doesn't state there is a default value to fill the memory with.

This is my test code, opened in two seperate power shells, shell 1:

I have some prolem about SharedMemory in python3.8,any help will be good.

SharedMemory has one parameter SIZE,the doc tell me the unit is byte.I created a instance of 1 byte size,then, let shm.buf=bytearray[1,2,3,4], it can work and no any exception!why?

why print buffer is a memory address?

why i set size is 1byte,but result show it allocate 4096byte?

why buffer address and buffer[3:4] address is 3X16X16byte far away?

why buffer[3:4] address same as buffer[1:3] address?

In a cross platform (Linux and windows) real-time application, I need the fastest way to share data between a C++ process and a python application that I both manage. I currently use sockets but it's too slow when using high-bandwith data (4K images at 30 fps).

I would ultimately want to use the multiprocessing shared memory but my first tries suggest it does not work. I create the shared memory in C++ using Boost.Interprocess and try to read it in python like this:

I've come into a problem recently where I had two separate processes that need to share two strings. (A dynamic IP address and a key) I'm used to using ROS for this, where I would define a ROS msg with the two strings and send it from one to the other.

However we are trying to go as simple as possible with our application here and thus avoid using third party SW as much as we can. To do so I originally planned to use shared memory to send a struct holding both std::string only to realize it is not a trivial problem as this struct's size is dynamic...

I've also thought of using other means like sockets or queues, but I always run into the problem of not knowing beforehand the size of this struct. How can one deal with this problem? Is there a way to do so that doesn't involve defining some protocol where you prepend the string with its size and end it with a null, or similar?

Here is a snip of my code that uses Qt to create a SharedMemory to pass this struct (unsuccessfully of course).

I would like to create an instance of multiprocessing.shared_memory.SharedMemory passing from outside the buffer to use to hold the data.

My use case is the following: