shared-array | SharedArray python/numpy extension | Machine Learning library

Here is a code after cleanup that works. Basically, the main problem is how the @distributed macro was trying to move the Python module around the cluster (it seems it does not know it is a library). So I packed it into a function which is always called locally at each given worker process (no risk of copying).

I assume that all your workers are on a single server and that you have actually added some workers using the addprocs command. The first problem with your code is that you create the SharedArray on all workers. Rather than that the syntax of a SharedArray is the following:

Per code, main.js represents Main process of Electron and filaA.js is for Renderer process. Since those 2 are different process, there is no way to share same object reference across processes: you should use IPC to ask one process's value if you want to achieve singleton across process.

SharedArrays works only within a single cluster node. In other words this is used to share RAM memory between processes running on the same server. When you add another server obviously you will not see that memory.

What you should do is to use DistributedArrays.jl instead:

There are the following problems in your code:

• You are spawning a remote task for each value of i a and this is just expensive and in the end it takes long. Basically the rule of thumb is to use @distributed macro for your load balancing across workers this will just evenly share the work.
• Never put addprocs inside your work function because every time you run it, every time you add new processes - spawning a new Julia process also takes lots of time and this was included in your measurements. In practice this means you want to run addprocs at some part of the script that performs the initialization or perhaps the processes are added via starting the julia process with -p or --machine-file parameter
• Finally, always run @time always twice - in the first measurement @time is also measuring compilation times and the compilation in a distributed environment takes much longer than in a single process.

Your function should look more or less like this

Note that your loops do very different things.

1. Int the first loop each thread keeps updating the same single cell the Array. Most likely since only a single memory cell is update in a single thread, the processor caching mechanism can be used to speed up things. On the other hand the second loop each process is updating several different memory cells and such caching is not possible.
2. The first Array holds Float64 values while the second holds Int64 values

After correcting those things the difference gets smaller (this is on my laptop, I have only 8 threads):

Update the package:

Can you benefit from threaded for instead for @distributed for? This works (Julia 1.4):

Did you try to call collectionView.reloadData() after changing the data arrays?