multicore | Parallel processing of R code | GPU library

I think you are missing the point of the fundamental locking primitives like Semaphores. If correct primitive is used, and used correctly, then the timing of the threads do not matter. They may well be simultaneous. The Operating System guarantees that no two thread will enter the critical section. Even on multicore machines, this bit is specially implemented (with lots of trickery even) to get that assurance.

To address your concerns specifically:

But that implies that the threads didn't really reach the CS exactly at the same microsecond.

No. The other threads could have reached in the same microsecond, BUT if the locking mechanism is correct, then only one the competing threads will "enter" the critical section and others will wait.

Although I guess it is quite rare, can it really happen, and what happens in this situation?

Rare or not, if the correct locking primitive is used, and used correctly, then no two threads will enter the critical section.

Can I simply assume the the program will malfunction?

Ideally the program should not malfunction. But any code will have bugs - so does your code and the Operating System code for the Semaphores. So it is safe to assume that in some edge cases the program will indeed malfunction. But this assumption is true for any code in general.

Locking and Critical Sections are rather tricky to correctly implement. So for non academic purposes we should always use the system provided locking primitives. All Operating Systems expose stuff like Semaphores which most programming languages have ways to use. Some programming languages have their own lightweight implementations which provide somewhat softer guarantees but at a higher performance. As I said, while doing Critical Sections, it is critical to choose the correct thing and also to implement it correctly.

ACCURACY is a 1 row matrix:

As you wished, VS IDE does support multi-core build process.

VS first will get a basic performance evaluation based on your current CPU hardware. Then, you should open the switch under Tools-->Options-->Projects and Solutions-->Build and Run-->and you will see maximum number of parallel project builds.

Set the number of build process based on your CPU performance.

Obviously, it is better to use multi-core build.

Note: value 1 means single-core and you should expand the value to enable multi-core build.

If you build c++ projects, there is another second option under Tools-->Options-->Projects and Solutions-->VC++ Project Settings-->Maximum concurrent c++ compilation.

In this suitation, value 0 means all CPU will be used.

This is a rewrite of the original post, that gives a better explanation of what went wrong, and fixes a possible bug in the workaround.

That looks like a bug in censboot. It doesn't handle the ... parameter correctly. (More explanation below.) The reason you don't get an error with parallel = 'no' is that the code follows a different path.

A workaround is to use "partial application" to create a 1-parameter statistic function, like this:

The basic problem is that some instructions read memory, modify the value read, then write a new value; and if the contents of memory change between the read and the write, (some) parallel code can end up in an inconsistent state.

A nice example is one CPU doing inc dword [foo] while another CPU does dec dword [foo]. After both instructions (on both CPUs) are executed the value should be the same as it originally was; but both CPUs could read the old value, then both CPUs could modify it, then both CPUs could write their new value; resulting in the value being 1 higher or 1 lower than you'd expect.

The solution was to use a #lock signal to prevent anything else from accessing the same piece of memory at the same time. E.g. the first CPU would assert #lock then do it's read/modify/write, then de-assert #lock; and anything else would see that the #lock is asserted and have to wait until the #lock is de-asserted before it can do any memory access. In other words, it's a simple form of mutual exclusion (like a spinlock, but in hardware).

Of course "everything else has to wait" has a performance cost; so it's mostly only done when explicitly requested by software (e.g. lock inc dword [foo] and not inc dword [foo]) but there are a few cases where it's done implicitly - xchg instruction when an operand uses memory, and updates to dirty/accessed/busy flags in some of the tables the CPU uses (for paging, and GDT/LDT/IDT entries). Also; later (Pentium Pro I think?), the behavior was optimized to work with cache coherency protocol so that the #lock isn't asserted if the cache line can be put in the exclusive state instead.

Note: In the past there have been 2 CPU bugs (Intel Pentium "0xF00F" bug and Cyrix "Coma" bug) where a CPU can be tricked into asserting the #lock signal and never de-asserting it; causing the entire system to lock up because nothing can access any memory.

1. Does this mean that any other computer device like GPU or something else can't have access to memory for example. Actually can other devices talk directly to RAM without passing first on the CPU.

Yes. If the #lock is asserted (which doesn't include cases where newer CPUs can put the cache line into the exclusive state instead); anything that accesses memory would have to wait for #lock to be de-asserted.

Note: Most modern devices can/do access memory directly (to transfer data to/from RAM without using the CPU to transfer data).

1. How does the processor know that it's in this locked state is it saved in a control or rflags register for example or what since I can't see how this operation works when having multicore CPU.

It's not saved in the contents of any register. It's literally an electronic signal on a bus or link. For an extremely over-simplified example; assume that the bus has 32 "address" wires, 32 "data" wires, plus a #lock wire; where "assert the #lock" means that the voltage on that #lock wire goes from 0 volts up to 3.3 volts. When anything wants to read or write memory (before attempting to change the voltages on the "address" wires or "data" wires) it checks the voltage on the #lock wire is 0 volts.

Note: A real bus is much more complicated and needs a few other signals (e.g. for direction of transfer, for collision avoidance, for "I/O port or physical memory", etc); and modern buses use serial lanes and not parallel wires; and modern systems use "point to point links" and not "common bus shared by all the things".

1. The websites I visited said lock any shared memory. does this mean that during this lock period the whole RAM is locked or just the memory page(or part of memory not all of it) that the instruction is performed on.

It's better to say that the bus is locked; where everything has to use the bus to access memory (and nothing else can use the bus when the bus is locked, even when something else is trying to use the bus for something that has nothing to do with memory - e.g. to send an IRQ to a CPU).

Of course (due to aggressive performance optimizations - primarily the "if the cache line can be put in the exclusive state instead" optimization) it's even better to say that the hardware can do anything it feels like as long as the result behaves as if there's a shared bus that was locked (even if there isn't a shared bus and nothing was actually locked).

Note: 80x86 supports misaligned accesses (e.g. you can lock inc dword [address] where the access can straddle a boundary), where if a memory access does straddle a boundary the CPU needs to combine 2 or more pieces (e.g. a few bytes from the end of one cache line and a few bytes from the start of the next cache line). Modern virtual memory means that if the virtual address straddles a page boundary the CPU needs to access 2 different virtual pages which may have "extremely unrelated" physical addresses. If a theoretical CPU tried to implement independent locks (a different lock for each memory area) then it would also need to support asserting multiple lock signals. This can cause deadlocks - e.g. one CPU locks "memory page 1" then tries to lock "memory page 2" (and can't because it's locked); while another CPU locks "memory page 2" then tries to lock "memory page 1" (and can't because it's locked). To fix that the theoretical CPU would have to use "global lock ordering" - always assert locks in a specific order. The end result would be a significant amount of complexity (where it's likely that the added complexity costs more performance than it saves).

I think what you are looking for is combinations from itertools, a package from the standard library.

Issue was found by HPC service provider. For unknown reason OMP_PLACES=cores variable which should tie threads/processes to specific cores, appeared to bind all processes to a single core only when running multi-core R jobs. Issue has been solved by rebuilding r-environment singularity-container.

Ok it is working again, I needed to delete the device in amazon, and run the mos aws setup command again.

So the real problem was I was flashing my device each time I updated my code and it would overwrite my wifi settings, my aws settings and my mdash settings.

Thanks to help from a different source this solved it for me

As we found out it is different on windows and ubuntu. It is probably because of this:

spawn The parent process starts a fresh python interpreter process. The child process will only inherit those resources necessary to run the process objects run() method. In particular, unnecessary file descriptors and handles from the parent process will not be inherited. Starting a process using this method is rather slow compared to using fork or forkserver.

Available on Unix and Windows. The default on Windows and macOS.

fork The parent process uses os.fork() to fork the Python interpreter. The child process, when it begins, is effectively identical to the parent process. All resources of the parent are inherited by the child process. Note that safely forking a multithreaded process is problematic.

Available on Unix only. The default on Unix.

Try adding this line to your code: