gil | git links tool to manage complex recursive repositories | Version Control System library

Start by reading the docs for Executor.shutdown(), which is called by magic with wait=True when the with block ends.

For the same reason, if you run this trivial program you'll see that you get no useful parallelism either:

I did a project like this recently; a couple of things I would suggest to make it easy:

1. Confirm that the instance of python set in your env variables (WIN+R, sysdm.cpl, Advanced, env variables) is that of the instance of python you wish to use (do this for your python search path too!)

2. Remove any lines attempting to set these in code; and instead handle errors if they are not found

Then, when you call you script from within your program; it only needs to look like this:

You should apply your condition to the series argument of plot(). The display argument is whether to enable or disable the plot by default and I believe it must be a constant.

And you probably want to change the style to something like plot.style_circles so your line won't be connected.

okay so python.net installation is really not well documented and the folks maintaining the python.net repository don't really help a lot since it's not a "support forum".

I solved this issue by installing the python.runtime.AllPlatflorms nuget package and pointing the environment variables to the right python folders/ files.

This works with python3.8 as well.

Unfortunately, you mainly listed all the current available options. Numba functions can be cached and the signature can be specified so to perform an eager compilation (compilation at the time of the function definition) instead of a lazy one (compilation during the first execution). Note that the cache=True flag is only meant to skip the compilation when it as already been done on the same platform before and not to share the code between multiple machine. AFAIK, the internal JIT used by Numba (LLVM-Lite) does not support that. In fact, it is exactly the purpose of the AOT compilation to do that. That being said, the AOT compilation requires the signatures to be provided (this is mandatory whatever the approach/tool used as long as the function is compiled) and it has quite strong limitations (eg. currently there is no support for parallel codes and fastmath). Keep in mind that Numba’s main use case is Just-in-Time compilation and not the ahead-of-time compilation.

Regarding your use-case, using Cython appears to make much more sense: the functions are pre-compiled once for some generic platforms and the compiled binaries can directly be provided to users without the need for recompilation on the target machine.

I don't care if the code is a little slower on a PC that doesn't have my specific processor.

Well, regarding your code, using a "generic" x86-64 code can be much slower. The main reasons lie in the use of SIMD instructions. Indeed, x86-64 processors all supports the SSE2 instruction set which provide basic 128-bit SIMD registers working on integers and floating-point numbers. Since about a decade, x86-processors supports the 256-bit AVX instruction set which significantly speed up floating-point computations. Since at least 7 years, almost all mainstream x86-64 processors supports the AVX-2 instruction set which mainly speed up integer computations (although it also improves floating-point thanks to new features). Since nearly a decade, the FMA instruction set can speed up codes using fuse-multiply adds by a factor of 2. Recent Intel processors support the 512-bit AVX-512 instruction set which not only double the number of items that can be computed per instruction but also adds many useful features. In the end, SIMD-friendly codes can be up to an order of magnitude faster with the new instruction sets compared to the obsolete "generic" SSE2 instruction set. Compilers (eg. GCC, Clang, ICC) are meant to generate a portable code by default and thus they only use SSE2 by default. Note that Numpy already uses such "new" features to speed up a lot many functions (see sorts, argmin/argmax, log/exp, etc.).

Receiving raw pointers usually^* means you don't assume ownership of the object. When you receive IReader* in the constructor of C, pybind11 assumes you will still hold the temporary PklReader() object and keep it alive outside. But you don't, so it gets freed and you get a segfault.

I think

C++ code compiles into machine code. This makes it faster compared to interpreter languages (however not every code written in C++ is faster than Python code if you don't know what you are doing). in C++ you can access the data pointers directly and use SIMD instructions on them to make them multiple times faster. You can also multi-thread your loops and codes to make them run even faster (either explicit multi-threading or tools like OpenMP). You can't do these things (at least properly) in a high level language).

Not every part of the code is worth optimizing. You should only optimize the parts that are computationally expensive and are a bottleneck of your program. These parts can be written in C or C++ and exposed to python by using bindings (by using pybind11 for example). Big machine learning libraries like PyTorch and TensorFlow do this.

Sometimes having a well optimized C++ CPU code is not enough. Then you can assess your problem and if it is suitable, you can use dedicated hardware. These hardware can go from low-level (like FPGA) to high-level hardware like dedicated graphics cards we usually have on our system (like CUDA programming for NVIDIA GPUs).

Using a language that compiles has great advantages even if you don't use multi-threading or SIMD operations. For example, looping over a C array or std::vector in C++ can be more than 100x faster compared to looping over Python arrays or using for in MATLAB (recently JIT compiling is being used to speed up high-level languages but still, the difference exists). This has many reasons, some of which are basic data types that are recognized at compile time and having contiguous arrays. This is why people recommend using Numpy vectorized operations over simple Python loops (same is recommended for MATLAB).

An answer from a core developer:

Unintended consequence of Mark Shannon's change that refactors fast opcode dispatching: https://github.com/python/cpython/commit/4958f5d69dd2bf86866c43491caf72f774ddec97 -- the INPLACE_ADD opcode no longer uses the "slow" dispatch path that checks for interrupts and such.

Note: The following only applies to HDDs, which have moving parts that can affect read throughput, not SDDs. The nature of the large performance difference makes it clear to me that this is a HDD-oriented problem, so this information operates under that assumption.

The problem is that while the threads may be operating in parallel, the data must be read sequentially from the hard drive as there is only the singular read head. Worse, however, is that since you've parallelized the I/O operations, the underlying OS will schedule these I/O tasks such that these files are processed only partially before switching to another thread--after all, even if you only have a single integer, the files headers still need to be processed as well--causing the read head to jump around much more wildly than in your strictly sequential code. All of this results in massively increased overhead compared to simply reading the entirety of each file in sequence, which doesn't require so many jumps.

This wouldn't be as much of a problem if, for instance, you had a single thread loading in large amounts of data from disk while a second thread performs some time-intensive processing of it, as that would allow the time-intensive processing to continue unblocked by the I/O operations. Your particular scenario is just a really, really bad case where you've given up a GIL bottleneck in exchange for a horrifically slow I/O bottleneck.

In short, you've understood correctly that I/O operations release the GIL, you just came to the wrong conclusion about parallelizing file reads.