xeon | module loader for bash scripts with require style | Script Programming library

"intersect" is less efficient than a composite index.

Have an index with these 4 columns in any order:

Saturating RAM is not as simple as is seems.

First of all, at first glance here is the apparent throughput we can compute from the provided numbers:

• Fill: 1 / 0.4263950000 = 2.34 GB/s (1 GB is read);
• Memcpy: 2 / 0.6350150000 = 3.15 GB/s (1 GB is read and 1 GB is written).

The thing is that the pages allocated by malloc are not mapped in physical memory on Linux systems. Indeed, malloc reserve some space in virtual memory, but the pages are only mapped in physical memory when a first touch is performed causing expensive page faults. AFAIK, the only way speed up this process is to use multiple cores or to prefill the buffers and reuse them later.

Additionally, due to architectural limitations (ie. latency), one core of a Xeon processor cannot saturate the RAM. Again, the only way to fix that is to use multiple cores.

If you try to use multiple core, then the result provided by the benchmark will be surprising since clock does not measure the wall-clock time but the CPU time (which is the sum of the time spent in all threads). You need to use another function. In C, you can use gettimeofday (which is not perfect as it is not monotonic) but certainly good-enough for your benchmark (related post: How can I measure CPU time and wall clock time on both Linux/Windows?). In C++, you should use std::steady_clock (which is monotonic as opposed to std::system_clock).

In addition, the write-allocate cache policy on x86-64 platform force cache lines to be read when they are written. This means that to write 1 GB, you actually need to read 1 GB! That being said, x86-64 processors provide non-temporal store instructions that does not cause this issue (assuming your array is aligned properly and big enough). Compilers can use that but GCC and Clang generally does not. memcpy is already optimized to use non-temporal stores on most machines. For more information, please read How do non temporal instructions work?.

Finally, you can parallelize the benchmark easily using OpenMP with simple #pragma omp parallel for directives on loops. Note that is also provide a user-friendly function for computing the wall-clock time correctly: omp_get_wtime. For the memcpy, the best is certainly to write a loop doing memcpy by (relatively big) chunks in parallel.

For more information about this subject, I advise you to read the great famous document: What Every Programmer Should Know About Memory. Since the document is a bit old, you can check the updating information about this here. The document also describe additional important things to understand why you may still not succeed saturate the RAM with the above information. One critical topic is NUMA.

It’s not a bug. Floating point arithmetic has rounding errors. For single arithmetic operations + - * / sqrt the results should be the same, but for floating-point functions you can’t really expect it.

In this case it seems the compiler itself produced the results at compile time. The processor you use is unlikely to make a difference. And we don’t know whether the new version is more or less precise than the old one.

Assuming that you are using an Intel® AVX-512 instruction-supported processor, try installing the Intel® Optimization for TensorFlow Wheel via PIP specifically build for AVX512. These packages are available as *.whl on the Intel® website for specific Python versions or can be installed using the following command for Python versions 3.7, 3.8, and 3.9 (Linux Only).

I changed save() to insert() - the speed has increased. Now all JSON (1107 lines) is imported in 40 seconds. Are there faster ways to load ready-made data from JSON into the database? What if there are 100 thousand lines or a million? Is it normal practice to wait a few hours?

The WebBrowser control in WPF internally uses a native ActiveX control that is kind of dated.

If you want a modern browser experience that supports the latest HTML and CSS in your app, you should switch to using the WebView2 control.

You are measuring the time it takes each worker to perform func() and observe performance decrease for a single process when going from 10 processes to 180 parallel processes.

This looks quite normal to me:

• Intel cores use hyper-threading so you actually have 96 cores (in more detail - a hyper-threaded core adds only 20-30% performance). It means that 168 of your processes need to share 84 hyper-threaded cores and 12 processes get full 12 cores.
• The CPU speed is determined by throttle temperature (https://en.wikipedia.org/wiki/Thermal_design_power) and of course there is so much more space when running 10 processes vs 180 processes
• Your tasks are obviously competing for memory. They make a total of over 5TB of memory allocations and you machine has much less than that. The last mile in garbage collecting is always the most expensive one - so if your garbage collectors are squeezed and competing for memory the performance is uneven with surprisingly longer garbage collection times.

Looking at this data I would recommend you to try:

-march native means -march $MY_HARDWARE. We have no idea what hardware you have. For you, that would be -march=skylake-avx512 (SkyLake SP) The results could be reproduced by specifying your hardware architecture explicitly.

It's quite possible that the errors will decrease with more modern instructions, specifically Fused-Multiply-and-Add (FMA). This is the operation a*b+c, but rounded once instead of twice. That saves one rounding error.

I've already left this code running for almost six hours and absolutely nothing happens.

A lot has been happening! Don't believe me? You ran ox.config(log_console=True), so look at your terminal and watch what's happening while it runs. You'll see a line like "2021-10-14 13:05:39 Requesting data within polygon from API in 1827 request(s)"... so you are making 1,827 requests to the Overpass server and the server is asking you to pause for rate limiting between many of those requests.

I know that due to the stipulated area the time is long, but what I wanted to know is if there is an alternative to this procedure or if there is a way to optimize so that the creation of the map is a little faster or if there is another way to load maps to route with osmnx and networkx without using queries to servers

Yes. This answer provides more details. There are tradeoffs between 1) model precision vs 2) area size vs 3) memory/speed. For faster modeling, you can load the network data from a .osm XML file instead of having to make numerous calls to the Overpass API. I'd also recommend using a custom_filter as described in the linked answer. OSMnx by default divides your query area into 50km x 50km pieces, then queries Overpass for each piece one a time to not exceed the server's per-query memory limits. You can configure this max_query_area_size parameter, as well as the server memory allocation, if you prefer to use OSMnx's API querying functions rather than its from-file functionality.