latency | Measure network round-trip latency | Networking library

Looking at the result of Cachegrind, it doesn't look like the memory is your bottleneck. The NN has to be stored in memory anyway, but if it's too large that your program's having a lot of L1 cache misses, then it's worth thinking to try to minimize L1 misses, but 1.7% of L1 (data) miss rate is not a problem.

So you're trying to make this run fast anyway. Looking at your code, what's happening at the most inner loop is very simple (load-> multiply -> add -> store), and it doesn't have any side effect other than the final store. This kind of code is easily parallelizable, for example, by multithreading or vectorizing. I think you'll know how to make this run in multiple threads seeing that you can write code with some complexity, and you asked in comments how to manually vectorize the code.

I will explain that part, but one thing to bear in mind is that once you choose to manually vectorize the code, it will often be tied to certain CPU architectures. Let's not consider non-AMD64 compatible CPUs like ARM. Still, you have the option of MMX, SSE, AVX, and AVX512 to choose as an extension for vectorized computation, and each extension has multiple versions. If you want maximum portability, SSE2 is a reasonable choice. SSE2 appeared with Pentium 4, and it supports 128-bit vectors. For this post I'll use AVX2, which supports 128-bit and 256-bit vectors. It runs fine on your CPU, and has reasonable portability these days, supported from Haswell (2013) and Excavator (2015).

The pattern you're using in the inner loop is called FMA (fused multiply and add). AVX2 has an instruction for this. Have a look at this function and the compiled output.

The throughput is reciprocal throughput if running a large block of just that instruction. (Or with dependency-breaking instructions for cases like adc or div where you can't make back-to-back executions not have a data dependency because of implicit register inputs/outputs, especially FLAGS). So 0.5 means it can run once per 0.5 cycles, i.e. 2/clock, as expected for CPUs that we know have 2 load ports.

Why are there sometimes two numbers for latency, e.g. [≤10;≤11]?

See also What do multiple values or ranges means as the latency for a single instruction? which used a load+ALU ALU instruction as an example. (I forgot how close a duplicate that was, not looking for it until I'd written the rest of this answer.)

Usually that indicates that latencies from different inputs to the output(s) can be different. e.g. a merge-masking load has to merge into the destination so that's one input, and the load address is another input (via integer registers). The recently-stored data in memory is a 3rd input (store-forwarding latency).

For cases like vector load-use latency, where the load result is in a different domain than the address registers, uops.info creates a dependency chain with an instruction sequence involving movd or vmovq rax, xmm0 to couple the load result back into the address for another load. It's hard to separately establish latencies for each part of that, so IIRC they assume that each other instruction in the chain is at least 1 cycle, and show the latency for the instruction under test as <= N, where N + rest of dep chain adds up to the total cycles per iteration of the test code.

Look at the details page for one of those results, showing the test sequence used to measure it. Every number in the table is also a link. Those details pages tell you which operand is which, and break down the latencies from each input to each output. Let's look at a zero-masked vmovdqa64 512-bit load (VMOVDQA64_Z (ZMM, K, M512)) which in asm they tested using vmovdqa64 zmm0{k1}{z},ZMMWORD PTR [r14]. The listed latency is [1;≤9].

They number the operands as

• 1 (write-only): the ZMM destination.
• 2 (read-only): the k0..7 mask register
• 3 (read-only): memory (later broken down into address vs. actual memory contents)

The 1 cycle latency part is latency from mask register to result, "Latency operand 2 → 1: 1". So the mask doesn't have to be ready until the load unit has fetched the data.

The <=9 is the latency from address base or index register to final ZMM result being ready.

Apparently with a store/reload case, bottlenecked on store-forwarding latency, "Latency operand 3 → 1 (memory): ≤6". They tested that with this sequence, described as "Chain latency: ≥6". vshufpd zmm is known to have 1 cycle latency, and I guess they're just counting the store as having 1 cycle latency? Like I said, they just assume everything is 1 cycle, even though it's kind of fishy to assign any latency at all to a store.

(Writing an answer instead of deleting, in case it is of use to others)

So turns out that since December 2020 Clang does support __attribute__((hot)), they just didn't document it. I nudged them. When tested in a newer clang version I see a difference in binaries.

BTW they did support __attribute__((cold)) long before that. And as I suspected, the clang front end did accept hot for sake of gcc compatibility.

Dndata frames can only have certain object classes as column types. A htest is not one of those. However, we can store lists as list-columns. If we adapt the current code to output lists htests as results, we can later extract elements of the tests separately.

Are you using an N1 or a non-N1 machine type? If you want to autoscale to zero, you must use non-N1 machines. See second note from node allocation:

Note: Versions that use a Compute Engine (N1) machine type cannot scale down to zero nodes. They can scale down to 1 node, at minimum.

Update: AI Platform supports scaling to zero, while Vertex AI currently does not. From the scaling documentation, nodes can scale but there is no mention that it can scale down to zero. Here's a public feature request for people who wants to track this issue.

With regards to latency requirements, the actual output will vary. However, one thing to note according to the documentation is that the service may not be able to bring nodes online fast enough to keep up with large spikes of request traffic. If your traffic regularly has steep spikes, and if reliably low latency is important to your application, you may want to consider manual scaling.

Additional Reference: https://cloud.google.com/ai-platform/prediction/docs/machine-types-online-prediction#automatic_scaling

An async function will return a Promise. Calling await myAsyncFunction(), will return the value of the Promise once the Promise is fulfilled. Note the return in the previous statement. It does not change the pointer in place.

E.g. calling await data does not change data.

You can get the output you expect by calling data = await data.

I cannot reproduce your issue using:

1. openjdk:8-jre-alpine docker image
2. JMeter 5.4.1
3. Test plan Test.jmx from extras folder of JMeter

Demo:

If you cannot reproduce the above behaviour I think you made some changes either to Results File Configuration or to Reporting Configuration or both so you need to inspect all the JMeter Properties which differ from the defaults and restore their values to the original ones.

If you need further support you need to share at least first 2 lines of your outFilePath.jtl results file. Better if possible the full file and all the .properties files from JMeter's "bin" folder.

After a deep dive in the gRPC source code, the following seems to work:

Turns out that Windows 10 Update Assistant incorrectly reported it upgraded my OS to 21H2 on my laptop. Checking Windows version by running winver reports that my OS is still 21H1. Of course CUDA in WSL2 will not work in Windows 10 without 21H2.

After successfully installing 21H2 I can confirm CUDA works with WSL2 even for laptops with Optimus NVIDIA cards.