pprof | Ruby Gem to list , filter , search and print Provisioning

Running this command:

Turns out the code in //do something with connection was more important than I thought. Even at the tls.Dial level you have to read off the "body". My, now obviously wrong assumption, was that tls.Dial just setup the connection and that since a GET / HTTP 1.1 request hadn't been sent yet no data needed to be read off the wire. This was causing all those buffers full of server response to sit around.

_, _= ioutil.ReadAll(tConn) fixed it all in one line. I feel much wiser and also dumb at the same time. As a side note, at this level, ReadAll() can hang for a long time if the server responds slowly. tConn.SetReadDeadline(time.Now().Add(time.Second * timeout)) solved that too.

You can use net/http/pprof for profiling.
fasthttp provides custom implementation for the same. You can use that same as the net/http/pprof.

To use this, register the handler as:

The solution I came along with is the following:

As you found, the default "randomness source" for the math/rand package is a locked source, suitable for use in concurrent goroutines. You will need to create one new source per goroutine using the NewSource function, and then one random number generator from that, using New. The pseudo-random numbers from this new generator will work the same way as the (single) locked source except that each generator will produce its own stream of identically-generated numbers if started from the same seed.

You'll therefore want to ensure that each seed you provide to NewSource is unique, so that each stream is different.

My best guess the problem is BIND_ON_IP=0.0.0.0. Each instance should use unique hostIP:Port as their address. Because it's all 0.0.0.0, every service will only work if running with one instance. Because more than instance will have conflict.

However, it's not a problem for frontend service because FE is stateles. Matching/History will run into this problem ---

HostA register it to mathcing service with 0.0.0.0:7935, and then HostB tries to do the same. This will cause the consistent hashing ring being unstable. The tasklist ownership keeps being switched between HostA and HostB.

To resolve this issue, you need to let each instance uses its own hostIP. Like in K8s uses PodIP.

After you resolve this issue, you will see in the logs in FE/history that they successfully connect to two Matching hosts:

net/http/pprof is a convenience package. It always registers handlers on DefaultServeMux, because DefaultServeMux is a global variable that it can actually do that with.

If you want to serve pprof results on some other ServeMux there's really nothing to it; all it takes is calling runtime/pprof.StartCPUProfile(w) with an http.ResponseWriter and then sleeping, or calling p.WriteTo(w, debug) on a runtime/pprof.Profile object. You can look at the source of net/http/pprof to see how it does it.

In a slightly better universe, net/http/pprof would have a RegisterHandlers(*http.ServeMux) function that could be used anywhere, you would be able to import it without anything being registered implicitly, and there would be another package (say net/http/pprof/sugar) that did nothing except call pprof.RegisterHandlers(http.DefaultServeMux) in its init. However, we don't live in that universe.

Ok, I believe that I've figured it out. It looks like Go allocates eagerly and the discrepancy is just due to the way the Go memory profiler takes samples.

Go allocates channel memory eagerly

The docs for make promise that

The channel's buffer is initialized with the specified buffer capacity.

I looked into the code for makechan, which gets called during make(chan chantype, size). It always calls mallocgc directly - no laziness.

Looking into the code for mallocgc, we can confirm that there's no laziness within mallocgc (besides the doc comment not mentioning laziness, mallocgc calls c.alloc directly).

pprof samples at the heap allocation level, not the calling function level

While looking around mallocgc, I found the profiling code. Within each mallocgc call, Go will check to see if its sampling condition is met. If so, it calls mProf_Malloc to add a record to the heap profile. I couldn't confirm that this is the profile used by pprof, but comments in that file suggest that it is.

The sampling condition is based on the number of bytes allocated since the previous sample was taken (it draws from an exponential distribution to sample, on average, after every runtime.MemProfileRate bytes are allocated).

The important part here is that each call to mallocgc has some probability of being sampled, rather than each call to foo. This means that if a call to foo makes multiple calls to mallocgc, we expect that only some of the mallocgc calls will be sampled.

Putting it all together

Every time my function foo is run, it will eagerly allocate memory for the 4 channels. At each memory allocation call, there is a chance that the Go will record a heap profile. On average, Go will record a heap profile every 512kB (the default value of runtime.MemProfileRate). Since the total size of these channels is 488kB, on average we expect only one allocation to be recorded each time foo is called. The profile I shared above was taken relatively soon after the service restarted, so the difference in number of allocated bytes is the result of pure statistical variance. After letting the service run for a day, the profile settled down to show that the number of bytes allocated by lines 142 and 146 were equal.

The operation you want to perform is called a positional population count on bytes. This is a well-known operation used in machine learning and some research has been done on fast algorithms to solve this problem.

Unfortunately, the implementation of these algorithms is fairly involved. For this reason, I have developed a custom algorithm that is much simpler to implement but only yields roughly half the performance of the other other method. However, at measured 10 GB/s, it should still be a decent improvement over what you had previously.

The idea of this algorithm is to gather corresponding bits from groups of 32 bytes using vpmovmskb and then to take a scalar population count which is then added to the corresponding counter. This allows the dependency chains to be short and a consistent IPC of 3 to be reached.

Note that compared to your algorithm, my code flips the order of bits around. You can change this by editing which counts array elements the assembly code accesses if you want. However, in the interest of future readers, I'd like to leave this code with the more common convention where the least significant bit is considered bit 0.

The complete source code can be found on github. The author has meanwhile developed this algorithm idea into a portable library that can be used like this: