sarama | Sarama is a Go library for Apache Kafka | Pub Sub library

NewConsumerGroup creates its own Consumer, so there's nothing for you to hook it to

Source: https://github.com/Shopify/sarama/blob/main/consumer_group.go#L105

Why doesn’t -call_tree affect this output of -tree

I think the -call_tree option doesn't change the output of -tree the outputs are not actually a tree, it outputs the nodes of the tree(more on this in the extra credit section).

In general, what does -call_tree do?

You can see a difference when you take a look at images generated with the -png flag, without the -call_tree flag: And with the -call_tree flag:

So instead of having 1 call tree, pprof attempts to create seperate trees based on context. In my case(will list them since the text on the image is not readable) the roots are:

• testing.(*B).launch (benchmark/test framework)
• runtime.gcBgMarkWorker (part of the runtime GC)
• runtime.bgsweep (part of the runtime GC)
• runtime.mcall (part of the runtime scheduler)
• runtime.morestack (something to do with the stack :) )

In the non -call_tree image these nodes are still present, but start mid tree, as if our code calls these background processed directly.

Basically basically what the option does is remove/hide infrequent calls between functions, so you will end up with a tree for each set of functions that call each other frequently.

I haven't tested this but I imagine that pprof will do this context aware tree separation for user code as well. All in all it returns a subjectively more readable tree, or at least a more relevant one.

What is the exact meaning of the output node I showed above?

The -tree option attempts to output the tree, as shows in the images. But since it is text output it shows you 1 node of the tree at a time, the non indented line in the context column is the current node, the rows above are nodes that call the current node, the rows below the nodes which this one calls (the arrows in the image).

The calls% is a the "weight" of the the incoming or outgoing edge, so indeed the percentage of calls from or to a function.

What documents could I read to answer the above questions?

If figured all of this out by looking at the source code, here are some key parts, in case you are interested:

• The file which does most of the output generation: https://github.com/google/pprof/blob/2007db6d4f53c44a417ddae675d50f56b8e8c2fd/internal/report/report.go
• The function for the -tree option: https://github.com/google/pprof/blob/2007db6d4f53c44a417ddae675d50f56b8e8c2fd/internal/report/report.go#L1047
• Line which explains when -call_tree is actually used: https://github.com/google/pprof/blob/2007db6d4f53c44a417ddae675d50f56b8e8c2fd/internal/report/report.go#L133

ioutil.Discard and ioutil.ReadAll has moved to io.Discard and io.ReadAll respectively as of Go 1.16, You should use Go 1.16 or use an older version of sarama (I think v1.20.1 should work for go1.13)

Also from sarama's README:

Sarama provides a "2 releases + 2 months" compatibility guarantee: we support the two latest stable releases of Kafka and Go, and we provide a two month grace period for older releases. This means we currently officially support Go 1.15 through 1.16, and Kafka 2.7 through 2.8, although older releases are still likely to work.

Did you see the section of the docs that serialized the event?

https://github.com/cloudevents/sdk-go#serializedeserialize-a-cloudevent

So it turns out that the problem was mainly a command line issue. I kept trying to use the -ca flag when I should have just used only the -certificate flag. I also needed to add the -verify option flag as well. So the command that allowed me to produce was using the following -

If you want to prune all the messages, another way to do that is to reduce the retention of the topic to a small value (e.g. 100ms). Wait for the brokers to remove all the records from the topic and then set the topic retention to its original value. Here’s how to do it.

First, set the retention time to 100 milliseconds.

You need to change ReaderConfig.MinBytes, otherwise segmentio/kafka-go will set it to 1e6 = 1 MB, and in that case, Kafka will wait for that much data to accumulate before answering the request.

It turns out I just needed to call

Your loop fetching the results may end early because you are using the same ctx context for iterating over the results which has a 10 seconds timeout.

Which means if retrieving and processing the 2 million records (including connecting) takes more than 10 seconds, the context will be cancelled and thus the cursor will also report an error.

Note that setting FindOptions.NoCursorTimeout to true is only to prevent cursor timeout for inactivity, it does not override the used context's timeout.

Use another context for executing the query and iterating over the results, one that does not have a timeout, e.g. context.Background().

Also note that for constructing the options for find, use the helper methods, so it may look as simple and as elegant as this: