benchmark | benchmark | Performance Testing library

When a divide-and-conquer recursive function doesn't yield runtimes low enough, which other improvements could be done?

Let's say, for example, this power function taken from here:

So I was really ripping my hair out why two different sessions of R with the same data were producing wildly different times to complete the same task. After a lot of restarting R, cleaning out all my variables, and really running a clean R, I found the issue: the new data structure provided by vroom and readr is, for some reason, super sluggish on my script. Of course the easiest thing to solve this is to convert your data into a tibble as soon as you load it in. Or is there some other explanation, like poor coding praxis in my functions that can explain the sluggish behavior? Or, is this a bug with recent updates of these packages? If so and if someone is more experienced with reporting bugs to tidyverse, then here is a repex showing the behavior cause I feel that this is out of my ballpark.

I'm learning that Golang channels are actually slower than many alternatives provided by the language. Of course, they are really easy to grasp but because they are a high level structure, they come with some overhead.

Reading some articles about it, I found someone benchmarking the channels here. He basically says that the channels can transfer 10 MB/s, which of course must be dependant on his hardware. He then says something that I haven't completely understood:

If you just want to move data quickly using channels then moving it 1 byte at a time is not sensible. What you really do with a channel is move ownership of the data, in which case the data rate can be effectively infinite, depending on the size of data block you transfer.

I've seen this "move ownership of data" in several places but I haven't seen a solid example illustrating how to do it instead of moving the data itself.

I wanted to see an example in order to understand this best practice.

Below is the code and the error that I'm getting while testing in Lambda. I'm a newbie in python & serverless. Please help. This is created for uploading the findings from the security hub to S3 for POC.

I am running a TPC-DS benchmark for Spark 3.0.1 in local mode and using sparkMeasure to get workload statistics. I have 16 total cores and SparkContext is available as

Spark context available as 'sc' (master = local[*], app id = local-1623251009819)

Q1. For local[*], driver and executors are created in a single JVM with 16 threads. Considering Spark's configuration which of the following will be true?

• 1 worker instance, 1 executor having 16 cores/threads
• 1 worker instance, 16 executors each having 1 core

For a particular query, sparkMeasure reports shuffle data as follows

shuffleRecordsRead => 183364403
shuffleTotalBlocksFetched => 52582
shuffleTotalBlocksFetched => 52582
shuffleLocalBlocksFetched => 52582
shuffleRemoteBlocksFetched => 0
shuffleTotalBytesRead => 1570948723 (1498.0 MB)
shuffleLocalBytesRead => 1570948723 (1498.0 MB)
shuffleRemoteBytesRead => 0 (0 Bytes)
shuffleRemoteBytesReadToDisk => 0 (0 Bytes)
shuffleBytesWritten => 1570948723 (1498.0 MB)
shuffleRecordsWritten => 183364480

Q2. Regardless of the query specifics, why is there data shuffling when everything is inside a single JVM?

In a heatmap, how could I create a three-color gradient, with blue for negative values, red for positive values and white for zero, such that with many zero values, much of the heatmap would be white (and not light red as with the default gradient).

I have a large DataFrame of distances that I want to classify.

I have two monotonic increasing vectors, v1 and v2 of unequal lengths. For each value in v1 (e.g., v1[1], v1[2], ...), I want to find the value in v2 that is just less than v1[i] and compute the difference.

My current code (see below) works correctly, but does not seem to scale up well. So I am looking for recommendations to improve my approach with the requirement of staying in R, or using a package I can call from R.

Example code:

I am in search of performance benchmarks for querying parquet ADLS files with the standard dedicated sql pool using external tables with polybase vs. serverless sql pool and OPENROWSET views. From my base queries on a 1.5 billion record table, it does appears OPENROWSET in serverless sql pool is around 30% more performant given time for the same query, but what are the architecture that power that? Are there any readily available performance benchmarks?