gib | Portable Solution for Developing Science Gateways

I think I also experienced this issue, although on macOS 12.1. I tried to debug it and found out that sometimes I also get another error:

Although ISO C++ does not support variable-length arrays, you seem to be using a compiler which supports them as an extension.

In the line

It is possible to do it in place like this:

In my case, since I had set a timeout of 1,500 milliseconds on the future and because of the slower CPU clock speed of the misbehaving machine, the class was not fully initialised when the timeout occurred. Turns out, OkHttp is more or less the culprit, it takes more than 5 seconds to inistalise the client on the given machine.

All in all, I am no longer applying any timeout on the first try of the connection test to give OkHttp enough time to initialise itself.

Note that this would not solve the problem if the initialisation of the HTTPClient was to fail at a different point in the application lifecycle. But since the first try of the connection test is the first place to call into HTTPClient, this is the only place where it can be initialised.

This is more of a comment, but worth pointing out.

The reason in general is indeed what talonmies commented, but you are summing up the numbers incorrectly. Let's see what happens when tensors are moved to GPU (I tried this on my PC with RTX2060 with 5.8G usable GPU memory in total):

Let's run the following python commands interactively:

I have no ways to test the code on your data, but here is one idea.

The idea is to conduct the wide to long transformation for a chunk of rows one at a time, store the outcome in a list. In the end, combine the list to the final data frame. Hopefully this reduces the memory usage.

If not working, try to see if melt from data.table can convert the data more efficiently.

One other idea that could be helpful. Perhaps subset the Df by removing column 1 to 16 before the wide to long transformation, just keep an ID column. You can join column 1 to 16 back to the converted data frame later.

You are measuring the time it takes each worker to perform func() and observe performance decrease for a single process when going from 10 processes to 180 parallel processes.

This looks quite normal to me:

• Intel cores use hyper-threading so you actually have 96 cores (in more detail - a hyper-threaded core adds only 20-30% performance). It means that 168 of your processes need to share 84 hyper-threaded cores and 12 processes get full 12 cores.
• The CPU speed is determined by throttle temperature (https://en.wikipedia.org/wiki/Thermal_design_power) and of course there is so much more space when running 10 processes vs 180 processes
• Your tasks are obviously competing for memory. They make a total of over 5TB of memory allocations and you machine has much less than that. The last mile in garbage collecting is always the most expensive one - so if your garbage collectors are squeezed and competing for memory the performance is uneven with surprisingly longer garbage collection times.

Looking at this data I would recommend you to try:

Learned a couple things here. The first is that 143 KILLED does not seem to actually be indicative of failure but rather of executors receiving a signal to shutdown once the job is finished. So, seems draconian when found in logs but is not.

What was confusing me was that I wasn't seeing any "Pi is roughly 3.1475357376786883" text on stdout/stderr. This led me to believe the computation never got that far, which was incorrect.

The issue here is what I was using --deploy-mode cluster when --deploy-mode client actually made a lot more sense in this situation. That is because I was running an ad-hoc container through kubectl run which was not part of the existing deployment. This fits the definition of client mode better, since the submission does not come from an existing Spark worker. When running in --deploy-mode=cluster, you'll never actually see stdout since input/output of the application are not attached to the console.

Once I changed --deploy-mode to client, I also needed to add --conf spark.driver.host as documented here and here, for the pods to be able to resolve back to the invoking host.

I don't think a better GPU was promised with Colab Pro+.