OperatorGraph | shape grammar base system and auto-tuner | GPU library

Interesting question. OpenCL defines a number of conceptual memories including local memory, constant memory, global memory, and private memory. And physically as you know, the hardware implementation of these memories is hardware dependent. For instance, some may emulate local memory using cache or system memory instead of having physical memory.

AFAIK, ARM Mali GPU does not have local memory, whereas Qualcomm Adreno GPU does have local memory.

For instance below table shows the definition of each memory in OpenCL and their relative latency and physical locations in Adreno GPU cited from OpenCL Optimization and Best Practices for Qualcomm Adreno GPUs∗

Answer updated:

as commented by SK-logic below, Mali6xx have a local memory (shared with cache).

For those who are trying to solve a similar issue, it might be coming from the java.exe executable not using the gpu you want as a default device, you can change that in Windows' settings.

According the docker image that the container is trying to pull (gke-nvidia-installer:fixed), it looks like you're trying use Ubuntu daemonset instead of cos.

You should run kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/cos/daemonset-preloaded.yaml

This will apply the right daemonset for your cos node pool, as stated here.

In addition, please verify your node pool has the https://www.googleapis.com/auth/devstorage.read_only scope which is needed to pull the image. You can should see it in your node pool page in GCP Console, under Security -> Access scopes (The relevant service is Storage).

For the benefit of community providing solution here

This problem is because when keras run with gpu, it uses almost all vram. So we needed to give memory_limit for each notebook as shown below

GLSL structs and C++ structs have different rules on alignment. For structs, the spec states:

If the member is a structure, the base alignment of the structure is N, where N is the largest base alignment value of any of its members, and rounded up to the base alignment of a vec4. The individual members of this substructure are then assigned offsets by applying this set of rules recursively, where the base offset of the first member of the sub-structure is equal to the aligned offset of the structure. The structure may have padding at the end; the base offset of the member following the sub-structure is rounded up to the next multiple of the base alignment of the structure.

Let's analyze the struct:

Here is the code:

Gradient tape triggers automatic differentiation which requires tracking gradients on all your weights and activations. Autodiff requires multiple more memory. This is normal. You'll have to manually tune your batch size until you find one that works, then tune your LR. Usually, the tune just means guess & check or grid search. (I am working on a product to do all of that for you but I'm not here to plug it).

Turns out that Windows 10 Update Assistant incorrectly reported it upgraded my OS to 21H2 on my laptop. Checking Windows version by running winver reports that my OS is still 21H1. Of course CUDA in WSL2 will not work in Windows 10 without 21H2.

After successfully installing 21H2 I can confirm CUDA works with WSL2 even for laptops with Optimus NVIDIA cards.

It looks like PyTorch support for the M1 GPU is in the works, but is not yet complete.

From @soumith on GitHub:

So, here's an update. We plan to get the M1 GPU supported. @albanD, @ezyang and a few core-devs have been looking into it. I can't confirm/deny the involvement of any other folks right now.

So, what we have so far is that we had a prototype that was just about okay. We took the wrong approach (more graph-matching-ish), and the user-experience wasn't great -- some operations were really fast, some were really slow, there wasn't a smooth experience overall. One had to guess-work which of their workflows would be fast.

So, we're completely re-writing it using a new approach, which I think is a lot closer to your good ole PyTorch, but it is going to take some time. I don't think we're going to hit a public alpha in the next ~4 months.

We will open up development of this backend as soon as we can.

That post: https://github.com/pytorch/pytorch/issues/47702#issuecomment-965625139

TL;DR: a public beta is at least 4 months out.