nvidia-driver | NVIDIA 's proprietary display driver for NVIDIA graphic cards | GPU library

You seem to have made manual changes to the environment.yaml that is provided in the git that you linked. From the github (your changes highlighted):

According the docker image that the container is trying to pull (gke-nvidia-installer:fixed), it looks like you're trying use Ubuntu daemonset instead of cos.

You should run kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/cos/daemonset-preloaded.yaml

This will apply the right daemonset for your cos node pool, as stated here.

In addition, please verify your node pool has the https://www.googleapis.com/auth/devstorage.read_only scope which is needed to pull the image. You can should see it in your node pool page in GCP Console, under Security -> Access scopes (The relevant service is Storage).

The exact mapping of workers and vectors will depend on the target device and implementation. Specifically when using NVHPC targeting NVIDIA GPUs, a "gang" maps to a CUDA Block, "worker" maps the the y dimension of a thread block, and "vector" to the x-dimension. The value used in "num_workers" or "vector_length" may be reduced given the constrains of the target. CUDA Blocks can contain up to a maximum 1024 threads so the "4096" value will be reduced to what is allowed by the hardware. Secondly, in order to support vector reductions in device routines, a maximum vector_length can be 32. In other words, you're "4096" value is actually "32" due to these constraints.

Note to see the max thread block size on your device, run the "nvaccelinfo" utility and look for the "Maximum Threads per Block" and "Maximum Block Dimensions" fields. Also, setting the environment variable "NV_ACC_TIME=1" will have the runtime produce some basic profiling information, including the actual number of blocks and thread block size used during the run.

In my understanding, a huge amount of threads should lie dormant when there are only as many as 10 iterations in the vector loop.

CUDA threads are grouped into a "warp" of 32 threads where all threads of a warp execute the same instructions concurrently (aka SIMT or single instruction multiple threads). Hence even though only 10 threads are doing useful work, the remaining 12 are not dormant. Plus they still take resources such as registers so adding too many threads for loops with lower trip counts, may actually hurt performance.

In this case setting the vector length to 1 is most likey the best case since the warp can now be comprised of the y-dimension threads. Setting it to 2, will cause a full 32 thread warp in the x-dimension, but only 2 doing useful work.

As to why some combinations give incorrect results, I didn't investigate. Routine worker, especially with reductions, is rarely used so it's possible we have some type of code gen issue, like an off-by one error in the reduction, at these irregular schedule sizes. I'll look into this later and determine if I need to file an issue report.

For #2, How you're determining it's getting run twice? Is just this based on the runtime?

looks like you trashed something (driver or cuda runtime) and you are not able to call any function related to cuda.

in my humble experience, I usually get these errors when my kernels runs for too long on a Windows machine and the Windows Display Driver Manager reset my GPU while i'm running the kernel.

Maybe you are experiencing some similiar issues on linux.

To fix this, have you tried to reset your GPU using the following bash command line ?

Can i do that with kubectl apply ? Ideally I would like to only run that yaml code onto the GPU node. How can I achieve that?

Yes, You can run the Deamon set on each node which will run the command on Nodes.

As you are on GKE and Daemon set will also run the command or script on New nodes also which are getting scaled up also.

Daemon set is mainly for running applications or deployment on each available node in the cluster.

We can leverage this deamon set and run the command on each node that exist and is also upcoming.

Example YAML :

Yes, this is actually possible with a "wrapper" defined resource type:

@DazWilkin's comment is correct:

It's not possible to install graphics driver to Cloud Run fully managed because it doesn't expose any GPU.

You should deploy it in Cloud Run Anthos(GKE). But you'll need to configure your GKE Cluster to install your GPU, According to the documentation, you should follow the steps:

1. Add a GPU-enabled node pool to your GKE cluster.

In this step, you can enable Enable Virtual Workstation (NVIDIA GRID). Please choose a GPU such as NVIDIA Tesla T4, P4 or P100 to enable NVIDIA GRID.

1. Install NVIDIA's device drivers to the nodes.

You can now create a service that will consume GPUs and deploy the image to Cloud Run for Anthos:

1. Setting up your service to consume GPUs

You can create symlinks inside of /usr/lib/x86_64-linux-gnu directory. I found it by:

1. Check your display server; Try to switch to Wayland or xorg:

https://docs.fedoraproject.org/en-US/quick-docs/configuring-xorg-as-default-gnome-session/

1. Have you tried boot with nomodeset var ? I had simmilar problem with laptop display with Intel UHD 620 after kernel update (fedora 33 → 34). Laptop screen was not working but second display was.

How to set nomodeset in GRUB

About drivers - if problem is related to drivers in fedora 34 you can use nvidia auto installer or rpmfusion