nvidia-smi | Vscode extension -- show GPU activities on status bar

From the TensorFlow docs

By default, TensorFlow maps nearly all of the GPU memory of all GPUs (subject to CUDA_VISIBLE_DEVICES) visible to the process. This is done to more efficiently use the relatively precious GPU memory resources on the devices by reducing memory fragmentation. To limit TensorFlow to a specific set of GPUs, use the tf.config.set_visible_devices method.

If you don't want TensorFlow to allocate the totality of your VRAM, you can either set a hard limit on how much memory to use or tell TensorFlow to only allocate as much memory as needed.

Configure a virtual GPU device as follows:

After trying it and solving a myriad of problems along the way, I have realized that it is simply the documentation that is out of date.

Adding the following yaml block to my docker-compose.yml resulted in nvidia-smi being available to use.

Your variable isn't getting set because right now your nvidia-smi command is throwing an error (to stdout, curiously) but skip=1 is skipping over it so there's nothing left to set the variable to.

= is one of the default delimiters for strings and so both of the = symbols in your command need to be escaped for your query to be executed correctly.

Solved by reinstalling the pytorch in my Conda Env.

You may try reinstalling the Pytorch or create a new Conda Environment to do it again.

This is more of a comment, but worth pointing out.

The reason in general is indeed what talonmies commented, but you are summing up the numbers incorrectly. Let's see what happens when tensors are moved to GPU (I tried this on my PC with RTX2060 with 5.8G usable GPU memory in total):

Let's run the following python commands interactively:

This happens becauce pytorch reserves the gpu memory for fast memory allocation. To learn more about it, see pytorch memory management. To solve this issue, you can use the following code:

Turns out that Windows 10 Update Assistant incorrectly reported it upgraded my OS to 21H2 on my laptop. Checking Windows version by running winver reports that my OS is still 21H1. Of course CUDA in WSL2 will not work in Windows 10 without 21H2.

After successfully installing 21H2 I can confirm CUDA works with WSL2 even for laptops with Optimus NVIDIA cards.

You can't have more than 1 GPU driver operational in this setting. Period. That driver is installed in the base machine. If you do something not recommended, like install it or attempt to install it in the container, it is still the one in the base machine that is in effect for the base machine as well as the container. Note that anything reported by nvidia-smi pertains to the GPU driver only, and therefore is using the driver installed in the base machine, whether you run it inside or outside of the container. There may be detailed reporting differences like visible GPUs, but this doesn't impact versions reported.

The CUDA runtime version will be the one that is installed in the container. Period. It has no ability to inspect what is outside the container. If it happens to match what you see outside the container, then it is simply the case that you have the same configuration outside the container as well as inside.

Probably most of your confusion would be resolved with this answer and perhaps your question is a duplicate of that one.

The problem is, with each call of get_named_entities, the amount of GPU memory allocated goes up.

You should detach your data as explained in the FAQ:

Don’t accumulate history across your training loop. By default, computations involving variables that require gradients will keep history. This means that you should avoid using such variables in computations which will live beyond your training loops, e.g., when tracking statistics. Instead, you should detach the variable or access its underlying data.

Edit

You can also use