cuda | Experiments with CUDA and Rust | GPU library

For the benefit of community providing solution here

This problem is because when keras run with gpu, it uses almost all vram. So we needed to give memory_limit for each notebook as shown below

As I wrote in the comment, I've been using a very similar Conda + Poetry setup in a data science project for the last year, for reasons similar to yours, and it's been working fine. The great majority of my dependencies are specified in pyproject.toml, but when there's something that's unavailable in PyPI, I add it to environment.yml.

Some additional tips:

1. Add Poetry, possibly with a version number (if needed), as a dependency in environment.yml, so that you get Poetry installed when you run conda env create, along with Python and other non-PyPI dependencies.
2. Consider adding conda-lock, which gives you lock files for Conda dependencies, just like you have poetry.lock for Poetry dependencies.

Here is the code:

In your posted system information, the last line

The problem is a problem in the Nvidia Driver 496.13. I had the same problem, and reverting to an older version fixed it

Turns out that Windows 10 Update Assistant incorrectly reported it upgraded my OS to 21H2 on my laptop. Checking Windows version by running winver reports that my OS is still 21H1. Of course CUDA in WSL2 will not work in Windows 10 without 21H2.

After successfully installing 21H2 I can confirm CUDA works with WSL2 even for laptops with Optimus NVIDIA cards.

You're using outdated imports for tf.keras. Layers can now be imported directly from tensorflow.keras.layers:

The output spatial dimensions of nn.ConvTranspose2d are given by:

Your kernel code looks good and the cache tiling implementation is correct. Only make sure that the number of bodies is a multiple of local size, or alternatively limit the inner for loop to the global size additionally.

OpenCL allows usage of multiple devices in parallel. You need to make a thread with a queue for each device separately. You also need to take care of device-device communications and synchronization manually. Data transfer happens over PCIe (you also can do remote direct memory access); but you can't use NVLink with OpenCL. This should not be an issue in your case though as you need only little data transfer compared to the amount of arithmetic.

A few more remarks:

• In many cases N-body requires FP64 to sum up the forces and resolve positions at very different length scales. However on the A6000, FP64 performance is very poor, just like on GeForce Ampere. FP32 would be significantly (~64x) faster, but is likely insufficient in terms of accuracy here. For efficient FP64 you would need an A100 or MI100.
• Instead of 1.0/sqrt, use rsqrt. This is hardware supported and almost as fast as a multiplication.
• Make sure to use either FP32 float (1.0f) or FP64 double (1.0) literals consistently. Using double literals with float triggers double arithmetic and casting of the result back to float which is much slower.

EDIT: To help you out with the error message: Most probably the error at clCreateKernel (what value does status have after calling clCreateKernel?) hints that program is invalid. This might be because you give clBuildProgram a vector of 2 devices, but set the number of devices to only 1 and also have context only for 1 device. Try