gpumatrix | array operation library on GPU with Eigen | GPU library

Parallelization with foreach or similar tools works because you have multiple CPUs (or a CPU with multiple cores), which can process multiple tasks at once. A GPU also has multiple cores, but these are already used to process a single task in parallel. So if you want to parallelize further, you will need multiple GPUs.

However, keep in mind that GPUs are faster than CPUs only for certain types of applications. Matrix operations with large matrices being a prime example! See the performance section here for a recent comparison of one particular example. So it might make sense for you to consider if the GPU is the right tool for you.

In addition: File IO will always go via the CPU.

This is a really annoying problem and it arises from the fact that for some reason a code compiled to be executed on CPU is not garbage-collected properly. So even though you are running a garbage collector - a compiled model is still on GPU. In order to overcome this, you may try a solution presented here (TLDR: run training in a separate process - as when process is finished - memory is cleared)

Your destructor always deletes the data pointer. However, the default copy constructor will have a copy of the original object's data pointer that it must not delete.

One way to fix this is to modify your class to hold a flag that says if the data pointer is owned by the class and needs to be deleted. Then define a copy constructor that sets that flag appropriately.

There are potential issues with this method if the copy outlives the original object, and the move constructor should be added as well. Then there's the copy assignment and move assignment operators. See this answer for more information.

There is fundamentally nothing wrong with the code you have posted. It is probably close to best practice. But it isn't compatible with the design idiom of your kernel.

As you can see here, your GPU is capable of running 2^31 - 1 or 2147483647 blocks. So you could change the code in question to this: