cuda-gdb | This directory contains various GNU compilers , assemblers | GPU library

Your in-kernel malloc operation are exceeding the device heap size.

Any time you are having trouble with a CUDA code that uses in kernel malloc or new, it's good practice (at least as a diagnostic) to check the returned pointer value for NULL, before attempting to use (i.e. dereference) it.

When I do that in your code right after the malloc operations in aligner::kmdist, I get the asserts being hit, indicating NULL return values. This is the indication you have exceeded the device heap. You can increase the device heap size.

When I increase the device heap size to 1GB, this particular issue disappears, and at that point cuda-memcheck may start reporting other errors (I don't know, your application may have other defects, but the proximal issue here is exceeding the device heap).

As an aside, I also recommend that you compile your code to match the architecture you are running on:

In the meantime, I figured out myself. You must add however that strangely, G++ and GCC version 4.9 is still not available, you must go with 4.8. By combining multiple sources, I constructed a way to install G++ and GCC 4.8.5 on your machine and configure them as the default ones:

After the helpful hint from @talonmies, I suspected this was a device linking problem. I simplified my build process, included all CUDA files in one translation unit, and turned off SEPARABLE COMPILATION.

Still, I did not see a cmake_device_link.o in either my main application binary or the component that called into my cuda library. And, still had the same error. Tried setting CUDA_RESOLVE_DEVICE_SYMBOLS to no effect.

Finally, I tried building the component that calls into my cuda library as SHARED. I saw the device linking step when building the .so in my cmake output, and the program runs fine. I do not know why building SHARED fixes what I suspect was a device linking problem - will accept any answer that deciphers that?

From Could not dlopen library 'libcudart.so.10.0'; we can get that you tensorflow package is built against CUDA 10.0. You should install CUDA 10.0 or build it from source (against CUDA 10.1 or 10.2) by yourself.

If you are using multiples streams, you may override Ad and Bd before using them.

Example with iters = 2 and nStream = 2 :

When a device-side error is detected while CUDA device code is running, that error is reported via the usual CUDA runtime API error reporting mechanism. The usual detected error in device code would be something like an illegal address (e.g. attempt to dereference an invalid pointer) but another type is a device-side assert. This type of error is generated whenever a C/C++ assert() occurs in device code, and the assert condition is false.

Such an error occurs as a result of a specific kernel. Runtime error checking in CUDA is necessarily asynchronous, but there are probably at least 3 possible methods to start to debug this.

1. Modify the source code to effectively convert asynchronous kernel launches to synchronous kernel launches, and do rigorous error-checking after each kernel launch. This will identify the specific kernel that has caused the error. At that point it may be sufficient simply to look at the various asserts in that kernel code, but you could also use step 2 or 3 below.

2. Run your code with cuda-memcheck. This is a tool something like "valgrind for device code". When you run your code with cuda-memcheck, it will tend to run much more slowly, but the runtime error reporting will be enhanced. It is also usually preferable to compile your code with -lineinfo. In that scenario, when a device-side assert is triggered, cuda-memcheck will report the source code line number where the assert is, and also the assert itself and the condition that was false. You can see here for a walkthrough of using it (albeit with an illegal address error instead of assert(), but the process with assert() will be similar.

3. It should also be possible to use a debugger. If you use a debugger such as cuda-gdb (e.g. on linux) then the debugger will have back-trace reports that will indicate which line the assert was, when it was hit.

Both cuda-memcheck and the debugger can be used if the CUDA code is launched from a python script.

At this point you have discovered what the assert is and where in the source code it is. Why it is there cannot be answered generically. This will depend on the developers intention, and if it is not commented or otherwise obvious, you will need some method to intuit that somehow. The question of "how to work backwards" is also a general debugging question, not specific to CUDA. You can use printf in CUDA kernel code, and also a debugger like cuda-gdb to assist with this (for example, set a breakpoint prior to the assert, and inspect machine state - e.g. variables - when the assert is about to be hit).

Try to change the code architecture (such as sm_20) to some higher version in CMakeLists.txt of stnbhwd that you are trying to install.

From:

I was able to do kernel source-level debugging on a nvrtc-generated kernel with cuda-gdb as follows:

• start with vectorAdd_nvrtc sample code
• modify the compileFileToPTX routine (provided by nvrtc_helper.h) to add the --device-debug switch during the compile-cu-to-ptx step.
• modify the loadPTX routine (provided by nvrtc_helper.h) to add the CU_JIT_GENERATE_DEBUG_INFO option (set to 1) for the cuModuleLoadDataEx load/JIT PTX-to-binary step.
• compile the main function (vectorAdd.cpp) with -g option.

Here is a complete test case/session. I'm only showing the vectorAdd.cpp file from the project because that is the only file I modified. Other project file(s) are identical to what is in the sample project:

Turns out that cudaMemcpy was not the issue. when examining device global memroy using cuda-gdb, one cannot do: x/10fw float_array. It will give incorrect values. To view, try this: p ((@global float*) float_array)[0]@10