cudamat | Private copy of cudamat | GPU library

I am benchmarking GPU matrix multiplication using PyCUDA, CUDAMat, and Numba and ran into some behavior I can't find a way to explain.
I calculate the time it takes for 3 different steps independently - sending the 2 matrices to device memory, calculating the dot product, and copying the results back to host memory.
The benchmarking for the dot product step is done in a loop since my applications will be doing many multiplications before sending the result back.

As I increase the number of loops, the dot product time increases linearly just as expected. But the part I can't understand is that the time it takes to send the final result back to host memory also increases linearly with the loop count, even though it is only copying one matrix back to host memory. The size of the result is constant no matter how many matrix multiplication loops you do, so this makes no sense. It behaves as if returning the final result requires returning all the intermediate results at each step in the loop.

Some interesting things to note are that the increase in time it takes has a peak. As I go above ~1000 dot products in a loop the time it takes to copy the final result back reaches a peak. Another thing is if inside the dot product loop I reinitialize the matrix that holds the result this behavior stops and the copy back time is the same no matter how many multiplies are done.
For example -