Multi GPU dgeqrf() problem

[jcbowden12]

Hi,

I have modified the testing_dgeqrf_mgpu.cpp test file so that it loops multiple times to call the magma_dgeqrf2_mgpu() function (or magma_dgeqrf4(), it doe not change the following issue).
When I run the program, it accumulates memory at the rate of the size of the input matrix (M * N * sizeof(double)) on each iteration.

I cannot find where the leak is happening. I have tested with valgrind and valgrind does not find any memory leak either. Each time I run it though it will freeze when 86% of my physical memory is used. Something is not releasing memory.

I have removed the trace output of magma_dgeqrf2_mgpu() and I comment out the lapck77_dgeqrf() function and use a memcpy to fill the h_R array, but otherwise the code is the same as in magma-1.2.0. I also tried removing use of pinned memory. Still no change in behaviour.

Any help will be appreciated. Code is attached and I have been running it like: ./testing_dgeqrf_mgpu -M 3500 -N 3500 -NGPU 1
System is running 3 x Fermi 2050s, 128GB RAM.

Regards,
Josh.

[Stan Tomov]

Hi Josh,
Thank you for this bug report. We didn't free a local work space in {s,d,c,z}geqrf_mgpu.cpp (and {s,d,c,z}geqrf_mgpu-trace.cpp). Please insert

Code: Select all

magma_free_host( local_work );


before

Code: Select all

return *info;


Also, there was a bug in file interface_cuda/alloc.cpp. Function magma_free_host should free memory with cudaFreeHost (instead of using cudaFree).

The fix is in the SVN and will be available with the next release.

Stan

[jcbowden12]

Hi Stan,
Yes, that fixed the problem. Thank you very much for finding it.
I should also say that the 'trace' code in the magma_dgeqrf2_mgpu() function degrades performance as iterations move along. It is easy to comment out the relevant code though.
Thanks for the great work,
Josh.

[jcbowden12]

Hello,

In case someone is calling this function from R, I'd just like to add that I continued to have problems with memory release when 'pinned' memory was used for the A input matrix, but only when functions were called from R. i.e. I could not allocate memory with magma_dmalloc_host(A_pinned), call magma_dgeqrf4(A_pinned), (memcpy A_pinned back to A) and then free with magma_free_host(A_pinned).

This procedure was fine when used in the testing_dgeqrf_mgpu program.

If anyone has a good reason why then I'd be interested to know.

thanks,
Josh.