Multi GPU dgeqrf() problem

Open discussion for MAGMA

Multi GPU dgeqrf() problem

Postby jcbowden12 » Wed Jun 13, 2012 2:36 am

Hi,

I have modified the testing_dgeqrf_mgpu.cpp test file so that it loops multiple times to call the magma_dgeqrf2_mgpu() function (or magma_dgeqrf4(), it doe not change the following issue).
When I run the program, it accumulates memory at the rate of the size of the input matrix (M * N * sizeof(double)) on each iteration.

I cannot find where the leak is happening. I have tested with valgrind and valgrind does not find any memory leak either. Each time I run it though it will freeze when 86% of my physical memory is used. Something is not releasing memory.

I have removed the trace output of magma_dgeqrf2_mgpu() and I comment out the lapck77_dgeqrf() function and use a memcpy to fill the h_R array, but otherwise the code is the same as in magma-1.2.0. I also tried removing use of pinned memory. Still no change in behaviour.

Any help will be appreciated. Code is attached and I have been running it like: ./testing_dgeqrf_mgpu -M 3500 -N 3500 -NGPU 1
System is running 3 x Fermi 2050s, 128GB RAM.

Regards,
Josh.
Attachments
testing_dgeqrf_mgpu.cpp
the modified multi-gpu test file
(4.9 KiB) Downloaded 101 times
jcbowden12
 
Posts: 5
Joined: Tue Mar 06, 2012 2:36 am

Re: Multi GPU dgeqrf() problem

Postby Stan Tomov » Tue Jun 19, 2012 2:06 am

Hi Josh,
Thank you for this bug report. We didn't free a local work space in {s,d,c,z}geqrf_mgpu.cpp (and {s,d,c,z}geqrf_mgpu-trace.cpp). Please insert
Code: Select all
magma_free_host( local_work );

before
Code: Select all
return *info;

Also, there was a bug in file interface_cuda/alloc.cpp. Function magma_free_host should free memory with cudaFreeHost (instead of using cudaFree).

The fix is in the SVN and will be available with the next release.

Stan
Stan Tomov
 
Posts: 251
Joined: Fri Aug 21, 2009 10:39 pm

Re: Multi GPU dgeqrf() problem

Postby jcbowden12 » Tue Jun 19, 2012 3:25 am

Hi Stan,
Yes, that fixed the problem. Thank you very much for finding it.
I should also say that the 'trace' code in the magma_dgeqrf2_mgpu() function degrades performance as iterations move along. It is easy to comment out the relevant code though.
Thanks for the great work,
Josh.
jcbowden12
 
Posts: 5
Joined: Tue Mar 06, 2012 2:36 am

Re: Multi GPU dgeqrf() problem

Postby jcbowden12 » Mon Jul 02, 2012 9:23 pm

Hello,

In case someone is calling this function from R, I'd just like to add that I continued to have problems with memory release when 'pinned' memory was used for the A input matrix, but only when functions were called from R. i.e. I could not allocate memory with magma_dmalloc_host(A_pinned), call magma_dgeqrf4(A_pinned), (memcpy A_pinned back to A) and then free with magma_free_host(A_pinned).

This procedure was fine when used in the testing_dgeqrf_mgpu program.

If anyone has a good reason why then I'd be interested to know.

thanks,
Josh.
jcbowden12
 
Posts: 5
Joined: Tue Mar 06, 2012 2:36 am


Return to User discussion

Who is online

Users browsing this forum: Google [Bot] and 2 guests