Slow tests sometimes

Open discussion for MAGMA

Slow tests sometimes

Postby waitzman » Mon Jul 29, 2013 11:56 am

I have a problem with test code sometimes running the test code testing_sgetri_gpu with increasingly larger matrixes in -c (check) mode.
When I get to a sufficiently large matrix, the test program fails with "!!!! magma_malloc failed for: d_A". That makes perfect sense as I don't have enough memory on the GPU for the matrix. But sometimes the performance of the test code is about half its normal performance, as shown below for one case. I am wondering if the crashed test program sometimes leaves the GPU in a partially cleaned up state, so it doesn't have the compute resources for later tests. The slowness problem goes away only after I reboot my machine.

Example: 285.30 GFlops/s version 123.98:
Code: Select all
   N   CPU GFlop/s (sec)   GPU GFlop/s (sec)   ||R||_F / ||A||_F
 5000    135.72 (   1.23)    285.30 (   0.58)       -inf

versus:
Code: Select all
   N   CPU GFlop/s (sec)   GPU GFlop/s (sec)   ||R||_F / ||A||_F
 5000    136.23 (   1.22)    123.98 (   1.34)       -inf


Is this a known problem?
Is there a way to clean up the GPU's state short of a reboot?

I am running on an iMac 27" 2012 (OS X 10.8.4) with its NVIDIA GeForce GTX 675MX and an Intel Core i5 at 3.2 GHz with 24GB of RAM.
I am using the CUDA 5.5 release candidate and the magma-1.4.0-beta2.

thanks.
waitzman
 
Posts: 3
Joined: Wed Jul 17, 2013 2:22 pm

Re: Slow tests sometimes

Postby mgates3 » Fri Aug 09, 2013 1:09 pm

Sometimes cards will decrease their performance if they overheat. I haven't personally observed this, though. Does waiting a few minutes help?

Are you using the provided make.inc.macos, or did you customize it in some way? If so, can you provide your make.inc?

Also, can you provide the COMPLETE input & output of the tester, including how you ran it? That provides some valuable information about your environment. E.g.,

Code: Select all
magma-trunk/testing> ./testing_sgetri_gpu -N 5000 -c
MAGMA 1.4.0 svn
device 0: GeForce GT 330M, 1100.0 MHz clock, 511.7 MB memory, capability 1.2
Usage: ./testing_sgetri_gpu [options] [-h|--help]

    N   CPU GFlop/s (sec)   GPU GFlop/s (sec)   ||R||_F / ||A||_F
=================================================================
 5000     29.67 (   5.62)     33.43 (   4.99)   2.18e-05


Thanks.
-mark
mgates3
 
Posts: 409
Joined: Fri Jan 06, 2012 2:13 pm


Return to User discussion

Who is online

Users browsing this forum: No registered users and 3 guests

cron