Slow tests sometimes

Open discussion for MAGMA library (Matrix Algebra on GPU and Multicore Architectures)

Slow tests sometimes

Postby waitzman » Mon Jul 29, 2013 11:56 am

I have a problem with test code sometimes running the test code testing_sgetri_gpu with increasingly larger matrixes in -c (check) mode.
When I get to a sufficiently large matrix, the test program fails with "!!!! magma_malloc failed for: d_A". That makes perfect sense as I don't have enough memory on the GPU for the matrix. But sometimes the performance of the test code is about half its normal performance, as shown below for one case. I am wondering if the crashed test program sometimes leaves the GPU in a partially cleaned up state, so it doesn't have the compute resources for later tests. The slowness problem goes away only after I reboot my machine.

Example: 285.30 GFlops/s version 123.98:
Code: Select all
   N   CPU GFlop/s (sec)   GPU GFlop/s (sec)   ||R||_F / ||A||_F
 5000    135.72 (   1.23)    285.30 (   0.58)       -inf

Code: Select all
   N   CPU GFlop/s (sec)   GPU GFlop/s (sec)   ||R||_F / ||A||_F
 5000    136.23 (   1.22)    123.98 (   1.34)       -inf

Is this a known problem?
Is there a way to clean up the GPU's state short of a reboot?

I am running on an iMac 27" 2012 (OS X 10.8.4) with its NVIDIA GeForce GTX 675MX and an Intel Core i5 at 3.2 GHz with 24GB of RAM.
I am using the CUDA 5.5 release candidate and the magma-1.4.0-beta2.

Posts: 3
Joined: Wed Jul 17, 2013 2:22 pm

Re: Slow tests sometimes

Postby mgates3 » Fri Aug 09, 2013 1:09 pm

Sometimes cards will decrease their performance if they overheat. I haven't personally observed this, though. Does waiting a few minutes help?

Are you using the provided, or did you customize it in some way? If so, can you provide your

Also, can you provide the COMPLETE input & output of the tester, including how you ran it? That provides some valuable information about your environment. E.g.,

Code: Select all
magma-trunk/testing> ./testing_sgetri_gpu -N 5000 -c
MAGMA 1.4.0 svn
device 0: GeForce GT 330M, 1100.0 MHz clock, 511.7 MB memory, capability 1.2
Usage: ./testing_sgetri_gpu [options] [-h|--help]

    N   CPU GFlop/s (sec)   GPU GFlop/s (sec)   ||R||_F / ||A||_F
 5000     29.67 (   5.62)     33.43 (   4.99)   2.18e-05

Posts: 566
Joined: Fri Jan 06, 2012 2:13 pm

Return to User discussion

Who is online

Users browsing this forum: No registered users and 1 guest