Trouble in running test programs

Open discussion for MAGMA

Trouble in running test programs

Postby scho » Wed Mar 09, 2011 12:26 am

I am new to MAGMA and just compiled library sources and testing programs.

But I have troubles in running test programs which are included in the distribution RC4.
- For example, executing testing_zgetrf yields 'Argument 7 of zgetrf had an illegal value.'
This error seems to come from magma_zgetrf.
- Execucting test_zgetrf_gpu gives '!!!! cublasAlloc failed for: d_A'

I have not changed the sources and tried with Intel compiler 11.1 and gnu (gcc 4.1.2).
I would appreciate it if someone would help me resolve these errors.

Suwon Cho
scho
 
Posts: 8
Joined: Wed Mar 09, 2011 12:07 am

Re: Trouble in running test programs

Postby mateo70 » Wed Mar 09, 2011 4:54 pm

Hi,

it just means that you don't have enough memory to run the default configuration.
You should try to specify M and N:
./testing_zgetrf_gpu -M 1024 -N 1024

Mathieu
mateo70
 
Posts: 41
Joined: Tue Mar 08, 2011 12:38 pm

Re: Trouble in running test programs

Postby fletchjp » Wed Mar 09, 2011 5:32 pm

The memory requirements of this test are about 6 Gbytes of main memory and 1.8 Gbytes of GPU memory.

I have 8 Gbytes of main memory and a 2 Gbyte card (GTX 460) but I guess many people do not.

This means I can run both testing_zgetrf and testing_zgetrf_gpu

The first one allocates and deallocates memory on both CPU and GPU as it goes along. The second one allocates GPU memory at the start (about 1.7 Gbytes).

John
fletchjp
 
Posts: 175
Joined: Mon Dec 27, 2010 7:29 pm

Re: Trouble in running test programs

Postby scho » Wed Mar 09, 2011 10:43 pm

Thanks for helpfull replies.
Now, testing_zgetrf works with some values of M and N.
However, I still get same errors with 'cublasAlloc'.

For whatever values of M and N, I got same error messages with testing_zgetrf_gpu (!!!! cublasAlloc failed for: d_A)
I also tried to run many other testing programs. No success at all (same type of errors).

Suwon
scho
 
Posts: 8
Joined: Wed Mar 09, 2011 12:07 am

Re: Trouble in running test programs

Postby fletchjp » Thu Mar 10, 2011 6:54 pm

What system and hardware do you have?
fletchjp
 
Posts: 175
Joined: Mon Dec 27, 2010 7:29 pm

Re: Trouble in running test programs

Postby scho » Fri Mar 11, 2011 8:34 pm

cpu: i7 980x
RAM: 12G
gpu: GTX580 with 1.5 GB

Thanks.
scho
 
Posts: 8
Joined: Wed Mar 09, 2011 12:07 am

Re: Trouble in running test programs

Postby fletchjp » Tue Mar 15, 2011 6:02 am

Those errors say that GPU memory allocation failed. Does it not succeed even with a small size for N, M?

John
fletchjp
 
Posts: 175
Joined: Mon Dec 27, 2010 7:29 pm

Re: Trouble in running test programs

Postby scho » Tue Mar 15, 2011 10:01 pm

All testing_*_gpu programs fail with any number. They fail even with -M 1 -N 1.

Suwon
scho
 
Posts: 8
Joined: Wed Mar 09, 2011 12:07 am

Re: Trouble in running test programs

Postby Stan Tomov » Wed Mar 16, 2011 1:27 am

Hello Suwon,
The GTX580 must show excellent performance in single precision; I would be interested to see it; of course if we manage to find the problem first!
From what you report so far it looks like there may be some incompatibility between the GPU, CUDA used and the driver. I would first check if you can run programs from the SDK. For example, what do you get after running deviceQuery. Run some of the other programs in the SDK to make sure the installation is correct, e.g., matrixMul and bandwidthTest.
If everything is fine with the SDK you can move to the magma testing directory. Make sure you used the same CUDA (sometimes system administrators may have installed several) and nvcc as in the SDK. Which CUDA do you use and which driver (e.g., 'cat /proc/driver/nvidia/version' will give you the driver).
You can first check that the LAPACK and BLAS installations are fine. You can run for example
./testing_zgetrf -M 20 -N 20
This is a small problem that will not try to use the GPU. All the code executed will be on the CPU using LAPACK. If you get ||PA-LU||/(||A||*N) to be zero your LAPACK most probably is not the problem.
Next try some entirely GPU kernels, e.g.,
./testing_sgemm -M 200 -N 200 -K 200
Next you can try some of the hybrid codes, e.g., the one that you mentioned is not working
./testing_zgetrf_gpu -M 1 -N 1
This call takes the 1x1 matrix from the GPU and copies it to the CPU, uses LAPACK to factor it, and moves the result back. So this is a small problem where no magma hybrid algorithms are used yet - only CUDA and CUBLAS calls related to initialization, memory allocation, and data transfers. If there is problem here the problem is most probably as I thought at the beginning the combination of CUDA, driver and GPU used.
Also, does the magma testing recognize the card - in general this is what the testing drivers print first. Is your device 0 recognized as the GTX580?
I hope these tests would help us to determine the problem.
Regards,
Stan
Stan Tomov
 
Posts: 251
Joined: Fri Aug 21, 2009 10:39 pm

Re: Trouble in running test programs

Postby scho » Wed Mar 16, 2011 7:27 am

Thank you very much, Stan.
I summarized what I tried according to your guidance as follows:

The GTX580 must show excellent performance in single precision; I would be interested to see it; of course if we manage to find the problem first!
From what you report so far it looks like there may be some incompatibility between the GPU, CUDA used and the driver. I would first check if you can run programs from the SDK. For example, what do you get after running deviceQuery. Run some of the other programs in the SDK to make sure the installation is correct, e.g., matrixMul and bandwidthTest.

---> What I have tried so far worked fine.

If everything is fine with the SDK you can move to the magma testing directory. Make sure you used the same CUDA (sometimes system administrators may have installed several) and nvcc as in the SDK. Which CUDA do you use and which driver (e.g., 'cat /proc/driver/nvidia/version' will give you the driver).

--> I reinstalled the driver which is in the download page of cudatooklit_3.2.16_linux_64_rhel5.5 & gpu..sdk_3.2.16.

--> 'cat /proc/driver/nvidia/version' yields
NVRM version: NVIDIA UNIX x86_64 Kernel Module 260.19.26 Mon Nov 29 00:53:44 PST 2010
GCC version: gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)

You can first check that the LAPACK and BLAS installations are fine. You can run for example
./testing_zgetrf -M 20 -N 20
This is a small problem that will not try to use the GPU. All the code executed will be on the CPU using LAPACK. If you get ||PA-LU||/(||A||*N) to be zero your LAPACK most probably is not the problem.

--> device 0: GeForce GTX 580, 1564.0 MHz clock, 1535.2 MB memory
testing_zgetrf -M 20 -N 20
M N CPU GFlop/s GPU GFlop/s ||PA-LU||/(||A||*N)
============================================================
20 20 0.01 1.40 8.655930e-18

Next try some entirely GPU kernels, e.g.,
./testing_sgemm -M 200 -N 200 -K 200

--> device 0: GeForce GTX 580, 1564.0 MHz clock, 1535.2 MB memory
!!!! cublasAlloc failed for: d_A

testing_sgemm -M 20 -N 20
device 0: GeForce GTX 580, 1564.0 MHz clock, 1535.2 MB memory
!!!! cublasAlloc failed for: d_A

Next you can try some of the hybrid codes, e.g., the one that you mentioned is not working
./testing_zgetrf_gpu -M 1 -N 1
This call takes the 1x1 matrix from the GPU and copies it to the CPU, uses LAPACK to factor it, and moves the result back. So this is a small problem where no magma hybrid algorithms are used yet - only CUDA and CUBLAS calls related to initialization, memory allocation, and data transfers. If there is problem here the problem is most probably as I thought at the beginning the combination of CUDA, driver and GPU used.
Also, does the magma testing recognize the card - in general this is what the testing drivers print first. Is your device 0 recognized as the GTX580?

-->./ testing_zgetrf -M 1 -N 1
device 0: GeForce GTX 580, 1564.0 MHz clock, 1535.2 MB memory
testing_zgetrf -M 1 -N 1
!!!! cublasAlloc failed for: d_A

==> HOWEVER, THE FORTRAN VERSION WORKS!
testing_zgetrf_gpu_f
Solving A x = b using LU factorization:
|| A || = 1.063E+03
|| b || = 9.994E-01
|| b - A x || / (||A|| ||b||) = 4.811E-16
Gflops = 35.0904631976274
scho
 
Posts: 8
Joined: Wed Mar 09, 2011 12:07 am

Next

Return to User discussion

Who is online

Users browsing this forum: Google [Bot], Yahoo [Bot] and 2 guests