Trouble in running test programs

[scho]

I am new to MAGMA and just compiled library sources and testing programs.

But I have troubles in running test programs which are included in the distribution RC4.
- For example, executing testing_zgetrf yields 'Argument 7 of zgetrf had an illegal value.'
This error seems to come from magma_zgetrf.
- Execucting test_zgetrf_gpu gives '!!!! cublasAlloc failed for: d_A'

I have not changed the sources and tried with Intel compiler 11.1 and gnu (gcc 4.1.2).
I would appreciate it if someone would help me resolve these errors.

Suwon Cho

[mateo70]

Hi,

it just means that you don't have enough memory to run the default configuration.
You should try to specify M and N:
./testing_zgetrf_gpu -M 1024 -N 1024

Mathieu

[fletchjp]

The memory requirements of this test are about 6 Gbytes of main memory and 1.8 Gbytes of GPU memory.

I have 8 Gbytes of main memory and a 2 Gbyte card (GTX 460) but I guess many people do not.

This means I can run both testing_zgetrf and testing_zgetrf_gpu

The first one allocates and deallocates memory on both CPU and GPU as it goes along. The second one allocates GPU memory at the start (about 1.7 Gbytes).

John

[scho]

Thanks for helpfull replies.
Now, testing_zgetrf works with some values of M and N.
However, I still get same errors with 'cublasAlloc'.

For whatever values of M and N, I got same error messages with testing_zgetrf_gpu (!!!! cublasAlloc failed for: d_A)
I also tried to run many other testing programs. No success at all (same type of errors).

Suwon

[fletchjp]

What system and hardware do you have?

[scho]

cpu: i7 980x
RAM: 12G
gpu: GTX580 with 1.5 GB

Thanks.

[fletchjp]

Those errors say that GPU memory allocation failed. Does it not succeed even with a small size for N, M?

John

[scho]

All testing_*_gpu programs fail with any number. They fail even with -M 1 -N 1.

Suwon

[Stan Tomov]

Hello Suwon,
The GTX580 must show excellent performance in single precision; I would be interested to see it; of course if we manage to find the problem first!
From what you report so far it looks like there may be some incompatibility between the GPU, CUDA used and the driver. I would first check if you can run programs from the SDK. For example, what do you get after running deviceQuery. Run some of the other programs in the SDK to make sure the installation is correct, e.g., matrixMul and bandwidthTest.
If everything is fine with the SDK you can move to the magma testing directory. Make sure you used the same CUDA (sometimes system administrators may have installed several) and nvcc as in the SDK. Which CUDA do you use and which driver (e.g., 'cat /proc/driver/nvidia/version' will give you the driver).
You can first check that the LAPACK and BLAS installations are fine. You can run for example
./testing_zgetrf -M 20 -N 20
This is a small problem that will not try to use the GPU. All the code executed will be on the CPU using LAPACK. If you get ||PA-LU||/(||A||*N) to be zero your LAPACK most probably is not the problem.
Next try some entirely GPU kernels, e.g.,
./testing_sgemm -M 200 -N 200 -K 200
Next you can try some of the hybrid codes, e.g., the one that you mentioned is not working
./testing_zgetrf_gpu -M 1 -N 1
This call takes the 1x1 matrix from the GPU and copies it to the CPU, uses LAPACK to factor it, and moves the result back. So this is a small problem where no magma hybrid algorithms are used yet - only CUDA and CUBLAS calls related to initialization, memory allocation, and data transfers. If there is problem here the problem is most probably as I thought at the beginning the combination of CUDA, driver and GPU used.
Also, does the magma testing recognize the card - in general this is what the testing drivers print first. Is your device 0 recognized as the GTX580?
I hope these tests would help us to determine the problem.
Regards,
Stan

[scho]

Thank you very much, Stan.
I summarized what I tried according to your guidance as follows:

The GTX580 must show excellent performance in single precision; I would be interested to see it; of course if we manage to find the problem first!
From what you report so far it looks like there may be some incompatibility between the GPU, CUDA used and the driver. I would first check if you can run programs from the SDK. For example, what do you get after running deviceQuery. Run some of the other programs in the SDK to make sure the installation is correct, e.g., matrixMul and bandwidthTest.

---> What I have tried so far worked fine.

If everything is fine with the SDK you can move to the magma testing directory. Make sure you used the same CUDA (sometimes system administrators may have installed several) and nvcc as in the SDK. Which CUDA do you use and which driver (e.g., 'cat /proc/driver/nvidia/version' will give you the driver).

--> I reinstalled the driver which is in the download page of cudatooklit_3.2.16_linux_64_rhel5.5 & gpu..sdk_3.2.16.

--> 'cat /proc/driver/nvidia/version' yields
NVRM version: NVIDIA UNIX x86_64 Kernel Module 260.19.26 Mon Nov 29 00:53:44 PST 2010
GCC version: gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)

You can first check that the LAPACK and BLAS installations are fine. You can run for example
./testing_zgetrf -M 20 -N 20
This is a small problem that will not try to use the GPU. All the code executed will be on the CPU using LAPACK. If you get ||PA-LU||/(||A||*N) to be zero your LAPACK most probably is not the problem.

--> device 0: GeForce GTX 580, 1564.0 MHz clock, 1535.2 MB memory
testing_zgetrf -M 20 -N 20
M N CPU GFlop/s GPU GFlop/s ||PA-LU||/(||A||*N)
============================================================
20 20 0.01 1.40 8.655930e-18

Next try some entirely GPU kernels, e.g.,
./testing_sgemm -M 200 -N 200 -K 200

--> device 0: GeForce GTX 580, 1564.0 MHz clock, 1535.2 MB memory
!!!! cublasAlloc failed for: d_A

testing_sgemm -M 20 -N 20
device 0: GeForce GTX 580, 1564.0 MHz clock, 1535.2 MB memory
!!!! cublasAlloc failed for: d_A

Next you can try some of the hybrid codes, e.g., the one that you mentioned is not working
./testing_zgetrf_gpu -M 1 -N 1
This call takes the 1x1 matrix from the GPU and copies it to the CPU, uses LAPACK to factor it, and moves the result back. So this is a small problem where no magma hybrid algorithms are used yet - only CUDA and CUBLAS calls related to initialization, memory allocation, and data transfers. If there is problem here the problem is most probably as I thought at the beginning the combination of CUDA, driver and GPU used.
Also, does the magma testing recognize the card - in general this is what the testing drivers print first. Is your device 0 recognized as the GTX580?

-->./ testing_zgetrf -M 1 -N 1
device 0: GeForce GTX 580, 1564.0 MHz clock, 1535.2 MB memory
testing_zgetrf -M 1 -N 1
!!!! cublasAlloc failed for: d_A

==> HOWEVER, THE FORTRAN VERSION WORKS!
testing_zgetrf_gpu_f
Solving A x = b using LU factorization:
|| A || = 1.063E+03
|| b || = 9.994E-01
|| b - A x || / (||A|| ||b||) = 4.811E-16
Gflops = 35.0904631976274