Magma 1.2.1 Release Notes and Multi-GPU

Open discussion for MAGMA

Magma 1.2.1 Release Notes and Multi-GPU

Postby kuruvinandan » Fri Jul 27, 2012 3:16 am

All,
I just downloaded Magma 1.2.1. I always had a notion that MAGMA is about accelerating Lacpack/blas routines on all CPU and GPU cores available in the system. However, 1.2.1. Release notes begins with:
"
MAGMA Release Notes

-----------------------------------------------------

MAGMA is intended for a single CUDA enabled NVIDIA GPU. It supports
Tesla and Fermi GPUs. For more details see the MAGMA 1.0 presentation.
....
"
Now, Does not MAGMA support multi-GPUs? In particular, I am looking @ LU dense Factorization, Matrix inversion, SVD
Thanks for any help,
Best Regards,
Kuruvinandan
kuruvinandan
 
Posts: 3
Joined: Fri Jul 27, 2012 3:11 am

Re: Magma 1.2.1 Release Notes and Multi-GPU

Postby mgates3 » Fri Jul 27, 2012 4:22 am

Yes, MAGMA supports multiple GPUs for certain routines. Currently, the release has multi-GPU support for LU, Cholesky, QR, as noted later in the release notes. Multi-GPU eigenvalue routines are also under development. I will update that first sentence of the release notes, though. Thanks.

-mark
mgates3
 
Posts: 424
Joined: Fri Jan 06, 2012 2:13 pm

Re: Magma 1.2.1 Release Notes and Multi-GPU

Postby kuruvinandan » Fri Jul 27, 2012 6:11 am

Thank you Mark! It was very helpful.
I hope the SVD ones are also multi-GPU enabled. Thanks,
kuruvinandan
 
Posts: 3
Joined: Fri Jul 27, 2012 3:11 am

Re: Magma 1.2.1 Release Notes and Multi-GPU

Postby neuron » Wed Aug 08, 2012 9:20 am

mgates3 wrote:Yes, MAGMA supports multiple GPUs for certain routines. Currently, the release has multi-GPU support for LU, Cholesky, QR, as noted later in the release notes. Multi-GPU eigenvalue routines are also under development. I will update that first sentence of the release notes, though. Thanks.

-mark


Hi,
I'd like to know if Magma has multi-gpu support for matrix-matrix product (SGEMM).
Presently I use Cublas, but in Cublas sgemm is only mono-gpu.
Thank you
neuron
 
Posts: 3
Joined: Wed Aug 08, 2012 9:08 am

Re: Magma 1.2.1 Release Notes and Multi-GPU

Postby mgates3 » Wed Aug 08, 2012 5:47 pm

No, a multi-GPU GEMM is not specifically provided in MAGMA. What size and distribution of the A, B, C matrices are you looking for?

Internally we have effectively done multi-GPU GEMM in several codes, based on cublas GEMM. The implementation depends on how your data is distributed. For C = A*B, the easiest is if A is duplicated on all GPUs, while B and C are distributed by block columns, then
Code: Select all
[ A ][ B1  B2  B3 ] = [ A*B1  A*B2  A*B3 ]

So on all GPUs, create products Ci = A*Bi. (This is done in the LU code, zgetrf.) Conveniently, each GPU needs only one GEMM, even if it has multiple blocks of columns.

Or if A and C are distributed by block rows, while B is duplicated on all GPUs, then
Code: Select all
[ A1 ]         [ A1*B ]
[ A2 ] [ B ] = [ A2*B ]
[ A3 ]         [ A3*B ]

So on all GPUs, create products Ci = Ai*B.

If A is stored block-column cyclically and B is stored block-row cyclically, then
Code: Select all
               [ B1 ]
[ A1  A2  A3 ] [ B2 ] = A1*B1 + A2*B2 + A3*B3
               [ B3 ]

So on all GPUs, create products Ai*Bi, then do a sum reduction (e.g., on the CPU) to get the final result. More general distributions require more communication between the GPUs.

Hope that helps.
-mark
mgates3
 
Posts: 424
Joined: Fri Jan 06, 2012 2:13 pm

Re: Magma 1.2.1 Release Notes and Multi-GPU

Postby neuron » Thu Aug 09, 2012 9:15 am

Thank you,
can you give me a pointer to some of your multi-gpu GEMM implementations?
neuron
 
Posts: 3
Joined: Wed Aug 08, 2012 9:08 am

Re: Magma 1.2.1 Release Notes and Multi-GPU

Postby mgates3 » Thu Aug 09, 2012 1:12 pm

They are not implemented as separate functions, but are part of the multi-GPU codes such as LU and QR, so I don't think they will be much use for you. For instance, see zgetrf_mgpu.cpp, the loop that ends with "end of gpu updates".
-mark
mgates3
 
Posts: 424
Joined: Fri Jan 06, 2012 2:13 pm


Return to User discussion

Who is online

Users browsing this forum: No registered users and 2 guests