Magma 1.2.1 Release Notes and Multi-GPU

[kuruvinandan]

All,
I just downloaded Magma 1.2.1. I always had a notion that MAGMA is about accelerating Lacpack/blas routines on all CPU and GPU cores available in the system. However, 1.2.1. Release notes begins with:
"
MAGMA Release Notes

-----------------------------------------------------

MAGMA is intended for a single CUDA enabled NVIDIA GPU. It supports
Tesla and Fermi GPUs. For more details see the MAGMA 1.0 presentation.
....
"
Now, Does not MAGMA support multi-GPUs? In particular, I am looking @ LU dense Factorization, Matrix inversion, SVD
Thanks for any help,
Best Regards,
Kuruvinandan

[mgates3]

Yes, MAGMA supports multiple GPUs for certain routines. Currently, the release has multi-GPU support for LU, Cholesky, QR, as noted later in the release notes. Multi-GPU eigenvalue routines are also under development. I will update that first sentence of the release notes, though. Thanks.

-mark

[kuruvinandan]

Thank you Mark! It was very helpful.
I hope the SVD ones are also multi-GPU enabled. Thanks,

[neuron]

Yes, MAGMA supports multiple GPUs for certain routines. Currently, the release has multi-GPU support for LU, Cholesky, QR, as noted later in the release notes. Multi-GPU eigenvalue routines are also under development. I will update that first sentence of the release notes, though. Thanks.

-mark

Hi,
I'd like to know if Magma has multi-gpu support for matrix-matrix product (SGEMM).
Presently I use Cublas, but in Cublas sgemm is only mono-gpu.
Thank you

[mgates3]

No, a multi-GPU GEMM is not specifically provided in MAGMA. What size and distribution of the A, B, C matrices are you looking for?

Internally we have effectively done multi-GPU GEMM in several codes, based on cublas GEMM. The implementation depends on how your data is distributed. For C = A*B, the easiest is if A is duplicated on all GPUs, while B and C are distributed by block columns, then

Code: Select all

[ A ][ B1  B2  B3 ] = [ A*B1  A*B2  A*B3 ]


So on all GPUs, create products Ci = A*Bi. (This is done in the LU code, zgetrf.) Conveniently, each GPU needs only one GEMM, even if it has multiple blocks of columns.

Or if A and C are distributed by block rows, while B is duplicated on all GPUs, then

Code: Select all

[ A1 ]         [ A1*B ]
[ A2 ] [ B ] = [ A2*B ]
[ A3 ]         [ A3*B ]


So on all GPUs, create products Ci = Ai*B.

If A is stored block-column cyclically and B is stored block-row cyclically, then

Code: Select all

               [ B1 ]
[ A1  A2  A3 ] [ B2 ] = A1*B1 + A2*B2 + A3*B3
               [ B3 ]


So on all GPUs, create products Ai*Bi, then do a sum reduction (e.g., on the CPU) to get the final result. More general distributions require more communication between the GPUs.

Hope that helps.
-mark

[neuron]

Thank you,
can you give me a pointer to some of your multi-gpu GEMM implementations?

[mgates3]

They are not implemented as separate functions, but are part of the multi-GPU codes such as LU and QR, so I don't think they will be much use for you. For instance, see zgetrf_mgpu.cpp, the loop that ends with "end of gpu updates".
-mark