Magma 1.2.1 Release Notes and Multi-GPU
-
- Posts: 3
- Joined: Fri Jul 27, 2012 3:11 am
Magma 1.2.1 Release Notes and Multi-GPU
All,
I just downloaded Magma 1.2.1. I always had a notion that MAGMA is about accelerating Lacpack/blas routines on all CPU and GPU cores available in the system. However, 1.2.1. Release notes begins with:
"
MAGMA Release Notes
-----------------------------------------------------
MAGMA is intended for a single CUDA enabled NVIDIA GPU. It supports
Tesla and Fermi GPUs. For more details see the MAGMA 1.0 presentation.
....
"
Now, Does not MAGMA support multi-GPUs? In particular, I am looking @ LU dense Factorization, Matrix inversion, SVD
Thanks for any help,
Best Regards,
Kuruvinandan
I just downloaded Magma 1.2.1. I always had a notion that MAGMA is about accelerating Lacpack/blas routines on all CPU and GPU cores available in the system. However, 1.2.1. Release notes begins with:
"
MAGMA Release Notes
-----------------------------------------------------
MAGMA is intended for a single CUDA enabled NVIDIA GPU. It supports
Tesla and Fermi GPUs. For more details see the MAGMA 1.0 presentation.
....
"
Now, Does not MAGMA support multi-GPUs? In particular, I am looking @ LU dense Factorization, Matrix inversion, SVD
Thanks for any help,
Best Regards,
Kuruvinandan
Re: Magma 1.2.1 Release Notes and Multi-GPU
Yes, MAGMA supports multiple GPUs for certain routines. Currently, the release has multi-GPU support for LU, Cholesky, QR, as noted later in the release notes. Multi-GPU eigenvalue routines are also under development. I will update that first sentence of the release notes, though. Thanks.
-mark
-mark
-
- Posts: 3
- Joined: Fri Jul 27, 2012 3:11 am
Re: Magma 1.2.1 Release Notes and Multi-GPU
Thank you Mark! It was very helpful.
I hope the SVD ones are also multi-GPU enabled. Thanks,
I hope the SVD ones are also multi-GPU enabled. Thanks,
Re: Magma 1.2.1 Release Notes and Multi-GPU
Hi,mgates3 wrote:Yes, MAGMA supports multiple GPUs for certain routines. Currently, the release has multi-GPU support for LU, Cholesky, QR, as noted later in the release notes. Multi-GPU eigenvalue routines are also under development. I will update that first sentence of the release notes, though. Thanks.
-mark
I'd like to know if Magma has multi-gpu support for matrix-matrix product (SGEMM).
Presently I use Cublas, but in Cublas sgemm is only mono-gpu.
Thank you
Re: Magma 1.2.1 Release Notes and Multi-GPU
No, a multi-GPU GEMM is not specifically provided in MAGMA. What size and distribution of the A, B, C matrices are you looking for?
Internally we have effectively done multi-GPU GEMM in several codes, based on cublas GEMM. The implementation depends on how your data is distributed. For C = A*B, the easiest is if A is duplicated on all GPUs, while B and C are distributed by block columns, then
So on all GPUs, create products Ci = A*Bi. (This is done in the LU code, zgetrf.) Conveniently, each GPU needs only one GEMM, even if it has multiple blocks of columns.
Or if A and C are distributed by block rows, while B is duplicated on all GPUs, then
So on all GPUs, create products Ci = Ai*B.
If A is stored block-column cyclically and B is stored block-row cyclically, then
So on all GPUs, create products Ai*Bi, then do a sum reduction (e.g., on the CPU) to get the final result. More general distributions require more communication between the GPUs.
Hope that helps.
-mark
Internally we have effectively done multi-GPU GEMM in several codes, based on cublas GEMM. The implementation depends on how your data is distributed. For C = A*B, the easiest is if A is duplicated on all GPUs, while B and C are distributed by block columns, then
Code: Select all
[ A ][ B1 B2 B3 ] = [ A*B1 A*B2 A*B3 ]
Or if A and C are distributed by block rows, while B is duplicated on all GPUs, then
Code: Select all
[ A1 ] [ A1*B ]
[ A2 ] [ B ] = [ A2*B ]
[ A3 ] [ A3*B ]
If A is stored block-column cyclically and B is stored block-row cyclically, then
Code: Select all
[ B1 ]
[ A1 A2 A3 ] [ B2 ] = A1*B1 + A2*B2 + A3*B3
[ B3 ]
Hope that helps.
-mark
Re: Magma 1.2.1 Release Notes and Multi-GPU
Thank you,
can you give me a pointer to some of your multi-gpu GEMM implementations?
can you give me a pointer to some of your multi-gpu GEMM implementations?
Re: Magma 1.2.1 Release Notes and Multi-GPU
They are not implemented as separate functions, but are part of the multi-GPU codes such as LU and QR, so I don't think they will be much use for you. For instance, see zgetrf_mgpu.cpp, the loop that ends with "end of gpu updates".
-mark
-mark