Multicore and MultiGPU use of MAGMA

[fletchjp]

I have been working with MAGMA on a system with 8 Intel CPU cores and 8 Gbytes of RAM and one GPU (NVIDIA GTX 460 with 2 Gbytes memory). I now have a second system, the same except that the CPU side has 16 Gbytes and the GTX 460 only 1 Gbyte memory as I could not get a second card the same as the first!! I am currently running MAGMA with a 4 core version of gotoBLAS (as the 8 core version is buggy).

I want to use this system to test schemes for improving the throughput of calculations which presently are run using ScaLAPACK on systems without a GPU, to assist in specifying a system for large matrix problems which need large CPU memories.

In that context I have the following questions:

1. Does MAGMA support two tasks on the same GPU, running together with the 4 core gotoBLAS and therefore sharing the GPU and using 4 cores each?
2. How far does MAGMA 1.1.0 help me in working on a system with GPU's attached to different sets of CPU cores?
3. Are you intending to link MAGMA to ScaLAPACK in order to speed up the different single process units within SCALAPACK. If so could that work with 2 tasks on a given set of cores, sharing the GPU as in Q1?

Any thoughts on this, including directions to other relevant software would be appreciated.

John

[fletchjp]

Since posting the above I have been experimenting with PLASMA as well. That seems to be really designed for large shared memory systems and to offer little advantage on my small system. Also, it does not enable me to share a matrix across more than one system in the way ScaLAPACK does. I can use ScaLAPACK to run problems on my two computers, but that does not use the GPU. What would be really interesting would be to be able to enhance PDGESV, PDGETRF and PDGETRS and the Z equivalents to use the GPU as well. It may be that quite a lot of that can be done by getting into the level where ScaLAPACK is using LAPACK and changing the calls.

Has anyone else the same interest?

Happy New Year

John

[Sidio47]

John,

I, too, am interested in what will be happening when we talk about mixing the local (shared memory) heterogeneous architecture of CPU+GPU with the non-local (distributed memory) architecture. All of the HPC's will be going this route (as the introduction pages of PLASMA and MAGMA elude to). I use ScaLAPACK right now and I am definitely interested in knowing what the future of ScaLAPACK-type implementations will entail.

Cheers,
Cyrus

[fletchjp]

Hello there. I posted this question more than a month ago and I really would like an answer.

I do know now from a reading of "CUDA application design and development" by Rob Farber a good deal more about how to send more than one task to a GPU. I am using his examples and I have one working to send several tasks to a GPU from all of a set of tasks running under OpenMPI. I have also adapted one of the Scalapack 2 examples in FORTRAN to call a CUDA C routine and do the same tasks. That could be adapted so that the GPU would set up the array data for the Scalapack task. It would be nice to have the Scalapack calls also have the option to use MAGMA to speed up the Scalapack use of matrix calculations on each process. I see that as a general purpose action. That is why I am asking you whether you are thinking in the same way.

Both the above examples also work when I run them on two computers each with a GPU. The only difference is the hostfile.

Thank you

John

[mgates3]

Hi John,

Both of these goals -- running multiple problems simultaneously on a single GPU and running MAGMA distributed across several GPUs on several nodes -- are of interest to the MAGMA team. We are moving in that direction, with MAGMA 1.1 adding support for streams in the MAGMA BLAS and support for multiple GPU within a single node with LU, Cholesky, and QR. (We are currently working on multi-GPU eigenvalue code, too.) However, we are not quite at the point of achieving your goals.

-mark

[fletchjp]

Mark

Thank you, it is good to know that you have some work in progress. You talk of

running MAGMA distributed across several GPUs on several nodes

without saying how this may be achieved.

Our problems are ones which can generate matrices too big to fit in the memory of even a modern CPU system with a large memory e.g. 16 Gbytes. So I want to harness more that one and at present I use Scalapack for this. We want to gain the advantage of the GPU for these large problems. I know now, largely from work with my recently acquired copy of "CUDA Application Design and Development", that I can combine CUDA and MPI and run e.g. 4 tasks on 2 CPUs each with a GPU and have the GPU kernels run work for more than one MPI task. That opens the possibility for Scalapack to take advantage of this.

I hope that you have this somewhere on your route map, and that is what I am asking. Do you envisage this as (a) worthwhile and (b) are your team going to do it or (c) is someone else going to do it and make it available?

An alternative would be for MAGMA to achieve its objective separately from Scalapack, by having its own version of e.g. BLACS. There would have to be some way to distribute the matrix across different tasks.

We are end users and not CUDA specialists, so I am reluctant to take this on here.

If we know it is coming then that will affect our choices for new hardware for our problems.

It may be that I just have to be patient.

John

[mgates3]

John,

I would say distributed use of MAGMA is of interest to us, but we don't currently have any code that does this, so you may have to wait a while for that functionality. Hooking MAGMA into Scalapack is one potential route. Another route could be using DAGue, which currently extends PLASMA to distributed systems:

http://icl.cs.utk.edu/dague/
http://icl.cs.utk.edu/plasma/

-mark

[fletchjp]

Thank you I will look into DAGue.

Unfortunately it has the following statement on http://icl.cs.utk.edu/dague/software/index.html :

- the GPU supports has been temporarily disabled.

(sic)

I had not heard of it.

John

[fletchjp]

Is there any progress on this topic, even an indication of what is being considered would be helpful?

John

[fletchjp]

Hello

Is there any more information on this topic?

Thanks

John