Support for matrices > 2GB in size

[mtspurrier]

I am able to run plasma_zgeqrf with a matrix that is 73728x1800 (1.977539 GB), but when I run a matrix that is 73728x1825 (2.005005 GB) it fails because it cannot allocate the memory. I am calling the routine from Fortran.

PLASMA ERROR: plasma_desc_mat_alloc(): malloc() failed
PLASMA ERROR: PLASMA_zgeqrf(): plasma_shared_alloc() failed
QRF INFO= -106

We have 132 GB of memory on the system and no one else is on it at the moment so there is plenty of memory free. There is likely something defined as an int when it should be something larger?

[mateo70]

Hi,

could you give us more information about the system and the compiler you are using ?
And can you try to replace the line control/descriptor.c:152 by the following one and tell me if you still have the problem:

Code: Select all

    size = (size_t)desc->lm * (size_t)desc->ln * (size_t)plasma_element_size(desc->dtyp);


By the way, with a shape of matrix like yours, maybe you will be interested by using a CAQR algorithm instead of QR. For that, after your call to PLASMA_Init and before the call to PLASMA_zgeqrf, you need to add the two following lines :

Code: Select all

        PLASMA_Set( PLASMA_HOUSEHOLDER_MODE, PLASMA_TREE_HOUSEHOLDER );
        PLASMA_Set( PLASMA_HOUSEHOLDER_SIZE, nbtileperdomain );


Regards,
Mathieu

[mtspurrier]

We are running on an SGI Altix UV 100 with 4 8-core Xeon 7550 processors. It has 132 GB of memory. We are using Intel's Composer XE 2011.

I made the change to control/descriptor.c and that appears to fix the issue. Thanks!

Thank you for the information on CAQR. I thought calling zgeqrf automatically did that for you. Is this usage in any of the documentation yet? What is a good value for nbtileperdomain? The integer values need to be added to plasmaf.h too.

We are actually trying to run with taller and skinnier matrices, but found that there is no performance gain. Perhaps the nbtileperdomain will help. There just didn't seem to be enough column tiles.

Thanks for the assistance.

[mateo70]

Thanks for the information, I will include that in the next release (bug fix and fortran interface). We will also include in the documentation how to use the CAQR algorithm.

Concerning the choice of nbtileperdomain, if you put 1 , you will have a binary tree and if you put the number of tile of your matrix, it will be a flat tree and so, the same algorithm than the classic tile-QR. A good choice is probably to fix it to have as many domains as your number of sockets or NUMA node, or even two or three domains per instance. It really depends on your architecture and your test case.