Support for matrices > 2GB in size

Open forum for general discussions relating to PLASMA.

Support for matrices > 2GB in size

Postby mtspurrier » Mon Jan 31, 2011 4:53 pm

I am able to run plasma_zgeqrf with a matrix that is 73728x1800 (1.977539 GB), but when I run a matrix that is 73728x1825 (2.005005 GB) it fails because it cannot allocate the memory. I am calling the routine from Fortran.

PLASMA ERROR: plasma_desc_mat_alloc(): malloc() failed
PLASMA ERROR: PLASMA_zgeqrf(): plasma_shared_alloc() failed
QRF INFO= -106

We have 132 GB of memory on the system and no one else is on it at the moment so there is plenty of memory free. There is likely something defined as an int when it should be something larger?
mtspurrier
 
Posts: 3
Joined: Thu Dec 09, 2010 8:54 am

Re: Support for matrices > 2GB in size

Postby mateo70 » Mon Jan 31, 2011 5:11 pm

Hi,

could you give us more information about the system and the compiler you are using ?
And can you try to replace the line control/descriptor.c:152 by the following one and tell me if you still have the problem:
Code: Select all
    size = (size_t)desc->lm * (size_t)desc->ln * (size_t)plasma_element_size(desc->dtyp);


By the way, with a shape of matrix like yours, maybe you will be interested by using a CAQR algorithm instead of QR. For that, after your call to PLASMA_Init and before the call to PLASMA_zgeqrf, you need to add the two following lines :
Code: Select all
        PLASMA_Set( PLASMA_HOUSEHOLDER_MODE, PLASMA_TREE_HOUSEHOLDER );
        PLASMA_Set( PLASMA_HOUSEHOLDER_SIZE, nbtileperdomain );


Regards,
Mathieu
mateo70
 
Posts: 92
Joined: Fri May 07, 2010 3:48 pm

Re: Support for matrices > 2GB in size

Postby mtspurrier » Tue Feb 01, 2011 12:40 pm

We are running on an SGI Altix UV 100 with 4 8-core Xeon 7550 processors. It has 132 GB of memory. We are using Intel's Composer XE 2011.

I made the change to control/descriptor.c and that appears to fix the issue. Thanks!

Thank you for the information on CAQR. I thought calling zgeqrf automatically did that for you. Is this usage in any of the documentation yet? What is a good value for nbtileperdomain? The integer values need to be added to plasmaf.h too.

We are actually trying to run with taller and skinnier matrices, but found that there is no performance gain. Perhaps the nbtileperdomain will help. There just didn't seem to be enough column tiles.

Thanks for the assistance.
mtspurrier
 
Posts: 3
Joined: Thu Dec 09, 2010 8:54 am

Re: Support for matrices > 2GB in size

Postby mateo70 » Tue Feb 01, 2011 1:07 pm

Thanks for the information, I will include that in the next release (bug fix and fortran interface). We will also include in the documentation how to use the CAQR algorithm.

Concerning the choice of nbtileperdomain, if you put 1 , you will have a binary tree and if you put the number of tile of your matrix, it will be a flat tree and so, the same algorithm than the classic tile-QR. A good choice is probably to fix it to have as many domains as your number of sockets or NUMA node, or even two or three domains per instance. It really depends on your architecture and your test case.
mateo70
 
Posts: 92
Joined: Fri May 07, 2010 3:48 pm


Return to User discussion

Who is online

Users browsing this forum: No registered users and 1 guest