Hello Desheng,
the patch is in revision mode. After tested it on 64bit platform,
here is a better revision:
LIWMIN = NUMROC( DESCA( M_ ) + DESCA( MB_ ) * NPROW
$ + MOD ( IA  1, DESCA( MB_ ) ), DESCA ( NB_ ),
$ MYCOL, DESCA( CSRC_ ), NPCOL ) +
$ MAX ( DESCA( MB_ ) * ICEIL ( ICEIL(
$ NUMROC( DESCA( M_ ) + DESCA( MB_) * NPROW,
$ DESCA( MB_ ), MYROW, DESCA( RSRC_ ), NPROW ),
$ DESCA( MB_ ) ), LCM / NPROW ), DESCA( NB_) )
and the pdgetri.f with the modification is attached.
You can use it right now and this should replace my previous email.
Still testing this. Please let me know if that works on your side.
Julien.
On Thu, 4 May 2006, Julien Langou wrote:
Hello Desheng,
yes there seems to have a problem with PxGETRI with rectangular grids when
one uses workspace query to get the optimal size of IWORK. The value returned
in IWORK(1) after a workspace query (LWORK=1) is too small. The
documentation gives the same workspace size as the one returned, so the
documentation of the routine seems to be incorrect as well.
Here is a temporary fix. It will need more testing to be validated but this
works fine on one of our cluster. Let me know if this works as well for you.
Replace Line 221222 of pdgetri.f  LIWMIN = NQ + MAX( ICEIL(
ICEIL( MP, DESCA( MB_ ) ),
 $ LCM / NPROW ), DESCA( NB_ ) )
with the following seven lines
+ LIWMIN = NUMROC( DESCA( M_ ) + DESCA( MB_ ) * NPROW
+ $ + MOD ( IA  1, MB_P ), DESCA ( NB_ ), MYCOL,
+ $ DESCA( CSRC_ ), NPCOL ) + MAX ( DESCA( MB_ )
+ $ * ICEIL ( ICEIL( NUMROC( DESCA( M_ )
+ $ + DESCA( MB_ ) * NPROW, DESCA( MB_ ), MYROW,
+ $ DESCA( RSRC_ ), NPROW ), DESCA( MB_ ) ),
+ $ LCM / NPROW ), DESCA( NB_ ) )
I have tested matrices of size n=[550], with nb=[13 32] for all the grids
possible with less than 16 processors ( 11, 12, 21, 13, 31, 14, 22,
41, 15, .... , 161), the routine PDGETRI with this modification works
fine, whereas the previous one was only working on square matrices if one was
using the minimum LIWORK. You were right.
So, temporary fix, you compile pdgetri.f with your fortran compiler, link
with scalapack but put pdgetri.o before the libscalapack.a library and this
should work.
We'll take a bit of time here to have a second look and certainly release a
more official patch soon.
Feel free for question, remarks, comments.
Julien
 next part 
SUBROUTINE PDGETRI( N, A, IA, JA, DESCA, IPIV, WORK, LWORK,
$ IWORK, LIWORK, INFO )
*
*  ScaLAPACK routine (version 1.7) 
* University of Tennessee, Knoxville, Oak Ridge National Laboratory,
* and University of California, Berkeley.
* May 1, 1997
*
* .. Scalar Arguments ..
INTEGER IA, INFO, JA, LIWORK, LWORK, N
* ..
* .. Array Arguments ..
INTEGER DESCA( * ), IPIV( * ), IWORK( * )
DOUBLE PRECISION A( * ), WORK( * )
* ..
*
* Purpose
* =======
*
* PDGETRI computes the inverse of a distributed matrix using the LU
* factorization computed by PDGETRF. This method inverts U and then
* computes the inverse of sub( A ) = A(IA:IA+N1,JA:JA+N1) denoted
* InvA by solving the system InvA*L = inv(U) for InvA.
*
* Notes
* =====
*
* Each global data object is described by an associated description
* vector. This vector stores the information required to establish
* the mapping between an object element and its corresponding process
* and memory location.
*
* Let A be a generic term for any 2D block cyclicly distributed array.
* Such a global array has an associated description vector DESCA.
* In the following comments, the character _ should be read as
* "of the global array".
*
* NOTATION STORED IN EXPLANATION
*   
* DTYPE_A(global) DESCA( DTYPE_ )The descriptor type. In this case,
* DTYPE_A = 1.
* CTXT_A (global) DESCA( CTXT_ ) The BLACS context handle, indicating
* the BLACS process grid A is distribu
* ted over. The context itself is glo
* bal, but the handle (the integer
* value) may vary.
* M_A (global) DESCA( M_ ) The number of rows in the global
* array A.
* N_A (global) DESCA( N_ ) The number of columns in the global
* array A.
* MB_A (global) DESCA( MB_ ) The blocking factor used to distribute
* the rows of the array.
* NB_A (global) DESCA( NB_ ) The blocking factor used to distribute
* the columns of the array.
* RSRC_A (global) DESCA( RSRC_ ) The process row over which the first
* row of the array A is distributed.
* CSRC_A (global) DESCA( CSRC_ ) The process column over which the
* first column of the array A is
* distributed.
* LLD_A (local) DESCA( LLD_ ) The leading dimension of the local
* array. LLD_A >= MAX(1,LOCr(M_A)).
*
* Let K be the number of rows or columns of a distributed matrix,
* and assume that its process grid has dimension p x q.
* LOCr( K ) denotes the number of elements of K that a process
* would receive if K were distributed over the p processes of its
* process column.
* Similarly, LOCc( K ) denotes the number of elements of K that a
* process would receive if K were distributed over the q processes of
* its process row.
* The values of LOCr() and LOCc() may be determined via a call to the
* ScaLAPACK tool function, NUMROC:
* LOCr( M ) = NUMROC( M, MB_A, MYROW, RSRC_A, NPROW ),
* LOCc( N ) = NUMROC( N, NB_A, MYCOL, CSRC_A, NPCOL ).
* An upper bound for these quantities may be computed by:
* LOCr( M ) <= ceil( ceil(M/MB_A)/NPROW )*MB_A
* LOCc( N ) <= ceil( ceil(N/NB_A)/NPCOL )*NB_A
*
* Arguments
* =========
*
* N (global input) INTEGER
* The number of rows and columns to be operated on, i.e. the
* order of the distributed submatrix sub( A ). N >= 0.
*
* A (local input/local output) DOUBLE PRECISION pointer into the
* local memory to an array of dimension (LLD_A,LOCc(JA+N1)).
* On entry, the local pieces of the L and U obtained by the
* factorization sub( A ) = P*L*U computed by PDGETRF. On
* exit, if INFO = 0, sub( A ) contains the inverse of the
* original distributed matrix sub( A ).
*
* IA (global input) INTEGER
* The row index in the global array A indicating the first
* row of sub( A ).
*
* JA (global input) INTEGER
* The column index in the global array A indicating the
* first column of sub( A ).
*
* DESCA (global and local input) INTEGER array of dimension DLEN_.
* The array descriptor for the distributed matrix A.
*
* IPIV (local input) INTEGER array, dimension LOCr(M_A)+MB_A
* keeps track of the pivoting information. IPIV(i) is the
* global row index the local row i was swapped with. This
* array is tied to the distributed matrix A.
*
* WORK (local workspace/local output) DOUBLE PRECISION array,
* dimension (LWORK)
* On exit, WORK(1) returns the minimal and optimal LWORK.
*
* LWORK (local or global input) INTEGER
* The dimension of the array WORK.
* LWORK is local input and must be at least
* LWORK = LOCr(N+MOD(IA1,MB_A))*NB_A. WORK is used to keep a
* copy of at most an entire column block of sub( A ).
*
* If LWORK = 1, then LWORK is global input and a workspace
* query is assumed; the routine only calculates the minimum
* and optimal size for all work arrays. Each of these
* values is returned in the first entry of the corresponding
* work array, and no error message is issued by PXERBLA.
*
* IWORK (local workspace/local output) INTEGER array,
* dimension (LIWORK)
* On exit, IWORK(1) returns the minimal and optimal LIWORK.
*
* LIWORK (local or global input) INTEGER
* The dimension of the array IWORK used as workspace for
* physically transposing the pivots.
* LIWORK is local input and must be at least
* if NPROW == NPCOL then
* LIWORK = LOCc( N_A + MOD(JA1, NB_A) ) + NB_A,
* else
* LIWORK = LOCc( N_A + MOD(JA1, NB_A) ) +
* MAX( CEIL(CEIL(LOCr(M_A)/MB_A)/(LCM/NPROW)),
* NB_A )
* where LCM is the least common multiple of process
* rows and columns (NPROW and NPCOL).
* end if
*
* If LIWORK = 1, then LIWORK is global input and a workspace
* query is assumed; the routine only calculates the minimum
* and optimal size for all work arrays. Each of these
* values is returned in the first entry of the corresponding
* work array, and no error message is issued by PXERBLA.
*
* INFO (global output) INTEGER
* = 0: successful exit
* < 0: If the ith argument is an array and the jentry had
* an illegal value, then INFO = (i*100+j), if the ith
* argument is a scalar and had an illegal value, then
* INFO = i.
* > 0: If INFO = K, U(IA+K1,IA+K1) is exactly zero; the
* matrix is singular and its inverse could not be
* computed.
*
* =====================================================================
*
* .. Parameters ..
INTEGER BLOCK_CYCLIC_2D, CSRC_, CTXT_, DLEN_, DTYPE_,
$ LLD_, MB_, M_, NB_, N_, RSRC_
PARAMETER ( BLOCK_CYCLIC_2D = 1, DLEN_ = 9, DTYPE_ = 1,
$ CTXT_ = 2, M_ = 3, N_ = 4, MB_ = 5, NB_ = 6,
$ RSRC_ = 7, CSRC_ = 8, LLD_ = 9 )
DOUBLE PRECISION ZERO, ONE
PARAMETER ( ZERO = 0.0D+0, ONE = 1.0D+0 )
* ..
* .. Local Scalars ..
LOGICAL LQUERY
INTEGER I, IACOL, IAROW, ICOFF, ICTXT, IROFF, IW, J,
$ JB, JN, LCM, LIWMIN, LWMIN, MP, MYCOL, MYROW,
$ NN, NP, NPCOL, NPROW, NQ
* ..
* .. Local Arrays ..
INTEGER DESCW( DLEN_ ), IDUM1( 2 ), IDUM2( 2 )
* ..
* .. External Subroutines ..
EXTERNAL BLACS_GRIDINFO, CHK1MAT, DESCSET, PCHK1MAT,
$ PDGEMM, PDLACPY, PDLASET, PDLAPIV,
$ PDTRSM, PDTRTRI, PXERBLA
* ..
* .. External Functions ..
INTEGER ICEIL, ILCM, INDXG2P, NUMROC
EXTERNAL ICEIL, ILCM, INDXG2P, NUMROC
* ..
* .. Intrinsic Functions ..
INTRINSIC DBLE, MAX, MIN, MOD
* ..
* .. Executable Statements ..
*
* Get grid parameters
*
ICTXT = DESCA( CTXT_ )
CALL BLACS_GRIDINFO( ICTXT, NPROW, NPCOL, MYROW, MYCOL )
*
* Test the input parameters
*
INFO = 0
IF( NPROW.EQ.1 ) THEN
INFO = (500+CTXT_)
ELSE
CALL CHK1MAT( N, 1, N, 1, IA, JA, DESCA, 5, INFO )
IF( INFO.EQ.0 ) THEN
IROFF = MOD( IA1, DESCA( MB_ ) )
ICOFF = MOD( JA1, DESCA( NB_ ) )
IAROW = INDXG2P( IA, DESCA( MB_ ), MYROW, DESCA( RSRC_ ),
$ NPROW )
NP = NUMROC( N+IROFF, DESCA( MB_ ), MYROW, IAROW, NPROW )
LWMIN = NP * DESCA( NB_ )
*
MP = NUMROC( DESCA( M_ ), DESCA( MB_ ), MYROW,
$ DESCA( RSRC_ ), NPROW )
NQ = NUMROC( DESCA( N_ ), DESCA( NB_ ), MYCOL,
$ DESCA( CSRC_ ), NPCOL )
IF( NPROW.EQ.NPCOL ) THEN
LIWMIN = NQ + DESCA( NB_ )
ELSE
LCM = ILCM( NPROW, NPCOL )
LIWMIN = NUMROC( DESCA( M_ ) + DESCA( MB_) * NPROW
$ + MOD ( IA  1, DESCA( MB_ ) ), DESCA ( NB_ ),
$ MYCOL, DESCA( CSRC_ ), NPCOL ) +
$ MAX ( DESCA( MB_ ) * ICEIL ( ICEIL(
$ NUMROC( DESCA( M_ ) + DESCA( MB_) * NPROW,
$ DESCA( MB_ ), MYROW, DESCA( RSRC_ ), NPROW ),
$ DESCA( MB_ )), LCM / NPROW ), DESCA( NB_))
END IF
*
WORK( 1 ) = DBLE( LWMIN )
IWORK( 1 ) = LIWMIN
LQUERY = ( LWORK.EQ.1 .OR. LIWORK.EQ.1 )
IF( IROFF.NE.ICOFF .OR. IROFF.NE.0 ) THEN
INFO = 4
ELSE IF( DESCA( MB_ ).NE.DESCA( NB_ ) ) THEN
INFO = (500+NB_)
ELSE IF( LWORK.LT.LWMIN .AND. .NOT.LQUERY ) THEN
INFO = 8
ELSE IF( LIWORK.LT.LIWMIN .AND. .NOT.LQUERY ) THEN
INFO = 10
END IF
END IF
IF( LWORK.EQ.1 ) THEN
IDUM1( 1 ) = 1
ELSE
IDUM1( 1 ) = 1
END IF
IDUM2( 1 ) = 8
IF( LIWORK.EQ.1 ) THEN
IDUM1( 2 ) = 1
ELSE
IDUM1( 2 ) = 1
END IF
IDUM2( 2 ) = 10
CALL PCHK1MAT( N, 1, N, 1, IA, JA, DESCA, 5, 2, IDUM1, IDUM2,
$ INFO )
END IF
*
IF( INFO.NE.0 ) THEN
CALL PXERBLA( ICTXT, 'PDGETRI', INFO )
RETURN
ELSE IF( LQUERY ) THEN
RETURN
END IF
*
* Quick return if possible
*
IF( N.EQ.0 )
$ RETURN
*
* Form inv(U). If INFO > 0 from PDTRTRI, then U is singular,
* and the inverse is not computed.
*
CALL PDTRTRI( 'Upper', 'Nonunit', N, A, IA, JA, DESCA, INFO )
IF( INFO.GT.0 )
$ RETURN
*
* Define array descriptor for working array WORK
*
JN = MIN( ICEIL( JA, DESCA( NB_ ) ) * DESCA( NB_ ), JA+N1 )
NN = ( ( JA+N2 ) / DESCA( NB_ ) ) * DESCA( NB_ ) + 1
IACOL = INDXG2P( NN, DESCA( NB_ ), MYCOL, DESCA( CSRC_ ), NPCOL )
CALL DESCSET( DESCW, N+IROFF, DESCA( NB_ ), DESCA( MB_ ),
$ DESCA( NB_ ), IAROW, IACOL, ICTXT, MAX( 1, NP ) )
IW = IROFF + 1
*
* Solve the equation inv(A)*L=inv(U) for inv(A) using blocked code.
*
DO 10 J = NN, JN+1, DESCA( NB_ )
JB = MIN( DESCA( NB_ ), JA+NJ )
I = IA + J  JA
*
* Copy current block column of L to WORK and replace with zeros.
*
CALL PDLACPY( 'Lower', JA+N1J, JB, A, I+1, J, DESCA,
$ WORK, IW+JJA+1, 1, DESCW )
CALL PDLASET( 'Lower', JA+N1J, JB, ZERO, ZERO, A, I+1, J,
$ DESCA )
*
* Compute current block column of inv(A).
*
IF( J+JB.LE.JA+N1 )
$ CALL PDGEMM( 'No transpose', 'No transpose', N, JB,
$ JA+NJJB, ONE, A, IA, J+JB, DESCA, WORK,
$ IW+J+JBJA, 1, DESCW, ONE, A, IA, J, DESCA )
CALL PDTRSM( 'Right', 'Lower', 'No transpose', 'Unit', N, JB,
$ ONE, WORK, IW+JJA, 1, DESCW, A, IA, J, DESCA )
DESCW( CSRC_ ) = MOD( DESCW( CSRC_ ) + NPCOL  1, NPCOL )
*
10 CONTINUE
*
* Handle the last block of columns separately
*
JB = JNJA+1
*
* Copy current block column of L to WORK and replace with zeros.
*
CALL PDLACPY( 'Lower', N1, JB, A, IA+1, JA, DESCA, WORK, IW+1,
$ 1, DESCW )
CALL PDLASET( 'Lower', N1, JB, ZERO, ZERO, A, IA+1, JA, DESCA )
*
* Compute current block column of inv(A).
*
IF( JA+JB.LE.JA+N1 )
$ CALL PDGEMM( 'No transpose', 'No transpose', N, JB,
$ NJB, ONE, A, IA, JA+JB, DESCA, WORK, IW+JB, 1,
$ DESCW, ONE, A, IA, JA, DESCA )
CALL PDTRSM( 'Right', 'Lower', 'No transpose', 'Unit', N, JB,
$ ONE, WORK, IW, 1, DESCW, A, IA, JA, DESCA )
*
* Use the row pivots and apply them to the columns of the global
* matrix.
*
CALL DESCSET( DESCW, DESCA( M_ ) + DESCA( MB_ )*NPROW, 1,
$ DESCA( MB_ ), 1, DESCA( RSRC_ ), MYCOL, ICTXT,
$ MP+DESCA( MB_ ) )
CALL PDLAPIV( 'Backward', 'Columns', 'Column', N, N, A, IA,
$ JA, DESCA, IPIV, IA, 1, DESCW, IWORK )
*
WORK( 1 ) = DBLE( LWMIN )
IWORK( 1 ) = LIWMIN
*
RETURN
*
* End of PDGETRI
*
END
