ScaLAPACK Archives

[Scalapack] Possible bug in SCALAPACK/PBLAS

Dear SCALAPACK development team:

I have been trying use the SCALAPACK routine PCTREVC recently with no success. 
My program crashes with the following error log when the PBLAS routine PSCASUM 
is called by PCTREVC. 

On examining the source code of the PSCASUM routine, I noticed that the format 
of the descriptor that PSCASUM expects is different from the standard 
descriptor format used elsewhere in SCALAPACK. The format of the descriptor 
array is given at the beginning of the PSCASUM source file. This is not the 
same format that is used elsewhere in SCALAPACK. For example, see the 
descriptor format that is shown in the source code of PCTREVC. 

The size of the descriptor array that PSCASUM expects is 11, whereas the size 
of the standard descriptor used elsewhere in SCALAPACK is 9. I believe this 
mismatch is what is causing PSCASUM to crash. I examined the source codes of 
few other routines in PBLAS and it seems that they are also expecting the same 
descriptor format which PSCASUM is expecting.

FYI, my SCALAPACK library version is 2.0.1, LAPACK library version is 3.4.0 and 
PBLAS library version is 2.0.

Could you help me solve this problem? I really need to get PCTREVC running.

I will greatly appreciate any help you can provide about this matter.

Ali Uzun


ERROR LOG:

PBLAS ERROR 'Parameter number 602 had an illegal value'
from {-1,-1}, pnum=-1, Contxt=0, in routine 'PSCASUM'.

{-1,-1}, pnum=5632, Contxt=0, killed other procs, exiting with error #-602.

PBLAS ERROR 'Parameter number 602 had an illegal value'
from {-1,-1}, pnum=-1, Contxt=0, in routine 'PSCASUM'.

{-1,-1}, pnum=1024, Contxt=0, killed other procs, exiting with error #-602.

aborting job:
Fatal error in PMPI_Waitall: Invalid MPI_Request, error stack:
PMPI_Waitall(274): MPI_Waitall(count=1, req_array=0xb7ae70, 
status_array=0x2aaab2ef8010) failed
PMPI_Waitall(250): The supplied request in array element 0 was invalid (kind=15)
aborting job:
Fatal error in PMPI_Waitall: Invalid MPI_Request, error stack:
PMPI_Waitall(274): MPI_Waitall(count=1, req_array=0xb7ae70, 
status_array=0x2aaab2ef8010) failed
PMPI_Waitall(250): The supplied request in array element 0 was invalid (kind=15)
aborting job:
Fatal error in PMPI_Waitall: Invalid MPI_Request, error stack:
PMPI_Waitall(274): MPI_Waitall(count=1, req_array=0xb7ae70, 
status_array=0x2aaab2ef8010) failed
PMPI_Waitall(250): The supplied request in array element 0 was invalid (kind=15)
_pmii_daemon(SIGCHLD): [NID 04497] [c16-3c0s4n1] [Tue Aug 14 09:50:50 2012] PE 
627 exit signal Segmentation fault
_pmii_daemon(SIGCHLD): [NID 04190] [c16-0c2s7n2] [Tue Aug 14 09:50:50 2012] PE 
227 exit signal Segmentation fault
_pmii_daemon(SIGCHLD): [NID 05259] [c20-1c0s2n3] [Tue Aug 14 09:50:50 2012] PE 
2227 exit signal Segmentation fault
_pmii_daemon(SIGCHLD): [NID 05150] [c20-0c0s7n2] [Tue Aug 14 09:50:50 2012] PE 
1827 exit signal Segmentation fault
_pmii_daemon(SIGCHLD): [NID 04516] [c16-3c1s1n0] [Tue Aug 14 09:50:50 2012] PE 
827 exit signal Segmentation fault
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2d728, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13322, comm=0xc4000003, request=0xb7afb0) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2d5e8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13322, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2d5e8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13322, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2d5e8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13322, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2d5e8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13322, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2d5e8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13322, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2d5e8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13322, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2d5e8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13322, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2d5e8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13322, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2d5e8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13322, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2d5e8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13322, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2d5e8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13322, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2d5e8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13322, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2d5e8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13322, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2d5e8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13322, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2d5e8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13322, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2d5e8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13322, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2d5e8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13322, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2d5e8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13322, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2d5e8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13322, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2d5e8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13322, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2d5e8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13322, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
application called MPI_Abort(MPI_COMM_WORLD, -602) - process 5632
_pmii_daemon(SIGCHLD): [NID 12815] [c13-0c0s3n3] [Tue Aug 14 09:50:50 2012] PE 
5632 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 04663] [c18-0c1s5n3] [Tue Aug 14 09:50:50 2012] PE 
1227 exit signal Segmentation fault
aborting job:
application called MPI_Abort(MPI_COMM_WORLD, -602) - process 1024
_pmii_daemon(SIGCHLD): [NID 04537] [c16-3c1s6n1] [Tue Aug 14 09:50:50 2012] PE 
1024 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2f780, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13407, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 04685] [c18-0c2s3n1] [Tue Aug 14 09:50:50 2012] PE 
1427 exit signal Segmentation fault
_pmii_daemon(SIGCHLD): [NID 05408] [c20-2c1s0n0] [Tue Aug 14 09:50:50 2012] PE 
2427 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2f780, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13407, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2f780, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13407, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2f780, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13407, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 11403] [c19-1c0s2n3] [Tue Aug 14 09:50:50 2012] PE 
2627 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 05084] [c18-3c2s7n0] [Tue Aug 14 09:50:50 2012] PE 
1627 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2ee20, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13383, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2ee20, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13383, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2ee20, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13383, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2ee20, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13383, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2ee20, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13383, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2ee20, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13383, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2ee20, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13383, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2ee20, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13383, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
[NID 04537] 2012-08-14 05:50:50 Apid 2595604: initiated application termination
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2ee20, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13383, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2ee20, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13383, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2ee20, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13383, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2ee20, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13383, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc312d8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13477, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc312d8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13477, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc312d8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13477, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2eb00, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13375, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2eb00, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13375, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2dbd8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7afb0) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2d5e8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13322, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2f780, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13407, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 05166] [c20-0c1s3n2] [Tue Aug 14 09:50:50 2012] PE 
2027 exit signal Segmentation fault
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2f8c0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13407, comm=0xc4000003, request=0xb7afb0) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 04752] [c18-1c0s4n0] [Tue Aug 14 09:50:50 2012] PE 
1542 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2f780, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13407, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 04267] [c16-1c1s2n3] [Tue Aug 14 09:50:50 2012] PE 
427 exit signal Segmentation fault
aborting job:
Fatal error in MPI_Recv: Invalid communicator, error stack:
MPI_Recv(186): MPI_Recv(buf=0x29ff570, count=1, MPI_FLOAT, src=1, tag=13406, 
comm=0xc4000003, status=0x2aaab2ef8010) failed
MPI_Recv(101): Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2eb00, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13375, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc316c0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13487, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2f780, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13407, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2f780, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13407, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2ee20, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13383, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2f140, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13391, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2f8c0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13407, comm=0xc4000003, request=0xb7afb0) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2f780, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13407, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc316c0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13487, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc316c0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13487, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc316c0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13487, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30ab8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7afb0) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2d5e8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13322, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30ab8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7afb0) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30400, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13439, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30400, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13439, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30220, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7afb0) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2f780, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13407, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2f780, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13407, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2f780, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13407, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2ee20, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13383, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2ee20, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13383, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 04664] [c18-0c1s6n0] [Tue Aug 14 09:50:50 2012] PE 
1242 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2ee20, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13383, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2f780, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13407, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2f780, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13407, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc316c0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13487, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc316c0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13487, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc316c0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13487, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 13079] [c13-2c0s5n3] [Tue Aug 14 09:50:50 2012] PE 
7799 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30400, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13439, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30400, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13439, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2f780, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13407, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2f780, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13407, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2f780, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13407, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2ee20, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13383, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2f780, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13407, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Recv: Invalid communicator, error stack:
MPI_Recv(186): MPI_Recv(buf=0x29ff570, count=1, MPI_FLOAT, src=1, tag=13390, 
comm=0xc4000003, status=0x2aaab2ef8010) failed
MPI_Recv(101): Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc316c0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13487, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30400, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13439, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2f780, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13407, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2ee20, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13383, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2f140, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13391, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2f140, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13391, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc305c8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30b80, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7afb0) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc312d8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13477, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc312d8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13477, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30400, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13439, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc312d8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13477, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2ef60, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13383, comm=0xc4000003, request=0xb7afb0) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2ee20, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13383, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2f140, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13391, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc312d8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13477, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc312d8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13477, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc312d8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13477, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc312d8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13477, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc312d8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13477, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2ee20, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13383, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2ee20, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13383, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc312d8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13477, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc312d8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13477, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc312d8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13477, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc312d8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13477, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc310f8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7afb0) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30220, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7afb0) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 04260] [c16-1c1s1n0] [Tue Aug 14 09:50:50 2012] PE 
342 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc312d8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13477, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc312d8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13477, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc312d8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13477, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc312d8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13477, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc312d8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13477, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc312d8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13477, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc312d8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13477, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 12476] [c15-1c1s7n0] [Tue Aug 14 09:50:50 2012] PE 
3905 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc312d8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13477, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc31418, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13477, comm=0xc4000003, request=0xb7afb0) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc312d8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13477, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc300e0, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13431, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc312d8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13477, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc310f8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7afb0) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc312d8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13477, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc310f8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7afb0) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30dd8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13461, comm=0xc4000003, request=0xb7afb0) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2da98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13333, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 13105] [c13-2c1s4n1] [Tue Aug 14 09:50:50 2012] PE 
8105 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 13104] [c13-2c1s4n0] [Tue Aug 14 09:50:50 2012] PE 
8099 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30798, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7afb0) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 13103] [c13-2c1s3n3] [Tue Aug 14 09:50:50 2012] PE 
8077 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 11534] [c19-2c0s3n2] [Tue Aug 14 09:50:50 2012] PE 
3254 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 11711] [c19-3c1s7n3] [Tue Aug 14 09:50:50 2012] PE 
3565 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 13101] [c13-2c1s3n1] [Tue Aug 14 09:50:50 2012] PE 
8061 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 13097] [c13-2c1s2n1] [Tue Aug 14 09:50:50 2012] PE 
8005 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 11709] [c19-3c1s7n1] [Tue Aug 14 09:50:50 2012] PE 
3542 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 13388] [c11-0c2s3n0] [Tue Aug 14 09:50:50 2012] PE 
9599 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2f258, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 13227] [c13-3c1s2n3] [Tue Aug 14 09:50:50 2012] PE 
9005 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 13202] [c13-3c0s4n2] [Tue Aug 14 09:50:50 2012] PE 
8705 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 13130] [c13-2c2s2n2] [Tue Aug 14 09:50:50 2012] PE 
8405 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 13015] [c13-1c2s5n3] [Tue Aug 14 09:50:50 2012] PE 
7405 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 12990] [c13-1c1s7n2] [Tue Aug 14 09:50:50 2012] PE 
7105 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 12965] [c13-1c1s1n1] [Tue Aug 14 09:50:50 2012] PE 
6805 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 12689] [c15-3c0s4n1] [Tue Aug 14 09:50:50 2012] PE 
4505 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 12546] [c15-2c0s0n2] [Tue Aug 14 09:50:50 2012] PE 
4205 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 12475] [c15-1c1s6n3] [Tue Aug 14 09:50:50 2012] PE 
3899 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 11706] [c19-3c1s6n2] [Tue Aug 14 09:50:50 2012] PE 
3505 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 11533] [c19-2c0s3n1] [Tue Aug 14 09:50:50 2012] PE 
3242 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 11467] [c19-1c2s2n3] [Tue Aug 14 09:50:50 2012] PE 
2842 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2ee20, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13383, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2ee20, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13383, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 04521] [c16-3c1s2n1] [Tue Aug 14 09:50:50 2012] PE 
842 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2f140, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13391, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 04224] [c16-1c0s0n0] [Tue Aug 14 09:50:50 2012] PE 
242 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 13363] [c11-0c1s4n3] [Tue Aug 14 09:50:50 2012] PE 
9305 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 13129] [c13-2c2s2n1] [Tue Aug 14 09:50:50 2012] PE 
8399 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 13072] [c13-2c0s4n0] [Tue Aug 14 09:50:50 2012] PE 
7705 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 12739] [c15-3c2s0n3] [Tue Aug 14 09:50:50 2012] PE 
5105 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 12678] [c15-3c0s1n2] [Tue Aug 14 09:50:50 2012] PE 
4499 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 12545] [c15-2c0s0n1] [Tue Aug 14 09:50:50 2012] PE 
4199 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 12471] [c15-1c1s5n3] [Tue Aug 14 09:50:50 2012] PE 
3842 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 11530] [c19-2c0s2n2] [Tue Aug 14 09:50:50 2012] PE 
3205 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 11459] [c19-1c2s0n3] [Tue Aug 14 09:50:50 2012] PE 
2742 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2f140, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13391, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 13446] [c11-1c0s1n2] [Tue Aug 14 09:50:50 2012] PE 
9899 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 13219] [c13-3c1s0n3] [Tue Aug 14 09:50:50 2012] PE 
8905 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 13194] [c13-3c0s2n2] [Tue Aug 14 09:50:50 2012] PE 
8605 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 13080] [c13-2c0s6n0] [Tue Aug 14 09:50:50 2012] PE 
7805 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 12764] [c15-3c2s7n0] [Tue Aug 14 09:50:50 2012] PE 
5405 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 12738] [c15-3c2s0n2] [Tue Aug 14 09:50:50 2012] PE 
5099 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 12718] [c15-3c1s3n2] [Tue Aug 14 09:50:50 2012] PE 
4854 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 12368] [c15-0c2s4n0] [Tue Aug 14 09:50:50 2012] PE 
3805 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 11699] [c19-3c1s4n3] [Tue Aug 14 09:50:50 2012] PE 
3454 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 11524] [c19-2c0s1n0] [Tue Aug 14 09:50:50 2012] PE 
3142 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 05085] [c18-3c2s7n1] [Tue Aug 14 09:50:50 2012] PE 
1642 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 04482] [c16-3c0s0n2] [Tue Aug 14 09:50:50 2012] PE 
542 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 03881] [c14-2c1s2n1] [Tue Aug 14 09:50:50 2012] PE 
42 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 13355] [c11-0c1s2n3] [Tue Aug 14 09:50:50 2012] PE 
9205 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 13226] [c13-3c1s2n2] [Tue Aug 14 09:50:50 2012] PE 
8999 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 13122] [c13-2c2s0n2] [Tue Aug 14 09:50:50 2012] PE 
8305 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 12847] [c13-0c1s3n3] [Tue Aug 14 09:50:50 2012] PE 
6005 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 12714] [c15-3c1s2n2] [Tue Aug 14 09:50:50 2012] PE 
4805 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 12639] [c15-2c2s7n3] [Tue Aug 14 09:50:50 2012] PE 
4405 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 11698] [c19-3c1s4n2] [Tue Aug 14 09:50:50 2012] PE 
3442 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 11484] [c19-1c2s7n0] [Tue Aug 14 09:50:50 2012] PE 
3042 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2ee20, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13383, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2f140, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13391, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 03889] [c14-2c1s4n1] [Tue Aug 14 09:50:50 2012] PE 
142 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 13362] [c11-0c1s4n2] [Tue Aug 14 09:50:50 2012] PE 
9299 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 13094] [c13-2c1s1n2] [Tue Aug 14 09:50:50 2012] PE 
7977 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 12998] [c13-1c2s1n2] [Tue Aug 14 09:50:50 2012] PE 
7205 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 12874] [c13-0c2s2n2] [Tue Aug 14 09:50:50 2012] PE 
6305 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 12845] [c13-0c1s3n1] [Tue Aug 14 09:50:50 2012] PE 
5999 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 12821] [c13-0c0s5n1] [Tue Aug 14 09:50:50 2012] PE 
5705 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 12756] [c15-3c2s5n0] [Tue Aug 14 09:50:50 2012] PE 
5305 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 12731] [c15-3c1s6n3] [Tue Aug 14 09:50:50 2012] PE 
5005 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 12713] [c15-3c1s2n1] [Tue Aug 14 09:50:50 2012] PE 
4799 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 12506] [c15-1c2s6n2] [Tue Aug 14 09:50:50 2012] PE 
4105 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 11735] [c19-3c2s5n3] [Tue Aug 14 09:50:50 2012] PE 
3742 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 11480] [c19-1c2s6n0] [Tue Aug 14 09:50:50 2012] PE 
2999 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 11452] [c19-1c1s7n0] [Tue Aug 14 09:50:50 2012] PE 
2654 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2ee20, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13383, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2ee20, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13383, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2d5a8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13341, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 12997] [c13-1c2s1n1] [Tue Aug 14 09:50:50 2012] PE 
7199 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 12973] [c13-1c1s3n1] [Tue Aug 14 09:50:50 2012] PE 
6905 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 12948] [c13-1c0s5n0] [Tue Aug 14 09:50:50 2012] PE 
6605 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 11542] [c19-2c0s5n2] [Tue Aug 14 09:50:50 2012] PE 
3354 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 05169] [c20-0c1s4n1] [Tue Aug 14 09:50:50 2012] PE 
2054 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 04288] [c16-1c2s0n0] [Tue Aug 14 09:50:50 2012] PE 
442 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2ee20, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13383, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc301c8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 13102] [c13-2c1s3n2] [Tue Aug 14 09:50:50 2012] PE 
8065 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 13023] [c13-1c2s7n3] [Tue Aug 14 09:50:50 2012] PE 
7505 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 12947] [c13-1c0s4n3] [Tue Aug 14 09:50:50 2012] PE 
6599 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 12813] [c13-0c0s3n1] [Tue Aug 14 09:50:50 2012] PE 
5605 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 12706] [c15-3c1s0n2] [Tue Aug 14 09:50:50 2012] PE 
4705 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 11727] [c19-3c2s3n3] [Tue Aug 14 09:50:50 2012] PE 
3642 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 11541] [c19-2c0s5n1] [Tue Aug 14 09:50:50 2012] PE 
3342 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 11476] [c19-1c2s5n0] [Tue Aug 14 09:50:50 2012] PE 
2942 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2ee20, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13383, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Recv: Invalid communicator, error stack:
MPI_Recv(186): MPI_Recv(buf=0x29ff570, count=1, MPI_FLOAT, src=1, tag=13382, 
comm=0xc4000003, status=0x2aaab2ef8010) failed
MPI_Recv(101): Invalid communicator
_pmii_daemon(SIGCHLD): [NID 13201] [c13-3c0s4n1] [Tue Aug 14 09:50:50 2012] PE 
8699 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30c98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13461, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 12866] [c13-0c2s0n2] [Tue Aug 14 09:50:50 2012] PE 
6205 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 12838] [c13-0c1s1n2] [Tue Aug 14 09:50:50 2012] PE 
5905 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 12710] [c15-3c1s1n2] [Tue Aug 14 09:50:50 2012] PE 
4754 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 12510] [c15-1c2s7n2] [Tue Aug 14 09:50:50 2012] PE 
4154 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 11722] [c19-3c2s2n2] [Tue Aug 14 09:50:50 2012] PE 
3605 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 11538] [c19-2c0s4n2] [Tue Aug 14 09:50:50 2012] PE 
3305 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 04304] [c16-1c2s4n0] [Tue Aug 14 09:50:50 2012] PE 
534 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 04259] [c16-1c1s0n3] [Tue Aug 14 09:50:50 2012] PE 
334 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30c98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13461, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 12940] [c13-1c0s3n0] [Tue Aug 14 09:50:50 2012] PE 
6505 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30c98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13461, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 12763] [c15-3c2s6n3] [Tue Aug 14 09:50:50 2012] PE 
5399 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 11713] [c19-3c2s0n1] [Tue Aug 14 09:50:50 2012] PE 
3599 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 11537] [c19-2c0s4n1] [Tue Aug 14 09:50:50 2012] PE 
3299 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 03880] [c14-2c1s2n0] [Tue Aug 14 09:50:50 2012] PE 
34 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 13022] [c13-1c2s7n2] [Tue Aug 14 09:50:50 2012] PE 
7499 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30c98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13461, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30c98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13461, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30c98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13461, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30978, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13453, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 12752] [c15-3c2s4n0] [Tue Aug 14 09:50:50 2012] PE 
5261 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 11535] [c19-2c0s3n3] [Tue Aug 14 09:50:50 2012] PE 
3265 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 11486] [c19-1c2s7n2] [Tue Aug 14 09:50:50 2012] PE 
3065 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 04191] [c16-0c2s7n3] [Tue Aug 14 09:50:50 2012] PE 
234 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2d1a8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13341, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 13011] [c13-1c2s4n3] [Tue Aug 14 09:50:50 2012] PE 
7365 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 12972] [c13-1c1s3n0] [Tue Aug 14 09:50:50 2012] PE 
6899 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 12675] [c15-3c0s0n3] [Tue Aug 14 09:50:50 2012] PE 
4454 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 12472] [c15-1c1s6n0] [Tue Aug 14 09:50:50 2012] PE 
3854 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 11710] [c19-3c1s7n2] [Tue Aug 14 09:50:50 2012] PE 
3554 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 11477] [c19-1c2s5n1] [Tue Aug 14 09:50:50 2012] PE 
2954 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2d5a8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13341, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc2dfa8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13341, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 11700] [c19-3c1s5n0] [Tue Aug 14 09:50:50 2012] PE 
3465 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 11526] [c19-2c0s1n2] [Tue Aug 14 09:50:50 2012] PE 
3165 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30c98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13461, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 12820] [c13-0c0s5n0] [Tue Aug 14 09:50:50 2012] PE 
5699 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30fb8, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13469, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30c98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13461, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30c98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13461, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 11543] [c19-2c0s5n3] [Tue Aug 14 09:50:50 2012] PE 
3365 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30c98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13461, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 12873] [c13-0c2s2n1] [Tue Aug 14 09:50:50 2012] PE 
6299 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30a40, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13455, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30c98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13461, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30c98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13461, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 12711] [c15-3c1s1n3] [Tue Aug 14 09:50:50 2012] PE 
4765 exit signal Aborted
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30c98, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13461, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
aborting job:
Fatal error in MPI_Irecv: Invalid communicator, error stack:
MPI_Irecv(147): MPI_Irecv(buf=0xc30658, count=1, MPI_FLOAT, src=MPI_ANY_SOURCE, 
tag=13445, comm=0xc4000003, request=0xb7ae70) failed
MPI_Irecv(92).: Invalid communicator
_pmii_daemon(SIGCHLD): [NID 13093] [c13-2c1s1n1] [Tue Aug 14 09:50:50 2012] PE 
7965 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 12494] [c15-1c2s3n2] [Tue Aug 14 09:50:50 2012] PE 
3965 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 13085] [c13-2c0s7n1] [Tue Aug 14 09:50:50 2012] PE 
7865 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 12502] [c15-1c2s5n2] [Tue Aug 14 09:50:50 2012] PE 
4065 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 12511] [c15-1c2s7n3] [Tue Aug 14 09:50:50 2012] PE 
4165 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 12694] [c15-3c0s5n2] [Tue Aug 14 09:50:50 2012] PE 
4566 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 12735] [c15-3c1s7n3] [Tue Aug 14 09:50:50 2012] PE 
5054 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 12702] [c15-3c0s7n2] [Tue Aug 14 09:50:50 2012] PE 
4661 exit signal Aborted
_pmii_daemon(SIGCHLD): [NID 12719] [c15-3c1s3n3] [Tue Aug 14 09:50:50 2012] PE 
4866 exit signal Aborted



<Prev in Thread] Current Thread [Next in Thread>
  • [Scalapack] Possible bug in SCALAPACK/PBLAS, Ali Uzun <=


For additional information you may use the LAPACK/ScaLAPACK Forum.
Or one of the mailing lists, or