PLASMA 2.4.0 and OpenMP compatibility issue?

[srdegraaf]

Mathieu,

That's great that you found a/the bug! As far as I can tell, I'm NOT using hwloc, so I'm looking forward to your more general patch tomorrow.

Thanks!

Stuart

[srdegraaf]

Mathieu,

Actually, I don't know what hwloc is beyond a cursory googling of it. However, when I do
find / -mount -name "*hwloc*"
it doesn't turn anything up, and neither does
rpm -qa |grep hwloc
so I conclude that I'm not "using" it.

I tried to do "patch <yourpatchfile" down in the build/plasma_2.4.0/control directory to update the three files, and patch barfed with the message:
patch: **** malformed patch at line 16: }

Anyway, I did the patching by hand, and then figured out how to rebuild (since the top level python installer over-writes the three patched files with their originals). I cd-ed to build/plasma_2.4.0 and did a "make lib", and then copied the resulting lib/libplasma.a up into the install/lib directory.

In any event, having done this and linking with the new library, I see no change in my code's behaviour. I guess/hope that this confirms that I'm not using hwloc.

Thanks,

Stuart

[srdegraaf]

Mathieu,

I just realized that "using" hwloc is a conscious choice, i.e. not something over which I have no control.
So, I did "yum install hwloc hwloc-devel", determined where I needed to add link libraries and include files, rebuilt plasma with your patch, and my code.

!!! IT WORKS NOW !!!

Thank you very much, sir!

Stuart

[mateo70]

Glad to hear that .

So just to add some informations:
- hwloc is a library developed by the OpenMPI team and which provides functionalities to bind threads, information about the hierarchy of the machine, memory placement, ... and so it's better to use it because it will hide the different systems and their different interfaces to PLASMA. The most important point in using hwloc is that it will detect if hyperthreading is activated or not and if yes, will bind only one thread per physical core.
- The way PLASMA detect if you have hwloc or not, and so the way you can detect if you have it or not, is to use the command line:

Code: Select all

pkg-config --libs hwloc

- Concerning the patch, you have to go in the PLASMA root directory and do :

Code: Select all

patch -p 0 < patch_file

As I said, I will try to add the fix for every architecture we support without hwloc today, and I will add the new release as soon as it is done.

However, this is not the best solution for the general case because if you have a loop over the block (PLASMA call, OpenMP section) you don't want to call init and finalize at each iteration. In this case, the solution is to add a fake openmp section before the first call to PLASMA. This way openmp will create non-binded threads and PLASMA will create later its own binded thread without interfering with openmp.

Regards,
Mathieu

[uhle89]

Mathieu,
I can also confirm, even iterating OMP/PLASMA/OMP/PLASMA... now works fine for my application,
omp_get_num_procs() always yields correct results.

Thank you for your fast response and efforts, Stephan

[mateo70]

Perfect !!!

Just to know, do you call PLASMA_Init and Finalize in your loop, or is it done outside the loop ?

Mathieu

[uhle89]

Dear Mathieu,
I call PLASMA_Init and Finalize within the loop. As as far as I understand this is still
neccessary if I want to change the number of OMP threads to be used dynamically.
However, the potential overhead of calling PLASMA_Init and Finalize wont bother me, since
each step in my loop takes minutes to hours.
regards stephan

[nchilton]

Hi,

I'm having the same problem as in this old thread and it seems to be similar to viewtopic.php?f=2&t=61 and viewtopic.php?f=2&t=224.

I have tried installing hwloc and rebuilding plasma, but no luck. I have seen the mention of using plasma_init_affinity(cores,affinity) or using a fake openMP parallel section, but I'm not exactly sure what is meant. Here is how I am using OpenMP/plasma currently:

Code: Select all

call OMP_set_num_threads(2)
!$OMP PARALLEL SECTIONS SHARED(totaldim) COPYIN(LWORK)
   !$OMP SECTION
      ThreadID = OMP_get_thread_num()
      allocate(dummyR(1,1))
      allocate(EigVals(totaldim))
      call PLASMA_INIT(2,plas_info)
      write(6,*) ThreadID,"plasma initialized",plas_info
      call PLASMA_ALLOC_WORKSPACE_DSYEV(totaldim,totaldim,T,plas_info)
      write(6,*) ThreadID,"plasma allocated",plas_info
      call PLASMA_DSYEV(301,121,totaldim,EigVecsR,totaldim,EigVals,T,dummyR,totaldim,plas_info)
      write(6,*) ThreadID,"plasma diaged!",plas_info
      call PLASMA_DEALLOC_HANDLE(T,plas_info)
      write(6,*) ThreadID,"plasma dealloced",plas_info
      call PLASMA_FINALIZE(plas_info)
      write(6,*) ThreadID,"plasma finalized",plas_info
      deallocate(dummyR,EigVals)
   !$OMP SECTION            
      ThreadID = OMP_get_thread_num()
      allocate(dummyR(1,1))
      allocate(EigVals(totaldim))
      call PLASMA_INIT(2,plas_info)
      write(6,*) ThreadID,"plasma initialized",plas_info
      call PLASMA_ALLOC_WORKSPACE_DSYEV(totaldim,totaldim,T,plas_info)
      write(6,*) ThreadID,"plasma allocated",plas_info
      call PLASMA_DSYEV(301,121,totaldim,EigVecsR,totaldim,EigVals,T,dummyR,totaldim,plas_info)
      write(6,*) ThreadID,"plasma diaged!",plas_info
      call PLASMA_DEALLOC_HANDLE(T,plas_info)
      write(6,*) ThreadID,"plasma dealloced",plas_info
      call PLASMA_FINALIZE(plas_info)
      write(6,*) ThreadID,"plasma finalized",plas_info
      deallocate(dummyR,EigVals)
!$OMP END PARALLEL SECTIONS

This code executes with only two cores (not the 2*2 = 4 as expected) and the diagonalizations complete at virtually the same time (always one then the other however).

Any suggestions?

Cheers,
Nick