Using MAGMA with CUDA/PGI

[jah87]

I'm using an interface block to call magma_dgesv_gpu:

Code: Select all

module dgesv_gpu_magma
  use cudafor
  interface dgesv_gpu
    subroutine MdgesvDev(n, nrhs, a, lda, ipiv, b, ldb, info) bind(c,name='magma_dgesv_gpu')
      use iso_c_binding
      integer(c_int), value :: n, nrhs, lda, ldb, info
      real(c_double), device, dimension(n,n) :: a
      real(c_double), device, dimension(n,nrhs) :: b
      integer(c_int), device, dimension(n) :: ipiv
    end subroutine MdgesvDev
    subroutine cublasinit() bind(c,name='cublasInit')
    end subroutine cublasinit
  end interface
end module dgesv_gpu_magma

The execution of the subroutine is as follows:

Code: Select all

  Real(8), Device, Dimension(:,:), Allocatable :: jac_dev
  Integer, Device, Dimension(:), Allocatable :: indx_dev
  Real(8), Device, Dimension(:), Allocatable :: dy_dev
...
  call cublasinit()
  call MdgesvDev(ny,1,jac_dev,ny,indx_dev,dy_dev,ny,info)

My makefile includes:

Code: Select all

-Mcuda -I$(INCLUDE_PATH) -L$(LIBRARY_PATH) -lmagma -lmagmablas -lmagma -lcublas

When I try to compile, I get:

/lib/libmagma.a(xerbla.o): In function `magma_xerbla':
xerbla.cpp:(.text+0x1c): undefined reference to `xerbla_'
/lib/libmagma.a(dgetrf_gpu.o): In function `magma_dgetrf_gpu':
dgetrf_gpu.cpp:(.text+0x2b9): undefined reference to `cuCtxSynchronize'
dgetrf_gpu.cpp:(.text+0x313): undefined reference to `dgetrf_'
dgetrf_gpu.cpp:(.text+0x7e6): undefined reference to `cuCtxSynchronize'
dgetrf_gpu.cpp:(.text+0x820): undefined reference to `dgetrf_'
dgetrf_gpu.cpp:(.text+0xa66): undefined reference to `dgetrf_'
/lib/libmagma.a(dgetrs_gpu.o): In function `magma_dgetrs_gpu':
dgetrs_gpu.cpp:(.text+0x50): undefined reference to `lsame_'
dgetrs_gpu.cpp:(.text+0xea): undefined reference to `lsame_'
dgetrs_gpu.cpp:(.text+0x109): undefined reference to `lsame_'
dgetrs_gpu.cpp:(.text+0x29d): undefined reference to `dlaswp_'
dgetrs_gpu.cpp:(.text+0x341): undefined reference to `dlaswp_'

What am I doing wrong?

[jah87]

I added -lcuda and -lacml flags, but now when I run I get a segmentation fault.

[jeremiahpalmer]

This post that I submitted yesterday might help with the seg fault:

http://icl.cs.utk.edu/magma/forum/viewtopic.php?f=2&t=486

[mgates3]

How are you allocating memory on the GPU? To use the GPU interface, magma_dgesv_gpu, at some point you need to call cudaMalloc to allocate GPU memory, which I don't see in your code snippets. If you use the CPU interface, magma_dgesv, then it expects matrices in host memory (and internally copies them to the GPU to do the computation).
-mark

[jah87]

I was naively allocating them with 'Allocate'. My assumption was that the PGI compiler would translate this to a cudaMalloc for the device.

[mgates3]

Ah, you may be right, I missed that you were using PGI and that it has builtin support for GPUs. I'm not familiar with their compiler extensions.
-mark

[kaushikragavan]

Hi,

I have been trying to call magma_dsygvd from fortran code and got a generic error:

Here`s my code:
MAGMA INTERFACE

module magmafor
interface dsygvd
subroutine dsygvdDev (ITYPE, JOBZ, UPLO, N, A, LDA, B, LDB, W, WORK, &
LWORK,IWORK, LIWORK, INFO) bind (c, name = 'magma_dsygvd')
use iso_c_binding
integer(c_int), value :: ITYPE, N, LDA, LDB, LWORK, LIWORK, INFO
character(kind = c_char), value :: JOBZ, UPLO
double precision(c_double), device, allocatable :: A(:,:), B(:,:), W(:), WORK(:)
! double precision(c_double), device, dimension (:) :: W(N), WORK(LWORK)
integer(c_int), device, ALLOCATABLE :: IWORK(:)
end subroutine dsygvdDev
end interface
end module magmafor


program main

use magmafor
integer :: ITYPE,NIN,NOUT,N,NMAX,NB,LDA,LDB,LWORK, LIWORK,INFO,stat
double precision, device, allocatable :: dA(:,:), dB(:,:), dW(:), dWORK(:)
double precision, allocatable :: A(:,:),B(:,:),W(:),WORK(:)
integer, device, allocatable :: dIWORK(:)
integer, allocatable :: IWORK(:)

ITYPE=0;
NIN=5
NOUT=6
NMAX=10
NB=64
LDA=NMAX
LDB=NMAX
LWORK=1+(6+NB+2*NMAX)*NMAX
LIWORK=3+5*NMAX

WRITE (NOUT,*) 'DSYGVD Example Program Results'
WRITE (NOUT,*)
! Skip heading in data file
READ (NIN,*)
READ (NIN,*) N

IF (N.LE.NMAX) THEN
allocate (A(LDA,NMAX), B(LDA,NMAX), W(NMAX), WORK(LWORK))
! allocate (dA(LDA,NMAX), dB(LDA,NMAX), dWORK(LWORK), dW(NMAX))
allocate (IWORK(LIWORK))
READ (NIN,*) ((A(I,J),J=I,N),I=1,N)
READ (NIN,*) ((B(I,J),J=I,N),I=1,N)

call dsygvd(2,'V','U',N,A,LDA,B,LDB,W,WORK,LWORK,IWORK,LIWORK,INFO)

if (INFO.EQ.0) then
write (NOUT,*) 'Eigenvalues'
write (NOUT,9999) (W(J),J=1,N)
write (NOUT,*) 'EIGENVECTORS'
write (NOUT,9999) ((A(I,J),J=1,N),I=1,N)

end if
!double precision, device, allocate (dW(N),dWORK(N))
!real*8, dimension(LDA,N) :: A,B

!double precision, device, deallocate(dW,dWORK)
9999 FORMAT (3X,(6F11.4))
!deallocate (dA, dB, dWORK, dW, dIWORK)
deallocate (A, B, WORK, W, IWORK)
END IF
end program

Makefile:
magmatest2: magmatest2.f90
pgf90 -Mcuda -c $<
pgf90 -Mcuda -o $@ $@.o -L /sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib -R /sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib/ -lmagma -lmagmablas -lcublas -lcuda -lm

When I compile I get the foll error:

pgf90 -Mcuda -c magmatest2.f90
PGF90-S-0155-Could not resolve generic procedure dsygvd (magmatest2.f90: 50)
0 inform, 0 warnings, 1 severes, 0 fatal for main
make: *** [magmatest2] Error 2

I have verified all the arguments with the subroutine and also with the source file : magma_dsygv.cpp.

[jah87]

I've debugged the initial problem to the point of now I receive a segmentation fault at line 247 in dgetrf_gpu.cpp:

Code: Select all

magma_free_host( work );

Since work is entirely an internal variable to MAGMA, I'm not sure what I've done (or didn't do) to cause this problem.

Any advice?