Linking errors while calling magma from fortran

[kaushikragavan]

Hi,

I getting numerous linking errors when I call the magma_dsygvd from a fortran code. I am using the Pgf90 compiler.

Here`s my code:
! DSYGVD Example Program Text


! MAGMA INTERFACE

module magmafor
interface dsygvdmagma
subroutine dsygvdDev (ITYPE, JOBZ, UPLO, N, A, LDA, B, LDB, W, WORK, &
LWORK,IWORK, LIWORK, INFO) bind (c, name = 'magma_dsygvd')
use iso_c_binding
integer(c_int), value :: ITYPE, N, LDA, LDB, LWORK, LIWORK, INFO
character(kind = c_char), value :: JOBZ, UPLO
double precision(c_double), device, allocatable :: A(:,:), B(:,:), W(:), WORK(:)
! double precision(c_double), device, dimension (:) :: W(N), WORK(LWORK)
integer(c_int), device, ALLOCATABLE :: IWORK(:)
end subroutine dsygvdDev
end interface
end module magmafor

program main

use magmafor

INTEGER NIN, NOUT
PARAMETER (NIN=5,NOUT=6)
INTEGER NB, NMAX
PARAMETER (NB=64,NMAX=10)
INTEGER LDA, LDB, LIWORK, LWORK, error
PARAMETER (LDA=NMAX,LDB=NMAX,LIWORK=3+5*NMAX,&
LWORK=1+(6+2*NMAX)*NMAX)

! .. Local Scalars ..
DOUBLE PRECISION, allocatable :: A(:,:), B(:,:), &
W(:), WORK(:)
double precision, device, allocatable :: dA(:,:), dB(:,:), &
dW(:), dWORK(:)
INTEGER IWORK(LIWORK)
integer, device, allocatable :: dIWORK(:)
! .. Executable Statements ..

WRITE (NOUT,*) 'DSYGVD Example Program Results'
WRITE (NOUT,*)
! Skip heading in data file
READ (NIN,*)
READ (NIN,*) N
IF (N.LE.NMAX) THEN
!
allocate (A(LDA,NMAX),B(LDA,NMAX), W(NMAX), WORK(LWORK), stat=error)
if(stat1.ne.0) then
print*,'error: couldn`t alloc memory'
end if

! Read the upper triangular parts of the matrices A and B

READ (NIN,*) ((A(I,J),J=I,N),I=1,N)
READ (NIN,*) ((B(I,J),J=I,N),I=1,N)

allocate (dA(LDA,NMAX), dB(LDA,NMAX), dW(NMAX), dWORK(LWORK))
allocate (dIWORK(LIWROK))

dA = A
dB = B
dW = W
dWORK = WORK

! Magma call

call dsygvdmagma(1,'V','U',N,dA,LDA,dB,LDB,dW,dWORK,LWORK,dIWORK,LIWORK,INFO)
!
! Solve the generalized symmetric eigenvalue problem
! A*B*x = lambda*x (ITYPE = 2)
!
! CALL DSYGVD(2,'Vectors','Upper',N,A,LDA,B,LDB,W,WORK,LWORK, &
! IWORK,LIWORK,INFO)
LWOPT = WORK(1)
LIWOPT = IWORK(1)
!

IF (INFO.EQ.0) THEN
!
W = dW
A = dA
! Print solution
!
WRITE (NOUT,*) 'Eigenvalues'
WRITE (NOUT,99999) (W(J),J=1,N)
WRITE (NOUT,*) 'Eigenvectors'
WRITE (NOUT,99999) ((A(I,J),J=1,N),I=1,N)

END IF



! Print workspace information
!
IF (LWORK.LT.LWOPT) THEN
WRITE (NOUT,*)
WRITE (NOUT,99995) 'Optimum workspace required = ', LWOPT, &
'Workspace provided = ', LWORK
END IF
IF (LIWORK.LT.LIWOPT) THEN
WRITE (NOUT,*)
WRITE (NOUT,99995) 'Integer workspace required = ', LIWOPT, &
'Integer workspace provided = ', LIWORK
END IF
ELSE
WRITE (NOUT,*) 'NMAX too small'
END IF

!
99999 FORMAT (3X,(6F11.4))
99998 FORMAT (4X,1P,6E11.1)
99997 FORMAT (1X,A,I4,A)
99996 FORMAT (1X,A,I4)
99995 FORMAT (1X,A,I5,/1X,A,I5)
deallocate(A)
deallocate(B)
deallocate(W,WORK)
deallocate(dA)
deallocate(dB)
deallocate(dW)
deallocate(dWORK)
deallocate(dIWORK)

end program

I compiled with:
dsygvdmagma: dsygvdmagma.f90
pgf90 -Mcuda -c $<
pgf90 -Mcuda -o $@ $@.o -L/sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib -R/sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.o/lib -lmagma -lmagmablas -lcublas -lm

And I get these errors:
pgf90 -Mcuda -c dsygvdmagma.f90
pgf90 -Mcuda -o dsygvdmagma dsygvdmagma.o -L/sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib -lmagma -lmagmablas -lcublas -lm
/sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib/libmagma.a(dsygvd.o): In function `magma_dsygvd':
dsygvd.cpp:(.text+0x8a): undefined reference to `lsame_'
dsygvd.cpp:(.text+0x9d): undefined reference to `lsame_'
dsygvd.cpp:(.text+0x2b9): undefined reference to `lsame_'
dsygvd.cpp:(.text+0x31f): undefined reference to `lsame_'
/sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib/libmagma.a(dpotrf_gpu.o): In function `magma_dpotrf_gpu':
dpotrf_gpu.cpp:(.text+0x58): undefined reference to `lsame_'
dpotrf_gpu.cpp:(.text+0x1a8): undefined reference to `dpotrf_'
dpotrf_gpu.cpp:(.text+0x399): undefined reference to `lsame_'
dpotrf_gpu.cpp:(.text+0x3ea): undefined reference to `dpotrf_'
dpotrf_gpu.cpp:(.text+0x508): undefined reference to `dpotrf_'
/sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib/libmagma.a(dsygst_gpu.o): In function `magma_dsygst_gpu':
dsygst_gpu.cpp:(.text+0x5c): undefined reference to `lsame_'
dsygst_gpu.cpp:(.text+0x60f): undefined reference to `dsygs2_'
dsygst_gpu.cpp:(.text+0x74e): undefined reference to `lsame_'
dsygst_gpu.cpp:(.text+0xc73): undefined reference to `dsygs2_'
dsygst_gpu.cpp:(.text+0xf19): undefined reference to `dsygs2_'
dsygst_gpu.cpp:(.text+0x16a6): undefined reference to `dsygs2_'
/sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib/libmagma.a(xerbla.o): In function `magma_xerbla':
xerbla.cpp:(.text+0x1c): undefined reference to `xerbla_'
/sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib/libmagma.a(dsyevd_gpu.o): In function `magma_dsyevd_gpu':
dsyevd_gpu.cpp:(.text+0x86): undefined reference to `lsame_'
dsyevd_gpu.cpp:(.text+0x99): undefined reference to `lsame_'
dsyevd_gpu.cpp:(.text+0x264): undefined reference to `dlamch_'
dsyevd_gpu.cpp:(.text+0x276): undefined reference to `dlamch_'
dsyevd_gpu.cpp:(.text+0x40f): undefined reference to `dsterf_'
dsyevd_gpu.cpp:(.text+0x4c9): undefined reference to `lsame_'
dsyevd_gpu.cpp:(.text+0x4f0): undefined reference to `lsame_'
dsyevd_gpu.cpp:(.text+0x5de): undefined reference to `dstedc_'
dsyevd_gpu.cpp:(.text+0x75c): undefined reference to `dscal_'
/sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib/libmagma.a(dsytrd_gpu.o): In function `magma_dsytrd_gpu':
dsytrd_gpu.cpp:(.text+0x62): undefined reference to `lsame_'
dsytrd_gpu.cpp:(.text+0x435): undefined reference to `dsytrd_'
dsytrd_gpu.cpp:(.text+0x524): undefined reference to `lsame_'
dsytrd_gpu.cpp:(.text+0x921): undefined reference to `dsytrd_'
/sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib/libmagma.a(dlatrd.o): In function `magma_dlatrd':
dlatrd.cpp:(.text+0xaa): undefined reference to `lsame_'
dlatrd.cpp:(.text+0x160): undefined reference to `dlarfg_'
dlatrd.cpp:(.text+0x344): undefined reference to `dscal_'
dlatrd.cpp:(.text+0x39d): undefined reference to `ddot_'
dlatrd.cpp:(.text+0x3f4): undefined reference to `daxpy_'
dlatrd.cpp:(.text+0x4c9): undefined reference to `dgemv_'
dlatrd.cpp:(.text+0x556): undefined reference to `dgemv_'
dlatrd.cpp:(.text+0x67a): undefined reference to `dgemv_'
dlatrd.cpp:(.text+0x6ed): undefined reference to `dgemv_'
dlatrd.cpp:(.text+0x76e): undefined reference to `dlarfg_'
dlatrd.cpp:(.text+0x9c1): undefined reference to `dgemv_'
dlatrd.cpp:(.text+0xa35): undefined reference to `dgemv_'
dlatrd.cpp:(.text+0xab5): undefined reference to `dgemv_'
dlatrd.cpp:(.text+0xb41): undefined reference to `dgemv_'
dlatrd.cpp:(.text+0xb78): undefined reference to `dscal_'
dlatrd.cpp:(.text+0xbdf): undefined reference to `ddot_'
dlatrd.cpp:(.text+0xc41): undefined reference to `daxpy_'
dlatrd.cpp:(.text+0xcca): undefined reference to `dgemv_'
dlatrd.cpp:(.text+0xd5a): undefined reference to `dgemv_'
dlatrd.cpp:(.text+0xddc): undefined reference to `dgemv_'
dlatrd.cpp:(.text+0xe76): undefined reference to `dgemv_'
dlatrd.cpp:(.text+0xeb3): undefined reference to `daxpy_'
/sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib/libmagma.a(dormtr_gpu.o): In function `magma_dormtr_gpu':
dormtr_gpu.cpp:(.text+0x87): undefined reference to `lsame_'
dormtr_gpu.cpp:(.text+0x97): undefined reference to `lsame_'
dormtr_gpu.cpp:(.text+0xc7): undefined reference to `lsame_'
dormtr_gpu.cpp:(.text+0x133): undefined reference to `lsame_'
dormtr_gpu.cpp:(.text+0x1a9): undefined reference to `lsame_'
/sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib/libmagma.a(dormtr_gpu.o):dormtr_gpu.cpp:(.text+0x1cc): more undefined references to `lsame_' follow
/sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib/libmagma.a(dormqr2_gpu.o): In function `magma_dormqr2_gpu':
dormqr2_gpu.cpp:(.text+0x547): undefined reference to `dlarft_'
/sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib/libmagma.a(dormql2_gpu.o): In function `magma_dormql2_gpu':
dormql2_gpu.cpp:(.text+0x103): undefined reference to `lsame_'
dormql2_gpu.cpp:(.text+0x118): undefined reference to `lsame_'
dormql2_gpu.cpp:(.text+0x1f9): undefined reference to `lsame_'
dormql2_gpu.cpp:(.text+0x221): undefined reference to `lsame_'
dormql2_gpu.cpp:(.text+0x52b): undefined reference to `dlarft_'
/sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib/libmagmablas.a(dauxiliary.cu_o): In function `cpu_gpu_ddiff':
tmpxft_0000388c_00000000-1_dauxiliary.cudafe1.cpp:(.text+0x6ad): undefined reference to `daxpy_'
tmpxft_0000388c_00000000-1_dauxiliary.cudafe1.cpp:(.text+0x6d5): undefined reference to `dlange_'
/sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib/libmagmablas.a(dauxiliary.cu_o): In function `magmablas_dlaset':
tmpxft_0000388c_00000000-1_dauxiliary.cudafe1.cpp:(.text+0x7b4): undefined reference to `magma_stream'
tmpxft_0000388c_00000000-1_dauxiliary.cudafe1.cpp:(.text+0x801): undefined reference to `magma_stream'
tmpxft_0000388c_00000000-1_dauxiliary.cudafe1.cpp:(.text+0x867): undefined reference to `magma_stream'
/sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib/libmagmablas.a(dauxiliary.cu_o): In function `dzero_32x32_block':
tmpxft_0000388c_00000000-1_dauxiliary.cudafe1.cpp:(.text+0xaab): undefined reference to `magma_stream'
/sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib/libmagmablas.a(dauxiliary.cu_o): In function `dzero_nbxnb_block':
tmpxft_0000388c_00000000-1_dauxiliary.cudafe1.cpp:(.text+0xb9e): undefined reference to `magma_stream'
/sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib/libmagmablas.a(dauxiliary.cu_o):tmpxft_0000388c_00000000-1_dauxiliary.cudafe1.cpp:(.text+0xd17): more undefined references to `magma_stream' follow
make: *** [dsygvdmagma] Error 2

[kaushikragavan]

The errors were due to passing the device pointers. There are implicit cudamallocs and cudamemcpys for the magma_dsygvd routine.
Now I am getting this error when I change it to the host pointers.

pgf90 -Mcuda -c magmatest2.f90
pgf90 -Mcuda -o magmatest2 magmatest2.o -L /sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib -R /sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib/ -lmagma -lmagmablas -lcublas -lcuda -lm
magmatest2.o: In function `main':
/nics/c/home/krajag1/Research/./magmatest2.f90:45: undefined reference to `magma_dsygvd'
make: *** [magmatest2] Error 2

Hope somebody can point out what`s going wrong.

[mgates3]

In the first post, you need to add BLAS and LAPACK libraries such as ATLAS or Intel MKL. See the various make.inc files for examples. The dpotrf_, lsame_, dsygs2_, etc. are LAPACK routine names. As for magma_stream not being defined, you may need to repeat -lmagma, as:
-lmagma -lmagmablas -lmagma

I don't really understand what code you changed for the second post.

-mark

[kaushikragavan]

Hi mgates3,

Thank you so much for pointing it out.
Now, after linking MKL and CUDA libraries, I still get the linking errors for cublas

Here`s my new code:

! MAGMA INTERFACE

module magmafor
interface dsygvdmagma
subroutine dsygvdDev (ITYPE, JOBZ, UPLO, N, A, LDA, B, LDB, W, WORK, &
LWORK,IWORK, LIWORK, INFO) bind (c, name = 'magma_dsygvd')
use iso_c_binding
integer(c_int), value :: ITYPE, N, LDA, LDB, LWORK, LIWORK, INFO
character(kind = c_char), value :: JOBZ, UPLO
double precision(c_double), allocatable :: A(:,:), B(:,:), W(:), WORK(:)
integer(c_int), allocatable :: IWORK(:)
end subroutine dsygvdDev
end interface
end module magmafor

program main

use magmafor

INTEGER NIN, NOUT
PARAMETER (NIN=5,NOUT=6)
INTEGER NB, NMAX
PARAMETER (NB=64,NMAX=10)
INTEGER LDA, LDB, LIWORK, LWORK, error
PARAMETER (LDA=NMAX,LDB=NMAX,LIWORK=3+5*NMAX,&
LWORK=1+(6+2*NMAX)*NMAX)

DOUBLE PRECISION, allocatable :: A(:,:), B(:,:), &
W(:), WORK(:)
integer, allocatable :: IWORK(:)

WRITE (NOUT,*) 'DSYGVD Example Program Results'
WRITE (NOUT,*)

! Skip heading in data file
READ (NIN,*)
READ (NIN,*) N
IF (N.LE.NMAX) THEN
allocate (A(LDA,NMAX),B(LDA,NMAX), W(NMAX), WORK(LWORK),IWORK(LIWORK), stat=error)
if(stat.ne.0) then
print*,'error: couldn`t alloc memory'
end if

! Read the upper triangular parts of the matrices A and B

READ (NIN,*) ((A(I,J),J=I,N),I=1,N)
READ (NIN,*) ((B(I,J),J=I,N),I=1,N)

! Magma call

call dsygvdmagma(1,'V','U',N,A,LDA,B,LDB,W,WORK,LWORK,IWORK,LIWORK,INFO)

LWOPT = WORK(1)
LIWOPT = IWORK(1)
!
IF (INFO.EQ.0) THEN
!
! Print solution
!
WRITE (NOUT,*) 'Eigenvalues'
WRITE (NOUT,99999) (W(J),J=1,N)
WRITE (NOUT,*) 'Eigenvectors'
WRITE (NOUT,99999) ((A(I,J),J=1,N),I=1,N)

END IF

! Print workspace information

IF (LWORK.LT.LWOPT) THEN
WRITE (NOUT,*)
WRITE (NOUT,99995) 'Optimum workspace required = ', LWOPT, &
'Workspace provided = ', LWORK
END IF
IF (LIWORK.LT.LIWOPT) THEN
WRITE (NOUT,*)
WRITE (NOUT,99995) 'Integer workspace required = ', LIWOPT, &
'Integer workspace provided = ', LIWORK
END IF
ELSE
WRITE (NOUT,*) 'NMAX too small'
END IF

99999 FORMAT (3X,(6F11.4))
99998 FORMAT (4X,1P,6E11.1)
99997 FORMAT (1X,A,I4,A)
99996 FORMAT (1X,A,I4)
99995 FORMAT (1X,A,I5,/1X,A,I5)
deallocate(A)
deallocate(B)
deallocate(W,WORK)
deallocate(IWORK)
end program

Makefile:

MKLROOT = /opt/intel/mkl/lib/intel64
CUDADIR = /sw/keeneland/cuda/4.1/linux_binary/lib64
MAGMALIB = /sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib
LIB = -L$(MKLROOT)/ -lmkl_intel_ilp64 -lmkl_pgi_thread -lmkl_core -mp -lpthread -pgf90libs -L$(CUDADIR)/ -lcublas -lcudart -lm

magmatest2: magmatest2.f90
pgfortran -Mcuda -c $<
pgfortran -Mcuda -o $@ $@.o -L$(MAGMALIB) -lcuda -lmagma -lmagmablas -lmagma $(LIB)

And this time I get the cublas error msg:


pgfortran -Mcuda -c magmatest2.f90
pgfortran -Mcuda -o magmatest2 magmatest2.o -L/sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib -lcuda -lmagma -lmagmablas -lmagma -L/opt/intel/mkl/lib/intel64/ -lmkl_intel_ilp64 -lmkl_pgi_thread -lmkl_core -mp -lpthread -pgf90libs -L/sw/keeneland/cuda/4.1/linux_binary/lib64/ -lcublas -lcudart -lm
/sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib/libmagma.a(dsygvd.o): In function `magma_dsygvd':
dsygvd.cpp:(.text+0x5c3): undefined reference to `cublasGetCurrentCtx'
dsygvd.cpp:(.text+0x638): undefined reference to `cublasSetError'
dsygvd.cpp:(.text+0x67c): undefined reference to `cublasGetCurrentCtx'
dsygvd.cpp:(.text+0x6d1): undefined reference to `cublasSetError'
/sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib/libmagma.a(dpotrf_gpu.o): In function `magma_dpotrf_gpu':
dpotrf_gpu.cpp:(.text+0x281): undefined reference to `cublasGetCurrentCtx'
dpotrf_gpu.cpp:(.text+0x2d1): undefined reference to `cublasSetError'
dpotrf_gpu.cpp:(.text+0x5e0): undefined reference to `cublasGetCurrentCtx'
dpotrf_gpu.cpp:(.text+0x62a): undefined reference to `cublasSetError'
/sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib/libmagma.a(dsygst_gpu.o): In function `magma_dsygst_gpu':
dsygst_gpu.cpp:(.text+0x2fb): undefined reference to `cublasGetCurrentCtx'
dsygst_gpu.cpp:(.text+0x356): undefined reference to `cublasSetError'
dsygst_gpu.cpp:(.text+0x390): undefined reference to `cublasGetCurrentCtx'
dsygst_gpu.cpp:(.text+0x3f1): undefined reference to `cublasSetError'
dsygst_gpu.cpp:(.text+0x42e): undefined reference to `cublasGetCurrentCtx'
dsygst_gpu.cpp:(.text+0x489): undefined reference to `cublasSetError'
dsygst_gpu.cpp:(.text+0x4ba): undefined reference to `cublasGetCurrentCtx'
dsygst_gpu.cpp:(.text+0x525): undefined reference to `cublasSetError'
dsygst_gpu.cpp:(.text+0x53a): undefined reference to `cublasGetCurrentCtx'
dsygst_gpu.cpp:(.text+0x59e): undefined reference to `cublasSetError'
dsygst_gpu.cpp:(.text+0x971): undefined reference to `cublasGetCurrentCtx'
dsygst_gpu.cpp:(.text+0x9cd): undefined reference to `cublasSetError'
dsygst_gpu.cpp:(.text+0xa03): undefined reference to `cublasGetCurrentCtx'
dsygst_gpu.cpp:(.text+0xa5f): undefined reference to `cublasSetError'
dsygst_gpu.cpp:(.text+0xa9c): undefined reference to `cublasGetCurrentCtx'
dsygst_gpu.cpp:(.text+0xaf7): undefined reference to `cublasSetError'
dsygst_gpu.cpp:(.text+0xb28): undefined reference to `cublasGetCurrentCtx'
dsygst_gpu.cpp:(.text+0xb84): undefined reference to `cublasSetError'
dsygst_gpu.cpp:(.text+0xba3): undefined reference to `cublasGetCurrentCtx'
dsygst_gpu.cpp:(.text+0xc02): undefined reference to `cublasSetError'
dsygst_gpu.cpp:(.text+0x106a): undefined reference to `cublasGetCurrentCtx'
dsygst_gpu.cpp:(.text+0x10f5): undefined reference to `cublasSetError'
dsygst_gpu.cpp:(.text+0x113c): undefined reference to `cublasGetCurrentCtx'
dsygst_gpu.cpp:(.text+0x11cd): undefined reference to `cublasSetError'
dsygst_gpu.cpp:(.text+0x120f): undefined reference to `cublasGetCurrentCtx'
dsygst_gpu.cpp:(.text+0x12a6): undefined reference to `cublasSetError'
dsygst_gpu.cpp:(.text+0x1349): undefined reference to `cublasGetCurrentCtx'
dsygst_gpu.cpp:(.text+0x13da): undefined reference to `cublasSetError'
dsygst_gpu.cpp:(.text+0x1400): undefined reference to `cublasGetCurrentCtx'
dsygst_gpu.cpp:(.text+0x1488): undefined reference to `cublasSetError'
dsygst_gpu.cpp:(.text+0x1809): undefined reference to `cublasGetCurrentCtx'
dsygst_gpu.cpp:(.text+0x188c): undefined reference to `cublasSetError'
dsygst_gpu.cpp:(.text+0x18c6): undefined reference to `cublasGetCurrentCtx'
dsygst_gpu.cpp:(.text+0x1952): undefined reference to `cublasSetError'
dsygst_gpu.cpp:(.text+0x1993): undefined reference to `cublasGetCurrentCtx'
dsygst_gpu.cpp:(.text+0x1a1d): undefined reference to `cublasSetError'
dsygst_gpu.cpp:(.text+0x1abd): undefined reference to `cublasGetCurrentCtx'
dsygst_gpu.cpp:(.text+0x1b4e): undefined reference to `cublasSetError'
dsygst_gpu.cpp:(.text+0x1b74): undefined reference to `cublasGetCurrentCtx'
dsygst_gpu.cpp:(.text+0x1bed): undefined reference to `cublasSetError'
/sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib/libmagma.a(dsytrd_gpu.o): In function `magma_dsytrd_gpu':
dsytrd_gpu.cpp:(.text+0x28e): undefined reference to `cublasGetCurrentCtx'
dsytrd_gpu.cpp:(.text+0x307): undefined reference to `cublasSetError'
dsytrd_gpu.cpp:(.text+0x711): undefined reference to `cublasGetCurrentCtx'
dsytrd_gpu.cpp:(.text+0x78b): undefined reference to `cublasSetError'
/sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib/libmagma.a(dlatrd.o): In function `magma_dlatrd':
dlatrd.cpp:(.text+0x21f): undefined reference to `cublasGetCurrentCtx'
dlatrd.cpp:(.text+0x282): undefined reference to `cublasSetError'
dlatrd.cpp:(.text+0x827): undefined reference to `cublasGetCurrentCtx'
dlatrd.cpp:(.text+0x8c2): undefined reference to `cublasSetError'
/sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib/libmagma.a(dlarfb_gpu.o): In function `magma_dlarfb_gpu':
dlarfb_gpu.cpp:(.text+0xaf): undefined reference to `cublasGetCurrentCtx'
dlarfb_gpu.cpp:(.text+0x120): undefined reference to `cublasSetError'
dlarfb_gpu.cpp:(.text+0x134): undefined reference to `cublasGetCurrentCtx'
dlarfb_gpu.cpp:(.text+0x1b0): undefined reference to `cublasSetError'
dlarfb_gpu.cpp:(.text+0x1d3): undefined reference to `cublasGetCurrentCtx'
dlarfb_gpu.cpp:(.text+0x241): undefined reference to `cublasSetError'
dlarfb_gpu.cpp:(.text+0x2ca): undefined reference to `cublasGetCurrentCtx'
dlarfb_gpu.cpp:(.text+0x33a): undefined reference to `cublasSetError'
dlarfb_gpu.cpp:(.text+0x362): undefined reference to `cublasGetCurrentCtx'
dlarfb_gpu.cpp:(.text+0x3e5): undefined reference to `cublasSetError'
dlarfb_gpu.cpp:(.text+0x408): undefined reference to `cublasGetCurrentCtx'
dlarfb_gpu.cpp:(.text+0x479): undefined reference to `cublasSetError'
dlarfb_gpu.cpp:(.text+0x494): undefined reference to `cublasGetCurrentCtx'
dlarfb_gpu.cpp:(.text+0x518): undefined reference to `cublasSetError'
/sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib/libmagmablas.a(dlansy.cu_o): In function `magmablas_dlansy':
tmpxft_000039ec_00000000-1_dlansy.cudafe1.cpp:(.text+0xa88): undefined reference to `cublasGetCurrentCtx'
tmpxft_000039ec_00000000-1_dlansy.cudafe1.cpp:(.text+0xaa6): undefined reference to `cublasSetError'
/sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib/libmagmablas.a(dgemm_fermi.cu_o): In function `magmablas_dgemm':
tmpxft_00004066_00000000-1_dgemm_fermi.cudafe1.cpp:(.text+0xa81): undefined reference to `cublasGetCurrentCtx'
tmpxft_00004066_00000000-1_dgemm_fermi.cudafe1.cpp:(.text+0xb37): undefined reference to `cublasSetError'
/sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib/libmagma.a(common.o): In function `magmablasSetKernelStream':
common.cpp:(.text+0x1c): undefined reference to `cublasGetCurrentCtx'
make: *** [magmatest2] Error 2

[mgates3]

Couple of problems. You should use mkl_intel_lp64, not mkl_intel_ilp64. lp64 means long and pointers are 64 bit, integers are 32 bit, while with ilp64 integers are also 64 bit. We're working on making MAGMA so it can be compiled with 64 bit integers, if desired.

On Keeneland, MAGMA 1.1 was compiled with CUDA 4.0, not CUDA 4.1. You can check the make.inc at
/sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/make.inc
You can either re-compile MAGMA on your own, or switch to using and linking with CUDA 4.0.

-mark

[kaushikragavan]

Hi mgates3,

I found the bug!
It was due to linking of cuda-4.1 libs. The makefile in magma-1.1 has a pointer to the cuda-4.0 installation.

Now I am getting a segmentation fault for the same test data from http://www.nag.com/lapack-ex/node104.html

Thank you once again for your help.

[kaushikragavan]

sorry for that! I didn notice your reply.

[kaushikragavan]

Testing lapack call with magma

Here`s my new code:


! MAGMA INTERFACE

module magmafor
interface dsygvdmagma
subroutine dsygvdDev (ITYPE, JOBZ, UPLO, N, A, LDA, B, LDB, W, WORK, &
LWORK,IWORK, LIWORK, INFO) bind (c, name = 'magma_dsygvd')
use iso_c_binding
integer(c_int), value :: ITYPE, N, LDA, LDB, LWORK, LIWORK, INFO
character(kind = c_char), value :: JOBZ, UPLO
double precision(c_double), allocatable :: A(:,:), B(:,:), W(:), WORK(:)
integer(c_int), allocatable :: IWORK(:)
end subroutine dsygvdDev
end interface
end module magmafor

program main

use magmafor

INTEGER NIN, NOUT
PARAMETER (NIN=5,NOUT=6)
INTEGER NB, NMAX
PARAMETER (NB=64,NMAX=10)
INTEGER LDA, LDB, LIWORK, LWORK, error
PARAMETER (LDA=NMAX,LDB=NMAX,LIWORK=3+5*NMAX,&
LWORK=1+(6+NB+2*NMAX)*NMAX)

DOUBLE PRECISION, allocatable :: A(:,:), B(:,:), &
W(:), WORK(:)

DOUBLE PRECISION, allocatable :: A1(:,:), B1(:,:), &
W1(:), WORK1(:)
integer, allocatable :: IWORK(:), IWORK1(:)

WRITE (NOUT,*) 'DSYGVD Example Program Results'
WRITE (NOUT,*)

! Skip heading in data file
READ (NIN,*)
READ (NIN,*) N
IF (N.LE.NMAX) THEN

allocate (A(LDA,NMAX),B(LDA,NMAX), W(NMAX), WORK(LWORK),IWORK(LIWORK), stat=error)

allocate (A1(LDA,NMAX),B1(LDA,NMAX), W1(NMAX), WORK1(LWORK),IWORK1(LIWORK))

if(stat.ne.0) then
print*,'error: couldn`t alloc memory'
end if

! Read the upper triangular parts of the matrices A and B

READ (NIN,*) ((A(I,J),J=I,N),I=1,N)
READ (NIN,*) ((B(I,J),J=I,N),I=1,N)

A1 = A
B1 = B

call DSYGVD(2,'V','U',N,A,LDA,B,LDB,W,WORK,LWORK,IWORK,LIWORK,INFO)

IF (INFO.EQ.0) THEN
!
! Print solution
!
WRITE (NOUT,*) 'Eigenvalues'
WRITE (NOUT,99999) (W(J),J=1,N)
WRITE (NOUT,*) 'Eigenvectors'
WRITE (NOUT,99999) ((A(I,J),J=1,N),I=1,N)

END IF
! Magma call

call dsygvdmagma(1,'V','U',N,A1,LDA,B1,LDB,W1,WORK1,LWORK,IWORK1,LIWORK,INFO)

LWOPT = WORK(1)
LIWOPT = IWORK(1)
IF (INFO.EQ.0) THEN
!
! Print solution
!
WRITE (NOUT,*) 'Eigenvalues'
WRITE (NOUT,99999) (W1(J),J=1,N)
WRITE (NOUT,*) 'Eigenvectors'
WRITE (NOUT,99999) ((A1(I,J),J=1,N),I=1,N)

END IF

! Print workspace information
!
IF (LWORK.LT.LWOPT) THEN
WRITE (NOUT,*)
WRITE (NOUT,99995) 'Optimum workspace required = ', LWOPT, &
'Workspace provided = ', LWORK
END IF
IF (LIWORK.LT.LIWOPT) THEN
WRITE (NOUT,*)
WRITE (NOUT,99995) 'Integer workspace required = ', LIWOPT, &
'Integer workspace provided = ', LIWORK
END IF
ELSE
WRITE (NOUT,*) 'NMAX too small'
END IF

!
99999 FORMAT (3X,(6F11.4))
99998 FORMAT (4X,1P,6E11.1)
99997 FORMAT (1X,A,I4,A)
99996 FORMAT (1X,A,I4)
99995 FORMAT (1X,A,I5,/1X,A,I5)
deallocate(A)
deallocate(B)
deallocate(W,WORK)
deallocate(IWORK)

I am getting the accurate result for the lapack call, whereas I get a segmentation fault for that of magma.

Here`s the result:

[krajag1@kidlogin2 Research]$ make
pgfortran -Mcuda -c magmatest2.f90
pgfortran -Mcuda -o magmatest2 magmatest2.o -L/sw/keeneland/magma/1.1/centos5.5_gnu4.4/magma_1.1.0/lib -lcuda -lmagma -lmagmablas -lmagma -lcublas -L/opt/intel/mkl/lib/intel64/ -L/sw/keeneland/cuda/4.0/linux_binary/lib64 -lmkl_intel_lp64 -lmkl_pgi_thread -lmkl_core -mp -lpthread -lcudart -lm -pgf90libs
[krajag1@kidlogin2 Research]$ ./magmatest2 < dat
DSYGVD Example Program Results

Eigenvalues
-3.5411 -0.3347 0.2983 2.2544
Eigenvectors
0.0356 -0.1039 -0.7459 0.1909 -0.3809 0.4322
-0.7845 0.3540 0.2943 1.5644 -0.7144 0.5665
0.3186 -1.0647 1.1184 0.3859
Segmentation fault

[kaushikragavan]

Hi mgates,

I tried profiling the code using pgdbg and got the foll:

pgdbg> run
libcuda.so.1 loaded by ld-linux-x86-64.so.2.
libcublas.so.4 loaded by ld-linux-x86-64.so.2.
libmkl_intel_lp64.so loaded by ld-linux-x86-64.so.2.
libmkl_pgi_thread.so loaded by ld-linux-x86-64.so.2.
libmkl_core.so loaded by ld-linux-x86-64.so.2.
libpthread.so.0 loaded by ld-linux-x86-64.so.2.
libcudart.so.4 loaded by ld-linux-x86-64.so.2.
libm.so.6 loaded by ld-linux-x86-64.so.2.
libnuma.so loaded by ld-linux-x86-64.so.2.
librt.so.1 loaded by ld-linux-x86-64.so.2.
libc.so.6 loaded by ld-linux-x86-64.so.2.
libstdc++.so.6 loaded by ld-linux-x86-64.so.2.
libz.so.1 loaded by ld-linux-x86-64.so.2.
libdl.so.2 loaded by ld-linux-x86-64.so.2.
libgcc_s.so.1 loaded by ld-linux-x86-64.so.2.
DSYGVD Example Program Results

Signalled SIGSEGV at 0x406138, function magma_dsygvd
0x406138: 41 C7 4 24 0 0 0 0 movl $0x0,(%r12)

Thanks,
kaushik

[kaushikragavan]

Hi mgates3,

The seg fault was to due to the dynamic memory allocation. I have attached the modified code and makefile for your reference.
This time I am getting an error for the parameter 1 of magma call

Note: I have tried both numerals and passing an integer variable.