dgesv MAGMA Fortran Device Interface w/ CRAY+OpenACC

Open discussion for MAGMA

dgesv MAGMA Fortran Device Interface w/ CRAY+OpenACC

Postby jah87 » Thu May 31, 2012 2:41 pm

Has anyone had success using the dgesv MAGMA Fortran Device Interface with CRAY's compiler (and OpenACC directives)?

I was able to get the host interface to work correctly, but I get a segmentation fault when using the device interface.

Code: Select all
!$acc data copyin(jac_dev) copy(dy_dev)
!$acc host_data use_device(jac_dev,dy_dev)
!  Call magmaf_dgesv(ny,1,jac_dev,ny,indx_dev,dy_dev,ny,info)               ! MAGMA Host-Interface Solve
  Call magmaf_dgesv_gpu(ny,1,jac_dev,ny,indx_dev,dy_dev,ny,info)           ! MAGMA Device-Interface Solve
!$acc end host_data
!$acc end data
jah87
 
Posts: 21
Joined: Tue May 01, 2012 1:54 pm

Re: dgesv MAGMA Fortran Device Interface w/ CRAY+OpenACC

Postby jah87 » Fri Jun 01, 2012 11:17 am

In the magma_dfortran.f90 file, I notice that jac_dev and dy_dev are being defined as dA and dB, "magma_devptr_t" data types. jac_dev and dy_dev are both REAL(KIND=8) types, so why is magma_devptr_t defined in the Makefile.internal file as "integer(kind=8)"? I'm assuming that the segmentation fault I'm getting has something to do with this scheme of changing jac_dev and dy_dev to device pointers, but I'm not well-versed enough in GPU memory allocation to pinpoint the issue exactly. I'm using -sreal64 at compile time, but this shouldn't affect jac_dev or dy_dev because they are both explicitly declared with KIND=8 (Double precision types remain 64 bit). I tried to account for this with my MAGMA compile (using -fdefault-real-8 and -fdefault-double-8 in FOPTS), but I still get segmentation faults.
jah87
 
Posts: 21
Joined: Tue May 01, 2012 1:54 pm

Re: dgesv MAGMA Fortran Device Interface w/ CRAY+OpenACC

Postby mgates3 » Fri Jun 01, 2012 3:10 pm

Where are jac_dev and dy_dev defined? Are those variables in your application code? Please post more complete source code that includes all relevant definitions.
They should be arrays that can be passed as pointers, not real scalars.
-mark
mgates3
 
Posts: 442
Joined: Fri Jan 06, 2012 2:13 pm

Re: dgesv MAGMA Fortran Device Interface w/ CRAY+OpenACC

Postby jah87 » Fri Jun 01, 2012 3:46 pm

Sorry, here is a more complete picture of the source code:

Code: Select all
Real(8), Dimension(:,:), Allocatable :: jac_dev
Real(8), Dimension(:), Allocatable :: dy_dev
Integer, Dimension(:), Allocatable :: indx_dev

...
Code: Select all
Allocate(jac_dev(ny,ny))
Allocate(dy_dev(ny))
Allocate(indx_dev(ny))

...
Code: Select all
!$acc data copyin(jac_dev) copy(dy_dev)
!$acc host_data use_device(jac_dev,dy_dev)
!  Call magmaf_dgesv(ny,1,jac_dev,ny,indx_dev,dy_dev,ny,info)               ! MAGMA Host-Interface Solve
  Call magmaf_dgesv_gpu(ny,1,jac_dev,ny,indx_dev,dy_dev,ny,info)           ! MAGMA Device-Interface Solve
!$acc end host_data
!$acc end data

...
Code: Select all
Deallocate(jac_dev,dy_dev,indx_dev


My understanding (correct me if I'm wrong), is that the host_data use_device directive(list) "directs the compiler to use the device address of any entry in list, for instance, when passing a variable to procedure." Shouldn't the device address be the pointer you are referring to?
jah87
 
Posts: 21
Joined: Tue May 01, 2012 1:54 pm

Re: dgesv MAGMA Fortran Device Interface w/ CRAY+OpenACC

Postby mgates3 » Fri Jun 01, 2012 5:15 pm

Okay, so jac_dev is not "Real(8)", that is, a scalar. It is "Real(8), Dimension(:,:)", that is, an array (pointer). We define the magma_devptr_t to be an integer the same size as a device pointer, e.g., 64 bits. The pointer size has nothing to do with the floating point size. Integer, single, double, complex, etc. arrays all have 64 bit pointers. (Assuming a 64 bit environment.)

I'm not familiar with the OpenACC extensions. From a quick look through the documentation, yes, your code looks reasonable.

What are your make.inc and Fortran compile & link commands?

-mark
mgates3
 
Posts: 442
Joined: Fri Jan 06, 2012 2:13 pm

Re: dgesv MAGMA Fortran Device Interface w/ CRAY+OpenACC

Postby jah87 » Sat Jun 02, 2012 4:25 pm

My make.inc:

Code: Select all
GPU_TARGET = Fermi

CC        = gcc
NVCC      = nvcc
FORT      = gfortran

ARCH      = ar
ARCHFLAGS = cr
RANLIB    = ranlib

OPTS      = -O3 -DADD_ -fPIC
FOPTS     = -O3 -DADD_ -fdefault-real-8 -fdefault-double-8 -x f95-cpp-input
NVOPTS    = --compiler-options -fno-strict-aliasing -DUNIX -O3 -DADD_
LDOPTS    = -fPIC -Xlinker -zmuldefs

LIB       = -lacml -lpthread -lcublas -lm

CUDADIR1   = /opt/cray/nvidia/default/
CUDADIR2   = /opt/nvidia/cudatoolkit/4.1.28/

LIBDIR    = -L/opt/acml/4.4.0/gfortran64/lib \
            -L$(CUDADIR1)/lib64 \
            -L$(CUDADIR2)/lib64
INC       = -I$(CUDADIR1)/include \
            -I$(CUDADIR2)/include


Relevant Makefile instructions:

Code: Select all
INCLUDE_PATH = ../magma-1.2.0/include
LIBRARY_PATH = ../magma-1.2.0/lib
 FC     = ftn
 FCP    = ftn
 LDR    = ftn
 LDRP   = ftn
OPT     = -O2 -h noomp
R8      = -sreal64
MAGMA = -I$(INCLUDE_PATH) -L$(LIBRARY_PATH) -lcublas -lmagma -lmagmablas -lmagma -lcublas
FFLAGS  = $(OPT) $(R8)   # active compiler options
LDRFLAGS= $(OPT) $(R8) # linker flags
BASE = control.o data.o common.o net_preprocess.o ffn.o full_net.o solve_be.o solve_bd.o match.o net.o flux.o eosnom90.o screen.o
xnet: $(BASE) jacobian_dense.o
        $(LDR) $(LDRFLAGS) -o xnet $(BASE) jacobian_dense.o $(MAGMA)
.f90.o:
        $(FC) $(FFLAGS) -c $<
jah87
 
Posts: 21
Joined: Tue May 01, 2012 1:54 pm

Re: dgesv MAGMA Fortran Device Interface w/ CRAY+OpenACC

Postby keitat » Mon Jun 04, 2012 5:15 pm

Have you ever tried libsci_acc bundled with Cray OpenACC compiler?
keitat
 
Posts: 8
Joined: Tue Jan 24, 2012 2:19 pm

Re: dgesv MAGMA Fortran Device Interface w/ CRAY+OpenACC

Postby jah87 » Mon Jun 04, 2012 5:41 pm

keitat wrote:Have you ever tried libsci_acc bundled with Cray OpenACC compiler?


Yes, I have. I was hoping to do some direct comparisons between libsci_acc (device-interface) and MAGMA (device-interface).
jah87
 
Posts: 21
Joined: Tue May 01, 2012 1:54 pm


Return to User discussion

Who is online

Users browsing this forum: No registered users and 4 guests

cron