dgesv MAGMA Fortran Device Interface w/ CRAY+OpenACC

[jah87]

Has anyone had success using the dgesv MAGMA Fortran Device Interface with CRAY's compiler (and OpenACC directives)?

I was able to get the host interface to work correctly, but I get a segmentation fault when using the device interface.

Code: Select all

!$acc data copyin(jac_dev) copy(dy_dev)
!$acc host_data use_device(jac_dev,dy_dev)
!  Call magmaf_dgesv(ny,1,jac_dev,ny,indx_dev,dy_dev,ny,info)               ! MAGMA Host-Interface Solve
  Call magmaf_dgesv_gpu(ny,1,jac_dev,ny,indx_dev,dy_dev,ny,info)           ! MAGMA Device-Interface Solve
!$acc end host_data
!$acc end data

[jah87]

In the magma_dfortran.f90 file, I notice that jac_dev and dy_dev are being defined as dA and dB, "magma_devptr_t" data types. jac_dev and dy_dev are both REAL(KIND=8) types, so why is magma_devptr_t defined in the Makefile.internal file as "integer(kind=8)"? I'm assuming that the segmentation fault I'm getting has something to do with this scheme of changing jac_dev and dy_dev to device pointers, but I'm not well-versed enough in GPU memory allocation to pinpoint the issue exactly. I'm using -sreal64 at compile time, but this shouldn't affect jac_dev or dy_dev because they are both explicitly declared with KIND=8 (Double precision types remain 64 bit). I tried to account for this with my MAGMA compile (using -fdefault-real-8 and -fdefault-double-8 in FOPTS), but I still get segmentation faults.

[mgates3]

Where are jac_dev and dy_dev defined? Are those variables in your application code? Please post more complete source code that includes all relevant definitions.
They should be arrays that can be passed as pointers, not real scalars.
-mark

[jah87]

Sorry, here is a more complete picture of the source code:

Code: Select all

Real(8), Dimension(:,:), Allocatable :: jac_dev
Real(8), Dimension(:), Allocatable :: dy_dev
Integer, Dimension(:), Allocatable :: indx_dev

...

Code: Select all

Allocate(jac_dev(ny,ny))
Allocate(dy_dev(ny))
Allocate(indx_dev(ny))

...

Code: Select all

!$acc data copyin(jac_dev) copy(dy_dev)
!$acc host_data use_device(jac_dev,dy_dev)
!  Call magmaf_dgesv(ny,1,jac_dev,ny,indx_dev,dy_dev,ny,info)               ! MAGMA Host-Interface Solve
  Call magmaf_dgesv_gpu(ny,1,jac_dev,ny,indx_dev,dy_dev,ny,info)           ! MAGMA Device-Interface Solve
!$acc end host_data
!$acc end data

...

Code: Select all

Deallocate(jac_dev,dy_dev,indx_dev

My understanding (correct me if I'm wrong), is that the host_data use_device directive(list) "directs the compiler to use the device address of any entry in list, for instance, when passing a variable to procedure." Shouldn't the device address be the pointer you are referring to?

[mgates3]

Okay, so jac_dev is not "Real(8)", that is, a scalar. It is "Real(8), Dimension(:,:)", that is, an array (pointer). We define the magma_devptr_t to be an integer the same size as a device pointer, e.g., 64 bits. The pointer size has nothing to do with the floating point size. Integer, single, double, complex, etc. arrays all have 64 bit pointers. (Assuming a 64 bit environment.)

I'm not familiar with the OpenACC extensions. From a quick look through the documentation, yes, your code looks reasonable.

What are your make.inc and Fortran compile & link commands?

-mark

[jah87]

My make.inc:

Code: Select all

GPU_TARGET = Fermi

CC        = gcc
NVCC      = nvcc
FORT      = gfortran

ARCH      = ar
ARCHFLAGS = cr
RANLIB    = ranlib

OPTS      = -O3 -DADD_ -fPIC
FOPTS     = -O3 -DADD_ -fdefault-real-8 -fdefault-double-8 -x f95-cpp-input
NVOPTS    = --compiler-options -fno-strict-aliasing -DUNIX -O3 -DADD_
LDOPTS    = -fPIC -Xlinker -zmuldefs

LIB       = -lacml -lpthread -lcublas -lm

CUDADIR1   = /opt/cray/nvidia/default/
CUDADIR2   = /opt/nvidia/cudatoolkit/4.1.28/

LIBDIR    = -L/opt/acml/4.4.0/gfortran64/lib \
            -L$(CUDADIR1)/lib64 \
            -L$(CUDADIR2)/lib64
INC       = -I$(CUDADIR1)/include \
            -I$(CUDADIR2)/include


Relevant Makefile instructions:

Code: Select all

INCLUDE_PATH = ../magma-1.2.0/include
LIBRARY_PATH = ../magma-1.2.0/lib
 FC     = ftn
 FCP    = ftn
 LDR    = ftn
 LDRP   = ftn
OPT     = -O2 -h noomp
R8      = -sreal64
MAGMA = -I$(INCLUDE_PATH) -L$(LIBRARY_PATH) -lcublas -lmagma -lmagmablas -lmagma -lcublas
FFLAGS  = $(OPT) $(R8)   # active compiler options
LDRFLAGS= $(OPT) $(R8) # linker flags
BASE = control.o data.o common.o net_preprocess.o ffn.o full_net.o solve_be.o solve_bd.o match.o net.o flux.o eosnom90.o screen.o
xnet: $(BASE) jacobian_dense.o
        $(LDR) $(LDRFLAGS) -o xnet $(BASE) jacobian_dense.o $(MAGMA)
.f90.o:
        $(FC) $(FFLAGS) -c $<


[keitat]

Have you ever tried libsci_acc bundled with Cray OpenACC compiler?

[jah87]

Have you ever tried libsci_acc bundled with Cray OpenACC compiler?

Yes, I have. I was hoping to do some direct comparisons between libsci_acc (device-interface) and MAGMA (device-interface).