Magma with large mem req. + Fortran90

Open discussion for MAGMA

Magma with large mem req. + Fortran90

Postby Dynetrekk » Wed Mar 07, 2012 11:02 am

Hi, I've noticed that Magma now supports several features that are very interesting. In particular, I want to use the Fortran 90 CPU interface to LU factorization (the getrf routine). In that context, I have some questions.

Does the CPU interface to Magma automagically handle matrices which are too large to keep in GPU memory (but can be held on the regular CPU memory)?

If not, will a multi-GPU setup handle a matrix which is larger than what a single GPU can hold, but smaller than what all the GPUs combined can hold?

Can anyone point me to some example code (preferably with compilation instructions) using the CPU+Fortran90 interface?
Dynetrekk
 
Posts: 3
Joined: Wed Mar 07, 2012 10:51 am

Re: Magma with large mem req. + Fortran90

Postby mgates3 » Thu Mar 08, 2012 5:50 pm

For LU, Cholesky, and QR, there is multi-GPU support which handles matrices that will fit in the combined memory of multiple GPUs. Set the environment variable MAGMA_NUM_GPUS to specify the number of GPUs to use, and call the normal CPU interface. For example:

export MAGMA_NUM_GPUS=2

For LU and Cholesky, there are non-resident versions that are automatically called if the matrix does not fit into a single GPU's memory. Call the normal CPU interface.

-mark
mgates3
 
Posts: 330
Joined: Fri Jan 06, 2012 2:13 pm

Re: Magma with large mem req. + Fortran90

Postby Dynetrekk » Fri Mar 09, 2012 6:11 am

That's great news! Now if only I could get my code to link. I've found all the libraries and set all the paths (at least I think so), but the ordering of the libraries is still incorrect. So far I've tried this (and some other combinations):
LIBS = -lcuda -lcudart -lcublas -lacml -lpthread -lm -lmagma -lmagmablas
gfortran cgetrf_f_edited.f90 -O3 -DADD_ -x f95-cpp-input -Dmagma_devptr_t="integer(kind=8)" $(LIBS) -L/export/cuda/cuda4.1.28/lib64 -L. ${MAGMA_INCLUDE_OPTS} ${MAGMA_PRE_COMPILE_OPTS} ${MAGMA_POST_COMPILE_OPTS} -L/home/gy07/gy07/gnag/magma/magma1.0_RC5_ATLAS/cuda4/lib -I/home/gy07/gy07/gnag/magma/magma1.0_RC5_ATLAS/cuda4/include -L/export/acmlg/gfortran64/lib

I'd appreciate any example code or example compilation instructions.
Dynetrekk
 
Posts: 3
Joined: Wed Mar 07, 2012 10:51 am

Re: Magma with large mem req. + Fortran90

Postby Dynetrekk » Fri Mar 09, 2012 6:13 am

By the way - the errors I encounter in compilation and linking are all of this form:
/tmp/ccATe4Gy.o: In function `MAIN__':
cgetrf_f_edited.f90:(.text+0x75): undefined reference to `cublas_init_'
cgetrf_f_edited.f90:(.text+0xba8): undefined reference to `cublas_shutdown_'
/home/gy07/gy07/gnag/magma/magma1.0_RC5_ATLAS/cuda4/lib/libmagma.a(cpotrf.o): In function `magma_cpotrf':
cpotrf.cpp:(.text+0x35b): undefined reference to `cublasGetCurrentCtx'
cpotrf.cpp:(.text+0x3c5): undefined reference to `cublasSetError'
cpotrf.cpp:(.text+0x49e): undefined reference to `cublasGetCurrentCtx'
cpotrf.cpp:(.text+0x52e): undefined reference to `cublasSetError'
cpotrf.cpp:(.text+0x5c2): undefined reference to `cublasGetCurrentCtx'

...and so on.
Dynetrekk
 
Posts: 3
Joined: Wed Mar 07, 2012 10:51 am

Re: Magma with large mem req. + Fortran90

Postby mgates3 » Fri Mar 09, 2012 3:09 pm

See the testing directory. Probably magma has to go before cuda & cublas. This is what works on my machines:

# on Linux with MKL
gcc -O3 -DADD_ -DGPUSHMEM=200 -fPIC testing_zgetrf.o -o testing_zgetrf \
lin/liblapacktest.a -L${MAGMA_DIR}/lib -lcuda -lmagma -lmagmablas -lmagma -L${MKLROOT}/lib/intel64 -L${CUDADIR}/lib64 -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_core -lpthread -lcublas -lm -fopenmp

# on Mac OS
gcc -m32 -Wall -O3 -DADD_ -DGPUSHMEM=130 -fPIC testing_zgetrf.o -o testing_zgetrf \
lin/liblapacktest.a -L${MAGMA_DIR}/lib -lcuda -lmagma -lmagmablas -lmagma -L${CUDADIR}/lib -framework Accelerate -lcublas -lcudart -lm

where paths are system dependent, but for instance,
CUDADIR = /mnt/scratch/cuda
MKLROOT = /mnt/scratch/sw/intel/2011.2.137/composerxe-2011.2.137/mkl
MAGMA_DIR = /mnt/scratch/magma_1.1.0
mgates3
 
Posts: 330
Joined: Fri Jan 06, 2012 2:13 pm


Return to User discussion

Who is online

Users browsing this forum: No registered users and 2 guests