Magma with large mem req. + Fortran90

[Dynetrekk]

Hi, I've noticed that Magma now supports several features that are very interesting. In particular, I want to use the Fortran 90 CPU interface to LU factorization (the getrf routine). In that context, I have some questions.

Does the CPU interface to Magma automagically handle matrices which are too large to keep in GPU memory (but can be held on the regular CPU memory)?

If not, will a multi-GPU setup handle a matrix which is larger than what a single GPU can hold, but smaller than what all the GPUs combined can hold?

Can anyone point me to some example code (preferably with compilation instructions) using the CPU+Fortran90 interface?

[mgates3]

For LU, Cholesky, and QR, there is multi-GPU support which handles matrices that will fit in the combined memory of multiple GPUs. Set the environment variable MAGMA_NUM_GPUS to specify the number of GPUs to use, and call the normal CPU interface. For example:

export MAGMA_NUM_GPUS=2

For LU and Cholesky, there are non-resident versions that are automatically called if the matrix does not fit into a single GPU's memory. Call the normal CPU interface.

-mark

[Dynetrekk]

That's great news! Now if only I could get my code to link. I've found all the libraries and set all the paths (at least I think so), but the ordering of the libraries is still incorrect. So far I've tried this (and some other combinations):
LIBS = -lcuda -lcudart -lcublas -lacml -lpthread -lm -lmagma -lmagmablas
gfortran cgetrf_f_edited.f90 -O3 -DADD_ -x f95-cpp-input -Dmagma_devptr_t="integer(kind=8)" $(LIBS) -L/export/cuda/cuda4.1.28/lib64 -L. ${MAGMA_INCLUDE_OPTS} ${MAGMA_PRE_COMPILE_OPTS} ${MAGMA_POST_COMPILE_OPTS} -L/home/gy07/gy07/gnag/magma/magma1.0_RC5_ATLAS/cuda4/lib -I/home/gy07/gy07/gnag/magma/magma1.0_RC5_ATLAS/cuda4/include -L/export/acmlg/gfortran64/lib

I'd appreciate any example code or example compilation instructions.

[Dynetrekk]

By the way - the errors I encounter in compilation and linking are all of this form:
/tmp/ccATe4Gy.o: In function `MAIN__':
cgetrf_f_edited.f90:(.text+0x75): undefined reference to `cublas_init_'
cgetrf_f_edited.f90:(.text+0xba8): undefined reference to `cublas_shutdown_'
/home/gy07/gy07/gnag/magma/magma1.0_RC5_ATLAS/cuda4/lib/libmagma.a(cpotrf.o): In function `magma_cpotrf':
cpotrf.cpp:(.text+0x35b): undefined reference to `cublasGetCurrentCtx'
cpotrf.cpp:(.text+0x3c5): undefined reference to `cublasSetError'
cpotrf.cpp:(.text+0x49e): undefined reference to `cublasGetCurrentCtx'
cpotrf.cpp:(.text+0x52e): undefined reference to `cublasSetError'
cpotrf.cpp:(.text+0x5c2): undefined reference to `cublasGetCurrentCtx'

...and so on.

[mgates3]

See the testing directory. Probably magma has to go before cuda & cublas. This is what works on my machines:

# on Linux with MKL
gcc -O3 -DADD_ -DGPUSHMEM=200 -fPIC testing_zgetrf.o -o testing_zgetrf \
lin/liblapacktest.a -L${MAGMA_DIR}/lib -lcuda -lmagma -lmagmablas -lmagma -L${MKLROOT}/lib/intel64 -L${CUDADIR}/lib64 -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_core -lpthread -lcublas -lm -fopenmp

# on Mac OS
gcc -m32 -Wall -O3 -DADD_ -DGPUSHMEM=130 -fPIC testing_zgetrf.o -o testing_zgetrf \
lin/liblapacktest.a -L${MAGMA_DIR}/lib -lcuda -lmagma -lmagmablas -lmagma -L${CUDADIR}/lib -framework Accelerate -lcublas -lcudart -lm

where paths are system dependent, but for instance,
CUDADIR = /mnt/scratch/cuda
MKLROOT = /mnt/scratch/sw/intel/2011.2.137/composerxe-2011.2.137/mkl
MAGMA_DIR = /mnt/scratch/magma_1.1.0