magma linking with cublas

[davelarge]

Hi all.

I'm sure that I'm missing something really simple here, but I'm at my wits end so perhaps someone can take the time to help me out.

I compile magma-1.2.1 using CUDA-4.2, where cuda is installed in /usr/local/cuda and I copy the magma libraries to /usr/local/magma/lib and the headers to /usr/local/magma/include.

When I run ldconfig -p | grep cuda, the linker tells me that it knows about all the libraries in /usr/local/cuda/lib

I then have some of my own code which I am trying to link against these libraries. The Makefile is as follows:

CC = nvcc
CFLAGS = -arch=sm_12
LIBDIR = -L/usr/local/magma/lib
INCDIR = -I/usr/local/magma/include -I/usr/local/cuda/include
LIBS = -lcublas -lmagma -lmagmablas -llapack -lblas
OBJECTS = Functions.o

cuPrintf.o : cuPrintf.cu
${CC} ${CFLAGS} -c $<

Functions.o : Functions.cu Functions.h cuPrintf.o
${CC} ${CFLAGS} -c ${INCDIR} $<

testdiagonalization : testdiagonalization.cu ${OBJECTS}
${CC} -o $@ ${CFLAGS} ${INCDIR} ${LIBDIR} ${LIBS} $@.cu ${OBJECTS}

When I run make testdiagonalization, I get a string of "undefined reference" errors, all to cublas functions. It seems to me as if the magma libraries cannot make contact with the cuda libraries. How can I fix this?

Many thanks.

[mgates3]

I think it's the order of your LIBS. Try putting -lcublas at the end. You probably also need the runtime, -lcudart, though nvcc may add that automatically. You may need to repeat -lmagma after -lmagmablas.

-mark

[davelarge]

mgates3, putting -lcublas at the end did the trick. Thanks.

Is there any way that I should have known that? It seems simultaneously absurdly simple and hugely counterintuitive that the order of the libraries should matter like that.

[mgates3]

Mostly experience. The gcc compiler man page says for -l library:

It makes a difference where in the command you write this option; the linker
searches and processes libraries and object files in the order they are
specified. Thus, foo.o -lz bar.o searches library z after file foo.o but
before bar.o. If bar.o refers to functions in z, those functions may not be
loaded.

Most compilers (gcc, icc, nvcc, ...) follow the same rule. Determining the correct order is sometimes a non-intuitive, trial-and-error process. In this case, Magma uses Lapack, blas, and cublas, so Magma has to be first. Lapack uses blas, so blas has to be after Lapack. At the moment, Magma and Magmablas have a circular dependency, so one may need to be repeated.

-mark

[davelarge]

Thanks mark - most helpful.