MAGMA GEMM Sources for Fermi Released

[mgates3]

Can you please cite specific code for your assertions? Perhaps there is a misunderstanding of the code.

In the current MAGMA release, magmablas/dgemm_fermi.cu has this code to call cublas (via magma_dgemm) if the memory exceeds the texture buffer size:

Code: Select all

    size_t CUBLAS_MAX_1DBUF_SIZE = ((1 << 27) - 512);
    if ( sizeA >= CUBLAS_MAX_1DBUF_SIZE ||
         sizeB >= CUBLAS_MAX_1DBUF_SIZE )
    {
        magma_dgemm( transA, transB, m, n, k, alpha,
                     dA, ldda, dB, lddb,
                     beta, dC, lddc, queue );
        return;
    }

That's the only time it calls cublas. In all other cases, it launches dgemm_kernel_fermi_XX, where XX is the transposition of A and B. Note the CUDA grid is ceiling( m / blocksize ) x ceiling( n / blocksize ), for various block sizes. That is part of dealing with arbitrary m, n, k.

-mark

[anikam]

Hello Mark,
Thanks a lot.
I am facing a lot of troubles installing the Magma library on my pc.
Whenever I try to compile the example in magma/example folder, I get these type of errors.
/usr/local/magma/lib/libmagma.so: undefined reference to `dorgql_'
/usr/local/magma/lib/libmagma.so: undefined reference to `ssteqr_'
/usr/local/magma/lib/libmagma.so: undefined reference to `dlaqp2_'
/usr/local/magma/lib/libmagma.so: undefined reference to `cungql_'
/usr/local/magma/lib/libmagma_sparse.so: undefined reference to `zlarnv_'
I am pretty sure, I have set the paths correctly.
I am using Cuda 7.5 and OpenBlas.

Can you please help me with this?


Thanks and Regards
Abhishek Nikam

[mgates3]

Those are all LAPACK routines. I think OpenBLAS usually provides a complete copy of LAPACK. Can you provide information about your computer, e.g., OS, compiler, version of BLAS, etc.?
Are you compiling using Make? If so, what is your make.inc file?
If you are compiling with CMake, how is it configuring MAGMA?
It's possible that your OpenBLAS assumes a different Fortran naming scheme. Presumably, you used -DADD_ to get these names with an underscore. There's also -DNOCHANGE and -DUPCASE, depending on your BLAS library. For instance:

On my machine, OpenBLAS assumes trailing underscore (-DADD_):

Code: Select all

bunsen slate/blaspp> nm /mnt/scratch/openblas/lib/libopenblas.so | grep -i cunglq
0000000000a00ff0 T LAPACKE_cunglq
0000000000a01170 T LAPACKE_cunglq_work
0000000000765f70 T cunglq_

While Intel MKL allows trailing underscore (-DADD_), upcase (-DUPCASE), or no underscope (-DNOCHANGE):

Code: Select all

bunsen slate/blaspp> nm $MKLROOT/lib/intel64/libmkl_gf_lp64.so | grep -i cunglq
00000000001b2410 T CUNGLQ
00000000003bb8a0 T LAPACKE_cunglq
00000000003bba40 T LAPACKE_cunglq_work
00000000001b2410 T cunglq
00000000001b2410 T cunglq_
                 U mkl_lapack_cunglq

-mark

[anikam]

Hello Mark,
I really thank you for the reply.
I am using an Intel-I5 machine with Ubuntu 14 on it.
I am using gcc 4.6.4 and g++ 5.4.1.

I am compiling it with make and I am posting my make.inc file for your reference.
I have commented out or deleted any options related to fortran compilation as I intend not to use fortran.
I have downloaded OpenBlas using apt-get
sudo apt-get install libopenblas-dev

My libopenblas.so file is present in usr/lib folder.
I am not exactly sure on how to give the path for openblas
Should I add export OPENBLASDIR=/usr/ in my bashrc?

Also, Isnt there a way in which I can install Magma Library without using any LAPACK routines as I am mainly interested in using Magmablas sgemm and dgemm routines.

My make.inc

export CUDADIR=/usr/local/cuda-7.5
export prefix=/usr/local/magma

CC = gcc
CXX = g++
NVCC = nvcc
ARCH = ar
ARCHFLAGS = cr
RANLIB = ranlib
FPIC = -fPIC


CFLAGS = -O3 $(FPIC) -DADD_ -Wall -std=c++11 -fopenmp -DMAGMA_SETAFFINITY
FFLAGS = -O3 $(FPIC) -DADD_ -Wall -Wno-unused-dummy-argument
NVCCFLAGS = -O3 -DADD_ -Xcompiler "-fno-strict-aliasing $(FPIC)" -cudart shared
LDFLAGS = $(FPIC) -fopenmp

LIB = -lopenblas -lcublas -lcudart
CUDADIR =/usr/local/cuda-7.5
OPENBLASDIR=/usr/

-include make.check-openblas
-include make.check-cuda

LIBDIR = -L$(CUDADIR)/lib64 \
-L$(OPENBLASDIR)/lib

INC = -I$(CUDADIR)/include

Also the error

g++ -fPIC -fopenmp -Wl,-rpath,../lib \
-o testing/testing_zaxpy testing/testing_zaxpy.o \
-L./testing -ltest \
-L./lib -lmagma \
-L./testing/lin -llapacktest \
-L/usr/local/cuda-7.5/lib64 -L/usr//lib -lopenblas -lcublas -lcudart
testing/testing_zaxpy.o: In function `main':
testing_zaxpy.cpp:(.text.startup+0x2e5): undefined reference to `zlarnv_'
testing_zaxpy.cpp:(.text.startup+0x2fa): undefined reference to `zlarnv_'

[anikam]

Hello Mark,
I tried multiple things but when Make test tries to compile testing_zaxpy.o, it fails.

g++ -fPIC -fopenmp -Wl,-rpath,../lib \
-o testing/testing_zaxpy testing/testing_zaxpy.o \
-L./testing -ltest \
-L./lib -lmagma \
-L./testing/lin -llapacktest \
-L/usr/local/cuda-7.5/lib64 -L/usr//lib -lopenblas -lcublas -lcudart -lcusparse
testing/testing_zaxpy.o: In function `main':
testing_zaxpy.cpp:(.text.startup+0x2e5): undefined reference to `zlarnv_'
testing_zaxpy.cpp:(.text.startup+0x2fa): undefined reference to `zlarnv_'


This is the error message.
I think so there can be a couple of factors which might be affecting it.
1) I have not mentioned any fortran compiler
2)The path to my OpenBlas is wrong.

I am not sure what exactly is messing up the make test.
The make test runs for quite a significant amount of time and builds everything bit I guess the linking fails.
I request for help, I dont have any clue as to what to do next.

[mgates3]

Yes, setting OPENBLASDIR=/usr is fine. (No need for a trailing slash, though it shouldn't hurt -- just gives a double // in your link line.) If your path were wrong, you would get a link error like:
/usr/bin/ld: cannot find -lopenblas

Please use "nm" to check your OpenBLAS library for the missing routines, as I demonstrated. Probably Ubuntu doesn't include LAPACK routines in its OpenBLAS package. In that case, install the Ubuntu LAPACK package and add it to LIB in your make.inc.

MAGMA depends on LAPACK. You can extract pieces of MAGMA that might not, but as a whole it uses LAPACK extensively. For instance, even the gemm tester use LAPACK's larnv to initialize matrices with random numbers, and lange to check the norm of the error.

-mark

[anikam]

Hello Mark,

Thanks for your help.
As, I just need the MagmaBlas dgemm function I just wrote a sample code.
In my sample test code, I malloced some host arrays and some device arrays.
Initialized it and called dgemm function.
The code compile properly after giving the -lmagma and -lcusparse flags.
I had a couple of questions for you, I request you to please help me with that

1)I am using the Magmablass gemm written by you for GPU Hardware simulation
2)So, I need to run it on gpgpu sim
3)While doing that, I am facing some issues like
4)CUDA runtime error: no CUDA-capable device is detected (38) in magma_getdevice at interface_cuda/interface.cpp:552
5)Error: magmablasGetQueue requires magma_init() to be called first for MAGMA v1 compatability.
6)Cannot bind A to texture: no CUDA-capable device is detected (38)

I think so, I can figure out the 4th issue, can you please suggest me something such that I can hack the 5th and 6th issue.
Also, The magmablas would work on matrix sizes like 10,000 * 10,000, right?

Thanks a lot for the help.

[mgates3]

Do you include <magma.h> or <magma_v2.h>? I highly recommend using <magma_v2.h>.

In either case, you need to do magma_init() before calling other MAGMA routines, and magma_finalize() afterwards. You can repeat and nest init/finalize.

The MAGMA gemm works on sizes up to the maximum CUDA texture memory size, 2^27 = 134217728 entries. For square matrices, that's about m = n = k = 11585. Beyond that, it calls cuBLAS gemm.

Remember that cuBLAS gemm is now better optimized than MAGMA gemm.

-mark

[anikam]

Hello Mark,

Thanks a lot.
Cannot we just use the Global Memory for larger sizes in MAGMABLAS Gemm?

[mgates3]

You can use global memory, but you lose considerable performance by doing so. It would be better to build a large gemm using smaller gemms. For instance,

Code: Select all

C = [ A1 A2 A3 ] [ B1 ]
                 [ B2 ]
                 [ B3 ]
  = A1*B1 + A2*B2 + A3*B3.

Otherwise, undefining TEXTURE_1D in magma/magmablas/*gemm_fermi_kernels.h should select the non-texture version. The code is not setup to compile both versions together.

NB: The size restriction is for entries of A or B. The size of C is not restricted.

Again, I recommend using cuBLAS gemm.

-mark