Building MAGMA 1.3 on Mac OS X Mountain Lion

[mgalloy]

I am trying to build MAGMA 1.3 on Mac OS X Mountain Lion. I am using the make.inc.macos that is included. I'm getting the following error when building:

Code: Select all

gcc -m32 -O3 -DADD_ -DHAVE_CUBLAS -DGPUSHMEM=130 -I/usr/local/cuda/include -I../include -I../control -c zhetrd_hb2st.cpp -o zhetrd_hb2st.o
zhetrd_hb2st.cpp:100: error: ‘pthread_barrier_t’ does not name a type
zhetrd_hb2st.cpp: In constructor ‘magma_zbulge_data::magma_zbulge_data(magma_int_t, magma_int_t, magma_int_t, magma_int_t, magma_int_t, magma_int_t, magma_int_t, cuDoubleComplex*, magma_int_t, cuDoubleComplex*, magma_int_t, cuDoubleComplex*, cuDoubleComplex*, magma_int_t, volatile magma_int_t*)’:
zhetrd_hb2st.cpp:77: error: ‘barrier’ was not declared in this scope
zhetrd_hb2st.cpp:77: error: ‘pthread_barrier_init’ was not declared in this scope
zhetrd_hb2st.cpp: In destructor ‘magma_zbulge_data::~magma_zbulge_data()’:
zhetrd_hb2st.cpp:82: error: ‘barrier’ was not declared in this scope
zhetrd_hb2st.cpp:82: error: ‘pthread_barrier_destroy’ was not declared in this scope
zhetrd_hb2st.cpp: In function ‘void* magma_zhetrd_hb2st_parallel_section(void*)’:
zhetrd_hb2st.cpp:354: error: ‘pthread_barrier_t’ was not declared in this scope
zhetrd_hb2st.cpp:354: error: ‘barrier’ was not declared in this scope
zhetrd_hb2st.cpp:354: error: ‘class magma_zbulge_data’ has no member named ‘barrier’
zhetrd_hb2st.cpp:412: error: ‘pthread_barrier_wait’ was not declared in this scope
zhetrd_hb2st.cpp:444: error: ‘pthread_barrier_wait’ was not declared in this scope
make[1]: *** [zhetrd_hb2st.o] Error 1
make: *** [libmagma] Error 2

My understanding is that pthread barriers are not supported on Mac OS X. Is it possible to build MAGMA on Mac OS X?

[mgates3]

Indeed, Mac OS X does not implement pthread barrier, which is an optional part of the pthread standard. You can remove the few offending functions from the Makefiles, or add pthread_barrier. I've used the implementation found here:
http://stackoverflow.com/questions/3640 ... v-s-dispat
to compile MAGMA on Mac OS X, though I have NOT tested the functions that use the pthread barrier (the 2-stage symmetric eigenvalue). I put those prototypes in magma/control/common_magma.h, with appropriate #if around it:

Code: Select all

#ifdef __APPLE__

#include <pthread.h>

#ifdef __cplusplus
extern "C" {
#endif

// from http://stackoverflow.com/questions/3640853/performance-test-sem-t-v-s-dispatch-semaphore-t-and-pthread-once-t-v-s-dispat

// *sigh* OSX does not have pthread_barrier
typedef int pthread_barrierattr_t;
typedef struct {
    pthread_mutex_t mutex;
    pthread_cond_t cond;
    int count;
    int tripCount;
} pthread_barrier_t;

int pthread_barrier_init( pthread_barrier_t *barrier,
                          const pthread_barrierattr_t *attr, unsigned int count );

int pthread_barrier_destroy( pthread_barrier_t *barrier );

int pthread_barrier_wait( pthread_barrier_t *barrier );

#ifdef __cplusplus
}
#endif

#endif        // __APPLE__


[mgalloy]

Thanks, this getting me farther along! It is good to know that someone has gotten this to work on Mac OS X already.

-Mike

[mgalloy]

OK, I am trying to build 64-bit (have removed the -m32 from the various OPTS variables in make.inc):

Code: Select all

OPTS      = -O3 -DADD_
F77OPTS   = -O3 -DADD_
FOPTS     = -O3 -DADD_ -x f95-cpp-input
NVOPTS    = -DADD_ --compiler-options -fno-strict-aliasing -DUNIX -O3
LDOPTS    = -fPIC

But, I get stuck once we start linking in the testing directory:

Code: Select all

gcc -O3 -DADD_ -DHAVE_CUBLAS -DGPUSHMEM=130 -I/usr/local/cuda/include -I../include -I../control -c testing_zgemm.cpp -o testing_zgemm.o
gcc -fPIC -DGPUSHMEM=130  testing_zgemm.o  -o testing_zgemm \
   libtest.a lin/liblapacktest.a -L../lib -lmagma -lmagmablas -lmagma \
   -L/usr/local/cuda/lib  \
   -framework Accelerate -lcublas -lcudart -lm -lstdc++.6
ld: warning: ignoring file ../lib/libmagmablas.a, file was built for archive which is not the architecture being linked (x86_64): ../lib/libmagmablas.a
gcc -O3 -DADD_ -DHAVE_CUBLAS -DGPUSHMEM=130 -I/usr/local/cuda/include -I../include -I../control -c testing_zgemv.cpp -o testing_zgemv.o
gcc -fPIC -DGPUSHMEM=130  testing_zgemv.o  -o testing_zgemv \
   libtest.a lin/liblapacktest.a -L../lib -lmagma -lmagmablas -lmagma \
   -L/usr/local/cuda/lib  \
   -framework Accelerate -lcublas -lcudart -lm -lstdc++.6
ld: warning: ignoring file ../lib/libmagmablas.a, file was built for archive which is not the architecture being linked (x86_64): ../lib/libmagmablas.a
Undefined symbols for architecture x86_64:
  "_magmablas_zgemv", referenced from:
      _main in testing_zgemv.o
ld: symbol(s) not found for architecture x86_64
collect2: ld returned 1 exit status
make[1]: *** [testing_zgemv] Error 1
make: *** [test] Error 2

I'm not familiar with ar/ranlib, are these flags set correctly? Is there some other mistake in how libmagma/libmagmablas are created?

Code: Select all

ARCH      = ar
ARCHFLAGS = cr
RANLIB    = ranlib

[mgates3]

First, notice this in your output:

warning: ignoring file ../lib/libmagmablas.a, file was built for archive which is not the architecture being linked (x86_64)

So it looks like you didn't compile all of MAGMA as 64-bit. If you are compiling in the same directory where you previously used -m32, then you need to 'make clean' and then 'make'.

Second, there are some problems with Mac OS X 64-bit veclib. It's really a prototype problem. A few single-precision functions we expect to return a float, but in 64-bit veclib they actually return a double. For instance,
float snrm2_( const int* n, const float *x, const int* incx );
is (apparently) actually
double snrm2_( const int* n, const float *x, const int* incx );
These need to be fixed in magma/include/magma_slapack.h and magma_clapack.h. Possibly other places. Without this fix, some functions like sgeev, cgeev, and most of the single-precision testers will fail. This problem is not evident when compiling 32-bit, though I'm not exactly sure why not. I generally compile -m32 on MacOS to avoid the issue.

-mark

[mgalloy]

I noticed the warning about the archive not built for x86_64, but I definitely did a "make clean" beforehand. There are no .a files in lib/ (actually no .o, .a, .so anywhere in the MAGMA distribution) when I start the build:

Code: Select all

$ find . -name '*.o'
$ find . -name '*.so'
$ find . -name '*.a'

I'm not sure why it would give the warning that something was not built for x86_64. Is it picking up something outside MAGMA, like CUDA? I did notice CUDA includes both architectures, but they are dylib's and shouldn't be included in the archive, right?

Code: Select all

lib$ file libcudart.dylib
libcudart.dylib: Mach-O universal binary with 2 architectures
libcudart.dylib (for architecture x86_64):   Mach-O 64-bit dynamically linked shared library x86_64
libcudart.dylib (for architecture i386):   Mach-O dynamically linked shared library i386

I also noticed these warnings in the output:

Code: Select all

/usr/bin/ranlib: file: ../lib/libmagma.a(magma_param.o) has no symbols
/usr/bin/ranlib: file: ../lib/libmagma.a(magma.o) has no symbols
/usr/bin/ranlib: file: ../lib/libmagma.a(get_nb_fermi.o) has no symbols
/usr/bin/ranlib: file: ../lib/libmagma.a(get_nb_kepler.o) has no symbols
/usr/bin/ranlib: file: ../lib/libmagma.a(trace.o) has no symbols
/usr/bin/ranlib: file: ../lib/libmagma.a(zgetrf_incpiv_gpu.o) has no symbols
/usr/bin/ranlib: file: ../lib/libmagma.a(ztstrf_gpu.o) has no symbols
/usr/bin/ranlib: file: ../lib/libmagma.a(cgetrf_incpiv_gpu.o) has no symbols
/usr/bin/ranlib: file: ../lib/libmagma.a(ctstrf_gpu.o) has no symbols
/usr/bin/ranlib: file: ../lib/libmagma.a(dgetrf_incpiv_gpu.o) has no symbols
/usr/bin/ranlib: file: ../lib/libmagma.a(dtstrf_gpu.o) has no symbols
/usr/bin/ranlib: file: ../lib/libmagma.a(sgetrf_incpiv_gpu.o) has no symbols
/usr/bin/ranlib: file: ../lib/libmagma.a(ststrf_gpu.o) has no symbols
ranlib ../lib/libmagma.a
ranlib: file: ../lib/libmagma.a(magma_param.o) has no symbols
ranlib: file: ../lib/libmagma.a(magma.o) has no symbols
ranlib: file: ../lib/libmagma.a(get_nb_fermi.o) has no symbols
ranlib: file: ../lib/libmagma.a(get_nb_kepler.o) has no symbols
ranlib: file: ../lib/libmagma.a(trace.o) has no symbols
ranlib: file: ../lib/libmagma.a(zgetrf_incpiv_gpu.o) has no symbols
ranlib: file: ../lib/libmagma.a(ztstrf_gpu.o) has no symbols
ranlib: file: ../lib/libmagma.a(cgetrf_incpiv_gpu.o) has no symbols
ranlib: file: ../lib/libmagma.a(ctstrf_gpu.o) has no symbols
ranlib: file: ../lib/libmagma.a(dgetrf_incpiv_gpu.o) has no symbols
ranlib: file: ../lib/libmagma.a(dtstrf_gpu.o) has no symbols
ranlib: file: ../lib/libmagma.a(sgetrf_incpiv_gpu.o) has no symbols
ranlib: file: ../lib/libmagma.a(ststrf_gpu.o) has no symbols

I have the full output from the build here:

http://michaelgalloy.com/wp-content/uploads/2013/01/magma-build-macos64.txt

I will also make the corrections to magma_slapack.h and magma_clapack.h.

[mgates3]

Ah, nvcc compiles 32-bit by default, while gcc compiles 64-bit by default. So the magmablas lib, which contains nvcc objects, is 32-bit. The solution is to explicitly put -m64 in NVOPTS and re-compile the magmablas directory. I would put it in all the OPTS for clarity.

Code: Select all

src/magma-1.3.0-updated> otool -hv lib/libmagma.a
Archive : lib/libmagma.a
lib/libmagma.a(zpanel_to_q.o):
Mach header
      magic cputype cpusubtype  caps    filetype ncmds sizeofcmds      flags
MH_MAGIC_64  X86_64        ALL  0x00      OBJECT     3        336 SUBSECTIONS_VIA_SYMBOLS
...

src/magma-1.3.0-updated> otool -hv lib/libmagmablas.a
Archive : lib/libmagmablas.a
lib/libmagmablas.a(zauxiliary.cu_o):
Mach header
      magic cputype cpusubtype  caps    filetype ncmds sizeofcmds      flags
   MH_MAGIC    I386        ALL  0x00      OBJECT     3        772 SUBSECTIONS_VIA_SYMBOLS
...

AFTER RE-COMPILING USING -m64:
magma-1.3.0-updated> otool -hv lib/libmagmablas.a
Archive : lib/libmagmablas.a
lib/libmagmablas.a(zauxiliary.cu_o):
Mach header
      magic cputype cpusubtype  caps    filetype ncmds sizeofcmds      flags
MH_MAGIC_64  X86_64        ALL  0x00      OBJECT     3        816 SUBSECTIONS_VIA_SYMBOLS
...


The "no symbols" warnings are normal. Some files get compiled on only some architectures, elsewhere the entire contents are #ifdef'd out.

-mark

[mgalloy]

Specifying -m64 explicitly is getting me through the build, once I updated my gfortran compiler (which also fixed the problem I was having with the 32-bit build). As a note to any others trying to build on Mac OS X Mountain Lion, I had to install an updated gfortran compiler to get past an "ld: library not found for -lgcc" error when linking objects produced by gfortran, even though I had the most current version of Xcode. I used the one here:

http://web.mit.edu/mfloyd/www/computing/mac/gfortran/

Also, the 32-bit build's tests are now working!

But, of course, I need the 64-bit build and it's tests are not working. I haven't made the changes to the prototypes to match the veclib that you suggested before because I couldn't find snrm2_ that you mentioned was in magma_slapack.h/magma_clapack. I do see:

Code: Select all

include$ grep snrm2 *.h
cblas.h:float  cblas_snrm2(const int N, const float *X, const int incX);

This seems to match the veclib documentation:

Code: Select all

float cblas_snrm2 (
   const int N,
   const float *X,
   const int incX
);

at the Apple docs:

http://bit.ly/Wwgb2O

What needs to change?

Currently, I get output like:

Code: Select all

testing$ ./testing_sgetrf -c -l
MAGMA 1.3.0
device 0: GeForce GT 330M, 1100.0 MHz clock, 255.7 MB memory, capability 1.2

Usage: ./testing_sgetrf -N <m,n> -c -c2 -l
  -N  can be repeated up to 10 times. If only m is given, then m=n.
  -c  or setting $MAGMA_TESTINGS_CHECK checks result, PA - LU.
  -c2 for square matrices, solves one RHS and checks residual, Ax - b.
  -l  or setting $MAGMA_RUN_LAPACK runs LAPACK.

    M     N   CPU GFlop/s (sec)   GPU GFlop/s (sec)   |PA-LU|/(N*|A|)
=========================================================================
 1024  1024     17.17 (   0.04)      8.01 (   0.09)   -9.77e-04
 2048  2048     22.09 (   0.26)     21.12 (   0.27)   -9.01e+15
 3072  3072     24.78 (   0.78)     29.20 (   0.66)   3.26e-04
 4032  4032     26.73 (   1.63)     36.13 (   1.21)        nan
 5184  5184     25.72 (   3.61)     43.92 (   2.11)   -6.56e+34
 6016  6016     26.13 (   5.56)     47.33 (   3.07)        nan
 7040  7040     25.58 (   9.09)     46.65 (   4.99)   -0.00e+00
 8064  8064     26.04 (  13.42)     48.17 (   7.26)   -1.24e-04
 9088  9088     24.82 (  20.16)     48.51 (  10.31)   5.97e-24
10112 10112     26.49 (  26.02)     46.76 (  14.74)   -9.89e-05

[mgalloy]

Has anyone gotten a 64-bit Mac OS build working? I need to link to a 64-bit application, so the 32-bit build won't suffice.

[mgates3]

The functions that (probably) need prototypes fixed are below. However, I haven't tried fixing them or tested whether changing the return types to double fixes all problems in the 64-bit Mac OS X build.

in magma_slapack.h:
float lapackf77_slange
float lapackf77_slansy
float lapackf77_slanst
float lapackf77_slansy
float lapackf77_slamc3
float lapackf77_slapy3
float lapackf77_sqpt01

in magma_clapack.h:
float lapackf77_clange
float lapackf77_clanhe
float lapackf77_clanht
float lapackf77_clansy
float lapackf77_slamc3
float lapackf77_slapy3
float lapackf77_cqpt01

For the BLAS functions like snrm2 and sdot, we've solved the problem by using the cblas interface instead of the Fortran interface, hence those functions are no longer in the magma headers.

Let us know if changing those fixes your problems.
-mark