[SOLVED] compiling using pathscale on opterons with atlas

[wazoo42]

Hi,
The end goal of all of this is to be able to compile a parallel version of molpro. I have already been able to do this with the pathscale compiler 2.3 and prebuild atlas libraries. I am hoping if I can compile atlas (already done, worked fine) and lapack using the pathscale compiler I will get bettter overall performance. As of right now I can't get the molpro compile to accept the lapack library b/c it has a bunch of undefined reference errors which I think are coming from its incomplete lapack library.

Now to the problem at hand. I am unsure which blas library I should link to in the lapack make.inc file. I have tried libptf77blas.a and libatlas.a from the precompiled atlas 3.6.0 and from the 3.6.0 I compiled with pathscale 2.2.1 and 3.6.0 I compiled with 2.3. The libatlas.a give me errors of the sort:

libptf77blas.a errors
-------------------------------------
/ab/atlas_dirs/3.6.0_test/libptf77blas.a(ATL_F77wrap_sptrotg.o): In function `at
l_f77wrap_srotg__':
: undefined reference to `ATL_srotg'
/ab/atlas_dirs/3.6.0_test/libptf77blas.a(ATL_F77wrap_sptscal.o): In function `at
l_f77wrap_sscal__':
: undefined reference to `ATL_sscal'
/ab/atlas_dirs/3.6.0_test/libptf77blas.a(ATL_F77wrap_sptnrm2.o): In function `at
l_f77wrap_snrm2__':
: undefined reference to `ATL_snrm2'
/ab/atlas_dirs/3.6.0_test/libptf77blas.a(ATL_F77wrap_sptasum.o): In function `at
l_f77wrap_sasum__':


libatlas.a errors
-------------------------------------
../../lapack_AMD64.a(slauu2.o): In function `slauu2_':
/comp/ab/LAPACK/SRC/slauu2.f:112: undefined reference to `sscal_'
/comp/ab/LAPACK/SRC/slauu2.f:108: undefined reference to `sdot_'
/comp/ab/LAPACK/SRC/slauu2.f:109: undefined reference to `sgemv_'
/comp/ab/LAPACK/SRC/slauu2.f:127: undefined reference to `sscal_'
/comp/ab/LAPACK/SRC/slauu2.f:123: undefined reference to `sdot_'
/comp/ab/LAPACK/SRC/slauu2.f:124: undefined reference to `sgemv_'

There are many more, this was just the last one it spit out. I believe I have included my make.inc file below. Notably, I changed the NOOPT variable to nothing b/c pathscale didn't recognize either flag.

####################################################################
# LAPACK make include file. #
# LAPACK, Version 3.0 #
# June 30, 1999 #
####################################################################
#
SHELL = /bin/sh
#
# The machine (platform) identifier to append to the library names
#
PLAT = _AMD64
#
# Modify the FORTRAN and OPTS definitions to refer to the
# compiler and desired compiler options for your machine. NOOPT
# refers to the compiler options desired when NO OPTIMIZATION is
# selected. Define LOADER and LOADOPTS to refer to the loader and
# desired load options for your machine.
#
FORTRAN = /opt/pathscale/bin/pathf90
#OPTS = -O4 -u -f -mt
#OPTS = -u -f -dalign -native -xO5 -xarch=v8plusa
#OPTS = -u -f -dalign -native -xO5 -xarch=v8plusa
OPTS = -i8 -O3 -OPT:Ofast
DRVOPTS = $(OPTS)
#NOOPT = -u -f
NOOPT = -i8
#NOOPT = -u -f -mt
LOADER = /opt/pathscale/bin/pathf90
#LOADOPTS = -mt
#LOADOPTS = -f -dalign -native -xO5 -xarch=v8plusa
LOADOPTS = -i8 -O3 -OPT:Ofast
#
# The archiver and the flag(s) to use when building archive (library)
# If you system has no ranlib, set RANLIB = echo.
#
ARCH = ar
ARCHFLAGS= cr
RANLIB = echo
#
# The location of the libraries to which you will link. (The
# machine-specific, optimized BLAS library should be used whenever
# possible.)
#
#BLASLIB = ../../blas$(PLAT).a
#BLASLIB = -xlic_lib=sunperf_mt
#BLASLIB = -xlic_lib=sunperf
BLASLIB = /ab/atlas_dirs/3.6.0_test/libatlas.a
LAPACKLIB = lapack$(PLAT).a
TMGLIB = tmglib$(PLAT).a
EIGSRCLIB = eigsrc$(PLAT).a
LINSRCLIB = linsrc$(PLAT).a


Any help would be greatly appreciated.

[wazoo42]

This is more of an addendum I think, than an update or a change b/c I think I will run into this problem with atlas. I tried linking to the GOTO blas I have (I put blaslib=/ab/libgoto.so) and it gave me an error saying it couldn't find pthread_create and pthread_join. How do I tell lapack installation to use /usr/lib64/libpthread.a?

[Julien Langou]

Hello,

your BLASLIB variable is wronlgy configure in both cases (ATLAS BLAS or Goto BLAS).

For ATLAS BLAS without thread:
BLASLIB=-L${ATLASBLASLIBDIR} -lf77blas -latlas

For ATLAS BLAS with threads:
BLASLIB=-L${ATLASBLASLIBDIR} -lptf77blas -latlas -lpthread

For Goto BLAS:
BLASLIB=-L${GOTOBLASLIBDIR} -lgoto -lpthread
(Goto BLAS library is a shared library so do not forget to set your LD_LIBRARY_PATH environment variable so that it contains the directory of the Goto BLAS library)

Should work better like this.

Julien

[wazoo42]

Thanks for the help. I'm still running into problems with atlas of the form:

/ab/atlas_dirs/3.6.0_test/libptf77blas.a(strmm.o): In function `strmm_':
/comp/ab/atlas_dirs/3.6.0_source/interfaces/blas/F77/src/strmm.f:277: undefined reference to `atl_f77wrap_strmm__'
/ab/atlas_dirs/3.6.0_test/libptf77blas.a(strsm.o): In function `strsm_':
/comp/ab/atlas_dirs/3.6.0_source/interfaces/blas/F77/src/strsm.f:279: undefined reference to `atl_f77wrap_strsm__'

Any ideas?

[Julien Langou]

I am not an ATLAS expert but I would say there is a problem
in your libptf77blas.a library. The reference from strmm_ in libptf77blas.a to atl_f77wrap_strmm__ should be in this same library. (a priori).

Is the nonthreaded version of ATLAS ok?
( BLASLIB=-L${ATLASBLASLIBDIR} -lf77blas -latlas )

Julien

[wazoo42]

Thanks for the help, the GOTO blas readily worked (at least it compiled, it is still testing after many hours).

I tried the single threaded version of atlas and it has the same problem. I'm going to post on their help request forum as well. Perhaps it is an issue of compiler flags. Unfortunately nobody at Pathscale has compiled atlas.

[wazoo42]

Thanks for the help again. I got it working after a few changes. I set up the linking as you suggested, and when building atlas I had to use F2CDEFS = -DAdd__ -DStringSunStyle and I used the GOTO blas for BLASLIB. I included my lapack make.inc and my ATLAS Make.arch for posterity.

####################################################################
# LAPACK make include file. #
# LAPACK, Version 3.0 #
# June 30, 1999 #
####################################################################
#
SHELL = /bin/sh
#
# The machine (platform) identifier to append to the library names
#
PLAT = _LINUX
#
# Modify the FORTRAN and OPTS definitions to refer to the
# compiler and desired compiler options for your machine. NOOPT
# refers to the compiler options desired when NO OPTIMIZATION is
# selected. Define LOADER and LOADOPTS to refer to the loader and
# desired load options for your machine.
#
FORTRAN = /opt/pathscale/bin/pathf90
#OPTS = -O4 -u -f -mt
#OPTS = -u -f -dalign -native -xO5 -xarch=v8plusa
#OPTS = -u -f -dalign -native -xO5 -xarch=v8plusa
OPTS = -i8 -fno-second-underscore -woff -align64 -O3 -OPT:Ofast
DRVOPTS = $(OPTS)
#NOOPT = -u -f
NOOPT = -i8 -fno-second-underscore -woff -align64
#NOOPT = -u -f -mt
LOADER = /opt/pathscale/bin/pathf90
#LOADOPTS = -mt
#LOADOPTS = -f -dalign -native -xO5 -xarch=v8plusa
LOADOPTS = -i8 -fno-second-underscore -woff -align64 -O3 -OPT:Ofast
#
# The archiver and the flag(s) to use when building archive (library)
# If you system has no ranlib, set RANLIB = echo.
#
ARCH = ar
ARCHFLAGS= cr
RANLIB = echo
#
# The location of the libraries to which you will link. (The
# machine-specific, optimized BLAS library should be used whenever
# possible.)
#
#BLASLIB = ../../blas$(PLAT).a
#BLASLIB = -xlic_lib=sunperf_mt
#BLASLIB = -xlic_lib=sunperf
BLASLIB = -L/ab/atlas_dirs/3.6.0_source/lib/Linux_HAMMER64SSE2_2 -lptf77blas -latlas -lpthread
LAPACKLIB = lapack$(PLAT).a
TMGLIB = tmglib$(PLAT).a
EIGSRCLIB = eigsrc$(PLAT).a
LINSRCLIB = linsrc$(PLAT).a
-----------------------------------------------------------------------------------------------------------------


# -----------------------------
# Make.ARCH for ATLAS3.6.0
# -----------------------------

# ----------------------------------
# Make sure we get the correct shell
# ----------------------------------
SHELL = /bin/sh

# -------------------------------------------------
# Name indicating the platform to configure BLAS to
# -------------------------------------------------
ARCH = Linux_HAMMER64SSE2_2

# -------------------
# Various directories
# -------------------
TOPdir = /ab/atlas_dirs/3.6.0_source
INCdir = $(TOPdir)/include/$(ARCH)
SYSdir = $(TOPdir)/tune/sysinfo/$(ARCH)
GMMdir = $(TOPdir)/src/blas/gemm/$(ARCH)
UMMdir = $(GMMdir)
GMVdir = $(TOPdir)/src/blas/gemv/$(ARCH)
GR1dir = $(TOPdir)/src/blas/ger/$(ARCH)
L1Bdir = $(TOPdir)/src/blas/level1/$(ARCH)
L2Bdir = $(TOPdir)/src/blas/level2/$(ARCH)
L3Bdir = $(TOPdir)/src/blas/level3/$(ARCH)
TSTdir = $(TOPdir)/src/testing/$(ARCH)
AUXdir = $(TOPdir)/src/auxil/$(ARCH)
CBLdir = $(TOPdir)/interfaces/blas/C/src/$(ARCH)
FBLdir = $(TOPdir)/interfaces/blas/F77/src/$(ARCH)
BINdir = $(TOPdir)/bin/$(ARCH)
LIBdir = $(TOPdir)/lib/$(ARCH)
PTSdir = $(TOPdir)/src/pthreads
MMTdir = $(TOPdir)/tune/blas/gemm/$(ARCH)
MVTdir = $(TOPdir)/tune/blas/gemv/$(ARCH)
R1Tdir = $(TOPdir)/tune/blas/ger/$(ARCH)
L1Tdir = $(TOPdir)/tune/blas/level1/$(ARCH)
L3Tdir = $(TOPdir)/tune/blas/level3/$(ARCH)

# ---------------------------------------------------------------------
# Name and location of scripts for running executables during tuning
# ---------------------------------------------------------------------
ATLRUN = $(BINdir)/ATLrun.sh
ATLFWAIT = $(BINdir)/xatlas_waitfile

# ---------------------
# Libraries to be built
# ---------------------
ATLASlib = $(LIBdir)/libatlas.a
CBLASlib = $(LIBdir)/libcblas.a
F77BLASlib = $(LIBdir)/libf77blas.a
PTCBLASlib = $(LIBdir)/libptcblas.a
PTF77BLASlib = $(LIBdir)/libptf77blas.a
LAPACKlib = $(LIBdir)/liblapack.a

TESTlib = $(LIBdir)/libtstatlas.a
# -------------------------------------------
# Upper bound on largest cache size, in bytes
# -------------------------------------------
L2SIZE = -DL2SIZE=1048576

# ---------------------------------------
# Command setting up correct include path
# ---------------------------------------
INCLUDES = -I$(TOPdir)/include -I$(TOPdir)/include/$(ARCH) \
-I$(TOPdir)/include/contrib

# -------------------------------------------
# Defines for setting up F77/C interoperation
# -------------------------------------------
F2CDEFS = -DAdd__ -DStringSunStyle

# --------------------------------------
# Special defines for user-supplied GEMM
# --------------------------------------
UMMDEFS =

# ------------------------------
# Architecture identifying flags
# ------------------------------
ARCHDEFS = -DATL_OS_Linux -DATL_ARCH_HAMMER64 -DATL_SSE2 -DATL_SSE1 -DATL_GAS_x8664 -m64

# NM is the flag required to name a compiled object/executable
# OJ is the flag required to compile to object rather than executable
# These flags are used by all compilers.
# -------------------------------------------------------------------
NM = -o
OJ = -c


# ---------------------------------------------------------------------------
# Fortran 77 compiler and the flags to use. Presently, ATLAS does not itself
# use any Fortran 77, but vendor BLAS are typically written for Fortran, so
# any links that include non-ATLAS BLAS will use FLINKER instead of CLINKER
# ---------------------------------------------------------------------------
F77 = /opt/pathscale/bin/pathf90
F77FLAGS = -O3 -OPT:Ofast
FLINKER = $(F77)
FLINKFLAGS = $(F77FLAGS)
FCLINKFLAGS = $(FLINKFLAGS)


# ---------------------------------------------------------------------------
# Various C compilers, and the linker to be used when we are not linking in
# non-ATLAS BLAS (which usually necessitate using the Fortran linker).
# The C compilers recognized by ATLAS are:
# CC : Compiler to use to compile regular, non-generated code
# MCC : Compiler to use to compile generated, highly-optimized code
# XCC : Compiler to be used on the compile engine of a cross-compiler
# These will typically all be the same. An example of where this is not
# the case would be DEC ALPHA 21164, where you want to use gcc for MCC,
# because DEC's cc does not allow the programmer access to all 32 floating
# point registers. However, on normal C code, DEC's cc produces much faster
# code than gcc, so you CC set to cc. Of course, any system where you are
# cross-compiling, you will need to set XCC differently than CC & MCC.
# ---------------------------------------------------------------------------
CDEFS = $(L2SIZE) $(INCLUDES) $(F2CDEFS) $(ARCHDEFS) -DATL_NCPU=2

GOODGCC = gcc
CC = /opt/pathscale/bin/pathcc
CCFLAG0 = -O3 -OPT:Ofast
CCFLAGS = $(CDEFS) $(CCFLAG0)
MCC = /opt/pathscale/bin/pathcc
MMFLAGS = -O3 -OPT:Ofast
XCC = /usr/bin/gcc
XCCFLAGS = $(CDEFS) -fomit-frame-pointer -O -mfpmath=387 -m64
CLINKER = $(CC)
CLINKFLAGS = $(CCFLAGS)
BC = $(CC)
BCFLAGS = $(CCFLAGS)
ARCHIVER = ar
ARFLAGS = r
RANLIB = echo

# -------------------------------------
# tar, gzip, gunzip, and parallel make
# -------------------------------------
TAR = /bin/tar
GZIP = /usr/bin/gzip
GUNZIP = /usr/bin/gunzip
PMAKE = $(MAKE) -j 2

# ------------------------------------
# Reference and system libraries
# ------------------------------------
BLASlib = -L /usr/lib64 -lgoto
FBLASlib =
FLAPACKlib =
LIBS = /usr/lib64/libpthread.so -lm

# ----------------------------------------------------------
# ATLAS install resources (include arch default directories)
# ----------------------------------------------------------
ARCHDEF =
MMDEF =
INSTFLAGS =

# ---------------------------------------
# Generic targets needed by all makefiles
# ---------------------------------------
waitfile: