[Llvm-bgq-discuss] llvm-bgq

[Chris Samuel]

Hi there,

First of all great to see this happening!  We finally have hope of
building newer versions of OpenFOAM for our 4 rack system - it needs a
newer version of GCC than is available for BG/Q, but does seem to claim
to support this clang version.

One wrinkle I've found is that if I try and use the MPI wrappers to build a
simple "hello world" program they do not correctly determine the library
locations:

[samuel at avoca Test]$ bash -x mpicc -O3 -o hello-bgclang hello.c
++ readlink /bgsys/drivers/ppcfloor
+ ppcflr=V1R2M0/ppc64/
+ pamilibdir=V1R2M0/ppc64//comm/sys/lib
+ newlayout=0
+ '[' -d V1R2M0/ppc64//comm/lib ']'
+ '[' 0 = 1 ']'
+ prefix=V1R2M0/ppc64//comm/gcc
+ exec_prefix=V1R2M0/ppc64//comm/gcc
+ sysconfdir=V1R2M0/ppc64//comm/gcc/etc
+ libdir=V1R2M0/ppc64//comm/gcc/lib
+ includedir=V1R2M0/ppc64//comm/gcc/include
+ CC=/usr/local/llvm/r203443-20140309/bin/bgclang
+ '[' 0 = 1 ']'
+ CFLAGS='  -IV1R2M0/ppc64/ -IV1R2M0/ppc64//comm/sys/include -IV1R2M0/ppc64//spi/include -IV1R2M0/ppc64//spi/include/kernel/cnk'
+ CPPFLAGS=' '
+ LDFLAGS=' -LV1R2M0/ppc64//spi/lib -LV1R2M0/ppc64//comm/sys/lib -LV1R2M0/ppc64//spi/lib -LV1R2M0/ppc64//comm/sys/lib   -LV1R2M0/ppc64//spi/lib'
+ LIBS='-lopa -lmpl  -lpami -lSPI -lSPI_cnk -lrt -lpthread -lstdc++ -lpthread  '

[...much elided...]

+ bgclang -IV1R2M0/ppc64/ -IV1R2M0/ppc64//comm/sys/include -IV1R2M0/ppc64//spi/include -IV1R2M0/ppc64//spi/include/kernel/cnk -LV1R2M0/ppc64//spi/lib -LV1R2M0/ppc64//comm/sys/lib -LV1R2M0/ppc64//spi/lib -LV1R2M0/ppc64//comm/sys/lib -LV1R2M0/ppc64//spi/lib -O3 -o hello-bgclang hello.c -IV1R2M0/ppc64//comm/gcc/include -LV1R2M0/ppc64//comm/gcc/lib -Wl,-rpath -Wl,V1R2M0/ppc64//comm/sys/lib -Wl,-rpath -Wl,V1R2M0/ppc64//comm/gcc/lib -lmpich -lopa -lmpl -lpami -lSPI -lSPI_cnk -lrt -lpthread -lstdc++ -lpthread
/usr/local/llvm/r203443-20140309/bin/clang: /lib64/libz.so.1: no version information available (required by /usr/local/llvm/r203443-20140309/bin/../lib/libLLVM-3.5.0svn.so)
/usr/local/llvm/r203443-20140309/bin/clang: /lib64/libz.so.1: no version information available (required by /usr/local/llvm/r203443-20140309/bin/../lib/libLLVM-3.5.0svn.so)
/usr/local/llvm/r203443-20140309/binutils/bin/as: /lib64/libz.so.1: no version information available (required by /usr/local/llvm/r203443-20140309/binutils/bin/as)
/usr/local/llvm/r203443-20140309/binutils/bin/ld: /lib64/libz.so.1: no version information available (required by /usr/local/llvm/r203443-20140309/binutils/bin/ld)
/usr/local/llvm/r203443-20140309/binutils/bin/ld: cannot find crtbeginT.o: No such file or directory
/usr/local/llvm/r203443-20140309/binutils/bin/ld: cannot find -lmpich
/usr/local/llvm/r203443-20140309/binutils/bin/ld: cannot find -lopa
/usr/local/llvm/r203443-20140309/binutils/bin/ld: cannot find -lmpl
/usr/local/llvm/r203443-20140309/binutils/bin/ld: cannot find -lpami
/usr/local/llvm/r203443-20140309/binutils/bin/ld: cannot find -lSPI
/usr/local/llvm/r203443-20140309/binutils/bin/ld: cannot find -lSPI_cnk
/usr/local/llvm/r203443-20140309/binutils/bin/ld: cannot find -lrt
/usr/local/llvm/r203443-20140309/binutils/bin/ld: cannot find -lpthread
/usr/local/llvm/r203443-20140309/binutils/bin/ld: cannot find -lstdc++
/usr/local/llvm/r203443-20140309/binutils/bin/ld: cannot find -lpthread
/usr/local/llvm/r203443-20140309/binutils/bin/ld: cannot find -lgcc
/usr/local/llvm/r203443-20140309/binutils/bin/ld: cannot find -lgcc_eh
/usr/local/llvm/r203443-20140309/binutils/bin/ld: cannot find -ldl
/usr/local/llvm/r203443-20140309/binutils/bin/ld: cannot find -lc
/usr/local/llvm/r203443-20140309/binutils/bin/ld: cannot find -lnss_files
/usr/local/llvm/r203443-20140309/binutils/bin/ld: cannot find -lnss_dns
/usr/local/llvm/r203443-20140309/binutils/bin/ld: cannot find -lresolv
/usr/local/llvm/r203443-20140309/binutils/bin/ld: cannot find -lm
clang: error: linker command failed with exit code 1 (use -v to see invocation)
+ rc=1
+ exit 1


My guess this is a result of how the link is set:

[samuel at avoca Test]$ ls -ld /bgsys/drivers/ppcfloor
lrwxrwxrwx 1 root root 13 Jan 30 11:23 /bgsys/drivers/ppcfloor -> V1R2M0/ppc64/

The fix for the MPI/PAMI/SPI side may be as simple as testing if
the destination of the link is relative (as it is here) and if so
then prepending the path of the directory containing the link
to it.

However, if I add those by hand I am still left with this puzzling set:

/usr/local/llvm/r203443-20140309/bin/clang: /lib64/libz.so.1: no version information available (required by /usr/local/llvm/r203443-20140309/bin/../lib/libLLVM-3.5.0svn.so)
/usr/local/llvm/r203443-20140309/bin/clang: /lib64/libz.so.1: no version information available (required by /usr/local/llvm/r203443-20140309/bin/../lib/libLLVM-3.5.0svn.so)
/usr/local/llvm/r203443-20140309/binutils/bin/as: /lib64/libz.so.1: no version information available (required by /usr/local/llvm/r203443-20140309/binutils/bin/as)
/usr/local/llvm/r203443-20140309/binutils/bin/ld: /lib64/libz.so.1: no version information available (required by /usr/local/llvm/r203443-20140309/binutils/bin/ld)
/usr/local/llvm/r203443-20140309/binutils/bin/ld: cannot find crtbeginT.o: No such file or directory
/usr/local/llvm/r203443-20140309/binutils/bin/ld: cannot find -lrt
/usr/local/llvm/r203443-20140309/binutils/bin/ld: cannot find -lpthread
/usr/local/llvm/r203443-20140309/binutils/bin/ld: cannot find -lstdc++
/usr/local/llvm/r203443-20140309/binutils/bin/ld: cannot find -lpthread
/usr/local/llvm/r203443-20140309/binutils/bin/ld: cannot find -lgcc
/usr/local/llvm/r203443-20140309/binutils/bin/ld: cannot find -lgcc_eh
/usr/local/llvm/r203443-20140309/binutils/bin/ld: cannot find -ldl
/usr/local/llvm/r203443-20140309/binutils/bin/ld: cannot find -lc
/usr/local/llvm/r203443-20140309/binutils/bin/ld: cannot find -lnss_files
/usr/local/llvm/r203443-20140309/binutils/bin/ld: cannot find -lnss_dns
/usr/local/llvm/r203443-20140309/binutils/bin/ld: cannot find -lresolv
/usr/local/llvm/r203443-20140309/binutils/bin/ld: cannot find -lm
clang: error: linker command failed with exit code 1 (use -v to see invocation)

I can fix most of those by hand by adding the library paths by hand with:

-L/bgsys/drivers/toolchain/V1R2M0-efix23/gnu-linux/lib/gcc/powerpc64-bgq-linux/4.4.6 \
-L/bgsys/drivers/toolchain/V1R2M0-efix23/gnu-linux/powerpc64-bgq-linux/lib

but it needs some way to automate finding them I'd say.

That still leaves me with the ones I cannot solve:

/usr/local/llvm/r203443-20140309/bin/clang: /lib64/libz.so.1: no version information available (required by /usr/local/llvm/r203443-20140309/bin/../lib/libLLVM-3.5.0svn.so)
/usr/local/llvm/r203443-20140309/bin/clang: /lib64/libz.so.1: no version information available (required by /usr/local/llvm/r203443-20140309/bin/../lib/libLLVM-3.5.0svn.so)
/usr/local/llvm/r203443-20140309/binutils/bin/as: /lib64/libz.so.1: no version information available (required by /usr/local/llvm/r203443-20140309/binutils/bin/as)
/usr/local/llvm/r203443-20140309/binutils/bin/ld: /lib64/libz.so.1: no version information available (required by /usr/local/llvm/r203443-20140309/binutils/bin/ld)
/usr/local/llvm/r203443-20140309/binutils/bin/ld: cannot find crtbeginT.o: No such file or directory


Thoughts?

cheers!
Chris
-- 
 Christopher Samuel        Senior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545
 http://www.vlsci.org.au/      http://twitter.com/vlsci