[Llvm-bgq-discuss] Details behind MPI wrapper for bgclang++

Hal Finkel hfinkel at anl.gov
Thu Feb 28 19:24:58 CST 2013 

----- Original Message -----
> From: "Jack Poulson" <jack.poulson at gmail.com>
> To: "Hal Finkel" <hfinkel at anl.gov>
> Cc: "Jeff Hammond" <jhammond at alcf.anl.gov>, llvm-bgq-discuss at lists.alcf.anl.gov
> Sent: Thursday, February 28, 2013 6:51:08 PM
> Subject: Re: [Llvm-bgq-discuss] Details behind MPI wrapper for bgclang++
> 
> Thanks Hal, that helps clear things up a bit. I guess I should be a
> little more clear about what exactly I'm doing. Since CMake's
> FindMPI module seems to attempt to link to MPICH incorrectly if I
> directly specify the mpic++11, I have to instead manually specify
> its behaviour (and thankfully Jeff took care of this a month or two
> ago). One of my link commands generated by CMake looks like this:
> 
> 
> 
> /home/projects/llvm/bin/bgclang++ -Wall -std=c++11 -O3 -stdlib=libc++
> -L/bgsys/drivers/ppcfloor/comm/gcc/lib
> -L/bgsys/drivers/ppcfloor/comm/sys/lib
> -L/bgsys/drivers/ppcfloor/spi/lib
> CMakeFiles/HypRadon-2d.dir/test/transform/HypRadon-2d.cpp.o -o
> bin/transform/HypRadon-2d -rdynamic libcmake-dummy-lib.a
> -L/soft/libraries/alcf/current/gcc/LAPACK/lib -llapack
> -L/soft/libraries/essl/current/essl/5.1/lib64 -lesslbg
> -L/soft/compilers/ibmcmp-nov2012/xlf/bg/14.1/bglib64 -lxlf90_r
> -L/soft/compilers/ibmcmp-nov2012/xlsmp/bg/3.1/bglib64 -lxlomp_ser
> -L/soft/compilers/ibmcmp-nov2012/xlmass/bg/7.3/bglib64 -lmassv
> -lmass -lxlopt -lxlfmath -lxl -lgfortran -lm -lpthread -ldl
> -Wl,--allow-multiple-definition -lcxxmpich -lmpich -lopa -lmpl -ldl
> -lpami -lSPI -lSPI_cnk -lpthread -lrt -lstdc++
> 
> 
> Upon typing this, I noticed that both libc++ and libstdc++ are both
> being used (which I assume is bad).

Actually, this is okay (although this is not obvious). When you specify -stdlib=libc++ the clang driver automatically rewrites -lstdc++ to some system-specific set of libraries necessary to link with libc++. In our case, this is -lc++ (and -lrt -lpthread -lstdc++ when statically linking). As a result, using -lstdc++ is fine here. You can verify this by passing -v and examining the linking command line.

I should also note that we have a *special* libc++ install which is partially based on libstdc++ so that we can link statically with PAMI without symbol definition conflicts. Everything *should* work, but I've only done some limited testing.

So using the command-line above succeeds but produces an executable that crashes?

Thanks again,
Hal

> However, I believe that
> libstdc++ is depended upon by PAMI. I tried changing my build to
> replace the libstdc++ library with
> /home/projects/llvm/libc++/lib/libc++.so and received a long list of
> undefined references in PAMI to various entities in the std
> namespace. For example,
> 
> 
> /bgsys/drivers/ppcfloor/comm/sys/lib/libpami.a(component-pami--COLLECTIVESGEOM.cnk.o):(.toc+0x888):
> undefined reference to `std::ios_base::Init::~Init()'
> /bgsys/drivers/ppcfloor/comm/sys/lib/libpami.a(component-pami--PROGRESS_EXTENSION.cnk.o):
> In function `__static_initialization_and_destruction_0':
> /bgsys/drivers/toolchain/V1R2M0/gnu-linux/lib/gcc/powerpc64-bgq-linux/4.4.6/../../../../powerpc64-bgq-linux/include/c++/4.4.6/iostream:72:
> undefined reference to `std::ios_base::Init::Init()'
> /bgsys/drivers/ppcfloor/comm/sys/lib/libpami.a(component-pami--PROGRESS_EXTENSION.cnk.o):(.toc+0x38):
> undefined reference to `std::ios_base::Init::~Init()'
> 
> 
> Any ideas?
> 
> 
> 
> Jack
> 
> On Thu, Feb 28, 2013 at 4:36 PM, Hal Finkel < hfinkel at anl.gov >
> wrote:
> 
> 
> 
> 
> 
> ----- Original Message -----
> > From: "Jack Poulson" < jack.poulson at gmail.com >
> > To: "Hal Finkel" < hfinkel at anl.gov >
> > Cc: "Jeff Hammond" < jhammond at alcf.anl.gov >,
> > llvm-bgq-discuss at lists.alcf.anl.gov
> > Sent: Thursday, February 28, 2013 6:12:21 PM
> > Subject: Re: [Llvm-bgq-discuss] Details behind MPI wrapper for
> > bgclang++
> > 
> 
> > Hi Hal,
> > 
> > 
> > As far as I can tell, there are only two distinct wrappers in the
> > /home/projects/llvm/mpi/clang/bin folder, mpicc and mpicxx, and all
> > of the others are simply symlinks to them. Is it possible that I am
> > misunderstanding?
> 
> You're correct, but the wrapper scripts alter their behavior slightly
> depending on the invoking name. Specifically, if you use mpic++ then
> you get the GNU C++03 mode by default with libstdc++ as the primary
> C++ STL implementation (and you still get libstdc++ in C++11 mode if
> you specify -std=c++11, etc.). If you use the mpic++11 wrapper, then
> you get libc++ as the primary C++ STL implementation (and the GNU
> C++11 mode by default). The libc++ implementation, being much more
> recent than the system libstdc++, provides significantly better
> C++11 support.
> 
> Can you give me access to the directory with the core files so that I
> can take a look?
> 
> Thanks again,
> Hal
> 
> 
> 
> > 
> > 
> > I am using the mpicxx wrapper.
> > 
> > 
> > Jack
> > 
> > 
> > On Thu, Feb 28, 2013 at 4:08 PM, Hal Finkel < hfinkel at anl.gov >
> > wrote:
> > 
> > 
> > Jack,
> > 
> > What Jeff said is correct. Jeff, I did not specifically update the
> > wrapper scripts after the V1R2 transition. Is there anything that I
> > need to do for that?
> > 
> > Also, Jack, are you using the mpic++ (with -std=c++11) wrapper or
> > the
> > mpic++11 wrapper?
> > 
> > -Hal
> > 
> > 
> > 
> > ----- Original Message -----
> > > From: "Jeff Hammond" < jhammond at alcf.anl.gov >
> > > To: "Jack Poulson" < jack.poulson at gmail.com >
> > > Cc: llvm-bgq-discuss at lists.alcf.anl.gov
> > > Sent: Thursday, February 28, 2013 5:59:08 PM
> > > Subject: Re: [Llvm-bgq-discuss] Details behind MPI wrapper for
> > > bgclang++
> > > 
> > > IBM does not provide MPI builds as part of the driver because
> > > they
> > > do
> > > not - and are not required to - provide LLVM builds of MPI.
> > > 
> > > ALCF can provide /home/projects/pami/llvm/mpi.... if you want an
> > > MPI
> > > build using LLVM. I will do this next week during MiraCon as part
> > > of
> > > building MPICH2 1.5 + IBM patches from source.
> > > 
> > > Best,
> > > 
> > > Jeff
> > > 
> > > On Thu, Feb 28, 2013 at 6:56 PM, Jack Poulson
> > > < jack.poulson at gmail.com > wrote:
> > > > Hello All,
> > > > 
> > > > I am trying to run a C++11 code on BGQ and have been running
> > > > into
> > > > strange
> > > > sporadic segfaults with non-helpful stack traces. I started
> > > > looking
> > > > through
> > > > the details of the driver
> > > > /home/projects/llvm/mpi/clang/bin/mpicxx and was
> > > > surprised by the 31st line:
> > > > prefix=/bgsys/drivers/ppcfloor/comm/gcc
> > > > When I look in the /bgsys/drivers/ppcfloor/comm folder, I
> > > > notice
> > > > that there
> > > > are directories for gcc and xl, but none for Clang. Is there
> > > > some
> > > > clear
> > > > explanation for this?
> > > > 
> > > > Thanks!
> > > > Jack
> > > > 
> > > > _______________________________________________
> > > > llvm-bgq-discuss mailing list
> > > > llvm-bgq-discuss at lists.alcf.anl.gov
> > > > https://lists.alcf.anl.gov/mailman/listinfo/llvm-bgq-discuss
> > > > 
> > > 
> > > 
> > > 
> > > --
> > > Jeff Hammond
> > > Argonne Leadership Computing Facility
> > > University of Chicago Computation Institute
> > > jhammond at alcf.anl.gov / (630) 252-5381
> > > http://www.linkedin.com/in/jeffhammond
> > > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond
> > > _______________________________________________
> > > llvm-bgq-discuss mailing list
> > > llvm-bgq-discuss at lists.alcf.anl.gov
> > > https://lists.alcf.anl.gov/mailman/listinfo/llvm-bgq-discuss
> > > 
> > 
> > 
> 
>

More information about the llvm-bgq-discuss mailing list