[Llvm-bgq-discuss] Details behind MPI wrapper for bgclang++

[Jeff Hammond]

In my experience, it is an artifact of reply-to-all in the IBM email
client that adds bounces.  That is the only time I see it.

Jeff

On Fri, Mar 1, 2013 at 11:03 AM, Hal Finkel <hfinkel at anl.gov> wrote:
> Michael,
>
> Did you get a duplicate of this message? It looks like llvm-bgq-discuss-bounces somehow also ended up in the CC list, and I wonder if that was causing the problem.
>
>  -Hal
>
> ----- Original Message -----
>> From: "Michael Blocksome" <blocksom at us.ibm.com>
>> To: "Jack Poulson" <jack.poulson at gmail.com>
>> Cc: "Hal Finkel" <hfinkel at anl.gov>, llvm-bgq-discuss at lists.alcf.anl.gov, llvm-bgq-discuss-bounces at lists.alcf.anl.gov
>> Sent: Friday, March 1, 2013 8:30:56 AM
>> Subject: Re: [Llvm-bgq-discuss] Details behind MPI wrapper for bgclang++
>>
>> All,
>>
>> I seem to be getting duplicates of each email in this thread .. any
>> idea what's going on with the mailing list?
>>
>> Michael Blocksome
>> Blue Gene Messaging
>> blocksom at us.ibm.com
>>
>>
>>
>>
>> From: Jack Poulson <jack.poulson at gmail.com>
>> To: Hal Finkel <hfinkel at anl.gov>,
>> Cc: llvm-bgq-discuss at lists.alcf.anl.gov
>> Date: 02/28/2013 09:44 PM
>> Subject: Re: [Llvm-bgq-discuss] Details behind MPI wrapper for
>> bgclang++
>> Sent by: llvm-bgq-discuss-bounces at lists.alcf.anl.gov
>>
>>
>>
>>
>> On Thu, Feb 28, 2013 at 7:36 PM, Hal Finkel < hfinkel at anl.gov >
>> wrote:
>> ----- Original Message -----
>> > From: "Jack Poulson" < jack.poulson at gmail.com >
>> > To: "Hal Finkel" < hfinkel at anl.gov >
>> > Cc: "Jeff Hammond" < jhammond at alcf.anl.gov >,
>> > llvm-bgq-discuss at lists.alcf.anl.gov
>> > Sent: Thursday, February 28, 2013 7:40:19 PM
>> > Subject: Re: [Llvm-bgq-discuss] Details behind MPI wrapper for
>> > bgclang++
>> >
>> > On Thu, Feb 28, 2013 at 5:24 PM, Hal Finkel < hfinkel at anl.gov >
>> > wrote:
>> >
>> >
>> >
>> >
>> > ----- Original Message -----
>> > > From: "Jack Poulson" < jack.poulson at gmail.com >
>> > > To: "Hal Finkel" < hfinkel at anl.gov >
>> > > Cc: "Jeff Hammond" < jhammond at alcf.anl.gov >,
>> > > llvm-bgq-discuss at lists.alcf.anl.gov
>> >
>> > > Sent: Thursday, February 28, 2013 6:51:08 PM
>> > > Subject: Re: [Llvm-bgq-discuss] Details behind MPI wrapper for
>> > > bgclang++
>> > >
>> >
>> > > Thanks Hal, that helps clear things up a bit. I guess I should be
>> > > a
>> > > little more clear about what exactly I'm doing. Since CMake's
>> > > FindMPI module seems to attempt to link to MPICH incorrectly if I
>> > > directly specify the mpic++11, I have to instead manually specify
>> > > its behaviour (and thankfully Jeff took care of this a month or
>> > > two
>> > > ago). One of my link commands generated by CMake looks like this:
>> > >
>> > >
>> > >
>> > > /home/projects/llvm/bin/bgclang++ -Wall -std=c++11 -O3
>> > > -stdlib=libc++
>> > > -L/bgsys/drivers/ppcfloor/comm/gcc/lib
>> > > -L/bgsys/drivers/ppcfloor/comm/sys/lib
>> > > -L/bgsys/drivers/ppcfloor/spi/lib
>> > > CMakeFiles/HypRadon-2d.dir/test/transform/HypRadon-2d.cpp.o -o
>> > > bin/transform/HypRadon-2d -rdynamic libcmake-dummy-lib.a
>> > > -L/soft/libraries/alcf/current/gcc/LAPACK/lib -llapack
>> > > -L/soft/libraries/essl/current/essl/5.1/lib64 -lesslbg
>> > > -L/soft/compilers/ibmcmp-nov2012/xlf/bg/14.1/bglib64 -lxlf90_r
>> > > -L/soft/compilers/ibmcmp-nov2012/xlsmp/bg/3.1/bglib64 -lxlomp_ser
>> > > -L/soft/compilers/ibmcmp-nov2012/xlmass/bg/7.3/bglib64 -lmassv
>> > > -lmass -lxlopt -lxlfmath -lxl -lgfortran -lm -lpthread -ldl
>> > > -Wl,--allow-multiple-definition -lcxxmpich -lmpich -lopa -lmpl
>> > > -ldl
>> > > -lpami -lSPI -lSPI_cnk -lpthread -lrt -lstdc++
>> > >
>> > >
>> > > Upon typing this, I noticed that both libc++ and libstdc++ are
>> > > both
>> > > being used (which I assume is bad).
>> >
>> > Actually, this is okay (although this is not obvious). When you
>> > specify -stdlib=libc++ the clang driver automatically rewrites
>> > -lstdc++ to some system-specific set of libraries necessary to link
>> > with libc++. In our case, this is -lc++ (and -lrt -lpthread
>> > -lstdc++
>> > when statically linking). As a result, using -lstdc++ is fine here.
>> > You can verify this by passing -v and examining the linking command
>> > line.
>> >
>> > I should also note that we have a *special* libc++ install which is
>> > partially based on libstdc++ so that we can link statically with
>> > PAMI without symbol definition conflicts. Everything *should* work,
>> > but I've only done some limited testing.
>> >
>> > So using the command-line above succeeds but produces an executable
>> > that crashes?
>> >
>> >
>> >
>> >
>> >
>> >
>> > Hi Hal,
>> >
>> >
>> > Thanks, that makes a lot of sense! The executable Backproj-2d in
>> > /home/poulson/dist-butterfly/build/clang/bin/transform produced
>> > core.{0,6,8,10,11}. Please let me know if you do not have access
>> > for
>> > some reason, and thank you again for your help.
>>
>> Interesting; so the backtrace is messed up, but the problem seems
>> legitimate. The code is ending up in steady_clock::now(), which uses
>> clock_gettime(CLOCK_MONOTONIC, ...), which is not supported on the
>> BG/Q compute nodes. I'll need to change it to use CLOCK_REALTIME
>> instead; I'll let you know when to retry.
>>
>> Thanks again,
>> Hal
>>
>>
>> Thanks Hal! I was thrown off by the fact that the trace contained
>> MPI_Gather and the program had failed before it should have reached
>> any MPI_Gather calls. I very much appreciate the help.
>>
>> Jack _______________________________________________
>> llvm-bgq-discuss mailing list
>> llvm-bgq-discuss at lists.alcf.anl.gov
>> https://lists.alcf.anl.gov/mailman/listinfo/llvm-bgq-discuss
>>
>>
> _______________________________________________
> llvm-bgq-discuss mailing list
> llvm-bgq-discuss at lists.alcf.anl.gov
> https://lists.alcf.anl.gov/mailman/listinfo/llvm-bgq-discuss



-- 
Jeff Hammond
Argonne Leadership Computing Facility
University of Chicago Computation Institute
jhammond at alcf.anl.gov / (630) 252-5381
http://www.linkedin.com/in/jeffhammond
https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond