[Llvm-bgq-discuss] clang/bgclang

Tue May 6 12:43:22 CDT 2014

Tom,

Just to be clear, John is running *on* his IONs, not on the compute nodes. Regardless, I do agree that LD_DEBUG is a good suggestion.

Thanks again,
Hal

----- Original Message -----
> From: "Thomas Gooding" <tgooding at us.ibm.com>
> To: "John A. Biddiscombe" <biddisco at cscs.ch>
> Cc: llvm-bgq-discuss at lists.alcf.anl.gov
> Sent: Tuesday, May 6, 2014 12:36:36 PM
> Subject: Re: [Llvm-bgq-discuss] clang/bgclang
> 
> 
> 
> 
> Sounds like some debug info would help isolate the problem library,
> i'd suggest enabling the dynamic loader's trace facility. Set this
> environment variable:
> LD_DEBUG=files,libs
> 
> or
> 
> LD_DEBUG=all
> 
> You could also use runjob's "--strace 0" option (or slurm/cobalt
> equivalent) to get a system call trace, but that might be slightly
> too low-level detail.
> 
> Hope that helps,
> Tom
> 
> Tom Gooding
> Senior Engineer / Blue Gene SW Lead / CAPI
> tgooding at us.ibm.com 507-253-0747
> 
> 
> llvm-bgq-discuss-bounces at lists.alcf.anl.gov wrote on 05/06/2014
> 11:11:37 AM:
> 
> > From: "Biddiscombe, John A." <biddisco at cscs.ch>
> > To: "llvm-bgq-discuss at lists.alcf.anl.gov" <llvm-bgq-
> > discuss at lists.alcf.anl.gov>
> > Date: 05/06/2014 11:11 AM
> > Subject: [Llvm-bgq-discuss] clang/bgclang
> > Sent by: llvm-bgq-discuss-bounces at lists.alcf.anl.gov
> > 
> > Hal,
> > 
> > In my ongoing struggle to succeed despite my repeated failures, I
> > have successfully compiled boost/hpx/otherstuff with just clang++
> > and linked to libc++ from the c++11 toolchain. However, when I
> > startup programs compiled as part of HPX, I get a segfault in some
> > _dl
> > _fixup going on during the initialization of a static path/dir
> > object in boost filesystem.
> > 
> > The best explanation I’ve found from stackoverflow says (http://
> > stackoverflow.com/questions/10578315/segmentation-fault-in-dl-runtime-resolve)
> > 1. Heap corruption (overflow) or
> > 2. Mismatched parts of glibc itself.
> > and 2) is clearly a possibility in my case since we’re using
> > libc.so.6 => /bgsys/drivers/ppcfloor/gnu-linux-4.7.2/powerpc64-bgq-
> > linux/lib/libc.so.6 (0x00000fff78420000)
> > which is non-standard.
> > 
> > Question: Is there anything in the bgclang script that does
> > important setting of any var/flag which might influence how the
> > exes
> > get linked. I see stuff in particular that sets
> > BGCLANG_TARGET_INC="-Xclang -isystem -Xclang
> > /gpfs/bbp.cscs.ch/home/
> > biddisco/apps/clang/bgclang/r206161-20140414/libc++/include
> > $BGCLANG_TARGET_INC"
> > and also
> > BGCLANG_TARGET_FLAGS="$BGCLANG_TARGET_FLAGS -L
> > $BGSYS_FLOOR/$GNU_LINUX/powerpc64-bgq-linux/lib"
> > and I wonder about the -Xclang items which I understand to be
> > instructions to clang plugins.
> > 
> > If I have my system configured as follows
> > CFLAGS="-mcpu=a2"
> > CXXFLAGS="-mcpu=a2 -std=c++11 -stdlib=libc++ -D_bgq_linux -I/gpfs/
> > bbp.cscs.ch/home/biddisco/apps/clang/bgclang/r206161-20140414/libc+
> > +/include -I/bgsys/drivers/ppcfloor"
> > LDFLAGS="-dynamic -L/gpfs/bbp.cscs.ch/home/biddisco/apps/clang/
> > bgclang/r206161-20140414/libc++/lib -L/bgsys/drivers/ppcfloor/gnu-
> > linux-4.7.2/powerpc64-bgq-linux/lib"
> > (the -D_bgq_linux is mine and used by certain cpp files to #include
> > stuff)
> > 
> > can you imagine what might be going wrong with my build/link to
> > munge my libc initialization in any way? Have I linked to the wrong
> > libc/c++ stuff perhaps?
> > Typically my exes come out looking like this (ldd below) as far as
> > libs are used, which seems ok, but they do segfault on start which
> > is really annoying me a lot (understatement).
> > 
> > thanks
> > 
> > JB
> > 
> > bbpbg2:~/bgas/clang/build/cpp$ ldd ../hpx/bin/network_storage
> > linux-vdso64.so.1 => (0x00000fff7d580000)
> > libhpxd.so.0 => /gpfs/bbp.cscs.ch/home/biddisco/
> > bgas/clang/build/hpx/lib/hpx/libhpxd.so.0 (0x00000fff79b00000)
> > libhpx_serializationd.so.0 => /gpfs/bbp.cscs.ch/
> > home/biddisco/bgas/clang/build/hpx/lib/hpx/libhpx_serializationd.so.
> > 0 (0x00000fff79ac0000)
> > libboost_date_time-clang35-mt-d-1_55.so.1.55.0 => /
> > gpfs/bbp.cscs.ch/home/biddisco/apps/clang/boost_1_55_0/lib/
> > libboost_date_time-clang35-mt-d-1_55.so.1.55.0 (0x00000fff79a90000)
> > libboost_filesystem-clang35-mt-d-1_55.so.1.55.0 => /
> > gpfs/bbp.cscs.ch/home/biddisco/apps/clang/boost_1_55_0/lib/
> > libboost_filesystem-clang35-mt-d-1_55.so.1.55.0
> > (0x00000fff79a40000)
> > libboost_program_options-clang35-mt-d-1_55.so.1.55.0
> > => /gpfs/bbp.cscs.ch/home/biddisco/apps/clang/boost_1_55_0/lib/
> > libboost_program_options-clang35-mt-d-1_55.so.1.55.0
> > (0x00000fff79920000)
> > libboost_regex-clang35-mt-d-1_55.so.1.55.0 => /gpfs/
> > bbp.cscs.ch/home/biddisco/apps/clang/boost_1_55_0/lib/
> > libboost_regex-clang35-mt-d-1_55.so.1.55.0 (0x00000fff79710000)
> > libboost_serialization-clang35-mt-d-1_55.so.1.55.0
> > => /gpfs/bbp.cscs.ch/home/biddisco/apps/clang/boost_1_55_0/lib/
> > libboost_serialization-clang35-mt-d-1_55.so.1.55.0
> > (0x00000fff79530000)
> > libboost_system-clang35-mt-d-1_55.so.1.55.0 => /
> > gpfs/bbp.cscs.ch/home/biddisco/apps/clang/boost_1_55_0/lib/
> > libboost_system-clang35-mt-d-1_55.so.1.55.0 (0x00000fff79510000)
> > libboost_thread-clang35-mt-d-1_55.so.1.55.0 => /
> > gpfs/bbp.cscs.ch/home/biddisco/apps/clang/boost_1_55_0/lib/
> > libboost_thread-clang35-mt-d-1_55.so.1.55.0 (0x00000fff794c0000)
> > libboost_atomic-clang35-mt-d-1_55.so.1.55.0 => /
> > gpfs/bbp.cscs.ch/home/biddisco/apps/clang/boost_1_55_0/lib/
> > libboost_atomic-clang35-mt-d-1_55.so.1.55.0 (0x00000fff794a0000)
> > libboost_chrono-clang35-mt-d-1_55.so.1.55.0 => /
> > gpfs/bbp.cscs.ch/home/biddisco/apps/clang/boost_1_55_0/lib/
> > libboost_chrono-clang35-mt-d-1_55.so.1.55.0 (0x00000fff79480000)
> > libboost_context-clang35-mt-d-1_55.so.1.55.0 => /
> > gpfs/bbp.cscs.ch/home/biddisco/apps/clang/boost_1_55_0/lib/
> > libboost_context-clang35-mt-d-1_55.so.1.55.0 (0x00000fff79460000)
> > libmpich.so.3 => /usr/lib64/mvapich2/lib/
> > libmpich.so.3 (0x00000fff790f0000)
> > libibverbs.so.1 => /usr/lib64/libibverbs.so.1
> > (0x000000808dbe0000)
> > libopa.so.1 => /usr/lib64/mvapich2/lib/libopa.so.1
> > (0x00000fff790a0000)
> > libmpl.so.1 => /usr/lib64/mvapich2/lib/libmpl.so.1
> > (0x00000080afc00000)
> > librt.so.1 => /bgsys/drivers/ppcfloor/gnu-linux-4.7.
> > 2/powerpc64-bgq-linux/lib/librt.so.1 (0x00000fff78f70000)
> > libpthread.so.0 => /bgsys/drivers/ppcfloor/gnu-
> > linux-4.7.2/powerpc64-bgq-linux/lib/libpthread.so.0
> > (0x00000fff78e50000)
> > librdmacm.so.1 => /usr/lib64/librdmacm.so.1
> > (0x000000808dc10000)
> > libhwloc.so.5 => /gpfs/bbp.cscs.ch/home/biddisco/
> > apps/clang/hwloc-1.8.1/lib/libhwloc.so.5 (0x00000fff78df0000)
> > libdl.so.2 => /bgsys/drivers/ppcfloor/gnu-linux-4.7.
> > 2/powerpc64-bgq-linux/lib/libdl.so.2 (0x00000fff78cd0000)
> > libc++.so.1 => /gpfs/bbp.cscs.ch/home/biddisco/apps/
> > clang/bgclang/libc++/lib/libc++.so.1 (0x00000fff78ab0000)
> > libstdc++.so.6 => /bgsys/drivers/ppcfloor/gnu-
> > linux-4.7.2/powerpc64-bgq-linux/lib/libstdc++.so.6
> > (0x00000fff78880000)
> > libm.so.6 => /bgsys/drivers/ppcfloor/gnu-linux-4.7.
> > 2/powerpc64-bgq-linux/lib/libm.so.6 (0x00000fff78760000)
> > libgcc_s.so.1 => /bgsys/drivers/ppcfloor/gnu-
> > linux-4.7.2/powerpc64-bgq-linux/lib/libgcc_s.so.1
> > (0x00000fff78650000)
> > libc.so.6 => /bgsys/drivers/ppcfloor/gnu-linux-4.7.
> > 2/powerpc64-bgq-linux/lib/libc.so.6 (0x00000fff78420000)
> > libibumad.so.3 => /usr/lib64/libibumad.so.3
> > (0x00000080a0b60000)
> > /lib64/ld64.so.1 (0x0000000020a80000)
> > 
> > _______________________________________________
> > llvm-bgq-discuss mailing list
> > llvm-bgq-discuss at lists.alcf.anl.gov
> > https://lists.alcf.anl.gov/mailman/listinfo/llvm-bgq-discuss
> 
> _______________________________________________
> llvm-bgq-discuss mailing list
> llvm-bgq-discuss at lists.alcf.anl.gov
> https://lists.alcf.anl.gov/mailman/listinfo/llvm-bgq-discuss
> 

-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory