<html><body>
<p><font size="2" face="sans-serif">Hi Hal,</font><br>
<br>
<font size="2" face="sans-serif">CNK has support for .tbss/.tdata segments (thread specific), which is what glibc uses to track thread-specific locale information. The rest of the support is entirely within glibc, I don't recall that it was disabled. As I recall, if you don't have support for tbss/tdata, programs will crashes when printing floating-point values (glibc needs to know whether to print a comma or decimal per the locale). </font><br>
<br>
<font size="2" face="sans-serif">Tom</font><br>
<br>
<font size="2" face="sans-serif">Tom Gooding<br>
Senior Engineer / Blue Gene SW Lead / C2<br>
tgooding@us.ibm.com 507-253-0747<br>
</font><br>
<br>
<img width="16" height="16" src="cid:1__=08BBF616DFE1A0478f9e8a93df938@us.ibm.com" border="0" alt="Inactive hide details for Hal Finkel ---02/20/2014 01:39:09 PM-------- Original Message ----- > From: "Thomas Heller" <thom.hel"><font size="2" color="#424282" face="sans-serif">Hal Finkel ---02/20/2014 01:39:09 PM-------- Original Message ----- > From: "Thomas Heller" <thom.heller@gmail.com></font><br>
<br>
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td width="1%"><img width="96" height="1" src="cid:2__=08BBF616DFE1A0478f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<ul style="padding-left: 4pt"><font size="1" color="#5F5F5F" face="sans-serif">From:</font></ul>
</td><td width="100%"><img width="1" height="1" src="cid:2__=08BBF616DFE1A0478f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<font size="1" face="sans-serif">Hal Finkel <hfinkel@anl.gov></font></td></tr>
<tr valign="top"><td width="1%"><img width="96" height="1" src="cid:2__=08BBF616DFE1A0478f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<ul style="padding-left: 4pt"><font size="1" color="#5F5F5F" face="sans-serif">To:</font></ul>
</td><td width="100%"><img width="1" height="1" src="cid:2__=08BBF616DFE1A0478f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<font size="1" face="sans-serif">thom heller <thom.heller@gmail.com></font></td></tr>
<tr valign="top"><td width="1%"><img width="96" height="1" src="cid:2__=08BBF616DFE1A0478f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<ul style="padding-left: 4pt"><font size="1" color="#5F5F5F" face="sans-serif">Cc:</font></ul>
</td><td width="100%" valign="middle"><img width="1" height="1" src="cid:2__=08BBF616DFE1A0478f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<font size="1" face="sans-serif">llvm-bgq-discuss@lists.alcf.anl.gov</font></td></tr>
<tr valign="top"><td width="1%"><img width="96" height="1" src="cid:2__=08BBF616DFE1A0478f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<ul style="padding-left: 4pt"><font size="1" color="#5F5F5F" face="sans-serif">Date:</font></ul>
</td><td width="100%"><img width="1" height="1" src="cid:2__=08BBF616DFE1A0478f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<font size="1" face="sans-serif">02/20/2014 01:39 PM</font></td></tr>
<tr valign="top"><td width="1%"><img width="96" height="1" src="cid:2__=08BBF616DFE1A0478f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<ul style="padding-left: 4pt"><font size="1" color="#5F5F5F" face="sans-serif">Subject:</font></ul>
</td><td width="100%"><img width="1" height="1" src="cid:2__=08BBF616DFE1A0478f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<font size="1" face="sans-serif">Re: [Llvm-bgq-discuss] trouble with latest clang install</font></td></tr>
<tr valign="top"><td width="1%"><img width="96" height="1" src="cid:2__=08BBF616DFE1A0478f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<ul style="padding-left: 4pt"><font size="1" color="#5F5F5F" face="sans-serif">Sent by:</font></ul>
</td><td width="100%"><img width="1" height="1" src="cid:2__=08BBF616DFE1A0478f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<font size="1" face="sans-serif">llvm-bgq-discuss-bounces@lists.alcf.anl.gov</font></td></tr>
</table>
<hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br>
<br>
<br>
<tt><font size="2">----- Original Message -----<br>
> From: "Thomas Heller" <thom.heller@gmail.com><br>
> To: "John A. Biddiscombe" <biddisco@cscs.ch><br>
> Cc: llvm-bgq-discuss@lists.alcf.anl.gov, "Hal Finkel" <hfinkel@anl.gov><br>
> Sent: Friday, February 14, 2014 2:02:27 PM<br>
> Subject: Re: [Llvm-bgq-discuss] trouble with latest clang install<br>
> <br>
> Hi all,<br>
> <br>
> Ok, I think i tracked it down.<br>
> If my suspicions are correct, the segfault isn't caused by bgclang or<br>
> hpx<br>
> directly. It looks like parts of boost can't deal with locales<br>
> correctly on<br>
> John's system. Here is how it happens:<br>
> On a regular BGQ compute node, you don't have interactive access and<br>
> i think<br>
> no locale information available. However, John's scenario is slightly<br>
> different:<br>
> 1) He uses SLURM to get on the nodes (interactively or through batch<br>
> jobs)<br>
> 2) He uses the BGAS nodes directly<br>
> <br>
> Now, using 1) has the implication of a feature of SLURM which makes<br>
> the bash<br>
> it spawns once the job has enough resources inherit all the<br>
> environment<br>
> variables the job submission had set (this includes LANG. LC_*). It<br>
> looks like<br>
> some flavors of linux (especially in the embedded world) have a<br>
> problem with<br>
> this. I ran into a similar problem when porting HPX to the Xeon Phi.<br>
> Everything was working nicely on our local machine (no job control,<br>
> direct<br>
> access through ssh etc.). I then moved on to Stampede, when logging<br>
> into one<br>
> of the Phis directly, everything still worked great. But only until i<br>
> stopped<br>
> using an interactive mode and started to submit jobs through the<br>
> batch system.<br>
> Which lead to similar problems John is running into right now ...<br>
> About 2) ... I am not exactly sure how this is related to the problem<br>
> at hand<br>
> ...<br>
> <br>
> Anyway, I was able to reproduce the problem on one of the CNK based<br>
> compute<br>
> nodes on JUQUEEN by using this jobscript:<br>
> # @ job_name = HPX_Hello_World<br>
> # @ comment = "HPX Hello World testrun"<br>
> # @ error = $(job_name).$(jobid).err<br>
> # @ output = $(job_name).$(jobid).out<br>
> # @ environment = COPY_ALL<br>
> # @ wall_clock_limit = 00:30:00<br>
> # @ notification = error<br>
> # @ notify_user = thom.heller@gmail.com<br>
> # @ job_type = bluegene<br>
> # @ bg_size = 32<br>
> # @ queue<br>
> <br>
> APP="$HOME/build/hpx/debug/bin/hello_world"<br>
> <br>
> ENVS="LANG=en_US LC_CTYPE=\"en_US\" LC_NUMERIC=\"en_US\"<br>
> LC_TIME=en_GB<br>
> LC_COLLATE=\"en_US\" LC_MONETARY=\"en_US\" LC_MESSAGES=\"en_US\"<br>
> LC_PAPER=\"en_US\" LC_NAME=\"en_US\" LC_ADDRESS=\"en_US\"<br>
> LC_TELEPHONE=\"en_US\" LC_MEASUREMENT=\"en_US\"<br>
> LC_IDENTIFICATION=\"en_US\"<br>
> LC_ALL=\"en_US\""<br>
> <br>
> runjob --ranks-per-node 1 --exe $APP --args "-t1" --envs $ENVS<br>
> <br>
> Which lead to the exact same error. What I am unsure about though is<br>
> who's<br>
> fault it is. The stack trace John posted earlier comes out of the<br>
> static<br>
> section of the binary which initializes some globals out of the boost<br>
> filesystem library. So we have three candidates: 1) Boost.Filesystem<br>
> 2) libc++<br>
> 3) the libc/posix on the BGAS node.<br>
<br>
This sounds right. CNK (and, specifically, its associated build of glibc) don't have locale support enabled. As a result, as I recall, only the default ('C') is supported.<br>
<br>
If it turns out that this is a bug in libc++, then we should fix it there. Maybe it is worthwhile to have a Boost build that is patched to avoid this problem as well?<br>
<br>
In any case, thanks for investigating this and sharing your findings!<br>
<br>
-Hal<br>
<br>
> <br>
> The solution to this problem is btw to unset all those environment<br>
> variables.<br>
> I commited a fix for HPX for working around this problem which should<br>
> not<br>
> require to manually unset those environment variables<br>
> (</font></tt><tt><font size="2"><a href="https://github.com/STEllAR-GROUP/hpx/commit/65ce125466ae43e68e19e89b3e50ece0721786de">https://github.com/STEllAR-GROUP/hpx/commit/65ce125466ae43e68e19e89b3e50ece0721786de</a></font></tt><tt><font size="2">).<br>
> Thanks for the patience.<br>
> <br>
> Regards,<br>
> Thomas<br>
> <br>
> On Friday, February 14, 2014 12:52:24 Biddiscombe, John A. wrote:<br>
> > Hal<br>
> > <br>
> > Apologies, I didn’t realize I was using the wrong wrapper.<br>
> > <br>
> > I recompiled using the bgclang++11 wrapper and things work much<br>
> > better.<br>
> > I first compiled boost ok, but had trouble linking to it - I ran<br>
> > into the<br>
> > cxxABI link error with boost program_options:: __1 etc etc<br>
> > <br>
> > After a bit of goggling around explained to me the std c++ lib<br>
> > issues, so<br>
> > I had another go using the following settings …<br>
> > <br>
> > export<br>
> > CC=/gpfs/bbp.cscs.ch/home/biddisco/bgas/apps/clang/bin/bgclang<br>
> > export<br>
> > CXX=/gpfs/bbp.cscs.ch/home/biddisco/bgas/apps/clang/bin/bgclang++11<br>
> > export<br>
> > PATH=/gpfs/bbp.cscs.ch/home/biddisco/bgas/apps/clang/bin:$PATH<br>
> > <br>
> > I found some info about building boost with clang and followed<br>
> > instructions here<br>
> > </font></tt><tt><font size="2"><a href="http://stackoverflow.com/questions/11081818/linking-troubles-with-boostprog">http://stackoverflow.com/questions/11081818/linking-troubles-with-boostprog</a></font></tt><tt><font size="2"><br>
> > ram-options-on-osx-using-llvm?lq=1<br>
> > I modified tools/build/v2/user-config.jam to include the clang-11<br>
> > option<br>
> > using clang : 11<br>
> > <br>
> > :<br>
> > "/gpfs/bbp.cscs.ch/home/biddisco/bgas/apps/clang/bin/bgclang++11"<br>
> > : <cxxflags>"-std=c++11 -stdlib=libc++ -ftemplate-depth=512"<br>
> > <br>
> > <linkflags>"-stdlib=libc++"<br>
> > ;<br>
> > <br>
> > <br>
> > And then proceeded to building boost using the following commands<br>
> > ./bootstrap.sh --with-toolset=clang-11<br>
> > ./b2 -j 16 toolset=clang-11 cxxflags="-fPIC" --threading=multi<br>
> > --without-mpi --without-python<br>
> > --prefix=/gpfs/bbp.cscs.ch/home/biddisco/apps/clang/boost_1_54_0<br>
> > <br>
> > And boost compiles fine.<br>
> > "The Boost C++ Libraries were successfully built!"<br>
> > <br>
> > To test, I compiled the boost serialisation demo from this page<br>
> > </font></tt><tt><font size="2"><a href="http://www.boost.org/doc/libs/1_42_0/libs/serialization/example/demo.cpp">http://www.boost.org/doc/libs/1_42_0/libs/serialization/example/demo.cpp</a></font></tt><tt><font size="2"><br>
> > And also a simple boost::program_options demo and boost::filesystem<br>
> > demo<br>
> > they all run fine<br>
> > <br>
> > Thank you very much for the help and all the work you’ve put in<br>
> > getting<br>
> > the clang stuff running..<br>
> > <br>
> > But…<br>
> > <br>
> > when I run simple demos from the HPX library<br>
> > <br>
> > bbpbg2:~/bgas/build/hpx$ bin/hello_world<br>
> > terminate called after throwing an instance of<br>
> > 'std::__1::runtime_error'<br>
> > what(): collate_byname<char>::collate_byname failed to construct<br>
> > for<br>
> > Aborted (core dumped)<br>
> > <br>
> > <br>
> > gdb shows me a trace …<br>
> > (gdb) where<br>
> > #0 0x00000fffb3458c5c in raise (sig=6) at<br>
> > ../nptl/sysdeps/unix/sysv/linux/raise.c:67<br>
> > #1 0x00000fffb345abd4 in abort () at abort.c:92<br>
> > #2 0x00000fffb3aa7b00 in __gnu_cxx::__verbose_terminate_handler ()<br>
> > at<br>
> > /bgsys/drivers/V1R2M1/ppc64/toolchain/gnu/gcc-4.4.6/libstdc++-v3/libsupc++/<br>
> > vterminate.cc:93<br>
> > #3 0x00000fffb3aa4d74 in __cxxabiv1::__terminate (handler=<value<br>
> > optimized out>)<br>
> > at<br>
> > /bgsys/drivers/V1R2M1/ppc64/toolchain/gnu/gcc-4.4.6/libstdc++-v3/libsupc++/<br>
> > eh_terminate.cc:38<br>
> > #4 0x00000fffb3aa4db8 in std::terminate () at<br>
> > /bgsys/drivers/V1R2M1/ppc64/toolchain/gnu/gcc-4.4.6/libstdc++-v3/libsupc++/<br>
> > eh_terminate.cc:48<br>
> > #5 0x00000fffb47b1c14 in .__clang_call_terminate () from<br>
> > /gpfs/bbp.cscs.ch/home/biddisco/apps/clang/boost_1_54_0/lib/libboost_filesy<br>
> > stem.so.1.54.0<br>
> > #6 0x00000fffb47b48a0 in<br>
> > ._ZNK5boost10filesystem4path7compareERKS1_ ()<br>
> > from<br>
> > /gpfs/bbp.cscs.ch/home/biddisco/apps/clang/boost_1_54_0/lib/libboost_filesy<br>
> > stem.so.1.54.0<br>
> > Backtrace stopped: frame did not save the PC<br>
> > <br>
> > <br>
> > It looks very suspicious as there are some stdlib++ appearances in<br>
> > there.<br>
> > <br>
> > Does anything here give you any idea of what might have gone wrong.<br>
> > I’ve<br>
> > tried a number of rebuilds and the error persists, whilst simple<br>
> > demos run<br>
> > ok. I’m not sure where to look to diagnose what’s up (I’ve<br>
> > contacted the<br>
> > HPX people as well). One question is why the shared clang libc++<br>
> > links to<br>
> > the stdlibc++ one. If I do an<br>
> > <br>
> > bbpbg2:~/bgas/build/c++test$ ldd<br>
> > /gpfs/bbp.cscs.ch/home/biddisco/bgas/apps/clang/libc++/lib/libc++.so.1.0<br>
> > <br>
> > linux-vdso64.so.1 => (0x00000fff9ad40000)<br>
> > libpthread.so.0 =><br>
> > /bgsys/drivers/V1R2M1/ppc64/gnu-linux/powerpc64-bgq-linux/lib/libpthread.so<br>
> > .0 (0x00000fff9ab00000)<br>
> > librt.so.1 =><br>
> > /bgsys/drivers/V1R2M1/ppc64/gnu-linux/powerpc64-bgq-linux/lib/librt.so.1<br>
> > (0x00000fff9a9d0000)<br>
> > libc.so.6 =><br>
> > /bgsys/drivers/V1R2M1/ppc64/gnu-linux/powerpc64-bgq-linux/lib/libc.so.6<br>
> > (0x00000fff9a790000)<br>
> > libstdc++.so.6 =><br>
> > /bgsys/drivers/V1R2M1/ppc64/gnu-linux/powerpc64-bgq-linux/lib/libstdc++.so.<br>
> > 6 (0x00000fff9a550000)<br>
> > /lib64/ld64.so.1 (0x0000000032420000)<br>
> > libm.so.6 =><br>
> > /bgsys/drivers/V1R2M1/ppc64/gnu-linux/powerpc64-bgq-linux/lib/libm.so.6<br>
> > (0x00000fff9a430000)<br>
> > libgcc_s.so.1 =><br>
> > /bgsys/drivers/V1R2M1/ppc64/gnu-linux/powerpc64-bgq-linux/lib/libgcc_s.so.1<br>
> > (0x00000fff9a320000)<br>
> > <br>
> > <br>
> > It seems odd. Could this be causing the trouble? (the demos run<br>
> > fine<br>
> > though, so I guess not).<br>
> > <br>
> > Anyway, I’ll keep poking around, if anything comes to mind, I’m<br>
> > grateful<br>
> > for help.<br>
> > <br>
> > Thanks<br>
> > <br>
> > JB<br>
> > <br>
> > <br>
> > _______________________________________________<br>
> > llvm-bgq-discuss mailing list<br>
> > llvm-bgq-discuss@lists.alcf.anl.gov<br>
> > </font></tt><tt><font size="2"><a href="https://lists.alcf.anl.gov/mailman/listinfo/llvm-bgq-discuss">https://lists.alcf.anl.gov/mailman/listinfo/llvm-bgq-discuss</a></font></tt><tt><font size="2"><br>
> <br>
> <br>
<br>
-- <br>
Hal Finkel<br>
Assistant Computational Scientist<br>
Leadership Computing Facility<br>
Argonne National Laboratory<br>
_______________________________________________<br>
llvm-bgq-discuss mailing list<br>
llvm-bgq-discuss@lists.alcf.anl.gov<br>
</font></tt><tt><font size="2"><a href="https://lists.alcf.anl.gov/mailman/listinfo/llvm-bgq-discuss">https://lists.alcf.anl.gov/mailman/listinfo/llvm-bgq-discuss</a></font></tt><tt><font size="2"><br>
</font></tt><br>
<br>
</body></html>