<html><body>
<p><font size="2" face="sans-serif">I'm not sure how much faith to put in the ldd output. It probably be better to see the following output:</font><br>
<font size="2" face="sans-serif">* objdump -p <filename></font><br>
<font size="2" face="sans-serif">* run with --strace 0 enabled. (or the slurm/cobalt equivalent) </font><br>
<br>
<font size="2" face="sans-serif">The 'ldd' output isn't precisely how dynamic libraries are loaded on CNK. There are some subtle differences:</font><br>
<font size="2" face="sans-serif">1) There's no vdso layer on CNK, so the "</font><tt><font size="2">linux-vdso64.so.1 => (0x00000fff9ae40000)</font></tt><font size="2" face="sans-serif">" line is incorrect. </font><br>
<font size="2" face="sans-serif">2) 'ldd' doesn't use the same ld.so cache file. CNK uses: '/etc/ld.so.bgq.cache' Whereas 'ldd' appears to use: </font><font size="2" face="Menlo-Regular">"/etc/ld.so.cache</font><font size="2" face="sans-serif">"</font><br>
<font size="2" face="sans-serif">3) File system paths and permissions may differ between the ionode and FEN. (although I don't suspect that here)</font><br>
<br>
<font size="2" face="sans-serif">Comparing a simple dynamically-linked hello world with run on CNK with an strace enabled. You'll see they pick up different paths:</font><br>
<br>
<font size="2" face="Menlo-Regular">% ldd dynhelloworld.elf </font><br>
<font size="2" face="Menlo-Regular"> linux-vdso64.so.1 => (0x00000fffab9a0000)</font><br>
<font size="2" face="Menlo-Regular"> libpthread.so.0 => /lib64/libpthread.so.0 (0x0000008078a70000)</font><br>
<font size="2" face="Menlo-Regular"> libc.so.6 => /lib64/libc.so.6 (0x0000008078880000)</font><br>
<font size="2" face="Menlo-Regular"> /lib64/bgq/ld64.so.1 => /lib64/ld64.so.1 (0x0000000033c70000)</font><br>
<br>
<font size="2" face="sans-serif">whereas:</font><br>
<br>
<font size="2" face="Menlo-Regular">% runjob --block R00-M0-N00 --np 1 --strace 0 : dynhelloworld.elf</font><br>
<font size="2" face="sans-serif">...</font><br>
<font size="2" face="Menlo-Regular">(I) sc_open[0.0:1]: pathname="dynhelloworld.elf", flags=0x00000000 mode=</font><br>
<font size="2" face="Menlo-Regular">(I) sc_read[0.0:1]: fd=3, buf=0x1dbfffb130, cnt=832</font><br>
<font size="2" face="Menlo-Regular">(I) sc_open[0.0:1]: pathname="/etc/ld.so.bgq.cache", flags=0x00000000 mode=</font><br>
<font size="2" face="Menlo-Regular">(I) sc_open[0.0:1]: pathname="/usr/lib64/bgq/tls/libpthread.so.0", flags=0x00000000 mode=</font><br>
<font size="2" face="Menlo-Regular">(I) sc_open[0.0:1]: pathname="/usr/lib64/bgq/libpthread.so.0", flags=0x00000000 mode=</font><br>
<font size="2" face="Menlo-Regular">(I) sc_open[0.0:1]: pathname="/lib64/bgq/tls/libpthread.so.0", flags=0x00000000 mode=</font><br>
<font size="2" face="Menlo-Regular">(I) sc_open[0.0:1]: pathname="/lib64/bgq/libpthread.so.0", flags=0x00000000 mode=</font><br>
<font size="2" face="Menlo-Regular">(I) sc_read[0.0:1]: fd=3, buf=0x1dbfffab70, cnt=832</font><br>
<font size="2" face="Menlo-Regular">(I) sc_open[0.0:1]: pathname="/usr/lib64/bgq/libc.so.6", flags=0x00000000 mode=</font><br>
<font size="2" face="Menlo-Regular">(I) sc_open[0.0:1]: pathname="/lib64/bgq/libc.so.6", flags=0x00000000 mode=</font><br>
<font size="2" face="Menlo-Regular">(I) sc_read[0.0:1]: fd=3, buf=0x1dbfffab30, cnt=832</font><br>
<br>
<font size="2" face="sans-serif">ld.so is very search-y when looking for libraries. But it found:</font><br>
<font size="2" face="sans-serif"> libpthread.so.0 at </font><font size="2" face="Menlo-Regular">/lib64/bgq/libpthread.so.0 (not /lib64/libpthread.so.0)</font><br>
<font size="2" face="sans-serif"> libc.so.6 at </font><font size="2" face="Menlo-Regular">/lib64/bgq/libc.so.6 (not /lib64/libc.so.6)</font><br>
<br>
<font size="2" face="sans-serif">The files in /lib64/bgq/* were compiled for CNK. Where as /lib64/* are Linux libraries. </font><br>
<br>
<font size="2" face="sans-serif">Hope this helps,</font><br>
<font size="2" face="sans-serif">Tom</font><br>
<br>
<font size="2" face="sans-serif">Tom Gooding<br>
Senior Engineer / Blue Gene SW Lead / CAPI<br>
tgooding@us.ibm.com 507-253-0747<br>
</font><br>
<br>
<tt><font size="2">llvm-bgq-discuss-bounces@lists.alcf.anl.gov wrote on 05/09/2014 08:24:04 AM:<br>
<br>
> From: Hal Finkel <hfinkel@anl.gov></font></tt><br>
<tt><font size="2">> To: Phil Miller <mille121@illinois.edu></font></tt><br>
<tt><font size="2">> Cc: llvm-bgq-discuss@lists.alcf.anl.gov</font></tt><br>
<tt><font size="2">> Date: 05/09/2014 08:25 AM</font></tt><br>
<tt><font size="2">> Subject: Re: [Llvm-bgq-discuss] Dynamic linking failure</font></tt><br>
<tt><font size="2">> Sent by: llvm-bgq-discuss-bounces@lists.alcf.anl.gov</font></tt><br>
<tt><font size="2">> <br>
> ----- Original Message -----<br>
> > From: "Phil Miller" <mille121@illinois.edu><br>
> > To: "Hal Finkel" <hfinkel@anl.gov><br>
> > Cc: llvm-bgq-discuss@lists.alcf.anl.gov<br>
> > Sent: Thursday, May 8, 2014 7:33:23 PM<br>
> > Subject: Re: [Llvm-bgq-discuss] Dynamic linking failure<br>
> > <br>
> > <br>
> > So, when I forced the matter by setting<br>
> > <br>
> > LD_LIBRARY_PATH=/bgsys/drivers/toolchain/<br>
> > V1R2M1_base_4.7.2/gnu-linux-4.<br>
> > 7.2/powerpc64-bgq-linux/lib/:/bgsys/drivers/ppcfloor/comm/lib/<br>
> > <br>
> > in my execution, it was successful. I'm not sure whether that should<br>
> > have worked automatically or not.<br>
> > <br>
> <br>
> Yes, I think so. If it works with ldd on the login then it <br>
> definitely should work without any special LD_LIBRARY_PATH.<br>
> <br>
> -Hal<br>
> <br>
> > <br>
> > <br>
> > <br>
> > On Thu, May 8, 2014 at 7:21 PM, Hal Finkel < hfinkel@anl.gov > wrote:<br>
> > <br>
> > <br>
> > <br>
> > <br>
> > ----- Original Message -----<br>
> > > From: "Phil Miller" < mille121@illinois.edu ><br>
> > > To: llvm-bgq-discuss@lists.alcf.anl.gov<br>
> > > Sent: Thursday, May 8, 2014 1:53:10 PM<br>
> > > Subject: [Llvm-bgq-discuss] Dynamic linking failure<br>
> > > <br>
> > > <br>
> > > <br>
> > > <br>
> > > I've compiled my application using bgclang/bgclang++ on Vesta, and<br>
> > > the process goes smoothly. When I use a static linked build of the<br>
> > > system, it runs cleanly.<br>
> > > <br>
> > > I want to try out Address Sanitizer (aka 'asan', activated with<br>
> > > '-fsanitize=address'), which requires dynamic linking. Sadly, that<br>
> > > lets me compile and link, but fails to run. Here's what I'm seeing,<br>
> > > again on Vesta:<br>
> > > <br>
> > > ===============<br>
> > > $ file check<br>
> > > check: ELF 64-bit MSB executable, 64-bit PowerPC or cisco 7500,<br>
> > > version 1 (SYSV), dynamically linked (uses shared libs), for<br>
> > > GNU/Linux 2.4.21, not stripped<br>
> > > <br>
> > > $ echo $LD_LIBRARY_PATH | tr : '\n'<br>
> > > /bgsys/drivers/ppcfloor/comm/lib<br>
> > > /bgsys/drivers/ppcfloor/comm/gcc/lib<br>
> > > /soft/compilers/ibmcmp-feb2014/vac/bg/12.1/bglib64<br>
> > > /soft/compilers/ibmcmp-feb2014/vacpp/bg/12.1/bglib64<br>
> > > /soft/compilers/ibmcmp-feb2014/xlf/bg/14.1/bglib64<br>
> > > /soft/compilers/ibmcmp-feb2014/xlmass/bg/7.3/bglib64<br>
> > > /soft/compilers/ibmcmp-feb2014/xlsmp/bg/3.1/bglib64<br>
> > > /dbhome/db2cat/sqllib/lib64<br>
> > > /dbhome/db2cat/sqllib/lib32<br>
> > > <br>
> > > $ ldd check<br>
> > > linux-vdso64.so.1 => (0x00000fff9ae40000)<br>
> > > libdl.so.2 =><br>
> > > /bgsys/drivers/toolchain/V1R2M1_base_4.7.2/gnu-linux-4.7.2/<br>
> powerpc64-bgq-linux/lib/libdl.so.2<br>
> > > (0x00000fff9ad20000)<br>
> > > libpami-gcc.so => /bgsys/drivers/ppcfloor/comm/lib/libpami-gcc.so<br>
> > > (0x00000fff9a7b0000)<br>
> > > libpthread.so.0 =><br>
> > > /bgsys/drivers/toolchain/V1R2M1_base_4.7.2/gnu-linux-4.7.2/<br>
> powerpc64-bgq-linux/lib/libpthread.so.0<br>
> > > (0x00000fff9a690000)<br>
> > > librt.so.1 =><br>
> > > /bgsys/drivers/toolchain/V1R2M1_base_4.7.2/gnu-linux-4.7.2/<br>
> powerpc64-bgq-linux/lib/librt.so.1<br>
> > > (0x00000fff9a560000)<br>
> > > libm.so.6 =><br>
> > > /bgsys/drivers/toolchain/V1R2M1_base_4.7.2/gnu-linux-4.7.2/<br>
> powerpc64-bgq-linux/lib/libm.so.6<br>
> > > (0x00000fff9a440000)<br>
> > > libstdc++.so.6 =><br>
> > > /bgsys/drivers/toolchain/V1R2M1_base_4.7.2/gnu-linux-4.7.2/<br>
> powerpc64-bgq-linux/lib/libstdc++.so.6<br>
> > > (0x00000fff9a210000)<br>
> > > libgcc_s.so.1 =><br>
> > > /bgsys/drivers/toolchain/V1R2M1_base_4.7.2/gnu-linux-4.7.2/<br>
> powerpc64-bgq-linux/lib/libgcc_s.so.1<br>
> > > (0x00000fff9a100000)<br>
> > > libc.so.6 =><br>
> > > /bgsys/drivers/toolchain/V1R2M1_base_4.7.2/gnu-linux-4.7.2/<br>
> powerpc64-bgq-linux/lib/libc.so.6<br>
> > > (0x00000fff99ed0000)<br>
> > > /lib64/ld64.so.1 (0x000000003cc80000)<br>
> > > <br>
> > > $ qsub -t 10 -A PARTS -n 1 --mode c1 ./check<br>
> > > 190745<br>
> > > <br>
> > > $ cat 190745.error<br>
> > > 2014-05-08 18:45:16.828 (INFO ) [0x40000a3bc20]<br>
> > > 27642:tatu.runjob.client: scheduler job id is 190745<br>
> > > 2014-05-08 18:45:16.829 (DEBUG) [0x40000a3bc20]<br>
> > > 27642:tatu.runjob.client: the environment variable COBALT_RESID did<br>
> > > not contain a Cobalt reservation id<br>
> > > 2014-05-08 18:45:16.844 (INFO ) [0x400004034d0]<br>
> > > 27642:tatu.runjob.monitor: monitor started<br>
> > > 2014-05-08 18:45:16.855 (INFO ) [0x40000a3bc20]<br>
> > > 27642:ibm.runjob.AbstractOptions: using properties file<br>
> > > /bgsys/local/etc/bg.properties<br>
> > > 2014-05-08 18:45:16.856 (INFO ) [0x40000a3bc20]<br>
> > > 27642:ibm.runjob.AbstractOptions: max open file descriptors: 65536<br>
> > > 2014-05-08 18:45:16.856 (INFO ) [0x40000a3bc20]<br>
> > > 27642:ibm.runjob.AbstractOptions: core file limit:<br>
> > > 18446744073709551615<br>
> > > 2014-05-08 18:45:16.977 (INFO ) [0x400004034d0]<br>
> > > 27642:tatu.runjob.monitor: task record 645048 created<br>
> > > 2014-05-08 18:45:16.978 (INFO ) [0x40000a3bc20]<br>
> > > VST-20420-31531-32:27642:ibm.runjob.client.options.Parser: set<br>
> > > local<br>
> > > socket to runjob_mux from properties file<br>
> > > 2014-05-08 18:45:17.782 (INFO ) [0x400004034d0]<br>
> > > 27642:tatu.runjob.monitor: tracklib completed<br>
> > > 2014-05-08 18:45:19.162 (INFO ) [0x40000a3bc20]<br>
> > > VST-20420-31531-32:848093:ibm.runjob.client.Job: job 848093 started<br>
> > > /gpfs/vesta-home/phil/charm-6.6/pamilrts-bluegeneq-asan-clang/<br>
> tests/util/./check:<br>
> > > error while loading shared libraries: libpami-gcc.so: cannot open<br>
> > > shared object file: No such file or directory<br>
> > > 2014-05-08 18:45:20.952 (INFO ) [0x40000a3bc20]<br>
> > > VST-20420-31531-32:848093:ibm.runjob.client.Job: exited with status<br>
> > > 127<br>
> > > 2014-05-08 18:45:20.952 (WARN ) [0x40000a3bc20]<br>
> > > VST-20420-31531-32:848093:ibm.runjob.client.Job: normal termination<br>
> > > with status 127 from rank 0<br>
> > > 2014-05-08 18:45:20.952 (INFO ) [0x40000a3bc20] tatu.runjob.client:<br>
> > > task exited with status 127<br>
> > > 2014-05-08 18:45:20.952 (INFO ) [0x400004034d0]<br>
> > > 27642:tatu.runjob.monitor: monitor terminating<br>
> > > 2014-05-08 18:45:20.956 (INFO ) [0x40000a3bc20] tatu.runjob.client:<br>
> > > monitor completed<br>
> > > =========<br>
> > > <br>
> > > Have a missed a step in running dynamically linked binaries on<br>
> > > BG/Q?<br>
> > <br>
> > No, I don't think so. I see no reason why this should not work. Can<br>
> > you e-mail support about this?<br>
> > <br>
> > -Hal<br>
> > <br>
> > > <br>
> > > _______________________________________________<br>
> > > llvm-bgq-discuss mailing list<br>
> > > llvm-bgq-discuss@lists.alcf.anl.gov<br>
> > > <a href="https://lists.alcf.anl.gov/mailman/listinfo/llvm-bgq-discuss">https://lists.alcf.anl.gov/mailman/listinfo/llvm-bgq-discuss</a><br>
> > > <br>
> > <br>
> > --<br>
> > Hal Finkel<br>
> > Assistant Computational Scientist<br>
> > Leadership Computing Facility<br>
> > Argonne National Laboratory<br>
> > <br>
> > <br>
> <br>
> -- <br>
> Hal Finkel<br>
> Assistant Computational Scientist<br>
> Leadership Computing Facility<br>
> Argonne National Laboratory<br>
> _______________________________________________<br>
> llvm-bgq-discuss mailing list<br>
> llvm-bgq-discuss@lists.alcf.anl.gov<br>
> <a href="https://lists.alcf.anl.gov/mailman/listinfo/llvm-bgq-discuss">https://lists.alcf.anl.gov/mailman/listinfo/llvm-bgq-discuss</a><br>
> <br>
</font></tt></body></html>