[Llvm-bgq-discuss] Error With Dynamic Linking on Mira Head Node

[Hal Finkel]

Hi Derek,

bgclang itself is not setup to create executables for the login nodes. 
To some extent, the problems with the configure scripts are unavoidable, 
especially if MPI is required. The assertions from 
PAMI::BgqJobPersonality come from static initializers inside the MPI 
library which run when the MPI library is loaded. I can provide you with 
a work-around for this problem, but you'll still run into problems if 
the compiler generates any QPX (i.e. SIMD) instructions because those 
won't work on the login node.

In any case, you can work around the static initializer problem by 
creating a version of the libpami-gcc.so library that is edited so that 
it won't run its static initializers and then add that library to your 
LD_LIBRARY_PATH when running the configure scripts. Here's the magic 
incarnation I've used for this purpose:

cp -a /bgsys/drivers/ppcfloor/comm/lib/libpami-gcc.so ./

printf '\x4e\x80\x00\x20' | dd of=libpami-gcc.so bs=1 seek=$(echo 
"ibase=16;obase=A;$(objdump -d libpami-gcc.so | grep 
__do_global_ctors_aux | grep '^0' | awk '{ print $1 }' | tr '[a-f]' 
'[A-F]')" | bc) count=4 conv=notrunc

Hopefully, this will be enough to get you past the configure scripts. If 
not, or if you have any questions, please let me know.

  -Hal

On 12/29/2016 11:27 AM, Derek Gaston wrote:
> Having trouble getting a dynamically linked executable working on 
> Mira/Cetus head nodes.  The executable _is_ working if I submit it to 
> the queue.
>
> Compiling a static executable works both on the head node and on the 
> compute nodes.
>
> Is there some special environment sauce I need that will enable me to 
> run these executables on the head node? Obviously I don't want to run 
> much on the head node... but not even being able to test the 
> executables without a trip through the queue is problematic!  For 
> instance... "configure" scripts kind of rely on being able to 
> compile/run simple test programs!
>
> My .soft is:
>
> +mpiwrapper-bgclang-mpi3.legacy.ndebug
> @default
>
> (Note: I have also tried essentially all the other bgclang options and 
> they all produce the same error).
>
>
> I'm trying to compile a test.c:
>
> int main()
> {
>   return 0;
> }
>
> My compile line is:
>
> mpicc -o test -dynamic test.c
>
>
> The error I'm seeing is:
>
> $ ./test
> ERROR: ld.so: object '/soft/buildtools/trackdeps/${LIB}/trackdeps.so' 
> from LD_PRELOAD cannot be preloaded: ignored.
> /bgsys/source/srcV1R2M2.15270/comm/sys/buildtools/pami/common/bgq/BgqPersonality.h:102: 
>
> /bgsys/source/srcV1R2M2.15270/comm/sys/buildtools/pami/common/bgq/BgqPersonality.h<102>
> Aborted (core dumped)
>
>
> Getting a stack trace out of the core file gives:
>
> (gdb) bt
> #0  0x00000fff837fec40 in raise (sig=Unhandled dwarf expression opcode 
> 0xf3
> ) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
> #1  0x00000fff838008b8 in abort () at abort.c:90
> #2  0x00000fff841a12b8 in PAMI::BgqJobPersonality::BgqJobPersonality 
> (this=<value optimized out>) at 
> /bgsys/source/srcV1R2M2.15270/comm/sys/buildtools/pami/common/bgq/BgqPersonality.h:102
> #3  0x00000fff841aeff4 in PAMI::Global::Global (this=0xfff84316848) at 
> /bgsys/source/srcV1R2M2.15270/comm/sys/buildtools/pami/common/bgq/Global.h:110
> #4  0x00000fff8419fa1c in __static_initialization_and_destruction_0 () 
> at 
> /bgsys/source/srcV1R2M2.15270/comm/sys/buildtools/pami/common/bgq/BgqGlobal.cc:25
> #5  global constructors keyed to BgqGlobal.cc(void) () at 
> /bgsys/source/srcV1R2M2.15270/comm/sys/buildtools/pami/common/bgq/BgqGlobal.cc:70
> #6  0x00000fff841d2b6c in .__do_global_ctors_aux () from 
> /bgsys/drivers/V1R2M2/ppc64/comm/lib/libpami-gcc.so
> #7  0x00000fff83f93cac in ._init () from 
> /bgsys/drivers/V1R2M2/ppc64/comm/lib/libpami-gcc.so
> #8  0x00000fff8492493c in call_init (l=0xfff84a17138, argc=Unhandled 
> dwarf expression opcode 0xf3
> ) at dl-init.c:69
> #9  call_init (l=0xfff84a17138, argc=Unhandled dwarf expression opcode 
> 0xf3
> ) at dl-init.c:34
> #10 0x00000fff84924acc in _dl_init (main_map=0xfff84a13610, 
> argc=Unhandled dwarf expression opcode 0xf3
> ) at dl-init.c:133
> #11 0x00000fff849140bc in ._dl_start_user () from 
> /bgsys/drivers/V1R2M2/ppc64/gnu-linux-4.7.2/powerpc64-bgq-linux/lib/ld64.so.1
>
>
> Thanks for any help!
>
> Derek
>
>
> _______________________________________________
> llvm-bgq-discuss mailing list
> llvm-bgq-discuss at lists.alcf.anl.gov
> https://lists.alcf.anl.gov/mailman/listinfo/llvm-bgq-discuss

-- 
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alcf.anl.gov/pipermail/llvm-bgq-discuss/attachments/20170102/a6891112/attachment.html>