[Llvm-bgq-discuss] bad elf

Sat Feb 2 20:45:31 CST 2013

----- Original Message -----
> From: "Geoffrey Irving" <irving at naml.us>
> To: "Hal Finkel" <hfinkel at anl.gov>
> Cc: llvm-bgq-discuss at lists.alcf.anl.gov
> Sent: Saturday, February 2, 2013 7:30:42 PM
> Subject: Re: [Llvm-bgq-discuss] bad elf
> 
> On Fri, Feb 1, 2013 at 6:27 AM, Hal Finkel <hfinkel at anl.gov> wrote:
> > ----- Original Message -----
> >> From: "Geoffrey Irving" <irving at naml.us>
> >> To: llvm-bgq-discuss at lists.alcf.anl.gov
> >> Sent: Friday, February 1, 2013 1:31:46 AM
> >> Subject: [Llvm-bgq-discuss] bad elf
> >>
> >> I switched to clang mpi wrapper compilers, but still get the bad
> >> elf
> >> error:
> >>
> >>     ...
> >>     2013-02-01 07:06:51.870 (FATAL) [0xfff829da210]
> >> VST-02640-13751-32:22357:ibm.runjob.client.Job: could not start
> >> job:
> >> job failed to start
> >>     2013-02-01 07:06:51.871 (FATAL) [0xfff829da210]
> >> VST-02640-13751-32:22357:ibm.runjob.client.Job: Load failed on
> >> Q02-I7-J00: Application executable ELF header contains invalid
> >> value,
> >> errno 8 Exec format error
> >
> > This almost always indicates that you'd somehow ended up linking to
> > the wrong c-runtime-library startup routines. You've picked up the
> > login-node versions and not the compute-node versions. I don't
> > know whether or not those get included in the shared libraries as
> > well.
> >
> > If you run the 'file' utility on the various .so files and the main
> > executable it should report a 'linux version' associated with the
> > file. Which version does it say? Are they all the same?
> 
> It says "for GNU/Linux 2.6.18" for some of them

Unfortunately, I think that this is indicative of the problem. Furthermore, the libraries that say this seem to be coming from /lib64. Nothing that runs on the compute node should come from there (or /lib*, /usr/lib*, etc.). Do you know why those files are in your list?

> 
> vesta:~% file ~/otherlab/other/install/release/bin/endgame-mpi
> ~/otherlab/other/install/release/lib/lib*.so ~/lib/liblzma.so.5.0.4
> /usr/lib64/libpython2.6.so.1.0 /lib64/libz.so.1.2.3
> /home/irving/lib/libsnappy.so.1.1.3
> /bgsys/drivers/ppcfloor/comm/gcc/lib/libcxxmpich.so.3.3
> /bgsys/drivers/ppcfloor/comm/gcc/lib/libmpich.so.3.3
> /bgsys/drivers/ppcfloor/comm/gcc/lib/libopa.so.1.0.0
> /bgsys/drivers/ppcfloor/comm/gcc/lib/libmpl.so.1.1.0
> /lib64/librt-2.12.so /lib64/libdl-2.12.so /lib64/libpthread-2.12.so
> /usr/lib64/libstdc++.so.6.0.13 /lib64/libm-2.12.so
> /lib64/libgcc_s-4.4.6-20110824.so.1 /lib64/libc-2.12.so
> /lib64/libutil-2.12.so
> /home/irving/otherlab/other/install/release/bin/endgame-mpi:
> ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
> (SYSV), dynamically linked, not stripped
> /home/irving/otherlab/other/install/release/lib/libother_core.so:
> ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
> (SYSV), dynamically linked, not stripped
> /home/irving/otherlab/other/install/release/lib/libpentago_core.so:
> ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
> (SYSV), dynamically linked, not stripped
> /home/irving/otherlab/other/install/release/lib/libpentago_mpi.so:
> ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
> (SYSV), dynamically linked, not stripped
> /home/irving/lib/liblzma.so.5.0.4:
> ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
> (SYSV), dynamically linked, not stripped
> /usr/lib64/libpython2.6.so.1.0:
> ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
> (SYSV), dynamically linked, stripped
> /lib64/libz.so.1.2.3:
> ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
> (SYSV), dynamically linked, stripped
> /home/irving/lib/libsnappy.so.1.1.3:
> ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
> (SYSV), dynamically linked, not stripped
> /bgsys/drivers/ppcfloor/comm/gcc/lib/libcxxmpich.so.3.3:
> ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
> (SYSV), dynamically linked, not stripped
> /bgsys/drivers/ppcfloor/comm/gcc/lib/libmpich.so.3.3:
> ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
> (SYSV), dynamically linked, not stripped
> /bgsys/drivers/ppcfloor/comm/gcc/lib/libopa.so.1.0.0:
> ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
> (SYSV), dynamically linked, not stripped
> /bgsys/drivers/ppcfloor/comm/gcc/lib/libmpl.so.1.1.0:
> ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
> (SYSV), dynamically linked, not stripped
> /lib64/librt-2.12.so:
> ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
> (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18,
> not stripped
> /lib64/libdl-2.12.so:
> ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
> (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18,
> not stripped
> /lib64/libpthread-2.12.so:
> ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
> (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18,
> not stripped
> /usr/lib64/libstdc++.so.6.0.13:
> ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
> (SYSV), dynamically linked, stripped
> /lib64/libm-2.12.so:
> ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
> (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18,
> not stripped
> /lib64/libgcc_s-4.4.6-20110824.so.1:
> ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
> (SYSV), dynamically linked, stripped
> /lib64/libc-2.12.so:
> ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
> (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18,
> not stripped
> /lib64/libutil-2.12.so:
> ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
> (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18,
> not stripped
> 
> Switching over to static linking seems to have fixed the problem, and
> the code now runs on the compute nodes.

Okay, good. Nevertheless, I'd like to get dynamic linking to work as well.

Thanks again,
Hal

> 
> Thanks so much for the help!
> 
> Geoffrey
> 