[Llvm-bgq-discuss] bad elf

[Geoffrey Irving]

On Fri, Feb 1, 2013 at 6:27 AM, Hal Finkel <hfinkel at anl.gov> wrote:
> ----- Original Message -----
>> From: "Geoffrey Irving" <irving at naml.us>
>> To: llvm-bgq-discuss at lists.alcf.anl.gov
>> Sent: Friday, February 1, 2013 1:31:46 AM
>> Subject: [Llvm-bgq-discuss] bad elf
>>
>> I switched to clang mpi wrapper compilers, but still get the bad elf
>> error:
>>
>>     ...
>>     2013-02-01 07:06:51.870 (FATAL) [0xfff829da210]
>> VST-02640-13751-32:22357:ibm.runjob.client.Job: could not start job:
>> job failed to start
>>     2013-02-01 07:06:51.871 (FATAL) [0xfff829da210]
>> VST-02640-13751-32:22357:ibm.runjob.client.Job: Load failed on
>> Q02-I7-J00: Application executable ELF header contains invalid value,
>> errno 8 Exec format error
>
> This almost always indicates that you'd somehow ended up linking to the wrong c-runtime-library startup routines. You've picked up the login-node versions and not the compute-node versions. I don't know whether or not those get included in the shared libraries as well.
>
> If you run the 'file' utility on the various .so files and the main executable it should report a 'linux version' associated with the file. Which version does it say? Are they all the same?

It says "for GNU/Linux 2.6.18" for some of them

vesta:~% file ~/otherlab/other/install/release/bin/endgame-mpi
~/otherlab/other/install/release/lib/lib*.so ~/lib/liblzma.so.5.0.4
/usr/lib64/libpython2.6.so.1.0 /lib64/libz.so.1.2.3
/home/irving/lib/libsnappy.so.1.1.3
/bgsys/drivers/ppcfloor/comm/gcc/lib/libcxxmpich.so.3.3
/bgsys/drivers/ppcfloor/comm/gcc/lib/libmpich.so.3.3
/bgsys/drivers/ppcfloor/comm/gcc/lib/libopa.so.1.0.0
/bgsys/drivers/ppcfloor/comm/gcc/lib/libmpl.so.1.1.0
/lib64/librt-2.12.so /lib64/libdl-2.12.so /lib64/libpthread-2.12.so
/usr/lib64/libstdc++.so.6.0.13 /lib64/libm-2.12.so
/lib64/libgcc_s-4.4.6-20110824.so.1 /lib64/libc-2.12.so
/lib64/libutil-2.12.so
/home/irving/otherlab/other/install/release/bin/endgame-mpi:
ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
(SYSV), dynamically linked, not stripped
/home/irving/otherlab/other/install/release/lib/libother_core.so:
ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
(SYSV), dynamically linked, not stripped
/home/irving/otherlab/other/install/release/lib/libpentago_core.so:
ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
(SYSV), dynamically linked, not stripped
/home/irving/otherlab/other/install/release/lib/libpentago_mpi.so:
ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
(SYSV), dynamically linked, not stripped
/home/irving/lib/liblzma.so.5.0.4:
ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
(SYSV), dynamically linked, not stripped
/usr/lib64/libpython2.6.so.1.0:
ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
(SYSV), dynamically linked, stripped
/lib64/libz.so.1.2.3:
ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
(SYSV), dynamically linked, stripped
/home/irving/lib/libsnappy.so.1.1.3:
ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
(SYSV), dynamically linked, not stripped
/bgsys/drivers/ppcfloor/comm/gcc/lib/libcxxmpich.so.3.3:
ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
(SYSV), dynamically linked, not stripped
/bgsys/drivers/ppcfloor/comm/gcc/lib/libmpich.so.3.3:
ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
(SYSV), dynamically linked, not stripped
/bgsys/drivers/ppcfloor/comm/gcc/lib/libopa.so.1.0.0:
ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
(SYSV), dynamically linked, not stripped
/bgsys/drivers/ppcfloor/comm/gcc/lib/libmpl.so.1.1.0:
ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
(SYSV), dynamically linked, not stripped
/lib64/librt-2.12.so:
ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
(SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18,
not stripped
/lib64/libdl-2.12.so:
ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
(SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18,
not stripped
/lib64/libpthread-2.12.so:
ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
(SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18,
not stripped
/usr/lib64/libstdc++.so.6.0.13:
ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
(SYSV), dynamically linked, stripped
/lib64/libm-2.12.so:
ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
(SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18,
not stripped
/lib64/libgcc_s-4.4.6-20110824.so.1:
ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
(SYSV), dynamically linked, stripped
/lib64/libc-2.12.so:
ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
(SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18,
not stripped
/lib64/libutil-2.12.so:
ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
(SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18,
not stripped

Switching over to static linking seems to have fixed the problem, and
the code now runs on the compute nodes.

Thanks so much for the help!

Geoffrey