[Llvm-bgq-discuss] bad elf
Geoffrey Irving
irving at naml.us
Sat Feb 2 21:03:57 CST 2013
On Sat, Feb 2, 2013 at 6:45 PM, Hal Finkel <hfinkel at anl.gov> wrote:
> ----- Original Message -----
>> From: "Geoffrey Irving" <irving at naml.us>
>> To: "Hal Finkel" <hfinkel at anl.gov>
>> Cc: llvm-bgq-discuss at lists.alcf.anl.gov
>> Sent: Saturday, February 2, 2013 7:30:42 PM
>> Subject: Re: [Llvm-bgq-discuss] bad elf
>>
>> On Fri, Feb 1, 2013 at 6:27 AM, Hal Finkel <hfinkel at anl.gov> wrote:
>> > ----- Original Message -----
>> >> From: "Geoffrey Irving" <irving at naml.us>
>> >> To: llvm-bgq-discuss at lists.alcf.anl.gov
>> >> Sent: Friday, February 1, 2013 1:31:46 AM
>> >> Subject: [Llvm-bgq-discuss] bad elf
>> >>
>> >> I switched to clang mpi wrapper compilers, but still get the bad
>> >> elf
>> >> error:
>> >>
>> >> ...
>> >> 2013-02-01 07:06:51.870 (FATAL) [0xfff829da210]
>> >> VST-02640-13751-32:22357:ibm.runjob.client.Job: could not start
>> >> job:
>> >> job failed to start
>> >> 2013-02-01 07:06:51.871 (FATAL) [0xfff829da210]
>> >> VST-02640-13751-32:22357:ibm.runjob.client.Job: Load failed on
>> >> Q02-I7-J00: Application executable ELF header contains invalid
>> >> value,
>> >> errno 8 Exec format error
>> >
>> > This almost always indicates that you'd somehow ended up linking to
>> > the wrong c-runtime-library startup routines. You've picked up the
>> > login-node versions and not the compute-node versions. I don't
>> > know whether or not those get included in the shared libraries as
>> > well.
>> >
>> > If you run the 'file' utility on the various .so files and the main
>> > executable it should report a 'linux version' associated with the
>> > file. Which version does it say? Are they all the same?
>>
>> It says "for GNU/Linux 2.6.18" for some of them
>
> Unfortunately, I think that this is indicative of the problem. Furthermore, the libraries that say this seem to be coming from /lib64. Nothing that runs on the compute node should come from there (or /lib*, /usr/lib*, etc.). Do you know why those files are in your list?
No, I've ruled out pollution from my other libraries, since I still
see /lib64/libc.so.6 if I make an .so that links against nothing
explicitly. Specifically,
/home/projects/llvm/mpi/clang.legacy/bin/mpiclang -o
build/compute/release/core/libother_core.so -shared -shared
-Wl,-rpath=/gpfs/vesta_home/irving/otherlab/other/install/compute/release/lib
build/compute/release/core/module.os <MANY_OTHER_OS_FILES>
-Linstall/compute/release/lib
vesta:other% echo $LD_LIBRARY_PATH
vesta:other% ldd build/compute/release/core/libother_core.so
linux-vdso64.so.1 => (0x00000fff90ca0000)
libmpich.so.3 => not found
libopa.so.1 => not found
libmpl.so.1 => not found
librt.so.1 => /lib64/librt.so.1 (0x00000fff90a30000)
libdl.so.2 => /lib64/libdl.so.2 (0x00000fff90a00000)
libpami.so => not found
libpthread.so.0 => /lib64/libpthread.so.0 (0x00000fff909c0000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00000fff90820000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00000fff907f0000)
libc.so.6 => /lib64/libc.so.6 (0x00000fff90600000)
/lib64/ld64.so.1 (0x000000004d180000)
libm.so.6 => /lib64/libm.so.6 (0x00000fff90500000)
Thus, it seems to be intrinsic to mpiclang. Here's the full link command:
/home/projects/llvm/mpi/clang.legacy/bin/mpiclang -o
build/compute/release/core/libother_core.so -shared -shared
-Wl,-rpath=/gpfs/vesta_home/irving/otherlab/other/install/compute/release/lib
build/compute/release/core/module.os
build/compute/release/core/openmesh/TriMesh.os
build/compute/release/core/openmesh/decimate.os
build/compute/release/core/openmesh/module.os
build/compute/release/core/openmesh/triangulator.os
build/compute/release/core/openmesh/visualize.os
build/compute/release/core/python/Buffer.os
build/compute/release/core/python/Class.os
build/compute/release/core/python/ClassTest.os
build/compute/release/core/python/ExceptionValue.os
build/compute/release/core/python/Object.os
build/compute/release/core/python/Ref.os
build/compute/release/core/python/repr.os
build/compute/release/core/python/exceptions.os
build/compute/release/core/python/from_python.os
build/compute/release/core/python/function.os
build/compute/release/core/python/module.os
build/compute/release/core/python/numpy.os
build/compute/release/core/python/wrap_constructor.os
build/compute/release/core/python/wrap_field.os
build/compute/release/core/python/wrap_function.os
build/compute/release/core/python/wrap_method.os
build/compute/release/core/python/wrap_property.os
build/compute/release/core/python/try_convert.os
build/compute/release/core/mesh/PolygonMesh.os
build/compute/release/core/mesh/SegmentMesh.os
build/compute/release/core/mesh/TriangleMesh.os
build/compute/release/core/mesh/TriangleSubdivision.os
build/compute/release/core/mesh/module.os
build/compute/release/core/array/Array.os
build/compute/release/core/array/Array2d.os
build/compute/release/core/array/NdArray.os
build/compute/release/core/array/NestedArray.os
build/compute/release/core/array/RawArray.os
build/compute/release/core/array/module.os
build/compute/release/core/force/ConstitutiveModel.os
build/compute/release/core/force/DiagonalizedIsotropicStressDerivative.os
build/compute/release/core/force/EtherDrag.os
build/compute/release/core/force/FiniteVolume.os
build/compute/release/core/force/Force.os
build/compute/release/core/force/Gravity.os
build/compute/release/core/force/LinearBendingElements.os
build/compute/release/core/force/LinearFiniteVolume.os
build/compute/release/core/force/LinearFiniteVolumeHex.os
build/compute/release/core/force/NeoHookean.os
build/compute/release/core/force/PlasticityModel.os
build/compute/release/core/force/RotatedLinear.os
build/compute/release/core/force/Springs.os
build/compute/release/core/force/StrainMeasure.os
build/compute/release/core/force/StrainMeasureHex.os
build/compute/release/core/force/module.os
build/compute/release/core/geometry/AnalyticImplicit.os
build/compute/release/core/geometry/BoxScalar.os
build/compute/release/core/geometry/BoxTree.os
build/compute/release/core/geometry/BoxVector.os
build/compute/release/core/geometry/Cylinder.os
build/compute/release/core/geometry/FrameImplicit.os
build/compute/release/core/geometry/Implicit.os
build/compute/release/core/geometry/ParticleTree.os
build/compute/release/core/geometry/Plane.os
build/compute/release/core/geometry/Ray.os
build/compute/release/core/geometry/Segment2d.os
build/compute/release/core/geometry/Segment3d.os
build/compute/release/core/geometry/SimplexTree.os
build/compute/release/core/geometry/Sphere.os
build/compute/release/core/geometry/Triangle2d.os
build/compute/release/core/geometry/Triangle3d.os
build/compute/release/core/geometry/mass_properties.os
build/compute/release/core/geometry/module.os
build/compute/release/core/geometry/platonic.os
build/compute/release/core/geometry/polygon.os
build/compute/release/core/image/ExrFile.os
build/compute/release/core/image/Image.os
build/compute/release/core/image/JpgFile.os
build/compute/release/core/image/MovFile.os
build/compute/release/core/image/PngFile.os
build/compute/release/core/image/color_utils.os
build/compute/release/core/image/module.os
build/compute/release/core/math/hash.os
build/compute/release/core/math/module.os
build/compute/release/core/math/numeric_limits.os
build/compute/release/core/math/uint128.os
build/compute/release/core/random/Random.os
build/compute/release/core/random/Sobol.os
build/compute/release/core/random/counter.os
build/compute/release/core/random/module.os
build/compute/release/core/random/permute.os
build/compute/release/core/solver/brent.os
build/compute/release/core/solver/module.os
build/compute/release/core/solver/powell.os
build/compute/release/core/solver/pattern_max.os
build/compute/release/core/structure/Tuple.os
build/compute/release/core/utility/CloneArray.os
build/compute/release/core/utility/DebugPrint.os
build/compute/release/core/utility/Log.os
build/compute/release/core/utility/LogEntry.os
build/compute/release/core/utility/LogScope.os
build/compute/release/core/utility/ProgressIndicator.os
build/compute/release/core/utility/base64.os
build/compute/release/core/utility/curry.os
build/compute/release/core/utility/debug.os
build/compute/release/core/utility/format.os
build/compute/release/core/utility/interrupts.os
build/compute/release/core/utility/module.os
build/compute/release/core/utility/path.os
build/compute/release/core/utility/process.os
build/compute/release/core/utility/resource.os
build/compute/release/core/utility/safe_bool.os
build/compute/release/core/utility/stream.os
build/compute/release/core/utility/time.os
build/compute/release/core/value/Action.os
build/compute/release/core/value/Compute.os
build/compute/release/core/value/Prop.os
build/compute/release/core/value/ConstValue.os
build/compute/release/core/value/Listen.os
build/compute/release/core/value/PropManager.os
build/compute/release/core/value/Value.os
build/compute/release/core/value/module.os
build/compute/release/core/vector/Frame.os
build/compute/release/core/vector/Matrix.os
build/compute/release/core/vector/Matrix2x2.os
build/compute/release/core/vector/Matrix3x2.os
build/compute/release/core/vector/Matrix3x3.os
build/compute/release/core/vector/Matrix4x4.os
build/compute/release/core/vector/Register.os
build/compute/release/core/vector/Rotation.os
build/compute/release/core/vector/SolidMatrix.os
build/compute/release/core/vector/SparseMatrix.os
build/compute/release/core/vector/SymmetricMatrix3x3.os
build/compute/release/core/vector/TetrahedralGroup.os
build/compute/release/core/vector/Vector.os
build/compute/release/core/vector/blas.os
build/compute/release/core/vector/group.os
build/compute/release/core/vector/module.os
build/compute/release/core/vector/test_matrix.os
-Linstall/compute/release/lib
This simpler test case may be useful:
vesta:other% /home/projects/llvm/mpi/clang.legacy/bin/mpiclang -o
blah.so -shared
vesta:other% ldd blah.so
linux-vdso64.so.1 => (0x00000fff90e00000)
libmpich.so.3 => not found
libopa.so.1 => not found
libmpl.so.1 => not found
librt.so.1 => /lib64/librt.so.1 (0x00000fff90ca0000)
libdl.so.2 => /lib64/libdl.so.2 (0x00000fff90c70000)
libpami.so => not found
libpthread.so.0 => /lib64/libpthread.so.0 (0x00000fff90c30000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00000fff90a90000)
libc.so.6 => /lib64/libc.so.6 (0x00000fff908a0000)
/lib64/ld64.so.1 (0x0000000053160000)
libm.so.6 => /lib64/libm.so.6 (0x00000fff907a0000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00000fff90770000)
For more completeness, here's my environment:
vesta:other% export
declare -x ARCH="linux-rhel_6-ppc64"
declare -x BASH_COMPLETION="/home/irving/src/bin/bash_completion/bash_completion"
declare -x BGQDRV="V1R1M2"
declare -x CLASSPATH="/usr/share/java/jdom.jar:/dbhome/db2cat/sqllib/java/db2java.zip:/dbhome/db2cat/sqllib/java/db2jcc.jar:/dbhome/db2cat/sqllib/java/sqlj.zip:/dbhome/db2cat/sqllib/function:/dbhome/db2cat/sqllib/java/db2jcc_license_cu.jar:."
declare -x COMM_BASE="/bgsys/drivers/ppcfloor/comm"
declare -x COMM_SELECT="gcc"
declare -x CVSIGNORE=".sconsign *output* log*.txt Thumbs.db *.pyc"
declare -x CVSROOT=":ext:physbam:/physbam_repository"
declare -x CVS_RSH="ssh"
declare -x DB2INSTANCE="db2cat"
declare -x DEFAULT_USER="irving"
declare -x DISTCC_DIR="/tmp/irving-distcc"
declare -x DUCK="/home/irving/duck"
declare -x EDITOR="/usr/bin/vim"
declare -x FIGNORE="svn:.pyc:.sln:.vcproj:.hi:.o:ite_scons:.DS_Store"
declare -x GHCRTS=""
declare -x GOODFONT="-misc-fixed-medium-r-semicondensed-*-13-*-*-*-*-*-*-*"
declare -x G_BROKEN_FILENAMES="1"
declare -x HISTCONTROL="ignoredups"
declare -x HISTSIZE="1000"
declare -x HOME="/home/irving"
declare -x HOSTNAME="vestalac1"
declare -x IBMCMP_INCLUDE="/soft/compilers/ibmcmp-nov2012/xlsmp/bg/3.1/include"
declare -x IBM_CCMP_DIR="/soft/compilers/ibmcmp-nov2012/vac/bg/12.1"
declare -x IBM_CPPCMP_DIR="/soft/compilers/ibmcmp-nov2012/vacpp/bg/12.1"
declare -x IBM_FCMP_DIR="/soft/compilers/ibmcmp-nov2012/xlf/bg/14.1"
declare -x IBM_MAIN_DIR="/soft/compilers/ibmcmp-nov2012"
declare -x IGNOREEOF="1000"
declare -x INCLUDE_PATH="/bgsys/drivers/ppcfloor/comm/gcc/include"
declare -x JAVA_HOME="/usr/lib/jvm/jre-1.6.0-ibm.ppc64"
declare -x JOBS="24"
declare -x LANG="en_US.UTF-8"
declare -x LD_LIBRARY_PATH=""
declare -x LESSOPEN="|/usr/bin/lesspipe.sh %s"
declare -x LM_LICENSE_FILE=""
declare -x LOGNAME="irving"
declare -x LS_COLORS=""
declare -x MAIL="/var/spool/mail/irving"
declare -x MANPATH=""
declare -x OBJECT_MODE="64"
declare -x OCAMLRUNPARAM="b1"
declare -x OLDPWD="/home/irving/otherlab/other/core"
declare -x OTHER="/home/irving/otherlab/other"
declare -x P4CONFIG=".p4config"
declare -x P4DIFF="diff"
declare -x P4EDITOR="vim"
declare -x P4MERGEDIFF="meld"
declare -x PATH="/bin:/usr/lib64/qt-3.3/bin:/home/irving/bin:/soft/environment/softenv-1.6.2/bin:/usr/bin:/usr/local/bin:/bgsys/drivers/V1R1M2/ppc64/gnu-linux/bin:/bgsys/drivers/V1R1M2/ppc64/hlcs/bin:/soft/debuggers/scripts/bin:/soft/compilers/ibmcmp-nov2012/vac/bg/12.1/bin:/soft/compilers/ibmcmp-nov2012/vacpp/bg/12.1/bin:/soft/compilers/ibmcmp-nov2012/xlf/bg/14.1/bin:/bgsys/drivers/ppcfloor/comm/gcc/bin:/bgsys/drivers/ppcfloor/bin:/bgsys/drivers/ppcfloor/sbin:/dbhome/db2cat/sqllib/bin:/dbhome/db2cat/sqllib/adm:/opt/bin:/sbin:/usr/sbin:/home/irving/src/bin:/home/irving/otherlab/other/core/bin:/home/irving/.local/bin"
declare -x PERL5LIB=""
declare -x PETSC_ARCH="darwin10.6.0-cxx-debug"
declare -x PETSC_DIR="/usr/local/petsc"
declare -x PHYSBAM=""
declare -x PLATFORM="nocona"
declare -x PS1="\$(if [ \`whoami\` != \"irving\" ]; then echo \\u@;
fi)vesta:\\W% "
declare -x PWD="/home/irving/otherlab/other"
declare -x PYTHONPATH="/home/irving:/home/irving/otherlab/other/..:/home/irving/otherlab/other/install/flavor/lib:/home/irving/.local/lib/python2.6/site-packages"
declare -x PYTHONSTARTUP="/home/irving/.pythonrc"
declare -x PYTHONVERSION="2.6"
declare -x QTDIR="/usr/lib64/qt-3.3"
declare -x QTINC="/usr/lib64/qt-3.3/include"
declare -x QTLIB="/usr/lib64/qt-3.3/lib"
declare -x RAW_LS="ls"
declare -x RAW_RMDIR="rmdir"
declare -x RAW_WHICH="/usr/bin/which"
declare -x SHELL="/bin/bash"
declare -x SHLVL="1"
declare -x SOFTENV_ALIASES="/soft/environment/softenv-1.6.2/etc/softenv-aliases.sh"
declare -x SOFTENV_LOAD="/soft/environment/softenv-1.6.2/etc/softenv-load.sh"
declare -x SSH_AUTH_SOCK="/tmp/ssh-wGRgD20792/agent.20792"
declare -x SSH_CLIENT="71.198.59.62 45420 22"
declare -x SSH_CONNECTION="71.198.59.62 45420 140.221.69.5 22"
declare -x SSH_TTY="/dev/pts/50"
declare -x TAO_DIR="/home/irving/otherlab/download/tao-1.10.1"
declare -x TERM="xterm-256color"
declare -x TEXINPUTS=""
declare -x USER="irving"
declare -x VALGRIND_OPTS="
--suppressions=/home/irving/src/etc/misc.supp
--suppressions=/home/irving/src/etc/mkl.supp
--suppressions=/home/irving/src/etc/opengl.supp
--suppressions=/home/irving/src/etc/python.supp"
declare -x VISUAL="/usr/bin/vim"
declare -x WHATAMI="linux-rhel_6-ppc64"
declare -x XLC_USR_CONFIG="/soft/compilers/ibmcmp-nov2012/vacpp/bg/12.1/etc/V1R1M2.vac.cfg.rhel6.2.gcc446"
declare -x XLF_USR_CONFIG="/soft/compilers/ibmcmp-nov2012/xlf/bg/14.1/etc/V1R1M2.xlf.cfg.rhel6.2.gcc446"
declare -x XL_DEFMSG="/soft/compilers/ibmcmp-nov2012/xlf/bg/14.1/msg/en_US"
Thanks,
Geoffrey
More information about the llvm-bgq-discuss
mailing list