[Llvm-bgq-discuss] Details behind MPI wrapper for bgclang++

[Hal Finkel]

----- Original Message -----
> From: "Jack Poulson" <jack.poulson at gmail.com>
> To: "Hal Finkel" <hfinkel at anl.gov>
> Cc: "Jeff Hammond" <jeff.science at gmail.com>, "Jeff Hammond" <jhammond at alcf.anl.gov>,
> llvm-bgq-discuss at lists.alcf.anl.gov
> Sent: Friday, March 1, 2013 5:18:43 PM
> Subject: Re: [Llvm-bgq-discuss] Details behind MPI wrapper for bgclang++
> 
> On Fri, Mar 1, 2013 at 1:57 PM, Hal Finkel < hfinkel at anl.gov > wrote:
> 
> 
> 
> 
> ----- Original Message -----
> > From: "Jeff Hammond" < jeff.science at gmail.com >
> > To: "Hal Finkel" < hfinkel at anl.gov >
> > Cc: "Jack Poulson" < jack.poulson at gmail.com >, "Jeff Hammond" <
> > jhammond at alcf.anl.gov >,
> > llvm-bgq-discuss at lists.alcf.anl.gov
> 
> > Sent: Friday, March 1, 2013 3:50:01 PM
> > Subject: Re: [Llvm-bgq-discuss] Details behind MPI wrapper for
> > bgclang++
> > 
> 
> > Should be deterministic. I've tested that before (a year ago maybe)
> > because of symmetric heap implementations. However, that's with
> > static
> > linkage.
> 
> For some reason I thought that he was linking dynamically, but that
> may not have been the case (at some point he deleted the executable,
> but before he did I ran ldd on it and it did not say that it was
> dynamically linked). Also, the bgclang wrapper script links
> statically by default.
> 
> Jack, is the problem itself deterministic?
> 
> 
> 
> -Hal
> 
> 
> 
> Sorry, I had a meeting that I had to run off to and am back now. 

What? You had something else to do?? ;)

> The
> problem is only non-deterministic because I am seeding from
> std::random_device. I could use a fixed unsigned integer if you
> would like. I then randomly instantiate a large number of sources
> throughout the my 2D domain and then apply the adjoint of a Radon
> transform on them (backprojection).

For testing, it will help if this is deterministic (I think)... or at least if you have the option to reproduce any particular run that will be useful.

> 
> I will happily try the flags you mentioned. Unfortunately, when I
> switched to using static linkage, I ran into a rather strange
> compilation problem:
> 
> [ 9%] Building CXX object
> CMakeFiles/Backproj-2d.dir/test/transform/Backproj-2d.cpp.o
> /home/projects/llvm/r175919-20130222/bin/bgclang++ -Wall -std=c++11
> -static -o
> CMakeFiles/Backproj-2d.dir/test/transform/Backproj-2d.cpp.o -c
> /home/poulson/dist-butterfly/test/transform/Backproj-2d.cpp
> /home/poulson/dist-butterfly/test/transform/Backproj-2d.cpp:8:10:
> fatal error:
> 'dist-butterfly.hpp' file not found
> #include "dist-butterfly.hpp"

This is the real problem. Are you missing a -I<include/path> or something like that?

FWIW, the bgclang++ wrapper script will pass -static by default, so unless you're specifically passing -shared or setting BGCLANG_STATIC_LINKING=no, then you were probably statically linking all along (assuming you were linking through the wrapper script as well). My fear was that the wrapper script was compiling assuming static linking (which is the default), but then CMake was invoking the linker directly assuming dynamic linking, and this was causing problems.

> ^
> 1 error generated.
> Assembler messages:
> Error: can't open /tmp/Backproj-2d-u4LbNL.s for reading: No such file
> or directory
> clang: error: assembler command failed with exit code 1 (use -v to
> see invocation)
> make[2]: ***
> [CMakeFiles/Backproj-2d.dir/test/transform/Backproj-2d.cpp.o] Error
> 1
> make[2]: Leaving directory
> `/gpfs/vesta-home/poulson/dist-butterfly/build/clang'
> make[1]: *** [CMakeFiles/Backproj-2d.dir/all] Error 2
> make[1]: Leaving directory
> `/gpfs/vesta-home/poulson/dist-butterfly/build/clang'
> make: *** [all] Error 2
> 
> Have any of you ever seen this type of issue before? It looks like
> invalid temporary files are being created. I'm not sure whether this
> is due to clang or CMake...(though I have good money on which one
> Jeff will choose!)

Like Jeff, I've had my fare share of problems with CMake ;)

Thanks again,
Hal

> 
> Jack
>