<html><head><style>body{font-family:Helvetica,Arial;font-size:13px}</style></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">Hi,</div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;"><br></div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">Super-brief summary:</div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;"><br></div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">The bug occurs at high optimization (-O3) with the vec_ld() or vec_st() functions when calling the (overloaded) version with single-precision arguments (i.e., float pointers).</div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;"><br></div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">This is reproducible even with a trivial piece of code that tries to load from single precision memory and then write out the contents of the (double precision) vector4doubles by storing to a double variable.</div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;"><br></div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">- Everything works fine with the double-precision functions, regardless of optimization</div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">- It works fine with clang in single precision without optimization </div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">- It works fine with xlc.</div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;"><br></div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">Cheers,</div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;"><br></div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">Erik</div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;"><br></div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;"><br></div><div class="airmail_ext_on" style="color:black">From: <span style="color:black">Mark Abraham</span> <a href="mailto:mark.j.abraham@gmail.com"><mark.j.abraham@gmail.com></a><br>Reply: <span style="color:black">Mark Abraham</span> <a href="mailto:mark.j.abraham@gmail.com"><mark.j.abraham@gmail.com>></a><br>Date: <span style="color:black">10 May 2015 at 16:04:51</span><br>To: <span style="color:black">Hal Finkel</span> <a href="mailto:hfinkel@anl.gov"><hfinkel@anl.gov>></a><br>Cc: <span style="color:black">llvm-bgq-discuss@lists.alcf.anl.gov</span> <a href="mailto:llvm-bgq-discuss@lists.alcf.anl.gov"><llvm-bgq-discuss@lists.alcf.anl.gov>></a>, <span style="color:black">Erik Lindahl</span> <a href="mailto:erik.lindahl@gmail.com"><erik.lindahl@gmail.com>></a><br>Subject: <span style="color:black"> Re: [Llvm-bgq-discuss] New bgclang nighty builds (and other updates) <br></span></div><br> <blockquote type="cite" class="clean_bq"><span><div><div></div><div>
<title></title>
<div dir="ltr">Hi,<br>
<br>
Bump. Erik Lindahl has also observed this on bgclang on JUQUEEN. Is
it helpful if I produce a small piece of code that reproduces some
of the issues on bgclang 3.6?<br>
<div><br></div>
<div>Mark</div>
<br>
<div class="gmail_quote">On Wed, Mar 25, 2015 at 4:16 PM Mark
Abraham <<a href="mailto:mark.j.abraham@gmail.com">mark.j.abraham@gmail.com</a>>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Tue, Mar 24, 2015 at 7:29 PM, Hal
Finkel <span dir="ltr"><<a href="mailto:hfinkel@anl.gov" target="_blank">hfinkel@anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<span>----- Original Message -----<br>
> From: "Mark Abraham" <<a href="mailto:mark.j.abraham@gmail.com" target="_blank">mark.j.abraham@gmail.com</a>><br>
> To: "Hal Finkel" <<a href="mailto:hfinkel@anl.gov" target="_blank">hfinkel@anl.gov</a>><br>
> Cc: <a href="mailto:llvm-bgq-discuss@lists.alcf.anl.gov" target="_blank">llvm-bgq-discuss@lists.alcf.anl.gov</a><br></span><span>>
Sent: Tuesday, March 24, 2015 12:35:20 PM<br>
> Subject: Re: [Llvm-bgq-discuss] New bgclang nighty builds (and
other updates)<br>
><br>
><br></span><span>> Hi Hal,<br>
><br>
><br>
> Thanks very much for the update & effort.<br>
><br>
<br></span>You're very welcome.<br>
<span><br>
><br>
> I tried out the default bgclang 3.6.0 on vesta, but found a
bunch of<br>
> the GROMACS SIMD-layer unit tests failing. These need correct
QPX<br>
> vector intrinsics available. From memory, things worked fine
with<br>
> bgclang in ~August last year, but I no longer have those
results. Is<br>
> there a simple way I can compile on vesta with older bgclang
to see<br>
> where a problem might lie? Otherwise / depending what I learn,
I'll<br>
> break out a debugger.<br>
<br></span>The old builds are still all installed. Just use the
MPI wrappers from:<br>
<br>
/home/projects/llvm/<whatever>/mpi/bgclang/bin -- I don't
know exactly which build you were using in August of last year,
r209570-20140527 maybe?<br></blockquote>
<div><br></div>
</div>
</div>
</div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>Yes, that was it, looking at some cruft in my former build
script. In any case, the latest GROMACS SIMD single-precision unit
tests pass on that old compiler version (3.5.0), and many of them
fail in a release build on the default 3.6.0 bgclang on vesta. For
some of the cases, it looks like some junk memory gets loaded,
somehow. A few tests pass, but no theme for passing or failing
tests leaps out at me. Double precision unit tests are OK,
though.</div>
<div><br></div>
<div>Naturally, in debug mode most of the tests pass. However, the
two I looked at in ddt were fine up until they called
vec_extract(x, 0) e.g. on lines <a href="https://github.com/gromacs/gromacs/blob/master/src/gromacs/simd/impl_ibm_qpx/impl_ibm_qpx.h#L315" target="_blank">https://github.com/gromacs/gromacs/blob/master/src/gromacs/simd/impl_ibm_qpx/impl_ibm_qpx.h#L315</a>
(where a SIMD vector of 1 2 3 1 was summed to 3, rather than
7) and <a href="https://github.com/gromacs/gromacs/blob/master/src/gromacs/simd/impl_ibm_qpx/impl_ibm_qpx.h#L462" target="_blank">https://github.com/gromacs/gromacs/blob/master/src/gromacs/simd/impl_ibm_qpx/impl_ibm_qpx.h#L462</a>
(where a SIMD vector dot product of the first 3 lanes returns a
garbage answer). So maybe that is a productive lead?</div>
<div><br></div>
<div>I couldn't inspect the disassembly in ddt, so I'm not sure
where we can take this from here, Hal. Tarball of test results
attached.</div>
<div><br></div>
<div>Thanks,</div>
<div><br></div>
<div>Mark</div>
</div>
</div>
</div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br></div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<span><font color="#888888"><br>
-Hal<br></font></span>
<div>
<div><br>
><br>
><br>
> Thanks,<br>
><br>
><br>
> Mark<br>
><br>
><br>
><br>
><br>
> On Fri, Mar 20, 2015 at 12:15 AM, Hal Finkel < <a href="mailto:hfinkel@anl.gov" target="_blank">hfinkel@anl.gov</a>
><br>
> wrote:<br>
><br>
><br>
> Hello everyone,<br>
><br>
> First, let me apologize to everyone, this is a few months
late...<br>
> but, hopefully, this will never be a problem again...<br>
><br>
> I now have a system setup which automatically pulls in
upstream<br>
> changes and tries to merge those with the bgclang-specific
patches,<br>
> and then builds the resulting suite of bgclang RPMs. When
this<br>
> succeeds, the RPMs should be posted automatically to:<br>
><br>
> <a href="http://www.mcs.anl.gov/~hfinkel/bgclang/" target="_blank">http://www.mcs.anl.gov/~hfinkel/bgclang/</a><br>
> (note that installing a build from here now also requires both
the<br>
> 'stage1' and 'stage2' RPMs as well)<br>
><br>
> The first such nightly build, r232720-20150319, has been
posted to<br>
> that page.<br>
><br>
> And, for the curious, the local repositories used for version
control<br>
> are now mirrored to github:<br>
><br>
> <a href="https://github.com/hfinkel/clang-bgq" target="_blank">https://github.com/hfinkel/clang-bgq</a><br>
> <a href="https://github.com/hfinkel/llvm-bgq" target="_blank">https://github.com/hfinkel/llvm-bgq</a><br>
> <a href="https://github.com/hfinkel/bgclang-aux" target="_blank">https://github.com/hfinkel/bgclang-aux</a><br>
> <a href="https://github.com/hfinkel/compiler-rt-bgq" target="_blank">https://github.com/hfinkel/compiler-rt-bgq</a><br>
> <a href="https://github.com/hfinkel/libcxx-bgq" target="_blank">https://github.com/hfinkel/libcxx-bgq</a><br>
> <a href="https://github.com/hfinkel/openmp-bgq" target="_blank">https://github.com/hfinkel/openmp-bgq</a><br>
> <a href="https://github.com/hfinkel/sleef-bgq" target="_blank">https://github.com/hfinkel/sleef-bgq</a><br>
><br>
> Compared to the latest "released" version (r220548-20141024),
the<br>
> most-recent nightly build does show some performance
regressions,<br>
> and there are a few things I've not even tested yet (LTO,
ASan,<br>
> etc.), but it also contains a number of bug fixes and
improvements,<br>
> so feel free to test on your applications.<br>
><br>
> One particular noteworthy improvement is that our OpenMP
runtime<br>
> library now has affinity support enabled. This means that all
of the<br>
> OpenMP 4 affinity features should work, and also that the
default<br>
> thread<->core bindings are now sensible.<br>
><br>
> The bgclang wrapper script no longer disables 'fast-isel'
instruction<br>
> selection at -O0, so your debug builds should now be faster
too.<br>
> Also, the automated vectorization of math functions using our
SLEEF<br>
> library adaptation is controlled using the new -fveclib flag
(so the<br>
> wrapper script contains -fveclib=SLEEF, and you can add<br>
> -fveclib=none to turn it off if desired for whatever
reason).<br>
><br>
> Also, the core QPX support has been contributed upstream
(although<br>
> not yet the Clang-level intrinsics support); so if you're
using LLVM<br>
> as a library, and want to just build from upstream sources
instead<br>
> of depending on the bgclang builds, that is now
possible.<br>
><br>
> Thanks again everyone, and please let me know if you
experience any<br>
> difficulties,<br>
> Hal<br>
><br>
> --<br>
> Hal Finkel<br>
> Assistant Computational Scientist<br>
> Leadership Computing Facility<br>
> Argonne National Laboratory<br>
> _______________________________________________<br>
> llvm-bgq-discuss mailing list<br>
> <a href="mailto:llvm-bgq-discuss@lists.alcf.anl.gov" target="_blank">llvm-bgq-discuss@lists.alcf.anl.gov</a><br>
> <a href="https://lists.alcf.anl.gov/mailman/listinfo/llvm-bgq-discuss" target="_blank">https://lists.alcf.anl.gov/mailman/listinfo/llvm-bgq-discuss</a><br>
><br>
><br>
<br>
--<br>
Hal Finkel<br>
Assistant Computational Scientist<br>
Leadership Computing Facility<br>
Argonne National Laboratory<br></div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div></div></span></blockquote> <div id="bloop_sign_1431267784243195136" class="bloop_sign"><div style="font-family:helvetica,arial;font-size:13px">-- <br><span style="font-family: 'helvetica Neue', helvetica;">Erik Lindahl <</span><a href="mailto:erik.lindahl@gmail.com" style="font-family: 'helvetica Neue', helvetica;">erik.lindahl@gmail.com</a><span style="font-family: 'helvetica Neue', helvetica;">> </span><br style="font-family: 'helvetica Neue', helvetica;"><span style="font-family: 'helvetica Neue', helvetica;">Professor of Biophysics, Dept. Biochemistry & Biophysics, Stockholm </span><span style="font-family: 'helvetica Neue', helvetica;">University </span><br style="font-family: 'helvetica Neue', helvetica;"><span style="font-family: 'helvetica Neue', helvetica;">Professor of Theoretical biophysics, Dept. Theoretical Physics, Royal Inst. </span><span style="font-family: 'helvetica Neue', helvetica;">Technology </span><br style="font-family: 'helvetica Neue', helvetica;"><span style="font-family: 'helvetica Neue', helvetica;">Science for Life Laboratory, Box 1031, 17121 Solna, Sweden</span></div></div></body></html>