And what happens if you turn off OpenMP at compile time? I wonder if the LLVM OpenMP runtime just sucks too much right now on BGQ. Hal and I have looked at it enough that I would believe this explanation.<div><br>Jeff<br>
<br><div class="gmail_quote">On Tue, Mar 25, 2014 at 10:25 AM, Biddiscombe, John A. <span dir="ltr"><<a href="mailto:biddisco@cscs.ch" target="_blank">biddisco@cscs.ch</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-GB" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Tom<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Well, I’m not using openMP myself, I am using HPX which has its own thread scheduling (Thomas Heller reads this list and knows the details).<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">My best results so far have been obtained using a commandline which passes some location setting via hwloc<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p>bin/H5FDdsmRaw_bandwidth_rw --hpx:print-bind --hpx:threads=15 --hpx:bind=thread:0-14=socket:0-14 2048 Block 16777216 VirtualRAM<u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">here I’m attempting to place one thread on each of the 15 cpus that I can see with hwloc. Now If there’s a way I can avoid the IOnode services which are running
(for example there are always 2xbgvrnic processes running consuming 2x100% cpu - these are servicing io requests from CNK I assume).<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I was planning on asking that exact question to the IBM contacts here to see if they know how to skip the cores that the services are using (if just one). the
problem is that hwloc doesn’t seem to give the correct results either so I’m experimenting a bit.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I just looked in my email from last week and I see that for bgvrnic “</span><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">If there's no communication
with the compute nodes, they are "just" spin-waiting and shouldn't have an impact - unless you get processes scheduled onto the same core (i.e. CPU 56-59).”</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">.
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">you mention that they are running on 66/67 - is it possible to reconcile these numbers by taking into account a different counting method? (i.e, not including
some)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">JB<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Thomas Gooding [mailto:<a href="mailto:tgooding@us.ibm.com" target="_blank">tgooding@us.ibm.com</a>]
<br>
<b>Sent:</b> 25 March 2014 15:58<br>
<b>To:</b> Biddiscombe, John A.<br>
<b>Cc:</b> <a href="mailto:llvm-bgq-discuss@lists.alcf.anl.gov" target="_blank">llvm-bgq-discuss@lists.alcf.anl.gov</a><br>
<b>Subject:</b> Re: [Llvm-bgq-discuss] clang on BGQ performance<u></u><u></u></span></p>
</div>
</div><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<p style="margin-bottom:12.0pt"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">John,</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">ionodes have 68 hwthreads available, however there are a few services running on the ionode that will take CPU. Core 0 takes PCIe interrupts (impacts performance on "cpus" 0-3) and bgvrnic takes
cpus 66 and 67. I'm not sure how clang's OMP binds software threads to cpus - - maybe there's a way to avoid those cpus. </span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">I assume you're seeing this (lack of) performance only with the OpenMP builds? </span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Tom</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Tom Gooding<br>
Senior Engineer / Blue Gene SW Lead / CAPI<br>
<a href="mailto:tgooding@us.ibm.com" target="_blank">tgooding@us.ibm.com</a> <a href="tel:507-253-0747" value="+15072530747" target="_blank">507-253-0747</a><br>
</span><br>
<br>
<img border="0" width="16" height="16" src="cid:image001.gif@01CF4845.64476EA0" alt="Inactive hide details for "Biddiscombe, John A." ---03/25/2014 08:58:04 AM---Dear people I'd had terrible performance of my app"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#424282">"Biddiscombe,
John A." ---03/25/2014 08:58:04 AM---Dear people I'd had terrible performance of my application which is intended to run on IO nodes, so</span><u></u><u></u></p>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%">
<tbody>
<tr>
<td width="1%" valign="top" style="width:1.0%;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><img border="0" width="96" height="1" src="cid:image003.png@01CF4845.64476EA0"><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#5f5f5f">From:</span><u></u><u></u></p>
</td>
<td width="100%" valign="top" style="width:100.0%;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><img border="0" width="1" height="1" src="cid:image004.png@01CF4845.64476EA0"><br>
<span style="font-size:7.5pt;font-family:"Arial","sans-serif"">"Biddiscombe, John A." <<a href="mailto:biddisco@cscs.ch" target="_blank">biddisco@cscs.ch</a>></span><u></u><u></u></p>
</td>
</tr>
<tr>
<td width="1%" valign="top" style="width:1.0%;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><img border="0" width="96" height="1" src="cid:image003.png@01CF4845.64476EA0"><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#5f5f5f">To:</span><u></u><u></u></p>
</td>
<td width="100%" valign="top" style="width:100.0%;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><img border="0" width="1" height="1" src="cid:image004.png@01CF4845.64476EA0"><br>
<span style="font-size:7.5pt;font-family:"Arial","sans-serif"">"<a href="mailto:llvm-bgq-discuss@lists.alcf.anl.gov" target="_blank">llvm-bgq-discuss@lists.alcf.anl.gov</a>" <<a href="mailto:llvm-bgq-discuss@lists.alcf.anl.gov" target="_blank">llvm-bgq-discuss@lists.alcf.anl.gov</a>></span><u></u><u></u></p>
</td>
</tr>
<tr>
<td width="1%" valign="top" style="width:1.0%;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><img border="0" width="96" height="1" src="cid:image003.png@01CF4845.64476EA0"><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#5f5f5f">Date:</span><u></u><u></u></p>
</td>
<td width="100%" valign="top" style="width:100.0%;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><img border="0" width="1" height="1" src="cid:image004.png@01CF4845.64476EA0"><br>
<span style="font-size:7.5pt;font-family:"Arial","sans-serif"">03/25/2014 08:58 AM</span><u></u><u></u></p>
</td>
</tr>
<tr>
<td width="1%" valign="top" style="width:1.0%;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><img border="0" width="96" height="1" src="cid:image003.png@01CF4845.64476EA0"><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#5f5f5f">Subject:</span><u></u><u></u></p>
</td>
<td width="100%" valign="top" style="width:100.0%;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><img border="0" width="1" height="1" src="cid:image004.png@01CF4845.64476EA0"><br>
<span style="font-size:7.5pt;font-family:"Arial","sans-serif"">[Llvm-bgq-discuss] clang on BGQ performance</span><u></u><u></u></p>
</td>
</tr>
<tr>
<td width="1%" valign="top" style="width:1.0%;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><img border="0" width="96" height="1" src="cid:image003.png@01CF4845.64476EA0"><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#5f5f5f">Sent by:</span><u></u><u></u></p>
</td>
<td width="100%" valign="top" style="width:100.0%;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><img border="0" width="1" height="1" src="cid:image004.png@01CF4845.64476EA0"><br>
<span style="font-size:7.5pt;font-family:"Arial","sans-serif""><a href="mailto:llvm-bgq-discuss-bounces@lists.alcf.anl.gov" target="_blank">llvm-bgq-discuss-bounces@lists.alcf.anl.gov</a></span><u></u><u></u></p>
</td>
</tr>
</tbody>
</table>
<div class="MsoNormal">
<hr size="2" width="100%" noshade style="color:#8091a5" align="left">
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
<br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Dear people</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> </span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">I’d had terrible performance of my application which is intended to run on IO nodes, so I’ve been poking around to try to find out what might be wrong.</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> </span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Today I compiled a simple stream memory writing test from
</span><a href="http://www.cs.virginia.edu/stream/FTP/Code/" target="_blank"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">http://www.cs.virginia.edu/stream/FTP/Code/</span></a><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> </span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">I’ve run it using openmp threads up to 60, (because for reasons I don’t understand, the IO node only shows 15*4 threads)</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> </span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">The results for bgclang seem to echo what I’ve been finding with my code. I have not tested my stuff fully with gcc as I only just got that installed recently.</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> </span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Any advice on what I might try to improve the bgclang numbers? in some cases gcc looks 2x better.
</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> </span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Note that my program doesn’t use openmp so I don’t directly care much about this particular example, but the trend mirrors what I’m seeing with HPX threads</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> </span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">thanks</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> </span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">JB</span><br>
<span style="font-size:10.0pt;font-family:"Courier New""> </span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">using bgclang version 20140309</span><br>
<span style="font-size:10.0pt;font-family:"Courier New""> </span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">export OMP_NUM_THREADS=1</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">-------------------------------------------------------------</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Function Best Rate MB/s Avg time Min time Max time</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Copy: 659.5 0.242635 0.242601 0.242724</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Scale: 536.2 0.298403 0.298376 0.298535</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Add: 828.5 0.289701 0.289669 0.289839</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Triad: 711.8 0.337206 0.337151 0.337325</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">-------------------------------------------------------------</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">export OMP_NUM_THREADS=2</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">-------------------------------------------------------------</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Function Best Rate MB/s Avg time Min time Max time</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Copy: 1318.8 0.121335 0.121322 0.121360</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Scale: 1072.5 0.149223 0.149185 0.149375</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Add: 1657.2 0.144868 0.144823 0.145036</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Triad: 1423.8 0.168611 0.168565 0.168755</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">-------------------------------------------------------------</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">export OMP_NUM_THREADS=4</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">-------------------------------------------------------------</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Function Best Rate MB/s Avg time Min time Max time</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Copy: 2636.4 0.060729 0.060688 0.060919</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Scale: 2236.9 0.071580 0.071529 0.071774</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Add: 3311.2 0.072555 0.072482 0.072750</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Triad: 2845.6 0.084426 0.084341 0.084540</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">-------------------------------------------------------------</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">export OMP_NUM_THREADS=8</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">-------------------------------------------------------------</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Function Best Rate MB/s Avg time Min time Max time</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Copy: 5265.6 0.030446 0.030386 0.030614</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Scale: 4468.1 0.035848 0.035809 0.036030</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Add: 6611.9 0.036341 0.036298 0.036526</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Triad: 5684.9 0.042258 0.042217 0.042420
</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">-------------------------------------------------------------</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">export OMP_NUM_THREADS=16</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">-------------------------------------------------------------</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Function Best Rate MB/s Avg time Min time Max time</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Copy: 9390.8 0.018977 0.017038 0.025704</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Scale: 7688.2 0.021786 0.020811 0.029255</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Add: 11985.7 0.020990 0.020024 0.028394</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Triad: 10875.0 0.023131 0.022069 0.031470
</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">-------------------------------------------------------------</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">export OMP_NUM_THREADS=32</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">-------------------------------------------------------------</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Function Best Rate MB/s Avg time Min time Max time</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Copy: 15556.4 0.011463 0.010285 0.012906</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Scale: 13361.1 0.013228 0.011975 0.014883</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Add: 20438.0 0.012872 0.011743 0.014259</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Triad: 18047.8 0.014270 0.013298 0.016016
</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">-------------------------------------------------------------</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">export OMP_NUM_THREADS=60</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">-------------------------------------------------------------</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Function Best Rate MB/s Avg time Min time Max time</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Copy: 11472.0 0.016570 0.013947 0.022287</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Scale: 10145.1 0.019031 0.015771 0.028346</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Add: 15317.9 0.018322 0.015668 0.025756</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Triad: 14106.8 0.018959 0.017013 0.025986
</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">-------------------------------------------------------------</span><br>
<span style="font-size:10.0pt;font-family:"Courier New""> </span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">using GCC 4.8.2</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">export OMP_NUM_THREADS=1</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">-------------------------------------------------------------</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Function Best Rate MB/s Avg time Min time Max time</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Copy: 3534.4 0.045289 0.045270 0.045306</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Scale: 1318.8 0.121390 0.121325 0.121632</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Add: 1899.0 0.126403 0.126384 0.126428</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Triad: 1910.3 0.125667 0.125637 0.125724</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">-------------------------------------------------------------</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">export OMP_NUM_THREADS=2</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">-------------------------------------------------------------</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Function Best Rate MB/s Avg time Min time Max time</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Copy: 7053.2 0.022716 0.022685 0.022744</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Scale: 2613.9 0.061247 0.061211 0.061278</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Add: 3794.3 0.063271 0.063252 0.063292</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Triad: 3794.4 0.063288 0.063251 0.063449</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">-------------------------------------------------------------</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">export OMP_NUM_THREADS=4</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">-------------------------------------------------------------</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Function Best Rate MB/s Avg time Min time Max time</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Copy: 13999.4 0.011470 0.011429 0.011494</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Scale: 5218.5 0.030683 0.030660 0.030729</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Add: 7585.3 0.031647 0.031640 0.031681</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Triad: 7583.4 0.031663 0.031648 0.031690</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">-------------------------------------------------------------</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">export OMP_NUM_THREADS=8</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">-------------------------------------------------------------</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Function Best Rate MB/s Avg time Min time Max time</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Copy: 25910.8 0.006205 0.006175 0.006233</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Scale: 10432.9 0.015373 0.015336 0.015484</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Add: 15130.5 0.015922 0.015862 0.016092</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Triad: 15116.2 0.015971 0.015877 0.016139</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">-------------------------------------------------------------</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">export OMP_NUM_THREADS=16</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">-------------------------------------------------------------</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Function Best Rate MB/s Avg time Min time Max time</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Copy: 28433.5 0.005643 0.005627 0.005665</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Scale: 20547.1 0.007831 0.007787 0.007860</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Add: 27006.3 0.008922 0.008887 0.008948</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Triad: 27758.5 0.008658 0.008646 0.008672</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">-------------------------------------------------------------</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">export OMP_NUM_THREADS=32</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">-------------------------------------------------------------</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Function Best Rate MB/s Avg time Min time Max time</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Copy: 28368.6 0.005673 0.005640 0.005742</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Scale: 26302.8 0.006115 0.006083 0.006175</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Add: 27164.4 0.008878 0.008835 0.008960</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Triad: 27691.3 0.008702 0.008667 0.008744</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">-------------------------------------------------------------</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">export OMP_NUM_THREADS=60</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">-------------------------------------------------------------</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Function Best Rate MB/s Avg time Min time Max time</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Copy: 25715.2 0.008484 0.006222 0.012176</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Scale: 22472.2 0.012979 0.007120 0.021724</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Add: 25319.6 0.014178 0.009479 0.023234</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Triad: 25591.9 0.013839 0.009378 0.023146</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">-------------------------------------------------------------</span><br>
<span style="font-size:10.0pt;font-family:"Courier New""> </span><br>
<span style="font-size:10.0pt;font-family:"Courier New""> </span><br>
<span style="font-size:10.0pt;font-family:"Courier New""> </span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">-- </span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">John Biddiscombe, email:biddisco @.at.@ <a href="http://cscs.ch" target="_blank">cscs.ch</a></span><br>
<a href="http://www.cscs.ch/" target="_blank"><span style="font-size:10.0pt;font-family:"Courier New"">http://www.cscs.ch/</span></a><br>
<span style="font-size:10.0pt;font-family:"Courier New"">CSCS, Swiss National Supercomputing Centre | Tel: <a href="tel:%2B41%20%2891%29%20610.82.07" value="+41916108207" target="_blank">+41 (91) 610.82.07</a></span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">Via Trevano 131, 6900 Lugano, Switzerland | Fax: <a href="tel:%2B41%20%2891%29%20610.82.82" value="+41916108282" target="_blank">+41 (91) 610.82.82</a></span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> </span><tt><span style="font-size:10.0pt">_______________________________________________</span></tt><span style="font-size:10.0pt;font-family:"Courier New""><br>
<tt>llvm-bgq-discuss mailing list</tt><br>
<tt><a href="mailto:llvm-bgq-discuss@lists.alcf.anl.gov" target="_blank">llvm-bgq-discuss@lists.alcf.anl.gov</a></tt><br>
<tt><a href="https://lists.alcf.anl.gov/mailman/listinfo/llvm-bgq-discuss" target="_blank">https://lists.alcf.anl.gov/mailman/listinfo/llvm-bgq-discuss</a></tt><br>
<br>
</span><u></u><u></u></p>
</div></div></div>
</div>
</div>
<br>_______________________________________________<br>
llvm-bgq-discuss mailing list<br>
<a href="mailto:llvm-bgq-discuss@lists.alcf.anl.gov">llvm-bgq-discuss@lists.alcf.anl.gov</a><br>
<a href="https://lists.alcf.anl.gov/mailman/listinfo/llvm-bgq-discuss" target="_blank">https://lists.alcf.anl.gov/mailman/listinfo/llvm-bgq-discuss</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div>Jeff Hammond</div><div>Argonne Leadership Computing Facility</div><div>University of Chicago Computation Institute</div><div><a href="mailto:jhammond@alcf.anl.gov" target="_blank">jhammond@anl.gov</a> / <a href="mailto:jhammond@uchicago.edu" target="_blank">jhammond@uchicago.edu</a> / (630) 252-5381</div>
<div><a href="http://www.linkedin.com/in/jeffhammond" target="_blank">http://www.linkedin.com/in/jeffhammond</a></div><div><a href="https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond" target="_blank">https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond</a></div>
<div><br></div>
</div>