<html><body>
<p><font size="2" face="sans-serif">John,</font><br>
<br>
<font size="2" face="sans-serif">ionodes have 68 hwthreads available, however there are a few services running on the ionode that will take CPU. Core 0 takes PCIe interrupts (impacts performance on "cpus" 0-3) and bgvrnic takes cpus 66 and 67. I'm not sure how clang's OMP binds software threads to cpus - - maybe there's a way to avoid those cpus. </font><br>
<br>
<font size="2" face="sans-serif">I assume you're seeing this (lack of) performance only with the OpenMP builds? </font><br>
<br>
<font size="2" face="sans-serif">Tom</font><br>
<br>
<font size="2" face="sans-serif">Tom Gooding<br>
Senior Engineer / Blue Gene SW Lead / CAPI<br>
tgooding@us.ibm.com 507-253-0747<br>
</font><br>
<br>
<img width="16" height="16" src="cid:1__=08BBF635DFC316F68f9e8a93df938@us.ibm.com" border="0" alt="Inactive hide details for "Biddiscombe, John A." ---03/25/2014 08:58:04 AM---Dear people I'd had terrible performance of my app"><font size="2" color="#424282" face="sans-serif">"Biddiscombe, John A." ---03/25/2014 08:58:04 AM---Dear people I'd had terrible performance of my application which is intended to run on IO nodes, so</font><br>
<br>
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td width="1%"><img width="96" height="1" src="cid:2__=08BBF635DFC316F68f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<ul style="padding-left: 4pt"><font size="1" color="#5F5F5F" face="sans-serif">From:</font></ul>
</td><td width="100%"><img width="1" height="1" src="cid:2__=08BBF635DFC316F68f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<font size="1" face="sans-serif">"Biddiscombe, John A." <biddisco@cscs.ch></font></td></tr>
<tr valign="top"><td width="1%"><img width="96" height="1" src="cid:2__=08BBF635DFC316F68f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<ul style="padding-left: 4pt"><font size="1" color="#5F5F5F" face="sans-serif">To:</font></ul>
</td><td width="100%"><img width="1" height="1" src="cid:2__=08BBF635DFC316F68f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<font size="1" face="sans-serif">"llvm-bgq-discuss@lists.alcf.anl.gov" <llvm-bgq-discuss@lists.alcf.anl.gov></font></td></tr>
<tr valign="top"><td width="1%"><img width="96" height="1" src="cid:2__=08BBF635DFC316F68f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<ul style="padding-left: 4pt"><font size="1" color="#5F5F5F" face="sans-serif">Date:</font></ul>
</td><td width="100%"><img width="1" height="1" src="cid:2__=08BBF635DFC316F68f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<font size="1" face="sans-serif">03/25/2014 08:58 AM</font></td></tr>
<tr valign="top"><td width="1%"><img width="96" height="1" src="cid:2__=08BBF635DFC316F68f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<ul style="padding-left: 4pt"><font size="1" color="#5F5F5F" face="sans-serif">Subject:</font></ul>
</td><td width="100%"><img width="1" height="1" src="cid:2__=08BBF635DFC316F68f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<font size="1" face="sans-serif">[Llvm-bgq-discuss] clang on BGQ performance</font></td></tr>
<tr valign="top"><td width="1%"><img width="96" height="1" src="cid:2__=08BBF635DFC316F68f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<ul style="padding-left: 4pt"><font size="1" color="#5F5F5F" face="sans-serif">Sent by:</font></ul>
</td><td width="100%"><img width="1" height="1" src="cid:2__=08BBF635DFC316F68f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<font size="1" face="sans-serif">llvm-bgq-discuss-bounces@lists.alcf.anl.gov</font></td></tr>
</table>
<hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br>
<br>
<br>
<font size="2" face="sans-serif">Dear people</font><br>
<font size="2" face="sans-serif"> </font><br>
<font size="2" face="sans-serif">I’d had terrible performance of my application which is intended to run on IO nodes, so I’ve been poking around to try to find out what might be wrong.</font><br>
<font size="2" face="sans-serif"> </font><br>
<font size="2" face="sans-serif">Today I compiled a simple stream memory writing test from </font><a href="http://www.cs.virginia.edu/stream/FTP/Code/"><font size="2" color="#0000FF" face="sans-serif"><u>http://www.cs.virginia.edu/stream/FTP/Code/</u></font></a><font size="2" face="sans-serif"> </font><br>
<font size="2" face="sans-serif">I’ve run it using openmp threads up to 60, (because for reasons I don’t understand, the IO node only shows 15*4 threads)</font><br>
<font size="2" face="sans-serif"> </font><br>
<font size="2" face="sans-serif">The results for bgclang seem to echo what I’ve been finding with my code. I have not tested my stuff fully with gcc as I only just got that installed recently.</font><br>
<font size="2" face="sans-serif"> </font><br>
<font size="2" face="sans-serif">Any advice on what I might try to improve the bgclang numbers? in some cases gcc looks 2x better. </font><br>
<font size="2" face="sans-serif"> </font><br>
<font size="2" face="sans-serif">Note that my program doesn’t use openmp so I don’t directly care much about this particular example, but the trend mirrors what I’m seeing with HPX threads</font><br>
<font size="2" face="sans-serif"> </font><br>
<font size="2" face="sans-serif">thanks</font><br>
<font size="2" face="sans-serif"> </font><br>
<font size="2" face="sans-serif">JB</font><br>
<font size="2" face="Courier New"> </font><br>
<font size="2" face="Courier New">using bgclang version 20140309</font><br>
<font size="2" face="Courier New"> </font><br>
<font size="2" face="Courier New">export OMP_NUM_THREADS=1</font><br>
<font size="2" face="Courier New">-------------------------------------------------------------</font><br>
<font size="2" face="Courier New">Function Best Rate MB/s Avg time Min time Max time</font><br>
<font size="2" face="Courier New">Copy: 659.5 0.242635 0.242601 0.242724</font><br>
<font size="2" face="Courier New">Scale: 536.2 0.298403 0.298376 0.298535</font><br>
<font size="2" face="Courier New">Add: 828.5 0.289701 0.289669 0.289839</font><br>
<font size="2" face="Courier New">Triad: 711.8 0.337206 0.337151 0.337325</font><br>
<font size="2" face="Courier New">-------------------------------------------------------------</font><br>
<font size="2" face="Courier New">export OMP_NUM_THREADS=2</font><br>
<font size="2" face="Courier New">-------------------------------------------------------------</font><br>
<font size="2" face="Courier New">Function Best Rate MB/s Avg time Min time Max time</font><br>
<font size="2" face="Courier New">Copy: 1318.8 0.121335 0.121322 0.121360</font><br>
<font size="2" face="Courier New">Scale: 1072.5 0.149223 0.149185 0.149375</font><br>
<font size="2" face="Courier New">Add: 1657.2 0.144868 0.144823 0.145036</font><br>
<font size="2" face="Courier New">Triad: 1423.8 0.168611 0.168565 0.168755</font><br>
<font size="2" face="Courier New">-------------------------------------------------------------</font><br>
<font size="2" face="Courier New">export OMP_NUM_THREADS=4</font><br>
<font size="2" face="Courier New">-------------------------------------------------------------</font><br>
<font size="2" face="Courier New">Function Best Rate MB/s Avg time Min time Max time</font><br>
<font size="2" face="Courier New">Copy: 2636.4 0.060729 0.060688 0.060919</font><br>
<font size="2" face="Courier New">Scale: 2236.9 0.071580 0.071529 0.071774</font><br>
<font size="2" face="Courier New">Add: 3311.2 0.072555 0.072482 0.072750</font><br>
<font size="2" face="Courier New">Triad: 2845.6 0.084426 0.084341 0.084540</font><br>
<font size="2" face="Courier New">-------------------------------------------------------------</font><br>
<font size="2" face="Courier New">export OMP_NUM_THREADS=8</font><br>
<font size="2" face="Courier New">-------------------------------------------------------------</font><br>
<font size="2" face="Courier New">Function Best Rate MB/s Avg time Min time Max time</font><br>
<font size="2" face="Courier New">Copy: 5265.6 0.030446 0.030386 0.030614</font><br>
<font size="2" face="Courier New">Scale: 4468.1 0.035848 0.035809 0.036030</font><br>
<font size="2" face="Courier New">Add: 6611.9 0.036341 0.036298 0.036526</font><br>
<font size="2" face="Courier New">Triad: 5684.9 0.042258 0.042217 0.042420 </font><br>
<font size="2" face="Courier New">-------------------------------------------------------------</font><br>
<font size="2" face="Courier New">export OMP_NUM_THREADS=16</font><br>
<font size="2" face="Courier New">-------------------------------------------------------------</font><br>
<font size="2" face="Courier New">Function Best Rate MB/s Avg time Min time Max time</font><br>
<font size="2" face="Courier New">Copy: 9390.8 0.018977 0.017038 0.025704</font><br>
<font size="2" face="Courier New">Scale: 7688.2 0.021786 0.020811 0.029255</font><br>
<font size="2" face="Courier New">Add: 11985.7 0.020990 0.020024 0.028394</font><br>
<font size="2" face="Courier New">Triad: 10875.0 0.023131 0.022069 0.031470 </font><br>
<font size="2" face="Courier New">-------------------------------------------------------------</font><br>
<font size="2" face="Courier New">export OMP_NUM_THREADS=32</font><br>
<font size="2" face="Courier New">-------------------------------------------------------------</font><br>
<font size="2" face="Courier New">Function Best Rate MB/s Avg time Min time Max time</font><br>
<font size="2" face="Courier New">Copy: 15556.4 0.011463 0.010285 0.012906</font><br>
<font size="2" face="Courier New">Scale: 13361.1 0.013228 0.011975 0.014883</font><br>
<font size="2" face="Courier New">Add: 20438.0 0.012872 0.011743 0.014259</font><br>
<font size="2" face="Courier New">Triad: 18047.8 0.014270 0.013298 0.016016 </font><br>
<font size="2" face="Courier New">-------------------------------------------------------------</font><br>
<font size="2" face="Courier New">export OMP_NUM_THREADS=60</font><br>
<font size="2" face="Courier New">-------------------------------------------------------------</font><br>
<font size="2" face="Courier New">Function Best Rate MB/s Avg time Min time Max time</font><br>
<font size="2" face="Courier New">Copy: 11472.0 0.016570 0.013947 0.022287</font><br>
<font size="2" face="Courier New">Scale: 10145.1 0.019031 0.015771 0.028346</font><br>
<font size="2" face="Courier New">Add: 15317.9 0.018322 0.015668 0.025756</font><br>
<font size="2" face="Courier New">Triad: 14106.8 0.018959 0.017013 0.025986 </font><br>
<font size="2" face="Courier New">-------------------------------------------------------------</font><br>
<font size="2" face="Courier New"> </font><br>
<font size="2" face="Courier New">using GCC 4.8.2</font><br>
<font size="2" face="Courier New">export OMP_NUM_THREADS=1</font><br>
<font size="2" face="Courier New">-------------------------------------------------------------</font><br>
<font size="2" face="Courier New">Function Best Rate MB/s Avg time Min time Max time</font><br>
<font size="2" face="Courier New">Copy: 3534.4 0.045289 0.045270 0.045306</font><br>
<font size="2" face="Courier New">Scale: 1318.8 0.121390 0.121325 0.121632</font><br>
<font size="2" face="Courier New">Add: 1899.0 0.126403 0.126384 0.126428</font><br>
<font size="2" face="Courier New">Triad: 1910.3 0.125667 0.125637 0.125724</font><br>
<font size="2" face="Courier New">-------------------------------------------------------------</font><br>
<font size="2" face="Courier New">export OMP_NUM_THREADS=2</font><br>
<font size="2" face="Courier New">-------------------------------------------------------------</font><br>
<font size="2" face="Courier New">Function Best Rate MB/s Avg time Min time Max time</font><br>
<font size="2" face="Courier New">Copy: 7053.2 0.022716 0.022685 0.022744</font><br>
<font size="2" face="Courier New">Scale: 2613.9 0.061247 0.061211 0.061278</font><br>
<font size="2" face="Courier New">Add: 3794.3 0.063271 0.063252 0.063292</font><br>
<font size="2" face="Courier New">Triad: 3794.4 0.063288 0.063251 0.063449</font><br>
<font size="2" face="Courier New">-------------------------------------------------------------</font><br>
<font size="2" face="Courier New">export OMP_NUM_THREADS=4</font><br>
<font size="2" face="Courier New">-------------------------------------------------------------</font><br>
<font size="2" face="Courier New">Function Best Rate MB/s Avg time Min time Max time</font><br>
<font size="2" face="Courier New">Copy: 13999.4 0.011470 0.011429 0.011494</font><br>
<font size="2" face="Courier New">Scale: 5218.5 0.030683 0.030660 0.030729</font><br>
<font size="2" face="Courier New">Add: 7585.3 0.031647 0.031640 0.031681</font><br>
<font size="2" face="Courier New">Triad: 7583.4 0.031663 0.031648 0.031690</font><br>
<font size="2" face="Courier New">-------------------------------------------------------------</font><br>
<font size="2" face="Courier New">export OMP_NUM_THREADS=8</font><br>
<font size="2" face="Courier New">-------------------------------------------------------------</font><br>
<font size="2" face="Courier New">Function Best Rate MB/s Avg time Min time Max time</font><br>
<font size="2" face="Courier New">Copy: 25910.8 0.006205 0.006175 0.006233</font><br>
<font size="2" face="Courier New">Scale: 10432.9 0.015373 0.015336 0.015484</font><br>
<font size="2" face="Courier New">Add: 15130.5 0.015922 0.015862 0.016092</font><br>
<font size="2" face="Courier New">Triad: 15116.2 0.015971 0.015877 0.016139</font><br>
<font size="2" face="Courier New">-------------------------------------------------------------</font><br>
<font size="2" face="Courier New">export OMP_NUM_THREADS=16</font><br>
<font size="2" face="Courier New">-------------------------------------------------------------</font><br>
<font size="2" face="Courier New">Function Best Rate MB/s Avg time Min time Max time</font><br>
<font size="2" face="Courier New">Copy: 28433.5 0.005643 0.005627 0.005665</font><br>
<font size="2" face="Courier New">Scale: 20547.1 0.007831 0.007787 0.007860</font><br>
<font size="2" face="Courier New">Add: 27006.3 0.008922 0.008887 0.008948</font><br>
<font size="2" face="Courier New">Triad: 27758.5 0.008658 0.008646 0.008672</font><br>
<font size="2" face="Courier New">-------------------------------------------------------------</font><br>
<font size="2" face="Courier New">export OMP_NUM_THREADS=32</font><br>
<font size="2" face="Courier New">-------------------------------------------------------------</font><br>
<font size="2" face="Courier New">Function Best Rate MB/s Avg time Min time Max time</font><br>
<font size="2" face="Courier New">Copy: 28368.6 0.005673 0.005640 0.005742</font><br>
<font size="2" face="Courier New">Scale: 26302.8 0.006115 0.006083 0.006175</font><br>
<font size="2" face="Courier New">Add: 27164.4 0.008878 0.008835 0.008960</font><br>
<font size="2" face="Courier New">Triad: 27691.3 0.008702 0.008667 0.008744</font><br>
<font size="2" face="Courier New">-------------------------------------------------------------</font><br>
<font size="2" face="Courier New">export OMP_NUM_THREADS=60</font><br>
<font size="2" face="Courier New">-------------------------------------------------------------</font><br>
<font size="2" face="Courier New">Function Best Rate MB/s Avg time Min time Max time</font><br>
<font size="2" face="Courier New">Copy: 25715.2 0.008484 0.006222 0.012176</font><br>
<font size="2" face="Courier New">Scale: 22472.2 0.012979 0.007120 0.021724</font><br>
<font size="2" face="Courier New">Add: 25319.6 0.014178 0.009479 0.023234</font><br>
<font size="2" face="Courier New">Triad: 25591.9 0.013839 0.009378 0.023146</font><br>
<font size="2" face="Courier New">-------------------------------------------------------------</font><br>
<font size="2" face="Courier New"> </font><br>
<font size="2" face="Courier New"> </font><br>
<font size="2" face="Courier New"> </font><br>
<font size="2" face="Courier New">-- </font><br>
<font size="2" face="Courier New">John Biddiscombe, email:biddisco @.at.@ cscs.ch</font><br>
<a href="http://www.cscs.ch/"><font size="2" color="#0000FF" face="Courier New"><u>http://www.cscs.ch/</u></font></a><br>
<font size="2" face="Courier New">CSCS, Swiss National Supercomputing Centre | Tel: +41 (91) 610.82.07</font><br>
<font size="2" face="Courier New">Via Trevano 131, 6900 Lugano, Switzerland | Fax: +41 (91) 610.82.82</font><br>
<font size="2" face="sans-serif"> </font><tt><font size="2">_______________________________________________<br>
llvm-bgq-discuss mailing list<br>
llvm-bgq-discuss@lists.alcf.anl.gov<br>
</font></tt><tt><font size="2"><a href="https://lists.alcf.anl.gov/mailman/listinfo/llvm-bgq-discuss">https://lists.alcf.anl.gov/mailman/listinfo/llvm-bgq-discuss</a></font></tt><tt><font size="2"><br>
</font></tt><br>
<br>
</body></html>