[Llvm-bgq-discuss] address sanitizer

Tue Jul 9 15:35:57 CDT 2013

Hi Hal,

Could you say a bit more on the need for 3 shadow regions?  Is it because
the address space is discontiguous, or are you referring to a different
limitation?  If discontiguous, how does it work under normal Linux that has
discontiguous ranges for executable, dynamic libraries, heap, and stack?

I am not sure how the address sanitizer works internally, but would it be
possible to grab the shadow memory from the front, by being the first
caller of 'brk'?

Is the dynamic library requirement a statement of the llvm runtime build
(e.g., missing "libanalyzer.a" library)?  Or is the analyzer runtime
effectively an "ELF interpreter"?  Or some other requirement?

Thanks!
Tom

Tom Gooding
Senior Engineer / Blue Gene Kernels
507-253-0747  (internal:  553-0747)

|------------>
| From:      |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |Hal Finkel <hfinkel at anl.gov>                                                                                                                      |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| To:        |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |llvm-bgq-discuss at lists.alcf.anl.gov,                                                                                                              |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Date:      |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |07/06/2013 04:49 PM                                                                                                                               |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Subject:   |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |[Llvm-bgq-discuss] address sanitizer                                                                                                              |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Sent by:   |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |llvm-bgq-discuss-bounces at lists.alcf.anl.gov                                                                                                       |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|

Hello everyone,

One of my motivations for working on LLVM/Clang for the BG/Q was to enable
use of the sanitizer debugging projects on the BG/Q. Over the last couple
of weeks, I've taken initial steps toward that goal. For those of you who
don't know, address sanitizer is a tool for detecting memory allocation and
use errors: use-after-free, double-free stack and heap overruns, etc.
Because this works using instrumentation, address sanitizer, in general,
has much lower overhead than tools like valgrind (which reply on processor
emulation). This should make it feasible to use address sanitizer to debug
memory misuse errors on the Q, including those which only show up at scale.

To use this feature, pass -fsanitize=address to the compiler (when
compiling and linking). I highly recommend using at least -O1 (if not -O3)
and -g as well.

Address sanitizer requires dynamic linking. When you provide
-fsanitize=address, the wrapper script will automatically switch into
non-static-linking mode. I've made a number of improvements to the wrapper
scripts (both bgclang and the mpi scripts), and small fix to Clang that
applies to dynamically linking in C++11 mode, to better support dynamic
linking. In short, hopefully this will now all *just work*.

Because of a limitation of the LLVM PowerPC backend (it cannot do dynamic
stack realignment yet), the ability of the current build to detect stack
overruns is limited. I'll make the necessary improvements in the PowerPC
backend soon, and then the ability to detect stack overruns will be the
same as on other platforms.

Some details on overheads: address sanitizer introduces runtime and memory
overheads in several different ways. First, the runtime allocates a
'shadow' memory region which it uses to record state information on
allocated memory regions. As I have it configured, this uses 1 byte of
'shadow' memory for every 32 bytes. The 'normal' upstream address sanitizer
uses a simple mapping between addresses and 'shadow' bytes. Unfortunately,
due to limitations imposed by CNK on virtual memory use and mapping, I've
had to divide this shadow region into three distinct pieces (one for
segments from the executable image, one for the heap/stack, and one for
things in /dev/shm). Selecting between these regions introduces an extra
penalty from the instrumentation. Nevertheless, the additional overhead
does not seem too bad. Also, because of CNK restrictions, this shadow area
needs to be allocated somewhere in the heap/stack region. I'm currently
placing it in the middle, so if your application c
 urrently uses more than 8GB of stack in c1 mode, for example, then this
configuration won't work for you. 'red zones' are also allocated around
every heap allocation and stack variable, further increasing the memory
overhead. I've tried this on HACC, and the runtime slowdown on various
stages was between 3x and 50x. If you're code spends 95% of its time in
dgemm, however, you'll probably not notice anything ;)

For those maintaining their own installs:
As of yet, I've not bumped the version number of the install (I'll do that
the next time I rebase). Nevertheless, there are obviously new parts of the
patchset, build scripts, etc. You'll find a new archive (-v2) on the trac
page https://trac.alcf.anl.gov/projects/llvm-bgq -- please note: do not
checkout compiler-rt into the llvm/projects subdirectory as you would for a
normal build (and as specified on the clang web page). The compiler-rt
library needs to be cross-compiled using the bgclang-wrapped compiler. Just
checkout compiler-rt into its own top-level directory, create an empty
build directory for it, and use the build script in the archive (after
adjusting paths as appropriate).

Happy bug hunting! (and please let me know if you encounter any problems).

 -Hal

--
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory
_______________________________________________
llvm-bgq-discuss mailing list
llvm-bgq-discuss at lists.alcf.anl.gov
https://lists.alcf.anl.gov/mailman/listinfo/llvm-bgq-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alcf.anl.gov/pipermail/llvm-bgq-discuss/attachments/20130709/44f3c475/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://lists.alcf.anl.gov/pipermail/llvm-bgq-discuss/attachments/20130709/44f3c475/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ecblank.gif
Type: image/gif
Size: 45 bytes
Desc: not available
URL: <http://lists.alcf.anl.gov/pipermail/llvm-bgq-discuss/attachments/20130709/44f3c475/attachment-0001.gif>