<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Hello, Christoph,<br>
<br>
We have already discussed it some time in the past: our distribution
does not have examples directory, we only have bin, doc, include, and
lib.<br>
<br>
Could you please send as an example tared directory?<br>
<br>
But you have answered to another question I had: IHPCT_BASE should
point to the root of the HPCT distribution tree. It illuminates the
problem with peekperf, which does not complain about this variable
anymore. However, it does not give a source anyway.<br>
<br>
Second problem: your hpm installation does not correspond to ours: if I
link against -L/bgsys/local/ihpct-2.2.2/lib -llicense -lhpm, I will get
unresolved dlopen and dlsym symbols. <br>
<br>
Third problem: you must be using gnu compiler, we are using IBM XL
compiler, -dynamic is not part of XL compiler options. If I am using
analogous -qnostaticlink, it does not help much.<br>
<br>
Forth problem: if I finally compiled it with provided mirror.so:<br>
<br>
mpixlf77 -g -o hpct-hcpm hpct-hcpm.o -L/home/morozov/soft/hpct_bgp/lib/
-lhpm -llicense -qnostaticlink
-L/bgsys/drivers/ppcfloor/gnu-linux/powerpc-bgp-linux/lib -ldl<br>
<br>
and run with LD_LIBRARY_PATH=$IHPCT_BASE, I am getting crash. Teh core
file locates the crush in utils/hpm_dlfcn.c:56. So, I assume it is an
hpm issue.<br>
<br>
Basically, I am trying to make the version I-Hsin was sending us two
months ago. I know, that previous versions were working just find until
this dlopen call has appeared.<br>
<br>
Best, and thank you very much for your response,<br>
<br>
Vitali<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
Christoph Pospiech wrote:
<blockquote cite="mid:200809041543.09215.Christoph.Pospiech@de.ibm.com"
type="cite">
<pre wrap="">On Wednesday 03 September 2008, Vitali A. Morozov wrote:
</pre>
<blockquote type="cite">
<pre wrap="">8) HPM part does not work: core file at runtime. Please, give us example
of 1) how to compile HPM-instrumented program, 2) what is the result of
the run.
</pre>
</blockquote>
<pre wrap=""><!---->
Sorry - long post and technical content. Please feel free to ask if something
is unclear.
The following commands have been executed on another customer site, at
Forschungszentrum Jülich. They use a home grown script "llrun" to set up
LoadL scripts and start the job. If interested, I can provide a contact to
the authors of this script. Otherwise, replace every occurrence of "llrun"
below with "mpirun -partition <whatever>".
I used the same software as has been delivered to ANL. I have also used
hpct/examples/hpm, which I am not sure about having been shipped to ANL.
[pospiech@jugene1 ~]$ cd hpct_test/
[pospiech@jugene1 hpct_test]$ cp -r /bgsys/local/ihpct-2.2.2/examples/ .
[pospiech@jugene1 hpct_test]$ cd examples/hpm
[pospiech@jugene1 hpm]$ export IHPCT_BASE=/bgsys/local/ihpct-2.2.2
[pospiech@jugene1 hpm]$ make ARCH=bgp_linux swim_mpi
/bgsys/drivers/ppcfloor/comm/bin/mpif90 -c -O3 -g -I/bgsys/local/ihpct-2.2.2/include -x
f77-cpp-input -std=gnu -Wp,-DHPM swim_mpi.f
/bgsys/drivers/ppcfloor/comm/bin/mpif90 -O3 swim_mpi.o -o
swim_mpi -dynamic -L/bgsys/local/ihpct-2.2.2/lib -llicense -lhpm
[pospiech@jugene1 hpm]$ llrun -np 16 -mode VN -env
LD_LIBRARY_PATH=$IHPCT_BASE/lib -env HPM_UNIQUE_FILE_NAME=yes ./swim_mpi
Creating LoadLeveler Job
Submiting LoadLeveler Interactive Job
Wait for job jugene1b.42567.0 to be started:......
.
SPEC benchmark 102.swim
NUMBER OF POINTS IN THE X DIRECTION 512
NUMBER OF POINTS IN THE Y DIRECTION 512
GRID SPACING IN THE X DIRECTION 25000.
GRID SPACING IN THE Y DIRECTION 25000.
TIME STEP 20.
TIME FILTER PARAMETER 0.001
NUMBER OF ITERATIONS 102
wALLclock time= 0.924954108235294
--------------------------------------------------------------------------------------
remarks and CAVEATS
--------------------------------------------------------------------------------------
The most important part of the llrun (or mpirun -partition <whatever>) command
are
-env LD_LIBRARY_PATH=$IHPCT_BASE/lib
(otherwise you get "cannot load swim_mpi)
-env HPM_UNIQUE_FILE_NAME=yes
(otherwise all tasks write to the same file and you get a pile of
garbage).
The above example writes files *.hpm and *.viz (for viewing with peekperf),
one for each MPI task. These might be many files. To only reduce it to a
single file (say, for MPI task 12), proceed as follwos.
[pospiech@jugene1 hpm]$ llrun -np 16 -mode VN -env
LD_LIBRARY_PATH=$IHPCT_BASE/lib -env HPM_UNIQUE_FILE_NAME=yes -env
HPM_AGGREGATE=single.so -env HPM_PRINT_TASK=12 ./swim_mpi
Creating LoadLeveler Job
Submiting LoadLeveler Interactive Job
Wait for job jugene1b.42569.0 to be started:......
To get a single file with the average values over all MPI tasks, the following
command should work.
[pospiech@jugene1 hpm]$ rm *.viz *.hpm
[pospiech@jugene1 hpm]$ llrun -np 16 -mode VN -env
LD_LIBRARY_PATH=$IHPCT_BASE/lib -env HPM_UNIQUE_FILE_NAME=yes -env
HPM_AGGREGATE=average.so ./swim_mpi
Creating LoadLeveler Job
Submiting LoadLeveler Interactive Job
Wait for job jugene1b.42571.0 to be started:..
.
SPEC benchmark 102.swim
NUMBER OF POINTS IN THE X DIRECTION 512
NUMBER OF POINTS IN THE Y DIRECTION 512
GRID SPACING IN THE X DIRECTION 25000.
GRID SPACING IN THE Y DIRECTION 25000.
TIME STEP 20.
TIME FILTER PARAMETER 0.001
NUMBER OF ITERATIONS 102
wALLclock time= 0.924827647058823
[pospiech@jugene1 hpm]$ ls -l *.viz
-rw-r--r-- 1 pospiech ibm 139131 2008-08-22 22:52
swim_mpi_R00-M1-N09_0_22.08.2008_22.52.12.viz
[pospiech@jugene1 hpm]$
The previous example is using GNU fortran and runtime linking of the
aggregation module. This module has been specified by the environment
variable HPM_AGGREGATE=average.so .
To use IBM Fortran and static linking, the dlopen stubs have to be reverted to
ordinary "unresolved references". The is a "fake_dlfcn.o" provided, which
does just that. Then the static version of the aggregation module (e.g.
average.o) has to be used. The following commands accomplish this task.
[pospiech@jugene1 hpm]$ make ARCH=bgp_linux clean
rm -rf *.o *.out swim_mpi swim0 swim_omp \
redBlackSOR *.viz *.hpm core*
[pospiech@jugene1
hpm]$ /bgsys/drivers/ppcfloor/comm/bin/mpixlf77 -c -O3 -g -I/bgsys/local/ihpct-2.2.2/include -qsuffix=cpp=f -DHPM
swim_mpi.f
** shalow === End of Compilation 1 ===
** inital === End of Compilation 2 ===
** calc1 === End of Compilation 3 ===
** calc2 === End of Compilation 4 ===
** calc3z === End of Compilation 5 ===
** calc3 === End of Compilation 6 ===
** f_hpm_errchk === End of Compilation 7 ===
"swim_mpi.f", 1500-036 (I) The NOSTRICT option (default at OPT(3)) has the
potential to alter the semantics of a program. Please refer to documentation
on the STRICT/NOSTRICT option for more information.
"swim_mpi.f", 1500-036 (I) The NOSTRICT option (default at OPT(3)) has the
potential to alter the semantics of a program. Please refer to documentation
on the STRICT/NOSTRICT option for more information.
"swim_mpi.f", 1500-036 (I) The NOSTRICT option (default at OPT(3)) has the
potential to alter the semantics of a program. Please refer to documentation
on the STRICT/NOSTRICT option for more information.
1501-510 Compilation successful for file swim_mpi.f.
[pospiech@jugene1 hpm]$ /bgsys/drivers/ppcfloor/comm/bin/mpixlf77 -g \
swim_mpi.o -o swim_mpi \
/bgsys/local/ihpct-2.2.2/lib/fake_dlfcn.o \
/bgsys/local/ihpct-2.2.2/lib/average.o \
-L/bgsys/local/ihpct-2.2.2/lib -lhpm -llicense
[pospiech@jugene1 hpm]$ llrun -np 16 -mode VN -env
HPM_UNIQUE_FILE_NAME=yes ./swim_mpi
Creating LoadLeveler Job
Submiting LoadLeveler Interactive Job
Wait for job jugene1b.42573.0 to be started:..
.
SPEC benchmark 102.swim
NUMBER OF POINTS IN THE X DIRECTION 512
NUMBER OF POINTS IN THE Y DIRECTION 512
GRID SPACING IN THE X DIRECTION 25000.
GRID SPACING IN THE Y DIRECTION 25000.
TIME STEP 20.
TIME FILTER PARAMETER .001
NUMBER OF ITERATIONS 102
wALLclock time= 0.719031374117646949
[pospiech@jugene1 hpm]$ ls -lrt *.viz
-rw-r--r-- 1 pospiech ibm 139123 2008-08-22 23:18
swim_mpi_R00-M1-N09_0_22.08.2008_23.18.11.viz
[pospiech@jugene1 hpm]$
Please note that for static linking, the setting of LD_LIBRARY_PATH and
HPM_AGGREGATE is no longer needed.
</pre>
</blockquote>
<br>
</body>
</html>