[hpct] Example of HPM usage

Thu Sep 4 08:43:08 CDT 2008

On Wednesday 03 September 2008, Vitali A. Morozov wrote:
> 8) HPM part does not work: core file at runtime. Please, give us example
> of 1) how to compile HPM-instrumented program, 2) what is the result of
> the run.

Sorry - long post and technical content. Please feel free to ask if something 
is unclear.

The following commands have been executed on another customer site, at 
Forschungszentrum Jülich. They use a home grown script "llrun" to set up 
LoadL scripts and start the job. If interested, I can provide a contact to 
the authors of this script. Otherwise, replace every occurrence of "llrun" 
below with "mpirun -partition <whatever>".

I used the same software as has been delivered to ANL. I have also used 
hpct/examples/hpm, which I am not sure about having been shipped to ANL.

[pospiech at jugene1 ~]$ cd hpct_test/
[pospiech at jugene1 hpct_test]$ cp -r /bgsys/local/ihpct-2.2.2/examples/ .
[pospiech at jugene1 hpct_test]$ cd examples/hpm
[pospiech at jugene1 hpm]$ export IHPCT_BASE=/bgsys/local/ihpct-2.2.2
[pospiech at jugene1 hpm]$ make ARCH=bgp_linux swim_mpi
/bgsys/drivers/ppcfloor/comm/bin/mpif90 -c  -O3 -g -I/bgsys/local/ihpct-2.2.2/include -x 
f77-cpp-input -std=gnu -Wp,-DHPM swim_mpi.f
/bgsys/drivers/ppcfloor/comm/bin/mpif90 -O3 swim_mpi.o -o 
swim_mpi -dynamic -L/bgsys/local/ihpct-2.2.2/lib -llicense -lhpm
[pospiech at jugene1 hpm]$ llrun -np 16 -mode VN -env 
LD_LIBRARY_PATH=$IHPCT_BASE/lib -env HPM_UNIQUE_FILE_NAME=yes ./swim_mpi
Creating LoadLeveler Job
Submiting LoadLeveler Interactive Job
 Wait for job jugene1b.42567.0 to be started:......
.
 SPEC benchmark 102.swim

 NUMBER OF POINTS IN THE X DIRECTION     512
 NUMBER OF POINTS IN THE Y DIRECTION     512
 GRID SPACING IN THE X DIRECTION      25000.
 GRID SPACING IN THE Y DIRECTION      25000.
 TIME STEP                               20.
 TIME FILTER PARAMETER                 0.001
 NUMBER OF ITERATIONS                    102
 wALLclock time=  0.924954108235294

--------------------------------------------------------------------------------------
remarks and CAVEATS
--------------------------------------------------------------------------------------

The most important part of the llrun (or mpirun -partition <whatever>) command 
are
  -env LD_LIBRARY_PATH=$IHPCT_BASE/lib 
    (otherwise you get "cannot load swim_mpi)
  -env HPM_UNIQUE_FILE_NAME=yes 
    (otherwise all tasks write to the same file and you get a pile of 
garbage).

The above example writes files *.hpm and *.viz (for viewing with peekperf), 
one for each MPI task. These might be many files. To only reduce it to a 
single file (say, for MPI task 12), proceed as follwos.

[pospiech at jugene1 hpm]$ llrun -np 16 -mode VN -env 
LD_LIBRARY_PATH=$IHPCT_BASE/lib -env HPM_UNIQUE_FILE_NAME=yes -env 
HPM_AGGREGATE=single.so -env HPM_PRINT_TASK=12 ./swim_mpi
Creating LoadLeveler Job
Submiting LoadLeveler Interactive Job
 Wait for job jugene1b.42569.0 to be started:......

To get a single file with the average values over all MPI tasks, the following 
command should work.

[pospiech at jugene1 hpm]$ rm *.viz *.hpm
[pospiech at jugene1 hpm]$ llrun -np 16 -mode VN -env 
LD_LIBRARY_PATH=$IHPCT_BASE/lib -env HPM_UNIQUE_FILE_NAME=yes -env 
HPM_AGGREGATE=average.so ./swim_mpi
Creating LoadLeveler Job
Submiting LoadLeveler Interactive Job
 Wait for job jugene1b.42571.0 to be started:..
.
 SPEC benchmark 102.swim

 NUMBER OF POINTS IN THE X DIRECTION     512
 NUMBER OF POINTS IN THE Y DIRECTION     512
 GRID SPACING IN THE X DIRECTION      25000.
 GRID SPACING IN THE Y DIRECTION      25000.
 TIME STEP                               20.
 TIME FILTER PARAMETER                 0.001
 NUMBER OF ITERATIONS                    102
 wALLclock time=  0.924827647058823

[pospiech at jugene1 hpm]$ ls -l *.viz
-rw-r--r-- 1 pospiech ibm 139131 2008-08-22 22:52 
swim_mpi_R00-M1-N09_0_22.08.2008_22.52.12.viz
[pospiech at jugene1 hpm]$

The previous example is using GNU fortran and runtime linking of the 
aggregation module. This module has been specified by the environment 
variable  HPM_AGGREGATE=average.so .

To use IBM Fortran and static linking, the dlopen stubs have to be reverted to 
ordinary "unresolved references". The is a  "fake_dlfcn.o" provided, which 
does just that. Then the static version of the aggregation module (e.g. 
average.o) has to be used. The following commands accomplish this task.

[pospiech at jugene1 hpm]$ make ARCH=bgp_linux clean
rm -rf *.o *.out swim_mpi swim0 swim_omp \
        redBlackSOR *.viz *.hpm core*
[pospiech at jugene1 
hpm]$ /bgsys/drivers/ppcfloor/comm/bin/mpixlf77 -c  -O3 -g -I/bgsys/local/ihpct-2.2.2/include -qsuffix=cpp=f -DHPM 
swim_mpi.f
** shalow   === End of Compilation 1 ===
** inital   === End of Compilation 2 ===
** calc1   === End of Compilation 3 ===
** calc2   === End of Compilation 4 ===
** calc3z   === End of Compilation 5 ===
** calc3   === End of Compilation 6 ===
** f_hpm_errchk   === End of Compilation 7 ===
"swim_mpi.f", 1500-036 (I) The NOSTRICT option (default at OPT(3)) has the 
potential to alter the semantics of a program.  Please refer to documentation 
on the STRICT/NOSTRICT option for more information.
"swim_mpi.f", 1500-036 (I) The NOSTRICT option (default at OPT(3)) has the 
potential to alter the semantics of a program.  Please refer to documentation 
on the STRICT/NOSTRICT option for more information.
"swim_mpi.f", 1500-036 (I) The NOSTRICT option (default at OPT(3)) has the 
potential to alter the semantics of a program.  Please refer to documentation 
on the STRICT/NOSTRICT option for more information.
1501-510  Compilation successful for file swim_mpi.f.
[pospiech at jugene1 hpm]$ /bgsys/drivers/ppcfloor/comm/bin/mpixlf77 -g \
swim_mpi.o -o swim_mpi \
/bgsys/local/ihpct-2.2.2/lib/fake_dlfcn.o \
/bgsys/local/ihpct-2.2.2/lib/average.o \
-L/bgsys/local/ihpct-2.2.2/lib -lhpm -llicense

[pospiech at jugene1 hpm]$ llrun -np 16 -mode VN -env 
HPM_UNIQUE_FILE_NAME=yes ./swim_mpi
Creating LoadLeveler Job
Submiting LoadLeveler Interactive Job
 Wait for job jugene1b.42573.0 to be started:..
.
 SPEC benchmark 102.swim

 NUMBER OF POINTS IN THE X DIRECTION     512
 NUMBER OF POINTS IN THE Y DIRECTION     512
 GRID SPACING IN THE X DIRECTION      25000.
 GRID SPACING IN THE Y DIRECTION      25000.
 TIME STEP                               20.
 TIME FILTER PARAMETER                  .001
 NUMBER OF ITERATIONS                    102
 wALLclock time= 0.719031374117646949

[pospiech at jugene1 hpm]$ ls -lrt *.viz
-rw-r--r-- 1 pospiech ibm 139123 2008-08-22 23:18 
swim_mpi_R00-M1-N09_0_22.08.2008_23.18.11.viz
[pospiech at jugene1 hpm]$

Please note that for static linking, the setting of LD_LIBRARY_PATH and 
HPM_AGGREGATE is no longer needed.
-- 

Mit freundlichen Gruessen/Best Regards

Dr. Christoph Pospiech
High Performance & Parallel Computing
Advanced Computing Technology Center
Phone +49-351-86269826, eMail: Christoph.Pospiech at de.ibm.com
Mobile +49-171 765 5871

IBM Deutschland GmbH
Vorsitzender des Aufsichtsrats: Erich Clementi
Geschäftsführung: Martin Jetter (Vorsitzender), Christian Diedrich,
Christoph Grandpierre, Matthias Hartmann, Thomas Fell, Michael Diemer
Sitz der Gesellschaft: Stuttgart
Registergericht: Amtsgericht Stuttgart, HRB 14562 WEEE-Reg.-Nr. DE 99369940