[hpct] Example of HPM usage
Vitali A. Morozov
morozov at anl.gov
Thu Sep 4 09:40:07 CDT 2008
Hello, Christoph,
We have already discussed it some time in the past: our distribution
does not have examples directory, we only have bin, doc, include, and lib.
Could you please send as an example tared directory?
But you have answered to another question I had: IHPCT_BASE should point
to the root of the HPCT distribution tree. It illuminates the problem
with peekperf, which does not complain about this variable anymore.
However, it does not give a source anyway.
Second problem: your hpm installation does not correspond to ours: if I
link against -L/bgsys/local/ihpct-2.2.2/lib -llicense -lhpm, I will get
unresolved dlopen and dlsym symbols.
Third problem: you must be using gnu compiler, we are using IBM XL
compiler, -dynamic is not part of XL compiler options. If I am using
analogous -qnostaticlink, it does not help much.
Forth problem: if I finally compiled it with provided mirror.so:
mpixlf77 -g -o hpct-hcpm hpct-hcpm.o -L/home/morozov/soft/hpct_bgp/lib/
-lhpm -llicense -qnostaticlink
-L/bgsys/drivers/ppcfloor/gnu-linux/powerpc-bgp-linux/lib -ldl
and run with LD_LIBRARY_PATH=$IHPCT_BASE, I am getting crash. Teh core
file locates the crush in utils/hpm_dlfcn.c:56. So, I assume it is an
hpm issue.
Basically, I am trying to make the version I-Hsin was sending us two
months ago. I know, that previous versions were working just find until
this dlopen call has appeared.
Best, and thank you very much for your response,
Vitali
Christoph Pospiech wrote:
> On Wednesday 03 September 2008, Vitali A. Morozov wrote:
>
>> 8) HPM part does not work: core file at runtime. Please, give us example
>> of 1) how to compile HPM-instrumented program, 2) what is the result of
>> the run.
>>
>
> Sorry - long post and technical content. Please feel free to ask if something
> is unclear.
>
> The following commands have been executed on another customer site, at
> Forschungszentrum Jülich. They use a home grown script "llrun" to set up
> LoadL scripts and start the job. If interested, I can provide a contact to
> the authors of this script. Otherwise, replace every occurrence of "llrun"
> below with "mpirun -partition <whatever>".
>
> I used the same software as has been delivered to ANL. I have also used
> hpct/examples/hpm, which I am not sure about having been shipped to ANL.
>
> [pospiech at jugene1 ~]$ cd hpct_test/
> [pospiech at jugene1 hpct_test]$ cp -r /bgsys/local/ihpct-2.2.2/examples/ .
> [pospiech at jugene1 hpct_test]$ cd examples/hpm
> [pospiech at jugene1 hpm]$ export IHPCT_BASE=/bgsys/local/ihpct-2.2.2
> [pospiech at jugene1 hpm]$ make ARCH=bgp_linux swim_mpi
> /bgsys/drivers/ppcfloor/comm/bin/mpif90 -c -O3 -g -I/bgsys/local/ihpct-2.2.2/include -x
> f77-cpp-input -std=gnu -Wp,-DHPM swim_mpi.f
> /bgsys/drivers/ppcfloor/comm/bin/mpif90 -O3 swim_mpi.o -o
> swim_mpi -dynamic -L/bgsys/local/ihpct-2.2.2/lib -llicense -lhpm
> [pospiech at jugene1 hpm]$ llrun -np 16 -mode VN -env
> LD_LIBRARY_PATH=$IHPCT_BASE/lib -env HPM_UNIQUE_FILE_NAME=yes ./swim_mpi
> Creating LoadLeveler Job
> Submiting LoadLeveler Interactive Job
> Wait for job jugene1b.42567.0 to be started:......
> .
> SPEC benchmark 102.swim
>
> NUMBER OF POINTS IN THE X DIRECTION 512
> NUMBER OF POINTS IN THE Y DIRECTION 512
> GRID SPACING IN THE X DIRECTION 25000.
> GRID SPACING IN THE Y DIRECTION 25000.
> TIME STEP 20.
> TIME FILTER PARAMETER 0.001
> NUMBER OF ITERATIONS 102
> wALLclock time= 0.924954108235294
>
>
> --------------------------------------------------------------------------------------
> remarks and CAVEATS
> --------------------------------------------------------------------------------------
>
> The most important part of the llrun (or mpirun -partition <whatever>) command
> are
> -env LD_LIBRARY_PATH=$IHPCT_BASE/lib
> (otherwise you get "cannot load swim_mpi)
> -env HPM_UNIQUE_FILE_NAME=yes
> (otherwise all tasks write to the same file and you get a pile of
> garbage).
>
> The above example writes files *.hpm and *.viz (for viewing with peekperf),
> one for each MPI task. These might be many files. To only reduce it to a
> single file (say, for MPI task 12), proceed as follwos.
>
> [pospiech at jugene1 hpm]$ llrun -np 16 -mode VN -env
> LD_LIBRARY_PATH=$IHPCT_BASE/lib -env HPM_UNIQUE_FILE_NAME=yes -env
> HPM_AGGREGATE=single.so -env HPM_PRINT_TASK=12 ./swim_mpi
> Creating LoadLeveler Job
> Submiting LoadLeveler Interactive Job
> Wait for job jugene1b.42569.0 to be started:......
>
> To get a single file with the average values over all MPI tasks, the following
> command should work.
>
> [pospiech at jugene1 hpm]$ rm *.viz *.hpm
> [pospiech at jugene1 hpm]$ llrun -np 16 -mode VN -env
> LD_LIBRARY_PATH=$IHPCT_BASE/lib -env HPM_UNIQUE_FILE_NAME=yes -env
> HPM_AGGREGATE=average.so ./swim_mpi
> Creating LoadLeveler Job
> Submiting LoadLeveler Interactive Job
> Wait for job jugene1b.42571.0 to be started:..
> .
> SPEC benchmark 102.swim
>
> NUMBER OF POINTS IN THE X DIRECTION 512
> NUMBER OF POINTS IN THE Y DIRECTION 512
> GRID SPACING IN THE X DIRECTION 25000.
> GRID SPACING IN THE Y DIRECTION 25000.
> TIME STEP 20.
> TIME FILTER PARAMETER 0.001
> NUMBER OF ITERATIONS 102
> wALLclock time= 0.924827647058823
>
> [pospiech at jugene1 hpm]$ ls -l *.viz
> -rw-r--r-- 1 pospiech ibm 139131 2008-08-22 22:52
> swim_mpi_R00-M1-N09_0_22.08.2008_22.52.12.viz
> [pospiech at jugene1 hpm]$
>
> The previous example is using GNU fortran and runtime linking of the
> aggregation module. This module has been specified by the environment
> variable HPM_AGGREGATE=average.so .
>
> To use IBM Fortran and static linking, the dlopen stubs have to be reverted to
> ordinary "unresolved references". The is a "fake_dlfcn.o" provided, which
> does just that. Then the static version of the aggregation module (e.g.
> average.o) has to be used. The following commands accomplish this task.
>
> [pospiech at jugene1 hpm]$ make ARCH=bgp_linux clean
> rm -rf *.o *.out swim_mpi swim0 swim_omp \
> redBlackSOR *.viz *.hpm core*
> [pospiech at jugene1
> hpm]$ /bgsys/drivers/ppcfloor/comm/bin/mpixlf77 -c -O3 -g -I/bgsys/local/ihpct-2.2.2/include -qsuffix=cpp=f -DHPM
> swim_mpi.f
> ** shalow === End of Compilation 1 ===
> ** inital === End of Compilation 2 ===
> ** calc1 === End of Compilation 3 ===
> ** calc2 === End of Compilation 4 ===
> ** calc3z === End of Compilation 5 ===
> ** calc3 === End of Compilation 6 ===
> ** f_hpm_errchk === End of Compilation 7 ===
> "swim_mpi.f", 1500-036 (I) The NOSTRICT option (default at OPT(3)) has the
> potential to alter the semantics of a program. Please refer to documentation
> on the STRICT/NOSTRICT option for more information.
> "swim_mpi.f", 1500-036 (I) The NOSTRICT option (default at OPT(3)) has the
> potential to alter the semantics of a program. Please refer to documentation
> on the STRICT/NOSTRICT option for more information.
> "swim_mpi.f", 1500-036 (I) The NOSTRICT option (default at OPT(3)) has the
> potential to alter the semantics of a program. Please refer to documentation
> on the STRICT/NOSTRICT option for more information.
> 1501-510 Compilation successful for file swim_mpi.f.
> [pospiech at jugene1 hpm]$ /bgsys/drivers/ppcfloor/comm/bin/mpixlf77 -g \
> swim_mpi.o -o swim_mpi \
> /bgsys/local/ihpct-2.2.2/lib/fake_dlfcn.o \
> /bgsys/local/ihpct-2.2.2/lib/average.o \
> -L/bgsys/local/ihpct-2.2.2/lib -lhpm -llicense
>
> [pospiech at jugene1 hpm]$ llrun -np 16 -mode VN -env
> HPM_UNIQUE_FILE_NAME=yes ./swim_mpi
> Creating LoadLeveler Job
> Submiting LoadLeveler Interactive Job
> Wait for job jugene1b.42573.0 to be started:..
> .
> SPEC benchmark 102.swim
>
> NUMBER OF POINTS IN THE X DIRECTION 512
> NUMBER OF POINTS IN THE Y DIRECTION 512
> GRID SPACING IN THE X DIRECTION 25000.
> GRID SPACING IN THE Y DIRECTION 25000.
> TIME STEP 20.
> TIME FILTER PARAMETER .001
> NUMBER OF ITERATIONS 102
> wALLclock time= 0.719031374117646949
>
> [pospiech at jugene1 hpm]$ ls -lrt *.viz
> -rw-r--r-- 1 pospiech ibm 139123 2008-08-22 23:18
> swim_mpi_R00-M1-N09_0_22.08.2008_23.18.11.viz
> [pospiech at jugene1 hpm]$
>
> Please note that for static linking, the setting of LD_LIBRARY_PATH and
> HPM_AGGREGATE is no longer needed.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alcf.anl.gov/pipermail/hpct/attachments/20080904/014f0be2/attachment.htm>
More information about the hpct
mailing list