[hpct] MPI_trace: Issues: Problem 1

[Vitali A. Morozov]

Dear HPCT experts,

I would like to make available to everybody the list of outstanding 
issues I have with Part 1 of the HPCT: MPI profiler and tracer.

To keep the issues separate, I will shoot one problem at a time.

Problem 1: A Fortran 90 code intentionally contains MPI_Send/MPI_Recv 
from/to non-zero MPI ranks. Manual instrumentation is used to bracket 
the region of interest with  MT_trace_start()/MT_trace_stop() functions 
to trace only enclosed calls. Both compiling and linking were done with 
-g. The results are confusing.

We expect

a) To see MPI_Send/MPI_Recv calls in mpi_profile.0 file, because, as 
follows from the manual, this file contains the calls from all mpi tasks.

b) Do not see MPI_Comm_size(), MPI_Comm_rank() calls in mpi_profile.0, 
as this calls are outside the bracketed region.

c) Post-process *.viz file with peekperf:
   
c1) "peekperf mpi_profile_0.viz trace"  opens a file dialog, asking for 
IHPCT_BASE directory. Any choice says "Cannot generate the binary 
analysis file!" and opens the window without sources.

c2) peekperf does not have MPI_Send/MPI_Recv calls.

d) Post-process single_trace file with "peekperf single_trace". There is 
no MPI calls in the trace. However, mpi_profile.2 file registered 
MPI_Recv(), which is even more confusing, because this call is outside 
the brackets.

d1) In case the trace stamps are large compared to a single 
communication, we would like to be able to manipulate by this length.

e) Error file contains "Error getting the executable name, line number 
information will not be available" message. What else should be done 
additionally to "-g" option?


=========
program selective
implicit none
include 'mpif.h'

integer ierr, rank, nprocs, istat( MPI_STATUS_SIZE )
real(8) d

call MPI_Init( ierr )
call MPI_Comm_size( MPI_COMM_WORLD, nprocs, ierr )
call MPI_Comm_rank( MPI_COMM_WORLD, rank, ierr )

call MT_trace_start_()
if ( rank .eq.  4 ) call MPI_Send( d, 1, MPI_DOUBLE_PRECISION, 8, 0, 
MPI_COMM_WORLD, ierr )
if ( rank .eq.  8 ) call MPI_Recv( d, 1, MPI_DOUBLE_PRECISION, 4, 0, 
MPI_COMM_WORLD, istat, ierr )
call MT_trace_stop_()

if ( rank .eq. 12 ) call MPI_Send( d, 1, MPI_DOUBLE_PRECISION,  2, 0, 
MPI_COMM_WORLD, ierr )
if ( rank .eq.  2 ) call MPI_Recv( d, 1, MPI_DOUBLE_PRECISION, 12, 0, 
MPI_COMM_WORLD, istat, ierr )

call MPI_Finalize()
stop

end program selective
============

build it with

mpixlf90 -g -c trace.f90
mpixlf90 -g -o trace trace.o -L/home/morozov/soft/hpct_bgp/lib/ 
-lmpitrace -llicense -lgfortran

run it with
--env TRACE_SEND_PATTERN=yes:TRACE_ALL_TASKS=yes:TRACE_ALL_EVENTS=no, 16 
nodes, smp.

Thank you

Vitali Morozov


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alcf.anl.gov/pipermail/hpct/attachments/20080813/497cfecb/attachment.html>