<br><font size=2 face="sans-serif">I've been working on a new README file
for the comm stack. Please take a look at it, since I hope that it
will answer some of the recent questions. It is a little rough, but
I wanted to get it out fast.</font>
<br><font size=2 face="sans-serif"><br>
Thanks,</font>
<br><font size=2 face="sans-serif">Joe Ratterman</font>
<br><font size=2 face="sans-serif">jratt@us.ibm.com</font>
<br>
<br>
<br><tt><font size=2>About the DCMF messaging stack<br>
This is the DCMF messaging stack, currently for use on the
IBM Blue<br>
Gene/P platform. The package does contain more than just
the DCMF<br>
libraries. Everything is collected together to ensure that
it works<br>
correctly.<br>
<br>
This file is generated from the POD in README.pod. Please
only change<br>
that file.<br>
<br>
DCMF<br>
DCMF stands for *Deep Computing Messaging Framework*.
It forms the<br>
basis of the active message layer in use, and
is the prefix on all C<br>
API symbols.<br>
<br>
CCMI<br>
CCMI stands for *Common Collective Messaging
Interface*. It is a<br>
general system for writing cross-platform collectives.
Internally,<br>
it uses the CCMI namespace, but the external
C API still uses the<br>
DCMF prefix.<br>
<br>
Directory layout<br>
The directories are designed to organize components effectively.
There<br>
is some historical carry-over, but the layout is intended
to group<br>
related items.<br>
<br>
tools/<br>
This is the simplest directory. It contains programs and
scripts that<br>
run on the FENs and help make your life better.<br>
<br>
sys/<br>
The sys directory contains low-level software written entirely
by IBM.<br>
It is generally assumed to be for software that talks to
the HW/SPIs or<br>
implements a new external API.<br>
<br>
sys/include/<br>
The API header files exist in this directory.
They are installed by<br>
the build process of the tool that owns them.<br>
<br>
sys/messaging/<br>
This is the home of the DCMF layer. This directory
contains a<br>
"configure" script that can be used
to setup the build with various<br>
options. The defaults are generally good for
our setup, but other<br>
people may have different goals. It is not
necessary to run<br>
"configure" from inside the messaging
directory. This allows<br>
multiple different builds from the same source.<br>
<br>
There are also a number of older tools in this
directory that should<br>
be moved in the future.<br>
<br>
sys/collectives/<br>
This is the home of the CCMI layer. This directory
also contains a<br>
configure script that works very much like
the one in<br>
sys/messaging/.<br>
<br>
sys/tests/<br>
This is a small (but growing) test repository,
able to test many<br>
different layers.<br>
<br>
sys/examples/<br>
These files are examples of how to use DCMF
C API documentation.<br>
They are also included in the documentation
that is generated by<br>
doxygen.<br>
<br>
sys/build-*/<br>
These directories are the build paths for the
source directories.<br>
Since it is so easy to avoid building in the
src path, we try to<br>
avoid mixing the source and objects--it simplifies
cleaning up and<br>
building with multiple compilers.<br>
<br>
lib/<br>
This is generally for imported external packages.<br>
<br>
lib/mpi/<br>
This is "beefed up" version of the
MPICH2 distribution from ANL. It<br>
contains a modified version of MPICH2 1.0.4p1
in lib/mpi/mpich2/.<br>
Mostly, the modifications consist of a new
MPID device, a new MPI-IO<br>
FS, and some changes to the test bucket scripts.<br>
<br>
The m4 and autoconf directories are from when
MPICH2 required a<br>
version of those tools that was not installed
on the build systems.<br>
They are not currently used on the BG/P platform.<br>
<br>
The sowing directory contains the source for
the doctext tool. It is<br>
used to create the manual and html pages to
document the MPI2 API.<br>
<br>
The build and stage dirs are for object files
created by the<br>
building the components. Stage is the install
location for<br>
sowing/doctext and, when used, m4 and autoconf.
There are two<br>
version of the MPICH2 build under lib/mpi/build/mpich2/.
The *dcmf*<br>
version is the standard version compiled with
GNU compiler suite and<br>
shipped. The *dcmf-fast* version, which isn't
actually known to be<br>
faster, is compiled with the IBM XL compiler
suite.<br>
<br>
lib/ga/<br>
This is the GA-ARMCI distribution from PNNL.
It has been slightly<br>
modified to work with the DCMF API. This is
a reference<br>
implementation and has been released to the
GA community.<br>
<br>
Build system<br>
The build system in use for the messaging stack is a bit
of a kludging<br>
together of the more sane build systems found in most of
the<br>
sub-components.<br>
<br>
Make files<br>
Generally, one should only really have to look at a few specific
files<br>
to see how certain sections are built:<br>
<br>
Makefile<br>
This is top-level make file and is designed
to help build individual<br>
components.<br>
<br>
Make.rules*<br>
Make.rules is a symlink that is intended to
point to another<br>
Make.rules.* file. To aid in porting to other
build systems, the<br>
Make.rules file is included (eventually) by
every other make file in<br>
the system--they must first define a ROOTDIR
variable to store the<br>
path that contains Make.rules. The Make.rules.floor
contains logical<br>
defaults for building everything based on the
installed floor on the<br>
FEN. Make.rules.ibm contains commands to get
the same information<br>
from the internal build system.<br>
<br>
lib/mpi/Makefile<br>
This file accommodates the large number of
options that must be<br>
passed to the MPICH2 configure. Unlike the
DCMF and CCMI parts,<br>
MPICH2 is a cross-platform system that has
no concept of the BG/P<br>
install layout. Obviously MPICH2 can be configured
on it's own, but<br>
it's quite hard to get the options correct.<br>
<br>
Make targets<br>
The build system is designed to detect the presence or lack
of the IBM<br>
XL compiler suite. To force the build system to avoid XL,
add "XL=" to<br>
your make command.<br>
<br>
default & install<br>
This is the default, and builds everything
below except for "tests".<br>
<br>
all This builds everything including "tests".<br>
<br>
docs<br>
DCMF, CCMI, and the DCMF MPID device in MPICH2
all contain extensive<br>
documentation. This rule builds and installs
it.<br>
<br>
dcmf[-fast]<br>
These are two different versions of the DCMF
library. The -fast<br>
version is build without asset macros.<br>
<br>
ccmi[-fast]<br>
Again, two versions of the CCMI library. The
-fast version is<br>
compiled by the IBM XL compilers without assert
macros. It is not<br>
known if it is actually faster.<br>
<br>
mpi This simply runs the makefile in lib/mpi/.<br>
<br>
ga Like mpi, this simply runs the makefile in lib/ga/.<br>
<br>
tests<br>
This builds the tests in sys/tests/.<br>
<br>
</font></tt>