<br><font size=2 face="sans-serif">This is how I do it in the GA/ARMCI
reference implementation (see: comm/lib/ga/toolkit/armci/src/x/dcmf/armcix_impl.c)
--- I added some comments for the purpose of this email....</font>
<br>
<br><tt><font size=2>int ARMCIX_Init ()</font></tt>
<br><tt><font size=2>{</font></tt>
<br><tt><font size=2> // This is the only DCMF call that can be invoked
before</font></tt>
<br><tt><font size=2> // DCMF_Messager_initialize(). Interrupt processing
is turned off</font></tt>
<br><tt><font size=2> // in a critical section. Doing this here prevents
the kernel from</font></tt>
<br><tt><font size=2> // firing an interrupt before the local node
has been initialized.</font></tt>
<br><tt><font size=2> DCMF_CriticalSection_enter(0);</font></tt>
<br>
<br><tt><font size=2> DCMF_Messager_initialize ();</font></tt>
<br>
<br><tt><font size=2> ...</font></tt>
<br>
<br><tt><font size=2> DCMF_Configure_t config;</font></tt>
<br><tt><font size=2> memset (&config, 0x00, sizeof(DCMF_Configure_t));</font></tt>
<br><tt><font size=2> config.interrupts = (interrupts==0)?DCMF_INTERRUPTS_OFF:DCMF_INTERRUPTS_ON;</font></tt>
<br><tt><font size=2> </font></tt>
<br><tt><font size=2> // First (input) parameter is the "requested"
configuration, the second</font></tt>
<br><tt><font size=2> // (output) parameter is the actual configuration.
This is important</font></tt>
<br><tt><font size=2> // because multiple independent software libraries
can coexist using</font></tt>
<br><tt><font size=2> // the same dcmf runtime (for example, MPI
and ARMCI). Interrupt mode </font></tt>
<br><tt><font size=2> // in particular can only be enabled, but not
disabled. Because the</font></tt>
<br><tt><font size=2> // application using the independent software
libraries may invoke the</font></tt>
<br><tt><font size=2> // init-type routine for those libraries in
any order it is possible that</font></tt>
<br><tt><font size=2> // the first may request to configure interrupts
on and then second may</font></tt>
<br><tt><font size=2> // request to configure interrupts off. In
this case the second configure</font></tt>
<br><tt><font size=2> // request will not "degrade" the
interrupt level and interrupts will</font></tt>
<br><tt><font size=2> // enabled for the dcmf runtime. Whew!</font></tt>
<br><tt><font size=2> DCMF_Messager_configure (&config, &config);</font></tt>
<br>
<br><tt><font size=2> ...</font></tt>
<br>
<br><tt><font size=2> // Ready to process interrupts now.</font></tt>
<br><tt><font size=2> DCMF_CriticalSection_exit(0);</font></tt>
<br><tt><font size=2>}</font></tt>
<br>
<br>
<br><font size=2 face="sans-serif">I have to point out that, specifically
on BG/P, interrupt mode may not be as useful as you may think. BG/P has
a DMA engine that offloads communication from the cores which allows BG/P
to achieve high overlap of computation and communication. In my experience
with ARMCI - another one-sided API - the remote nodes end up polling "enough"
that progress is always made. In fact, in most ARMCI and GA test cases
the performance is actually lower in interrupt mode.</font>
<br>
<br><font size=2 face="sans-serif">One thing to remember is that a single
call to DCMF_Messager_advance() may invoke many different callbacks (local
completion callbacks, recv notify callbacks, recv done callbacks, etc),
so it's not like you have to call DCMF_Messager_advance() a gajillion times.
Usually you are looping on a variable that will be changed by one of the
callback functions. Other callbacks may be invoked and may be altering
state on other variables, but the loop doesn't exit until the particular
condition is met. Check out this snip (again from the ARMCI reference implementation)
from the file: comm/lib/ga/toolkit/armci/src/x/dcmf/armcix_get.c
:</font>
<br>
<br><tt><font size=2>int ARMCIX_Get(void * src, void * dst, int bytes,
int proc)</font></tt>
<br><tt><font size=2>{</font></tt>
<br><tt><font size=2> DCMF_CriticalSection_enter (0);</font></tt>
<br>
<br><tt><font size=2> volatile unsigned active = 1;</font></tt>
<br><tt><font size=2> DCMF_Callback_t cb_wait = { ARMCIX_DCMF_cb_decrement,
(void *)&active };</font></tt>
<br><tt><font size=2> DCMF_Request_t request;</font></tt>
<br>
<br><tt><font size=2> DCMF_Get ( &__get_protocol,</font></tt>
<br><tt><font size=2> &request,</font></tt>
<br><tt><font size=2> cb_wait,</font></tt>
<br><tt><font size=2> DCMF_MATCH_CONSISTENCY,</font></tt>
<br><tt><font size=2> proc,</font></tt>
<br><tt><font size=2> bytes,</font></tt>
<br><tt><font size=2> (char
*) dst,</font></tt>
<br><tt><font size=2> (char
*) src);</font></tt>
<br>
<br><tt><font size=2> while (active) DCMF_Messager_advance ();</font></tt>
<br>
<br><tt><font size=2> DCMF_CriticalSection_exit (0);</font></tt>
<br>
<br><tt><font size=2> return 0;</font></tt>
<br><tt><font size=2>}</font></tt>
<br>
<br><font size=2 face="sans-serif">A couple things to note here ..</font>
<ol>
<li value=1><font size=2 face="sans-serif">the 'volatile' keyword is important
or the compiler may optimize out the while loop.</font>
<li value=2><font size=2 face="sans-serif">the code is wrapped in a critical
section. no point in taking the overhead for an interrupt if you
are actively polling!</font>
<li value=3><font size=2 face="sans-serif">the "cb_wait" callback
just decrements the "active" variable - non-blocking operations
may free the DCMF_Request_t that was used for the operation. Since this
is blocking the DCMF_Request_t was placed on the stack.</font></ol>
<br>
<br>
<br><font size=2 face="sans-serif">Michael Blocksome<br>
Blue Gene Messaging Team Lead<br>
Advanced Systems SW Development<br>
blocksom@us.ibm.com<br>
</font>
<br>
<br><tt><font size=2>rajesh.nishtala@gmail.com wrote on 02/07/2008 03:18:24
PM:<br>
<br>
> Thanks for the answers! They definitely help. In order to reconfigure<br>
> one the messaging layer to use Polling versus interrupts does one
need<br>
> to reinstall/reconfigure the software or is there a way to do it<br>
> within a particular installation. In either case how would one enable<br>
> or disable the 'interrupt mode'. Is this the 'DCMF_Interrupts'
field<br>
> in the DCMF_Configure_t type?<br>
> <br>
> Thanks,<br>
> -rajesh<br>
> <br>
> On Feb 7, 2008 12:12 PM, Michael Blocksome <blocksom@us.ibm.com>
wrote:<br>
> ><br>
> > Rajesh,<br>
> ><br>
> > These are excellent questions! We need to provide better documentation
on<br>
> > how the callback flow works ... baring that, I'll answer your
questions as<br>
> > they come and maybe we can pull together a document after answering
these<br>
> > questions on the mailing list.<br>
> ><br>
> ><br>
> > There are two types of callbacks - think of them as "local
completion" and<br>
> > "remote notification" callbacks.<br>
> ><br>
> > The local completion callbacks are specified not at registration
time, but<br>
> > when an individual operation is started (DCMF_Send(), for example).
This<br>
> > callback is invoked by the dcmf runtime, when the local node
calls the<br>
> > DCMF_Messager_advance() function, after the source buffer has
been<br>
> > completely sent. Once the local completion callback is
invoked all buffers<br>
> > associated with the operation may be deallocated, etc.<br>
> ><br>
> > The callbacks that are registered for DCMF_Send (and DCMF_Control,etc)
with<br>
> > the DCMF_Send_register() function are invoked by the dcmf runtime
on the<br>
> > remote node when that node call the DCMF_Messager_advance() function.<br>
> > Typically all nodes in the system will periodically poll with<br>
> > DCMF_Messager_advance() to make progress, however the BGP messager
can be<br>
> > configured to enable an interupt to be fired when a core receives
a packet.<br>
> > In this interrupt mode active polling is not required - although
you do take<br>
> > a performance hit because of the overhead of processing the interrupts.<br>
> ><br>
> > The remote callbacks for DCMF_Send are invoked before any data
has been<br>
> > written to the remote node. There are two callback types
and each has<br>
> > slightly different use by the application programmer.<br>
> ><br>
> > The DCMF_RecvSendShort ("short") callbacks are invoked
when the entire<br>
> > message has been received by the remote node into a temporary
location (on<br>
> > BGP this is a single packet of data that has been received by
the DMA into a<br>
> > memory fifo). The application's responsibility is to copy
the data out of<br>
> > the temporary buffer and into the final destination buffer. This
callback<br>
> > type was created specifically to allow the dcmf implementation
to optimize<br>
> > the performance for small messages.<br>
> ><br>
> > The DCMF_RecvSend ("long" or "asynhcronous")
callbacks are invoked when the<br>
> > control information has been received by the remote node into
a temporary<br>
> > location (on BGP the control information will be contained in
a single<br>
> > packet). The application's responsibility is to allocate
memory<br>
> > (DCMF_Request_t) for the dcmf runtime to use to receive the rest
of the<br>
> > data, as well as specify the destination buffer and length and
a<br>
> > ("recv_done") callback. This "recv_done"
callback is invoked by the dcmf<br>
> > runtime when the data has been completely received and written
to the<br>
> > destination buffer. Typically applications will free/deallocate
the<br>
> > DCMF_Request_t memory that was allocated previously.<br>
> ><br>
> ><br>
> ><br>
> > DCMF_Put and memory regions (i.e., registration, pinning)<br>
> ><br>
> > The DCMF_Put in the library is just stubbed in as we didn't have
a need for<br>
> > it in release 1 of the BG/P software. However, we are actively
working on<br>
> > adding the DCMF_Put() into the API which will also require a
memory region<br>
> > API. The existing DCMF_Get API will be updated to use these new
memory<br>
> > regions objects. Perhaps we should go into more detail
on the memory region<br>
> > API in a separate email.<br>
> ><br>
> ><br>
> > I hope this helps!<br>
> ><br>
> > Michael Blocksome<br>
> > Blue Gene Messaging Team Lead<br>
> > Advanced Systems SW Development<br>
> > blocksom@us.ibm.com<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > "Rajesh Nishtala" <rajeshn@eecs.berkeley.edu><br>
> > Sent by: dcmf-bounces@lists.anl-external.org<br>
> ><br>
> > 02/07/2008 12:01 PM<br>
> ><br>
> > To dcmf@lists.anl-external.org<br>
> ><br>
> > cc upc-devel@lbl.gov<br>
> ><br>
> > Subject [dcmf] Question about DCMF Library from a new user<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > Hi,<br>
> > I am porting GASNet, our portable runtime layer for the
Berkeley UPC<br>
> > compiler, to the BlueGene/P and i'm using DCMF as the lower
level<br>
> > messaging layer. I have some high level questions regarding
the<br>
> > library that will influence the design of our BlueGene/P
port. The<br>
> > main difference between our library and MPI is that we
focus on<br>
> > one-sided communication so my main questions are regarding
these<br>
> > issues.<br>
> ><br>
> > + When does the callback that gets registered with DCMF_Send()
get<br>
> > called? Does it get called after the data has been committed
to the<br>
> > memory on the remote node or does it simply imply that
the data buffer<br>
> > is safe to reuse on the local node?<br>
> ><br>
> > + I notice that when I do an nm on the dcmf libraries there
is<br>
> > DCMF_Put() function, however when i waded through the code
a little<br>
> > bit more I noticed that the function simply called an abort
which to<br>
> > me implies that it is not implemented. Is this why it doesn't
show up<br>
> > in the dcmf.h header files?<br>
> ><br>
> > + I have heard that the BlueGene/P supports RDMA operations.
Are there<br>
> > any special considerations for managing the memory registration
(i.e.<br>
> > memory pinning) to enable these operations or is this done<br>
> > automatically under the covers?<br>
> ><br>
> ><br>
> > Thanks in advance for any help!<br>
> ><br>
> > Sincerely,<br>
> > Rajesh Nishtala<br>
> > _______________________________________________<br>
> > dcmf mailing list<br>
> > dcmf@lists.anl-external.org<br>
> > http://lists.anl-external.org/cgi-bin/mailman/listinfo/dcmf<br>
> > http://dcmf.anl-external.org/wiki<br>
> ><br>
> ><br>
</font></tt>