[alcf-discuss] [Fwd: Re: Parallel hdf5 I/O of strided data]

[John R. Cary]

yes to collective.  This is the code:


//  Set collective
    xferPropList = H5Pcreate(H5P_DATASET_XFER);
    status = H5Pset_dxpl_mpio(xferPropList, H5FD_MPIO_COLLECTIVE);

// Then write
    status = H5Dwrite(dsetId, this->h5Types[dtype], mspaceId,
        dspaceId, xferPropList, dataPtr);

The code for setting collective is invoked for only the strided field
data.  Particle data (which writes out naturally one processor after 
another)
is not done collectively, and it works fine.

In fact, we tried not doing the collective, and then we got failures
on franklin.

We have a small test code, but that does not fail.

Our code has no problem with parallel I/O on franklin.nersc.gov.

Thx.....John

Tim Tautges wrote:
> [forwarding 'cuz it might be useful to have archived...]
>
>
> -------- Original Message --------
> Subject: Re: [alcf-discuss] Parallel hdf5 I/O of strided data
> Date: Wed, 6 May 2009 16:46:40 -0500
> From: Rob Latham <robl at mcs.anl.gov>
> To: Tim Tautges <tautges at mcs.anl.gov>
> References: <4A01FE12.20909 at mcs.anl.gov> 
> <20090506213103.GA21963 at mcs.anl.gov> <4A02034F.4060402 at mcs.anl.gov>
>
> On Wed, May 06, 2009 at 04:38:23PM -0500, Tim Tautges wrote:
>> Is that right?  The website you pointed to earlier  
>> (http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetDxplMpio)  
>> implies the opposite, at least in the documentation for  
>> H5Pset_dxpl_mpio_collective_opt...
>
>>>
>>> HDF5 disables collective I/O to a dataset by default, which is not
>>> ideal on bluegene. 
>
> OK, I see where the confusion lies.  We're talking about two different
> functions here.
>
> H5Pset_dxpl_mpio() sets up a property list that will guide how HDF5
> does dataset access: it can either be independent (the default) or
> collective.  Every HDF5 application on BlueGene should call this, set
> the H5FD_MPIO_COLLECTIVE flag, and pass the resulting property list to
> the HDF5 read/write routines.
>
> H5Pset_dxpl_mpio_collective_opt() is something else altogether.  Note
> "This function should be used only when H5FD_MPIO_COLLECTIVE  has been
> set through H5Pset_dxpl_mpio".    I don't have any idea why anybody
> would ever call H5Pset_dxpl_mpio_collective_opt() but I presume the
> HDF5 guys had a good reason.
>
> My advice: don't ever call H5Pset_dxpl_mpio_collective_opt() -- it
> doesn't do what you think it will do, and what it *does* do isn't good
> for BlueGene anyway.
>
> ==rob
>