Hi Dave,
Take a look at the Paramfit source code in AMBER 11 (not the git tree
version unless you go back in time as we have since ripped out all the MPI
stuff and just use OpenMP), it will show you exactly how to do this in C.
Essentially you just build your own MPI Datatypes containing all the offsets
etc and you can send structs around to your heart's content. No need to mess
with aligning / padding anything. It works great.
Now, should you be sending structs etc? - Probably not for performance
reasons, but it can be useful during startup to broadcast parameters etc.
The pain is you HAVE TO remember to update the data types etc if you ever
modify the structures.
Look at void mpi_setup_data_types(mpi_struct *mpi_data, parm_struct
*parm_data)
/*ATOM_STRUCT*/
mpi_data->MPI_atom_struct_type_construct[0] = MPI_DOUBLE;
mpi_data->MPI_atom_struct_type_construct[1] = MPI_DOUBLE;
mpi_data->MPI_atom_struct_type_construct[2] = MPI_INT;
mpi_data->MPI_atom_struct_type_construct[3] = MPI_INT;
mpi_data->MPI_atom_struct_type_construct[4] = MPI_INT;
...
mpi_data->MPI_atom_struct_type_construct[9] = MPI_CHAR;
mpi_data->MPI_atom_struct_type_blocklen[0] = 1;
mpi_data->MPI_atom_struct_type_blocklen[1] = 1;
mpi_data->MPI_atom_struct_type_blocklen[2] = 1;
mpi_data->MPI_atom_struct_type_blocklen[3] = 1;
...
/*compute displacements*/
MPI_Address(parm_data->atom, &mpi_data->MPI_atom_struct_type_disp[0]);
MPI_Address(&parm_data->atom[0].amass,
&mpi_data->MPI_atom_struct_type_disp[1]);
MPI_Address(&parm_data->atom[0].join,
&mpi_data->MPI_atom_struct_type_disp[2]);
MPI_Address(&parm_data->atom[0].irotat,
&mpi_data->MPI_atom_struct_type_disp[3]);
...
for (i=(NUM_ELEMENTS_IN_ATOM_STRUCT-1);i>=0;--i)
mpi_data->MPI_atom_struct_type_disp[i] -=
mpi_data->MPI_atom_struct_type_disp[0];
/*build datatype*/
MPI_Type_struct(NUM_ELEMENTS_IN_ATOM_STRUCT,
mpi_data->MPI_atom_struct_type_blocklen,
mpi_data->MPI_atom_struct_type_disp,
mpi_data->MPI_atom_struct_type_construct,
&mpi_data->MPI_atom_struct_type );
MPI_Type_commit(&mpi_data->MPI_atom_struct_type);
/*END ATOM_STRUCT*/
All the best
Ross
> -----Original Message-----
> From: dcerutti.rci.rutgers.edu [mailto:dcerutti.rci.rutgers.edu]
> Sent: Sunday, September 25, 2011 9:38 AM
> To: AMBER Developers Mailing List
> Subject: [AMBER-Developers] (no subject)
>
> Hi Amber Devs,
>
> I've been making progress on the MPI-based parallel implementation of
> mdgx, but I'm having a problem when it comes to sending structs and I'm
> not sure how to proceed. There is a very convenient method for
> user-defined data types in MPI, MPI_Type_struct, and I'd like to use
> this
> to define my own data types and pass them in messages.
>
> http://www.mcs.anl.gov/research/projects/mpi/www/www3/MPI_Type_struct.h
> tml
>
> However, it seems that a struct like
>
> struct iddd {
> int it1;
> double db1;
> double db2;
> double db3;
> };
> typedef struct iddd id3;
>
> will show up as 32 bytes if I do sizeof(id3) even though
>
> sizeof(int) + 3* sizeof(double) = 28
>
> Similarly, with a struct like
>
> struct iiddd {
> int it1;
> int it2;
> double db1;
> double db2;
> double db3;
> };
> typedef struct iiddd i2d3;
>
> the result of sizeof(i2d3) is 32, which fortunately is in agreement
> with
>
> 2*sizeof(int) + 3* sizeof(double) = 32
>
> What's going on here, and is there any way I can work with
> MPI_Type_struct
> and be assured that what I'm doing is safe? The main types that I'd
> like
> to define for the bulk of communications have two integers followed by
> 3,
> 6, or 18 doubles, so in that sense it seems like I might be OK on
> architectures where 2*sizeof(int) = sizeof(double) if sizeof(iNdK)
> (where
> iNdK is a struct with N integers followed K doubles) is rounding up to
> the
> nearest increment of sizeof(double).
>
> It seems that a struct like
>
> struct ididd {
> int it1;
> double db1;
> int it2;
> double db2;
> double db3;
> };
> typedef struct ididd idid2;
>
> would show up as sizeof(idid2) = 40 (I've been testing this), so that
> jibes with other things I've heard about packing the smallest elements
> of
> a struct together to conserve memory. But, I'm still left with my
> fundamental question about how to send structs efficiently.
>
> It seems that I could send them all as type MPI_BYTE using
> N*sizeof(mystruct) to send N elements. That would work in most cases,
> it
> seems, but it might get broken if someone had a heterogeneous cluster.
> One other solution, suggested by the website above, might be to
> determine
> the offsets for various fields in the struct by pointer arithmetic:
>
> int offset1, offset2, offset3;
> idid2 mystruct;
>
> offset1 = &mystruct.db1 - &mystruct;
> offset2 = &mystruct.it2 - &mystruct;
> offset3 = &mystruct.db2 - &mystruct;
>
> However when I test that, I get offset1=1, offset2=2, and offset3=3.
> Consider the 64-bit architecture, that agrees with doubles being 8
> bytes,
> but it still seems that there's a lot of stuff I'm not able to control
> as
> I'd like to. It may be that there isn't in fact a safe way to send
> messages with heterogeneous data types in a heterogeneous cluster. Can
> anyone suggest a strong solution?
>
> Thanks,
> Dave
>
>
> _______________________________________________
> AMBER-Developers mailing list
> AMBER-Developers.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber-developers
_______________________________________________
AMBER-Developers mailing list
AMBER-Developers.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber-developers
Received on Mon Sep 26 2011 - 09:30:02 PDT
