Re: [AMBER-Developers] (no subject)

From: Duke, Robert E Jr <>
Date: Sun, 25 Sep 2011 17:47:42 +0000

There are alignment requirements for data types, and what you are seeing is the typical padding required to keep all the dbl's aligned properly in an array. The only way around it I know of, off the top of my head, is to marshal the data into and out of byte arrays, and for this example, it is probably not worth it. The thing about marshalling is that it has very real costs too.

From: []
Sent: Sunday, September 25, 2011 9:37 AM
To: AMBER Developers Mailing List
Subject: [AMBER-Developers] (no subject)

Hi Amber Devs,

I've been making progress on the MPI-based parallel implementation of
mdgx, but I'm having a problem when it comes to sending structs and I'm
not sure how to proceed. There is a very convenient method for
user-defined data types in MPI, MPI_Type_struct, and I'd like to use this
to define my own data types and pass them in messages.

However, it seems that a struct like

struct iddd {
  int it1;
  double db1;
  double db2;
  double db3;
typedef struct iddd id3;

will show up as 32 bytes if I do sizeof(id3) even though

sizeof(int) + 3* sizeof(double) = 28

Similarly, with a struct like

struct iiddd {
  int it1;
  int it2;
  double db1;
  double db2;
  double db3;
typedef struct iiddd i2d3;

the result of sizeof(i2d3) is 32, which fortunately is in agreement with

2*sizeof(int) + 3* sizeof(double) = 32

What's going on here, and is there any way I can work with MPI_Type_struct
and be assured that what I'm doing is safe? The main types that I'd like
to define for the bulk of communications have two integers followed by 3,
6, or 18 doubles, so in that sense it seems like I might be OK on
architectures where 2*sizeof(int) = sizeof(double) if sizeof(iNdK) (where
iNdK is a struct with N integers followed K doubles) is rounding up to the
nearest increment of sizeof(double).

It seems that a struct like

struct ididd {
  int it1;
  double db1;
  int it2;
  double db2;
  double db3;
typedef struct ididd idid2;

would show up as sizeof(idid2) = 40 (I've been testing this), so that
jibes with other things I've heard about packing the smallest elements of
a struct together to conserve memory. But, I'm still left with my
fundamental question about how to send structs efficiently.

It seems that I could send them all as type MPI_BYTE using
N*sizeof(mystruct) to send N elements. That would work in most cases, it
seems, but it might get broken if someone had a heterogeneous cluster.
One other solution, suggested by the website above, might be to determine
the offsets for various fields in the struct by pointer arithmetic:

int offset1, offset2, offset3;
idid2 mystruct;

offset1 = &mystruct.db1 - &mystruct;
offset2 = &mystruct.it2 - &mystruct;
offset3 = &mystruct.db2 - &mystruct;

However when I test that, I get offset1=1, offset2=2, and offset3=3.
Consider the 64-bit architecture, that agrees with doubles being 8 bytes,
but it still seems that there's a lot of stuff I'm not able to control as
I'd like to. It may be that there isn't in fact a safe way to send
messages with heterogeneous data types in a heterogeneous cluster. Can
anyone suggest a strong solution?


AMBER-Developers mailing list
AMBER-Developers mailing list
Received on Sun Sep 25 2011 - 11:00:02 PDT
Custom Search