Hi,
Well i think we are all more or less in agreement,
but i have a couple of nit pick belows.
On Mon, Sep 26, 2011 at 11:22:12AM -0700, Ross Walker wrote:
> > you might think. So you either 1) live with the extra bytes "over the
> > wire" because any other solution has equivalent costs, 2) marshal into
> > and out of a byte stream or 3) there is a third option - send the data
> > in sequential arrays.
>
> The 3rd option here is what I was essentially trying to refer to when I said
> sending structs was non-optimal. The problem you have in C is that, unlike
> Fortran, structures are not linear in memory to begin with. If you allocate
> arrays in structures what you really do is have a pointer in there that
> points to some other piece of memory in a different location. When you go to
No this is not technically correct, but if you know C then you
know what Ross is trying to say.
struct { int stuff[4] } bla
vs
struct { int * pstuff } ha
int mystuff[4]
ha->pstuff = mystuff
bla is linear, ha is linear, mystuff is linear, but the
combo ha and mystuff is not guaranteed to be linear.
In addition, the same padding and alignment issues we've been talking
about in C can arise in Fortran; see for example:
http://earth.uni-muenster.de/~joergs/doc/f90/unix-um/dfum_018.html
> 2 D arrays in structures it gets even worse. This is why allocatable
> variables were not allowed in structures in the F95 standard (they were
> added reluctantly as part of F2003) since it prevents the data in your
> structures being linear in memory.
>
> So while using the offset approach to build a custom MPI datatype works, and
> works well given it avoids all the packing etc that Bob is referring to it
> does nothing to address the fact that using structures in the first place
> will be hurting your performance. Then MPI sends of structures etc mean
> going and getting data from lots of different locations in memory which
> hurts performance even before you start transmitting over the infiniband
> 'line'.
>
> Thus your best option would probably be to go back to the old school
> approach of putting things in linear arrays and sending them. Note, I do NOT
> mean copying the data into linear arrays I mean actually having it in linear
> arrays to begin with. Essentially you should write down exactly how you will
> traverse the data in your code and exactly how you will communicate it
> between nodes. Then you need to lay out your arrays etc to match these two
> cases as best you can. This of course will make the code more ugly and most
> likely more difficult to follow but it is a necessary evil if you want to
> get the performance.
I don't agree that such a code is necessarily ugly.
scott
_______________________________________________
AMBER-Developers mailing list
AMBER-Developers.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber-developers
Received on Mon Sep 26 2011 - 12:30:07 PDT
