> you might think. So you either 1) live with the extra bytes "over the
> wire" because any other solution has equivalent costs, 2) marshal into
> and out of a byte stream or 3) there is a third option - send the data
> in sequential arrays.
The 3rd option here is what I was essentially trying to refer to when I said
sending structs was non-optimal. The problem you have in C is that, unlike
Fortran, structures are not linear in memory to begin with. If you allocate
arrays in structures what you really do is have a pointer in there that
points to some other piece of memory in a different location. When you go to
2 D arrays in structures it gets even worse. This is why allocatable
variables were not allowed in structures in the F95 standard (they were
added reluctantly as part of F2003) since it prevents the data in your
structures being linear in memory.
So while using the offset approach to build a custom MPI datatype works, and
works well given it avoids all the packing etc that Bob is referring to it
does nothing to address the fact that using structures in the first place
will be hurting your performance. Then MPI sends of structures etc mean
going and getting data from lots of different locations in memory which
hurts performance even before you start transmitting over the infiniband
'line'.
Thus your best option would probably be to go back to the old school
approach of putting things in linear arrays and sending them. Note, I do NOT
mean copying the data into linear arrays I mean actually having it in linear
arrays to begin with. Essentially you should write down exactly how you will
traverse the data in your code and exactly how you will communicate it
between nodes. Then you need to lay out your arrays etc to match these two
cases as best you can. This of course will make the code more ugly and most
likely more difficult to follow but it is a necessary evil if you want to
get the performance.
All the best
Ross
/\
\/
|\oss Walker
---------------------------------------------------------
| Assistant Research Professor |
| San Diego Supercomputer Center |
| Adjunct Assistant Professor |
| Dept. of Chemistry and Biochemistry |
| University of California San Diego |
| NVIDIA Fellow |
|
http://www.rosswalker.co.uk |
http://www.wmd-lab.org/ |
| Tel: +1 858 822 0854 | EMail:- ross.rosswalker.co.uk |
---------------------------------------------------------
Note: Electronic Mail is not secure, has no guarantee of delivery, may not
be read every day, and should not be used for urgent or sensitive issues.
_______________________________________________
AMBER-Developers mailing list
AMBER-Developers.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber-developers
Received on Mon Sep 26 2011 - 11:30:03 PDT
