Okay, let's clarify a bit. Packing into a byte stream (and unpacking) IS costly, due to all the data movement in memory. Whether or not it is worthwhile depends on the extent of reduction in data that has to be sent "over the wire", and this is something that is highly hardware dependent, so you need to check it out. I think with the example you showed, you maybe would get on the order of a 10% reduction in data sent via MPI, so my rough guess was that the gain in doing this would be marginal; at 25% I would begin to consider some sort of marshalling approach much more seriously. You absolutely SHOULD NOT convert int's to dbl's. You get NO data compression from doing this, and add the overhead of the data conversion, which is typically more costly than you might think. So you either 1) live with the extra bytes "over the wire" because any other solution has equivalent costs, 2) marshal into and out of a byte stream or 3) there is a third option - send the data in sequential arrays. Thus, say you logica
lly have several ints and dbls that relate to one atom. Well, if you are sending data relating to 100's of atoms, you can just send all the ints first, and then all the dbls. As an aside, If memory serves, I actually used marshalling extensively in pmemd, but this was partly a function of the fact that I used dedicated mpi buffers to avoid a host of OTHER problems you can get into with buffers and mpi. I was also always having to send data out-of-order, so you have to move the stuff around anyway. I also was dealing with fortran, and had to do more cheesey things to get around various deficiencies associated with the language. Bottom line - how much do you care about performance? If you do, then you may choose to do a variety of things that would be considered "less clean" from a software engineering perspective; they will be more work and more error prone and in some senses less maintainable, but if you are writing in C and actually understand it well, all this low level stuff is dead easy. On the ot
her hand, the cleanest software engineering approach is the one that Ross refers to.
- Bob
________________________________________
From: dcerutti.rci.rutgers.edu [dcerutti.rci.rutgers.edu]
Sent: Monday, September 26, 2011 9:35 AM
To: AMBER Developers Mailing List
Subject: Re: [AMBER-Developers] Sending Structs in MPI
Hrmm, I seem to have gotten conflicting advice now; Bob said that
converting everything to byte streams is costly, you say that sending
things as structs is non-optimal. Should I convert the integers in my
structs to doubles for sending? That seems like it might be costly as
well. Alternatively, I could even devise a way (actually, I have, and
it's been in the code for a very long time anticipating that I might want
to do this) to pack doubles of small real numbers into ints and ship 'em
off for greater precision than floating point conversions would allow.
Just let me know what you think would give the best performance, and I'll
keep that in mind as I develop the parallel implementation. Much better
to get things right in the beginning.
Dave
> Hi Dave,
>
> Take a look at the Paramfit source code in AMBER 11 (not the git tree
> version unless you go back in time as we have since ripped out all the MPI
> stuff and just use OpenMP), it will show you exactly how to do this in C.
> Essentially you just build your own MPI Datatypes containing all the
> offsets
> etc and you can send structs around to your heart's content. No need to
> mess
> with aligning / padding anything. It works great.
>
> Now, should you be sending structs etc? - Probably not for performance
> reasons, but it can be useful during startup to broadcast parameters etc.
> The pain is you HAVE TO remember to update the data types etc if you ever
> modify the structures.
_______________________________________________
AMBER-Developers mailing list
AMBER-Developers.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber-developers
_______________________________________________
AMBER-Developers mailing list
AMBER-Developers.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber-developers
Received on Mon Sep 26 2011 - 10:30:02 PDT
