Re: [AMBER-Developers] New pmemd-GTI branch available for testing

From: David Cerutti <>
Date: Sat, 23 Dec 2017 13:35:45 -0500

I think this is some great work from Taisung, and he has taught me a number
of things about CUDA and C++. I think that the gpuContext consolidation is
definitely something that should be carried through to master, and I
believe that the linking will help debugging (the CUDA debugger has
requirements about things be compiled simultaneously).

While I am not yet ready to take a side on which TI implementation, I will
say that I've managed to integrate the AFE code from SDSC into some of my
new code and it has been cleanly integrated. Having a separate part of the
program handle TI and perform transactions with the equilibrium MD code (as
opposed to embedding the calculations in the workflow) will impose a
cost--no one can reach in and extract information for additional
computations as quickly as they could be done within the context of the
primary MD kernels--but that cost appears to be small, and an apparatus
that reaches in and extracts such information would serve as a model for
future program extensions. There are features of each code not found in
the other: the Berendsen barostat is supported by pmemd-GTI while MBAR is
supported by the AFE code.

Merry Christmas all,


On Sat, Dec 23, 2017 at 1:11 PM, David Cerutti <> wrote:

> Hi all:
> In an effort to evaluate and prepare to ultimately merge
> our GPU-accelerated TI implementation with the master branch, we created
> the “gti” branch that is synced to the current master branch, but that
> enabled both TI implementations - the pmemdGTI implementation from the
> Laboratory for Biomolecular Simulation Research (LBSR) at Rutgers, and the
> AFE implementation from Ross' group which is in the current master branch -
> to co-exist. This allows head to head tests and comparisons, code
> evaluation, consolidation and discussion of how we want to move forward in
> the next official release. I am sure we can learn a lot from looking at
> both code bases, and adopt best practices from each.
> Here is a summary of main changes from current main branch:
> *Major program changes—for cuda developers*
> *1. **all*
> The cSim and gpuContext objects are now wrapped as base classes. All
> variable definitions and initializations are done in base_gpuCpntext.{h,
> cpp} and base_simulationConst.{h, cpp}.
> *2. **Configured with –gti flag:*
> a. The current pmemd.cuda is compiled as a combination of several
> compiled cuda units. All units need to keep its own copy of cSim and that
> means ~10 copies of cSim are in the program at the same time and they need
> to share the GPU constant memory (64KB). *As a result, cSim is limited
> to size of ~5KB*. Furthermore, any change in cSim on the host needs to
> be passed to all copies of cSim’s on GPU.
> With –gti configured, the cuda kernels are compiled as portable units and
> linked as one cuda unit—hence there is only one copy of cSim at runtime. *The
> cSim now has the size limit of ~64KB. *And coping cSim from host to GPU
> only needs to be done once. You will be able to start putting much more
> things into cSim.
> b. The gpu context object is a local variable created in the heap
> space (gpu = new _gpuContext;). With –gti configured. It is now handled
> as a singleton object and can be accessed from anywhere by
> “gpu = theGPUContext::GetPointer(); “.
> *Test results*
> *1.without -git flag*
> Without turning on GTI (i.e., configured without –gti),
> gave
> 171 file comparisons passed
> 0 file comparison failed
> 0 test experienced an error
> *2. with -git flag*
> With -gti turned on (i.e., configured with), gave
> 161 file comparisons passed
> 10 file comparison failed
> 0 test experienced an error
> The “file comparison failed” is small numerical differences in the
> cellulose case and all AFE cases (due to different TI implementations).
> We will update all GTI functionalities and test cases after this initial
> merge.
> Please take a look at the “gti” branch and let us any problems: especially
> the AFE part: we need to modify some parts so that the “#ifdef GTI” can
> effectively distinguish the AFE and the GTI implementations. It seems
> there is no problem—all AFE tests are passed when –gti is not configured.
> Best,
> Taisung and Darrin
AMBER-Developers mailing list
Received on Sat Dec 23 2017 - 11:00:03 PST
Custom Search