Re: [AMBER-Developers] Parallel make failing

From: Gerald Monard <Gerald.Monard.univ-lorraine.fr>
Date: Fri, 27 May 2016 00:14:14 +0200

OK. If you like it, I let you push the changes in the master tree.
All the best,

Gerald.

On 05/26/2016 11:56 PM, Ray Luo wrote:
> Thanks again!
>
> Ray
> --
> Ray Luo, Ph.D.
> Professor
> Biochemistry, Molecular Biophysics, Chemical Physics,
> Chemical and Biomedical Engineering
> University of California, Irvine, CA 92697-3900
>
>
> On Thu, May 26, 2016 at 2:34 PM, Gerald Monard
> <Gerald.Monard.univ-lorraine.fr> wrote:
>> Ray,
>>
>> pbsa.SANDER.o (and parms.SANDER.o) should not be compiled (sorry).
>> Can you apply this new patch?
>>
>> Gerald.
>>
>>
>> On 05/26/2016 11:17 PM, Ray Luo wrote:
>>>
>>> Gerald,
>>>
>>> Thanks a lot! I'm testing with a "configure -mpi gnu" build in 16
>>> threads. Here are the error messages when building "libFpbsa.a".
>>>
>>> All the best,
>>> Ray
>>>
>>> ++++
>>>
>>> cd ../pbsa && make libFpbsa.a
>>> make[3]: Entering directory
>>> `/home/rayl/amber-master-fresh1/AmberTools/src/pbsa'
>>> /bin/rm -f decomp.o decomp.mod DECOMP.mod
>>> mpif90 -DBINTRAJ -DEMIL -DMPI -c -O3 -mtune=native -fPIC
>>> -ffree-form -I/home/rayl/amber-master-fresh1/include
>>> -I/home/rayl/amber-master-fresh1/include -o density.o density.F90
>>> mpif90 -DBINTRAJ -DEMIL -DMPI -c -O3 -mtune=native -fPIC
>>> -ffree-form -I/home/rayl/amber-master-fresh1/include
>>> -I/home/rayl/amber-master-fresh1/include -o gen_dx_file.o
>>> gen_dx_file.F90
>>> cd ../sander && make decomp.o
>>> mpif90 -DBINTRAJ -DEMIL -DMPI -DSANDER -c -O3 -mtune=native -fPIC
>>> -ffree-form -I/home/rayl/amber-master-fresh1/include
>>> -I/home/rayl/amber-master-fresh1/include -I../sander \
>>> -o interpX.SANDER.o interpX.F90
>>> mpif90 -DBINTRAJ -DEMIL -DMPI -DSANDER -c -O3 -mtune=native -fPIC
>>> -ffree-form -I/home/rayl/amber-master-fresh1/include
>>> -I/home/rayl/amber-master-fresh1/include -I../sander \
>>> -o timer.SANDER.o timer.F90
>>> mpif90 -DBINTRAJ -DEMIL -DMPI -c -O3 -mtune=native -fPIC
>>> -ffree-form -I/home/rayl/amber-master-fresh1/include
>>> -I/home/rayl/amber-master-fresh1/include -o decomp.o decomp.F90
>>> mpif90 -DBINTRAJ -DEMIL -DMPI -DSANDER -c -O3 -mtune=native -fPIC
>>> -ffree-form -I/home/rayl/amber-master-fresh1/include
>>> -I/home/rayl/amber-master-fresh1/include -I../sander \
>>> -o parms.SANDER.o parms.F90
>>> mpif90 -DBINTRAJ -DEMIL -DMPI -c -O3 -mtune=native -fPIC
>>> -ffree-form -I/home/rayl/amber-master-fresh1/include
>>> -I/home/rayl/amber-master-fresh1/include -o sa_driver.o
>>> sa_driver.F90
>>> make[4]: Entering directory
>>> `/home/rayl/amber-master-fresh1/AmberTools/src/sander'
>>> make[4]: `decomp.o' is up to date.
>>> make[4]: Leaving directory
>>> `/home/rayl/amber-master-fresh1/AmberTools/src/sander'
>>> cp -p ../sander/decomp.o ../pbsa
>>> cp -p ../sander/decomp.mod ../pbsa
>>> mpif90 -DBINTRAJ -DEMIL -DMPI -DSANDER -c -O3 -mtune=native -fPIC
>>> -ffree-form -I/home/rayl/amber-master-fresh1/include
>>> -I/home/rayl/amber-master-fresh1/include -I../sander \
>>> -o sa_driver.SANDER.o sa_driver.F90
>>> mpif90 -DBINTRAJ -DEMIL -DMPI -DSANDER -c -O3 -mtune=native -fPIC
>>> -ffree-form -I/home/rayl/amber-master-fresh1/include
>>> -I/home/rayl/amber-master-fresh1/include -I../sander \
>>> -o pb_p3m.SANDER.o pb_p3m.F90
>>> mpif90 -DBINTRAJ -DEMIL -DMPI -DSANDER -c -O3 -mtune=native -fPIC
>>> -ffree-form -I/home/rayl/amber-master-fresh1/include
>>> -I/home/rayl/amber-master-fresh1/include -I../sander \
>>> -o pbsa.SANDER.o pbsa.F90
>>> pbsa.F90:88.49:
>>>
>>> use parms, only : allocate_parms, clean_parms, bcast_parms, cn1, cn2
>>>
>>> 1
>>> Error: Symbol 'bcast_parms' referenced at (1) not found in module 'parms'
>>> make[3]: *** [pbsa.SANDER.o] Error 1
>>> make[3]: *** Waiting for unfinished jobs....
>>> make[3]: Leaving directory
>>> `/home/rayl/amber-master-fresh1/AmberTools/src/pbsa'
>>> make[2]: *** [libpbsa] Error 2
>>> make[2]: Leaving directory
>>> `/home/rayl/amber-master-fresh1/AmberTools/src/sander'
>>> make[1]: *** [parallel] Error 2
>>> make[1]: Leaving directory `/home/rayl/amber-master-fresh1/AmberTools/src'
>>> make: *** [install] Error 2
>>>
>>> --
>>> Ray Luo, Ph.D.
>>> Professor
>>> Biochemistry, Molecular Biophysics, Chemical Physics,
>>> Chemical and Biomedical Engineering
>>> University of California, Irvine, CA 92697-3900
>>>
>>>
>>> On Thu, May 26, 2016 at 1:53 PM, Gerald Monard
>>> <Gerald.Monard.univ-lorraine.fr> wrote:
>>>>
>>>> Ray,
>>>>
>>>> The patch was generated with git format-patch, so you should use git am
>>>> instead of git apply.
>>>>
>>>> Gerald.
>>>>
>>>> On 05/26/2016 10:49 PM, Ray Luo wrote:
>>>>>
>>>>> Gerald,
>>>>>
>>>>> Somehow I got the following error messages when applying it ...
>>>>>
>>>>> git apply
>>>>> ~/0001-libFpbsa-PBSA-library-for-sander-dependency-modifica.patch
>>>>>
>>>>> /home/rayl/0001-libFpbsa-PBSA-library-for-sander-dependency-modifica.patch:32:
>>>>> trailing whitespace.
>>>>> # in alphabetical order (GM/2016-05-26)
>>>>>
>>>>> /home/rayl/0001-libFpbsa-PBSA-library-for-sander-dependency-modifica.patch:33:
>>>>> trailing whitespace.
>>>>> SANDERPBSAOBJS= curv.o density.o dsvdc.o gen_dx_file.o GrToPr.o \
>>>>>
>>>>> /home/rayl/0001-libFpbsa-PBSA-library-for-sander-dependency-modifica.patch:34:
>>>>> trailing whitespace.
>>>>> iimod.o interpX.SANDER.o memory_module.o miniop.o
>>>>> np_force.SANDER.o \
>>>>>
>>>>> /home/rayl/0001-libFpbsa-PBSA-library-for-sander-dependency-modifica.patch:35:
>>>>> trailing whitespace.
>>>>> parms.SANDER.o pb_bldsys.o pb_crgview.o pb_direct.o pb_fddrv.o
>>>>> \
>>>>>
>>>>> /home/rayl/0001-libFpbsa-PBSA-library-for-sander-dependency-modifica.patch:36:
>>>>> trailing whitespace.
>>>>> pb_fdfrc.SANDER.o pb_force.SANDER.o pb_init.SANDER.o pb_list.o
>>>>> pb_lsolver.o \
>>>>> error: patch failed: AmberTools/src/pbsa/Makefile:67
>>>>> error: AmberTools/src/pbsa/Makefile: patch does not apply
>>>>> error: patch failed: AmberTools/src/pbsa/depend:37
>>>>> error: AmberTools/src/pbsa/depend: patch does not apply
>>>>> error: patch failed: AmberTools/src/pbsa/makedepend:16
>>>>> error: AmberTools/src/pbsa/makedepend: patch does not apply
>>>>>
>>>>> All the best,
>>>>> Ray
>>>>> --
>>>>> Ray Luo, Ph.D.
>>>>> Professor
>>>>> Biochemistry, Molecular Biophysics, Chemical Physics,
>>>>> Chemical and Biomedical Engineering
>>>>> University of California, Irvine, CA 92697-3900
>>>>>
>>>>>
>>>>> On Thu, May 26, 2016 at 1:30 PM, Ray Luo <rluo.uci.edu> wrote:
>>>>>>
>>>>>> Gerald,
>>>>>>
>>>>>> Will try your patch next ...
>>>>>>
>>>>>> Ray
>>>>>> --
>>>>>> Ray Luo, Ph.D.
>>>>>> Professor
>>>>>> Biochemistry, Molecular Biophysics, Chemical Physics,
>>>>>> Chemical and Biomedical Engineering
>>>>>> University of California, Irvine, CA 92697-3900
>>>>>>
>>>>>>
>>>>>> On Thu, May 26, 2016 at 1:18 PM, Gerald Monard
>>>>>> <Gerald.Monard.univ-lorraine.fr> wrote:
>>>>>>>
>>>>>>> Ray, Dan (+any volunteer): can you try (and test) that patch?
>>>>>>>
>>>>>>> I can do the same for LIBPBSA dependencies...;-)
>>>>>>>
>>>>>>> Gerald.
>>>>>>>
>>>>>>>
>>>>>>> On 05/26/2016 09:41 PM, Gerald Monard wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> Ray,
>>>>>>>>
>>>>>>>> Yes, my mistake: pb_force.o has a dependency for sa_driver.o.
>>>>>>>> But it's weirder(?) than this: if I track the compilation, a problem
>>>>>>>> appears for libFpbsa.a (pbsa library for sander). I can see this (in
>>>>>>>> serial order):
>>>>>>>>
>>>>>>>> gfortran [...] -DSANDER [...] -I../sander \ -o sa_driver.SANDER.o
>>>>>>>> sa_driver.F90
>>>>>>>> gfortran [...] -DSANDER [...] -I../sander \ -o pb_force.SANDER.o
>>>>>>>> pb_force.F90
>>>>>>>> gfortran [...] -DSANDER [...] -I../sander \ -o pb_p3m.SANDER.o
>>>>>>>> pb_p3m.F90
>>>>>>>> gfortran [...] -DSANDER [...] -I../sander \ -o interpX.SANDER.o
>>>>>>>> interpX.F90
>>>>>>>> gfortran [...] [...] -o pb_force.o pb_force.F90
>>>>>>>> gfortran [...] [...] -o pb_direct.o pb_direct.F90
>>>>>>>>
>>>>>>>> It means that pb_force is compiled twice (thus the error for the
>>>>>>>> parallel make: pb_direct needs pb_force that needs sa_driver which
>>>>>>>> will
>>>>>>>> create the solvent_accessibility module; while pb_force.SANDER.o
>>>>>>>> needs
>>>>>>>> sa_driver.SANDER.o that will create the solvent_accessibility
>>>>>>>> module!).
>>>>>>>>
>>>>>>>> Origin: makedepend has a definition for pb_direct.SANDER.o, but
>>>>>>>> Makefile
>>>>>>>> compiles pb_direct.o as requested by SANDERPBSAOBJS.
>>>>>>>> Two options
>>>>>>>> 1) It should be said pb_direct.SANDER.o in the Makefile (to be
>>>>>>>> consistent with makedepend)
>>>>>>>> 2) makedepend should not contain the $SANDER{"pb_direct"} directive
>>>>>>>> (to
>>>>>>>> be consistent with Makefile)
>>>>>>>>
>>>>>>>> I'm checking the discrepancy between Makefile and makedepend and will
>>>>>>>> commit a patch asap.
>>>>>>>>
>>>>>>>> Ray: you know more than me what is needed: to patch Makefile or to
>>>>>>>> patch
>>>>>>>> makedepend? (I guess it depends where #ifdef SANDER are present)
>>>>>>>>
>>>>>>>> Gerald.
>>>>>>>>
>>>>>>>> On 05/26/2016 09:21 PM, Ray Luo wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Gerald,
>>>>>>>>>
>>>>>>>>> Somehow in my "depend", as generated by "makedepend", "pb_force.o"
>>>>>>>>> lists "sa_driver.o" as one of the depends in all three cases, i.e.
>>>>>>>>> for
>>>>>>>>> PBSA, SANDER, and LIBPBSA ... I suppose "makedepend" is okay?
>>>>>>>>>
>>>>>>>>> All the best,
>>>>>>>>> Ray
>>>>>>>>> --
>>>>>>>>> Ray Luo, Ph.D.
>>>>>>>>> Professor
>>>>>>>>> Biochemistry, Molecular Biophysics, Chemical Physics,
>>>>>>>>> Chemical and Biomedical Engineering
>>>>>>>>> University of California, Irvine, CA 92697-3900
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, May 26, 2016 at 11:31 AM, Gerald Monard
>>>>>>>>> <Gerald.Monard.univ-lorraine.fr> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> FYI, based on Dan's error report: pbsa/pb_force.F90 needs the
>>>>>>>>>> solvent_accessibility module from sa_driver.F90, but makedepend
>>>>>>>>>> does not
>>>>>>>>>> recognize this dependency. So if you're lucky the first one get
>>>>>>>>>> compiled
>>>>>>>>>> before the second one (the files are in "correct" order in the
>>>>>>>>>> Makefile), if not, then you're doomed for a compile error.
>>>>>>>>>>
>>>>>>>>>> Gerald.
>>>>>>>>>>
>>>>>>>>>> On 05/26/2016 07:37 PM, Gerald Monard wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hello,
>>>>>>>>>>>
>>>>>>>>>>> This issue was raised some weeks/months ago (see thread
>>>>>>>>>>> [AMBER-Developers] Sander parallel build broken by PBSA updates).
>>>>>>>>>>> If
>>>>>>>>>>> it's still the same problem, it's a dependency problem in PBSA
>>>>>>>>>>> (makedepend not making its work correctly). You can't see it with
>>>>>>>>>>> make
>>>>>>>>>>> -j1 and it's a random problem with make -jX (X>1).
>>>>>>>>>>> I (partially) solved it last time. I'll check into it asap.
>>>>>>>>>>>
>>>>>>>>>>> Gerald.
>>>>>>>>>>>
>>>>>>>>>>> On 05/26/2016 07:23 PM, Ray Luo wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> First thing first, I always "git clean -xfd" before rebuilding.
>>>>>>>>>>>> I'll
>>>>>>>>>>>> try
>>>>>>>>>>>> different thead values to see whether I can reproduce the
>>>>>>>>>>>> problem.
>>>>>>>>>>>>
>>>>>>>>>>>> Also all object files for sander are separately defined and built
>>>>>>>>>>>> so
>>>>>>>>>>>> this
>>>>>>>>>>>> could be something higher up.
>>>>>>>>>>>>
>>>>>>>>>>>> Ray
>>>>>>>>>>>> On May 26, 2016 10:04 AM, "Daniel Roe" <daniel.r.roe.gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, May 26, 2016, Ray Luo wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Okay, I've just confirmed again that I can
>>>>>>>>>>>>> make -j 32 install
>>>>>>>>>>>>> if I turned off python building. I don't know too much how the
>>>>>>>>>>>>> python
>>>>>>>>>>>>> part is built ... Please take a look of the dependence relations
>>>>>>>>>>>>> related to python building.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Nope - still fails for me with '--skip-python'. It's almost
>>>>>>>>>>>> certainly
>>>>>>>>>>>> an issue with pbsa/libpbsa dependencies since that is where the
>>>>>>>>>>>> build
>>>>>>>>>>>> always fails for me ('make -j4'):
>>>>>>>>>>>>
>>>>>>>>>>>> ```
>>>>>>>>>>>> mpif90 -DBINTRAJ -DMPI -c -O3 -mtune=native -fPIC
>>>>>>>>>>>> -ffree-form
>>>>>>>>>>>> -I/home/droe/Amber/GIT/amber/include
>>>>>>>>>>>> -I/home/droe/Amber/GIT/amber/include -o pb_force.o
>>>>>>>>>>>> pb_force.F90
>>>>>>>>>>>> pb_force.F90:292:77:
>>>>>>>>>>>>
>>>>>>>>>>>> sa_init, sa_driver,
>>>>>>>>>>>> sa_free,
>>>>>>>>>>>> sa_free_mb, &
>>>>>>>>>>>>
>>>>>>>>>>>> 1
>>>>>>>>>>>> Error: Symbol ‘saslave_init’ referenced at (1) not found in
>>>>>>>>>>>> module
>>>>>>>>>>>> ‘solvent_accessibility’
>>>>>>>>>>>> Makefile:226: recipe for target 'pb_force.o' failed
>>>>>>>>>>>> make[3]: *** [pb_force.o] Error 1
>>>>>>>>>>>> ```
>>>>>>>>>>>>
>>>>>>>>>>>> -Dan
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> -------------------------
>>>>>>>>>>>> Daniel R. Roe, PhD
>>>>>>>>>>>> Department of Medicinal Chemistry
>>>>>>>>>>>> University of Utah
>>>>>>>>>>>> 30 South 2000 East, Room 307
>>>>>>>>>>>> Salt Lake City, UT 84112-5820
>>>>>>>>>>>> http://home.chpc.utah.edu/~cheatham/
>>>>>>>>>>>> (801) 587-9652
>>>>>>>>>>>> (801) 585-6208 (Fax)
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> AMBER-Developers mailing list
>>>>>>>>>>>> AMBER-Developers.ambermd.org
>>>>>>>>>>>> http://lists.ambermd.org/mailman/listinfo/amber-developers
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> AMBER-Developers mailing list
>>>>>>>>>>>> AMBER-Developers.ambermd.org
>>>>>>>>>>>> http://lists.ambermd.org/mailman/listinfo/amber-developers
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ____________________________________________________________________________
>>>>>>>>>>
>>>>>>>>>> Prof. Gerald MONARD
>>>>>>>>>> SRSMC, Université de Lorraine, CNRS
>>>>>>>>>> Boulevard des Aiguillettes B.P. 70239
>>>>>>>>>> F-54506 Vandoeuvre-les-Nancy, FRANCE
>>>>>>>>>>
>>>>>>>>>> e-mail : Gerald.Monard.univ-lorraine.fr
>>>>>>>>>> tel. : +33 (0)383.684.381
>>>>>>>>>> fax : +33 (0)383.684.371
>>>>>>>>>> web : http://www.monard.info
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ____________________________________________________________________________
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> AMBER-Developers mailing list
>>>>>>>>>> AMBER-Developers.ambermd.org
>>>>>>>>>> http://lists.ambermd.org/mailman/listinfo/amber-developers
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> AMBER-Developers mailing list
>>>>>>>>> AMBER-Developers.ambermd.org
>>>>>>>>> http://lists.ambermd.org/mailman/listinfo/amber-developers
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> ____________________________________________________________________________
>>>>>>>
>>>>>>> Prof. Gerald MONARD
>>>>>>> SRSMC, Université de Lorraine, CNRS
>>>>>>> Boulevard des Aiguillettes B.P. 70239
>>>>>>> F-54506 Vandoeuvre-les-Nancy, FRANCE
>>>>>>>
>>>>>>> e-mail : Gerald.Monard.univ-lorraine.fr
>>>>>>> tel. : +33 (0)383.684.381
>>>>>>> fax : +33 (0)383.684.371
>>>>>>> web : http://www.monard.info
>>>>>>>
>>>>>>>
>>>>>>> ____________________________________________________________________________
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> AMBER-Developers mailing list
>>>>>>> AMBER-Developers.ambermd.org
>>>>>>> http://lists.ambermd.org/mailman/listinfo/amber-developers
>>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> AMBER-Developers mailing list
>>>>> AMBER-Developers.ambermd.org
>>>>> http://lists.ambermd.org/mailman/listinfo/amber-developers
>>>>>
>>>>
>>>> --
>>>>
>>>> ____________________________________________________________________________
>>>>
>>>> Prof. Gerald MONARD
>>>> SRSMC, Université de Lorraine, CNRS
>>>> Boulevard des Aiguillettes B.P. 70239
>>>> F-54506 Vandoeuvre-les-Nancy, FRANCE
>>>>
>>>> e-mail : Gerald.Monard.univ-lorraine.fr
>>>> tel. : +33 (0)383.684.381
>>>> fax : +33 (0)383.684.371
>>>> web : http://www.monard.info
>>>>
>>>>
>>>> ____________________________________________________________________________
>>>>
>>>>
>>>> _______________________________________________
>>>> AMBER-Developers mailing list
>>>> AMBER-Developers.ambermd.org
>>>> http://lists.ambermd.org/mailman/listinfo/amber-developers
>>>
>>>
>>> _______________________________________________
>>> AMBER-Developers mailing list
>>> AMBER-Developers.ambermd.org
>>> http://lists.ambermd.org/mailman/listinfo/amber-developers
>>>
>>
>> --
>> ____________________________________________________________________________
>>
>> Prof. Gerald MONARD
>> SRSMC, Université de Lorraine, CNRS
>> Boulevard des Aiguillettes B.P. 70239
>> F-54506 Vandoeuvre-les-Nancy, FRANCE
>>
>> e-mail : Gerald.Monard.univ-lorraine.fr
>> tel. : +33 (0)383.684.381
>> fax : +33 (0)383.684.371
>> web : http://www.monard.info
>>
>> ____________________________________________________________________________
>>
>>
>> _______________________________________________
>> AMBER-Developers mailing list
>> AMBER-Developers.ambermd.org
>> http://lists.ambermd.org/mailman/listinfo/amber-developers
>>
>
> _______________________________________________
> AMBER-Developers mailing list
> AMBER-Developers.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber-developers
>

-- 
____________________________________________________________________________
  Prof. Gerald MONARD
  SRSMC, Université de Lorraine, CNRS
  Boulevard des Aiguillettes B.P. 70239
  F-54506 Vandoeuvre-les-Nancy, FRANCE
  e-mail : Gerald.Monard.univ-lorraine.fr
  tel.   : +33 (0)383.684.381
  fax    : +33 (0)383.684.371
  web    : http://www.monard.info
____________________________________________________________________________
_______________________________________________
AMBER-Developers mailing list
AMBER-Developers.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber-developers
Received on Thu May 26 2016 - 15:30:02 PDT
Custom Search