Hi Kellon,
I agree I don't see any issues with initializing the array during setup. There is a performance hit to initializing arrays when one doesn't need to but if it only occurs once during startup it should not be an issue. The issue is if such initialization occurs every step then it would be a bad idea. The IEEE warning is innocuous but if zeroing the array at startup immediately after allocating (I.e. doing it always not just when running with MPI) get rids of that warning then it has my vote.
All the best
Ross
> On Nov 9, 2018, at 14:28, Kellon Belfon <kellonbelfon.gmail.com> wrote:
>
> Thank you Josh and Ross for your response, advise and suggestions. I
> definitely appreciate your time and wisdom.
>
> This is what I gathered:
> (1) atm_frc is allocated,
> (2) values from atm_frc is uploaded to the GPU to populate pForce array
> during initial setup,
> (3) Whenever gpu_download_frc is called (mostly for MPI), then atm_frc is
> repopulated with the actual forces from pForce array.
> *gpu_download_frc is not called in my runs
>
> But it appears that the garbage values in the atm_frc array get overwritten
> by the actual force during the download_frc call, so it will not affect the
> calculations if atm_frc is not initialize. Therefore it is not really a
> bug. I was thinking it might be just a good approach to initialize the
> array to get rid of the overflow note.
>
> I ran the amber tests after initializing the atm_frc array to zero and they
> all passed. Also the garbage values in the atm_frc array are values that
> were previously stored at that memory and not from the current
> calculations.
>
> Respectfully,
>
> Kellon
>
> On Thu, Nov 8, 2018 at 10:24 AM Ross Walker <ross.rosswalker.co.uk> wrote:
>
>> I am not sure that's a bug. The atm_frc array needs to be initialized to
>> zero for MPI because it will do an mpi_reduce into that array so not every
>> element will be touched by every core and it will do an addition to the
>> existing element rather than an assignment. When running in serial values
>> are assigned to each atm_frc element rather than added/reduced and thus it
>> should not be necessary to initialize the array.
>>
>> I would trace through the code and find out exactly where values are
>> stored into the atm_frc array in serial to determine if the initialization
>> is needed.
>>
>> All the best
>> Ross
>>
>>> On Nov 8, 2018, at 00:16, Josh Berryman <
>> the.real.josh.berryman.gmail.com> wrote:
>>>
>>>>> I looked at the code and found that atm_frc is only initialized to
>> zero
>>> when MPI is called (inpcrd_dat.F90:481: atm_frc(:,:) = 0.d0 and
>>> parallel.F90:2146: atm_frc(:,:) = 0.d0).
>>> Well that looks like a classic memory management bug that you should
>> commit
>>> a fix for.
>>>
>>> If you are mailing the developer's list then I guess that means you have
>>> gitlab access?
>>>
>>> Before committing, maybe send an email to whoever else has been
>> committing
>>> the most to pmemd.F90 recently, but basically it looks as if you have
>>> sorted this out yourself.
>>>
>>> Josh
>>>
>>>
>>>
>>>
>>>
>>> On Wed, 7 Nov 2018 at 23:07, Kellon Belfon <kellonbelfon.gmail.com>
>> wrote:
>>>
>>>> Follow up to my previous email.
>>>>
>>>> I set atm_frc array to be zero to test the whether the large value in
>> the
>>>> atm_frc array is garbage values/old values.
>>>>
>>>> To do this, at line 414 in pmemd.F90 I added the following line
>>>> atm_frc(:,:) = 0.d0
>>>>
>>>> Then I ran the calculations in triplicate of 10 times and I did not get
>> the
>>>> overflow error anymore. I think this suggests that the large values in
>> the
>>>> atm_frc array might be garbage values since atm_frc is not initialized
>> to
>>>> be zero?
>>>> I looked at the code and found that atm_frc is only initialized to zero
>>>> when MPI is called (inpcrd_dat.F90:481: atm_frc(:,:) = 0.d0 and
>>>> parallel.F90:2146: atm_frc(:,:) = 0.d0).
>>>>
>>>> Also, I compared the mdout files with and without the atm_frc(:,:) =
>> 0.d0
>>>> at line 414 in pmemd.F90 and the results are the same for the system I
>>>> tested. I am not sure if this will break anything else. I can run the
>> amber
>>>> standard test case, if this might help?
>>>>
>>>> Respectfully,
>>>>
>>>> Kellon
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, Nov 7, 2018 at 1:40 PM Kellon Belfon <kellonbelfon.gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Josh,
>>>>>
>>>>> Thanks for the response. I think in this case it is not because of my
>>>>> system. We have tested other systems in our lab and all of these
>> systems
>>>>> give the same overflow notes sometime. This happens during production
>>>> runs
>>>>> too and the behavior is random.
>>>>>
>>>>> The overflow only occurs at initialization when the force array should
>> be
>>>>> zero. Displaying the values show that the array is mostly zero except
>>>> for a
>>>>> sporadic large number in between. This is not always on the same atom
>> and
>>>>> doing ten runs in triplicate showed the value is zero sometimes and
>>>>> sometimes it is this large number. I am thinking this might just be
>>>> garbage
>>>>> values or old values from previous use of the register. Is this a
>> normal
>>>>> behavior of the code or have you ever seen this issue before?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Kellon
>>>>>
>>>>>
>>>>> On Wed, Nov 7, 2018 at 3:13 AM Josh Berryman <
>>>>> the.real.josh.berryman.gmail.com> wrote:
>>>>>
>>>>>> Hi Kellon, if you are getting forces in the region 1e29 then your
>> system
>>>>>> has severe steric clashes in it: the main problem with Lennard-Jones
>> for
>>>>>> non-bonded interactions is that it diverges quickly for close approach
>>>> of
>>>>>> atoms. Probably the answer in that case is to run for 1-10ps with
>> very
>>>>>> high Langevin coupling and small timestep on the CPU, or to use xmin
>>>>>> option
>>>>>> to pre-stabilise your system (again on the CPU, where the 64 bit
>>>> datatype
>>>>>> for floats will give you more headroom against overflows).
>>>>>>
>>>>>> google suggests that (as you say you have seen yourself)
>> IEEE_DENORMAL
>>>> is
>>>>>> not a problem, it describes cases where a number comes close enough to
>>>>>> zero
>>>>>> that maybe it should just be rounded to zero anyway
>>>>>>
>>>>>> Josh
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, 6 Nov 2018 at 20:17, Kellon Belfon <kellonbelfon.gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Everyone,
>>>>>>>
>>>>>>> We recently upgraded our compiler (gnu 4.8.4 to 7.3.0) on our cluster
>>>>>> and
>>>>>>> started getting the following note for our gpu calculations in Amber.
>>>>>>> Note: The following floating-point exceptions are signalling:
>>>>>>> IEEE_UNDERFLOW_FLAG IEEE_DENORMAL
>>>>>>>
>>>>>>>> From this previous post (
>> http://archive.ambermd.org/201804/0130.html
>>>> ),
>>>>>>> the
>>>>>>> response was pretty much do not worry about them. Does this apply to
>>>>>>> IEEE_OVERFLOW_FLAG as well?
>>>>>>> Also running the calculation with pmemd.cuda_DPFP does not produce
>> the
>>>>>>> underflow note. I was thinking maybe it is from mixing floats and
>>>>>> doubles?
>>>>>>>
>>>>>>> We are getting the overflow note for some of our calculations, but it
>>>>>> does
>>>>>>> not affect the results. I also used cuda-gdb using -G -g
>>>>>>> -ffpe-trap=overflow flags, to stop the code where the overflow
>>>> occurs. I
>>>>>>> found that the overflow occurs in gpu_upload_frc(), during the first
>>>>>> upload
>>>>>>> of the forces as the system is being initialized on the GPU.
>>>>>>> Further debugging showed the note occurs when the atm_frc array has
>>>>>> values
>>>>>>> that are not zero but instead a large number (atm_frc[i][1] =
>>>>>>> -1.5739204096161189e+29). I think this large number causes the
>>>> overflow
>>>>>>> note since the calculation fails right after (The calculation fails
>>>>>> because
>>>>>>> I set the ffpe-trap).
>>>>>>>
>>>>>>> I then ran the same calculation ten times (with the ffpe-trap, if
>>>> there
>>>>>> is
>>>>>>> an overflow the calculations will fail) and I get the overflow 4 out
>>>> of
>>>>>> 10
>>>>>>> time. Then I repeat for another 10 times in triplicate (2/10, 5/10,
>>>> 1/10
>>>>>>> overflow). It seem like an unpredictable note. For really small
>>>> numbers,
>>>>>>> multiplying by the forcescale does the trick but for these large
>>>>>> numbers it
>>>>>>> is causing the overflow note and the behavior is unpredictable. Does
>>>>>> anyone
>>>>>>> has any advice on this? Should we just ignore, since the results are
>>>>>> okay?
>>>>>>>
>>>>>>> Below are the results for one of the trial:
>>>>>>> *Run 1:*
>>>>>>> Program received signal SIGFPE: Floating-point exception - erroneous
>>>>>>> arithmetic operation.
>>>>>>> Backtrace for this error:
>>>>>>> #0 0x7fb6597912da in ???
>>>>>>> #1 0x7fb659790503 in ???
>>>>>>> #2 0x7fb658a84f1f in ???
>>>>>>> #3 0x556a6ac73f76 in gpu_upload_frc_
>>>>>>> at /mnt/raidc2/kbelfon/amber18/src/pmemd/src/cuda/gpu.cpp:1675
>>>>>>> #4 0x556a6abeb897 in pmemd
>>>>>>> at /mnt/raidc2/kbelfon/amber18/src/pmemd/src/pmemd.F90:445
>>>>>>> #5 0x556a6abecbb3 in main
>>>>>>> at /mnt/raidc2/kbelfon/amber18/src/pmemd/src/pmemd.F90:77
>>>>>>> run_direct.sh: line 19: 12686 Floating point exception(core dumped)
>>>>>>> ${pmemdGPU} -O -i ./test.in -p ./5awl.opc.parm7 -c ./8md.rst7 -ref
>>>>>>> ./8md.rst7 -o ./md1.out -x ./md1.x -inf ./md1.info -r ./md1.rst7
>>>>>>> *Run 2: *
>>>>>>> Program received signal SIGFPE: Floating-point exception - erroneous
>>>>>>> arithmetic operation.
>>>>>>> Backtrace for this error:
>>>>>>> #0 0x7f8780b802da in ???
>>>>>>> #1 0x7f8780b7f503 in ???
>>>>>>> #2 0x7f877fe73f1f in ???
>>>>>>> #3 0x557f1d950f76 in gpu_upload_frc_
>>>>>>> at /mnt/raidc2/kbelfon/amber18/src/pmemd/src/cuda/gpu.cpp:1675
>>>>>>> #4 0x557f1d8c8897 in pmemd
>>>>>>> at /mnt/raidc2/kbelfon/amber18/src/pmemd/src/pmemd.F90:445
>>>>>>> #5 0x557f1d8c9bb3 in main
>>>>>>> at /mnt/raidc2/kbelfon/amber18/src/pmemd/src/pmemd.F90:77
>>>>>>> run_direct.sh: line 20: 12691 Floating point exception(core dumped)
>>>>>>> ${pmemdGPU} -O -i ./test.in -p ./5awl.opc.parm7 -c ./8md.rst7 -ref
>>>>>>> ./8md.rst7 -o ./md1.out -x ./md1.x -inf ./md1.info -r ./md1.rst7
>>>>>>> *Run 3:*
>>>>>>> Note: The following floating-point exceptions are signalling:
>>>>>>> IEEE_INVALID_FLAG IEEE_UNDERFLOW_FLAG IEEE_DENORMAL
>>>>>>> *Run 4:*
>>>>>>> Note: The following floating-point exceptions are signalling:
>>>>>>> IEEE_INVALID_FLAG IEEE_UNDERFLOW_FLAG IEEE_DENORMAL
>>>>>>> *Run 5:*
>>>>>>> Note: The following floating-point exceptions are signalling:
>>>>>>> IEEE_INVALID_FLAG IEEE_UNDERFLOW_FLAG IEEE_DENORMAL
>>>>>>> *Run 6:*
>>>>>>> Program received signal SIGFPE: Floating-point exception - erroneous
>>>>>>> arithmetic operation.
>>>>>>> Backtrace for this error:
>>>>>>> #0 0x7fa7cca442da in ???
>>>>>>> #1 0x7fa7cca43503 in ???
>>>>>>> #2 0x7fa7cbd37f1f in ???
>>>>>>> #3 0x559568d8cf76 in gpu_upload_frc_
>>>>>>> at /mnt/raidc2/kbelfon/amber18/src/pmemd/src/cuda/gpu.cpp:1675
>>>>>>> #4 0x559568d04897 in pmemd
>>>>>>> at /mnt/raidc2/kbelfon/amber18/src/pmemd/src/pmemd.F90:445
>>>>>>> #5 0x559568d05bb3 in main
>>>>>>> at /mnt/raidc2/kbelfon/amber18/src/pmemd/src/pmemd.F90:77
>>>>>>> run_direct.sh: line 24: 12708 Floating point exception(core dumped)
>>>>>>> ${pmemdGPU} -O -i ./test.in -p ./5awl.opc.parm7 -c ./8md.rst7 -ref
>>>>>>> ./8md.rst7 -o ./md1.out -x ./md1.x -inf ./md1.info -r ./md1.rst7
>>>>>>> *Run 7: *
>>>>>>> Note: The following floating-point exceptions are signalling:
>>>>>>> IEEE_INVALID_FLAG IEEE_UNDERFLOW_FLAG IEEE_DENORMAL
>>>>>>> *Run 8:*
>>>>>>> Program received signal SIGFPE: Floating-point exception - erroneous
>>>>>>> arithmetic operation.
>>>>>>> Backtrace for this error:
>>>>>>> #0 0x7f32888232da in ???
>>>>>>> #1 0x7f3288822503 in ???
>>>>>>> #2 0x7f3287b16f1f in ???
>>>>>>> #3 0x55ae7dec2f13 in gpu_upload_frc_
>>>>>>> at /mnt/raidc2/kbelfon/amber18/src/pmemd/src/cuda/gpu.cpp:1673
>>>>>>> #4 0x55ae7de3a897 in pmemd
>>>>>>> at /mnt/raidc2/kbelfon/amber18/src/pmemd/src/pmemd.F90:445
>>>>>>> #5 0x55ae7de3bbb3 in main
>>>>>>> at /mnt/raidc2/kbelfon/amber18/src/pmemd/src/pmemd.F90:77
>>>>>>> run_direct.sh: line 26: 12717 Floating point exception(core dumped)
>>>>>>> ${pmemdGPU} -O -i ./test.in -p ./5awl.opc.parm7 -c ./8md.rst7 -ref
>>>>>>> ./8md.rst7 -o ./md1.out -x ./md1.x -inf ./md1.info -r ./md1.rst7
>>>>>>> *Run 9:*
>>>>>>> Note: The following floating-point exceptions are signalling:
>>>>>>> IEEE_UNDERFLOW_FLAG IEEE_DENORMAL
>>>>>>> *Run 10:*
>>>>>>> Note: The following floating-point exceptions are signalling:
>>>>>>> IEEE_INVALID_FLAG IEEE_UNDERFLOW_FLAG IEEE_DENORMAL
>>>>>>>
>>>>>>> Thank you!
>>>>>>>
>>>>>>> Respectfully,
>>>>>>>
>>>>>>> Kellon
>>>>>>> _______________________________________________
>>>>>>> AMBER-Developers mailing list
>>>>>>> AMBER-Developers.ambermd.org
>>>>>>> http://lists.ambermd.org/mailman/listinfo/amber-developers
>>>>>>>
>>>>>> _______________________________________________
>>>>>> AMBER-Developers mailing list
>>>>>> AMBER-Developers.ambermd.org
>>>>>> http://lists.ambermd.org/mailman/listinfo/amber-developers
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Kellon A. A. Belfon, Graduate Student
>>>> Carlos Simmerling Laboratory
>>>> The Laufer Center for Physical and Quantitative Biology
>>>> The Department of Chemistry, Stony Brook University
>>>> Stony Brook, New York 11794
>>>> Phone: (347) 546-4237 <(347)+546+4237> Email:
>> kellon.belfon.stonybrook.
>>>> <kellon.belfon.stonybrook.edu>edu
>>>> _______________________________________________
>>>> AMBER-Developers mailing list
>>>> AMBER-Developers.ambermd.org
>>>> http://lists.ambermd.org/mailman/listinfo/amber-developers
>>>>
>>> _______________________________________________
>>> AMBER-Developers mailing list
>>> AMBER-Developers.ambermd.org
>>> http://lists.ambermd.org/mailman/listinfo/amber-developers
>>
>>
>> _______________________________________________
>> AMBER-Developers mailing list
>> AMBER-Developers.ambermd.org
>> http://lists.ambermd.org/mailman/listinfo/amber-developers
>>
>
>
> --
> Kellon A. A. Belfon, Graduate Student
> Carlos Simmerling Laboratory
> The Laufer Center for Physical and Quantitative Biology
> The Department of Chemistry, Stony Brook University
> Stony Brook, New York 11794
> Phone: (347) 546-4237 <(347)+546+4237> Email: kellon.belfon.stonybrook.
> <kellon.belfon.stonybrook.edu>edu
> _______________________________________________
> AMBER-Developers mailing list
> AMBER-Developers.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber-developers
_______________________________________________
AMBER-Developers mailing list
AMBER-Developers.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber-developers
Received on Fri Nov 09 2018 - 17:30:02 PST
