Yes, you want to do it before the OrientForcesKernel to avoid the troubles
caused by extra points.
My implementation of TI calls PME twice each iteration--what you only need
is do the netfrc stuff right after each gradsum call, that should be it.
Taisung
-----Original Message-----
From: Scott Le Grand [mailto:varelse2005.gmail.com]
Sent: Friday, June 26, 2020 8:02 PM
To: AMBER Developers Mailing List <amber-developers.ambermd.org>
Subject: Re: [AMBER-Developers] Faster NetFrc
Got sidetracked, back on this full-time...
So OK, if there are no extra points, I'm doing it in the beginning of the
update kernel.
But if extra points, I need to do it before the OrientForcesKernel?
So, what about Softcore TI? Do I need to do separate corrections for each TI
region? Are we doing separate gradsums there as well? Because I'm thinking
separate update regions too get that playing nice.
The first cut which is good for neither special case is checked into the
branch sl_netfrc. It's 7% faster for JAC NVE, and 1-3% faster for everything
else I benchmarked.
This is still a WIP only for single GPU ATM. I will fix that shortly. I want
to get all the special cases running single GPU before tackling that.
Scott
On Fri, May 22, 2020 at 3:06 PM David Cerutti <dscerutti.gmail.com> wrote:
> I agree with Taisung's idea--doing the netfrc correction immediately
> after (or even in the same kernel as) the PME gradient sum will have
> it happen before any spreading occurs, so any further netfrc
> accumulation after that will be solely a function of 1.) conserved
> forces that automagically... or
> 2.) fp32 roundoff error, if there is any, in the kOrientForces() kernel.
> (In my new frame types there are some places where I revert to fp32
> after getting displacements down to the level of local coordinate
> frames, as many more registers may be needed for several of the new
> frame styles.)
>
> Dave
>
>
> On Fri, May 22, 2020 at 5:41 PM Scott Le Grand <varelse2005.gmail.com>
> wrote:
>
> > Yep. there will be FPRE differences. We'll need to validate. I just
> > don't want to break the science.
> >
> > On Fri, May 22, 2020 at 2:16 PM <taisung.gmail.com> wrote:
> >
> > > Actually, I don't think you need to deal w/ the extra points at
> > > all if
> > you
> > > do netfrc right after PME gradsum. Just do the total accumulate
> > > and
> get
> > > rid
> > > of the resulting net forces for all atoms (including extra points).
> The
> > > forces on extra points will be transferred to nearby heavy atoms
> > > (hence become zero) in a later stage.
> > >
> > > The results could be tiny different though. But theoretically I
> believe
> > it
> > > is OK.
> > >
> > > Taisung
> > >
> > > -----Original Message-----
> > > From: Scott Le Grand [mailto:varelse2005.gmail.com]
> > > Sent: Friday, May 22, 2020 4:31 PM
> > > To: AMBER Developers Mailing List <amber-developers.ambermd.org>
> > > Subject: Re: [AMBER-Developers] Faster NetFrc
> > >
> > > So if I accumulate them from the extra points in gradsum, but then
> > > only apply to atoms that are not extra points, we are good?
> > >
> > > On Fri, May 22, 2020 at 1:29 PM David Cerutti
> > > <dscerutti.gmail.com>
> > wrote:
> > >
> > > > Our emails may have crossed in writing, but it appears that in
> > > > both CPU and GPU codes the netfrc is being subtracted after
> > > > extra points have had their forces transmitted to their massive
> > > > frame atoms. The solutions to dealing with the pollution that
> > > > netfrc correction causes with extra points differ in each code.
> > > >
> > > > Dave
> > > >
> > > >
> > > > On Fri, May 22, 2020 at 4:27 PM Scott Le Grand <
> varelse2005.gmail.com>
> > > > wrote:
> > > >
> > > > > Cool, and if it's an extra point should I or should I not
> > > > > subtract the netfrc? Dumb questions asked upfront to save time
later?
> > > > >
> > > > > On Fri, May 22, 2020 at 1:20 PM <cancersimulation.gmail.com>
> wrote:
> > > > >
> > > > > > It is static. Usually, it should be the same as "numextra"
> > > > > > here
> > > > > >
> > > > > > use prmtop_dat_mod, only : numextra
> > > > > >
> > > > > > I believe Dave Cerutti has implemented some extra-point
> > > > > > stuff and I
> > > > don't
> > > > > > know if the above statement still holds with his stuff.
> > > > > >
> > > > > > Taisung
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Scott Le Grand [mailto:varelse2005.gmail.com]
> > > > > > Sent: Friday, May 22, 2020 3:02 PM
> > > > > > To: AMBER Developers Mailing List
> > > > > > <amber-developers.ambermd.org>
> > > > > > Subject: Re: [AMBER-Developers] Faster NetFrc
> > > > > >
> > > > > > So also...
> > > > > >
> > > > > > I would assume the value of "ignored" is static. How do I
> > > > > > calculate it
> > > > a
> > > > > > priori? And it looks like I don't apply it to the extra points.
> > > > > > Feel
> > > > free
> > > > > > to
> > > > > > describe off-list, but I want to drill down deep and get
> > > > > > this right on
> > > > > the
> > > > > > first swing...
> > > > > >
> > > > > > On Thu, May 21, 2020 at 6:08 PM Scott Le Grand
> > > > > > <varelse2005.gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Excellent, that makes this much more straightforward.
> > > > > > > Should have something by early next week.
> > > > > > >
> > > > > > > On Thu, May 21, 2020 at 6:04 PM <taisung.gmail.com> wrote:
> > > > > > >
> > > > > > >> The force thresholds are for those "dummies" (not the
> > > > > > >> alchemical dummies but something like lone-pair points of
> water
> > > > > > >> models). The forces of those atoms are "transferred" to
> > > > > > >> other real atoms at the final force collection stage--and
> > > > > > >> hence need to be kept to zero during the netfrc stage.
> > > > > > >> Of course, you
> may
> > > > > > >> find better ways to do things. For example, as Scott
> > > > > > >> mentioned, the only non-conserved force part is PME
> > > > > > >> reciprocal part. If the netfrc is done in the PME
> > > > > > >> reciprocal part, there
> > is
> > > no need to have such force thresholds.
> > > > > > >>
> > > > > > >> Taisung
> > > > > > >>
> > > > > > >> -----Original Message-----
> > > > > > >> From: David Cerutti [mailto:dscerutti.gmail.com]
> > > > > > >> Sent: Thursday, May 21, 2020 8:37 PM
> > > > > > >> To: AMBER Developers Mailing List
> > > > > > >> <amber-developers.ambermd.org>
> > > > > > >> Subject: Re: [AMBER-Developers] Faster NetFrc
> > > > > > >>
> > > > > > >> As implied, the GTI code is the revision that introduced
this.
> > > > > > >> Taisung can comment more on his logic, but the presence
> > > > > > >> of
> this
> > > > > > >> "small" term reminds me of something he's got in the
> non-bonded
> > > > inner
> > > > > > >> loop as well. I'm not sure we ever determined why these
> > > > conditionals
> > > > > > >> were needed; I think the one in the non-bonded loop
> > > > > > >> should
> just
> > > > > > >> go away after some other revisions I made, but I'll wait
> > > > > > >> for more
> > > > input.
> > > > > > >>
> > > > > > >> Dave
> > > > > > >>
> > > > > > >>
> > > > > > >> On Thu, May 21, 2020 at 8:29 PM Scott Le Grand
> > > > > > >> <varelse2005.gmail.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > Hey guys,
> > > > > > >> > Back as an AMBER developer and one of the first things
> > > > > > >> > I'd like to do is optimize netfrc.
> > > > > > >> >
> > > > > > >> > Fixed point conservative forces have no net force. They
> > > > > > >> > automagically cancel out 100%. So we can ignore
> > > > > > >> > everything except non-conserved
> > > > > > >> forces.
> > > > > > >> >
> > > > > > >> > The only non-conserved force I know of is the Ewald
> > > > > > >> > Gradient
> > > Sum.
> > > > > > >> > So if I add up the net force there, and handle it upon
> either
> > > > force
> > > > > > >> > reduction or update, I can eliminate two passes on
> > > > > > >> > memory
> and
> > > > > > >> > the two kernels currently dedicated to adding and then
> > > > > > >> > subtracting
> > > > them.
> > > > > > >> >
> > > > > > >> > But... Are there any other non-conserved forces?
> > > > > > >> >
> > > > > > >> > And... It looks like the FORTRAN and the CUDA code do
> > > > > > >> > different
> > > > > > things.
> > > > > > >> > FORTRAN simply averages the forces and subtracts. But
> > > > > > >> > CUDA only averages over atoms with force thresholds
> > > > > > >> > above a predefined small amount. What's up with that?
> > > > > > >> >
> > > > > > >> > pme_ene.F90:
> > > > > > >> > if (netfrc .gt. 0 .and. onstep) then
> > > > > > >> >
> > > > > > >> > if (ti_mode .eq. 0) then
> > > > > > >> > do i = 1, atm_cnt
> > > > > > >> > net_frcs(:) = net_frcs(:) + frc(:, i)
> > > > > > >> > end do
> > > > > > >> >
> > > > > > >> > ! Now do the correction:
> > > > > > >> >
> > > > > > >> > net_frcs(:) = net_frcs(:) / dble(atm_cnt -
> > > > > > >> > numextra)
> > > > > > >> >
> > > > > > >> > do i = 1, atm_cnt
> > > > > > >> > frc(:, i) = frc(:, i) - net_frcs(:)
> > > > > > >> > end do
> > > > > > >> > else
> > > > > > >> > do i = 1, atm_cnt
> > > > > > >> > ti_net_frcs(1, :) = ti_net_frcs(1, :) +
> > > > > > >> > ti_nb_frc(1, :,
> > > > i)
> > > > > > >> > ti_net_frcs(2, :) = ti_net_frcs(2, :) +
> > > > > > >> > ti_nb_frc(2, :,
> > > > i)
> > > > > > >> > end do
> > > > > > >> >
> > > > > > >> > ti_net_frcs(1,:) =
> > > > > > >> > ti_net_frcs(1,:)/dble(ti_atm_cnt(1)-ti_numextra_pts(1))
> > > > > > >> > ti_net_frcs(2,:) =
> > > > > > >> > ti_net_frcs(2,:)/dble(ti_atm_cnt(2)-ti_numextra_pts(2))
> > > > > > >> > net_frcs(:) = ti_net_frcs(1,:) +
> > > > > > >> > ti_net_frcs(2,:)
> > > > > > >> >
> > > > > > >> > do i = 1, atm_cnt
> > > > > > >> > ! This matches how sander removes netfrcs in
> > > > > > >> > TI
> runs
> > > > > > >> > if (ti_lst(1,i) .ne. 0) then
> > > > > > >> > frc(:, i) = frc(:, i) - ti_net_frcs(1,:)
> > > > > > >> > else if (ti_lst(2,i) .ne. 0) then
> > > > > > >> > frc(:, i) = frc(:, i) - ti_net_frcs(2,:)
> > > > > > >> > else
> > > > > > >> > frc(:, i) = frc(:, i) - net_frcs(:)
> > > > > > >> > end if
> > > > > > >> > end do
> > > > > > >> > end if
> > > > > > >> > ! Any extra points must have their 0.d0 forces
> reset...
> > > > > > >> >
> > > > > > >> > if (numextra .gt. 0 .and. frameon .ne. 0) &
> > > > > > >> > call zero_extra_pnts_vec(frc, ep_frames,
> > > > > > >> > gbl_frame_cnt)
> > > > > > >> >
> > > > > > >> > end if
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > GTI path:
> > > > > > >> > while (pos < cSim.atoms) {
> > > > > > >> > PMEFloat fx = converter(pX[pos], ONEOVERFORCESCALE);
> > > > > > >> > PMEFloat fy = converter(pY[pos], ONEOVERFORCESCALE);
> > > > > > >> > PMEFloat fz = converter(pZ[pos], ONEOVERFORCESCALE);
> > > > > > >> > if (abs(fx) > small || abs(fy) > small || abs(fz) >
> > small) {
> > > > > > >> > pX[pos] -= nfX;
> > > > > > >> > pY[pos] -= nfY;
> > > > > > >> > pZ[pos] -= nfZ;
> > > > > > >> > }
> > > > > > >> > pos += increment;
> > > > > > >> > }
> > > > > > >> >
> > > > > > >> > Scott
> > > > > > >> > _______________________________________________
> > > > > > >> > AMBER-Developers mailing list
> > > > > > >> > AMBER-Developers.ambermd.org
> > > > > > >> > http://lists.ambermd.org/mailman/listinfo/amber-develop
> > > > > > >> > ers
> > > > > > >> >
> > > > > > >> _______________________________________________
> > > > > > >> AMBER-Developers mailing list
> > > > > > >> AMBER-Developers.ambermd.org
> > > > > > >> http://lists.ambermd.org/mailman/listinfo/amber-developer
> > > > > > >> s
> > > > > > >>
> > > > > > >>
> > > > > > >> _______________________________________________
> > > > > > >> AMBER-Developers mailing list
> > > > > > >> AMBER-Developers.ambermd.org
> > > > > > >> http://lists.ambermd.org/mailman/listinfo/amber-developer
> > > > > > >> s
> > > > > > >>
> > > > > > >
> > > > > > _______________________________________________
> > > > > > AMBER-Developers mailing list AMBER-Developers.ambermd.org
> > > > > > http://lists.ambermd.org/mailman/listinfo/amber-developers
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > AMBER-Developers mailing list AMBER-Developers.ambermd.org
> > > > > > http://lists.ambermd.org/mailman/listinfo/amber-developers
> > > > > >
> > > > > _______________________________________________
> > > > > AMBER-Developers mailing list
> > > > > AMBER-Developers.ambermd.org
> > > > > http://lists.ambermd.org/mailman/listinfo/amber-developers
> > > > >
> > > > _______________________________________________
> > > > AMBER-Developers mailing list
> > > > AMBER-Developers.ambermd.org
> > > > http://lists.ambermd.org/mailman/listinfo/amber-developers
> > > >
> > > _______________________________________________
> > > AMBER-Developers mailing list
> > > AMBER-Developers.ambermd.org
> > > http://lists.ambermd.org/mailman/listinfo/amber-developers
> > >
> > >
> > > _______________________________________________
> > > AMBER-Developers mailing list
> > > AMBER-Developers.ambermd.org
> > > http://lists.ambermd.org/mailman/listinfo/amber-developers
> > >
> > _______________________________________________
> > AMBER-Developers mailing list
> > AMBER-Developers.ambermd.org
> > http://lists.ambermd.org/mailman/listinfo/amber-developers
> >
> _______________________________________________
> AMBER-Developers mailing list
> AMBER-Developers.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber-developers
>
_______________________________________________
AMBER-Developers mailing list
AMBER-Developers.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber-developers
_______________________________________________
AMBER-Developers mailing list
AMBER-Developers.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber-developers
Received on Fri Jun 26 2020 - 20:30:04 PDT
