Re: [AMBER-Developers] regression test directory

From: Jason Swails <>
Date: Wed, 28 Jan 2015 10:49:38 -0500

On Wed, Jan 28, 2015 at 10:10 AM, Gerald Monard <> wrote:

> On 01/28/2015 03:53 PM, Jason Swails wrote:
> > On Wed, Jan 28, 2015 at 9:47 AM, Gerald Monard <
> >> wrote:
> >
> >> Hello,
> >>
> >> For the development/maintenance of the SEBOMD part and the QUICK part (=
> >> ab initio code from Kennie's group) in AmberTools, I would like to set
> >> up some tests that could be (very) time consuming. These would be mostly
> >> regression tests to ensure that what is developed doesn't break
> anything.
> >>
> >> Given that the current test/ directory (AmberTools+Amber) is ~2 Gbytes
> >> and that what I will add should not be of any use for a non-developer, I
> >> think it would be wiser to put my tests somewhere else.
> >>
> >> What I suggest is to create a reg-test/ directory (or any name that
> >> suits you) in Amber that would not be part of the distribution and that
> >> could serve for development testing. I would keep (for SEBOMD and
> >> quick), only relevant tests to ensure a proper compilation for the users
> >> and move the rest to this directory.
> >>
> >> Is this OK for you all?
> >>
> >
> > ​I would personally vote to put it in the test/ directory for the
> following
> > reasons:
> >
> > - If they are too big to include in the tarball, the test directory
> > containing them can be easily added to the tarball exclusion list so it's
> > omitted from the tarball (just as easily as if it was 'sequestered').
> If so, then I guess we should have a keyword or file to explicitly say
> that we don't want the directory to be included in the distribution
> ("touch NOTAR"?). I'm not sure that I should play with the release maker
> to specify one by one the directories in an excluded list.

​The way I was envisioning it was to add a single line to the excludes_at
file, if in fact the test files you plan to add are too big to be added.
There's no rule to "1 test per 1 directory" inside test/ -- so if all of
your regression tests are added to something like "sebomd-regression/", you
can simply omit that directory in excludes_at.

> - If they are not too big to include, they could serve as nice examples
> for
> > people wanting to use the functionality.​
> > - Keeps things simpler -- how does a new developer decide what goes into
> > "reg-test" vs "test"?
> May be I'm wrong, but for me, the test/ directory serves for the "user"
> to be sure that his/her compilation went right and that (s)he can trust
> the results of his/her simulations (computationally speaking of course).

The reg-test would be a standard regression test directory intended for
> developers to ensure that new functionalities don't break old ones or
> that bugs are fixed.

​This is currently what we use "test/" for, but I don't particularly like
that model. [1] What do you envision being the standard developer workflow
here? Are reg-tests something that I should run, for instance, if I make
any change to the sander source code? Or are they safety nets to make sure
possible errors are caught before they go out in the wild? (i.e., do we
run them before almost every push, or do we run them with vigor before
every release?) In my experience, mandatory tests that take "too long" do
a lot more to hinder any development than promote safe development.

e.g., to be sure that SEBOMD is correct after compilation, I could
> nearly end up with only 1 test with "all" functionalities included. That
> could be fast for a user. On the other way, I want to be able to check
> my functionalities one by one when I develop. I could need hundreds of
> tests, and some could be time consuming...
> Gerald.


[1] The model I think I like the most is to have an extensive suite of
short "white-box" unit-tests that try to cover all (or as many as possible)
of the possible code paths in the various programs. (white-box meaning that
the tests should be designed with the implementation details in mind,
specifically so we can maximize test coverage.) These are run before any
commit to the 'main' branch. Then a set of "black-box" regression tests
that run for a long time and test the various options and make sure that
ensemble results don't change from version to version. These are run a lot
less frequently (since they would need to run for a lot longer).

So I like your idea of separate "test/" and "reg-test/" directories -- I
just don't think that there's a practical difference between the reg-test
you propose and the test/ we have. But I think I've changed my mind -- if
this is a start in that type of direction, it's probably a Good Thing. (An
instructive README in the new directory would help new developers)

Jason M. Swails
Rutgers University
Postdoctoral Researcher
AMBER-Developers mailing list
Received on Wed Jan 28 2015 - 08:00:03 PST
Custom Search