On 01/28/2015 04:49 PM, Jason Swails wrote:
> On Wed, Jan 28, 2015 at 10:10 AM, Gerald Monard <
> Gerald.Monard.univ-lorraine.fr> wrote:
>
>>
>>
>> On 01/28/2015 03:53 PM, Jason Swails wrote:
>>> On Wed, Jan 28, 2015 at 9:47 AM, Gerald Monard <
>>> Gerald.Monard.univ-lorraine.fr> wrote:
>>>
>>>> Hello,
>>>>
>>>> For the development/maintenance of the SEBOMD part and the QUICK part (=
>>>> ab initio code from Kennie's group) in AmberTools, I would like to set
>>>> up some tests that could be (very) time consuming. These would be mostly
>>>> regression tests to ensure that what is developed doesn't break
>> anything.
>>>>
>>>> Given that the current test/ directory (AmberTools+Amber) is ~2 Gbytes
>>>> and that what I will add should not be of any use for a non-developer, I
>>>> think it would be wiser to put my tests somewhere else.
>>>>
>>>> What I suggest is to create a reg-test/ directory (or any name that
>>>> suits you) in Amber that would not be part of the distribution and that
>>>> could serve for development testing. I would keep (for SEBOMD and
>>>> quick), only relevant tests to ensure a proper compilation for the users
>>>> and move the rest to this directory.
>>>>
>>>> Is this OK for you all?
>>>>
>>>
>>> I would personally vote to put it in the test/ directory for the
>> following
>>> reasons:
>>>
>>> - If they are too big to include in the tarball, the test directory
>>> containing them can be easily added to the tarball exclusion list so it's
>>> omitted from the tarball (just as easily as if it was 'sequestered').
>>
>> If so, then I guess we should have a keyword or file to explicitly say
>> that we don't want the directory to be included in the distribution
>> ("touch NOTAR"?). I'm not sure that I should play with the release maker
>> to specify one by one the directories in an excluded list.
>>
>
> The way I was envisioning it was to add a single line to the excludes_at
> file, if in fact the test files you plan to add are too big to be added.
> There's no rule to "1 test per 1 directory" inside test/ -- so if all of
> your regression tests are added to something like "sebomd-regression/", you
> can simply omit that directory in excludes_at.
>
>> - If they are not too big to include, they could serve as nice examples
>> for
>>> people wanting to use the functionality.
>>> - Keeps things simpler -- how does a new developer decide what goes into
>>> "reg-test" vs "test"?
>>
>> May be I'm wrong, but for me, the test/ directory serves for the "user"
>> to be sure that his/her compilation went right and that (s)he can trust
>> the results of his/her simulations (computationally speaking of course).
>
> The reg-test would be a standard regression test directory intended for
>> developers to ensure that new functionalities don't break old ones or
>> that bugs are fixed.
>>
>
> This is currently what we use "test/" for, but I don't particularly like
> that model. [1] What do you envision being the standard developer workflow
> here?
> Are reg-tests something that I should run, for instance, if I make
> any change to the sander source code?
Exactly. The idea would be to run reg-tests on a "regular" basis for
//your// program (e.g., sander for me). It could also include unitary
tests. What I have in mind is to include things that could be a little
bit more complex than a "dacdif test" ;-) (like compiling unitary test
for fortran modules, running "long" simulations to ensure some
properties are kept, checking parallel scaling, etc.).
> Or are they safety nets to make sure
> possible errors are caught before they go out in the wild? (i.e., do we
> run them before almost every push, or do we run them with vigor before
> every release?) In my experience, mandatory tests that take "too long" do
> a lot more to hinder any development than promote safe development.
>
Ideally, one should not push before testing, but ... we all know how it
works in real life. So my plan would be to let developer decide of
course (and the dashboard could be of very good help here).
I can prepare something for Gainesville's meeting and we could start
working around it at that time.
Gerald.
> e.g., to be sure that SEBOMD is correct after compilation, I could
>> nearly end up with only 1 test with "all" functionalities included. That
>> could be fast for a user. On the other way, I want to be able to check
>> my functionalities one by one when I develop. I could need hundreds of
>> tests, and some could be time consuming...
>>
>> Gerald.
>>
>
> Thanks,
> Jason
>
>
> [1] The model I think I like the most is to have an extensive suite of
> short "white-box" unit-tests that try to cover all (or as many as possible)
> of the possible code paths in the various programs. (white-box meaning that
> the tests should be designed with the implementation details in mind,
> specifically so we can maximize test coverage.) These are run before any
> commit to the 'main' branch. Then a set of "black-box" regression tests
> that run for a long time and test the various options and make sure that
> ensemble results don't change from version to version. These are run a lot
> less frequently (since they would need to run for a lot longer).
>
> So I like your idea of separate "test/" and "reg-test/" directories -- I
> just don't think that there's a practical difference between the reg-test
> you propose and the test/ we have. But I think I've changed my mind -- if
> this is a start in that type of direction, it's probably a Good Thing. (An
> instructive README in the new directory would help new developers)
>
>
--
____________________________________________________________________________
Prof. Gerald MONARD
SRSMC, Université de Lorraine, CNRS
Boulevard des Aiguillettes B.P. 70239
F-54506 Vandoeuvre-les-Nancy, FRANCE
e-mail : Gerald.Monard.univ-lorraine.fr
tel. : +33 (0)383.684.381
fax : +33 (0)383.684.371
web : http://www.monard.info
____________________________________________________________________________
_______________________________________________
AMBER-Developers mailing list
AMBER-Developers.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber-developers
Received on Wed Jan 28 2015 - 08:30:04 PST
