Just my 2c,
I agree with Ben and Lachele. It is really annoying, not to say
counterproductive, when the tests stop because of one test that fails.
It would be much better to force it through all tests (failure or
not), then print a summary at the end. The filenames themselves don't
bother me much but, if it's easy to implement, why not?
Cheers,
Gustavo Seabra
Professor Adjunto
Departamento de Química Fundamental
Universidade Federal de Pernambuco
Fone: +55-81-2126-7450 Ext. 5021
Sent from Recife, Pernambuco, Brazil
On Tue, Feb 23, 2010 at 8:05 PM, Ben Roberts <roberts.qtp.ufl.edu> wrote:
> All,
>
> Is there a consensus or preferred approach when it comes to test failures? At the moment, some tests will fail and keep going, while if others fail (especially with "Program error") the test stops entirely. From my point of view this is unhelpful, since establishing precisely which tests fail may require several runs through.
>
> I know I brought this up briefly at the meeting, but forget what the preferred solution was.
>
> One that occurs to me is to precede each test with a - (in the Makefile). But this seems like a blunt instrument, insofar as if it's just done by itself the user may not necessarily be aware of catastrophic problems.
>
> What would people think about the possibility of a summary report at the end? For example, something like this:
>
> 73 tests were requested. Of these:
> 8 tests were skipped (system/environment requirements not met)
> 58 tests passed
> 4 tests failed diff - check output
> 3 tests encountered errors
>
> (Traffic on the earlier thread about processor requirements for parallel tests suggested that skipping tests is a bad idea, however I would expect that the real problem is not so much the skipping itself, as the fact that it is done silently.)
>
> What do people think? Bad idea, must-have, or good if someone puts in the time?
>
> Cheers,
> Ben
> _______________________________________________
> AMBER-Developers mailing list
> AMBER-Developers.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber-developers
>
_______________________________________________
AMBER-Developers mailing list
AMBER-Developers.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber-developers
Received on Wed Feb 24 2010 - 06:00:12 PST
