Re: [AMBER-Developers] General test failure question

From: Gustavo Seabra <>
Date: Sat, 27 Feb 2010 19:07:48 -0300

On Sat, Feb 27, 2010 at 2:41 AM, Ben Roberts <> wrote:
> On 26/02/2010, at 8:05 PM, Gustavo Seabra wrote:
>> Hi Ben,
>> On Fri, Feb 26, 2010 at 9:48 PM, Ben Roberts <> wrote:
>>> OK. Steps I've taken so far in my working copy:
>>> 1. Make sure that any "make" command in a Makefile is actually executed
>>> as
>>> "make -k". This change affects two or three statements that I've found.
>> Great. Is that a change to the Makefile or is it in the script? That
>> alone solves most of the problems. If it works fine, I'd say check
>> that in.
> I made a handful of changes to Makefiles (where one Makefile calls another).
> In general, however, I put the "make -k" in the scripts. Changes to
> Makefiles have been uploaded, as have the scripts themselves.

That should work, thanks.

>>> 2. Prepare the aforementioned scripts.
>> Myself, I'd still recommend awk. With awk, we can do more, like
>> finding separators (like the "-------" lines, and then looking for the
>> "PASSED" betwen them, etc. It gives us a lot more flexibility then
>> trying to fix every test. Also, tests that fail with segfaults, for
>> example, don't always print the "POSSIBLE FAILURE" statement, so we
>> need to look for other stuff. Awk allows you to make a easier to use
>> counter, save data about the failed tests, etc. If you want, I can
>> take a look at it later, after you check in what you already have.
> That would be appreciated. Since I don't know my way around awk very well, I
> would like to leave it to those (such as yourself) who do.

I'm checking out a new tree now, and will look into that. Cheking out
here in Brazil takes hours, literally :-(

>>> I haven't committed any of these changes yet. Is it appropriate to do so?
>>> I'm not quite sure how much boiler-busting will be involved, or whether
>>> this
>>> change counts as a new and poorly tested feature.
>> Most of the changes don't change anything for other users, except the
>> Makefile changes (if that's how you have done it) that will make sure
>> they don't get the annoying errors they have been getting, so I'd say
>> check them in.
>> This is great stuff Ben, thanks a lot!
> No problem. I hope it all works properly.
> As an aside, in my personal opinion, it would be helpful if all tests behave
> consistently - so that the same result generates the same report. But even
> apart from the feature freeze, I agree that sorting them out definitely
> falls into the "rainy day project" category. Having said that, enforcing
> consistency across the tests may be easier than trying to write awk commands
> to cover every contingency. What are your thoughts?

I totally agree with you. The only reason I'm voting on the awk script
at this point is that I think this is the "quick-and-dirty" way to do
it now. It would be nicer to have all tests working this consistently,
but I guess we'll have to wait for that rainy day to come... (and here
at least, those days are *not* all that common :-)


AMBER-Developers mailing list
Received on Sat Feb 27 2010 - 14:30:07 PST
Custom Search