On 1/03/2010, at 3:53 PM, case wrote:
> On Mon, Mar 01, 2010, Ben Roberts wrote:
>>
>> In a supplement to the recent flurry of emails about test scripts, in
>> my sandbox I've gone through and rewritten the NAB tests to use run
>> scripts and dacdif. Thus, the output is more standardised, and the
>> run
>> scripts also call a "program error" type function in the event of an
>> actual failure (previously, they would just report that the outputs
>> were different).
>
> Good...this is something that I have wanted to do for some time.
> But (see below), why don't you just update the Makefile instead to use
> dacdif rather than diff? That seems to me to the cleaner way to
> proceed.
>
>>
>> A potential downside here is that the actual differences are no
>> longer
>> printed to STDOUT, per dacdif's way of doing things. Another
>> potential
>> downside is that dacdif is doing things I don't entirely expect, and
>> that important differences are thereby being skipped.
>
> I'll take a look once things are checked in; but I don't expect any
> bad
> side effects...
>
>>
>> A third risk is: what environment variables are needed? In the run
>> scripts, BINDIR, NABFLAGS and SFX are clearly required. They are
>> passed from the nab Makefile, but aren't available independently,
>> because the Makefile in turn gets them from src/config.h. Also, does
>> NAB expect any environment variables to be present, and is that why a
>> run script approach has previously been avoided?
>
> But lots of make files import variables from config.h, e.g. both
> Makefile
> and Makefile_at in amber11/test. Why is it any less desirable here
> than
> elsewhere? Since things like the "test_at_serial.sh" scripts go
> through
> make to do their work, I guess I don't see the problem here(?).
I went via the shell script approach for (broad) consistency with
other tests - the consistency is not absolute, since I used sh instead
of csh. That could also be changed if desired.
The other reason was to do with reporting of program crashes
(conventionally reported as "Program error") cf. failures ("Possible
FAILURE in <output file>"). Historically, that difference was not
noted in the nab tests, with the result that sometimes the screen
would be filled up with "differences" (i.e., a program crashed so
there were "differences" on every line of the output). It seems harder
to distinguish between the two outcomes in Make, though, now that I
think about it, it probably just requires slightly fiddlier syntax.
Since from my point of view it's six of one and half a dozen of the
other as to what I upload, what (long-term) would we prefer? Shell
scripts, or Makefiles? I know this was brought up a few months ago,
and we put off making a decision until the meeting. To my knowledge,
though, no decision was made there. Your thoughts?
Ben
_______________________________________________
AMBER-Developers mailing list
AMBER-Developers.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber-developers
Received on Mon Mar 01 2010 - 13:30:02 PST
