Next: , Previous: Test Suites, Up: Regression


20.3 Running the Regression Tests

./test.sh blunder.tst runs the tests in blunder.tst and prints the results of the commands on numbered lines, which may look like:

     1 E5
     2 F9
     3 O18
     4 B7
     5 A4
     6 E4
     7 E3
     8 A3
     9 D9
     10 J9
     11 B3
     12 C6
     13 C6

This is usually not very informative, however. More interesting is ./eval.sh blunder.tst which also compares the results above against the correct ones in the test file and prints a report for each test on the form:

     1 failed: Correct '!E5', got 'E5'
     2 failed: Correct 'C9|H9', got 'F9'
     3 PASSED
     4 failed: Correct 'B5|C5|C4|D4|E4|E3|F3', got 'B7'
     5 PASSED
     6 failed: Correct 'D4', got 'E4'
     7 PASSED
     8 failed: Correct 'B4', got 'A3'
     9 failed: Correct 'G8|G9|H8', got 'D9'
     10 failed: Correct 'G9|F9|C7', got 'J9'
     11 failed: Correct 'D4|E4|E5|F4|C6', got 'B3'
     12 failed: Correct 'D4', got 'C6'
     13 failed: Correct 'D4|E4|E5|F4', got 'C6'

The result of a test can be one of four different cases:

If you want a less verbose report, ./regress.sh . blunder.tst does the same thing as the previous command, but only reports unexpected results. The example above is compressed to

     3 unexpected PASS!
     5 unexpected PASS!
     7 unexpected PASS!

For convenience the tests are also available as makefile targets. For example, make blunder runs the tests in the blunder test suite by executing eval.sh blunder.tst. make all_batches runs all test suites in a sequence using the regress.sh script.