Next: Running the Regressions, Previous: Regression Testing, Up: Regression
The regression tests are grouped into suites and stored in files as GTP commands. A part of a test suite can look as follows:
# Connecting with ko at B14 looks best. Cutting at D17 might be # considered. B17 (game move) is inferior. loadsgf games/strategy25.sgf 61 90 gg_genmove black #? [B14|D17] # The game move at P13 is a suicidal blunder. loadsgf games/strategy25.sgf 249 95 gg_genmove black #? [!P13] loadsgf games/strategy26.sgf 257 100 gg_genmove black #? [M16]*
Lines starting with a hash sign, or in general anything following a hash
sign, are interpreted as comments by the GTP mode and thus ignored by
the engine. GTP commands are executed in the order they appear, but only
those on numbered lines are used for testing. The comment lines starting
with #?
are magical to the regression testing scripts and
indicate correct results and expected pass/fail status. The string
within brackets is matched as a regular expression against the response
from the previous numbered GTP command. A particular useful feature of
regular expressions is that by using `|' it is possible to specify
alternatives. Thus B14|D17
above means that if either B14
or D17
is the move generated in test case 90, it passes. There is
one important special case to be aware of. If the correct result string
starts with an exclamation mark, this is excluded from the regular
expression but afterwards the result of the matching is negated. Thus
!P13
in test case 95 means that any move except P13
is
accepted as a correct result.
In test case 100, the brackets on the #?
line is followed by an
asterisk. This means that the test is expected to fail. If there is no
asterisk, the test is expected to pass. The brackets may also be
followed by a `&', meaning that the result is ignored. This is
primarily used to report statistics, e.g. how many tactical reading
nodes were spent while running the test suite.