In the earlier Item, Understand the Test Anywhere Protocal (TAP), you saw the very basics of that simple, line-oriented test report. You ran a single test and it output a single line to denote the status of the test, and possibly some diagnostic information. The TAP, however, didn’t organize any of the tests for you.
On the grand scale, you can separate tests for different parts of a system into separate files. However, even in those separate test files, you probably want to group various tests together as a logical unit.
Consider this test script, which I extracted from the HTML-SimpleLinkExtor-1.23 distribution, one of my CPAN offerings. I’m not proud of this test file, but it is what it is, and I’ll improve it:
use File::Spec;
use Test::More 'no_plan';
use_ok( "HTML::SimpleLinkExtor" );
ok( defined &HTML::SimpleLinkExtor::schemes, "schemes() is defined" );
my $file = 't/example2.html';
ok( -e $file, "Example file is there" );
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
{
my $p = HTML::SimpleLinkExtor->new;
ok( ref $p, "Made parser object" );
isa_ok( $p, 'HTML::SimpleLinkExtor' );
can_ok( $p, 'schemes' );
$p->parse_file( $file );
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
{
my @links = $p->schemes( 'http' );
my $links = $p->schemes( 'http' );
is( $links, 7, "Got the right number of HTTP links" );
is( scalar @links, $links, "Found the right number of links" );
}
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
{
my @links = $p->schemes( 'https' );
my $links = $p->schemes( 'https' );
is( $links, 2, "Got the right number of HTTPS links" );
is( scalar @links, $links, "Found the right number of links" );
}
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
{
my @links = $p->schemes( 'ftp' );
my $links = $p->schemes( 'ftp' );
is( $links, 1, "Got the right number of FTP links" );
is( scalar @links, $links, "Found the right number of links" );
}
}
There’s a few things you’ll notice about this test. First, there are some tests that are really meta-tests to check various things before the real tests start. Before you test a module, you want to check that it loads and perhaps that the method you want to test is there. You might be surprised how many times I’ve wasted time looking for the problem with a method when I had one name in the module and a different name in the tests, never noticing the dissonance. As such, I test for the right method name now. This test also needs an input file, so I ensure that it exists before I start:
use_ok( "HTML::SimpleLinkExtor" ); ok( defined &HTML::SimpleLinkExtor::schemes, "schemes() is defined" ); my $file = 't/example2.html'; ok( -e $file, "Example file is there" );
This particular test file exercises the schemes method that returns links for the give protocol. Before I can test that, I need the object, so there are some tests for that:
my $p = HTML::SimpleLinkExtor->new; ok( ref $p, "Made parser object" ); isa_ok( $p, 'HTML::SimpleLinkExtor' ); can_ok( $p, 'schemes' );
Notice that these tests are at the start of a scope created by a naked block. That parser in $p only exists in that block. After I call parse, I have a series of naked blocks to define scopes for tests that I logically group. In the TAP output, I can tell what’s going on by the labels:
$ perl5.14.1 -Mblib t/schemes.t
ok 1 - use HTML::SimpleLinkExtor;
ok 2 - schemes() is defined
ok 3 - Example file is there
ok 4 - Made parser object
ok 5 - The object isa HTML::SimpleLinkExtor
ok 6 - HTML::SimpleLinkExtor->can('schemes')
ok 7 - Got the right number of HTTP links
ok 8 - Found the right number of links
ok 9 - Got the right number of HTTPS links
ok 10 - Found the right number of links
...
ok 28 - Found the right number of links
1..28
However, this doesn’t really group the tests. I have to separate themselves because I choose the labels.
Version 0.94 of Test::More adds support for nested TAP, which I did not cover in Understand the Test Anywhere Protocal. This enhancement to TAP allows me to group many tests as part of one larger test. The subtest function in Test::More handles it for me. The subtest takes a label and a code reference. In this case, you use an anonymous subroutine that runs additional tests:
use Test::More;
subtest 'Roscoe tests' => sub {
use_ok( 'CGI' );
pass( 'First post' );
};
subtest 'Buster tests' => sub {
use_ok( 'CGI' );
pass( 'First post' );
fail( 'Oops' );
};
subtest 'Mimi tests' => sub {
can_ok( 'CGI', 'head' );
pass( 'First post' );
};
done_testing();
Notice the semicolon after each subtest statement. That anonymous subroutine definition is an expression, so you need a statement separator, just like you need for a do block.
Nested TAP works by indented each level of nested tests. If all of the nested tests pass, it prints an outdented line that is the summary for that group. The overall test passes if all of its subtests pass, and fails if any of its subtests fail:
$ perl5.14.1 nested.t
ok 1 - use CGI;
ok 2 - First post
1..2
ok 1 - Roscoe tests
ok 1 - use CGI;
ok 2 - First post
not ok 3 - Oops
# Failed test 'Oops'
# at nested.pl line 11.
1..3
# Looks like you failed 1 test of 3.
not ok 2 - Buster tests
# Failed test 'Buster tests'
# at nested.pl line 12.
ok 1 - CGI->can('head')
ok 2 - First post
1..2
ok 3 - Mimi tests
1..3
# Looks like you failed 1 test of 3.
Notice that each subtest group gets its own plan, test count, and indented diagnostics. The overall summary only counts the subtest groups. The summary for each subtest group shows up after the subtests (they have to run first, after all).
Going back to my HTML::SimpleLinkExtor test, I can reorganize the test into groups within subtest groups that label them by their logical task. I have basic sanity checks to ensure I have everything I need to test (“Sanity check”). I have the basic setup to start the tests (“Create object”). Once I have all of those, I have tests grouped by the scheme I want to check (“HTTP” and so on). There are three levels of nested TAP here:
use File::Spec;
use Test::More 0.96;
my $file = 't/example2.html';
subtest 'Sanity check' => sub {
use_ok( "HTML::SimpleLinkExtor" );
ok( defined &HTML::SimpleLinkExtor::schemes, "schemes() is defined" );
ok( -e $file, "Example file is there" );
};
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
subtest 'No base' => sub {
my $p;
subtest 'Create object' => sub {
$p = HTML::SimpleLinkExtor->new;
ok( ref $p, "Made parser object" );
isa_ok( $p, 'HTML::SimpleLinkExtor' );
can_ok( $p, 'schemes' );
$p->parse_file( $file );
};
subtest 'All links' => sub {
my @links = $p->links;
is( scalar @links, 26, "Found the right number of links" );
};
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
subtest 'HTTP' => sub {
my @links = $p->schemes( 'http' );
my $links = $p->schemes( 'http' );
is( $links, 7, "Got the right number of HTTP links" );
is( scalar @links, $links, "Found the right number of links" );
};
...
};
If something fails in that test, the indented diagnostics and subtests label help me figure out what logical task failed. In this case, testing the “No base” case for the “All links” test had a problem even though the absolute amount of output is much larger:
$ make test
/Users/brian/bin/perls/perl5.14.1 "-MTest::Manifest" "-e" "run_t_manifest(0, 'blib/lib', 'blib/arch', )"
t/compile.t ......... ok
t/pod.t ............. skipped: Test::Pod 1.00 required for testing POD
t/pod_coverage.t .... skipped: Test::Pod::Coverage required for testing POD
t/parse.t ........... ok
t/tags.t ............ ok
t/schemes.t ......... 1/?
# Failed test 'Found the right number of links'
# at t/schemes.t line 30.
# got: '26'
# expected: '27'
# Looks like you failed 1 test of 1.
# Failed test 'All links'
# at t/schemes.t line 31.
# Looks like you failed 1 test of 7.
# Failed test 'No base'
# at t/schemes.t line 78.
# Looks like you failed 1 test of 4.
t/schemes.t ......... Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/4 subtests
Since I recently had to update HTML::SimpleLinkExtor and its tests were almost already grouped into subtests, I converted them to use subtest. If you look at HTML-SimpleLinkExtor-1.24 or greater, you’ll see more examples of subtest.
Things to remember
- Use
subtestto group related tests by task - Nested TAP works by successive indenting of the TAP stream
- Each nested section has its own plan and test count