Locate bugs with source control bisection

As you work in Perl you store each step in source control. When you finish a little bit of work, you commit your work. Ideally, every commit deals with one thing so you’re only introducing one logical change in each revision. Somewhere along the process, you might discover that something is not working correctly. You think that it used to work but you’re not sure where things went pear-shaped, perhaps because the bug seemingly deals with something that you weren’t working on.

You might think that your test suite should catch those, but a test suite doesn’t protect you from bugs. Your tests might not have checked for the problem, or your tests might have been wrong, or all sorts of other things that you don’t expect. Your tests are only as good as you make them, and often the programmer creates his own tests and acts as his own quality control (which isn’t the best of arrangements).

Since you’re keeping your work in source control (we’ll give you the benefit of the doubt), you can easily, at least in process, figure out where the problem appears by bisecting your source tree until you find the revision that introduces the problem:

1. check out a revision where things don’t work
2. verify broken behavior with a test script
3. check out a revision where you think things work
4. verify working behavior with a test script
5. check out a revision halfway between the working and broken versions
6. run the test script to check if that revision works or breaks
7. repeat this bisection until you find revision that breaks

With each iteration, you narrow the window of revisions where you might have introduced the problem.

This is simple in process, but it’s tedious in practice. With any source control system, you can manually checkout a revision, run your check script, and see what happens. You can keep doing that until you find the problem. You can even automate it.

Some modern source control systems, however, provide a bisection feature for you so you don’t have to automate it yourself. For this Item, consider git, which is popular in the Perl community (and the perl source is in a git repository). Read perl5-porters for a couple of days and you’re likely to read about some of the developers using git bisect to find a bug in perl.

As a demonstration, you can use the Buster::Bean git repository. In that module which simulates my cat, there’s a complaino subroutine that should return a string that has “meow” in it, but somewhere along the line it broke. The t/export.t test checks for this, and you notice in the latest revision that the t/export.t test doesn’t pass.

# t/export.t
like( complaino(), qr/meow/i, 'complaino returns something like a meow' );

The current version of complaino is broken, but you don’t remember when it broke:

# lib/Bean.pm
sub complaino {
	return 'MEOOOOOW!' # this versions is broken
	}

Once you unpack the distribution, change into the Buster-Bean directory if you want to follow along.

To start your hunt, you tell git that you are starting a bisection:

% git bisect start

Next you have to set the initial window. As with many things in git, you can specify the revision in various way. In this case, you can use the start of the SHA-1 digest for each commit. You set the good and bad bounds of the window:

% git bisect good 4be027af
% git bisect bad 14883968

Once you’ve set the window for your bisection, you run the bisection by specifying a test script to run for each commit that git will check:

% git bisect run ./test-script.sh

The trick is to write a test script that tells git if the revision works or not. If your test script exits with 0, you’re telling git that the revision works. If your test script exits with 1-127, that commit fails (although exiting with 125 tells git to skip that revision).

How you write your test script depends on what you want to check. In the case of complaino, you want to find where the t/export.t test starts to fail. You might think that you can just run the your test script:

% git bisect run perl -Iblib/lib t/export.t

However, that might not put the right versions of the modules in blib. In this case, you need to run Makefile.PL each time then make the source (or ./Build it to put everything in the righ place. With the right files, you run the program that provides the final exit code to tell git what happened:

#!/bin/sh

perl Makefile.PL
make
perl -Iblib/lib t/export.t

Now you’re to run the bisection:

% git bisect run ./meow_test.sh

The output shows git‘s progress through the revision history. The first revision it tests is in the middle of the the window you specified. In this case, that one passes, so it’s now the good bound of the window. The next revision is 93d1747, where t/export.t fails. That narrows the window on the bad side. The rest of the bisection tries revisions that all pass, so 93d1747 must be the revision that introduces that brokenness. git reports 93d1747... is first bad commit:

running ./meow_test.sh
Writing Makefile for Buster::Bean
Skip blib/lib/Buster/Bean.pm (unchanged)
Manifying blib/man3/Buster::Bean.3
1..4
ok 1 - use Buster::Bean;
ok 2 - Buster::Bean->can('complaino')
ok 3 - complaino subroutine is defined
ok 4 - complaino returns something like a meow

Bisecting: 3 revisions left to test after this
[93d1747e4681ea21536a66aba25bb21e1cddda05] Make uppercase
running ./meow_test.sh
Writing Makefile for Buster::Bean
cp lib/Bean.pm blib/lib/Buster/Bean.pm
Manifying blib/man3/Buster::Bean.3
1..4
ok 1 - use Buster::Bean;
ok 2 - Buster::Bean->can('complaino')
ok 3 - complaino subroutine is defined
not ok 4 - complaino returns something like a meow
#   Failed test 'complaino returns something like a meow'
#   at t/export.t line 10.
#                   'MEOOOOOW!'
#     doesn't match '(?i-xsm:meow)'
# Looks like you failed 1 test of 4.

Bisecting: 1 revisions left to test after this
[980be46fa2ea3ee32372aedd62949a663c729058] * Use fewer exclamation points
running ./meow_test.sh
Writing Makefile for Buster::Bean
cp lib/Bean.pm blib/lib/Buster/Bean.pm
Manifying blib/man3/Buster::Bean.3
1..4
ok 1 - use Buster::Bean;
ok 2 - Buster::Bean->can('complaino')
ok 3 - complaino subroutine is defined
ok 4 - complaino returns something like a meow

Bisecting: 0 revisions left to test after this
[b5ab15611ca25c6925998463c9cb7b079fe87c8b] * Make it lowercase
running ./meow_test.sh
Writing Makefile for Buster::Bean
cp lib/Bean.pm blib/lib/Buster/Bean.pm
Manifying blib/man3/Buster::Bean.3
1..4
ok 1 - use Buster::Bean;
ok 2 - Buster::Bean->can('complaino')
ok 3 - complaino subroutine is defined
ok 4 - complaino returns something like a meow

93d1747e4681ea21536a66aba25bb21e1cddda05 is first bad commit
commit 93d1747e4681ea21536a66aba25bb21e1cddda05
Author: brian d foy <[email protected]>
Date:   Mon Jul 19 20:43:07 2010 -0500

    Make uppercase

:040000 040000 8360932159317f0685b98f2b2dba4753c53e6240 0e7a2e5d54de7db9f1b790ed4da0a704612ba130 M  lib
bisect run success

Graphically, that bisection looks like this, starting at the top and going toward the bottom for ① then on its way back toward the top for ②, alternating directions as it closes in on the bad revision:

When you are done with the your bisection, you need to tell git to return to the head of the source tree:

% git bisect reset 

Other source control systems have a bisection feature which do basically the same thing although their details might be different. If your source control system doesn’t have this feature, you can automate it (or switch to a system that does have it). Once set-up, you should be be able to locate bugs much quicker.