Tags

, , , ,

I like and recommend fossil for source code management.

One of the things that I like about it is that it has a friendly and open development team, and a larger group of friendly and helpful expert users who provide help and support via the mailing list. When a user’s problem turns out to expose an issue in fossil itself, the team is not at all shy about fixing it on the spot.

I recently decided to contribute some effort on the regression test suite, with the goal of moving one of my favorite optional features into the default configuration.

Part of why this kind of rapid fix development style works so well is that fossil itself implements the fossil web site, and that server is usually running a build that is very close to the current tip of trunk. So the latest version is moved to the production environment swiftly.

Another reason is that there is a fairly extensive regression test suite included in fossil’s source kit, and that test suite does get run from time to time, and certainly when the team decides it is time to mark a formal release. Many users with private builds also run the test suite from time to time.

Before I extend the existing suite, my first step was to make sure I could run it myself on a recent version of fossil. Along the way, I learned a few things that seemed worth documenting in one place.

Setting up for testing

In the past, the test suite required too much care to be practical for must people to run. It required that it be run from inside a checkout of fossil itself, and it modified the repository in ways that were undesirable to share. So a tester effectively needed to make a clone of the repository, open it, build and test fossil in that workspace, then discard the workspace and the clone without pushing any changes back up stream.

Since then, a more sensible practice became supported where most test cases create new, empty repositories of their own. Since it is generally an error to open a repository inside an open checkout, this change required that test cases be run in a separate tree.

Prerequisite Tools

First, you need to be able to build fossil, which will require a suitable C compiler and the usual suspect collection of programmer’s tools starting with make. On Windows, I’ve used [MinGW][] and MSYS for builds. On linux, you need the usual suite of developer tools.

I would like to be able to test on Windows, and am pleasantly surprised to find that the test suite largely works the same on Windows, at least if run from bash provided via MSYS.

With fossil built, you still need a working installation of Tcl. A full-featured desktop Linux distro likely has Tcl already. If not, sudo apt-get install tcl will generally do the trick.

In addition to Tcl, the test suite implicitly assumes you have the SQLite integration module installed in your Tcl. This is not typical for a stock distro, so you are likely to need to add it. On Ubuntu, sudo apt-get install libsqlite3-tcl will solve that. If building fossil itself to use Tcl, but not via the “stubs” mechanism that loads the available Tcl library at run time, you might need to also get the tcl-dev package.

Building Out of Tree

The basic recipe is first, with fossil’s repository open in ~/fossil/work, build out of tree:

$ W=~/fossil/work
$ B=~/fossil/build
$ mkdir $B; cd $B
$ $W/configure   #with options to taste
$ make

to get $B/fossil freshly built. See $W/configure --help for a list of all the options that are supported. On my testing box, I’m using --with-openssl=none for all tested configurations because I haven’t bothered to track down what packages I need to properly compile against OpenSSL.

Repeat this with additional build folders to make multiple flavors of fossil to easily compare behavior with different configuration options. Note that no matter how many build folders you configure, you only have one source tree, and that running configure does not change the source tree in any way.

In any build folder head -1 config.log will reveal the exact command line used to configure that build.

Test Out of Tree

You can and should run tests from out of the tree. The tests could be run in a build folder, or entirely separately.

We’ll start with some shell variables that we will assume point to the appropriate folders:

$ W=~/fossil/work
$ B=~/fossil/build
$ T=~/fossil/test
$ F=$B/fossil
$ mkdir $T; cd $T

With those definitions, the most direct way to run the entire test suite is as follows:

$ tclsh $W/test/tester.tcl $F

That will produce an immense amount of output, so a couple of additional command line options are useful to know about:

-prot    capture all output in a file named prot
-verbose include more detail about each test
-quiet   only report failed test output
-halt    halt the test run at the first failed test

The entire test suite consists of $W/test/tester.tcl itself, and all of the files $W/test/*.test which are loaded and run in alphabetical order by tester.tcl. All of the tests are written in Tcl, which must be installed.

Individual test files can be run rather than the entire suite by a command like:

$ tclsh $W/test/tester.tcl $F amend utf

which would run the tests in amend.test and utf.test only. This is particularly handy when chasing an issue covered by just one specific test. Otherwise, the current release has on the order of 30000 individual tests in 23 files, and takes a noticeable amount of time to run.

Results

This exercise seemed to have shaken some dust out of the corners of the test suite, and even revealed a bug in fossil when a failing test case couldn’t be entirely explained by my not yet having everything configured right.

As of checkin fc14f1431286cc51, the entire test suite runs out of tree and sees exactly the same two failing test cases whether or not Tcl support, TH1 hooks, and TH1 docs are configured.

The entire suite also runs on Windows (under MSYS bash), and sees the same two failing tests. The windows build was not configured with Tcl, TH1 hooks or TH1 docs at this point, so it is running a slightly smaller total set of tests.

If you have a project involving embedded systems, micro-controllers, electronics design, audio, video, or more we can help. Check out our main site and call or email us with your needs. No project is too small!

+1 626 303-1602
Cheshire Engineering Corp.
710 S Myrtle Ave, #315
Monrovia, CA 91016

(Written with StackEdit.)