Chapter 42
Doctests: Example-driven Unit Tests

Difficulty

Newcomer

Skills

Problem/Task

Unit tests are nice, but they are not the best implementation of what eXtreme Programming expects of testing. Testing should also serve as documentation, a requirement that the conventional unit test module pattern does not provide. This chapter will show you an alternative way of writing unit tests that can also serve well as documentation.

Solution

Python already provides docstrings for classes and methods, which serve - like the name suggests - as documentation for the object. If you would be able to write tests in the docstrings of classes and methods and execute them during test runs, all requirements of a testing framework are fulfilled. Even better, the tests would automatically become part of the documentation. This way the documentation reader would always see a working example of the code. Since most people learn by example, this will also speed up the learning process of the technology.

The solution to this problem are doctests, which have exactly the described behavior. If you embed Python-prompt-like sample code in the docstrings of a class and register the contained module as one having doctests, then the Python code in the docstrings is executed for testing. Each docstring will be counted as a single test.

42.1 Integrating the Doctest

So how will our example from the previous chapter change in light of docstests? First of all, you can completely get rid of the TestSample class. Next, add the following lines to the docstring of the Sample class:


1  Examples:
2  
3    >>> sample = Sample()
4  
5    Here you can see how the 'title' attribute works.
6  
7    >>> print sample.title
8    None
9    >>> sample.title = 'Title'
10    >>> print sample.title
11    Title
12  
13    The description is implemented using a accessor and mutator method
14  
15    >>> sample.getDescription()
16    ''
17    >>> sample.setDescription('Hello World')
18    >>> sample.getDescription()
19    'Hello World'
20    >>> sample.setDescription(u'Hello World')
21    >>> sample.getDescription()
22    u'Hello World'
23  
24    'setDescription()' only accepts regular and unicode strings
25  
26    >>> sample.setDescription(None)
27    Traceback (most recent call last):
28      File "<stdin>", line 1, in ?
29      File "test_sample.py", line 31, in setDescription
30        assert isinstance(value, (str, unicode))
31    AssertionError

Next you need to change the test suite construction to look for the doctests in the Sample class’ docstring. To do so, import the DocTestSuite class and change the test_suite() function as follows:


1  from zope.testing.doctestunit import DocTestSuite
2  
3  def test_suite():
4      return unittest.TestSuite((
5          DocTestSuite(),
6          ))

The first argument to the DocTestSuite constructor is a string that is the dotted Python path to the module that is to be searched for doctests. If no module is specified the current one is chosen. The constructor also takes two keyword arguments, setUp and tearDown, that specify functions that are called before and after each tests. These are the equivalent methods to the TestCase’s setUp() and tearDown() methods.

You can now execute the test as before using Pythontest_sample.py, except that ZOPE3/src must be in your PYTHONPATH. The output is the same as for unit tests.


1  .
2  ----------------------------------------------------------------------
3  Ran 1 test in 0.003s
4  
5  OK

As you can see, the three different unit tests collapsed to one doctest.

You will have to agree that doctests are a much more natural way to test your code. However, there are a couple of issues that one should be aware of when using doctests.

42.2 Shortcomings

The most obvious problem is that if you like to test attributes and properties, there is no docstring to place the tests. This problem is usually solved by testing attributes implicitly in context of other tests and/or place their tests in the class’ docstring. This solution is actually good, since attributes by themselves usually do not have much functionality, but are used in combination with methods to provide functionality.

Next, it is not easy to test for certain outputs. The prime example here is None, since it has no representation. The easy way around this is to make the testing statement a condition. So the statement methodReturningNone() which should return None is tested using methodReturningNone()isNone which should return True. There are also some issues when testing statements that return output whose representation is longer than a line, since the docstring checker is not smart enough the remove the indentation white space. A good example of such objects are lists and tuples. The best solution to this problem is to use the pretty printer module, pprint, which always represents objects in lines no longer than 80 characters and uses nice indentation rules for clarity.

Another problematic object to test are dictionaries, since their representation might change from one time to another based on the way the entries are hashed. The simplest solution to the problem is to always convert the dictionary to a list using the items() method and sorting the entries. This should give you a uniquely identifiable representation of the dictionary.

Over time I have noticed that using doctests tends to make me write sloppy tests. Since I think of the tests as examples to show how the class is supposed to work, I often neglect to test for all aspects and possible situation a piece of code could come into. This problem can be solved by either writing some additional classic unit tests or by creating a special testing module that contains further doctests.

While doctests cover 98% of all test situations well, there are some tests that require heavy programming. A good example of that is a test in the internationalization support that makes sure that all XML locale files can be parsed and some of the most important data is correctly evaluated. I found that it is totally okay to use regular unit tests for these scenarios.

Still, overall I think doctests are the way to go, due to their great integration into documentation!

Exercises

  1. As a matter of taste, some people like it better when each method is tested in the method docstring. Therefore, move the getDescription and setDescription tests to the methods’ docstrings and make sure that all three tests pass.
  2. Once you have split up the tests, you always have to setup the sample object over and over again. Use a setUp() function to setup the sample as you did for the unit tests in the previous chapter.
  3. (Equivalent to excercise 2 in the previous chapter.) Currently the test_ setDescription() test only verifies that None is not allowed as input value.

    1. Improve the test, so that all other builtin types are tested as well.
    2. Also, make sure that any objects inheriting from str or unicode pass as valid values.