Documentation Testing ​

If you often read Python's official documentation, you may have noticed that many documents include example code. For instance, the re module comes with several examples:

>>> import re
>>> m = re.search('(?<=abc)def', 'abcdef')
>>> m.group(0)
'def'

You can input and execute these example codes in Python's interactive environment, and the results match what is displayed in the documentation.

These code snippets, along with other explanations, can be written in comments, which can then be used by tools to automatically generate documentation. Since these codes can be pasted and run directly, can we also automatically execute the code written in comments?

The answer is yes.

When we write comments, we can include examples like this:

def abs(n):
    '''
    Function to get the absolute value of a number.
    
    Example:
    
    >>> abs(1)
    1
    >>> abs(-1)
    1
    >>> abs(0)
    0
    '''
    return n if n >= 0 else (-n)

This clearly informs the function's callers about the expected inputs and outputs.

Furthermore, Python’s built-in “documentation testing” (doctest) module can directly extract code from comments and execute tests.

doctest strictly checks the input and output based on the Python interactive command line to determine whether the test results are correct. When exceptions occur, you can use ... to indicate a long output.

Let's use doctest to test the Dict class we created earlier:

# mydict2.py
class Dict(dict):
    '''
    Simple dict but also support access as x.y style.

    >>> d1 = Dict()
    >>> d1['x'] = 100
    >>> d1.x
    100
    >>> d1.y = 200
    >>> d1['y']
    200
    >>> d2 = Dict(a=1, b=2, c='3')
    >>> d2.c
    '3'
    >>> d2['empty']
    Traceback (most recent call last):
        ...
    KeyError: 'empty'
    >>> d2.empty
    Traceback (most recent call last):
        ...
    AttributeError: 'Dict' object has no attribute 'empty'
    '''
    def __init__(self, **kw):
        super(Dict, self).__init__(**kw)

    def __getattr__(self, key):
        try:
            return self[key]
        except KeyError:
            raise AttributeError(r"'Dict' object has no attribute '%s'" % key)

    def __setattr__(self, key, value):
        self[key] = value

if __name__ == '__main__':
    import doctest
    doctest.testmod()

Running python mydict2.py:

$ python mydict2.py

There will be no output, which indicates that all the doctest tests we wrote are correct. If there are issues in the program (for instance, if we comment out the __getattr__() method), running it again will produce errors:

$ python mydict2.py
**********************************************************************
File "/Users/michael/Github/learn-python3/samples/debug/mydict2.py", line 10, in __main__.Dict
Failed example:
    d1.x
Exception raised:
    Traceback (most recent call last):
      ...
    AttributeError: 'Dict' object has no attribute 'x'
**********************************************************************
File "/Users/michael/Github/learn-python3/samples/debug/mydict2.py", line 16, in __main__.Dict
Failed example:
    d2.c
Exception raised:
    Traceback (most recent call last):
      ...
    AttributeError: 'Dict' object has no attribute 'c'
**********************************************************************
1 items had failures:
   2 of   9 in __main__.Dict
***Test Failed*** 2 failures.

Notice the last three lines of output. When the module is imported normally, doctest is not executed. It only runs when the module is executed directly from the command line. So there’s no need to worry about doctest being executed in a non-testing environment.

Exercise ​

Write a doctest for the function fact(n) and execute it:

def fact(n):
    '''
    Calculate 1*2*...*n
    
    >>> fact(1)
    1
    >>> fact(10)
    ?
    >>> fact(-1)
    ?
    '''
    if n < 1:
        raise ValueError()
    if n == 1:
        return 1
    return n * fact(n - 1)

if __name__ == '__main__':
    import doctest
    doctest.testmod()

Summary ​

doctest is extremely useful, not only for testing but also for providing example code. Through certain documentation generation tools, comments containing doctest can be automatically extracted. When users view the documentation, they also see doctest examples.