Adding a Test Harness to Komodo

We shipped a unittest harness in version 4.3 of Komodo, but strictly speaking it was closer to 1.0 functionality. We hard-wired four separate test harnesses, one for each of Perl, Python, Ruby, and PHP. Naturally, as soon as we launched 4.3 there was a barrage of requests for new test harnesses. With the functionality in 4.4 we're sharing the joy in writing tests. This document will explain how the test plans are structured, and what you need to do to write your own.

What is Sleuth?

"Sleuth" is a name we internally use for the unittest integration component. It has a well-defined meaning in English, but is rarely used, and therefore less likely to conflict or create confusion with existing systems. For example, Komodo has its own unittest framework. If we called this component "UnitTest", it could lead to confusion.

Structure of a Test Harness

The file sleuth.py defines two classes, KoSleuthHarness and KoSleuthHarnessRunner. KoSleuthHarnessRunner is the higher-level class, and is the one Komodo uses to run a set of tests. KoSleuthHarness is a lower-level class that launches the actual test process, reads its output, filters it, parses it, and writes information back to Komodo for it to display in the Test Results tab.

Neither class is an XPCOM class, so there are no IDL files that Komodo ships, but these are their external interfaces:

class KoSleuthHarness:
    def __init__(self, args, launch_dir=None):
        # args: array of strings
        # launch_dir: directory to run program from
    def filter_line(self, line, line_num):
        # line: string
        # Callback, return True if the line should be filtered (not parsed)
    def run(self):
        # An inner generator called by KoSleuthHarnessRunner.run
    def stop(self):
        # An inner function called by KoSleuthHarnessRunner.stop
        
class KoSleuthHarnessRunner:
    def __init__(self):
        # Initializes this class
    def lookupOutsideKoPath(self, programName):
        # programName : string, like "perl"
        # returns either the full path, or None
    def run(self):
        # A generator, yields a stream of lines read from the process
    def stop(self):
        # Called by Komodo to end the process

By itself, the code in sleuth.py cannot run any test harnesses. Each test harness is represented by a pure-Python module that needs to do the following things:

  • Live in a file with a name and location that Komodo can discover. The section on writing extensions will cover the directory structure to use. Harness filenames must end with "_harness.py".
  • Define a subclass of KoSleuthHarnessRunner.
  • Most harnesses will also need to define a subclass of KoSleuthHarness, although currently the koPHP_PHPUnit_harness.py harness defines a trivial subclass of it.
  • Set a module global called harnessName to the name of your harness. Harness names should be namespaced, to avoid collision -- currently the last harness loaded wins at run-time. These values will appear in harness selection menus in the Komodo dialogs for creating and editing test plans.
  • Define a register function that Komodo will call at startup time:
    class PHP_PHPUnit_SleuthHarness:
        # ...
    harness_name = "PHP - PHPUnit"
    def register(sleuthManager):
        sleuthManager.register('PHP', harness_name, PHP_PHPUnit_SleuthHarness)

Whenever Komodo wants to run your harness, it will create a new instance of the class passed in the third argument to the register method. This means you can't maintain persistent data in each of these classes; use module globals if you need to.

Have a look at the harnesses that Komodo ships. Notice that the PHP and Python harnesses are relatively thin wrappers around sleuth.py. By contrast, the Perl TAP harness has to do much more work, as it's parsing output, rather than plugging code into the underlying test harness.

Notice also that error messages are relayed back to the UI by setting the self._initial_msgs field to a list of sleuth-formatted error messages back to sleuth.py. The upper-level HarnessRunner class checks for these before launching the actual process; if it finds any, it normally relays them back to the caller (remember that the run method is a generator, not a function). This is a good way to present missing prerequisites back to the user.

Talking to Komodo: the Sleuth Language

These are the six different directives Komodo is watching for:

    @suite_started@:
    start a new test suite
    
    @test_started@: testname
    
    @fault@: line
    - a line of a possibly multi-line fault associated with the current test.
    
    @into@: line
    - non-error information the user might find interesting
    
    @test_result@: P | F | E
    
    @suite_finished@: N:n P:p F:f E:e; t
    - finished the current test suite. n, p, f, e stand for the number
    of tests, number passed, number failures, and number errors.
    t gives the elapsed time

Any other output is emitted to Komodo and treated as part of either the current fault or info event, or will be discarded.

Yes, tests that emit any of the above strings as part of their output will confuse Komodo.

Defining New Test Harnesses

In general, you'll need to spend more time figuring out how to plug in your test-runner into the framework you're targeting, or parse its output, then you should spend getting a new framework to work with Komodo. Test harness extensions are very simple, and follow this layout:

dir/
    - install.rdf
    - sleuth-harnesses/
        - *
            - *_harness.py

That's it. The sleuth-harnesses may contain more than one subdirectory: each of those subdirectories can define one or more test harnesses. The reason for the extra level of directories is to make it easier to package several different types of harnesses in a single extension.

The nose.xpi extension implements not one but two harnesses for working with Python's nose testing framework. Nose is good at "sniffing" out things to test in a directory, much better than the unittest-based test harness shipped with Komodo. I noticed that by adding a --with-doctests argument to the invocation of Nose, it would run docstring-based tests instead of tests that implemented unittest. I first duplicated the contents of as_nose_harness.py into as_nose_doctest_harness.py, got it to work (by adding one extra argument), and then refactored the common part of the two files into as_nose_common.py. These harnesses work with versions 0.9 and 0.10 of Nose.

Before you can use them, you need to install Nose, of course, and also install the sleuthplug.py module that I bundled in the extension. At some point I'll touch base with Brandon Corfman, who first directed my attention to this framework, to find out a way to run plugins without installing them. But meanwhile I've got a Ruby Test::Unit harness to write.