Package test

source code

This is a top level package, hosting the entire CSB test framework. It is divided into several major parts:

• test cases, located under csb.test.cases
• test data, in /csb/test/data (not a package)
• test console, in /csb/test/app.py

This module, csb.test, contains all the glue-code functions, classes and decorators you would need in order to write tests for CSB.

1. Configuration and Tree

   Config is a common config object shared between CSB tests. Each config instance contains properties like:

   • data: the data folder, automatically discovered and loaded in csb.test.Config.DATA at module import time
   • temp: a default temp folder, which test cases can use

   Each Config provides a convenient way to retrieve files from /csb/test/data. Be sure to check out Config.getTestFile and Config.getPickle. In case you need a temp file, use Config.getTempStream or have a look at csb.io.TempFile and csb.io.TempFolder.

   All test data files should be placed in the data folder. All test modules must be placed in the root package: csb.test.cases. There is a strict naming convention for test modules: the name of a test module should be the same as the name of the CSB API package it tests. For example, if you are writing tests for csb/bio/io/__init__.py, the test module must be csb/test/cases/bio/io/__init__.py. csb.test.cases is the root package of all test modules in CSB.

2. Writing Tests

   Writing a test is easy. All you need is to import csb.test and then create your own test cases, derived from csb.test.Case:

   >>> import csb.test
   >>> @csb.test.unit
       class TestSomeClass(csb.test.Case):
           def setUp(self):
               super(TestSomeClass, self).setUp()
               # do something with self.config here...

   In this way your test case instance is automatically equipped with a reference to the test config, so your test method can be:

   >>> @csb.test.unit
       class TestSomeClass(csb.test.Case):
           def testSomeMethod(self):
               myDataFile = self.config.getTestFile('some.file')
               self.assert...

   The "unit" decorator marks a test case as a collection of unit tests. All possibilities are: csb.test.unit, csb.test.functional, csb.test.custom, and csb.test.regression.

   Writing custom (a.k.a. "data", "slow", "dynamic") tests is a little bit more work. Custom tests must be functions, not classes. Basically a custom test is a function, which builds a unittest.TestSuite instance and then returns it when called without arguments.

   Regression tests are usually created in response to reported bugs. Therefore, the best practice is to mark each test method with its relevant bug ID:

   >>> @csb.test.regression
       class SomeClassRegressions(csb.test.Case)
           def testSomeFeature(self)
           """
           @see: [CSB 000XXXX] 
           """
           # regression test body...

3. Style Guide:
   • name test case packages as already described
   • group tests in csb.test.Case-s and name them properly
   • prefix test methods with "test", like "testParser" - very important
   • use camelCase for methods and variables. This applies to all the code under csb.test (including test) and does not apply to the rest of the library!
   • for functional tests it's okay to define just one test method: runTest
   • for unit tests you should create more specific test names, for example: "testParseFile" - a unit test for some method called "parse_file"
   • use csb.test decorators to mark tests as unit, functional, regression, etc.
   • make every test module executable:

        if __name__ == '__main__':
            csb.test.Console()   # Discovers and runs all test cases in the module
     

4. Test Execution

   Test discovery is handled by test builders and a test runner app. Test builders are subclasses of AbstractTestBuilder. For every test type (unit, functional, regression, custom) there is a corresponding test builder. AnyTestBuilder is a special builder which scans for unit, regression and functional tests at the same time.

   Test builder classes inherit the following test discovery methods:

   • loadTests - load tests from a test namespace. Wildcard namespaces are handled by loadAllTests
   • loadAllTests - load tests from the given namespace, and from all sub-packages (recursive)
   • loadFromFile - load tests from an absolute file name
   • loadMultipleTests - calls loadTests for a list of namespaces and combines all loaded tests in a single suite

   Each of those return test suite objects, which can be directly executed with python's unittest runner.

   Much simpler way to execute a test suite is to use our test app (csb/test/app.py), which is simply an instance of csb.test.Console:

      $ python csb/test/app.py --help
   

   The app has two main arguments:

   • test type - tells the app which TestBuilder to use for test dicsovery ("any" triggers AnyTestBuilder, "unit" - UnitTestBuilder, etc.)
   • test namespaces - a list of "dotted" test modules, for example:

         csb.test.cases.bio.io.*   # io and sub-packages
         csb.test.cases.bio.utils  # only utils
         .                         # current module
     

   In addition to running the app from the command line, you can run it also programmatically by instantiating csb.test.Console. You can construct a test console object by passing a list of test namespace(s) and a test builder class to the Console's constructor.

5. Commit Policies

   Follow these guidelines when making changes to the repository:

   • no bugs in "trunk": after fixing a bug or implementing a new feature, make sure at least the default test set passes by running the test console without any arguments. This is equivalent to: app.py -t any "csb.test.cases.*". (If no test case from this set covers the affected code, create a test case first, as described in the other policies)
   • no recurrent issues: when a bug is found, first write a regression test with a proper "@see: BugID" tag in the docstring. Run the test to make sure it fails. After fixing the bug, run the test again before you commit, as required by the previous policy
   • test all new features: there should be a test case for every new feature we implement. One possible approach is to write a test case first and make sure it fails; when the new feature is ready, run the test again to make sure it passes